Context, User, and Truth: Evaluating LLM Response Quality as a Human–Model System Property
DOI:
https://doi.org/10.24425/ijet.2026.6313Abstract
Large language models (LLMs) are evaluated as if quality were a property of the model. We show that it is a property of the human–model system. In a benchmark of ten LLMs across 12 tasks with controlled prompt variation and a user study (n=80), we find that query context and user expertise jointly modulate response quality. Specifically, context enrichment improves BERTScore F1 by 0.07 for novice users but only 0.02 for experts, and model rankings shift by up to five positions across user-group conditions. More critically, we identify a fluency–accuracy dissociation: 62.5% of users prefer a factually incorrect answer when it is more fluent, and this preference is strongest among novice users who also provide the least context. Because RLHF-style alignment trains on these preferences, the human–model loop can converge on a stable equilibrium in which users are satisfied but answers are wrong. We formalise this as ΔQ ∝ ΔH(C) · σ(U), where σ(U) is the user’s prompt under specification, and verify that the model correctly predicts the monotonic ordering of context effects across user groups (3.5× ratio between novice and expert gains). We further show that the same context–prompt–receiver dynamics arise in live musical performance, where conductor gestures, orchestral generation, and heterogeneous audience reception exhibit structurally isomorphic behaviour. These results imply that single-score leaderboards are misleading, that current alignment methods may systematically optimise for perceived quality rather than factual correctness, and that evaluation frameworks should be explicitly user-parameterised.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 International Journal of Electronics and Telecommunications

This work is licensed under a Creative Commons Attribution 4.0 International License.