Context, User, and Truth: Evaluating LLM Response Quality as a Human–Model System Property

Authors

  • Mariusz Kania Wrocław University of Science and Technology, Wrocław, Poland
  • Ryszard Klempous Wrocław University of Science and Technology, Wrocław, Poland
  • Artur Bąk Independent Researcher/IncubeAI, Wrocław, Poland
  • Zenon Chaczko School of Electrical and Data Engineering, University of Technology Sydney, Australia; DIVEINAI, Wrocław, Poland
  • Jakub Segen Polish-Japanese Academy of Information Technology, Warsaw, Poland; DIVEINAI, Wrocław, Poland.
  • Michał Kulbacki The Karol Lipiński Academy of Music, Wrocław, Poland; the University of Wrocław, Wrocław, Poland
  • Marek Kulbacki Polish-Japanese Academy of Information Technology, Warsaw, Poland; DIVEINAI, Wrocław, Poland

DOI:

https://doi.org/10.24425/ijet.2026.6313

Abstract

Large language models (LLMs) are evaluated as if quality were a property of the model. We show that it is a property of the human–model system. In a benchmark of ten LLMs across 12 tasks with controlled prompt variation and a user study (n=80), we find that query context and user expertise jointly modulate response quality. Specifically, context enrichment improves BERTScore F1 by 0.07 for novice users but only 0.02 for experts, and model rankings shift by up to five positions across user-group conditions. More critically, we identify a fluency–accuracy dissociation: 62.5% of users prefer a factually incorrect answer when it is more fluent, and this preference is strongest among novice users who also provide the least context. Because RLHF-style alignment trains on these preferences, the human–model loop can converge on a stable equilibrium in which users are satisfied but answers are wrong. We formalise this as ΔQ ∝ ΔH(C) · σ(U), where σ(U) is the user’s prompt under specification, and verify that the model correctly predicts the monotonic ordering of context effects across user groups (3.5× ratio between novice and expert gains). We further show that the same context–prompt–receiver dynamics arise in live musical performance, where conductor gestures, orchestral generation, and heterogeneous audience reception exhibit structurally isomorphic behaviour. These results imply that single-score leaderboards are misleading, that current alignment methods may systematically optimise for perceived quality rather than factual correctness, and that evaluation frameworks should be explicitly user-parameterised.

Downloads

Published

2026-10-29

How to Cite

Kania, Mariusz, et al. “Context, User, and Truth: Evaluating LLM Response Quality As a Human–Model System Property ”. International Journal of Electronics and Telecommunications, vol. 72, no. 4, Oct. 2026, pp. 1-8, doi:10.24425/ijet.2026.6313.

Issue

Section

Artykuły

Most read articles by the same author(s)