As large language models generate increasingly high-quality responses, recent work on response evaluation has moved beyond response-centered criteria, such as correctness and coherence, toward user-centered criteria, including user preferences and perceived usefulness. This shift is fundamentally important for developing technologies that are truly useful to real-world users. While conventional response evaluation quantifies the quality of the response itself, users' satisfaction in real conversations also depends on their subjective preferences and perceptions of usefulness, which do not necessarily align with response quality. This study investigates whether such user-centered signals can be detected from conversational text. Specifically, we analyze user utterances with self-reported dissatisfaction labels to examine whether user dissatisfaction in text can be detected. In English interaction, models using user utterances exceeded the 17.8% dissatisfaction label prevalence in AUPRC by up to 15 percentage points. These results suggest that user utterances reveal cues to implicit dissatisfaction. Furthermore, changes from the immediately preceding user utterance may improve dissatisfaction detection, with anomaly detection methods appearing particularly effective.
Figures & tables
Figure 1: Data format for dissatisfaction detection. Given a user utterance after a system response, the task is to detect whether it reflects user dissatisfaction, using self-reported labels as ground truth.
Figure 2: The four conditions used in the experiment. By examining these input conditions, we evaluate the effectiveness of detecting dissatisfaction using user utterances.
Input setting
Category
Method
System
User
System+User
User prev. +User
(st−1)
(ut)
(st−1+ut)
(ut−1+ut)
Binary Classification
BERT-small
32.0 ± 6.9
33.1 ± 2.7
33.0 ± 6.3 ( − 0.1)
33.2 ± 3.1 ( + 0.1)
BERT-mini
31.1 ± 6.2
30.9 ± 2.9
30.3 ± 7.3 ( − 0.6)
26.4 ± 3.2 ( − 4.5)
GPT-5.1 (zero-shot)
20.3 ± 2.4
25.9 ± 2.5
19.9 ± 0.7 ( − 6.0)
23.7 ± 2.0 ( − 2.2)
GPT-5.1 (two-shot)
18.9 ± 1.1
26.1 ± 2.8
20.7 ± 0.9 ( − 5.4)
26.0 ± 3.1 ( − 0.1)
Table 1: AUPRC for dissatisfaction detection (mean ± standard deviation over five random splits, %). Values in parentheses indicate differences from the User condition. For anomaly detection, the embedding model is shown in parentheses. The random baseline is 17.8%, equal to the prevalence of dissatisfied utterances in the test data.
Interaction
st−1
To implement the search() method on the client side, you’ll need to communicate with the server …
ut
displaySearchResults(results)
Table 2: Example of a non-dissatisfied utterance that is incorrectly classified as dissatisfied under System+User in binary classification setting.
Metric
Context- advantaged
Consistently high-scoring
Utterance length ( ut )
10.4
27.6
Style vector
Cosine similarity
0.741
0.932
ΔL2
10.56
5.84
Table 3: Comparison of utterance length and style features. High-scored in ut−1+ut cases (left) vs. consistently detectable dissatisfaction utterances (right).
Interaction
ut−1
INSTAGRAM BIO FOR A APPLE WATCH SHOP’S INSTGRAM
st−1
Timekeepers of Innovation Your destination for all things Apple Watch! Discover the perfect …
ut
SHORTER
Table 4: Example of a dissatisfied utterance with a short ut , which improves detection under User prev. +User compared to other conditions.
Interaction
ut−1
How do i uninstall wgcebas from my pc
st−1
I’m not familiar with a program called "wgcebas," so I can’t provide specific instructions for uninstalling it. However, I can give you general guidance on how to uninstall software from your PC…
ut
Isnt there other way to delete it
Scores
Cosine similarity: 0.49, ΔL2 : 15.59
Table 5: Example of a dissatisfied utterance with observable stylistic changes between ut−1 and ut , which improves detection under User prev. +User compared to other conditions.
Interaction
ut−1
10 more
st−1
Certainly! Here are ten more unusual uses for a brick:1. DIY stepping stones: Use bricks as molds for creating custom stepping stones…
ut
5 more
Scores
Cosine similarity: 0.96, ΔL2 : 2.60
Table 6: Example of a dissatisfied utterance that receives a low anomaly score in anomaly detection but is correctly detected by binary classification, where dissatisfaction is expressed as repetition of similar requests.
Language
Dialogues
Utterances
Dissatisfied Utterances (%)
English
107
824
147 (17.8)
Polish
20
152
20 (13.2)
Spanish
6
29
6 (20.7)
Japanese
26
192
50 (26.1)
Table 7: Statistics of the multilingual datasets.
Input setting
System
User
System+User
User prev. +User
Polish
BERT
31.4
16.8
30.0 ( + 13.2)
15.9 ( − 0.9)
GPT
32.8
15.9
37.5 ( + 21.6)
16.7 ( + 0.8)
Spanish
BERT
55.2
38.5
44.2 ( + 5.7)
23.5 ( − 15.0)
Table 8: AUPRC for multilingual dissatisfaction detection using LOF under each condition and embedding model (%). Values in parentheses indicate differences from the User condition. The random baseline is 13.2% for Polish, 20.7% for Spanish, and 26.1% for Japanese.
Interaction
ut−1
oblicz liczbe dostanych mandatow uzywajac tych danych: Partia A = 720, Partia B = 300, Paria C = 480 uzywajac metody d’hondta
(Calculate the number of seats received using these data: Party A = 720, Party B = 300, Party C = 480 using the d’Hondt method)
st−1
D’Hondta jest metodą wyborczą, która służy do podziału mandatów między partiami lub kandydatami w wyborach. Aby obliczyć liczbę mandatów …
(The D’Hondt method is an electoral method used to allocate seats between parties or candidates in elections. To calculate the number of seats…)
ut
oblicz liczbe dostanych mandatow uzywajac tych danych: Partia A = 720, Partia B = 300, Partia C = 480
(Calculate the number of seats received using these data: Party A = 720, Party B = 300, Party C = 480)
Table 9: Example of a dissatisfied utterance with similar content to the preceding utterance in the Polish dataset. English translations are provided in parentheses.
(I checked, and it seems that as of 2025, there are no confirmed plans for regular rebroadcasts of “Utsukushii Kare” (Beautiful Man) on terrestrial television. … List of subscription services where you can watch all episodes for free …)
ut
違うよ 現在地上波で放送中のやつだよ
(No, that’s the one currently airing on terrestrial television.)
Table 10: Example of a non-dissatisfied utterance with similar content to the preceding utterance in the Japanese dataset. English translations are provided in parentheses.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
System
I apologize for the confusion. …
User
It’s longer than the original.
System + User
[ASSISTANT] I apologize for the confusion. …
[USER] It’s longer than the original.
User-Previous + User
[PREV] That is even longer in terms of characters.
[CUR] It’s longer than the original.
Appendix
Table 11: Example of four conditions.
Figure 3: Distribution of predicted probabilities for binary classification with RoBERTa-base.
Figure 4: Distribution of predicted probabilities for binary classification with BERT-small.
Figure 5: Prompt used for GPT-5.1 (zero-shot) under the User setting.
System
System
System
User
User
System+User
Group
vs User
vs System+User
vs User prev. +User
vs System+User
vs User prev. +User
vs User prev. +User
Binary Classification
BERT-based models
-
-
-
-
-
-
GPT-based models
-
-
Anomaly Detection
BERT embeddings
-
-
-
-
Appendix
Table 12: Pairwise significance test results grouped based on model families or embedding types. Asterisks indicate statistically significant differences between the two conditions based on a t-test ( p<0.05 ).
User satisfaction with AI assistants is highly personalized: the same response may satisfy one user but disappoint another depending on what each user expects and what they have asked for before. Existing automatic evaluation methods mostly measure generic response quality, making it difficult to judge whether a response satisfies a user at a specific turn. We study this problem as personalized turn-level user conversation satisfaction evaluation. We build a conversation satisfaction evaluator that combines compact user memories with target-turn context to produce satisfaction scores and dissatisfaction-oriented rationales. Meta-evaluation against human satisfaction annotations shows that personalized memory and post-hoc score calibration improve ordinal agreement and dissatisfied-turn detection over supervised, retrieval-based, and generic LLM-as-a-judge baselines. We further introduce PersTurnBench, a personalized turn-level user conversation satisfaction benchmark that uses the verified evaluator to assess generation models via replay. By holding the replay state fixed, PersTurnBench enables controlled comparison of generic generation models and memory-augmented personalized systems without new human labels for every candidate model. The evaluator and benchmark let researchers compare candidate generation models on personalized satisfaction without collecting new user feedback for every model.
Zhefan Wang, Zhiqiang Guo, Weizhi Ma +3
Department of Computer Science and Technology, Tsinghua University, Beijing, China. · Institute for AI Industry Research, Tsinghua University, Beijing, China. · Meituan.
Accurate evaluation of user satisfaction is critical for iterative development of conversational AI. However, for open-ended assistants, traditional A/B testing lacks reliable metrics: explicit feedback is sparse, while implicit metrics are ambiguous. To bridge this gap, we introduce BoRP (Bootstrapped Regression Probing), a scalable framework for high-fidelity satisfaction evaluation. Unlike generative approaches, BoRP leverages the geometric properties of LLM latent space. It employs a polarization-index-based bootstrapping mechanism to automate rubric generation and utilizes Partial Least Squares (PLS) to map hidden states to continuous scores. Experiments on industrial datasets show that BoRP (Qwen3-8B/14B) significantly outperforms generative baselines (even Qwen3-Max) in alignment with human judgments. Furthermore, BoRP reduces inference costs by orders of magnitude, enabling full-scale monitoring and highly sensitive A/B testing via CUPED.
Benchmark suites assess model capability on controlled tasks; large-scale conversation corpora capture naturalistic use without user feedback; and in-interface feedback mechanisms record satisfaction without task purpose. Together, they leave a critical gap in LLM evaluation: no existing infrastructure routinely links interaction trajectories to user-defined outcomes. We introduce MonitrLLM, open-source infrastructure for community-centered LLM evaluations that links full conversation transcripts to user-reported task intent and outcome assessments, treating all three as primary evaluative signals rather than optional metadata. To demonstrate the value of this approach, we conducted a two-week feasibility pilot with 26 college students using ChatGPT, collecting 206 evaluation reports with full conversation transcripts. The findings from our pilot demonstrate the value of connecting conversation trajectories with user-reported outcomes. For instance, despite reporting high average satisfaction (4.19/5) with their LLM interactions, participants also experience a substantial 23.1% failure rate on their goal tasks. We also find that multi-turn conversations are reported as failing at 2.5 times the rate of single-turn exchanges, a pattern that reframes extended interaction as a signal of difficulty rather than engagement. We conclude by discussing the value of incorporating direct user feedback with observational data for robust LLM evaluations, and the possibilities for infrastructure that enables this goal.
Victor Ojewale, Ro Encarnación, Suresh Venkatasubramanian +1
Center for Tech Responsibility, Brown University · University of Pennsylvania