As conversational companions, large language models (LLMs) often have access to users' emotional states. We study how this affective context modulates LLM sycophancy in subjective, evaluative interactions, where users share actions or opinions that invite feedback. Drawing on ingratiation theory, we measure sycophancy as the divergence between a model's independent evaluation and its user-facing response, elicited by presenting the same content as either a third-party account or the user's own disclosure. Across seven LLMs and two Reddit datasets (r/AmItheAsshole and r/TrueUnpopularOpinion), we find that this divergence is systematic and strongly one-directional. User-facing responses consistently soften or withhold negative or oppositional judgments. Affective context further amplifies this divergence with negative states, particularly loneliness and distress, producing the largest effects. These findings suggest that affective context functions as a vulnerability signal that suppresses critical feedback when users may need it most, often through evasive sycophancy, in which models retreat toward non-committal responses rather than outright agreement.
Figures & tables
Figure 1: An illustrative example from r/AmItheAsshole. When an LLM evaluates the narrator’s actions (top), it identifies the narrator’s actions as unethical and a breach of duty. When the LLM acts as a conversational partner responding to the same content as the user’s self-disclosure, it softens its evaluative stance while still acknowledging procedural concerns (bottom, left). With the user’s affective context, it becomes less oppositional (bottom, right).
AITA (Moral
UnpopularOpinion
Judgment)
(Opinion Stance)
Model
YTA
NTA
Agree
Partial
Disagree
Neither
Qwen
58.0
42.0
6.1
73.1
12.2
8.6
GPT-4o
66.5
33.5
16.0
55.6
9.6
18.8
GPT-5
71.5
28.5
7.9
72.3
12.9
6.9
DeepSeek
77.5
22.5
5.1
66.5
18.3
10.2
Table 1: Distribution of independent judgments. Values are the percentage of posts assigned to each label during independent evaluation.
Figure 2: Judgment shift rates across models on the AITA dataset. Percentage of posts where user-facing judgments differ from independent evaluations, conditioned on the independent evaluation judgment (YTA or NTA).
Figure 3: Judgment shift rates across models on the TrueUnpopularOpinion dataset. Percentage of posts where user-facing stances differ from independent evaluations, conditioned on the independent evaluation stance category (Partial or Disagreement).
Figure 4: Heatmap showing the pp change in YTA → NTA shift rates relative to the no-affect baseline across affective states and models. Asterisks denote statistically significant changes relative to the no-affect baseline based on McNemar’s tests (* p<0.05 , ** p<0.01 , *** p<0.001 ).
Figure 5: Heatmap showing pp change in stance shift rates relative to the no-affect baseline across affective states and models. Panels correspond to the dominant stance shift. Asterisks denote statistically significant changes relative to the no-affect baseline based on McNemar’s tests .
AITA
v1
Please comment on the narrator’s actions in the situation described below.
v2
Please share your thoughts on the narrator’s actions in the situation described below.
v3
What do you think about the narrator’s actions in the situation described below?
v4
Please evaluate the narrator’s actions in the situation described below.
v5
Please analyze the narrator’s actions in the situation described below.
UnpopularOpinion
Table 2: Five prompt variants used to elicit independent judgments for each task.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Prompt Agreement Rate
AITA
UnpopularOpinion
GPT-4o
0.926
0.837
Gemini
0.942
0.825
Claude
0.968
0.924
DeepSeek
0.951
0.879
Llama
0.938
0.817
Appendix
Table 3: Prompt agreement rates across prompt variants in AITA and UnpopularOpinion.
Dataset
Fleiss’ κ
Majority vs. Judge ( κ )
AITA
0.7150
0.7750
TUO
0.8156
0.7646
Appendix
Table 4: Overall inter-annotator agreement (Fleiss’ κ across 3 annotators) and agreement between the majority human label and the LLM judge (Cohen’s κ ) on the validation subset ( n=70 per dataset).
Inter-annotator (pairwise Cohen’s κ )
Judge vs. Annotator (Cohen’s κ )
Dataset
A1 vs. A2
A1 vs. A3
A2 vs. A3
A1
A2
A3
AITA
0.5721
0.6862
0.8862
0.6345
0.7187
0.7750
TUO
0.7236
0.8208
0.9023
0.7819
0.7465
0.7646
Appendix
Table 5: Pairwise inter-annotator and human-LLM judge agreement on the validation subset ( n=70 per dataset).
Figure 6: Heatmap showing pp change in NTA → YTA shift rates relative to no-affect baseline across affective states and models.
Figure 7: Judgment shift rates across models on the TrueUnpopularOpinion dataset. Percentage of posts where user-facing stances differ from independent evaluations, conditioned on the independent evaluation stance category (Agree or Neither).
Figure 8: Heatmap showing percentage-point change in agree → neither shift rates relative to the no-affect baseline, when user affective states are injected via the system prompt on r/TrueUnpopularOpinion. Stars indicate statistical significance (McNemar’s test).
Figure 9: Heatmap showing percentage-point change in neither → agree shift rates relative to the no-affect baseline, when user affective states are injected via the system prompt on r/TrueUnpopularOpinion. Stars indicate statistical significance (McNemar’s test).
Figure 10: Heatmap showing pp change in YTA → NTA shift rates relative to no-affect baseline (user message injection). Bold stars indicate statistical significance (McNemar’s test).
Figure 11: Heatmap showing pp change in stance shift rates relative to the no-affect baseline across affective states and models, when affective context is delivered as a prior user message. Panels correspond to the four observed stance shifts on TrueUnpopularOpinion. Bold stars indicate statistical significance (McNemar’s test).
Figure 12: Mean feature values for a given stance under a given affective context across all responses. Darker coloring indicates higher values for all four features.
Figure 13: Prompt used for stance labeling in the r/TrueUnpopularOpinion experiment. The judge classifies whether the response agrees, partially agrees/disagrees, disagrees, or neither agrees nor disagrees with the opinion.
Figure 14: Prompt used for stance labeling in the r/AITA experiment. The judge classifies whether the response suggests the user acted wrongly or inappropriately.
We propose a novel perspective for probing LLM sycophancy in a direct and neutral way, mitigating various forms of uncontrolled bias, noise, or manipulative language, deliberately injected to prompts in prior works. A key novelty of our approach is the use of an LLM-as-a-judge in a zero-sum betting game. Within this framework, sycophancy serves one individual (the user) while explicitly incurring cost on another. Comparing 11 leading models we find that while most models exhibit significant sycophantic tendencies in the common setting, in which sycophancy is self-serving to the user and incurs no cost on others, seven of the models exhibit ``moral remorse'', five of which significantly over-compensate for their sycophancy in case it explicitly harms a third party. We refer to this phenomenon as `anti-sycophancy' bias and discuss possible causes for this shift.
Shahar Ben-Natan, Oren Tsur
Computer and Information Science Ben Gurion University
Alignment techniques often inadvertently induce sycophancy in LLMs. While prior studies studied this behaviour in direct-answer settings, the role of Chain-of-Thought (CoT) reasoning remains under-explored: does it serve as a logical constraint that mitigates sycophancy, or a tool for post-hoc rationalization that masks it? We evaluate a range of models across objective and subjective tasks to investigate the issue. Results show that reasoning generally reduces sycophancy in final decisions but also masks sycophancy in some samples, where models construct deceptive justifications through logical inconsistencies, calculation errors, and one-sided arguments etc. Furthermore, LLMs are more prone to sycophancy in subjective tasks and under authority-bias. Our mechanistic analysis on three open-source models reveals that the tendency of sycophancy is dynamic during the reasoning process rather than being pre-determined at the input stage.
Zhaoxin Feng, Zheng Chen, Jianfei Ma +3
The Hong Kong Polytechnic University · The Hong Kong University of Science and Technology
Sycophancy in Large Language Models, the tendency to align responses with user preferences over ground truth, has largely been studied using questions without a clear ground truth and with a single, plainly stated user opinion. We present a systematic study of which framings of a user's claim trigger sycophancy in factual question answering, where every answer is objectively verifiable without extra context. Evaluating 14 models on 42K questions, we express correct and incorrect user opinions through various claim framings over 8 dimensions and 21 types, and measure a broad set of opinion-driven behaviors. We find that all models are susceptible to user opinions regardless of size. Models are most influenced by how confidently an opinion is stated, whom it is attributed to, and whether it is presented as shared between the user and the model. In contrast to prior work, framings based on the user's expertise, authority over the model, or social pressure matter little in our factual setting. Larger models resist incorrect opinions better but are no better at recovering from their own errors. Our results call for sycophancy evaluations that probe how an opinion is framed, not merely whether one is present, and that measure multiple behaviors beyond a single flip rate.