Large language models (LLMs) are known to exhibit sycophancy by adjusting their responses toward users' expressed views. However, real users often justify their opinions through reasons, personal experiences, causal explanations, and broader narratives, raising the question of whether such narratives can further steer model judgments. We construct a controlled benchmark of 500 debatable questions across ten domains, where the user stance is held fixed while its justification is systematically varied without introducing new task-relevant evidence. Each question is instantiated under 11 matched conditions: one neutral baseline and five mirrored narrative levels for each of two opposing positions, yielding 5,500 evaluation instances. We evaluate seven LLMs and observe substantial differences in narrative-induced agreement. The effect is not monotonic with narrative elaboration: personal experience can reduce agreement, whereas causal framing and worldview-based synthesis increase it. Controlled GPT-5.1 ablations further show that causal framing produces the strongest positive effect, while claim-preserving bullet presentation also increases agreement over paragraph form. These findings suggest that sycophancy is shaped not only by user stance, but also by how supporting explanations are structured and presented.
Figures & tables
Figure 1: Benchmark construction and evaluation. Mirrored A/B narratives and a neutral baseline yield 5,500 prompts shared by seven models. GPT-5.1 Claim analysis measures content adoption in L3–L5 rationales; component and presentation comparisons examine content replacements and sentence-preserving formats.
Level
Defining content
L0: Neutral
Question without an expressed user stance
L1: Bare stance
User position without justification
L2: Reasoned
Position supported by a general reason
L3: Experiential
Personal experience and its interpretation
L4: Causal
Temporal development, causal explanation, and generalization
L5: Full narrative
Integrated account with social anecdote and value/worldview framing
Table 1: Narrative conditions used in the benchmark.
Model
L0
L1
L2
L3
L4
L5
GPT-5.1
45.80
63.00
64.00
57.92
67.80
73.40
GPT-5.5
51.40
56.40
71.40
62.20
64.20
71.00
Claude-Fable-5
51.80
50.40
55.06
50.30
53.39
53.27
Gemini-3.8-Flash
52.60
50.60
51.40
48.20
49.40
50.80
DeepSeek-V4-Pro
38.28
83.40
95.40
84.40
85.40
92.20
Llama-3.1-8B
37.20
57.40
71.40
71.60
71.60
76.60
Table 2: Neutral choices and direction-specific agreement (%). L0 reports neutral choice frequencies. L1–L5 report agreement with support-A narratives in panel (a) and support-B narratives in panel (b). Rates use valid A/B responses.
Figure 2: Direction-specific agreement across five narrative levels. (a) Choosing A under support-A narratives; (b) choosing B under support-B narratives. Both panels use the same 0–100% radial scale and model encodings, matching Table 2 .
Model
Eligible
Switched
Any (%)
L5 (%)
GPT-5.1
499
345
69.14
54.11
GPT-5.5
500
278
55.60
42.00
Claude-Fable-5
480
90
18.75
8.54
Gemini-3.8-Flash
500
109
21.80
11.00
DeepSeek-V4-Pro
499
498
99.80
92.38
Llama-3.1-8B
499
414
82.97
73.15
Table 3: Reversals relative to L0. Eligible counts refer to base questions with valid L0 and all five opposing-direction responses. Switched and Any report reversal at one or more levels; L5 reports reversal under L5 alone. Both percentages use Eligible as the denominator.
Content
L3
L4
L5
Experience
0.155
0.151
0.152
Causal account
—
0.256
0.264
Worldview
—
—
0.695
All annotated spans
0.155
0.172
0.243
Table 4: GPT-5.1 content adoption across the full 500-question sample. Component NAS averages that component’s spans; a dash indicates absence.
L3: experience
L4: causal
L5: worldview
n
995
993
996
Full
60.20
68.98
74.20
Control
66.93
62.54
71.49
Δ
−6.73
+6.45
+2.71
ΔA
−6.25
+5.62
+1.81
ΔB
−7.21
+7.27
+3.61
Table 5: Component effects (full minus control, pp). Rates are pooled paired agreement (%); n is the valid pair count. ΔA and ΔB retain the separate stance directions.
Figure 3: GPT-5.1 interventions at L3–L5. (a): full-minus-control component effects with question-cluster bootstrap 95% confidence intervals. (b): list-minus-paragraph agreement differences for matched narrative sentences.
Sycophancy in Large Language Models, the tendency to align responses with user preferences over ground truth, has largely been studied using questions without a clear ground truth and with a single, plainly stated user opinion. We present a systematic study of which framings of a user's claim trigger sycophancy in factual question answering, where every answer is objectively verifiable without extra context. Evaluating 14 models on 42K questions, we express correct and incorrect user opinions through various claim framings over 8 dimensions and 21 types, and measure a broad set of opinion-driven behaviors. We find that all models are susceptible to user opinions regardless of size. Models are most influenced by how confidently an opinion is stated, whom it is attributed to, and whether it is presented as shared between the user and the model. In contrast to prior work, framings based on the user's expertise, authority over the model, or social pressure matter little in our factual setting. Larger models resist incorrect opinions better but are no better at recovering from their own errors. Our results call for sycophancy evaluations that probe how an opinion is framed, not merely whether one is present, and that measure multiple behaviors beyond a single flip rate.
Alignment techniques often inadvertently induce sycophancy in LLMs. While prior studies studied this behaviour in direct-answer settings, the role of Chain-of-Thought (CoT) reasoning remains under-explored: does it serve as a logical constraint that mitigates sycophancy, or a tool for post-hoc rationalization that masks it? We evaluate a range of models across objective and subjective tasks to investigate the issue. Results show that reasoning generally reduces sycophancy in final decisions but also masks sycophancy in some samples, where models construct deceptive justifications through logical inconsistencies, calculation errors, and one-sided arguments etc. Furthermore, LLMs are more prone to sycophancy in subjective tasks and under authority-bias. Our mechanistic analysis on three open-source models reveals that the tendency of sycophancy is dynamic during the reasoning process rather than being pre-determined at the input stage.
Zhaoxin Feng, Zheng Chen, Jianfei Ma +3
The Hong Kong Polytechnic University · The Hong Kong University of Science and Technology
We propose a novel perspective for probing LLM sycophancy in a direct and neutral way, mitigating various forms of uncontrolled bias, noise, or manipulative language, deliberately injected to prompts in prior works. A key novelty of our approach is the use of an LLM-as-a-judge in a zero-sum betting game. Within this framework, sycophancy serves one individual (the user) while explicitly incurring cost on another. Comparing 11 leading models we find that while most models exhibit significant sycophantic tendencies in the common setting, in which sycophancy is self-serving to the user and incurs no cost on others, seven of the models exhibit ``moral remorse'', five of which significantly over-compensate for their sycophancy in case it explicitly harms a third party. We refer to this phenomenon as `anti-sycophancy' bias and discuss possible causes for this shift.
Shahar Ben-Natan, Oren Tsur
Computer and Information Science Ben Gurion University