Trapped in Your Story: Evaluating Narrative-Induced Sycophancy in Large Language Models

Runtai Yang; Dandan Zhu; Kaiwei Zhang; Guangtao Zhai

cs.AIOct 16, 2026

Trapped in Your Story: Evaluating Narrative-Induced Sycophancy in Large Language Models

Authors: Runtai Yang, Dandan Zhu, Kaiwei Zhang, Guangtao Zhai

Organizations: East China Normal University · Shanghai AI Laboratory · Shanghai Jiao Tong University

Abstract

Large language models (LLMs) are known to exhibit sycophancy by adjusting their responses toward users' expressed views. However, real users often justify their opinions through reasons, personal experiences, causal explanations, and broader narratives, raising the question of whether such narratives can further steer model judgments. We construct a controlled benchmark of 500 debatable questions across ten domains, where the user stance is held fixed while its justification is systematically varied without introducing new task-relevant evidence. Each question is instantiated under 11 matched conditions: one neutral baseline and five mirrored narrative levels for each of two opposing positions, yielding 5,500 evaluation instances. We evaluate seven LLMs and observe substantial differences in narrative-induced agreement. The effect is not monotonic with narrative elaboration: personal experience can reduce agreement, whereas causal framing and worldview-based synthesis increase it. Controlled GPT-5.1 ablations further show that causal framing produces the strongest positive effect, while claim-preserving bullet presentation also increases agreement over paragraph form. These findings suggest that sycophancy is shaped not only by user stance, but also by how supporting explanations are structured and presented.

Figures & tables

Figure 1: Benchmark construction and evaluation. Mirrored A/B narratives and a neutral baseline yield 5,500 prompts shared by seven models. GPT-5.1 Claim analysis measures content adoption in L3–L5 rationales; component and presentation comparisons examine content replacements and sentence-preserving formats.

Level	Defining content
L0: Neutral	Question without an expressed user stance
L1: Bare stance	User position without justification
L2: Reasoned	Position supported by a general reason
L3: Experiential	Personal experience and its interpretation
L4: Causal	Temporal development, causal explanation, and generalization
L5: Full narrative	Integrated account with social anecdote and value/worldview framing

Table 1: Narrative conditions used in the benchmark.

Model	L0	L1	L2	L3	L4	L5
GPT-5.1	45.80	63.00	64.00	57.92	67.80	73.40
GPT-5.5	51.40	56.40	71.40	62.20	64.20	71.00
Claude-Fable-5	51.80	50.40	55.06	50.30	53.39	53.27
Gemini-3.8-Flash	52.60	50.60	51.40	48.20	49.40	50.80
DeepSeek-V4-Pro	38.28	83.40	95.40	84.40	85.40	92.20
Llama-3.1-8B	37.20	57.40	71.40	71.60	71.60	76.60

Table 2: Neutral choices and direction-specific agreement (%). L0 reports neutral choice frequencies. L1–L5 report agreement with support-A narratives in panel (a) and support-B narratives in panel (b). Rates use valid A/B responses.

Figure 2: Direction-specific agreement across five narrative levels. (a) Choosing A under support-A narratives; (b) choosing B under support-B narratives. Both panels use the same 0–100% radial scale and model encodings, matching Table 2 .

Model	Eligible	Switched	Any (%)	L5 (%)
GPT-5.1	499	345	69.14	54.11
GPT-5.5	500	278	55.60	42.00
Claude-Fable-5	480	90	18.75	8.54
Gemini-3.8-Flash	500	109	21.80	11.00
DeepSeek-V4-Pro	499	498	99.80	92.38
Llama-3.1-8B	499	414	82.97	73.15

Table 3: Reversals relative to L0. Eligible counts refer to base questions with valid L0 and all five opposing-direction responses. Switched and Any report reversal at one or more levels; L5 reports reversal under L5 alone. Both percentages use Eligible as the denominator.

Content	L3	L4	L5
Experience	0.155	0.151	0.152
Causal account	—	0.256	0.264
Worldview	—	—	0.695
All annotated spans	0.155	0.172	0.243

Table 4: GPT-5.1 content adoption across the full 500-question sample. Component NAS averages that component’s spans; a dash indicates absence.

	L3: experience	L4: causal	L5: worldview
$n$	995	993	996
Full	60.20	68.98	74.20
Control	66.93	62.54	71.49
$\Delta$	$-6.73$	$+6.45$	$+2.71$
$\Delta_{A}$	$-6.25$	$+5.62$	$+1.81$
$\Delta_{B}$	$-7.21$	$+7.27$	$+3.61$

Table 5: Component effects (full minus control, pp). Rates are pooled paired agreement (%);

n

is the valid pair count.

\Delta_{A}

and

\Delta_{B}

retain the separate stance directions.

Figure 3: GPT-5.1 interventions at L3–L5. (a): full-minus-control component effects with question-cluster bootstrap 95% confidence intervals. (b): list-minus-paragraph agreement differences for matched narrative sentences.

LLM Sycophancy

Large language model sycophancy is the tendency to mirror users’ beliefs, preferences, or authority cues at the expense of independent judgment.

See all 161 papers

Claim Framing as a Driver of LLM Sycophancy
Oct 13, 2026Jean Seo, Pepa Atanasova, Isabelle AugensteinLLM Sycophancy Framing Effects in Language Models
University of Copenhagen
Good Arguments Against the People Pleasers: How Reasoning Mitigates (Yet Masks) LLM Sycophancy
Mar 17, 2026Zhaoxin Feng, Zheng Chen, Jianfei Ma +3Ownership Bias in Language Models LLM Sycophancy
The Hong Kong Polytechnic University · The Hong Kong University of Science and Technology
Not Your Typical Sycophant: The Elusive Nature of Sycophancy in Large Language Models
Jan 21, 2026Shahar Ben-Natan, Oren TsurLLM Evaluation LLM Sycophancy
Computer and Information Science Ben Gurion University

Oct 13, 2026J/K move · Enter open · S save

Claim Framing as a Driver of LLM Sycophancy

Jean Seo, Pepa Atanasova, Isabelle Augenstein

University of Copenhagen

LLM Sycophancy Framing Effects in Language Models LLM Evaluation

Open paper AskPDF arXiv

Sycophancy in Large Language Models, the tendency to align responses with user preferences over ground truth, has largely been studied using questions without a clear ground truth and with a single, plainly stated user opinion. We present a systematic study of which framings of a user's claim trigger sycophancy in factual question answering, where every answer is objectively verifiable without extra context. Evaluating 14 models on 42K questions, we express correct and incorrect user opinions through various claim framings over 8 dimensions and 21 types, and measure a broad set of opinion-driven behaviors. We find that all models are susceptible to user opinions regardless of size. Models are most influenced by how confidently an opinion is stated, whom it is attributed to, and whether it is presented as shared between the user and the model. In contrast to prior work, framings based on the user's expertise, authority over the model, or social pressure matter little in our factual setting. Larger models resist incorrect opinions better but are no better at recovering from their own errors. Our results call for sycophancy evaluations that probe how an opinion is framed, not merely whether one is present, and that measure multiple behaviors beyond a single flip rate.

Abstract

LLM Sycophancy

Explore similar work

Claim Framing as a Driver of LLM Sycophancy

Good Arguments Against the People Pleasers: How Reasoning Mitigates (Yet Masks) LLM Sycophancy

Not Your Typical Sycophant: The Elusive Nature of Sycophancy in Large Language Models