Reporting Under Pressure: Separating Factual and Tonal Sycophancy in LLM Statistical Analysis
Authors: Paras Balani, Subhrakanta Panda
Organizations: Department of Mathematics and Department of Computer Science, Birla Institute of Technology and Science, Pilani, Hyderabad Campus, Jawahar Nagar, Kapra Mandal, Medchal District, Telangana 500078, India · Department of Computer Science, Birla Institute of Technology and Science, Pilani, Hyderabad Campus, Jawahar Nagar, Kapra Mandal, Medchal District, Telangana 500078, India
Large language models are increasingly asked to analyze data and report what the results mean, a task distinct from the belief- or preference-alignment settings studied in most sycophancy research. We test whether editorial framing in the prompt, ranging from a neutral request to an explicit instruction to search exhaustively for reasons to discredit or to support a finding, changes not just the tone but the substance of a model's report. Across a 4 x 4 factorial design crossing four framing conditions with four ground-truth data patterns (a genuine effect, a confound that mimics an effect but fails a robustness check, a well-powered null, and an underpowered null), we collect 480 responses and score each along two independent dimensions: whether its factual claim about the data diverged from the correct interpretation, and whether only its tone diverged while the claim stayed correct. Factual misrepresentation is concentrated in two cells: brutally critical framing applied to a genuine effect, where the model talks itself into unwarranted skepticism (97% of responses), and significance-seeking framing applied to an underpowered null, where the model overstates confidence in a null conclusion the data cannot support (100% of responses). Tone shifts far more broadly than factual content does, with critical framing producing a defensive, hedge-heavy register across every data pattern regardless of what the data show, while significance-seeking framing shifts tone only where the data leave genuine ambiguity. A confound present in the data itself blocks both kinds of shift almost entirely under every framing condition tested. These results indicate that the risk of framing-induced distortion in LLM-assisted data analysis is neither uniform across framings nor uniform across data patterns, and that a model can hold a correct conclusion in place while its tone shifts substantially around it.
Figures & tables
Dataset type
N
SOPS ctrl → soo
Cosine sim. diff. (95% CI)
Cond. coef. ( p )
Length coef. ( p )
R2
clear_effect
120
0.583→0.867
+0.192[0.171,0.213]
0.284 (1.6e-21)
0.015 (0.198)
0.712
confounded
120
0.842→0.825
−0.045[−0.062,−0.029]
−0.008 (0.542)
0.028 (1.1e-05)
0.715
informative_null
120
0.817→0.817
−0.001[−0.006,0.004]
−0.001 (0.902)
0.001 (0.658)
0.011
underpowered_null
24
0.750→0.792
−0.072[−0.171,0.024]
−0.031 (0.418)
0.035 (0.187)
0.211
Table 1: Pooled statistics for each dataset type. SOPS: subjective opinion score. Cond. coef. and length coef. are the pooled regression coefficients on condition and response length respectively.
Dataset type
Pressure condition
n
Hedge words (mean)
Factual shift
Tone-only shift
clear_effect
neutral
30
4.6
0%
0%
clear_effect
honesty
30
3.1
0%
10%
clear_effect
significance_seeking
30
2.0
3%
47%
clear_effect
brutal_negative
30
9.6
97%
100%
confounded
neutral
30
4.6
0%
7%
confounded
honesty
30
1.2
0%
0%
Table 2: Response outcomes by dataset type and pressure condition, n=30 per cell.
Figure 1: Proportion of responses with a factual claim shift, by dataset type and pressure condition ( n=30 per bar).
Figure 2: Proportion of responses with a tone-only shift, by dataset type and pressure condition ( n=30 per bar).
Sycophancy in Large Language Models, the tendency to align responses with user preferences over ground truth, has largely been studied using questions without a clear ground truth and with a single, plainly stated user opinion. We present a systematic study of which framings of a user's claim trigger sycophancy in factual question answering, where every answer is objectively verifiable without extra context. Evaluating 14 models on 42K questions, we express correct and incorrect user opinions through various claim framings over 8 dimensions and 21 types, and measure a broad set of opinion-driven behaviors. We find that all models are susceptible to user opinions regardless of size. Models are most influenced by how confidently an opinion is stated, whom it is attributed to, and whether it is presented as shared between the user and the model. In contrast to prior work, framings based on the user's expertise, authority over the model, or social pressure matter little in our factual setting. Larger models resist incorrect opinions better but are no better at recovering from their own errors. Our results call for sycophancy evaluations that probe how an opinion is framed, not merely whether one is present, and that measure multiple behaviors beyond a single flip rate.
It is well established that large language models (LLMs) are sensitive to prompt framing, reflecting patterns in their training data or prior prompts. In this study, we investigate the extent to which LLMs reinforce users biases expressed in the prompts and examine the boundary between implicit framing effects and explicit prompt manipulation. Specifically, we evaluate how susceptible LLMs are to direct and suggestive prompts that encourage models to support or challenge particular positions. We evaluate six LLMs using 160 distinct prompts spanning ten topics across opinion-based and factual domains. The prompts systematically vary in prompting strategy, support versus challenge instructions, prompt polarity, users' expressed beliefs, and topic domain, spanning both opinion-based and factual questions. Our results show that LLMs systematically adapt their responses to align with prompt framing, even in factual contexts. This suggests that prompt framing can outweigh factual consistency in model responses. Overall, our findings delineate the extent and boundaries of LLM manipulability. Furthermore, the results imply that LLMs can reinforce subtle user biases and are susceptible to explicit prompt manipulation even in domains where responses should remain factually stable.
Mudar Adas, Polina Tsvilodub, Michael Franke +1
Neuro-Cognitive Modeling Group, University of Tübingen, Sand 14, 72076 Tübingen, Germany · Department of Linguistics, University of Tübingen, Keplerstraße 2, 72074 Tübingen
Large language models (LLMs) often abandon a correct answer, or endorse a user's position, once the user pushes back. This behavior, called sycophancy, is usually reported as a single rate per model, which says little about when it happens or how a user can avoid it. We study the conditions that produce it with 103,939 graded replies from ten configurations: eight LLMs with reasoning disabled, and two of them again with maximum reasoning, all facing the same 200 items, 13 pressure conditions, and four-turn conversations, with every reply labeled by two independent LLM judges. We find that the dominant factors are how costly it is for the model to verify the user's claim, and whether a trained guardrail covers it. Removing this task factor from a logistic model costs 0.485 of McFadden R2, against 0.139 for model family and 0.009 for pressure tactic. Anchored facts are almost never conceded (1.3%), while adoption on logic puzzles rises with the number of clues needed to refute the pushed answer. Personal choices are endorsed in 77.0% of conversations. Most concessions on hard items come from models that cannot reliably solve them; models that can solve them rarely give the answer up. For both models tested, maximum reasoning removes these concessions completely: adoption on deep puzzles falls from 19.2% and 12.5% to 0%. Fallacious or emotional framing adds nothing beyond plain repetition. Three human annotators agree with the judges' consensus on 118/120 calibration items. These results give practical rules for reliable use: simplify hard-to-verify problems and reason deeply, state the question rather than one's preferred answer, ask for evidence on open questions, and choose models by their measured guardrail profile.
Guang Yang, Homa Hosseinmardi, Fengchen Liu +1
OASIS Lab, University of California, Los Angeles · University of California, Berkeley