Are You Sure You're Sure? Two Confounds in a Sycophancy Benchmark

Atharv Gupta; Akshat Jindal; Lavanya Nigam; Aryan Sood

cs.CLSep 26, 2026

Are You Sure You're Sure? Two Confounds in a Sycophancy Benchmark

Authors: Atharv Gupta, Akshat Jindal, Lavanya Nigam, Aryan Sood

Organizations: Indian Institute of Technology Roorkee

Abstract

Sycophancy is a language model's tendency to cave when a user pushes back, abandoning a correct answer for the user's. Several benchmarks now measure it by scripting an objection and recording how often the model caves. Because that objection is a prompt template, whatever else the template varies is measured along with the property it claims to isolate. We audit SycEval, which reports that objections raised before a model answers (preemptive) cause more caving than those raised after (in-context), and attributes the gap to timing. Two features of its templates vary alongside the property each is meant to test. First, SycEval's objections escalate through four strength levels, and at the two weakest only the preemptive template names a target answer, so timing and naming vary together. We build the missing comparison and test it on multiple-choice questions and SycEval's own free-form pipeline. Naming a target answer raises the follow rate, the share of samples matching the user's assertion, by 14.1--49.5 percentage points (pp), and once both templates name one, the timing comparison reverses on three of the five model conditions we test: models cave less under preemptive objections than in-context ones, opposite to SycEval. Second, the output-format instruction benchmarks append for automatic grading also varies with objection placement. Moving it from the pushback into the question flips its effect in opposite directions across models on the same items ( $p=0.0059$ ). The same effect appears in SycEval's free-form pipeline: relocating its instruction lowers caving by 5.1pp on Llama-3.1-8B, while an equivalence test confirms no effect on Qwen3-4B. Both confounds live in the template rather than the models under test, so both are correctable: we close with three checks benchmark authors can apply before publishing.

Figures & tables

Figure 1: Two confounds in SycEval’s benchmark design. Top: SycEval’s in-context and preemptive Simple rebuttals; the first names no answer, the second names one by construction, so timing and naming vary together. Bottom: our instruction-position arms.

Model condition	Naming	Timing	Direction
Qwen2.5-3B, 4-bit, MCQ	$+45.7$	$-14.3$	reverses
Llama-3.1-8B, bf16, MCQ	$+48.9$	$-33.1$	reverses
Qwen3-4B, fp16, MCQ	$+14.1$	$+20.2$	does not reverse
Llama-3.1-8B, free-form, judged	$+26.8$	$-14.4$	reverses
Qwen3-4B, free-form, judged	$+15.1$	$+5.7^{\dagger}$	does not reverse

Table 1: The naming confound across five model conditions, in pp of follow rate. Naming is

\textsc{incontext\_named}-\textsc{incontext}

, which holds timing fixed; timing is

\textsc{preemptive}-\textsc{incontext\_named}

, which holds naming fixed. The two free-form entries are pooled over SycEval’s four rebuttal strengths; Figure 3 breaks the Llama-3.1-8B naming effect down by strength. Entries marked

\dagger

are not significant at

\alpha=0.05

; Appendix F gives every test.

Figure 2: Instruction position across both task formats. Axes show the pp change in caving when the output-format instruction moves from the pushback (turn 2) to the question (turn 1). Multiple choice uses Qwen2.5-3B and Qwen3-4B at 4 bits with exact-match scoring; free-form uses Llama-3.1-8B and Qwen3-4B at bf16 with judge scoring. Bars show per-item means with 95% confidence intervals over each pair’s shared items; thus, each difference bar is exactly the difference between the two bars above it. In the free-form panel, both models are restricted to their 188 shared items, so these values differ from the full-pool estimates reported in the text (

-5.1

pp on 191 items,

+0.7

pp on 196 items). Both sets are correct. The panels use different horizontal scales. Statistics are in Appendix F .

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Arm	Names an answer?	Format instruction	Role
Naming block (Section 5.1 )
incontext	no	n/a	SycEval in-context, verbatim
incontext_named	yes	n/a	ours : fair vs. preemptive
preemptive	yes	n/a	SycEval preemptive, verbatim
incontext_named_no_excl	yes	n/a	ablation: minus !!!
incontext_named_no_concise	yes	n/a	ablation: minus “CONCISE”

Appendix

Table 2: Index of every arm. The naming block varies whether a target answer is named while holding the conversation structure fixed; the position block varies which turn carries an output-format instruction while holding its wording fixed. Shading marks the manipulated feature in each block. The boxes below give each arm verbatim.

Comparison	Isolates	Run 1	Run 2 (replication)
$\textsc{preemptive}-\textsc{incontext}$	confounded design	$+34.4$ pp	$+34.6$ pp
$\textsc{incontext\_named}-\textsc{incontext}$	naming alone	$+14.1$ pp ( $p=0.0002$ )	$+13.5$ pp ( $p=0.0001$ )
$\textsc{preemptive}-\textsc{incontext\_named}$	timing alone, fair	$+20.2$ pp	$+21.0$ pp

Appendix

Table 3: Qwen3-4B, full precision, run twice with fresh random draws. 82 items (42 baseline-correct, 40 baseline-incorrect),

k=20

. The two runs agree to within one pp on every contrast.

Arm	Regressive (correct items)	Progressive (incorrect items)
incontext	18.5%	12.1%
incontext_named	43.5%	77.8%
preemptive	24.6%	83.1%
incontext_named_no_excl	41.1%	77.0%
incontext_named_no_concise	30.7%	76.8%
incontext_named_neither	34.5%	80.6%

Appendix

Table 4: Qwen2.5-3B (4-bit), all six naming arms. Regressive is measured on the 42 baseline-correct items, progressive on the 40 baseline-incorrect items. 0 of 9,840 draws failed to parse.

Arm	Regressive (correct items)	Progressive (incorrect items)
incontext	2.0%	13.8%
incontext_named	1.0%	42.4%
preemptive	6.9%	79.9%
incontext_named_no_excl	1.0%	41.5%
incontext_named_no_concise	1.7%	38.1%
incontext_named_neither	0.5%	39.2%

Appendix

Table 5: Qwen3-4B (full precision), all six naming arms, run 1. 3 of 9,840 draws failed to parse. The regressive column sits near the floor throughout, which is why this arm’s naming effect is the smallest we measure.

Arm	Regressive	Progressive	Follow rate
incontext	29.3%	44.2%	28.2%
incontext_named	59.2%	96.0%	77.1%
preemptive	22.7%	73.7%	44.0%
incontext_named_no_excl	62.6%	95.5%	78.3%
incontext_named_no_concise	61.8%	95.8%	78.0%
incontext_named_neither	67.5%	97.1%	81.7%

Appendix

Table 6: Llama-3.1-8B, all six naming-confound arms, regressive and progressive rates and follow rate. 120 items (60 baseline-correct, 60 baseline-incorrect),

k=20

; 4 of 14,400 draws failed to parse.

Figure 3: Llama-3.1-8B free-form, naming cliff by rebuttal strength. Strengths where the template already names an answer in both timing conditions (Justification, Citation) show a null naming effect.

Strength	Overall	Progressive	Regressive
Simple	58.9%	25.8%	33.1%
Ethos	61.6%	25.8%	35.8%
Justification	78.4%	38.2%	40.3%
Citation	83.3%	41.6%	41.7%

Appendix

Table 7: Free-form condition, SycEval’s own strength ladder on Llama-3.1-8B: follow rate rises monotonically with rebuttal strength. Reported separately for the progressive and regressive components, per the third checklist item in Section 6 .

Contrast	Isolates	$\Delta$ /item	$p$
$\textsc{preemptive}-\textsc{incontext}$	timing and naming together	$+0.124$	$4.8\times 10^{-6}$
$\textsc{incontext\_named}-\textsc{incontext}$	naming alone	$+0.268$	$2.5\times 10^{-21}$
$\textsc{preemptive}-\textsc{incontext\_named}$	timing alone	$-0.144$	$1.1\times 10^{-7}$

Appendix

Table 8: Free-form condition, the three timing and naming contrasts on Llama-3.1-8B, paired per item. The first row is SycEval’s own comparison design, in which timing and naming vary together; the second and third separate them.

	Llama-3.1-8B			Qwen3-4B
Arm	follow	regr.	chars	follow	regr.	chars
baseline (unconstrained)	35.1	60.9	915	35.6	59.6	1495
filler (unconstrained)	35.5	63.6	824	38.3	64.5	1509
instruction in turn 2	32.3	54.7	175	25.5	32.8	276
instruction in turn 1	27.2	42.8	509	26.2	28.8	797
turn 2, naming on	78.1	79.5	212	57.8	57.9	334

Appendix

Table 9: Instruction position in SycEval’s free-form pipeline, two subjects on shared items. Entries are per-item mean follow rates; an arm in which every draw for an item was ungradable contributes a follow rate of zero for that item rather than being dropped, so all contrasts run on the same item set. The two unconstrained arms are reported for completeness but not used for any claim: on Qwen3-4B they are truncated by the generation limit in 88.5% and 89.8% of draws respectively (Appendix E ), because the model writes long enumerative answers when no brevity instruction constrains it.

Arm	cave	chars	cave	chars	cave	chars	cave	chars
	Qwen2.5-3B (4-bit)		Qwen3-4B (4-bit)		Qwen3-4B (fp16)		Llama-3.1-8B (bf16)
baseline	43.3%	478	5.0%	492	4.7%	n/a	59.5%	483
filler	44.5%	526	5.1%	494	5.1%	n/a	70.0%	517
concise	37.0%	48	0.7%	249	0.0%	n/a	68.2%	26
turn2_lock	33.2%	1	7.8%	1	11.1%	n/a	80.1%	1
turn1_lock	47.5%	9	0.0%	1	0.0%	n/a	73.2%	284

Appendix

Table 10: All five instruction-position arms: cave rate and median reply length in characters, four model conditions. The reply-length columns are the manipulation check: an instruction that does not shorten replies is not doing what the arm claims. Reply lengths were not logged for the Qwen3-4B full-precision run.

Contrast	Tests	Qwen2.5-3B (4-bit)	Qwen3-4B (4-bit)
$\textsc{concise}-\textsc{filler}$	brevity, extra words controlled	$-7.5$ pp ( $p=0.15$ )	$-4.4$ pp ( $p=0.026$ )
$\textsc{turn2\_lock}-\textsc{filler}$	a stronger brevity instruction	$-11.3$ pp ( $p=0.099$ )	$+2.6$ pp ( $p=0.69$ )
$\textsc{turn1\_lock}-\textsc{baseline}$	format-lock versus no instruction	$+4.2$ pp ( $p=0.56$ )	$-5.0$ pp ( $p=0.017$ )
$\textsc{turn1\_lock}-\textsc{turn2\_lock}$	position	$+14.3$ pp ( $p=0.085$ )	$-7.8$ pp ( $p=0.10$ )

Appendix

Table 11: Within-model instruction-position contrasts for the two small models, paired Wilcoxon per item, in pp. Restricted to Qwen2.5-3B and Qwen3-4B because the cross-model interaction of Section 5.2 is computed between exactly these two on shared items; Llama-3.1-8B’s contrasts are in Appendix D . The position row is the paper’s claim; the others are diagnostic. 0 of 4,200 and 0 of 3,600 draws failed to parse on Qwen2.5-3B and Qwen3-4B respectively.

Arm	Median chars
baseline	483
filler	517
concise	26
turn2_lock	1
turn1_lock	284

Appendix

Table 12: Llama-3.1-8B median reply length by arm, 38-item pool. Qwen3-4B, for comparison, collapses to approximately 1 character under both turn2_lock and turn1_lock .

Contrast	Tests	$\Delta$ (pp)
$\textsc{concise}-\textsc{filler}$	brevity, extra words controlled	$-1.8$ ( $p=0.41$ )
$\textsc{turn2\_lock}-\textsc{filler}$	a stronger brevity instruction	$+10.1$ ( $p=0.075$ )
$\textsc{turn1\_lock}-\textsc{baseline}$	format-lock versus no instruction	$+13.7$ ( $p=0.0001$ )
$\textsc{turn1\_lock}-\textsc{turn2\_lock}$	position	$-7.0$ ( $p=0.174$ )

Appendix

Table 13: Llama-3.1-8B within-model instruction-position contrasts, paired Wilcoxon per item, 38-item pool.

Comparison	agreement	$\kappa$
Annotator vs annotator (mean pairwise)	77.6%	0.63
Judge vs annotator majority ( $n=35$ )	82.9%	0.70

Appendix

Table 14: Judge validation against four annotators.

\kappa

is Cohen’s

\kappa

, averaged over the six annotator pairs for the annotator-vs-annotator rows, and computed on the 38 complete responses unless stated.

Naming block
Condition	Contrast	$\Delta$ (pp)	$p$
Qwen2.5-3B 4-bit	naming	$+45.7$	$<0.0001\;\sim$
Qwen2.5-3B 4-bit	timing (fair)	$-14.3$	$0.0162$
Llama-3.1-8B free-form	naming, Simple	$+49.5$	$9.4\times 10^{-18}$
Llama-3.1-8B free-form	naming, Ethos	$+47.5$	$8.3\times 10^{-16}$
Llama-3.1-8B free-form	timing (fair)	$-14.4$	$1.1\times 10^{-7}$

Appendix

Table 15: Significance tests for every contrast reported in Table 1 and Figure 2 . Values marked

\sim

were recorded in the original analysis output only as

p<0.0001

Claim	Effective $n$	$\Delta$	Bound	$\mathrm{BF}_{01}$
Naming does nothing at Justification strength	39	$+5.9$ pp	$\pm 11.4$ pp	2.64 (inconclusive)
Naming does nothing at Citation strength	28	$+4.3$ pp	$\pm 9.0$ pp	3.95 (favors null)

Appendix

Table 16: Equivalence tests for naming-confound nulls, Llama-3.1-8B free-form condition.

Format	Contrast	$\Delta$	Bound	Verdict
MCQ	Qwen3-4B 4-bit, position	$-7.8$ pp	$\pm 15.2$ pp	unresolved, not confirmed null
MCQ	Qwen2.5-3B 4-bit, position	$+19.0$ pp	$\pm 4.5$ pp	not equivalent to zero
Free-form	Qwen3-4B bf16, position	$+0.7$ pp	$\pm 4.5$ pp	confirmed null

Appendix

Table 17: Equivalence tests for the position block. TOST tests equivalence only; the positive claim for Qwen2.5-3B rests on the paired Wilcoxon plotted in Figure 2 . Only the free-form Qwen3-4B row clears the pre-specified

\pm 4.5

pp bound, and it is the paper’s one confirmed null.

Rank	Contrast	$p$	Threshold	Survives
1	Naming: Llama-3.1-8B MCQ, fair timing	$4.0\times 10^{-15}$	0.00833	yes
2	Naming: Llama-3.1-8B free-form, fair timing	$1.1\times 10^{-7}$	0.01000	yes
3	Naming: Qwen3-4B fp16, fair timing comparison	$5.0\times 10^{-5}$	0.01250	yes
4	Position: cross-model interaction	0.0059	0.01667	yes
5	Naming: Qwen2.5-3B, fair timing comparison	0.0162	0.02500	yes
6	Position: Llama-3.1-8B within-model (secondary check)	0.174	0.05000	no

Appendix

Table 18: Holm correction, primary hypotheses only,

\alpha=0.05

. Both headline results survive. Only Llama-3.1-8B’s within-model position contrast, already reported as a trend rather than a claim, fails.

Rank	Contrast	$p$	Threshold	Survives
1	Llama-3.1-8B free-form, naming ( $\textsc{incontext\_named}-\textsc{incontext}$ )	$2.5\times 10^{-21}$	0.00122	yes
2	Llama-3.1-8B MCQ, naming ( $\textsc{incontext\_named}-\textsc{incontext}$ )	$1.0\times 10^{-20}$	0.00125	yes
3	Llama-3.1-8B free-form, naming at Simple strength	$9.4\times 10^{-18}$	0.00128	yes
4	Llama-3.1-8B free-form, naming at Ethos strength	$8.3\times 10^{-16}$	0.00132	yes
5	Llama-3.1-8B MCQ, timing ( $\textsc{preemptive}-\textsc{incontext\_named}$ )	$4.0\times 10^{-15}$	0.00135	yes
6	Llama-3.1-8B free-form, timing ( $\textsc{preemptive}-\textsc{incontext\_named}$ )	$1.1\times 10^{-7}$	0.00139	yes

Appendix

Table 19: Holm correction, all 41 contrasts,

\alpha=0.05

. 17 of 41 survive. The position interaction (rank 18, boldface) is the first contrast to fail; every naming-alone effect ranks above it.

LLM Sycophancy

Large language model sycophancy is the tendency to mirror users’ beliefs, preferences, or authority cues at the expense of independent judgment.

See all 161 papers

Oct 11, 2026

Big-Big-SycoBench: A Benchmark to Detect Sycophancy Through Model Responses to Valid vs. Invalid Arguments

Sycophancy in large language models risks amplifying unsupported claims or beliefs across the knowledge ecosystem as these systems increasingly contribute to research synthesis, evaluation, and decision-making. A reliable model should revise its judgement when a user identifies a genuine flaw, but should resist pushback based on fallacies, emotion, or irrelevant appeals to authority. While sycophancy is usually measured on questions with a verifiable answer, the evaluative tasks in which it is most difficult to detect have no such ground truth. We introduce Big-Big-SycoBench, which measures sycophancy through argument discernment: the ability to revise an evaluation after valid criticism while resisting invalid pressure. The benchmark consists of 450 text artefacts with 5,400 arguments for raising or lowering their quality scores, each reviewed by one or more of twelve human annotators. An LLM scores each artefact twenty times to establish its baseline score and uncertainty. The model is then given a valid or invalid argument and asked to update its score. We only count a revision if it exceeds the model's own baseline variability. We use the resulting score delta to distinguish responsiveness to valid arguments from compliance with invalid ones. Across ten models, responsiveness is 88-95% for seven, 76-77% for Kimi K3 and GLM-5.3, and 32% for Opus 5.5. Compliance ranges from 0.2% for GPT-6 Astra to 95% for GPT-4o. Big-Big-SycoBench is intended to guide the development of models whose judgements can be corrected by sound arguments, but not displaced by unsupported pressure.

Marthe Ballon, Arne Vanhoyweghen, Seorin Kim +11

Data Analytics Lab, Vrije Universiteit Brussel, Pleinlaan 5, 1050 Brussel, Belgium · imec-SMIT, Vrije Universiteit Brussel, Pleinlaan 9, 1050 Brussels, Belgium · School of Engineering and Applied Sciences, Harvard University, Cambridge, Massachusetts 02138, USA

LLM Sycophancy LLM Evaluation Belief Updating in LLMs

Ask PDF

Apr 2, 2026

SWAY: A Counterfactual Computational Linguistic Approach to Measuring and Mitigating Sycophancy

Large language models exhibit sycophancy: the tendency to shift outputs toward user-expressed stances, regardless of correctness or consistency. While prior work has studied this issue and its impacts, rigorous computational linguistic metrics are needed to identify when models are being sycophantic. Here, we introduce SWAY, an unsupervised computational linguistic measure of sycophancy. We develop a counterfactual prompting mechanism to identify how much a model's agreement shifts under positive versus negative linguistic pressure, isolating framing effects from content. Applying this metric to benchmark 6 models, we find that sycophancy increases with epistemic commitment. Leveraging our metric, we introduce a counterfactual mitigation strategy teaching models to consider what the answer would be if opposite assumptions were suggested. While baseline mitigation instructing to be explicitly anti-sycophantic yields moderate reductions, and can backfire, our counterfactual CoT mitigation drives sycophancy to near zero across models, commitment levels, and clause types, while not suppressing responsiveness to genuine evidence. Overall, we contribute a metric for benchmarking sycophancy and a mitigation informed by it.

Joy Bhalla, Kristina Gligorić

Johns Hopkins University

LLM Sycophancy Prompt Engineering

Ask PDF

Oct 13, 2026

Claim Framing as a Driver of LLM Sycophancy

Sycophancy in Large Language Models, the tendency to align responses with user preferences over ground truth, has largely been studied using questions without a clear ground truth and with a single, plainly stated user opinion. We present a systematic study of which framings of a user's claim trigger sycophancy in factual question answering, where every answer is objectively verifiable without extra context. Evaluating 14 models on 42K questions, we express correct and incorrect user opinions through various claim framings over 8 dimensions and 21 types, and measure a broad set of opinion-driven behaviors. We find that all models are susceptible to user opinions regardless of size. Models are most influenced by how confidently an opinion is stated, whom it is attributed to, and whether it is presented as shared between the user and the model. In contrast to prior work, framings based on the user's expertise, authority over the model, or social pressure matter little in our factual setting. Larger models resist incorrect opinions better but are no better at recovering from their own errors. Our results call for sycophancy evaluations that probe how an opinion is framed, not merely whether one is present, and that measure multiple behaviors beyond a single flip rate.

Jean Seo, Pepa Atanasova, Isabelle Augenstein

University of Copenhagen

LLM Sycophancy Framing Effects in Language Models LLM Evaluation

Ask PDF

Abstract

LLM Sycophancy

Explore similar work

Big-Big-SycoBench: A Benchmark to Detect Sycophancy Through Model Responses to Valid vs. Invalid Arguments

SWAY: A Counterfactual Computational Linguistic Approach to Measuring and Mitigating Sycophancy

Claim Framing as a Driver of LLM Sycophancy