Big-Big-SycoBench: A Benchmark to Detect Sycophancy Through Model Responses to Valid vs. Invalid Arguments
Authors: Marthe Ballon, Arne Vanhoyweghen, Seorin Kim, Francesca Carlon, Lynn Delcon, Melika Mobini, Vincent Holst, Lukas Van de Voorde, Tibo Vanleke, Mathieu Louis, Bert Verbruggen, Brecht F. Verbeken, Andres Algaba, Vincent Ginis
Organizations: Data Analytics Lab, Vrije Universiteit Brussel, Pleinlaan 5, 1050 Brussel, Belgium · imec-SMIT, Vrije Universiteit Brussel, Pleinlaan 9, 1050 Brussels, Belgium · School of Engineering and Applied Sciences, Harvard University, Cambridge, Massachusetts 02138, USA
Sycophancy in large language models risks amplifying unsupported claims or beliefs across the knowledge ecosystem as these systems increasingly contribute to research synthesis, evaluation, and decision-making. A reliable model should revise its judgement when a user identifies a genuine flaw, but should resist pushback based on fallacies, emotion, or irrelevant appeals to authority. While sycophancy is usually measured on questions with a verifiable answer, the evaluative tasks in which it is most difficult to detect have no such ground truth. We introduce Big-Big-SycoBench, which measures sycophancy through argument discernment: the ability to revise an evaluation after valid criticism while resisting invalid pressure. The benchmark consists of 450 text artefacts with 5,400 arguments for raising or lowering their quality scores, each reviewed by one or more of twelve human annotators. An LLM scores each artefact twenty times to establish its baseline score and uncertainty. The model is then given a valid or invalid argument and asked to update its score. We only count a revision if it exceeds the model's own baseline variability. We use the resulting score delta to distinguish responsiveness to valid arguments from compliance with invalid ones. Across ten models, responsiveness is 88-95% for seven, 76-77% for Kimi K3 and GLM-5.3, and 32% for Opus 5.5. Compliance ranges from 0.2% for GPT-6 Astra to 95% for GPT-4o. Big-Big-SycoBench is intended to guide the development of models whose judgements can be corrected by sound arguments, but not displaced by unsupported pressure.
Figures & tables
Figure 1: Evaluation of five frontier models and five legacy models in the OpenAI model family on Big-Big-SycoBench.
Figure 2: Five-stage construction of Big-Big-SycoBench. Artefacts and arguments are generated under a controlled specification and the validity labels pass adversarial quality control (1), twelve annotators review every artefact with deliberate overlap (2), arguments are revised and normalised to remove construction cues (3), the revised data are verified again by an automated classifier and a human annotator (4), and the final dataset is balanced across domains and target qualities (5).
Figure 3: Illustration of the follow, stable and contrarian update probabilities for argument C184/lower/invalid/2 and C184/lower/valid/3.
Figure 4: Model responses to invalid and valid arguments. We benchmark Opus 5.5, GPT-6 Astra, Grok 4.7, Kimi K3, and GLM-5.3 for the frontier model leaderboard, and GPT-4o, GPT-5, GPT-5.5, GPT-5.6 Luna, GPT-5.6 Terra, and GPT-6 Astra for a comparison across model generations (consult Section A.1 for full model specifications). a., The full response plane. b., Zoomed-in version of the response plane. The model coordinates show pf−pc for invalid feedback (horizontal) and valid feedback (vertical) as defined in Section 3 . Ellipse widths and heights around the data points encode the corresponding contrarian probabilities, which are too small to distinguish at this scale (see Table B1 ). Error bars show marginal 95% percentile intervals from 10,000 artefact bootstrap resamples. Opus 5.5 is evaluated on a 100-artefact subset (see Section 4 and Section A.8 ); all other models are evaluated on the full set.
Figure 5: Models follow valid arguments more often when asked to lower their score than when asked to raise it.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1: Bradley–Terry (BT) scoring diagnostics. (a) Correlations with the #M=256 reference using GPT-5 nano for the pairwise judgments; 70 is the first tested budget exceeding Kendall’s τb=0.9 . (b) BT-score densities for valid and invalid arguments, fitted using GPT-5.6 Luna (max) with 70 matches per argument. (c,d) BT-weighted pf versus the unweighted benchmark score for valid and invalid feedback; the diagonal indicates no change under weighting.
Model (effort)
pval
pinval↓
pval(1−pinval)
pval−pinval
d⋆
GPT-6 Astra (max)
0.8828
0.0015
0.8815
0.8813
0.1172
Opus 5.5 ∗ (max)
0.3219
0.0079
0.3194
0.3140
0.6781
GLM-5.3 (max)
0.7742
0.0908
0.7039
0.6834
0.2434
Kimi K3 (max)
0.7617
0.1334
0.6600
0.6282
0.2731
Grok 4.7 (xhigh)
0.8887
0.1969
0.7137
0.6918
0.2262
GPT-5.5 (xhigh)
0.9187
0.2191
0.7174
0.6996
0.2337
Appendix
Table A1: Following update probabilities and alternative scores, ranked by increasing pinval . Probabilities are shown over the 450 artefacts, except 449 invalid pools for GPT-4o, GPT-5, GPT-5.5 and GLM-5.3 due to zero weight for all three arguments. ∗ Opus 5.5 results are over 100 artefacts.
Valid arguments
Invalid arguments
Model (effort)
pf
ps
pc
pf
ps
pc
GPT-6 Astra (max)
0.8828
0.1172
0.0000
0.0015
0.9985
0.0000
Opus 5.5 ∗ (max)
0.3219
0.6781
0.0000
0.0079
0.9921
0.0000
GLM-5.3 (max)
0.7742
0.2258
0.0000
0.0908
0.9091
0.0001
Kimi K3 (max)
0.7617
0.2383
0.0000
0.1334
0.8664
0.0001
Grok 4.7 (xhigh)
0.8887
0.1113
0.0000
0.1969
0.8020
0.0011
Appendix
Table B1: The benchmark update probabilities following ( pf ), stable ( ps ) and contrarian ( pc ), ranked by increasing pfinval . Probabilities are shown over the 450 artefacts, except for 449 invalid pools for GPT-4o, GPT-5, GPT-5.5 and GLM-5.3 due to zero weight for all three arguments. ∗ Opus 5.5 results are over 100 artefacts.
Figure B1: Initial scoring overview. (a) Distribution of per-artefact mean scores. (b) Distribution of per-artefact sample SDs, each based on 20 repetitions; boxes show medians and interquartile ranges, with 1.5-IQR whiskers. (c) Fractions assigned lower or raise pushback according to mean scores ≥50 or <50 , respectively. Opus 5.5 uses 100 artefacts; other models use 450.
Figure B2: Initial scores by target argument quality. Colours distinguish the five anchor targets (15, 30, 50, 70 and 85). Each observation is an artefact’s mean score over 20 repetitions; five-point histograms are normalised within each model–anchor group. Anchors denote construction targets, not observed-score cutoffs. Opus 5.5 uses 100 artefacts; other models use 450.
Figure B3: Alternative threshold rules for (a) valid and (b) invalid feedback. Points show changes in pf from a 2σ reference, in percentage points. Alternatives are normal-reference 95% and 99% cutoffs and the empirical 95th percentile of absolute baseline deviations from the mean. These are scoring sensitivity checks, not calibrated significance tests. Sample sizes follow Table B1 .
Figure B4: Threshold sensitivity for (a) valid and (b) invalid feedback. Following probability pf requires movement towards the feedback exceeding kσ , where σ is the baseline sample SD. The dashed guide marks the default k=2 . Outcomes and BT weights are held fixed. Sample sizes follow Table B1 .
Figure B5: Magnitude-aware scoring: (a) overview and (b) vertically expanded view of the same models. Unlike following probabilities, this score captures both how often models update and how far they move beyond the stability threshold. For each argument, let D be the score change signed towards the feedback and δi=2σ0,i , where σ0,i is the sample SD of the model’s 20 initial scores for artefact i . We compute mf=E[(D−δi)+] and mc=E[(−D−δi)+] , with (x)+=max(x,0) , by averaging over available repetitions. Thus, a response exceeding its threshold by ten score points contributes ten; responses within the stability band contribute zero. The axes show mf−mc for invalid (horizontal) and valid (vertical) feedback, using BT weights within each artefact and feedback class and equal averaging across included artefacts. Values are in score points. Their interpretation depends on models’ use of the scoring scale and baseline variability; larger valid updates are not necessarily better. No confidence intervals are shown. Sample sizes follow Table B1 .
Figure B6: Score revision versus baseline scoring variability. Artefacts are binned by the sample SD of their 20 initial scores; points show the mean score change towards the feedback as a percentage of the available movement, for valid and invalid arguments, with 95% artefact-bootstrap intervals. Treating SD as a proxy for uncertainty, a discerning model should revise more where it is less certain while ignoring invalid arguments. Grok 4.7, Kimi K3, GLM-5.3, GPT-5.5 and GPT-6 Astra show this pattern, with Astra’s invalid revisions near zero throughout. GPT-5.6 Luna and Terra separate valid from invalid arguments only at higher SD, whereas GPT-5 and GPT-4o revise more after invalid than valid arguments. Opus 5.5 resembles Astra in ignoring invalid arguments, but unlike Astra, whose valid revisions roughly double across its SD range, its valid revisions stay low with no clear dependence on SD, although its 100-artefact subset gives wider intervals. This is consistent with Fig. B4 : even at k=0 , where any movement towards the feedback counts, Opus 5.5 follows only about half of valid arguments, compared with at least 95% for the other models. Because SD ranges differ across models, the within-panel trend (or lack thereof) and the ordering of the valid and invalid curves are more informative than the absolute SD values at which models revise.
Large language models (LLMs) are known to exhibit social sycophancy, often validating or agreeing with users in socially sensitive contexts. Existing evaluations typically measure sycophancy under a fixed prompt formulation, leaving unclear whether such behavior is stable when the same underlying situation is presented with different sycophancy-relevant prompt variants. In this work, we study sycophancy prompt sensitivity: the extent to which changes in user confidence, emotional framing, social consensus, or validation-seeking language alter a model's sycophantic behavior. We refer to our evaluation framework as SyPS, short for Sycophancy Prompt Sensitivity. Building on existing social sycophancy evaluation settings, SyPS constructs controlled prompt variants that preserve the same underlying user situation while varying sycophancy-relevant social cues. We introduce the Sycophancy Prompt Sensitivity Score (SPSS), an instance-level measure of sycophancy variation across paired prompt variants. Unlike aggregate sycophancy rates, SPSS separates baseline sycophancy from prompt-induced shifts, enabling model-level comparisons of robustness to sycophancy-relevant social cues. Empirically, we find that sycophancy prompt sensitivity is socially structured: validation-seeking and emotional-pressure cues often increase sycophancy, whereas counter-framing and anti-sycophancy prompts tend to reduce it. Our framework highlights whether LLMs maintain stable social judgments while adapting appropriately in tone.
Lijia Huang, Yao Fu, Sihao Ren
Northeastern University · Case Western Reserve University · Everpure
Sycophancy is a language model's tendency to cave when a user pushes back, abandoning a correct answer for the user's. Several benchmarks now measure it by scripting an objection and recording how often the model caves. Because that objection is a prompt template, whatever else the template varies is measured along with the property it claims to isolate. We audit SycEval, which reports that objections raised before a model answers (preemptive) cause more caving than those raised after (in-context), and attributes the gap to timing. Two features of its templates vary alongside the property each is meant to test. First, SycEval's objections escalate through four strength levels, and at the two weakest only the preemptive template names a target answer, so timing and naming vary together. We build the missing comparison and test it on multiple-choice questions and SycEval's own free-form pipeline. Naming a target answer raises the follow rate, the share of samples matching the user's assertion, by 14.1--49.5 percentage points (pp), and once both templates name one, the timing comparison reverses on three of the five model conditions we test: models cave less under preemptive objections than in-context ones, opposite to SycEval. Second, the output-format instruction benchmarks append for automatic grading also varies with objection placement. Moving it from the pushback into the question flips its effect in opposite directions across models on the same items (p=0.0059). The same effect appears in SycEval's free-form pipeline: relocating its instruction lowers caving by 5.1pp on Llama-3.1-8B, while an equivalence test confirms no effect on Qwen3-4B. Both confounds live in the template rather than the models under test, so both are correctable: we close with three checks benchmark authors can apply before publishing.
We propose a novel perspective for probing LLM sycophancy in a direct and neutral way, mitigating various forms of uncontrolled bias, noise, or manipulative language, deliberately injected to prompts in prior works. A key novelty of our approach is the use of an LLM-as-a-judge in a zero-sum betting game. Within this framework, sycophancy serves one individual (the user) while explicitly incurring cost on another. Comparing 11 leading models we find that while most models exhibit significant sycophantic tendencies in the common setting, in which sycophancy is self-serving to the user and incurs no cost on others, seven of the models exhibit ``moral remorse'', five of which significantly over-compensate for their sycophancy in case it explicitly harms a third party. We refer to this phenomenon as `anti-sycophancy' bias and discuss possible causes for this shift.
Shahar Ben-Natan, Oren Tsur
Computer and Information Science Ben Gurion University