cs.LGAug 26, 2026

Mitigating LLM sycophancy with RL-based fine-tuning: Bayesian Truth Serum approach

Authors: Serhii Mytsyk, Yiming Zhang, Vikram Krishnamurthy

Organizations: Center for Applied Mathematics Cornell University · Department of Electrical and Computer Engineering Cornell University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language models (LLMs) frequently exhibit sycophancy: they adapt their answers to a user's stated beliefs or preferences instead of reporting what they hold to be true, which lowers factual accuracy and can amplify misinformation. This paper proposes a methodology for mitigating sycophancy that employs the Bayesian Truth Serum (BTS), a peer-prediction mechanism, as the reward in Group Relative Policy Optimization (GRPO) to fine-tune an LLM. BTS pays an answer for being surprisingly common, that is, more frequent among respondents than those respondents themselves predicted. We treat a group of responses from a model for one question as those respondents, so the reward is a function of the model's own outputs and fine-tuning needs neither labels nor preference annotations. We prove that in the large-group limit a sycophantic response earns strictly lower expected reward than an honest one. We also prove that if the entire group agrees in advance on a symmetric answering rule, it cannot earn a higher information score than under truthful reporting. On our true/false benchmark the reference model's answer-flip rate under user pressure decreases from 23% to 4%, and its accuracy under that pressure increases from 80% to 93%. Our reward outperforms SMART and is comparable to synthetic-data fine-tuning and to pinpoint tuning, all three of which train on labels. It spends considerably more compute in exchange, which makes it suitable when labeled data is scarce. Peer Truth Serum, which also pays a premium for a rare answer but elicits no prediction report, reproduces the effect. A peer-prediction reward computed inside a single GRPO group therefore reduces sycophancy without labels, and comparing mechanisms suggests that the premium paid for a rarer answer drives the effect.

Figures & tables

Figure 1: Schematic of the BTS-guided GRPO fine-tuning iteration. For each question, we sample a group of GG LLM responses: each contains an answer and a prediction of the group’s answer distribution. The BTS reward, computed entirely from the group, drives the GRPO update in place of ground-truth supervision.
More in this topic

LLM Sycophancy

Large language model sycophancy is the tendency to mirror users’ beliefs, preferences, or authority cues at the expense of independent judgment.

See all 163 papers

Explore similar work

CardsList
Jan 21, 2026

Not Your Typical Sycophant: The Elusive Nature of Sycophancy in Large Language Models

We propose a novel perspective for probing LLM sycophancy in a direct and neutral way, mitigating various forms of uncontrolled bias, noise, or manipulative language, deliberately injected to prompts in prior works. A key novelty of our approach is the use of an LLM-as-a-judge in a zero-sum betting game. Within this framework, sycophancy serves one individual (the user) while explicitly incurring cost on another. Comparing 11 leading models we find that while most models exhibit significant sycophantic tendencies in the common setting, in which sycophancy is self-serving to the user and incurs no cost on others, seven of the models exhibit ``moral remorse'', five of which significantly over-compensate for their sycophancy in case it explicitly harms a third party. We refer to this phenomenon as `anti-sycophancy' bias and discuss possible causes for this shift.

Shahar Ben-Natan, Oren Tsur

Computer and Information Science Ben Gurion University

LLM EvaluationLLM SycophancyLanguage Model Bias Evaluation
AskPDF
Oct 13, 2026

Claim Framing as a Driver of LLM Sycophancy

Sycophancy in Large Language Models, the tendency to align responses with user preferences over ground truth, has largely been studied using questions without a clear ground truth and with a single, plainly stated user opinion. We present a systematic study of which framings of a user's claim trigger sycophancy in factual question answering, where every answer is objectively verifiable without extra context. Evaluating 14 models on 42K questions, we express correct and incorrect user opinions through various claim framings over 8 dimensions and 21 types, and measure a broad set of opinion-driven behaviors. We find that all models are susceptible to user opinions regardless of size. Models are most influenced by how confidently an opinion is stated, whom it is attributed to, and whether it is presented as shared between the user and the model. In contrast to prior work, framings based on the user's expertise, authority over the model, or social pressure matter little in our factual setting. Larger models resist incorrect opinions better but are no better at recovering from their own errors. Our results call for sycophancy evaluations that probe how an opinion is framed, not merely whether one is present, and that measure multiple behaviors beyond a single flip rate.

Jean Seo, Pepa Atanasova, Isabelle Augenstein

University of Copenhagen

LLM SycophancyFraming Effects in Language ModelsLLM Evaluation
AskPDF
Apr 12, 2026

Calibration Collapse Under Sycophancy Fine-Tuning: How Reward Hacking Breaks Uncertainty Quantification in LLMs

Modern large language models (LLMs) are increasingly fine-tuned via reinforcement learning from human feedback (RLHF) or related reward optimisation schemes. While such procedures improve perceived helpfulness, we investigate whether sycophantic reward signals degrade calibration -- a property essential for reliable uncertainty quantification. We fine-tune Qwen3-8B under three regimes: no fine-tuning (base), neutral supervised fine-tuning (SFT) on TriviaQA, and sycophancy-inducing Group Relative Policy Optimisation (GRPO) that rewards agreement with planted wrong answers. Evaluating on 1,0001{,}000 MMLU items across five subject domains with bootstrap confidence intervals and permutation testing, we find that sycophantic GRPO produces consistent directional calibration degradation -- ECE rises by +0.006+0.006 relative to the base model and MCE increases by +0.010+0.010 relative to neutral SFT -- though the effect does not reach statistical significance (p=0.41p = 0.41) at this training budget. Post-hoc matrix scaling applied to all three models reduces ECE by 4040--64%64\% and improves accuracy by 1.51.5--3.03.0 percentage points. However, the sycophantic model retains the highest post-scaling ECE relative to the neutral SFT control (0.0420.042 vs.\ 0.0370.037), suggesting that reward-induced miscalibration leaves a structured residual even after affine correction. These findings establish a methodology for evaluating the calibration impact of reward hacking and motivate calibration-aware training objectives.

Subramanyam Sahoo

Cambridge AI Safety Hub (CAISH)

Language Model CalibrationReward HackingRL Fine-TuningLLM Sycophancy
AskPDF