Sycophancy, the tendency of a language model to change its answer to match a user's stated belief, is a common alignment failure. Existing activation steering methods typically apply a single contrastive direction uniformly throughout the model, which is an unconditional intervention that alters activations even when no sycophantic behavior is present, trading knowledge retention for behavioral correction. In Mixture-of-Experts (MoE) models, prior work further suggests that behavior is encoded within expert computations rather than routing decisions alone, making precise behavioral steering particularly challenging. In this work, we introduce a shared contrastive signal, built from matched prompts with and without a stated belief, that identifies where sycophancy lives across the MoE hierarchy and drives interventions that act only where the behavior is present. We formulate localization as a causal search over a granularity ladder of MoE blocks, experts, attention blocks, and heads, and compare unconditional subtraction against two conditional alternatives: an analytic projection-based subtraction and a learned per-token gate that steers the model away from sycophancy while keeping its weights frozen. We evaluate on three MoE models measuring sycophancy alongside general knowledge and reasoning benchmarks. Our conditional interventions removed up to 90% of the belief-induced sycophancy. Our results demonstrate that sycophancy resides in identifiable computational subcircuits and can be selectively steered while maintaining a favorable removal-retention trade-off.
Figures & tables
Figure 1: The unit of intervention. Every attention head and expert carries a gate m ; the gated contributions Ai and Mi also include gated mechanisms at the output stream.
Nudge
No-nudge
Conforming
belief, agrees
neutral, agrees
Nonconforming
belief, opposes
neutral, opposes
Table 1: Each item appears in four matched forms. Nudge versus no-nudge defines the belief direction and belief push, whereas contrastive activation addition compares conforming and nonconforming answers only under the nudge condition.
Sycophancy data
Retention data
Split
Items
Balance
Set
Items
Per seed
Matched
8234
4117/4117
MMLU
1482
741
Train
4940
2470/2470
GSM8K
1319
659
Dev.
1646
823/823
TruthfulQA
817
408
Test
1648
824/824
Perplexity
1723
861
Table 2: Dataset splits and retention sets. Splits are balanced, identifier-disjoint, and verified to have zero leakage.
Method
Description
Raw
No intervention; fixes the baseline
Prompted
Instructed to be objective
FARE
Reweights the router away from belief experts
CAA subtraction
Condition direction, unconditional; swept over nine sets
Cond. subtraction
Condition direction, conditional; swept over nine sets
Learned gate
Conditional; all sets jointly
Table 3: The methods compared. The subtraction methods are swept over nine sets of the identified components, from a single component type up to all together.
Metric
Target after edit
Belief push, Eq. ( 3 )
to zero
Nudge shift
to zero
No-nudge rate (anchor)
unchanged
MMLU
≥ baseline
GSM8K
≥ baseline
TruthfulQA
≥ baseline
Table 4: The evaluation panel. The upper block measures the behavior we remove and the lower block the knowledge we keep.
Figure 6
Figure 4: The central result. For five methods across three models, each cell reports belief push removed together with GSM8K, MMLU, TruthfulQA, and perplexity, each shown relative to baseline (%); for accuracy this is the ratio of edited to baseline score, and for perplexity the inverted ratio, so that above 100% is better on every metric. The two conditional methods remove most of the push while holding GSM8K at or above baseline; routing and constant subtraction trade retention for removal.
Table 5: Design axes used in the conditional-gating sweep.
Figure 5: Every operating point for all methods and models, plotting removed belief push against GSM8K relative to baseline (%). The subtractions and FARE give many points because they are swept over component sets and coefficients, unlike the single-point gate and prompting.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: The frontier, one model at a time. Each point is one operating point, with belief push removed (%) on the horizontal axis and GSM8K retained (%) on the vertical axis; marker shape and colour denote the method (see legend), and the dashed line marks baseline retention. Points toward the upper right remove more push while keeping more reasoning. The learned gate and conditional subtraction lead in every model; the shaded band marks the high-removal region where retention stays near baseline.
Figure 7: A single ranking. For each model, the panel Z-score combines behavioral removal with retention into one all-round score (higher is better); bars are ordered within each panel and labelled with their value. The two conditional methods, conditional subtraction and the learned gate, rank first in every model, while the routing baseline (FARE) ranks last.
Figure 8: The carrying heads. Per-head causal effect on the belief push, shown as a heatmap over layer (horizontal) and head index (vertical) with colour giving the magnitude of the effect; the selected heads are boxed in magenta, eight in OLMoE, ten in Qwen, and thirteen in Granite. A few heads stand out against a near-silent background, all in the mid-to-late layers.
Figure 9: The carrying experts. Each selected expert is placed by layer (horizontal) and expert index (vertical) and labelled by its location, five in OLMoE and six in Granite. Qwen has none, because its always-on shared experts cannot appear in a routed contrast, the first sign that the shared-expert regime behaves differently.
Figure 10: Belief accumulates in the mid-to-late layers. For each model, the left panel tracks the model’s lean toward the user’s stated option (logits, vertical axis) as the residual stream passes through successive layers (horizontal axis), comparing a belief run against a matched no-belief run; the shaded region between the two curves is the preference caused by the stated opinion, and the dotted curve is a random-direction control. The right panel shows how much preference each individual layer contributes, with the dashed marker at the layer by which half of the total preference has formed. In every model the lean is flat through the early layers and rises sharply in the upper layers.
Figure 11: Where the belief signal is read and carried, per head. For each model, the left heatmap gives each head’s estimated causal effect on the belief push and the middle heatmap gives how much attention each head places on the stated-opinion text, both over layer (vertical) and head index (horizontal). The right panel plots attention on the opinion (horizontal) against causal effect (vertical), with colour showing the extra attention a head pays when an opinion is present; the labelled heads in the upper region both attend to the opinion and carry its effect, which we call transporter heads.
Figure 12: GSM8K holds until the coefficient grows large. Each panel plots GSM8K accuracy (vertical axis) against the conditional-subtraction strength c (horizontal axis). Colour marks the component set the edit is applied to: S1 attention blocks; S2 MoE blocks; S3 attention and MoE blocks; S4 attention heads; S5 experts; S6 heads and experts; S7 attention blocks and heads; S8 MoE blocks and experts; and S9 all located components. Within each set, thin lines are the individual seeds and the thick line is their mean. Accuracy stays roughly flat at low c and collapses once c becomes large, at a threshold that differs by model.
Figure 13: Main effect of each design axis on belief push removed, per model (one bar per model; higher removes more). The axes and levels are edit class (ComponentMask CM, VectorMaskFrozen VF, VectorMaskLearned VL), gate structure (FlatGate FG, CondGate CG, SharedGate SG), schedule (FullTrain FT, FineToCoarseTrain FC), and initialisation (WarmStart W, ColdStart C). The conditional gate (CG) and full schedule (FT) remove the most push across all three models, motivating those two choices in the selected configuration.
Figure 14: Choosing the gate. Each row is a candidate configuration named in the edit-gate-schedule-init convention of Figure 13 (for example VF-CG-FT-C), restricted to configurations that qualify on all three models. Rows are scored by their distance to complete removal under reasoning and fluency penalties, with over-removal runs discarded; lower is better, the starred row is the winner, and the final column notes why each runner-up was not chosen. The winner, VF-CG-FT-C, a frozen conditional gate (VectorMaskFrozen with CondGate) trained on a full schedule from a cold start, is the pick carried into the seed study.
Figure 15: Choosing the gate architecture, one territory per model. Each point is a configuration labelled edit-gate-schedule-init; the horizontal axis is nudge removed (%) and the vertical axis is the GSM8K change versus baseline (negative means reasoning lost). Each shaded region encloses one model’s top-ten configurations, and the star, pentagon, and cross mark that model’s first, second, and third best picks. The acronym key is given in the side panel.
Figure 16: Architecture sweep on the removal-against-perplexity plane, used for configuration selection. The horizontal axis is nudge removed (%) and the vertical axis is the perplexity change versus baseline (positive means fluency lost), with the dashed line marking the honest ceiling and the background shading giving the distance-to-ideal ranking score (brighter is better). Perplexity here is the decision-token-off control of Section 4.3 , so this plane screens configurations for gross perplexity regressions rather than measuring open-ended fluency; the selected configurations sit at high removal with perplexity near baseline.
Figure 17: The edit is sparse. Per-site edit amount ( 1−m ) of the selected gate at its checkpoint, one bar per component with the dashed line marking the inclusion threshold. A handful of attention and mixture-of-experts sites carry most of the edit, matching the concentrated placement of Figure 2 .
Figure 18: What the gate edits, and what it leaves. The most and least edited components of the selected gate, ranked by edit amount and coloured by component type. The heavily edited sites are mid-to-late-layer attention and expert components, while the least edited are individual attention heads.
Figure 19: The localization is stable. Gate activity across the three seeds (s1, s2, s4), with colour giving each component’s edit amount. The set of components the gate opens on is consistent from seed to seed, so the located support is a property of the model rather than of a single run.
Figure 20: Method comparison across the three MoE models. Each axis is a metric (Removal, Invariance, MMLU, GSM8K, TruthfulQA, PPL), normalised so that farther out is better, and each closed curve is one method. Conditional subtraction (Cond sub) and the learned gate achieve high removal while retaining capability, whereas prompting and FARE trade little removal for capability or fail to remove sycophancy.
Large language models exhibit sycophancy, the tendency to shift their stated positions toward perceived user preferences or authority cues regardless of evidence. Standard alignment methods fail to correct this because scalar reward models conflate two distinct failure modes into a single signal: pressure capitulation, where the model changes a correct answer under social pressure, and evidence blindness, where the model ignores the provided context entirely. We operationalise sycophancy through formal definitions of pressure independence and evidence responsiveness, serving as a working framework for disentangled training rather than a definitive characterisation of the phenomenon. We propose the first approach to sycophancy reduction via reward decomposition, introducing a multi-component Group Relative Policy Optimisation (GRPO) reward that decomposes the training signal into five terms: pressure resistance, context fidelity, position consistency, agreement suppression, and factual correctness. We train using a contrastive dataset pairing pressure-free baselines with pressured variants across three authority levels and two opposing evidence contexts. Across five base models, our two-phase pipeline consistently reduces sycophancy on all metric axes, with ablations confirming that each reward term governs an independent behavioural dimension. The learned resistance to pressure generalises beyond our training methodology and prompt structure, reducing answer-priming sycophancy by up to 17 points on SycophancyEval despite the absence of such pressure forms during training.
Muhammad Ahmed Mohsin, Ahsan Bilal, Muhammad Umer +1
Sycophantic agreement in language models refers to the tendency to overly affirm a user's stated beliefs or preferences, often at the expense of factual accuracy. Although it is widely recognized as an alignment failure, its underlying mechanisms remain poorly understood. In this work, we use causal mediation analysis to identify the mechanisms behind sycophantic agreement. We show that a stated opinion is incorporated into the residual stream of the final prompt token early, where it biases subsequent answer retrieval. A sparse set of early attention heads carries this opinion signal. Ablating these heads substantially reduces sycophancy while leaving factual accuracy largely intact. The same heads carry the opinion when it is explicitly stated, regardless of how it is phrased. When an opinion is not stated explicitly but instead conveyed through content-free pushback (e.g., ``Are you sure?"), we find a distinct set of heads that suppresses the model's original correct answer to promote a revised answer. By providing a mechanistic account of how opinions induce sycophantic agreement, this work takes a step toward developing more targeted and reliable alignment interventions.
Sixing Chen, Zhuofan Josh Ying, Logan Riggs Smith +2
New York University · Cambridge Boston Alignment Initiative · Columbia University +2
Large language models exhibit sycophancy: the tendency to shift outputs toward user-expressed stances, regardless of correctness or consistency. While prior work has studied this issue and its impacts, rigorous computational linguistic metrics are needed to identify when models are being sycophantic. Here, we introduce SWAY, an unsupervised computational linguistic measure of sycophancy. We develop a counterfactual prompting mechanism to identify how much a model's agreement shifts under positive versus negative linguistic pressure, isolating framing effects from content. Applying this metric to benchmark 6 models, we find that sycophancy increases with epistemic commitment. Leveraging our metric, we introduce a counterfactual mitigation strategy teaching models to consider what the answer would be if opposite assumptions were suggested. While baseline mitigation instructing to be explicitly anti-sycophantic yields moderate reductions, and can backfire, our counterfactual CoT mitigation drives sycophancy to near zero across models, commitment levels, and clause types, while not suppressing responsiveness to genuine evidence. Overall, we contribute a metric for benchmarking sycophancy and a mitigation informed by it.