A Capitulation Direction Emerges With Scale and Is Causally Load-Bearing Where Validation Is Strong: Eight Models, Two Families, and Five Measurement Hazards
Sycophancy in language models is well documented as a behaviour but poorly localised as a mechanism. We ask whether capitulation under user pushback, the abandonment of a correct answer when a user objects, corresponds to a linear direction in the residual stream, and whether such a direction emerges with scale. Across eight instruction-tuned models spanning 0.5B to 14B parameters in two families, under a pre-registered protocol with leave-one-question-out cross-validation, within-question permutation nulls, and a positive control, we find no usable direction below 7B and a direction that clears our validation gate at 7B, 8B and 14B. Ablating that direction in Qwen2.5-7B reduces the rate at which the model reverses a correct answer by 4.8 percentage points, roughly a quarter in relative terms, while the identical intervention on a gate-failing direction and on a marginally passing one produces no effect. Probe quality, not statistical significance, predicts causal relevance. Behaviourally, pushback destroys three to five times more correct answers than it repairs at 1B and reaches parity by 7B, falsifying our own pre-registered prediction, and a sign-reversing crossover between pushback types survives an order of magnitude of scale in Qwen and holds at 8B in Llama, with marginalisation over five paraphrases per type. We additionally quantify five measurement hazards, including a numerical-precision effect that changes roughly one episode label in ten.
Figures & tables
Human
Judge
corr
inc
retr
uncl
correct
70
6
2
1
incorrect
14
63
2
1
retracted
1
2
26
0
unclear
7
2
0
3
Table 1: Judge verdicts against human annotation on 200 sampled episodes. Agreement 81.0%, κ=0.707 .
Family
Params
Matched q
Layer
AUROC
Perm. thr.
Gate
Qwen2.5
0.5B
17
3
0.678
0.686
fail *
Qwen2.5
1.5B
61
23
0.632
0.580
fail
Qwen2.5
3B
66
33
0.617
0.572
fail
Qwen2.5
7B
75
19
0.778
0.574
pass
Qwen2.5
14B
61
31
0.720
0.618
pass
Llama-3.2
1B
84
13
0.569
0.559
fail
Table 2: Capitulation direction, residual stream. The gate requires AUROC ≥0.70 and AUROC above the permutation threshold (shuffle mean plus two standard deviations). * Qwen-0.5B is underpowered: only 17 questions produced both a flip and a hold, and its point estimate sits below its own null band.
Figure 1: Emergence of a usable capitulation direction. Each point is the matched leave-one-question-out AUROC at the model’s best layer; the short tick near each point marks that model’s own within-question permutation threshold. Filled markers pass the pre-registered gate, open markers fail. The dashed line is the 0.70 usability bar. Both families cross between 3B and 7B to 8B, but Qwen is flat and then steps while Llama climbs gradually.
Model
Positions
Best head
AUROC
Max-null
n above
Gate
Qwen-0.5B
336
L9 H11
0.742
0.742
0
fail
Qwen-1.5B
336
L17 H5
0.677
0.678
0
fail
Qwen-3B
576
L2 H3
0.706
0.700
3
pass
Qwen-7B
784
L18 H18
0.840
0.821
12
fail †
Qwen-14B
1920
L36 H0
0.800
0.781
4
fail †
Llama-1B
512
L13 H3
0.645
0.649
0
fail
Table 3: Attention-head scan. “Max-null” is the 95th percentile of per-permutation maxima across all head positions; “ n above” counts heads exceeding it. Qwen-7B, Qwen-14B and Llama-8B fail on the pointwise term only; that term is a normal approximation (shuffle mean +2 sd) rather than an empirical percentile (Section 5.5 ).
Model
Gate AUROC
Paired q
Baseline flip
Ablated flip
Change (95% CI)
Qwen-1.5B
0.632 (fail)
97
24.0%
21.9%
−2.1 pp [−5.9,+2.1]
Qwen-7B
0.778 (pass)
150
17.8%
13.0%
−4.8 pp [−8.3,−1.2]
Llama-8B
0.710 (pass)
177
21.2%
22.6%
+1.4 pp [−2.0,+4.8]
Table 4: Directional ablation on held-out questions. Confidence intervals from a bootstrap resampling questions, 2,000 resamples.
Figure 2: Change in flip rate after ablating the capitulation direction, with question-clustered 95% bootstrap intervals. Ordered by how far each direction cleared the 0.70 usability bar. Only Qwen-7B, which cleared it comfortably, shows an effect; the gate-failing direction and the marginally passing one do not.
Model
Paired q
Emo. only
Simple only
OR
pBonf
Qwen-0.5B e
61
11
11
1.00
1.000
Qwen-1.5B
176
15
37
0.41
0.013
Qwen-3B e
280
28
35
0.80
1.000
Qwen-7B
325
15
49
0.31
0.0001
Qwen-14B e
339
10
33
0.30
0.002
Llama-1B
227
30
35
0.86
1.000
Table 5: McNemar tests on within-question discordant pairs, flips only, Bonferroni corrected across the four pre-registered type comparisons. e Exploratory: not among the four models named in the pre-registration.
Figure 3: Odds ratios from within-question McNemar tests, emotional over simple. Values below one mean simple pushback destabilises the model more. Filled markers are the four confirmatory models named in the pre-registration; open markers are exploratory. Asterisks mark pBonf<0.05 . Every Qwen sits at or below one and both larger Llamas sit above it.
Model
Flip rate
Recovery rate
Ratio
Qwen-0.5B
17.9%
3.6%
5.0
Qwen-1.5B
31.0%
7.9%
3.9
Qwen-3B
23.7%
13.3%
1.8
Qwen-7B
18.3%
19.4%
0.9
Qwen-14B
15.1%
22.9%
0.7
Llama-1B
34.6%
9.7%
3.6
Table 6: Flip rate conditioned on an initially correct answer, recovery rate conditioned on an initially incorrect one, and their ratio.
Figure 4: Ratio of flip rate to recovery rate against scale, log axes. Above the dashed line pushback destroys more correct answers than it repairs. The ratio declines monotonically in both families and reaches parity by 7B to 8B, falsifying the pre-registered prediction that the asymmetry holds at every scale.