Consistency Training Suppresses Cue Following Without Teaching Cue Selectivity
Authors: Prakhar Gupta, Terry Jingchen Zhang, Zhijing Jin
Organizations: University of Michigan · Jinesis Lab, University of Toronto & Vector Institute · University of Oxford · EuroSafeAI · Max Planck Institute for Intelligent Systems, Tübingen, Germany
Language models often change answers because of a cue, a hint such as "I believe the answer is B" that gives no valid evidence. Consistency training (CT) trains a model to answer the same with and without the cue. Four CT methods apply this idea to output tokens (BCT), residual activations (ACT), attention patterns (AttCT) and sampled-answer rates (RMCT). Prior work suggests some learn different solutions. We train all four on Gemma-2-2B-it and Llama-3.1-8B-Instruct with the same data. We test predictions written in advance and inspect the models with a known circuit that carries cues to the answer. All four make the trained cue less effective, and their fixes share three properties. First, correct cues lose about as much influence as wrong ones. Second, the fix is usually large for cues before the answer options, as in training, and much smaller after them. Llama RMCT depends least on position. Third, the circuit heads still carry most remaining cue-induced errors. Making BCT targets depend on the cue's correctness mostly weakens the fix. CT thus weakens the trained cue without teaching models when a cue is right. Evaluations with only wrong cues at the trained position miss the first two properties.
Figures & tables
Figure 1: Consistency training weakens right and wrong cues alike, mostly before the answer options, and leaves the cue circuit in use. (a) Cue following on the same questions, cue before the options. (b) Harmful-cue following removed at each cue position. (c) Share of cue errors corrected by patching, which resets the circuit heads to uncued values. If CT had removed the circuit, the after-CT bar would drop toward the black lines, which show the best random heads (highest over models and methods). All bars and the lines in (b) average the two models. Bars in (a) and the after-CT bar in (c) also average the four methods. Whiskers show ranges, and Llama uses one seed in (c).
Figure 2: Every method cuts the trained cue, but transfer to new cue types is weaker. Whiskers show 95% intervals and dotted lines the original model (Table 6 ).
Figure 3: A cue that names the correct answer loses its effect as much as a wrong one. Arrows run from the original model (circle) to the model after CT, on questions that both models answer wrongly without a cue (cue before the options). A selective fix would keep the green arrows short.
Figure 4: The fix fades for cues placed after the answer options. Whiskers show 95% intervals. Training puts every cue at the front (shaded). On Gemma, the values of RMCT after the options come mostly from its preference for answering D (§ 4.3 ).
Figure 5: The circuit still carries the leftover errors, and the fix sits mainly before it. (a) Share of the remaining cue-induced errors that patching the circuit heads corrects. (b, c) We copy one block of attention layers from the trained model into the original model and show the share of the method’s fixes that this mixed model reproduces (our planned readout for Llama). The swap cannot test RMCT. Llama uses one seed.
Figure 6: No change to the training data of BCT makes the fix selective. Cue battery, seed means. (a) shows each BCT variant and the self-training control on Gemma, and (b) compares BCT with position-diverse BCT, which trains with the cue at all four positions.
Figure 7: Benign fine-tuning raises cue following in every model, the original one included. Harmful-cue following on the trained cue, with bands for the SD over seeds (the original model is one run).
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
code
registered (EDT)
prediction, as written
outcome
P1
26 Sep, 23:47
efficacy: every arm cuts trained-cue (suggested_answer test) flip ∣ knew by ≥ 50% vs untrained.
fails
P2
26 Sep, 23:47
transfer: BCT has the lowest mean held-out caving (5 held-out cues) of the four arms.
fails
P3
26 Sep, 23:47
reading vs following (wave-3 W2 pattern): early-reader cue-span mass on the trained cue falls by ≥ 30% for ACT and AttCT in all seeds, while on every held-out cue each arm’s relative change stays within ± 25% (seed mean).
fails
P4
26 Sep, 23:47
where edits land: per-head OV depth centroid (log 82.7 definition) is lower for AttCT and ACT than for RMCT in all seeds.
holds
P5
26 Sep, 23:47
content-blindness: every arm lowers paired helpful-cue uptake (gen3, W4c paired definition) in all seeds.
holds
F1
27 Sep, 14:21
Bayes-BCT(0.8): helpful-uptake drop ≤0.5× BCT’s AND harmful-caving drop ≥0.5× BCT’s.
fails
Appendix
Table 1: Every scored prediction as registered (2026). P1–P5, F1–F3 and R1 concern Gemma, and B1–B3 concern response-only BCT on Gemma. L1–L5, LF1–LF3, LR1–LR2 and D1 concern Llama. S1–S2 outcomes are given as Gemma / Llama. The 30 Sep 12:07 erratum withdrew the composition numbers of the 30 Sep block, including the 25% in LF1, and the Llama Bayes-BCT data for LF1 were rebuilt on the corrected readout. The P1–P5 section header in the file reads 27 Sep, and an erratum appended one minute later corrects it to 26 Sep. Outcomes are on the corrected, unmerged results.
appended (EDT)
amendment, erratum or note
27 Sep, 00:27
LR rule, Amendment 1 (the header says 00:45, corrected by an erratum). The grid-end test uses the best F1, extensions are sequential and at most two, and an untied published LR yields to the best-F1 LR. RMCT execution v2 uses shared-prefix sampling and a one-epoch run in resumable segments (we verified that the LoRA weights and the Adam step are identical across boundaries).
27 Sep, 02:07
Exploratory RMCT+anchor variant declared (anchor weight 0.5, the trainer’s default), after the pre-registered RMCT runs had collapsed to a constant answer at every LR evaluated so far. It is not part of P1–P5.
27 Sep, 02:31
LR rule, Amendment 2. A checkpoint that gives one letter on at least 95% of validation prompts is degenerate and excluded. If every grid point is degenerate, the grid extends downward.
30 Sep, 09:01
Erratum to D1. RMCT cannot be tested by the layer swap (its adapter trains MLP and unembedding weights), so it is dropped from that analysis.
30 Sep, 11:04
Llama relapse runs use micro-batches of 2 because of memory. The loss and gradient are unchanged up to bf16 numerics.
30 Sep, 12:07–13:27
Llama readout correction, in four notes. The first-token readout must include the “(L” tokens, with a readout gate of at least .97 agreement with greedy generation (v6, 12:07, with gate coverage and handling at 12:24). Because v6 fails for Llama BCT, the readout becomes the maximum over letter forms (v7), and Llama LR selection, test rates and relapse move to generated answers (12:53). Exact bf16 ties are broken as greedy decoding breaks them (v7.1, 13:27).
Appendix
Table 2: Amendments, errata and correction notes after the first scored predictions, in the order appended (2026). Notes on one topic appended within hours of each other share a row. The full texts are in the supplementary pre-registration file.
verdict
deciding quantity
threshold
value
95% interval
status
P1 (BCT)
trained flip ∣ knew −0.5× original ( ≤ 0 holds)
0.00
-0.129
[-0.196, -0.062]
resolved
P1 (ACT)
trained flip ∣ knew −0.5× original ( ≤ 0 holds)
0.00
-0.004
[-0.077, 0.069]
unresolved
P1 (AttCT)
trained flip ∣ knew −0.5× original ( ≤ 0 holds)
0.00
+0.012
[-0.058, 0.084]
unresolved
P1 (RMCT)
trained flip ∣ knew −0.5× original ( ≤ 0 holds)
0.00
-0.153
[-0.211, -0.093]
resolved
P2
BCT held-out − lowest other method ( < 0 holds)
0.00
+0.139
[0.094, 0.185]
resolved
L1 (BCT, trained)
trained method − original ( < 0 holds)
0.00
-0.204
[-0.293, -0.115]
resolved
Appendix
Table 3: Pre-registered verdicts that turn on a single estimate, with the deciding quantity, threshold, value and 95% interval. The intervals come from a bootstrap over questions for test-set, cue-battery and layer-swap rates, a binomial bootstrap for relapse fractions (Table 20 gives the seed t interval) and a seed-level t interval for LR2 and S1–S2. A verdict is unresolved when the interval contains the threshold.
Table 4: Stored and re-selected learning rates, with the validation F1 at every grid point (unmerged adapters, generated answers). d marks a degenerate point (excluded), and † an exploratory variant.
merged − unmerged, seed mean
method
agreement (range)
trained cue
held-out
accuracy
rule met
Gemma-2-2B-it
BCT
0.972–0.983
-0.010
-0.001
-0.003
0/3
ACT
0.975–0.988
-0.004
+0.003
-0.005
0/3
AttCT
0.987–0.991
+0.002
-0.002
-0.003
2/3
RMCT
0.924–0.943
+0.033
+0.059
+0.012
0/3
Appendix
Table 5: Merged (bf16 merge_and_unload ) and unmerged adapters, per method over its three seeds. The columns give the range of agreement of the generated test answers, the seed-mean difference (merged − unmerged) in trained-cue flip ∣ knew , held-out flip ∣ knew and uncued accuracy, and the number of seeds that meet the declared rule. The merge check came before the learning-rate re-selection, so the Gemma ACT and AttCT rows are the checkpoints at the superseded rates. Held-out rates use the 16-token answers before the few-shot correction. † marks an exploratory variant.
method
trained-cue flip ∣ knew
Δ [95% CI]
held-out (5 cues)
Δ [95% CI]
uncued accuracy ( Δ )
Gemma-2-2B-it
original
0.490 [0.39, 0.59]
–
0.546 [0.50, 0.59]
–
0.454 [0.39, 0.52]
BCT
0.116 ± 0.024
-0.37 [-0.48, -0.27]
0.421 ± 0.018
-0.12 [-0.17, -0.08]
0.406 ± 0.045 (-0.05)
ACT
0.241 ± 0.012
-0.25 [-0.34, -0.16]
0.479 ± 0.004
-0.07 [-0.10, -0.04]
0.468 ± 0.005 (+0.01)
AttCT
0.257 ± 0.010
-0.23 [-0.32, -0.15]
0.486 ± 0.003
-0.06 [-0.08, -0.03]
0.444 ± 0.005 (-0.01)
RMCT
0.091 ± 0.015
-0.40 [-0.50, -0.30]
0.282 ± 0.043
-0.26 [-0.31, -0.22]
0.458 ± 0.028 (+0.00)
Appendix
Table 6: All four methods reduce following of the trained cue, and transfer depends on the method and the model. Test set, generated answers. Harmful-cue following (flip ∣ knew ) for the trained cue at its original positions and averaged over the five held-out cue types, and uncued accuracy. Seed mean ± SD. Δ is the method minus the original model, with its 95% question-cluster interval (no interval for accuracy). The cells of the original model show their own interval.
method
trained cue
post hoc
distractor fact
wrong few-shot
distractor arg.
squares
Gemma-2-2B-it
original
0.490
0.794
0.255
0.504
0.737
0.440
BCT
0.116 (-0.37)
0.391 (-0.40)
0.086 (-0.17)
0.513 (+0.01 ∘ )
0.823 (+0.09)
0.292 (-0.15)
ACT
0.241 (-0.25)
0.679 (-0.12)
0.066 (-0.19)
0.543 (+0.04 ∘ )
0.667 (-0.07)
0.438 (+0.00 ∘ )
AttCT
0.257 (-0.23)
0.773 (-0.02 ∘ )
0.056 (-0.20)
0.488 (-0.02 ∘ )
0.688 (-0.05)
0.427 (-0.01 ∘ )
RMCT
0.091 (-0.40)
0.369 (-0.43)
0.015 (-0.24)
0.326 (-0.18)
0.506 (-0.23)
0.196 (-0.24)
Appendix
Table 7: Per-cue flip ∣ knew on the test set (generated answers, with few-shot cues scored at the test slot). The table gives the original value and, for each method, the seed mean with the change from the original model in parentheses. ∘ marks a change whose 95% question-cluster bootstrap interval includes zero, and † an exploratory variant.
method
trained: method / control
beyond prior
held-out: method / control
beyond prior
modal-letter share
Gemma-2-2B-it
BCT
0.115 / 0.439
0.324 ± 0.025
0.423 / 0.587
0.164 ± 0.023
0.31 (D/C/C)
ACT
0.238 / 0.471
0.233 ± 0.020
0.488 / 0.597
0.109 ± 0.003
0.27 (C/C/C)
AttCT
0.251 / 0.475
0.224 ± 0.012
0.516 / 0.603
0.086 ± 0.008
0.27 (C/C/C)
RMCT
0.098 / 0.336
0.237 ± 0.020
0.303 / 0.492
0.189 ± 0.012
0.54 (D/D/D)
RMCT+anchor †
0.142 / 0.462
0.320 ± 0.053
0.392 / 0.592
0.200 ± 0.050
0.30 (A/C/A)
Appendix
Table 8: Letter-prior control (exploratory, first-position readout, with the gate-failing Gemma checkpoints on the v7.1 readout). The table gives the flip ∣ knew of the method and of the original model shifted to the uncued letter distribution of the method, on the trained cue and the held-out mean. It also gives the drop beyond the prior (control − method, seed mean ± SD) and the share of uncued answers on the most frequent letter of the method (seed mean, with the letter per seed). † marks an exploratory variant.
method
helpful drop
harmful drop
ratio [95% CI]
rel. red. front
pre
post
end
Gemma-2-2B-it (original model’s harmful-cue following, front 0.493, pre 0.463, post 0.746, end 0.878)
BCT
0.384 ± 0.057
0.304 ± 0.077
1.26 [1.08, 1.49]
0.951
0.885
0.403
0.180
ACT
0.309 ± 0.004
0.252 ± 0.002
1.23 [1.10, 1.38]
0.920
0.791
0.127
0.093
AttCT
0.280 ± 0.006
0.212 ± 0.006
1.32 [1.16, 1.51]
0.912
0.784
0.064
0.025
RMCT
0.416 ± 0.048
0.377 ± 0.035
1.10 [0.96, 1.28]
0.924
0.918
0.542
0.382
Llama-3.1-8B-Instruct (original model’s harmful-cue following, front 0.381, pre 0.403, post 0.605, end 0.638)
Appendix
Table 9: Helpful cues drop together with harmful ones, and the fix fades when the cue moves after the options. Cue battery, pooled over six wordings and four positions. Paired drops in helpful- and harmful-cue following (seed mean ± SD), their ratio with a 95% question-bootstrap interval (“–” where the harmful drop exceeds .02 in fewer than 97.5% of samples), and the relative reduction of harmful-cue following at each position (only front is trained).
method
absolute [95% CI]
proportional [95% CI]
log-odds [95% CI]
Gemma-2-2B-it
BCT
1.26 [1.08, 1.49]
0.98 [0.89, 1.09]
1.30 [1.10, 1.54]
ACT
1.23 [1.09, 1.37]
0.99 [0.93, 1.06]
1.35 [1.18, 1.56]
AttCT
1.32 [1.15, 1.51]
1.06 [0.97, 1.15]
1.49 [1.26, 1.75]
RMCT
1.10 [0.95, 1.28]
0.87 [0.78, 0.97]
1.10 [0.93, 1.29]
RMCT+anchor †
1.04 [0.94, 1.15]
0.83 [0.78, 0.88]
1.08 [0.96, 1.22]
Appendix
Table 10: Selectivity on three scales. Each cell is the seed-mean helpful drop divided by the seed-mean harmful drop on the paired item sets, over all positions, with a 95% question-bootstrap interval (B = 2000, and “–” where the harmful drop exceeds .02 in fewer than 97.5% of samples). Exploratory. † marks an exploratory variant.
Table 11: Helpful-to-harmful ratio before the options (front and pre pooled) and at each position, with each cell on three scales (“–” on a scale where the harmful drop is below .02). Seed means on the paired item sets. Exploratory. † marks an exploratory variant.
method
correct cue
wrong cue
drop difference [95%]
log-odds gap
change [95%]
Gemma-2-2B-it
BCT
0.62 → 0.06
0.57 → 0.05
+0.043 [-0.028, 0.118]
+0.22 → +0.16
[-0.66, 0.55]
ACT
0.66 → 0.09
0.58 → 0.04
+0.028 [-0.033, 0.091]
+0.33 → +0.82
[-0.02, 1.05]
AttCT
0.66 → 0.12
0.59 → 0.08
+0.035 [-0.025, 0.097]
+0.33 → +0.48
[-0.26, 0.56]
RMCT
0.61 → 0.11
0.56 → 0.04
-0.022 [-0.094, 0.054]
+0.21 → +1.15
[0.33, 1.72]
gold-gated BCT
0.61 → 0.21
0.58 → 0.11
-0.074 [-0.153, 0.013]
+0.13 → +0.85
[0.18, 1.29]
Appendix
Table 12: Same questions, cue before the options. The table gives the following of a cue that names the correct answer and of a cue that names a wrong option on the same questions, original → trained (seed means). The questions are those that both the original and the trained model answer wrongly without a cue, and the option that the wrong cue names is neither model’s uncued answer. The last columns give the difference of the two drops and the change in their log-odds gap, with 95% question-bootstrap intervals. Exploratory.
strength and letter checks
training-family / novel wordings
method
matched: before / after
after coef.
end, non-correct
front
pre
post
end
Gemma-2-2B-it
BCT
0.93 / 0.46
-0.57
0.13
0.95 / 0.95
0.90 / 0.87
0.40 / 0.41
0.19 / 0.17
ACT
0.86 / 0.25
-0.66
0.08
0.92 / 0.92
0.80 / 0.78
0.11 / 0.15
0.06 / 0.12
AttCT
0.83 / 0.18
-0.72
0.02
0.90 / 0.93
0.79 / 0.78
0.04 / 0.10
0.00 / 0.05
RMCT
0.91 / 0.65
-0.30
0.28
0.92 / 0.93
0.91 / 0.93
0.57 / 0.51
0.41 / 0.35
Appendix
Table 13: Position checks (seed means, exploratory). Matched is the relative reduction of harmful-cue following in wording × position cells whose original harmful-cue following lies in the range shared by cells before and after the options. After coef. is the after-the-options coefficient of a regression of relative reduction on original harmful-cue following and position over the 24 cells. End, non-correct is the end-position relative reduction when every answer other than the correct one counts as following the cue. The right part gives the relative reduction by position for the three training-family wordings and the three novel wordings. † marks an exploratory variant.
relative reduction of trained-cue flip ∣ knew
method
front (trained)
pre
post
front − post [95%]
Gemma-2-2B-it
original flip ∣ knew ( n )
0.303 (33)
0.467 (30)
0.686 (35)
–
BCT
0.82
0.98
0.60
+0.22 [-0.077, 0.475]
ACT
0.80
0.78
0.25
+0.55 [0.204, 0.843]
AttCT
0.83
0.82
0.14
+0.69 [0.405, 0.947]
Appendix
Table 14: The trained cue of the test set (Table 6 ) split by the cue position of each source row (post hoc, exploratory). The table gives the original flip ∣ knew per position with the number of known items in parentheses, the relative reduction of flip ∣ knew per position (seed means), and the front − post gap with a 95% question-bootstrap interval (B = 2000, with the same resampled questions for the original model and every seed). † marks an exploratory variant.
checkpoint (seed 42)
harmful drop
helpful drop
ratio abs.
prop.
log-odds
rel. red. front
pre
post
end
Gemma
BCT, selected LR
0.382
0.446
1.17
0.93
1.16
0.97
0.90
0.55
0.29
BCT, LR 3.16e-5
0.127
0.231
1.82
1.42
2.04
0.87
0.72
-0.10
-0.05
BCT, LR 3.16e-4
0.350
0.449
1.28
0.99
1.28
0.95
0.92
0.61
0.25
ACT, selected LR
0.249
0.304
1.22
0.98
1.35
0.92
0.78
0.13
0.09
ACT, LR 3.16e-6
0.210
0.244
1.16
0.93
1.32
0.79
0.69
0.10
0.08
Appendix
Table 15: Robustness of the pattern to the learning rate and the training data (exploratory, seed 42). The table gives the paired drops on the cue battery, the helpful-to-harmful ratio on three scales and the relative reduction of harmful-cue following by position. It covers the selected learning rate, its non-degenerate neighbors, and the Llama checkpoints trained with the authors’ own data and configuration.
method
circuit [Wilson 95%] (recovered / flips)
writers
readers
circuit, trained-cue flips ( n )
best random set
Gemma-2-2B-it
original
0.818 [0.780, 0.850] (390/477)
0.71
0.26
0.82 (50)
0.07
BCT
0.843 [0.817, 0.866] (694/823)
0.78
0.26
0.97 (30)
0.06
ACT
0.844 [0.821, 0.864] (933/1106)
0.75
0.25
0.92 (72)
0.06
AttCT
0.861 [0.840, 0.880] (981/1139)
0.75
0.27
0.88 (73)
0.07
RMCT
0.910 [0.885, 0.931] (559/614)
0.87
0.43
0.90 (29)
0.12
Appendix
Table 16: Patch recovery. Each cell is the share of a method’s remaining cue-induced flips that return to the correct answer when the uncued answer-position activations of a head set are patched in, pooled over seeds (Gemma three seeds, Llama seed 42). The head sets are the circuit (with a Wilson 95% interval), its writer and reader subsets (the Llama writer subset is L17.H24 alone), the circuit on trained-cue flips only, and the best random head set (the maximum over three layer-matched sets and, for Gemma, over seeds). The Gemma circuit column and the subset and trained-cue columns are exploratory.
rel. change, labeled items
drop on followed trained-cue items
method
trained cue
held-out cues
fixed
still followed
fixed / gap
n fixed / still
OV depth centroid
BCT
-0.101 ± 0.153
-0.21 +0.37 -0.26 -0.06 -0.25
14%
13%
0.27
37 / 13
19.9 ± 0.4
ACT
-0.139 ± 0.016
-0.05 -0.05 -0.04 -0.12 -0.09
20%
2%
0.29
27 / 23
16.9 ± 0.7
AttCT
-0.109 ± 0.010
+0.00 -0.02 -0.01 -0.05 -0.01
20%
1%
0.25
24 / 26
4.9 ± 0.3
RMCT
-0.110 ± 0.046
-0.08 -0.05 -0.04 -0.07 -0.11
10%
5%
0.19
40 / 10
20.4 ± 0.1
RMCT+anchor †
-0.128 ± 0.051
+0.15 -0.07 +0.01 -0.02 -0.09
–
–
–
–
7.6 ± 0.5
Appendix
Table 17: Early reader heads of Gemma. The left part gives the relative change of their cue-span attention on the labeled items of the original model (seed mean ± SD, with held-out cues in the order post hoc, distractor fact, wrong few-shot, distractor argument and squares). The middle part (exploratory) uses the trained-cue items that the original model follows. It gives the relative drop on the items the method fixes (its cued answer is no longer the cued letter) and on the items it still follows (seed means), the fixed-item drop as a share of the gap of the original model between items it follows and items it resists, and the item counts per seed. The right part gives the OV depth centroid of the LoRA edit over the attention projections (layer index out of 26). † marks an exploratory variant.
mediated fraction: binary / own fixes / probability
method
fixed by trained model
EARLY
CARRIER
CIRCUIT
LATE
D1
Llama-3.1-8B-Instruct
BCT
72
0.71 / 0.51 / 0.38
0.72 / 0.53 / 0.73
0.28 / 0.21 / 0.17
0.25 / 0.19 / 0.16
fails (CARRIER)
ACT
64
0.73 / 0.69 / 0.59
0.70 / 0.66 / 0.58
0.11 / 0.09 / 0.06
0.08 / 0.08 / 0.01
holds (EARLY)
AttCT
57
0.72 / 0.65 / 0.61
0.70 / 0.61 / 0.59
0.14 / 0.11 / 0.06
0.07 / 0.05 / 0.01
holds (EARLY)
Gemma-2-2B-it
Appendix
Table 18: Layer swap. Each cell is the forward mediated fraction when one block of the trained model’s q, k, v and o projections is copied into the original model (Llama EARLY 0–8, CARRIER 9–14, CIRCUIT 15–18 and LATE 19–31, Gemma 0–6, 7–12, 13–18 and 19–25), on items whose cue the original model follows. The three readouts are the pre-registered binary readout, the share of the trained model’s own fixes and the share of the fall in the probability of the cued letter. For Llama (seed 42), the table also gives D1 (the largest block is EARLY or CARRIER, and LATE is below 0.15). For Gemma, it gives seed means over three seeds, the range over seeds of the number of fixed items, and the number of seeds that meet the same criterion (exploratory). RMCT cannot be tested this way.
method
trained-cue
held-out
accuracy
helpful drop
harmful drop
CAI
rel. red. front
end
Gemma-2-2B-it
BCT
0.116
0.421
0.406
0.384
0.304
1.26
0.951
0.180
response-only BCT
0.045
0.193
0.444
0.652
0.565
1.15
0.980
0.852
Bayes-BCT ( r=.5 )
0.161
0.460
0.412
0.308
0.234
1.32
0.935
0.111
Bayes-BCT ( r=.8 )
0.202
0.504
0.398
0.276
0.191
1.45
0.866
0.077
gold-gated BCT
0.159
0.452
0.438
0.243
0.229
1.06
0.842
0.037
Appendix
Table 19: Changing the training data of BCT. Seed means for the BCT variants of Section 4.5 (defined in Appendix B ). Response-only BCT is run on Gemma only. CAI (content-awareness index) is the helpful drop divided by the harmful drop ( ∗ marks a harmful drop below .05, which the registered criterion does not count as selective). Rel. red. is the relative reduction of harmful-cue following at the front and end positions.
Table 20: Relapse under 125 steps of benign LoRA fine-tuning (generated answers, three seeds per method, ± is the seed SD). The relapse fraction is (method at step 125 − method at step 0)/(original at step 0 − method at step 0), and its seed 95% CI is a t interval over seeds. Beyond original drift subtracts the change of the original model under the same fine-tuning. Log-odds rise is the change in the logit of trained-cue flip ∣ knew from step 0 to 125, with its ratio to that of the original model in parentheses. The drift-adjusted and log-odds columns are exploratory. Held-out fractions are given only where the held-out gap of the method is at least .05. The held-out columns use the 16-token answers without the few-shot correction (as does the original held-out rate in the header), and the held-out fraction beyond drift subtracts the held-out change of the original model.
checkpoint
MMLU BRR ± SE
held-out BRR ± SE
MMLU clean acc.
Anthropic rate
Gemma-2-2B-it original
0.540 ± 0.018
0.331 ± 0.019
0.596 [0.565, 0.626]
0.848
Gemma-2-2B-it BCT
0.145 ± 0.019
0.319 ± 0.019
0.512 [0.481, 0.543]
0.466
Gemma-2-2B-it ACT
0.081 ± 0.017
0.160 ± 0.017
0.602 [0.571, 0.632]
0.838
Gemma-2-2B-it AttCT
0.131 ± 0.018
0.176 ± 0.018
0.596 [0.565, 0.626]
0.849
Gemma-2-2B-it response-only BCT
0.011 ± 0.016
0.076 ± 0.018
0.536 [0.505, 0.567]
0.651
Gemma-2-2B-it BCT sweep, LR 1e-5
0.543 ± 0.018
0.421 ± 0.019
0.594 [0.563, 0.624]
0.656
Appendix
Table 21: The evaluator of the AttCT authors on seed-42 final checkpoints (adapter unmerged, their code). The columns give the MMLU bias rate (BRR = P(cued answer ∣ cued) − P(cued answer ∣ clean), n = 1000) with its unpaired SE, the held-out BRR (n = 951), clean MMLU accuracy (Wilson 95%) and the Anthropic model-written sycophancy rate (n = 999). Gemma ACT and AttCT use the re-selected learning rates (Appendix B.4 ), and the Gemma BCT sweep rows are three other learning rates of the BCT grid. § marks the authors’ own data and configuration run through their pipeline. Values in parentheses are those that the AttCT paper reports for its Llama checkpoints.
LR
evals
training-bias switch rate
held-out switch rate
authors’ logs (distractor argument)
authors’ logs (held-out)
base
2
0.650 [0.567, 0.724]
0.419 [0.383, 0.456]
0.650
–
1e-4
2
0.497 [0.417, 0.576]
0.414 [0.378, 0.450]
0.325/0.346
0.438/0.414
2.86e-4
2
0.362 [0.284, 0.449]
0.375 [0.337, 0.414]
0.141/0.132
0.320/0.322
Appendix
Table 22: RMCT in the authors’ own regime (their trainer and paper configuration, chain-of-thought answers, evaluated with their protocol on 100 questions per pass). The table gives the switch rate toward the bias on the training bias and pooled over five held-out biases, for final-step checkpoints pooled over two evaluation passes. Next to these are the values recomputed from the evaluation logs that the authors released (one training run, two passes). The authors’ base value comes from their 400-question logs, and their held-out value for the second pass pools four biases, while ours pools five. A second training run at 2.86e-4 (seed 43) gives 0.350 and 0.309 at its final step. We evaluate with the authors’ vLLM pipeline on bf16-merged weights, which weakened RMCT in our setting (Appendix D.3 ). The direction reproduces, but the size does not at 2.86e-4.
Reasoning consistency in language models measures whether conclusions and reasoning remain reliable across equivalent prompts, samples, steps, and contexts.
Language models often abandon a correct answer when the prompt contains a cue, a hint toward a wrong option such as "I think the answer is B." Sycophancy and related biases show this failure. They are often studied separately and blamed on alignment. We test this view causally, tracing the attention heads behind these biases through the stages of two post-training pipelines. In multiple-choice tests on most models, four cue types rely largely on the same few heads, a shared circuit already present in pretrained models. Most base models follow cues in plain text. Fine-tuning a base model on about 200 generic assistant examples without cues activates these heads in chat format, so cue-following can be a side effect of assistant tuning. The final reinforcement learning (RL) stage strengthens the effect of cues in almost every comparison. In Tulu-3-8B and OLMo-2-7B, we swap the attention patterns of six middle layers between the RL checkpoint and its predecessor. For suggested-answer cues, this shifts each checkpoint's cue effect most of the way to the other's level. Post-training thus seems to change when and how strongly models use this circuit. Mitigation for these multiple-choice biases could target this circuit.
Prakhar Gupta, Terry Jingchen Zhang, Zhijing Jin
University of Michigan · Jinesis Lab, University of Toronto & Vector Institute · University of Oxford +2
Large language models are often influenced by extraneous input features, such as cues revealing a user's preferred answer. Consistency training reduces this influence by training models to behave similarly across inputs with and without the extraneous feature. However, existing methods train for consistency over entire responses or internal activations, which also constrains whether the model verbalises said extraneous features. We show this leads to obfuscation, where the model learns not to mention a cue while remaining influenced by it, which may undermine monitorability. To address this, we introduce Rate Matching Consistency Training (RMCT), which trains for consistency over selected behavioural properties without constraining how this behaviour is expressed. RMCT matches the rate at which the model exhibits a target behaviour (e.g., following a bias cue) across input perturbations, rather than requiring paired inputs with and without the extraneous feature, extending consistency training to settings where the extraneous features cannot be removed. We evaluate RMCT on sycophancy reduction in two open-weight language models, achieving reductions in bias-following comparable to a standard consistency-training baseline on held-out bias types, while largely preserving the model's tendency to verbalise the bias cue. Further, we find that RMCT is more data-efficient at the expense of being less compute-efficient in our experiments. Overall, RMCT shows that consistency training can improve behavioural robustness without directly trading off against monitorability.
Sohaib Imran, Prakhar Gupta, Jannes Elstner +1
Independent · University of Michigan · UK AI Security Institute
In this paper, we study how training data creates associations between the tokens at the start of a base model's response and the reasoning behavior that follows. First, we demonstrate that fixing particular starting token cues makes a base model's performance competitive with that of its reinforcement learning (RL)-trained counterparts on math and coding. For instance, the cue ".\n\nOkay" raises Olmo-3-7B's MATH-500 pass@1 accuracy from 42% to 78%, while "Alright," raises Qwen3-14B's from 72% to 87%. Second, RL makes these cues more likely, while fixing them recovers much of its performance gain over the base model. Third, we trace the reasoning effects of token cues to the training data. We perform causal data interventions to turn an arbitrary word, such as "chicken", into an effective reasoning cue, or remove an existing cue's effect. A similar edit makes the prompt instruction "Think duck duck goose" as effective as "Think step by step" at eliciting reasoning. We also find that the hidden state representations induced by different cues correlate with different document types from the training set. Finally, we extend our study of token cues with a case study in language model safety, finding that different cues elicit distinct refusal and compliance behaviors that correspond to different types of training data.