cs.CLOct 16, 2026
SaveMany Biases, One Circuit: Post-Training Tunes a Pretrained Cue-Routing Mechanism
Organizations: University of Michigan · Jinesis Lab, University of Toronto & Vector Institute · University of Oxford · EuroSafeAI ·
Abstract
Language models often abandon a correct answer when the prompt contains a cue, a hint toward a wrong option such as "I think the answer is B." Sycophancy and related biases show this failure. They are often studied separately and blamed on alignment. We test this view causally, tracing the attention heads behind these biases through the stages of two post-training pipelines. In multiple-choice tests on most models, four cue types rely largely on the same few heads, a shared circuit already present in pretrained models. Most base models follow cues in plain text. Fine-tuning a base model on about 200 generic assistant examples without cues activates these heads in chat format, so cue-following can be a side effect of assistant tuning. The final reinforcement learning (RL) stage strengthens the effect of cues in almost every comparison. In Tulu-3-8B and OLMo-2-7B, we swap the attention patterns of six middle layers between the RL checkpoint and its predecessor. For suggested-answer cues, this shifts each checkpoint's cue effect most of the way to the other's level. Post-training thus seems to change when and how strongly models use this circuit. Mitigation for these multiple-choice biases could target this circuit.
Figures & tables
Figure 1: Our account of where cue-following comes from. Panels 1 to 3 summarize the multiple-choice results of Sections 3 to 5 . R and W mark reader and writer heads (§ 2 ), and B marks the cued wrong option. Panel 3 is schematic. Its swap uses suggested-answer cues in the DPO and RL checkpoints of Tulu-3-8B and OLMo-2-7B, in the six layers just below the reader heads.
Figure 2: Copying the base model’s query and output weights barely changes caving, and a little generic tuning is enough to reach SFT-level caving. (a) The random-heads bars average three random sets of six heads. Gemma-2-2b-it is tested on its own flips, so its caving starts at 1.00. (b) LoRA fine-tuning of the Llama-3.1-8B base, tested in chat format. At 200 examples, the marks show the mean and range of three data samples. Reader attention averages the three readers and partly reflects the chosen answer. The runs answer fewer uncued prompts correctly than the SFT model (0.48 to 0.54 at 200 examples, compared with 0.73).
Figure 3: Caving, and how prompt format distorts comparisons of base and aligned models. (a) Aligned models in their chat format, on items answered correctly without the cue. (b) Change in caving from base to aligned model on matched items. Chat format is the usual comparison. Both base and aligned models can read plain text. Filled markers are significant (exact McNemar test, , Appendix C.4 ).
Figure 4: One activation direction reproduces the caving of the SFT model. Both panels use the Llama-3.1-8B base. (a) One direction is added at the answer position (the SFT direction at layer 12, the in-context direction at layer 14). The shifted base answers fewer uncued prompts correctly than the SFT model (0.40 compared with 0.79 on these items). (b) Flips caused by eight generic dialogue turns or three multiple-choice examples in the prompt, reversed by patching each head set. Random heads average 10 draws (at most 7%).
Figure 5: From DPO to the final RL stage, susceptibility rises, and reader attention to the cued letter rises with it. (a) Blue tests are significant after Bonferroni correction over the 13 tests (values in Table 5 ). (b) Orange marks , and the grey Tulu test misses the threshold of that we set in advance. The reader attention rate partly reflects the chosen answer. In OLMo, attention to the text of the cue itself falls (Appendix E.7 ).
Figure 6: Changing attention in the middle band moves most of the stage difference in both pipelines. is the fraction of the stage difference in held-out susceptibility that an intervention moves. A value of 1 means the other checkpoint’s level. Panels (a) and (b) show 95% CIs. (a) Random six-layer sets exclude the middle-band, reader, and writer layers, and dashed lines mark . (c) The scale is chosen only to match the other checkpoint’s reader attention. Layer sets that close at most 43% of the gap in reader attention are not shown (Table 8 ). The all-layers row is a separate run (Table 7 ).
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Test | Models / checkpoints | Control | Uncertainty | Outcome | |
| The circuit and its origin | |||||
| Top-8 head patching | Llama-3.1-8B, Qwen-2.5-7B, Gemma-2-9b (instruct) | random heads of the same number, redrawn per flip | 151 to 156 flips per model | random maximum | reverses 69% to 92% of flips (in Llama partly on selection flips), random 8-head sets at most 0.08 (App. C.1 ) |
| Leave-one-cue-out head selection | Llama-3.1-8B, Gemma-2-9b | 20 random 8-head draws | 32 to 40 flips per held-out cue | random mean | recovers 0.60 to 0.98, random 0.01 to 0.06 (App. C.2 ) |
| Weight copies (query and output projections) | Llama-3.1-8B base and Tulu-3-8B-SFT, Gemma-2-2b base and instruct | 3 random 6-head sets | 400 flip items (320 for the positive control) | — | aligned caving barely changed with base weights (Tulu 0.81 to 0.82 against 0.80). The copy into the base is weak evidence (the bases answer 3% and 17% correctly). Copying early-layer SFT weights into the base does raise caving (positive control) (§ 3 , App. C.3 ) |
| Layer knockout | 5 models (3 families), base (plain text) and aligned (chat) | base against aligned | 12 to 74 caving items per model | Wilson 95% CI, paired | removes 51% (base) and 54% (aligned) of caving, (§ 3 , App. C.3 ) |
| Comparison with format held fixed | 6 models (4 families) | base and aligned model on the same items in each format | 31 to 294 matched items per cell | exact McNemar | the chat-format increase reverses or disappears in 3 models once the format is fixed (App. C.4 ) |
Table 1: Evidence at a glance. “—” means that the test reports point estimates without an interval.
| family | example readers | writer hub | top-8 | top-16 | monitor AUROC |
|---|---|---|---|---|---|
| gemma-2-2b | L18.H6, L14.H2, L13.H6 | L17.H3 | 0.80 ∗ | — | 0.995 |
| gemma-2-9b | L20.H14, L24.H10, L26.H11 | (distributed) | 0.69 | 0.88 | 0.995 |
| llama-3.1-8b | L18.H28, L16.H19, L20.H14 | L17.H24 | 0.92 | 0.94 | 0.869 |
| qwen-2.5-7b | L22.H25, L21.H14, L22.H7 | L26.H26, L24.H27 | 0.85 | 0.96 | 0.911 |
| qwen-2.5-14b | (distributed) | (distributed) | 0.44 | 0.58 | — |
| olmo-2-7b | L20.H7, L18.H19, L21.H18 | L18.H19 † | — | — | 0.954 |
Table 2: Circuit summary for each family. Top- values are held-out recovery rates for patching the top heads (for Llama, two of the ten evaluation flips per cell of cue and dataset were also in the selection slice). ∗ The set of 7 circuit heads on suggested-answer flips that were not used for head selection (one random 7-head set gives 0.00). On the selection slice the set recovers 0.89 (bootstrap CI ), and 30 random draws of the same size give a mean of 0.04 and a maximum of 0.29. The monitor head is selected on a labeled training half (Gemma-2-9b L30.H9, Qwen-7B L23.H11, OLMo-2 L20.H7, Tulu-SFT L20.H14), and Gemma-2-2b and Llama use fixed heads found by patching. † In OLMo-2 the strongest head both reads and writes (single-head recovery 0.63 at the SFT stage and 0.53 at the instruct stage). The main persuasion decision heads of Sun et al. (2026) have the same indices as our hub heads, namely L17.H24 in Llama-3-8B-Instruct (the predecessor of the Llama-3.1 model we study) and L18.H19 in the same OLMo-2-1124-7B-Instruct checkpoint. Their Llama auxiliary decision heads include L15.H8 and L15.H11, two of our Llama writers.
| family | format | base | aligned | discordant | (McNemar) | |
|---|---|---|---|---|---|---|
| gemma-2-2b | plain text | 93 | 0.699 | 0.624 | 22/15 | |
| gemma-2-9b | plain text | 207 | 0.705 | 0.251 | 100/6 | |
| llama-3.1-8b | plain text | 173 | 0.751 | 0.584 | 36/7 | |
| qwen-2.5-7b | plain text | 254 | 0.508 | 0.193 | 87/7 | |
| qwen-2.5-14b | plain text | 294 | 0.248 | 0.078 | 55/5 | |
| olmo-2-7b | plain text | 142 | 0.331 | 0.430 | 14/28 |
Table 3: Caving rates for base and aligned models on matched items that both answered correctly without the cue, with format held fixed (Figure 3 b). We exclude the 21 suggested-answer items, of 476 in all, whose cue the loader removed. In chat format, four base models rarely answer correctly (uncued accuracy 0.11 to 0.16 on these items), and for three of them the comparison reverses sign. The chat-format cells for these families are small precisely because the base rarely answers correctly in that format. Scored from parsed greedy generations, the plain-text rows keep their direction. Base and aligned caving are 0.65 and 0.58 for gemma-2-2b ( ), 0.57 and 0.24 for gemma-2-9b, 0.81 and 0.58 for llama-3.1-8b, 0.35 and 0.15 for qwen-2.5-7b, and 0.25 and 0.08 for qwen-2.5-14b, each with except gemma-2-2b, and 0.37 and 0.44 for olmo-2-7b ( ). When scored from generations, these four base models answer 3% to 19% of uncued prompts correctly in chat format.
| Condition | Uncued acc. | Caving | Susceptibility | Flips (known) | Reader attention |
|---|---|---|---|---|---|
| Base, plain text | 0.48 | 0.83 | 0.77 (183) | 0.40 | |
| Base, chat, no prefix | 0.03 | 0.23 | 0.00 (11) | 0.00 | |
| Base, chat, 8 role turns | 0.25 | 0.62 | 0.46 (97) | 0.11 | |
| Base, chat, same text without roles | 0.14 | 0.33 | 0.00 (53) | 0.02 | |
| Tuned (SFT), chat | 0.56 | 0.68 | 0.63 (216) | 0.49 |
Table 4: Eight generic dialogue turns lose their activating effect when the role structure is removed (unselected items). The prefix without roles stays near the inactive level in susceptibility, known-item flips, and reader attention. Fixed-set susceptibility does not condition on correctness. Flip rates are over items answered correctly without the cue, with counts in parentheses.
| test | susc | ||
|---|---|---|---|
| graded 8B (Tulu) | 232 | ||
| graded 7B (OLMo) | 212 | ||
| graded 13B (OLMo) | 244 | ||
| graded 32B (OLMo) | 305 | ||
| published 8B (Tulu) | 241 | ||
| published 7B (OLMo) | 243 |
Table 5: The 13 stage tests. Each row gives the paired change in susceptibility ( ) from DPO to the final stage (RLVR for Tulu, Instruct for OLMo) on items that both stages answer correctly without the cue. Graded, arithmetic, and 70B rows average the four mid-strength cue wordings, and the arithmetic counts unique problems. Twelve of the 13 changes are positive, and 11 are significant after Bonferroni correction. No change in uncued confidence survives the correction. Also counting fused letter tokens such as “(B” shrinks the Tulu-8B published and held-out differences by a quarter to a third, but both stay significant after correction (Appendix B.3 ).
| susc ( ) | Tulu 8B | OLMo 7B | OLMo 13B | OLMo 32B |
|---|---|---|---|---|
| suggested answer | (0.6) | (5.0) | (1.4) | (2.5) |
| prior commitment | (1.7) | (5.2) | (2.2) | (4.1) |
| distractor fact | (3.7) | (7.1) | ( ) | (4.5) |
| wrong few-shot | (5.0) | (2.7) | (0.7) | (0.8) |
| held-out: distractor argument | (3.2) | (6.8) | — | — |
| held-out: spurious few-shot | (3.7) | (2.0) | — | — |
Table 6: Change in susceptibility from DPO to the RL stage for each cue type (paired, matched items, to per cell). Nineteen of the 20 cells are positive. The exception is OLMo-13B distractor fact ( ), and OLMo-13B shows smaller changes than the other models overall. Which cue types rise most differs by recipe.
| Pipeline | Direction | Chosen | Held-out attention rate (matched / target) | (95% CI) |
|---|---|---|---|---|
| Tulu DPO RLVR | RLVR down | 0.85 | 0.434 / 0.432 | 0.51 (0.30, 1.00) |
| DPO up | 1.3 | 0.468 / 0.477 | 0.80 (0.50, 1.52) | |
| OLMo DPO Instruct | Instruct down | 0.7 | 0.631 / 0.674 | 1.39 (1.11, 1.72) |
| DPO up | 1.7 | 0.737 / 0.714 | 1.23 (0.89, 1.62) |
Table 7: Attention matching transfers at least half of the stage difference in susceptibility. The scale is chosen on calibration items from attention alone. is the transferred fraction of the susceptibility difference between DPO and the final stage on held-out items. The held-out attention rate averages all 44 held-out items.
| Layer set | Tulu | Tulu | OLMo | OLMo |
|---|---|---|---|---|
| reader heads (3) | ||||
| readers + writers (5) | — | — | ||
| reader layers (all heads) | ||||
| middle band (6 layers) | (0.73, 1.35) | (0.69, 1.45) | (0.61, 0.93) | (0.53, 1.37) |
| low band (6 layers) | n.m. | n.m. | ||
| high band (6 layers) | n.m. | n.m. |
Table 8: Most of the stage difference transfers through the six layers directly below the lowest reader layer , the middle band (Tulu L10–15, OLMo L12–17). The table gives transferred fractions, with 95% CIs for the middle band. “—” means not run. “n.m.” means that no grid point brings the reader attention rate within 0.05 of the target, so the layer set receives no transfer claim. The natural attention gaps between the stages are 0.052 (Tulu) and 0.047 (OLMo), about the size of this tolerance. On the calibration questions, the middle band, reader layers, reader heads, and circuit heads close at least 66% of the gap (some overshoot it). Every control set closes 43% or less, so the controls give no matched comparison. means that the nearest grid point is the natural condition, because the layer set moves attention so little. That arm is then the natural run. marks a calibration miss of 0.052, just past the threshold. The random three-layer sets exclude only reader layers, and Tulu’s set (L4, L5, L10) includes the middle-band layer L10. The network-wide row repeats Table 7 (a separate run on 44 held-out questions per pipeline). It is the reference for the 0.6 criterion that we set in advance. The other rows come from the layer-set sweep, and the text reports a larger separate run at the reader heads.
Figure 7: The caving direction predicts flips better than the confidence direction in 11 of 12 model cells. Held-out AUROC, averaged over probed layers. The confidence direction is scored so that lower confidence predicts caving, as the confidence account expects, and it is ahead only in Qwen-2.5-14B tuned, by 0.0004.
Figure 8: Reasoning first removes most caving on arithmetic but much less on knowledge questions. Tulu-3-8B, pooled over DPO and RLVR, on items answered correctly in both answering modes, with identical cue wording (111 knowledge and 129 arithmetic presentations).
Figure 9: Scaling attention to the cue moves caving smoothly and also removes the help of a correct cue. Gemma-2-2b-it, with attention to the cue scaled by in every layer. (a) Caving as the rate of answering on 80 unselected wrong-cue MMLU items, including items that the model does not know. (b) Accuracy on 60 correct-cue items with the plain opinion cue.
Figure 10: Attention to the cued letter separates caving from resisting items on held-out questions in five models. (a) Gemma-2-2b’s fixed readers, on the cue types used for selection and on two held-out cue types. (b) One head selected per model on a labeled training half. Only the OLMo-2 and Tulu-SFT heads are designated readers. The measure partly reads the answer that is forming (Appendix B.5 ).
Figure 11: Across 207 open-weight conditions, caving falls close to one curve of the post-cue margin. An indicator for base against tuned adds , and the residual standard deviation is 0.12. The largest departures are uncued cells, where caving is zero by definition, and base-model arithmetic cells. Stars are the API models on the weak ladder, with one wording changed.
Figure 12: Attention to the cued letter tracks caving across 15 checkpoints, mostly through the difference between base and post-trained checkpoints. The x-axis uses the most cue-attending head in a fixed layer band and partly reflects the chosen answer. The ten post-trained checkpoints alone give ( ), and each family’s final checkpoint caves on nearly all of these items by construction.