cs.CLOct 2, 2026
SaveCopying Before Suppression: What Drives a Below-Chance Dip During Language Model Training?
Organizations: University of Chicago · Independent Researcher
Abstract
Mechanistic interpretability usually studies fully trained models, yet the computations that drive a behaviour can change while the model is still learning the task. On the Indirect Object Identification task, a model should continue with the name mentioned once rather than the name mentioned twice. Pythia models pass through an early training window in which they prefer the repeated name, so accuracy in a choice between the two names falls below one half while language-model loss on a fixed text sample keeps decreasing across the same window. The window reflects a temporary imbalance between two computations. We identify one cause of the wrong preference by selecting a set of attention heads that write the repeated name, on prompts separate from those used for causal evaluation, keeping that selection fixed, and then replacing each head's final-token output with its average output on a separate set of non-repeated-name prompts. This improves the correct-minus-repeated logit difference in a separately trained 160M model and in the official 160M, 410M, and 1B models. At 160M, the head that lowers the repeated name in the mature model shows little of its mature behaviour at this point. It directs less than one percent of its attention to the repeated mention, and its output makes almost no direct contribution to lowering that name's logit. Both properties grow over the interval in which behaviour recovers. Across the 160M, 410M, and 1B models, transplanting the corresponding head's mature parameters into the early checkpoint recovers 35 to 68 percent of the total improvement in the correct-minus-repeated logit difference seen by the end of training. Related early-to-late reversals appear at further Pythia scales, in two independently trained GPT-2 models, and in OLMo. A mature circuit can therefore conceal a transient causal configuration that shaped behaviour earlier in training.
Figures & tables
Figure 1: The below-chance window. Panel A shows forced-choice accuracy at three model sizes falling below the chance level of one half and then recovering. Panel B shows the mean logit difference over the same interval. Panel A counts the fraction of prompts with positive logit difference, whereas Panel B averages its signed magnitude. The full-vocabulary output at the floor is shown in Figure 8 .
Figure 2: Position controls at Pythia 160M. Replacing the residual-stream state at the second subject mention reverses sign between the floor and maturity for both the matched and random replacements. The tested interventions at the first subject mention, indirect object, and structural control position do not reverse sign.
Figure 3: Writing the repeated name is causally harmful at the floor. Replacing each frozen head’s prompt-specific output at the final token with its control mean improves the correct minus repeated logit difference at all four evaluated floors. Bars are template-to-item nested intervals. The three-name margin is shown in Figure 18 .
Figure 4: The suppressing head develops during recovery. Panel A shows that at the 160M floor the head places under one percent of its final-token attention on the repeated mention, rising sharply as behaviour recovers. Panel B shows that the head’s direct output projection on the repeated name is close to zero at the floor and becomes strongly negative later. Panel C shows that transplanting the mature head’s parameters into the early checkpoint moves the logit difference a substantial fraction of the way to the mature value at all three sizes. The effect of replacing this head’s output in the mature model is shown in Figure 31 .
Appendix figures & tables50 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: The task and the account in one picture. The upper line shows a prompt with the indirect object mentioned once, the subject mentioned twice, and the position where the model must answer. The lower panels summarise what this paper establishes. Early in training the heads that copy the repeated subject are already causally effective while the head that later suppresses it is close to inert, and the model falls below chance. Later the suppressing computation grows and behaviour recovers.
Figure 6: Language-model loss and IOI forced-choice accuracy around the 160M floor. Loss decreases from steps 1000 to 5000, while IOI accuracy first falls and then recovers. The dashed line marks chance-level forced-choice accuracy.
Figure 7: Language-model loss on the same fixed text sample for the released Pythia 160M model and the separately trained 160M model. Loss decreases across the evaluated checkpoints in both runs.
| Run | Floor acc. | Native LD | Edited LD | LD |
| seed1 | 30.7% | |||
| seed3 | 14.7% | |||
| seed5 | 25.3% | |||
| data-seed1 | 30.7% | |||
| data-seed2 | 19.3% | |||
| data-seed3 | 36.3% |
Table 1: Results for the nine PolyPythia 160M variants. Floor accuracy is below the forced-choice baseline for every run. Native and edited LD are measured before and after removing the repeated relation.
Figure 8: Full-vocabulary output at the 160M floor. The repeated name receives about twice the probability of the correct name, while greedy decoding produces neither name on almost all prompts.
Figure 9: Vocabulary rank across training at 160M. Median rank of the repeated and correct names across checkpoints, on a logarithmic scale, with rank 1 the highest-scoring token, so a lower rank indicates greater model preference. The repeated name ranks ahead around the floor and the correct name overtakes it during recovery.
Figure 10: Repetition sensitivity across nine PolyPythia 160M runs. It shows change in logit difference at each run’s floor after replacing the second occurrence of the repeated name with a different name. All nine effects are positive. These runs are grouped by whether they vary the random seed, the data order, or the parameter initialisation.
Figure 11: Residual-stream replacement at the second mention, in two variants. Neither edits the evaluated prompt. Only the activation carried at that position is replaced, after each block in an early layer band. Panel A takes the replacement from a matched control prompt that differs only in the name at the second mention, so the inserted state carries another identity. Panel B instead samples a fresh independent direction for every prompt and every layer, scaling each to the norm of the state it replaces, so no name from any prompt enters. Comparing the two separates disruption of the state from content imported with it.
Figure 12: Position controls. The norm-matched random replacement of Figure 11 is applied separately at four sites in the same sentence. Values are the change in logit difference at the 160M floor and at maturity. The structural control is defined positionally, as the token immediately following the first subject mention, so the word marked in this example sentence varies across sentence templates. Replacement at the first mention increases the logit difference at both checkpoints, whereas replacement at the indirect object and at the structural control decreases it at both. Only replacement at the second mention changes sign. This is what makes the second-mention state specific rather than one of several positions carrying a causal effect.
| Position and replacement | Floor | Mature |
| Second mention, matched | ||
| Second mention, random | ||
| First mention, random | ||
| Indirect object, random | ||
| Structural position, random |
Table 2: Change in logit difference from state replacement at each position, at the 160M floor and mature checkpoints.
Figure 13: Input token edit. The second occurrence of the repeated name is replaced in the token sequence itself, and the model then runs normally. Because repetition status and token identity change together, a positive effect is evidence that the model is sensitive to the repeated relation, not an isolation of repetition alone. This is the only manipulation in the paper that alters the evaluated prompt.
Figure 14: Decomposition of the input edit at the 160M floor.
Figure 15: How the prompts are divided. After the head set is frozen, the remaining names and sentence templates are split into a control-mean set of non-repeated prompts, used only to build one fixed mean output per head, and an evaluation set of repeated-name prompts on which the effect is measured. The two share no names and no sentence templates, so the measured effect cannot borrow from the prompts that produced the replacement.
Figure 16: Selecting and freezing the copying heads. Four prompt resources play four distinct roles: a selection set for the candidate screen, a validation split held out from that screen, a control-mean set used only after freezing to build the replacement vectors, and an evaluation set. Candidate heads are kept when replacing each on its own improves both the three-name margin and the correct-name probability. The survivors are then ranked on a validation split that took no part in that screen, and the resulting set is frozen and never reselected. Table 3 lists the frozen set for every model. Only afterwards is the intervention effect measured, on evaluation prompts whose names and sentence templates never influenced selection.
| Setting | Control mean | Evaluation | Frozen head set | |
| retrained 160M | 64 | 192 | 12 | L2H11, L5H0, L5H7, L6H8, L7H1, L8H2, L9H1, L9H10, L10H4, L10H10, L11H10, L11H11 |
| official 160M | 64 | 192 | 21 | L2H10, L3H10, L4H0, L4H6, L4H8, L4H11, L5H5, L5H9, L6H0, L6H6, L7H6, L7H10, L8H1, L8H2, L9H3, L9H6, L9H7, L9H8, L10H1, L10H7, L11H11 |
| official 410M | 64 | 192 | 17 | L11H8, L19H10, L17H6, L13H5, L16H15, L12H4, L17H13, L10H14, L16H2, L14H7, L14H0, L20H4, L10H0, L10H9, L16H7, L11H14, L11H1 |
| official 1B | 64 | 192 | 22 | L5H6, L6H7, L9H6, L6H4, L7H1, L4H0, L5H7, L8H5, L11H2, L15H1, L11H1, L14H2, L5H2, L14H5, L15H0, L12H2, L4H1, L13H3, L3H1, L3H4, L13H4, L4H5 |
Table 3: Sizes in items and the frozen head sets. An item is a name pair evaluated in both orders. The control-mean half and the evaluation half share no names and no templates. Heads are written as layer and index, and the sets are screened, then ranked on a separate validation split, then frozen before the mean is estimated or the effect is measured.
Figure 17: Control-mean replacement of the copying heads. The replacement is one fixed vector per head, obtained by averaging that head’s output at the final token over many non-repeated control prompts spanning several names and both name orders. It is therefore not an activation borrowed from a single donor prompt, and it carries no particular competing name. On a separate evaluation prompt in which a name does repeat, each frozen head’s own output is replaced by its fixed mean while every other head, position and parameter is left alone.
| Setting at floor | ||
| retrained 160M | ||
| official 160M | ||
| official 410M | ||
Table 4: Effect of the control-mean output replacement, in which each frozen head’s prompt-specific output at the final token is replaced by its average output over non-repeated control prompts, with nested resampling intervals. Positive means the preference for the correct name improved.
Figure 18: Change in the three-name margin of equation 2 from the control-mean replacement. Filled circles indicate that the template-to-item nested interval lies above zero and open squares that it crosses zero. Under item-clustered resampling all four intervals lie above zero.
Figure 19: Both measures under both resampling procedures. Item-clustered intervals are narrower because the nested procedure additionally resamples a small number of template groups.
Figure 20: Source-position value replacement. The evaluated prompt and the identity at the second mention remain unchanged. For each frozen head, only the value vector read at the first mention is replaced by the corresponding value from a prompt carrying a different name there. If the heads merely wrote toward whichever name repeats, their downstream contribution would be unaffected; if it instead follows the substituted identity, the heads are carrying the name represented at their source. This distinction is what the term copying head refers to.
| Setting | Heads | Repeated | Substituted | Three-name margin |
| retrained 160M | 12 | |||
| official 160M | 21 | |||
| official 410M | 17 | |||
| official 1B | 22 |
Table 5: Component logit changes from replacing only the frozen ensemble’s value vectors at the first subject mention, holding the second mention fixed. All four settings use the same source-position value-replacement protocol. The repeated-name logit falls and the substituted name’s logit rises by a comparable amount at every setting, and the three-name margin improves at every setting. Table 6 gives that change under both uncertainty procedures.
| Setting | Change | Item-clustered | Template-nested |
| retrained 160M | |||
| official 160M | |||
| official 410M | |||
| official 1B |
Table 6: Change in the three-name margin from replacing the frozen ensemble’s value vectors at the first subject mention, under both uncertainty procedures. Every interval lies above zero. This experiment carries six to eight sentence templates, compared with three or four for the control-mean experiment, which is why its nested intervals widen less relative to the item-clustered ones.
Figure 21: The two computations measured on one scale. Each curve is the change in the three-name margin when the same intervention is applied to one of the two frozen head sets, with shaded 95 percent intervals. The left panel uses the retrained 160M model with dense checkpoints through the transition and the right panel uses the released 160M model over full training. A positive value means replacing that set improved behaviour, so the native set was working against the correct answer. A negative value means replacing it damaged behaviour, so the native set was helping.
| Setting | Measure | Point estimate | Item-clustered interval | Nested interval | Resolves under nested |
| retrained 160M | two-name | yes | |||
| three-name | no | ||||
| official 160M | two-name | yes | |||
| three-name | yes | ||||
| official 410M | two-name | yes | |||
| three-name | yes |
Table 7: Both measures under both resampling procedures, as 95 percent intervals from ten thousand draws.
Figure 22: The selected head set against two hundred feature-matched control sets, at the official 160M floor in the upper row and the retrained 160M floor in the lower row. The shaded band spans the full range of the two hundred control-set effects and the solid line marks their mean, with the diamond marking the selected set. The control ranges are to and to at the official model, and to and to at the retrained model. The separation is wide at the official model and narrow at the retrained model.
Figure 23: Passive attention split. The total attention a copying head places on the two subject mentions is preserved and divided equally between them, leaving the value vectors untouched. This is neither an ablation nor a value replacement, since the same mass is present and only its allocation changes. The donor attention row is taken, the mass it places on the two subject positions is divided equally between them, and the modified row is written back; every other entry of that row is the donor’s. All seven point estimates satisfy an absolute difference below , although some uncertainty intervals extend beyond it, so this is a small-effect comparison rather than a demonstration of equivalence.
Figure 24: Interaction term for the repeated-subject logit at each floor setting. An amplification mechanism predicts a positive value.
Figure 25: Removing the linearly decoded duplication direction at the 160M floor degrades accuracy in proportion to the removal strength.
| Setting | Mean | Item interval | Nested interval |
| 160M floor | |||
| 410M floor | |||
| 410M recovery | |||
| 410M mature | |||
| 1B floor | |||
| 1B recovery |
Table 8: Paired difference between the native-routing effect and the passive-split effect on the three-name margin. A negative value means native routing is more protective than the mechanical split, which is the direction an additional learned correction would predict. Six of the seven settings are positive, and all seven are smaller in size than the analysis bound of .
Figure 26: Paired differences between the native-routing effect and the passive-split effect, with the analysis bound shaded. Negative values would indicate native routing is more protective than the mechanical split.
Figure 27: Mature-head mean replacement, and how it differs from the control mean of Appendix D . Here the selected head’s activation is replaced by its own mean over the prompts and token positions of each sentence-template block being evaluated, so the mean comes from the evaluation data itself. In the copying experiment the mean is instead built from a separate set of non-repeated control prompts. The two operations share a word but not a data source, and they answer different questions. Because this head was also chosen by its replacement effect, the size of that effect is selection-conditioned.
Figure 28: Activation graft and parameter transplant. Both move something from the mature checkpoint into the floor model, but they move different objects. The graft copies the head’s realised output on the evaluated prompt, so the floor model does not recompute it. The transplant copies the head’s query, key, value and output weights together with its head-specific biases, after which the floor model runs normally and the head computes a fresh output from its own upstream representations. The graft asks whether the mature output is sufficient on these prompts; the transplant asks whether what the head has learned is sufficient inside the early network.
Figure 29: Cumulative reconstruction at 410M. Mature parameters are transplanted into the floor model cumulatively, beginning with the suppressing head alone and then adding the strongest correct-name writers in order. The result covers selected cumulative prefixes rather than every intermediate count: measurements exist at zero, one, two, three, five and eight writers. The suppressing head alone closes percent of the floor-to-mature gap and all eight writers together reach percent, at forced-choice accuracy . The experiment is available at this size only.
Figure 30: Magnitude amplification. The floor head’s own output is multiplied by two, four, eight and sixteen, with no mature parameters introduced. If the floor head were already performing the mature operation only weakly, scaling it should move behaviour toward maturity. Accuracy remains below chance at every multiplier, which is why the transplant result is read as supplying a changed computation rather than additional strength.
Figure 31: Replacing the suppressing head’s prompt-specific output with its mature-head mean, taken over the prompts and token positions of each sentence-template block. This is not the control mean of Appendix D . The logit difference falls substantially at all three sizes while forced-choice accuracy remains well above chance, so the head is a substantial contributor to the mature preference rather than solely responsible for it.
Figure 32: Injecting the mature head’s activations into the floor checkpoint, with an output-norm-matched control head as control.
Figure 33: Upstream dependence of the mature suppressing head. Panel A replaces the first feed-forward block’s output with its matched non-repeated counterpart and measures the suppressor’s attention to the second mention, which falls at all three sizes. Panel B isolates only the direct connection between the two, adding the first-block difference to the suppressor’s downstream input while holding every other upstream contribution fixed; that route carries none of the attention dependence and well under a tenth of a percent of the logit dependence. The influence therefore reaches the head through intermediate computation, drawn as an unresolved region because the experiments do not identify which components form the path.
| Checkpoint | Attention to | Direct output projection on |
| step 512 | ||
| step 1000 | ||
| step 2000 (floor) | ||
| step 3000 | ||
| step 5000 | ||
| step 143000 (mature) |
Table 9: The mature-defined suppressing head traced backward at 160M. It is barely engaged by these diagnostics at the floor; they are descriptive, not causal.
Figure 34: The head identified at maturity, traced backward at all three sizes. The upper row shows attention from the final token to the second subject mention and the lower row shows the effect of its output on the repeated-subject logit.
Figure 35: Parameter transplant of the single suppressing head against a bundle of eight randomly chosen heads. The comparison is asymmetric in the number of heads transplanted.
Figure 36: First feed-forward block replacement, also called the layer-0 MLP. The evaluated prompt is not edited. A matched control prompt in which no name repeats is run first, the output of that one module is cached, and the cached activation is inserted at the same site in the target run before the forward pass continues. Because the whole module output is replaced, the experiment does not isolate a duplication-specific feature: it establishes the first feed-forward block as an important early source of repetition-sensitive state rather than as a duplication detector.
| Model | Native floor | Change | 95% interval |
| Pythia 70M | |||
| Pythia 160M | |||
| Pythia 410M | |||
| Pythia 1B | |||
| Pythia 1.4B |
Table 10: Change in the floor logit difference when the first feed-forward block’s output is replaced by its matched non-repeated counterpart. The effect is positive at all five sizes and every interval lies above zero.
| Setting | Early state | Intervention | Early result | Mature result | Strongest supported conclusion |
| Pythia 70M to 1.4B | negative at all five floors | first feed-forward block replacement | positive at all five | not evaluated here | early repetition-sensitive computation is harmful across scale |
| Pythia 2.8B | acc. , | input edit | input-level sensitivity to repetition reverses | ||
| GPT-2 alias | at step 1500 | early-block replacement; future-suppressor replacement | ; | ; | early harmful computation precedes mature suppression |
| GPT-2 battlestar | at step 1500 | as above | ; | ; | ordering recurs with a different suppressor head |
| OLMo-1B | acc. , | second-mention residual-state replacement | state-level causal role reverses outside Pythia and GPT-2 |
Table 11: Extensions across scale and model family. Every entry is the effect of the intervention on the logit difference, so a positive value means the replacement moved the model toward the correct name and a negative value means the native component was helping. Interventions differ across rows, so their magnitudes are not directly comparable. The final column states only the conclusion supported by the intervention in that row.
Figure 37: Prompt structures for the subject-verb agreement and factual-recall tasks. The subject-verb agreement prompts place an attractor of the opposite number between the subject and the verb position, with the preposition or relative clause varying across templates. The factual-recall prompts state one fact, interpose a second, and then return to the first subject, so the correct continuation is the object belonging to the repeated subject.
| Task | Early behaviour | Intervention | Early result | Mature result | Strongest supported conclusion |
| Naming | below chance at 160M, 410M and 1B | control-mean copying-head output replacement | to | separate suppression experiments | full copying-to-suppression account in the primary models |
| Greater-Than | below chance at 160M and 1B; at 410M | residual-state replacement at | to | to | causal role of the year state reverses |
| Factual recall | dips to | mature-head mean replacement | near zero through the dip | replacement effect | behavioural dip with a later substantial suppressing component |
| Subject-verb agreement | no corresponding dip established | matched residual-state replacement | no reversal; negative case |
Table 12: Task extensions. Each row uses a task-specific intervention and outcome, so effect magnitudes should not be compared across tasks. The final column gives the strongest conclusion supported within that task.
Figure 38: Controlled task injection. Models are trained from scratch on ordinary text into which task-structured sequences are substituted at a controlled rate, and each checkpoint is then evaluated twice: on names never injected during training, and on names drawn from the injection pool. The gap between the two separates a failure to generalise to unseen names from a failure to learn the task at all.
Figure 39: Held-out accuracy for the four injection rates. The left panel shows the main run and the right panel two further seeds, with seed 43 solid and seed 44 dashed.
Figure 40: Final accuracy on the two evaluations for each injection rate in the main run.
Figure 41: Ten runs of the minimal task trained from scratch. All ten show the dip and nine show the sign reversal.
Figure 42: Two models trained on ordinary web text. Panel A shows forced-choice accuracy, panel B the logit difference, and panel C the change produced by replacing the residual-stream state at the second occurrence of the repeated name with the corresponding state from a matched control prompt. The evaluated prompt tokens are unchanged. Run A covers 200000 steps and run B covers 105000 steps.
More in this topic
Circuits in Language Models
Circuits in language models are interacting internal components that implement behaviors and can be traced with causal interventions.
See all 122 papers