Copying Before Suppression: What Drives a Below-Chance Dip During Language Model Training?

Tejas Dahiya; Cole Blondin

cs.CLOct 2, 2026

Copying Before Suppression: What Drives a Below-Chance Dip During Language Model Training?

Authors: Tejas Dahiya, Cole Blondin

Organizations: University of Chicago · Independent Researcher

Abstract

Mechanistic interpretability usually studies fully trained models, yet the computations that drive a behaviour can change while the model is still learning the task. On the Indirect Object Identification task, a model should continue with the name mentioned once rather than the name mentioned twice. Pythia models pass through an early training window in which they prefer the repeated name, so accuracy in a choice between the two names falls below one half while language-model loss on a fixed text sample keeps decreasing across the same window. The window reflects a temporary imbalance between two computations. We identify one cause of the wrong preference by selecting a set of attention heads that write the repeated name, on prompts separate from those used for causal evaluation, keeping that selection fixed, and then replacing each head's final-token output with its average output on a separate set of non-repeated-name prompts. This improves the correct-minus-repeated logit difference in a separately trained 160M model and in the official 160M, 410M, and 1B models. At 160M, the head that lowers the repeated name in the mature model shows little of its mature behaviour at this point. It directs less than one percent of its attention to the repeated mention, and its output makes almost no direct contribution to lowering that name's logit. Both properties grow over the interval in which behaviour recovers. Across the 160M, 410M, and 1B models, transplanting the corresponding head's mature parameters into the early checkpoint recovers 35 to 68 percent of the total improvement in the correct-minus-repeated logit difference seen by the end of training. Related early-to-late reversals appear at further Pythia scales, in two independently trained GPT-2 models, and in OLMo. A mature circuit can therefore conceal a transient causal configuration that shaped behaviour earlier in training.

Figures & tables

Figure 1: The below-chance window. Panel A shows forced-choice accuracy at three model sizes falling below the chance level of one half and then recovering. Panel B shows the mean logit difference over the same interval. Panel A counts the fraction of prompts with positive logit difference, whereas Panel B averages its signed magnitude. The full-vocabulary output at the floor is shown in Figure 8 .

Figure 2: Position controls at Pythia 160M. Replacing the residual-stream state at the second subject mention reverses sign between the floor and maturity for both the matched and random replacements. The tested interventions at the first subject mention, indirect object, and structural control position do not reverse sign.

Figure 3: Writing the repeated name is causally harmful at the floor. Replacing each frozen head’s prompt-specific output at the final token with its control mean improves the correct minus repeated logit difference at all four evaluated floors. Bars are template-to-item nested intervals. The three-name margin is shown in Figure 18 .

Figure 4: The suppressing head develops during recovery. Panel A shows that at the 160M floor the head places under one percent of its final-token attention on the repeated mention, rising sharply as behaviour recovers. Panel B shows that the head’s direct output projection on the repeated name is close to zero at the floor and becomes strongly negative later. Panel C shows that transplanting the mature head’s parameters into the early checkpoint moves the logit difference a substantial fraction of the way to the mature value at all three sizes. The effect of replacing this head’s output in the mature model is shown in Figure 31 .

Appendix figures & tables50 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: The task and the account in one picture. The upper line shows a prompt with the indirect object mentioned once, the subject mentioned twice, and the position where the model must answer. The lower panels summarise what this paper establishes. Early in training the heads that copy the repeated subject are already causally effective while the head that later suppresses it is close to inert, and the model falls below chance. Later the suppressing computation grows and behaviour recovers.

Figure 6: Language-model loss and IOI forced-choice accuracy around the 160M floor. Loss decreases from steps 1000 to 5000, while IOI accuracy first falls and then recovers. The dashed line marks chance-level forced-choice accuracy.

Figure 7: Language-model loss on the same fixed text sample for the released Pythia 160M model and the separately trained 160M model. Loss decreases across the evaluated checkpoints in both runs.

Run	Floor acc.	Native LD	Edited LD	$\Delta$ LD
seed1	30.7%	$-0.688$	$-0.106$	$+0.582$
seed3	14.7%	$-1.406$	$-0.343$	$+1.063$
seed5	25.3%	$-1.104$	$+0.075$	$+1.179$
data-seed1	30.7%	$-1.074$	$+0.176$	$+1.250$
data-seed2	19.3%	$-1.514$	$-0.161$	$+1.353$
data-seed3	36.3%	$-0.868$	$+0.161$	$+1.029$

Appendix

Table 1: Results for the nine PolyPythia 160M variants. Floor accuracy is below the

50\%

forced-choice baseline for every run. Native and edited LD are measured before and after removing the repeated relation.

Figure 8: Full-vocabulary output at the 160M floor. The repeated name receives about twice the probability of the correct name, while greedy decoding produces neither name on almost all prompts.

Figure 9: Vocabulary rank across training at 160M. Median rank of the repeated and correct names across checkpoints, on a logarithmic scale, with rank 1 the highest-scoring token, so a lower rank indicates greater model preference. The repeated name ranks ahead around the floor and the correct name overtakes it during recovery.

Figure 10: Repetition sensitivity across nine PolyPythia 160M runs. It shows change in logit difference at each run’s floor after replacing the second occurrence of the repeated name with a different name. All nine effects are positive. These runs are grouped by whether they vary the random seed, the data order, or the parameter initialisation.

Figure 11: Residual-stream replacement at the second mention, in two variants. Neither edits the evaluated prompt. Only the activation carried at that position is replaced, after each block in an early layer band. Panel A takes the replacement from a matched control prompt that differs only in the name at the second mention, so the inserted state carries another identity. Panel B instead samples a fresh independent direction for every prompt and every layer, scaling each to the norm of the state it replaces, so no name from any prompt enters. Comparing the two separates disruption of the state from content imported with it.

Figure 12: Position controls. The norm-matched random replacement of Figure 11 is applied separately at four sites in the same sentence. Values are the change in logit difference at the 160M floor and at maturity. The structural control is defined positionally, as the token immediately following the first subject mention, so the word marked in this example sentence varies across sentence templates. Replacement at the first mention increases the logit difference at both checkpoints, whereas replacement at the indirect object and at the structural control decreases it at both. Only replacement at the second mention changes sign. This is what makes the second-mention state specific rather than one of several positions carrying a causal effect.

Position and replacement	Floor	Mature
Second mention, matched	$+0.953$	$-4.123$
Second mention, random	$+0.920$	$-4.126$
First mention, random	$+2.152$	$+2.079$
Indirect object, random	$-3.536$	$-9.063$
Structural position, random	$-0.815$	$-4.450$

Appendix

Table 2: Change in logit difference from state replacement at each position, at the 160M floor and mature checkpoints.

Figure 13: Input token edit. The second occurrence of the repeated name is replaced in the token sequence itself, and the model then runs normally. Because repetition status and token identity change together, a positive effect is evidence that the model is sensitive to the repeated relation, not an isolation of repetition alone. This is the only manipulation in the paper that alters the evaluated prompt.

Figure 14: Decomposition of the input edit at the 160M floor.

Figure 15: How the prompts are divided. After the head set is frozen, the remaining names and sentence templates are split into a control-mean set of non-repeated prompts, used only to build one fixed mean output per head, and an evaluation set of repeated-name prompts on which the effect is measured. The two share no names and no sentence templates, so the measured effect cannot borrow from the prompts that produced the replacement.

Figure 16: Selecting and freezing the copying heads. Four prompt resources play four distinct roles: a selection set for the candidate screen, a validation split held out from that screen, a control-mean set used only after freezing to build the replacement vectors, and an evaluation set. Candidate heads are kept when replacing each on its own improves both the three-name margin and the correct-name probability. The survivors are then ranked on a validation split that took no part in that screen, and the resulting set is frozen and never reselected. Table 3 lists the frozen set for every model. Only afterwards is the intervention effect measured, on evaluation prompts whose names and sentence templates never influenced selection.

Setting	Control mean	Evaluation	$n$	Frozen head set
retrained 160M	64	192	12	L2H11, L5H0, L5H7, L6H8, L7H1, L8H2, L9H1, L9H10, L10H4, L10H10, L11H10, L11H11
official 160M	64	192	21	L2H10, L3H10, L4H0, L4H6, L4H8, L4H11, L5H5, L5H9, L6H0, L6H6, L7H6, L7H10, L8H1, L8H2, L9H3, L9H6, L9H7, L9H8, L10H1, L10H7, L11H11
official 410M	64	192	17	L11H8, L19H10, L17H6, L13H5, L16H15, L12H4, L17H13, L10H14, L16H2, L14H7, L14H0, L20H4, L10H0, L10H9, L16H7, L11H14, L11H1
official 1B	64	192	22	L5H6, L6H7, L9H6, L6H4, L7H1, L4H0, L5H7, L8H5, L11H2, L15H1, L11H1, L14H2, L5H2, L14H5, L15H0, L12H2, L4H1, L13H3, L3H1, L3H4, L13H4, L4H5

Appendix

Table 3: Sizes in items and the frozen head sets. An item is a name pair evaluated in both orders. The control-mean half and the evaluation half share no names and no templates. Heads are written as layer and index, and the sets are screened, then ranked on a separate validation split, then frozen before the mean is estimated or the effect is measured.

Figure 17: Control-mean replacement of the copying heads. The replacement is one fixed vector per head, obtained by averaging that head’s output at the final token over many non-repeated control prompts spanning several names and both name orders. It is therefore not an activation borrowed from a single donor prompt, and it carries no particular competing name. On a separate evaluation prompt in which a name does repeat, each frozen head’s own output is replaced by its fixed mean while every other head, position and parameter is left alone.

Setting at floor	$\Delta\mathrm{LD}$	$\Delta\mathrm{margin}_{3}$
retrained 160M	$+0.180$	$+0.115$
	$[0.004,0.362]$	$[-0.072,0.292]$
official 160M	$+0.804$	$+0.501$
	$[0.529,1.132]$	$[0.209,0.846]$
official 410M	$+0.285$	$+0.226$
	$[0.179,0.388]$	$[0.103,0.345]$

Appendix

Table 4: Effect of the control-mean output replacement, in which each frozen head’s prompt-specific output at the final token is replaced by its average output over non-repeated control prompts, with nested resampling intervals. Positive means the preference for the correct name improved.

Figure 18: Change in the three-name margin of equation 2 from the control-mean replacement. Filled circles indicate that the template-to-item nested interval lies above zero and open squares that it crosses zero. Under item-clustered resampling all four intervals lie above zero.

Figure 19: Both measures under both resampling procedures. Item-clustered intervals are narrower because the nested procedure additionally resamples a small number of template groups.

Figure 20: Source-position value replacement. The evaluated prompt and the identity at the second mention remain unchanged. For each frozen head, only the value vector read at the first mention is replaced by the corresponding value from a prompt carrying a different name there. If the heads merely wrote toward whichever name repeats, their downstream contribution would be unaffected; if it instead follows the substituted identity, the heads are carrying the name represented at their source. This distinction is what the term copying head refers to.

Setting	Heads	Repeated	Substituted	Three-name margin
retrained 160M	12	$-0.478$	$+0.550$	$+0.412$
official 160M	21	$-2.025$	$+2.257$	$+0.972$
official 410M	17	$-0.732$	$+0.754$	$+0.596$
official 1B	22	$-0.364$	$+0.417$	$+0.212$

Appendix

Table 5: Component logit changes from replacing only the frozen ensemble’s value vectors at the first subject mention, holding the second mention fixed. All four settings use the same source-position value-replacement protocol. The repeated-name logit falls and the substituted name’s logit rises by a comparable amount at every setting, and the three-name margin improves at every setting. Table 6 gives that change under both uncertainty procedures.

Setting	Change	Item-clustered	Template-nested
retrained 160M	$+0.412$	$[0.370,0.455]$	$[0.313,0.518]$
official 160M	$+0.972$	$[0.898,1.047]$	$[0.806,1.137]$
official 410M	$+0.596$	$[0.542,0.650]$	$[0.436,0.742]$
official 1B	$+0.212$	$[0.179,0.248]$	$[0.128,0.315]$

Appendix

Table 6: Change in the three-name margin from replacing the frozen ensemble’s value vectors at the first subject mention, under both uncertainty procedures. Every interval lies above zero. This experiment carries six to eight sentence templates, compared with three or four for the control-mean experiment, which is why its nested intervals widen less relative to the item-clustered ones.

Figure 21: The two computations measured on one scale. Each curve is the change in the three-name margin when the same intervention is applied to one of the two frozen head sets, with shaded 95 percent intervals. The left panel uses the retrained 160M model with dense checkpoints through the transition and the right panel uses the released 160M model over full training. A positive value means replacing that set improved behaviour, so the native set was working against the correct answer. A negative value means replacing it damaged behaviour, so the native set was helping.

Setting	Measure	Point estimate	Item-clustered interval	Nested interval	Resolves under nested
retrained 160M	two-name	$+0.180$	$[0.110,0.248]$	$[0.004,0.362]$	yes
	three-name	$+0.115$	$[0.047,0.184]$	$[-0.072,0.292]$	no
official 160M	two-name	$+0.804$	$[0.657,0.958]$	$[0.529,1.132]$	yes
	three-name	$+0.501$	$[0.346,0.655]$	$[0.209,0.846]$	yes
official 410M	two-name	$+0.285$	$[0.210,0.362]$	$[0.179,0.388]$	yes
	three-name	$+0.226$	$[0.146,0.309]$	$[0.103,0.345]$	yes

Appendix

Table 7: Both measures under both resampling procedures, as 95 percent intervals from ten thousand draws.

Figure 22: The selected head set against two hundred feature-matched control sets, at the official 160M floor in the upper row and the retrained 160M floor in the lower row. The shaded band spans the full range of the two hundred control-set effects and the solid line marks their mean, with the diamond marking the selected set. The control ranges are

-0.023

+0.056

and

-0.023

+0.054

at the official model, and

-0.007

+0.316

and

-0.016

+0.250

at the retrained model. The separation is wide at the official model and narrow at the retrained model.

Figure 23: Passive attention split. The total attention a copying head places on the two subject mentions is preserved and divided equally between them, leaving the value vectors untouched. This is neither an ablation nor a value replacement, since the same mass is present and only its allocation changes. The donor attention row is taken, the mass it places on the two subject positions is divided equally between them, and the modified row is written back; every other entry of that row is the donor’s. All seven point estimates satisfy an absolute difference below

0.02

, although some uncertainty intervals extend beyond it, so this is a small-effect comparison rather than a demonstration of equivalence.

Figure 24: Interaction term for the repeated-subject logit at each floor setting. An amplification mechanism predicts a positive value.

Figure 25: Removing the linearly decoded duplication direction at the 160M floor degrades accuracy in proportion to the removal strength.

Setting	Mean	Item interval	Nested interval
160M floor	$+0.016$	$[0.009,0.023]$	$[0.006,0.026]$
410M floor	$+0.018$	$[0.012,0.024]$	$[0.001,0.042]$
410M recovery	$+0.005$	$[-0.003,0.012]$	$[-0.020,0.035]$
410M mature	$+0.013$	$[0.003,0.023]$	$[-0.025,0.053]$
1B floor	$+0.019$	$[0.014,0.026]$	$[-0.002,0.045]$
1B recovery	$+0.018$	$[0.012,0.023]$	$[0.004,0.037]$

Appendix

Table 8: Paired difference between the native-routing effect and the passive-split effect on the three-name margin. A negative value means native routing is more protective than the mechanical split, which is the direction an additional learned correction would predict. Six of the seven settings are positive, and all seven are smaller in size than the analysis bound of

0.02

Figure 26: Paired differences between the native-routing effect and the passive-split effect, with the analysis bound shaded. Negative values would indicate native routing is more protective than the mechanical split.

Figure 27: Mature-head mean replacement, and how it differs from the control mean of Appendix D . Here the selected head’s activation is replaced by its own mean over the prompts and token positions of each sentence-template block being evaluated, so the mean comes from the evaluation data itself. In the copying experiment the mean is instead built from a separate set of non-repeated control prompts. The two operations share a word but not a data source, and they answer different questions. Because this head was also chosen by its replacement effect, the size of that effect is selection-conditioned.

Figure 28: Activation graft and parameter transplant. Both move something from the mature checkpoint into the floor model, but they move different objects. The graft copies the head’s realised output on the evaluated prompt, so the floor model does not recompute it. The transplant copies the head’s query, key, value and output weights together with its head-specific biases, after which the floor model runs normally and the head computes a fresh output from its own upstream representations. The graft asks whether the mature output is sufficient on these prompts; the transplant asks whether what the head has learned is sufficient inside the early network.

Figure 29: Cumulative reconstruction at 410M. Mature parameters are transplanted into the floor model cumulatively, beginning with the suppressing head alone and then adding the strongest correct-name writers in order. The result covers selected cumulative prefixes rather than every intermediate count: measurements exist at zero, one, two, three, five and eight writers. The suppressing head alone closes

67.9

percent of the floor-to-mature gap and all eight writers together reach

88.1

percent, at forced-choice accuracy

0.933

. The experiment is available at this size only.

Figure 30: Magnitude amplification. The floor head’s own output is multiplied by two, four, eight and sixteen, with no mature parameters introduced. If the floor head were already performing the mature operation only weakly, scaling it should move behaviour toward maturity. Accuracy remains below chance at every multiplier, which is why the transplant result is read as supplying a changed computation rather than additional strength.

Figure 31: Replacing the suppressing head’s prompt-specific output with its mature-head mean, taken over the prompts and token positions of each sentence-template block. This is not the control mean of Appendix D . The logit difference falls substantially at all three sizes while forced-choice accuracy remains well above chance, so the head is a substantial contributor to the mature preference rather than solely responsible for it.

Figure 32: Injecting the mature head’s activations into the floor checkpoint, with an output-norm-matched control head as control.

Figure 33: Upstream dependence of the mature suppressing head. Panel A replaces the first feed-forward block’s output with its matched non-repeated counterpart and measures the suppressor’s attention to the second mention, which falls at all three sizes. Panel B isolates only the direct connection between the two, adding the first-block difference to the suppressor’s downstream input while holding every other upstream contribution fixed; that route carries none of the attention dependence and well under a tenth of a percent of the logit dependence. The influence therefore reaches the head through intermediate computation, drawn as an unresolved region because the experiments do not identify which components form the path.

Checkpoint	Attention to $S2$	Direct output projection on $S$
step 512	$0.047$	$-0.005$
step 1000	$0.021$	$-0.016$
step 2000 (floor)	$0.009$	$-0.034$
step 3000	$0.849$	$-1.338$
step 5000	$0.692$	$-4.590$
step 143000 (mature)	$0.934$	$-6.220$

Appendix

Table 9: The mature-defined suppressing head traced backward at 160M. It is barely engaged by these diagnostics at the floor; they are descriptive, not causal.

Figure 34: The head identified at maturity, traced backward at all three sizes. The upper row shows attention from the final token to the second subject mention and the lower row shows the effect of its output on the repeated-subject logit.

Figure 35: Parameter transplant of the single suppressing head against a bundle of eight randomly chosen heads. The comparison is asymmetric in the number of heads transplanted.

Figure 36: First feed-forward block replacement, also called the layer-0 MLP. The evaluated prompt is not edited. A matched control prompt in which no name repeats is run first, the output of that one module is cached, and the cached activation is inserted at the same site in the target run before the forward pass continues. Because the whole module output is replaced, the experiment does not isolate a duplication-specific feature: it establishes the first feed-forward block as an important early source of repetition-sensitive state rather than as a duplication detector.

Model	Native floor $\mathrm{LD}$	Change	95% interval
Pythia 70M	$-2.327$	$+1.808$	$[1.694,1.926]$
Pythia 160M	$-0.757$	$+0.769$	$[0.687,0.854]$
Pythia 410M	$-0.616$	$+0.149$	$[0.101,0.198]$
Pythia 1B	$-0.651$	$+0.493$	$[0.449,0.539]$
Pythia 1.4B	$-0.517$	$+0.371$	$[0.338,0.406]$

Appendix

Table 10: Change in the floor logit difference when the first feed-forward block’s output is replaced by its matched non-repeated counterpart. The effect is positive at all five sizes and every interval lies above zero.

Setting	Early state	Intervention	Early result	Mature result	Strongest supported conclusion
Pythia 70M to 1.4B	$\mathrm{LD}$ negative at all five floors	first feed-forward block replacement	positive at all five	not evaluated here	early repetition-sensitive computation is harmful across scale
Pythia 2.8B	acc. $0.297$ , $\mathrm{LD}$ $-0.504$	input edit	$+0.456$	$-4.076$	input-level sensitivity to repetition reverses
GPT-2 alias	$\mathrm{LD}$ $-1.475$ at step 1500	early-block replacement; future-suppressor replacement	$+1.123$ ; $-0.002$	$-4.081$ ; $-1.928$	early harmful computation precedes mature suppression
GPT-2 battlestar	$\mathrm{LD}$ $-1.741$ at step 1500	as above	$+1.472$ ; $-0.002$	$-4.607$ ; $-1.506$	ordering recurs with a different suppressor head
OLMo-1B	acc. $0.437$ , $\mathrm{LD}$ $-0.220$	second-mention residual-state replacement	$+0.0448$	$-5.137$	state-level causal role reverses outside Pythia and GPT-2

Appendix

Table 11: Extensions across scale and model family. Every entry is the effect of the intervention on the logit difference, so a positive value means the replacement moved the model toward the correct name and a negative value means the native component was helping. Interventions differ across rows, so their magnitudes are not directly comparable. The final column states only the conclusion supported by the intervention in that row.

Figure 37: Prompt structures for the subject-verb agreement and factual-recall tasks. The subject-verb agreement prompts place an attractor of the opposite number between the subject and the verb position, with the preposition or relative clause varying across templates. The factual-recall prompts state one fact, interpose a second, and then return to the first subject, so the correct continuation is the object belonging to the repeated subject.

Task	Early behaviour	Intervention	Early result	Mature result	Strongest supported conclusion
Naming	below chance at 160M, 410M and 1B	control-mean copying-head output replacement	$+0.180$ to $+0.804$	separate suppression experiments	full copying-to-suppression account in the primary models
Greater-Than	below chance at 160M and 1B; $0.553$ at 410M	residual-state replacement at $YY$	$+0.082$ to $+0.319$	$-0.737$ to $-0.667$	causal role of the year state reverses
Factual recall	dips to $0.083$	mature-head mean replacement	near zero through the dip	replacement effect $-1.330$	behavioural dip with a later substantial suppressing component
Subject-verb agreement	no corresponding dip established	matched residual-state replacement	$+0.020$	$+0.022$	no reversal; negative case

Appendix

Table 12: Task extensions. Each row uses a task-specific intervention and outcome, so effect magnitudes should not be compared across tasks. The final column gives the strongest conclusion supported within that task.

Figure 38: Controlled task injection. Models are trained from scratch on ordinary text into which task-structured sequences are substituted at a controlled rate, and each checkpoint is then evaluated twice: on names never injected during training, and on names drawn from the injection pool. The gap between the two separates a failure to generalise to unseen names from a failure to learn the task at all.

Figure 39: Held-out accuracy for the four injection rates. The left panel shows the main run and the right panel two further seeds, with seed 43 solid and seed 44 dashed.

Figure 40: Final accuracy on the two evaluations for each injection rate in the main run.

Figure 41: Ten runs of the minimal task trained from scratch. All ten show the dip and nine show the sign reversal.

Figure 42: Two models trained on ordinary web text. Panel A shows forced-choice accuracy, panel B the logit difference, and panel C the change produced by replacing the residual-stream state at the second occurrence of the repeated name with the corresponding state from a matched control prompt. The evaluated prompt tokens are unchanged. Run A covers 200000 steps and run B covers 105000 steps.

Circuits in Language Models

Circuits in language models are interacting internal components that implement behaviors and can be traced with causal interventions.

See all 122 papers

Oct 19, 2026

The Development of Sycophancy in Language Models: Tracing Vulnerabilities from Pretraining to Post-Training

Language models can change a committed answer to accommodate a user's opinion or pushback without new evidence. However, when this behaviour develops during training and what a released model inherits from earlier stages remain unclear. We follow 741 checkpoints of 30 open model lineages, from early pretraining to released models. We plant initial verdicts on moral scenarios and measure how far a model follows a user's stated opinion or abandons its verdict under pressure. In most lineages the behaviour is present before post-training, and its average strength changes in either direction, at different stages across lineages. A clearer pattern emerges at the level of individual questions. Beyond responses without pressure, pretraining behaviour predicts little of which questions a released model yields on. That pattern forms during post-training: on routes with supervised fine-tuning, the fine-tuning checkpoint predicts part of it and the preference-optimization checkpoint most of it. A sibling trained from the same base along another route predicts far less: for Tülu-3 8B, its own preference-stage checkpoint removes 85% of the remaining prediction error, a sibling 19%. Internal states add predictive information without changing this ranking. Similar average scores conceal different vulnerabilities, and training history matters for auditing a released model.

Haonan Huang, Chengkun Cai

Princeton University · University College London

LLM Sycophancy Language Model Post-Training LLM Auditing

Ask PDF

Oct 23, 2026

The Emergence and Sparsification of a Syntactic Circuit

To understand the emergence of grammatical behavior in language models, we track circuits for subject-verb number agreement in pythia-1b and ten seeds of pythia-410m across training. We find that subject-verb agreement is learned early, and is initially handled by a diffuse collection of neurons. However, after agreement behavior has stabilized, the mechanisms responsible for it continue to reorganize. The attention heads which mediate agreement often shift during training: in the case of pythia-1b, behavior is controlled by two heads early in training but by a third (different) head later. Furthermore, the circuit shrinks over training, with pythia-1b's final circuit being strikingly sparse. We take these results to be informative about how linguistic knowledge emerges and is represented in a language model.

Qing Yao, Sasha Boguraev, Tiago Pimentel +1

The University of Texas at Austin · ETH Zurich

Circuits in Language Models Syntactic Generalization in Language Models

Ask PDF

Sep 22, 2026

Matryoshka attribution: Learning to attribute language model outputs to representations and weights

Attributing language model outputs to their internal computations is an open problem in interpretability. Existing methods, which use causal interventions, gradients, or learnable masks, either are infeasibly expensive or struggle to identify actual causally-important internal computations. We propose framing attribution as the problem of identifying nested subsets of internal components which minimise a downstream loss. To learn this task, we introduce Matryoshka Attribution (MAttr), a mask learning method that parametrises the mask with a simple differentiable sigmoid top-

k

operator. We supervise training over all sparsities simultaneously by randomising

k

over training, resulting in a learned ordering of components by attribution score. MAttr achieves number 1 on the official leaderboard of the Mechanistic Interpretability Benchmark (Mueller et al., 2025); our method identifies sparse and task-transferrable circuits across varying circuit bases. As a practical application, we show that MAttr can be trained with reinforcement learning to identify weight changes responsible for downstream behaviours in LLM finetuning. We train MAttr on refusal judge scores and find that restoring

1\%

of Llama 3.1 8B Instruct's weights to their base model state is sufficient to remove refusals while maintaining capabilities. We view MAttr as a successful formulation of interpretability into a learnable objective that we can tackle with gradient descent, and encourage future work along these lines.

Aryaman Arora, Kirill Acharya, Nathan Hu +4

Stanford University

LLM Interpretability Circuits in Language Models

Ask PDF

Abstract

Circuits in Language Models

Explore similar work

The Development of Sycophancy in Language Models: Tracing Vulnerabilities from Pretraining to Post-Training

The Emergence and Sparsification of a Syntactic Circuit

Matryoshka attribution: Learning to attribute language model outputs to representations and weights