The Emergence and Sparsification of a Syntactic Circuit

Qing Yao; Sasha Boguraev; Tiago Pimentel; Kyle Mahowald

cs.CLOct 23, 2026

The Emergence and Sparsification of a Syntactic Circuit

Authors: Qing Yao, Sasha Boguraev, Tiago Pimentel, Kyle Mahowald

Organizations: The University of Texas at Austin · ETH Zurich

Abstract

To understand the emergence of grammatical behavior in language models, we track circuits for subject-verb number agreement in pythia-1b and ten seeds of pythia-410m across training. We find that subject-verb agreement is learned early, and is initially handled by a diffuse collection of neurons. However, after agreement behavior has stabilized, the mechanisms responsible for it continue to reorganize. The attention heads which mediate agreement often shift during training: in the case of pythia-1b, behavior is controlled by two heads early in training but by a third (different) head later. Furthermore, the circuit shrinks over training, with pythia-1b's final circuit being strikingly sparse. We take these results to be informative about how linguistic knowledge emerges and is represented in a language model.

Figures & tables

Figure 1: The subject–verb agreement circuit for pythia-1b at step 2k (left) and step 143k (right). While agreement is acquired early (within

2\%

of training) and the model remains proficient thereafter, the underlying circuit continues to reorganize and sparsify. Circles represent the smallest subset of top-ranked neurons whose sufficiency and necessity exceed 50% (see Section 4.2 ).

Figure 2: Agreement accuracy vs. # of subject-neurons in the circuit through training across our models. Each point is a checkpoint. Typically, steps 1k–2k form the agreement acquisition phase. In the remaining steps, the circuit sparsifies before reaching some equilibrium.

Panel of Figure 3 — Figure 3: Attention-head reorganization. Top: In pythia-1b , attention heads reorganize across training checkpoints, with the heads important early, L15 H7 and L15 H2 , ceding importance to L13 H7 . All other heads are negligible. Bottom: The identities of the leading heads change across training. Circle area is proportional to each head’s individual $\min\{\operatorname{Nec},\operatorname{Suff}\}$ score, with each individual head a different color.

Figure 4: Top-32 ranked RelP neurons across positions and attention heads L13 H7 and L15 H7 across checkpoints as identified on PP training set (

N=400

). Neurons are colored by their correspondence to the respective heads, on a gradient between blue ( L15 H7 ) and red ( L13 H7 ), determined by path patching. Gray neurons are not attributed to either route. Line width indicates each head’s sufficiency. We see that throughout training, there is a gradual transition from an L15 H7 -routed mechanism to an L13 H7 -routed one.

Components $\mathcal{C}$	$\mathbf{G}$	Measure
Necessity	$\mathbf{G^{\mathcal{C}\leftarrow\texttt{avg}}}$	Nec.
$\mathcal{C}_{\mathtt{full}}$	$+0.76$	$78\%$
$\mathcal{C}_{\text{scope}}$	$+0.58$	$83\%$
Sufficiency	$\mathbf{G^{\mathcal{C}\leftarrow\mathbf{s}}}$	Suff.
$\mathcal{C}_{\mathtt{full}}$	$-2.29$	$83\%$
$\mathcal{C}_{\text{scope}}$	$-2.64$	$88\%$

Table 1: Necessity and sufficiency of the full circuit,

\mathcal{C}_{\mathtt{full}}

, relative to the scope,

\mathcal{C}_{\text{scope}}

, tested on an evaluation set of 192 PP items. The circuit is nearly as necessary and sufficient as the full scope, despite containing around 1/6000 as many MLP neurons.

Template	$\mathbf{G^{\mathbf{b}}}$	$\mathbf{G^{\mathbf{s}}}$	Nec.	Suff.
PP	$+3.48$	$-3.48$	$78\%$	$83\%$
SubjectRC	$+4.48$	$-4.48$	$86\%$	$89\%$
ObjectRC	$+4.30$	$-4.30$	$76\%$	$80\%$
Simple	$+6.49$	$-6.49$	$67\%$	$78\%$
WikiText	$+4.29$	$-3.75$	$73\%$	$82\%$

Table 2: Circuit necessity and sufficiency across templates (

N=32

for Simple ,

N=192

for other templates, and

N=80

for WikiText). Mean activations are computed over the larger training set for PP , and directly on the evaluation set otherwise.

Components $\mathbf{\mathcal{C}}$	$\mathbf{G}$	Measure
Necessity	$\mathbf{G^{\mathcal{C}\leftarrow\texttt{avg}}}$	Nec.
$\mathcal{C}_{\mathtt{H}}$ (Full)	$+1.15$	$67\%$
$\mathcal{C}_{\mathtt{H}}$ (1D subspace)	$+1.01$	$70.90\pm 0.03\%$
$\mathcal{C}_{\mathtt{H}}$ (255D Complement)	$+3.64$	$-4.50\pm 0.02\%$
Sufficiency	$\mathbf{G^{\mathcal{C}\leftarrow\mathbf{s}}}$	Suff.
$\mathcal{C}_{\mathtt{H}}$ (Full)	$-1.25$	$68\%$

Table 3: The 1D channel identified by DAS has a comparable effect to the entire 256D

\mathcal{C}_{\mathtt{H}}

. We report the mean necessity and sufficiency

\pm

standard error across three independent seeds of DAS.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Condition	Template prefix	Example prefix
Simple	The {subject}	The guard
SubjectRC	The {subject} that {verbed} the {object}	The guard that liked the doctor
ObjectRC	The {subject} that the {object} {verbed}	The guard that the doctor liked
PP	The {subject} {prep} the {object}	The guard near the doctors
PP + adjective-on-subject	The {adj} {subject} {prep} the {object}	The tall guard near the doctors
PP + adjective-on-object	The {subject} {prep} the {adj} {object}	The guard near the tall doctors

Appendix

Table 4: Agreement templates used in cross-template and perturbation generalization experiments. The object/distractor pool contains both singular and plural forms for PP -based templates, and base and source examples differ only in the subject number. We evaluate

N=192

pairs (

N=32

for Simple ).

Condition	Template / readout	Example
Held-out subjects	The {held-out-subject} {prep} the {object} ; readout is is/are , was/were , has/have	The lawyer near the doctors $\to$ is/are
Held-out verbs	The {subject} {prep} the {object} ; readout is the held-out verb-pair set	The guard near the doctors $\to$ runs/run

Appendix

Table 5: Lexical-generalization templates. We test a wider range of subject and verb tokens in the PP template to ensure circuit robustness.

		Percent Recovered
		Full pool		Top-16
Step	Scoring	Nec	Suff	Nec	Suff
2k	Direct (used)	$89.4$	$92.7$	$12.2$	$17.8$
2k	Total effect	$88.8$	$92.5$	$10.8$	$16.2$
143k	Direct (used)	$97.6$	$98.7$	$51.7$	$64.6$
143k	Total effect	$94.9$	$94.9$	$54.9$	$66.3$

Appendix

Table 6: Sparsification is robust to the stop gradient implementation. All values are percentages on the PP evaluation set. “Full pool” is the necessity/sufficiency of the 1024-neuron circuit with respective attention heads, and “Top-16” is the fraction of the gain from the head-only baseline to the full pool captured by the 16 highest-ranked neurons.

Run	2k	5k	8k	18k	50k	143k
Seed 0	766	152 [108/152]	55 [53/55]	15 [15/15]	14 [11/14]	12 [12/12]
Seed 1	462	170 [102/170]	85 [64/85]	22 [22/22]	12 [11/12]	6 [6/6]
Seed 2	345	153 [81/153]	39 [36/39]	29 [28/29]	23 [21/23]	48 [23/23]
Seed 5	477	319 [136/319]	311 [182/311]	178 [126/178]	58 [50/58]	29 [29/29]
Seed 6	569	198 [118/198]	109 [95/109]	15 [15/15]	9 [9/9]	8 [8/8]
Seed 7	473	90 [66/90]	29 [28/29]	17 [17/17]	16 [14/16]	14 [13/14]

Appendix

Table 7: Size and overlap of the

k_{50}

neuron sets across checkpoints. The first number is the

k_{50}

size, the numerator of the fraction is the overlap between the checkpoint’s and the previous checkpoint’s

k_{50}

set, and the denominator of the fraction is the size of the smaller

k_{50}

set of the two.

Figure 7:

k_{50}

rebound co-occurs with training regressions. Seed 3 rebound happens early and the model recovers a sparse neuron mechanism by end of training. Seed 4 suffered setbacks late and its

k_{50}

ended high.

Checkpoint	Head	$\min\{\mathrm{Nec},\mathrm{Suff}\}$
1k	L15 H2	$48.4\%$
	L15 H7	$42.5\%$
	L15 H0	$8.2\%$
	L15 H1	$7.8\%$
	L9 H6	$1.7\%$
2k	L15 H7	$49.4\%$

Appendix

Table 8: Top five attention heads at each checkpoint scored by

\min(\mathrm{Nec},\mathrm{Suff})

. The three main heads are bolded.

Run	Leader	Score	Leader	Score	Movement
	Step 2k		Step 143k
Seed 0	L22 H11	$69.9\%$	L22 H11	$65.2\%$	Stable
Seed 1	L19 H15	$89.9\%$	L19 H15	$80.7\%$	Stable
Seed 2	L23 H10	$49.4\%$	L21 H9	$33.9\%$	Earlier
Seed 5	L20 H15	$80.4\%$	L20 H15	$87.6\%$	Stable
Seed 6	L21 H6	$42.1\%$	L22 H4	$56.2\%$	Later

Appendix

Table 9: Leading head across ten seeds of pythia-410m at step 2k and step 143k. Score denotes

\min(\operatorname{Nec},\operatorname{Suff})

. There is no predictable pattern in how the models reorganize the transport channels.

Checkpoint	Head	subject $\to$ pred.	prep. $\to$ pred.	det. $\to$ pred.	object $\to$ pred.
256	L15 H7	$0.006$	$0.001$	$0.000$	$0.001$
256	L15 H2	$-0.012$	$-0.002$	$0.000$	$-0.002$
256	L13 H7	$0.009$	$0.002$	$0.000$	$0.000$
1k	L15 H7	$0.391$	$-0.004$	$0.000$	$-0.004$
1k	L15 H2	$0.446$	$0.000$	$0.000$	$0.000$
1k	L13 H7	$0.006$	$0.000$	$0.000$	$0.000$

Appendix

Table 10: Position-specific edge interventions into the prediction position. Entries are fractions of the full head sufficiency recovered by the single-edge intervention. L15 H7 and L15 H2 are concentrated on subject-to-prediction edges, while L13 H7 is a strong subject edge that increasingly uses the preposition-to-prediction edge late in training.

Checkpoint	$\text{pos}_{0}$ initial The	$\text{pos}_{1}$ subject	$\text{pos}_{2}$ prep.	$\text{pos}_{3}$ distractor the	$\text{pos}_{4}$ object/pred.
256	$0.00004$	$0.00025$	$0.00004$	$0.00004$	$0.00031$
1k	$0.00069$	$0.00598$	$0.00038$	$0.00014$	$0.00135$
2k	$0.00064$	$0.05164$	$0.00150$	$0.00051$	$0.01050$
5k	$0.00093$	$0.22402$	$0.00680$	$0.00200$	$0.07530$
8k	$0.00114$	$0.28259$	$0.01666$	$0.00528$	$0.15537$
18k	$0.00392$	$0.27067$	$0.06123$	$0.01214$	$0.25152$

Appendix

Table 11: Maximum absolute RelP score by PP template token position across checkpoints. Largest RelP scores occur at the subject and prediction-side positions after agreement emerges, while the sentence-initial determiner and the distractor determiner remain small throughout.

Dataset	Ablated	clean CE	ablated CE	$\Delta$ CE
PP	L0	$4.05$	$4.86$	$+0.82$
PP	L5	$4.05$	$4.03$	$-0.01$
WikiText	L0	$2.08$	$2.90$	$+0.83$
WikiText	L5	$2.08$	$2.07$	$-0.01$

Appendix

Table 12: Cross-entropy impact of full-MLP mean ablation at the subject position. CE is measured on the correct agreement target at the prediction position. PP evaluation set

(N=192)

, WikiText

(N=80)

Module	Components
$\mathcal{C}_{\mathtt{A}}$ ( $\text{pos}_{1}$ )	L1 n7988 , L5 n3793 , L8 n6308 , L9 n5060 , L9 n3101 , L10 n747 , L10 n6267 , L11 n6624 , L11 n239 , L12 n1535
$\mathcal{C}_{\mathtt{H}}$	L13 H7
$\mathcal{C}_{\mathtt{P}}$ ( $\text{pos}_{4}$ )	L14 n1160 (pl); L14 n4146 , L14 n6888 (sg)
$\mathcal{C}_{\mathtt{N}}$ ( $\text{pos}_{4}$ )	L14 n6221 , L15 n2511 , L15 n3382 , L15 n6710 (pl); L15 n2305 (sg)

Appendix

Table 13: Components of the subject–verb agreement circuit

\mathcal{C}_{\mathtt{full}}

in pythia-1b .

\mathcal{C}_{\mathtt{H}}

transports the number feature, and

\mathcal{C}_{\mathtt{P}}

and

\mathcal{C}_{\mathtt{N}}

neurons promote or suppress the labeled verb forms.

Neuron	sg mean	pl mean	non-noun mean
L1 n7988	$-0.117$	$1.796$	$-0.102$
L5 n3793	$-0.102$	$3.307$	$-0.105$
L8 n6308	$-0.091$	$2.831$	$-0.056$
L9 n5060	$-0.108$	$3.552$	$-0.090$
L9 n3101	$-0.085$	$1.190$	$-0.068$
L10 n747	$-0.106$	$1.466$	$-0.049$

Appendix

Table 14: Activations for

\mathcal{C}_{\mathtt{A}}

at the subject position. Means are post-activation MLP neuron values. The selected neurons distinguish plural nouns from singular nouns, while singular nouns remain close to non-nouns.

Neuron	module	sg mean	pl mean
L14 n1160	$\mathcal{C}_{\mathtt{P}}$ plural	$-0.06$	$2.88$
L14 n4146	$\mathcal{C}_{\mathtt{P}}$ singular	$1.67$	$-0.05$
L14 n6888	$\mathcal{C}_{\mathtt{P}}$ singular	$0.95$	$-0.11$
L14 n6221	$\mathcal{C}_{\mathtt{N}}$ plural-gated	$-0.05$	$0.69$
L15 n2511	$\mathcal{C}_{\mathtt{N}}$ plural-gated	$-0.05$	$1.36$
L15 n3382	$\mathcal{C}_{\mathtt{N}}$ plural-gated	$-0.08$	$1.22$

Appendix

Table 15: Readout neuron activations at the prediction position. The pro- and anti-readouts are gated by the number state reaching the prediction position: plural-side neurons activate on plural-subject examples, while singular-side neurons activate on singular-subject examples. Each mean is computed over

96

singular-subject and

96

plural-subject base sentences in the PP evaluation set (

N=192

Donor	all	sg	pl
Baseline	$+3.48$	$+3.61$	$+3.35$
Same-number donor	$+3.68$	$+3.71$	$+3.66$
Cross-number donor	$+3.54$	$+3.59$	$+3.48$

Appendix

Table 16: Agreement gap under off-circuit intervention (

N=192

N_{\text{donors}}=20

each; 1924 same-class and 1916 cross-class trials). With the circuit untouched, replacing relevant non-circuit activation with a donor’s activation leaves agreement intact regardless of donor class.

	Donor
Base	PP	SubjRC	ObjRC	Simple
PP	$\mathbf{84\%}$	$97\%$	$95\%$	$59\%$
SubjRC	$78\%$	$\mathbf{89\%}$	$88\%$	$58\%$
ObjRC	$72\%$	$82\%$	$\mathbf{80\%}$	$53\%$
Simple	$85\%$	$93\%$	$90\%$	$\mathbf{78\%}$

Appendix

Table 17: Sufficiency from cross-template intervention. Rows are the base template; columns are the donor template (each pair drawn at random, with the donor’s subject opposite-number from base’s). Donors are drawn to have opposite number to each base sentence.

N=32

base examples for Simple and

N=192

otherwise, balanced across sg/pl-base.

Perturbation	$G^{\mathbf{b}}$	$G^{\mathcal{C}\leftarrow\mathbf{s}}$	$G^{\mathbf{s}}$	$\operatorname{Suff.}$
None	$+3.48$	$-2.30$	$-3.48$	$83\%$
Adj on subj	$+3.26$	$-2.18$	$-3.29$	$83\%$
Adj on obj	$+3.08$	$-2.34$	$-3.19$	$86\%$
Adv after obj	$+3.70$	$+0.56$	$-3.87$	$\mathbf{42\%}$
Single prefix	$+3.00$	$-1.88$	$-3.00$	$81\%$
Loc. prefix	$+4.33$	$-3.02$	$-4.31$	$85\%$

Appendix

Table 18: On-circuit intervention on the perturbed PP template (

N=192

). The circuit applies to all settings apart from adding an adverb after the distractor object.

Circuits in Language Models

Circuits in language models are interacting internal components that implement behaviors and can be traced with causal interventions.

See all 122 papers

Copying Before Suppression: What Drives a Below-Chance Dip During Language Model Training?
Oct 2, 2026Tejas Dahiya, Cole BlondinCircuits in Language Models LLM Training
University of Chicago · Independent Researcher
Shared Circuits for Shared Grammar: Tracing Subject-Verb Agreement Across Languages
Aug 19, 2026Isabella Gidi, Antonio Almudévar, Core Francisco Park +2Multilingual Language Models Circuits in Language Models
Harvard University · University of Zaragoza · Boston University +1
The Kinetics of Training: A Driven-Nucleation Rate Law for Emergence, Plasticity Loss, and Circuit Control in Language Models
Jul 29, 2026Lei DongLoss of Plasticity Phase Transitions
Independent Researcher

Oct 2, 2026J/K move · Enter open · S save

Copying Before Suppression: What Drives a Below-Chance Dip During Language Model Training?

Tejas Dahiya, Cole Blondin

University of Chicago · Independent Researcher

Circuits in Language Models LLM Training Causal Interventions in Language Models

Open paper AskPDF arXiv

Mechanistic interpretability usually studies fully trained models, yet the computations that drive a behaviour can change while the model is still learning the task. On the Indirect Object Identification task, a model should continue with the name mentioned once rather than the name mentioned twice. Pythia models pass through an early training window in which they prefer the repeated name, so accuracy in a choice between the two names falls below one half while language-model loss on a fixed text sample keeps decreasing across the same window. The window reflects a temporary imbalance between two computations. We identify one cause of the wrong preference by selecting a set of attention heads that write the repeated name, on prompts separate from those used for causal evaluation, keeping that selection fixed, and then replacing each head's final-token output with its average output on a separate set of non-repeated-name prompts. This improves the correct-minus-repeated logit difference in a separately trained 160M model and in the official 160M, 410M, and 1B models. At 160M, the head that lowers the repeated name in the mature model shows little of its mature behaviour at this point. It directs less than one percent of its attention to the repeated mention, and its output makes almost no direct contribution to lowering that name's logit. Both properties grow over the interval in which behaviour recovers. Across the 160M, 410M, and 1B models, transplanting the corresponding head's mature parameters into the early checkpoint recovers 35 to 68 percent of the total improvement in the correct-minus-repeated logit difference seen by the end of training. Related early-to-late reversals appear at further Pythia scales, in two independently trained GPT-2 models, and in OLMo. A mature circuit can therefore conceal a transient causal configuration that shaped behaviour earlier in training.

Abstract

Circuits in Language Models

Explore similar work

Copying Before Suppression: What Drives a Below-Chance Dip During Language Model Training?

Shared Circuits for Shared Grammar: Tracing Subject-Verb Agreement Across Languages

The Kinetics of Training: A Driven-Nucleation Rate Law for Emergence, Plasticity Loss, and Circuit Control in Language Models