Sparse Transformer: a Controlled Testbed for Mechanistic Interpretability

Anton Korznikov; Alexey Dontsov; Elena Tutubalina

cs.AIOct 14, 2026

Sparse Transformer: a Controlled Testbed for Mechanistic Interpretability

Authors: Anton Korznikov, Alexey Dontsov, Elena Tutubalina

Abstract

Mechanistic interpretability aims to decompose neural network computations into human-understandable features, yet progress is limited by the absence of ground-truth units to evaluate against. We address this with a Sparse Transformer (ST): a 5-layer language model trained on TinyStories whose wide, TopK-sparse MLP forces neuron specialization. We show that ST neurons are highly interpretable: they outperform a loss-matched dense transformer by 43% on AutoInterp, and circuits of a median of 12 neurons suffice to reproduce individual model predictions. This testbed allows us to directly compare feature learning methods: Transcoder, which approximates the MLP input-output mapping, recovers ST's neuron vectors with 52.6% higher cosine similarity than Sparse Autoencoder, which only reconstructs outputs. Furthermore, the ST enables a controlled study of polysemanticity: distilling its sparse MLP into a dense counterpart reveals how compression leads to feature superposition. These results show how a known reference basis sharpens evaluation and understanding, a role we hope the ST can serve for the field.

Figures & tables

Figure 1: An interpretable 11-neuron circuit. Given the prompt “Jack was hungry so he went looking for something to” , nodes are MLP neurons grouped by layer (L0–L4); edges indicate causal influence. (top right) Neuron dashboards show top activating contexts for 3 circuit nodes. (bottom right) Faithfulness and completeness as neurons are added by beam search; 11 neurons suffice to reach faithfulness $\geq 0.95$ .

Panel of Figure 2 — Figure 2: Width–sparsity trade-off. (left) Validation loss vs. proportion of activated neurons for each MLP width multiplier $M$ ; loss decreases monotonically with both width and activation rate. (middle) Power-law predictions vs. true validation loss across all 35 models ( $R^{2}{=}0.944$ ). (right) Fraction of dead neurons across the $(M,K)$ grid; negligible for $M\leq 16$ but rising at large width and small $K$ .

Parameter	Estimate	95% CI
$c$	$1.696$	$[1.667,\ 1.725]$
$\alpha$	$-0.038$	$[-0.041,\ -0.034]$
$\beta$	$-0.015$	$[-0.018,\ -0.013]$

Table 1: Power-law fit coefficients for

\mathcal{L}(M,K)

Model	AutoInterp	95% CI
Random guessing	0.500	—
Dense ( $M{=}1$ )	0.565	[0.51, 0.61]
Dense ( $M{=}4$ , loss-matched)	0.605	[0.55, 0.65]
Dense ( $M{=}32$ , width-matched)	0.667	[0.61, 0.72]
ST ( $M{=}32$ , $K{=}8$ )	0.866	[0.82, 0.91]

Table 2: AutoInterp scores. TopK sparsity yields substantially more interpretable neurons: ST (

M{=}32

K{=}8

) scores 0.866 vs. 0.605 for the loss - matched and 0.667 for the width-matched dense baseline.

Panel of Figure 3 — Figure 3: Feature recovery results (Case Study #1) . (left) Schematic of the experiment: comparing Transcoder (trained to approximate the MLP’s input–output mapping) and SAE (trained to reconstruct MLP output activations) against the ground - truth ST neuron vectors. (middle) NMSE training curves for Transcoder and SAE. Transcoder achieves $3.8\times$ lower final NMSE than SAE. (right) Average cosine similarity between each true ST latent and its closest SAE/Transcoder latent. Transcoder value vectors achieve $+52.6\%$ improvement over SAE decoder vectors.

Panel of Figure 4 — Figure 4: Teacher-student mapping and polysemanticity. (left) Cumulative mass functions for both mapping directions. At the $80\%$ level, each teacher neuron $t_{i}$ maps to ${\sim}20$ student neurons $s_{j}$ , while each student neuron draws from ${\sim}150$ teacher neurons. (right) Student neuron $s_{310}$ superposes four semantically unrelated teacher neurons: body-part tokens, verb propagation after [they] , [again] / [in] propagation, and time-expression tokens.

Panel of Figure 5 — Figure 5: Feature composition and geometric alignment. (left) TVD between student-space distributions of teacher neuron pairs, stratified by PMI. High-PMI pairs show lower TVD, indicating co-occurring features map to overlapping student regions. (middle) Cosine similarity between student value vectors and the teacher-distribution-weighted average of teacher value vectors, far above the shuffled baseline ( ${\approx}0.15$ ). (right) Same analysis for key vectors, confirming geometric alignment holds for both input and output projections.

Appendix figures & tables38 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Method	NMSE $\downarrow$	MMCS enc./key $\uparrow$	MMCS dec./value $\uparrow$
TopK ( $K{=}8$ )	Transcoder	$0.0094\pm 0.0005$	$0.564\pm 0.0002$	$0.709\pm 0.0001$
	SAE	$0.0358\pm 0.0020$	$0.410\pm 0.0001$	$0.464\pm 0.0001$
TopK ( $K{=}4$ )	Transcoder	$0.0237\pm 0.0012$	$0.493\pm 0.0002$	$0.661\pm 0.0001$
	SAE	$0.0681\pm 0.0037$	$0.340\pm 0.0000$	$0.378\pm 0.0000$
TopK ( $K{=}16$ )	Transcoder	$0.0068\pm 0.0004$	$0.593\pm 0.0001$	$0.720\pm 0.0001$
	SAE	$0.0247\pm 0.0013$	$0.415\pm 0.0001$	$0.469\pm 0.0002$

Appendix

Table 3: SAE vs. Transcoder under alternative sparsity settings (mean

\pm

std over 3 seeds). As in Section 5 , SAE encoder and decoder vectors (“enc.”, “dec.”) are matched against the true value vectors, whereas Transcoder input and output vectors (“key”, “value”) are matched against the true keys and values, respectively.

#	Target	$P_{0}$ (%)	$n_{F}$	$F(n_{F})$	$C(n_{F})$	$n_{C}$	$F(20)$	$C(20)$
1	[eat]	98.3	11	0.956	0.198	11	1.010	0.142
2	[hug]	89.1	12	0.961	0.114	9	1.082	0.063
3	[chase]	81.5	13	0.968	0.467	39	1.112	0.338
4	[prince]	96.9	14	0.951	0.139	11	0.982	0.085
5	[voice]	94.8	5	0.986	0.331	6	1.054	0.057
6	[day]	99.1	12	0.954	0.149	10	0.996	0.089

Appendix

Table 4: Per-prompt circuit metrics.

P_{0}

is the full-model probability of the target token.

n_{F}

is the smallest circuit size with

F\geq 0.95

(reached by 15 of 16 prompts), and

F(n_{F})

C(n_{F})

are the metrics at that size;

n_{C}

is the smallest size with

C\leq 0.2

(reached by 14 of 16 prompts);

F(20)

C(20)

are the metrics at a fixed budget of 20 neurons. Entries marked – are not reached within the 50-neuron budget; medians of

n_{F}

and

n_{C}

are over prompts that reach the threshold. Row 1 is the circuit in Figure 1 .

Neurons $n$	1	2	3	5	8	10	15	20	25	30	40	50
$F$ mean	0.11	0.21	0.30	0.49	0.74	0.85	0.98	1.01	1.02	1.03	1.03	1.03
$F$ std	0.16	0.25	0.29	0.29	0.22	0.16	0.06	0.05	0.05	0.06	0.06	0.06
$F$ median	0.04	0.09	0.16	0.47	0.82	0.90	0.99	1.00	1.00	1.00	1.01	1.01
$C$ mean	0.92	0.84	0.76	0.62	0.45	0.35	0.26	0.20	0.17	0.12	0.09	0.07
$C$ std	0.14	0.18	0.23	0.24	0.26	0.28	0.25	0.22	0.17	0.12	0.09	0.07
$C$ median	0.96	0.89	0.86	0.66	0.41	0.26	0.19	0.12	0.11	0.08	0.06	0.06

Appendix

Table 5: Faithfulness (

F

, higher is better) and completeness (

C

, lower is better) as a function of circuit size

n

, aggregated over the 16 prompts (population std). Per-prompt values for every

n\in[1,50]

are in the released repository.

Figure 6: Circuit for “She felt very sad so her friend gave her a big warm” $\to$ [hug] ( $p{=}89.1\%$ ).

Figure 7: Circuit for “The cat saw a mouse and started to” $\to$ [chase] ( $p{=}81.5\%$ ).

Figure 8: Circuit for “The princess kissed the frog and it turned into a handsome” $\to$ [prince] ( $p{=}96.9\%$ ).

Figure 9: Circuit for “The robot beeped and said in a friendly” $\to$ [voice] ( $p{=}94.8\%$ ).

Figure 10: Circuit for “She planted a seed and watered it every” $\to$ [day] ( $p{=}99.1\%$ ).

Figure 11: Circuit for “The pirate looked at the map to find the hidden” $\to$ [treasure] ( $p{=}91.7\%$ ).

Figure 12: Circuit for “He washed his hands before sitting at the dinner” $\to$ [table] ( $p{=}98.8\%$ ).

Figure 13: Circuit for “She put on her shoes and ran out the” $\to$ [door] ( $p{=}99.9\%$ ).

Figure 14: Circuit for “She opened the book and started to” $\to$ [read] ( $p{=}88.8\%$ ).

Figure 15: Circuit for “He wrapped the gift in colorful shiny” $\to$ [paper] ( $p{=}97.7\%$ ).

Figure 16: Circuit for “She drank the hot soup and felt much” $\to$ [better] ( $p{=}99.7\%$ ).

Figure 17: Circuit for “She brushed her hair and looked in the” $\to$ [mirror] ( $p{=}99.9\%$ ).

Figure 18: Circuit for “She baked a cake and put it in the” $\to$ [oven] ( $p{=}99.6\%$ ).

Figure 19: Circuit for “The baby dropped her toy and started to” $\to$ [cry] ( $p{=}83.8\%$ ).

Figure 20: Circuit for “He stacked the blocks until the tower was very” $\to$ [tall] ( $p{=}87.2\%$ ).

Figure 21: L0 - N2286 activates on positive adjectives ( good, great ) and friendly interpersonal contexts.

Figure 22: L0 - N2993 detects the relative pronoun who (and its variants whom ) in narrative contexts, particularly after a character introduction.

Figure 23: L0 - N7792 responds to planning and discussion verbs ( discussed, brainstorm, plan ).

Figure 24: L1 - N222 activates on the preposition in in contexts where a character notices or discovers something in their surroundings (e.g. “a bright light in the sky” , “something in the distance” ). Its output weight vector promotes calm, positive continuations ( relax, comfortable, enjoy, trust ).

Figure 25: L1 - N3150 activates on the comma following a character’s speech or completed action, marking a narrative continuation point. Its output weight vector promotes continuation tokens ( and, he, it, change ), suggesting it signals that another clause or reaction is about to follow.

Figure 26: L1 - N3675 activates on the verb listen in negated contexts ( “did not listen” , “does not listen” ), typically in scenes involving disobedience or conflict between characters. Negative logits include tape, hides, reward .

Figure 27: L1 - N5005 detects geological objects: stone, mineral, crystal . Its top examples consistently mention finding or holding a stone.

Figure 28: L2 - N467 marks the word Today at the beginning of a narrative sentence, often followed by a comma, signalling a temporal shift in the story.

Figure 29: L2 - N1091 encodes a character’s enjoyment of an activity: top phrases are “loved to play” , “loved to crawl” , “loved to draw” .

Figure 30: L2 - N1441 activates on sentence - initial pronouns ( She, He, They ) that introduce a new action or dialogue, e.g. “She held her pocket tight” .

Figure 31: L2 - N3125 detects a moral or lesson learned: top activations contain “learned a lesson” , “learned a hard lesson” , often at the end of a story.

Figure 32: L3 - N1169 represents the action of throwing: top examples are “threw stones” , “threw the model” , “threw a newspaper” .

Figure 33: L3 - N1609 activates on the gerund being at moments of emotional or behavioural shift in a character - particularly when a character is described as being naughty , being stubborn , or stopped being a certain way. Its output weight vector promotes anger- and emotion-related tokens ( angry, upset, jealous, naughty, mad ).

Figure 34: L3 - N3412 activates on the preposition from in the construction “tired from [activity]” (e.g. “tired from playing” , “tired from working” , “tired from all the walking” ), signalling physical exhaustion caused by a preceding effort.

Figure 35: L3 - N3501 activates on the punctuation mark (colon or dash) that follows a phrase like “something they had never seen before” , “something beautiful” , or “something strange” - the moment just before the surprising object is revealed or described.

Figure 36: L4 - N2893 represents dizziness or spinning: top examples contain “dizzy” (often split as “dizz y” ) from spinning too much.

Figure 37: L4 - N3860 marks a temporal turning point: “From then on” , “From now on” , signalling that a character changes behaviour after an event.

Figure 38: L4 - N4150 detects loud sounds: top activations are “loud noise” , “loud bang” , “loud noises” , often from fireworks or broken objects.

Figure 39: L4 - N5981 encodes a commitment or resolution: “From that day on, [character] made sure to always” followed by a cautious behaviour.

Figure 40: L4 - N6211 activates on verbs describing a downward or de-escalating motion ( calmed, bent, fell, broke ) in the construction “[verb] down” , particularly when a character or situation settles or physically descends. Negative logits include up, down (as standalone tokens) and loves .

Mechanistic Interpretability

Mechanistic interpretability studies how neural networks implement behaviors through internal features, circuits, and causal information flow.

See all 371 papers

Comparing Sparsified MLPs and Transcoders: Many Transcoder Features Are Neurons in Disguise
Oct 17, 2026Stepan Shabalin, Mateusz PiotrowskiTransformer Interpretability Neuron-Level Interpretability
Anthropic · Georgia Institute of Technology
Individual Parameters in Weight-Sparse Transformers Appear Interpretable
Jul 3, 2026Arnau Marin-Llobet, Stefan HeimersheimTransformer Interpretability Transformer
School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138
Scaling Interpretable Transformers with Parity Bottleneck Layers
Jul 22, 2026Andrew Mack, Kraig Yuheng Tou, Mark Henry +2Feature Superposition Transformer Interpretability
Principles of Intelligence · Independent

Oct 17, 2026J/K move · Enter open · S save

Comparing Sparsified MLPs and Transcoders: Many Transcoder Features Are Neurons in Disguise

Stepan Shabalin, Mateusz Piotrowski

Anthropic · Georgia Institute of Technology

Transformer Interpretability Neuron-Level Interpretability Transformer FFNs

Open paper AskPDF arXiv

MLP neurons in transformer language models are widely understood to be polysemantic, motivating transcoders - sparse, learned approximations of MLP layers - as interpretable replacement models for mechanistic analysis. However, recent work shows that MLP neurons can be used to recover sparse, interpretable circuits (Arora et al. 2026, Ye et al. 2025), raising the question of whether transcoders are necessary for circuit analysis. We define neuron features as transcoder features whose weights are strongly aligned with individual MLP neurons, and show that they account for a substantial fraction of active features in late layers across 13 open-source transcoders. Only neuron features have a stable correspondence with something in the underlying MLP: other features vary more between training runs, do not match the MLP's Jacobian, and change less significantly from their random initialization. Our findings reframe MLP neurons as a stronger basis for interpretability than commonly assumed, and call for better methods of attributing the non-neuron structure that is used by the model.

Abstract

Mechanistic Interpretability

Explore similar work

Comparing Sparsified MLPs and Transcoders: Many Transcoder Features Are Neurons in Disguise

Individual Parameters in Weight-Sparse Transformers Appear Interpretable

Scaling Interpretable Transformers with Parity Bottleneck Layers