Mechanistic interpretability aims to decompose neural network computations into human-understandable features, yet progress is limited by the absence of ground-truth units to evaluate against. We address this with a Sparse Transformer (ST): a 5-layer language model trained on TinyStories whose wide, TopK-sparse MLP forces neuron specialization. We show that ST neurons are highly interpretable: they outperform a loss-matched dense transformer by 43% on AutoInterp, and circuits of a median of 12 neurons suffice to reproduce individual model predictions. This testbed allows us to directly compare feature learning methods: Transcoder, which approximates the MLP input-output mapping, recovers ST's neuron vectors with 52.6% higher cosine similarity than Sparse Autoencoder, which only reconstructs outputs. Furthermore, the ST enables a controlled study of polysemanticity: distilling its sparse MLP into a dense counterpart reveals how compression leads to feature superposition. These results show how a known reference basis sharpens evaluation and understanding, a role we hope the ST can serve for the field.
Figures & tables
Figure 1: An interpretable 11-neuron circuit. Given the prompt “Jack was hungry so he went looking for something to” , nodes are MLP neurons grouped by layer (L0–L4); edges indicate causal influence. (top right) Neuron dashboards show top activating contexts for 3 circuit nodes. (bottom right) Faithfulness and completeness as neurons are added by beam search; 11 neurons suffice to reach faithfulness ≥0.95 .
Figure 2: Width–sparsity trade-off. (left) Validation loss vs. proportion of activated neurons for each MLP width multiplier M ; loss decreases monotonically with both width and activation rate. (middle) Power-law predictions vs. true validation loss across all 35 models ( R2=0.944 ). (right) Fraction of dead neurons across the (M,K) grid; negligible for M≤16 but rising at large width and small K .
Parameter
Estimate
95% CI
c
1.696
[1.667,1.725]
α
−0.038
[−0.041,−0.034]
β
−0.015
[−0.018,−0.013]
Table 1: Power-law fit coefficients for L(M,K) .
Model
AutoInterp
95% CI
Random guessing
0.500
—
Dense ( M=1 )
0.565
[0.51, 0.61]
Dense ( M=4 , loss-matched)
0.605
[0.55, 0.65]
Dense ( M=32 , width-matched)
0.667
[0.61, 0.72]
ST ( M=32 , K=8 )
0.866
[0.82, 0.91]
Table 2: AutoInterp scores. TopK sparsity yields substantially more interpretable neurons: ST ( M=32 , K=8 ) scores 0.866 vs. 0.605 for the loss - matched and 0.667 for the width-matched dense baseline.
Figure 3: Feature recovery results (Case Study #1) . (left) Schematic of the experiment: comparing Transcoder (trained to approximate the MLP’s input–output mapping) and SAE (trained to reconstruct MLP output activations) against the ground - truth ST neuron vectors. (middle) NMSE training curves for Transcoder and SAE. Transcoder achieves 3.8× lower final NMSE than SAE. (right) Average cosine similarity between each true ST latent and its closest SAE/Transcoder latent. Transcoder value vectors achieve +52.6% improvement over SAE decoder vectors.
Figure 4: Teacher-student mapping and polysemanticity. (left) Cumulative mass functions for both mapping directions. At the 80% level, each teacher neuron ti maps to ∼20 student neurons sj , while each student neuron draws from ∼150 teacher neurons. (right) Student neuron s310 superposes four semantically unrelated teacher neurons: body-part tokens, verb propagation after [they] , [again] / [in] propagation, and time-expression tokens.
Figure 5: Feature composition and geometric alignment. (left) TVD between student-space distributions of teacher neuron pairs, stratified by PMI. High-PMI pairs show lower TVD, indicating co-occurring features map to overlapping student regions. (middle) Cosine similarity between student value vectors and the teacher-distribution-weighted average of teacher value vectors, far above the shuffled baseline ( ≈0.15 ). (right) Same analysis for key vectors, confirming geometric alignment holds for both input and output projections.
Appendix figures & tables38 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Method
NMSE ↓
MMCS enc./key ↑
MMCS dec./value ↑
TopK ( K=8 )
Transcoder
0.0094±0.0005
0.564±0.0002
0.709±0.0001
SAE
0.0358±0.0020
0.410±0.0001
0.464±0.0001
TopK ( K=4 )
Transcoder
0.0237±0.0012
0.493±0.0002
0.661±0.0001
SAE
0.0681±0.0037
0.340±0.0000
0.378±0.0000
TopK ( K=16 )
Transcoder
0.0068±0.0004
0.593±0.0001
0.720±0.0001
SAE
0.0247±0.0013
0.415±0.0001
0.469±0.0002
Appendix
Table 3: SAE vs. Transcoder under alternative sparsity settings (mean ± std over 3 seeds). As in Section 5 , SAE encoder and decoder vectors (“enc.”, “dec.”) are matched against the true value vectors, whereas Transcoder input and output vectors (“key”, “value”) are matched against the true keys and values, respectively.
#
Target
P0 (%)
nF
F(nF)
C(nF)
nC
F(20)
C(20)
1
[eat]
98.3
11
0.956
0.198
11
1.010
0.142
2
[hug]
89.1
12
0.961
0.114
9
1.082
0.063
3
[chase]
81.5
13
0.968
0.467
39
1.112
0.338
4
[prince]
96.9
14
0.951
0.139
11
0.982
0.085
5
[voice]
94.8
5
0.986
0.331
6
1.054
0.057
6
[day]
99.1
12
0.954
0.149
10
0.996
0.089
Appendix
Table 4: Per-prompt circuit metrics. P0 is the full-model probability of the target token. nF is the smallest circuit size with F≥0.95 (reached by 15 of 16 prompts), and F(nF) , C(nF) are the metrics at that size; nC is the smallest size with C≤0.2 (reached by 14 of 16 prompts); F(20) , C(20) are the metrics at a fixed budget of 20 neurons. Entries marked – are not reached within the 50-neuron budget; medians of nF and nC are over prompts that reach the threshold. Row 1 is the circuit in Figure 1 .
Neurons n
1
2
3
5
8
10
15
20
25
30
40
50
F mean
0.11
0.21
0.30
0.49
0.74
0.85
0.98
1.01
1.02
1.03
1.03
1.03
F std
0.16
0.25
0.29
0.29
0.22
0.16
0.06
0.05
0.05
0.06
0.06
0.06
F median
0.04
0.09
0.16
0.47
0.82
0.90
0.99
1.00
1.00
1.00
1.01
1.01
C mean
0.92
0.84
0.76
0.62
0.45
0.35
0.26
0.20
0.17
0.12
0.09
0.07
C std
0.14
0.18
0.23
0.24
0.26
0.28
0.25
0.22
0.17
0.12
0.09
0.07
C median
0.96
0.89
0.86
0.66
0.41
0.26
0.19
0.12
0.11
0.08
0.06
0.06
Appendix
Table 5: Faithfulness ( F , higher is better) and completeness ( C , lower is better) as a function of circuit size n , aggregated over the 16 prompts (population std). Per-prompt values for every n∈[1,50] are in the released repository.
Figure 6: Circuit for “She felt very sad so her friend gave her a big warm” → [hug] ( p=89.1% ).
Figure 7: Circuit for “The cat saw a mouse and started to” → [chase] ( p=81.5% ).
Figure 8: Circuit for “The princess kissed the frog and it turned into a handsome” → [prince] ( p=96.9% ).
Figure 9: Circuit for “The robot beeped and said in a friendly” → [voice] ( p=94.8% ).
Figure 10: Circuit for “She planted a seed and watered it every” → [day] ( p=99.1% ).
Figure 11: Circuit for “The pirate looked at the map to find the hidden” → [treasure] ( p=91.7% ).
Figure 12: Circuit for “He washed his hands before sitting at the dinner” → [table] ( p=98.8% ).
Figure 13: Circuit for “She put on her shoes and ran out the” → [door] ( p=99.9% ).
Figure 14: Circuit for “She opened the book and started to” → [read] ( p=88.8% ).
Figure 15: Circuit for “He wrapped the gift in colorful shiny” → [paper] ( p=97.7% ).
Figure 16: Circuit for “She drank the hot soup and felt much” → [better] ( p=99.7% ).
Figure 17: Circuit for “She brushed her hair and looked in the” → [mirror] ( p=99.9% ).
Figure 18: Circuit for “She baked a cake and put it in the” → [oven] ( p=99.6% ).
Figure 19: Circuit for “The baby dropped her toy and started to” → [cry] ( p=83.8% ).
Figure 20: Circuit for “He stacked the blocks until the tower was very” → [tall] ( p=87.2% ).
Figure 21: L0 - N2286 activates on positive adjectives ( good, great ) and friendly interpersonal contexts.
Figure 22: L0 - N2993 detects the relative pronoun who (and its variants whom ) in narrative contexts, particularly after a character introduction.
Figure 23: L0 - N7792 responds to planning and discussion verbs ( discussed, brainstorm, plan ).
Figure 24: L1 - N222 activates on the preposition in in contexts where a character notices or discovers something in their surroundings (e.g. “a bright light in the sky” , “something in the distance” ). Its output weight vector promotes calm, positive continuations ( relax, comfortable, enjoy, trust ).
Figure 25: L1 - N3150 activates on the comma following a character’s speech or completed action, marking a narrative continuation point. Its output weight vector promotes continuation tokens ( and, he, it, change ), suggesting it signals that another clause or reaction is about to follow.
Figure 26: L1 - N3675 activates on the verb listen in negated contexts ( “did not listen” , “does not listen” ), typically in scenes involving disobedience or conflict between characters. Negative logits include tape, hides, reward .
Figure 27: L1 - N5005 detects geological objects: stone, mineral, crystal . Its top examples consistently mention finding or holding a stone.
Figure 28: L2 - N467 marks the word Today at the beginning of a narrative sentence, often followed by a comma, signalling a temporal shift in the story.
Figure 29: L2 - N1091 encodes a character’s enjoyment of an activity: top phrases are “loved to play” , “loved to crawl” , “loved to draw” .
Figure 30: L2 - N1441 activates on sentence - initial pronouns ( She, He, They ) that introduce a new action or dialogue, e.g. “She held her pocket tight” .
Figure 31: L2 - N3125 detects a moral or lesson learned: top activations contain “learned a lesson” , “learned a hard lesson” , often at the end of a story.
Figure 32: L3 - N1169 represents the action of throwing: top examples are “threw stones” , “threw the model” , “threw a newspaper” .
Figure 33: L3 - N1609 activates on the gerund being at moments of emotional or behavioural shift in a character - particularly when a character is described as being naughty , being stubborn , or stopped being a certain way. Its output weight vector promotes anger- and emotion-related tokens ( angry, upset, jealous, naughty, mad ).
Figure 34: L3 - N3412 activates on the preposition from in the construction “tired from [activity]” (e.g. “tired from playing” , “tired from working” , “tired from all the walking” ), signalling physical exhaustion caused by a preceding effort.
Figure 35: L3 - N3501 activates on the punctuation mark (colon or dash) that follows a phrase like “something they had never seen before” , “something beautiful” , or “something strange” - the moment just before the surprising object is revealed or described.
Figure 36: L4 - N2893 represents dizziness or spinning: top examples contain “dizzy” (often split as “dizz y” ) from spinning too much.
Figure 37: L4 - N3860 marks a temporal turning point: “From then on” , “From now on” , signalling that a character changes behaviour after an event.
Figure 38: L4 - N4150 detects loud sounds: top activations are “loud noise” , “loud bang” , “loud noises” , often from fireworks or broken objects.
Figure 39: L4 - N5981 encodes a commitment or resolution: “From that day on, [character] made sure to always” followed by a cautious behaviour.
Figure 40: L4 - N6211 activates on verbs describing a downward or de-escalating motion ( calmed, bent, fell, broke ) in the construction “[verb] down” , particularly when a character or situation settles or physically descends. Negative logits include up, down (as standalone tokens) and loves .
MLP neurons in transformer language models are widely understood to be polysemantic, motivating transcoders - sparse, learned approximations of MLP layers - as interpretable replacement models for mechanistic analysis. However, recent work shows that MLP neurons can be used to recover sparse, interpretable circuits (Arora et al. 2026, Ye et al. 2025), raising the question of whether transcoders are necessary for circuit analysis. We define neuron features as transcoder features whose weights are strongly aligned with individual MLP neurons, and show that they account for a substantial fraction of active features in late layers across 13 open-source transcoders. Only neuron features have a stable correspondence with something in the underlying MLP: other features vary more between training runs, do not match the MLP's Jacobian, and change less significantly from their random initialization. Our findings reframe MLP neurons as a stronger basis for interpretability than commonly assumed, and call for better methods of attributing the non-neuron structure that is used by the model.
A central goal of mechanistic interpretability is to understand how neural networks work and what each individual component does. Dominant circuit-finding approaches focus on a specific behavior and reverse-engineer the role of components on the associated sub-distribution. However, past work has shown that components can have different functions that are active on different subsets of the input distribution. In this work we ask whether a single weight can be understood globally across the full training distribution by characterizing when it matters (the inputs on which ablating it changes the model's predictions). We introduce an automated LLM pipeline that writes a short, human-readable description of when a weight matters and verifies it on held-out text, crediting a weight only if its description generalizes. Across two sparse and two dense transformers, the fraction of weights that are interpretable (in this sense) is higher in sparse transformers than in dense ones, a gap that widens once unreliable descriptions are discarded. Our results show that a meaningful fraction of a sparse transformer model's weights can be interpreted: 12 to 31% of weights have a single short description that identifies what the weight is used for.
Arnau Marin-Llobet, Stefan Heimersheim
School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138
Language models are thought to exhibit the phenomenon of superposition, representing many more features than dimensions in their residual streams. Sparse autoencoders (SAEs) are designed to recover such features post-hoc, but training models that are interpretable by construction has remained impractical, as a per-layer over-complete bottleneck is prohibitively expensive in both memory and compute. To overcome this issue, we introduce the ParityTransformer, a GPT-2-scale architecture whose intermediate representations are efficient and wide / sparse by design. At each layer, a Deep Parity Bottleneck (DPB) replaces a learned over-complete basis with a parameter-free algebraic dictionary, providing a deterministic incoherence guarantee and eliminating the memory requirements that have prevented per-layer interpretable bottlenecks at scale. A DPB is a hierarchically structured sparse bottleneck which efficiently enforces sparsity using a multi-level mixture-of-experts approach: a hardware-aware implementation that closes the cost gap between activation sparse and dense training to a manageable interpretability tax. Empirically, ParityTransformers perform at least as well as post-hoc SAEs on sparse probing tasks, while out-performing on measures of feature absorption, steering effectiveness, and fine-grained causal interventions. Because subsequent computation acts only on features that survive the sparse bottleneck, the ParityTransformer's features are native to the model's forwards pass by construction, addressing the question of whether SAEs probe features the model actually uses during computation. We see this as a step toward training models whose internal representations are interpretable by design rather than recovered post hoc.