Comparing Sparsified MLPs and Transcoders: Many Transcoder Features Are Neurons in Disguise

Stepan Shabalin; Mateusz Piotrowski

cs.AIOct 17, 2026

Comparing Sparsified MLPs and Transcoders: Many Transcoder Features Are Neurons in Disguise

Authors: Stepan Shabalin, Mateusz Piotrowski

Organizations: Anthropic · Georgia Institute of Technology

Abstract

MLP neurons in transformer language models are widely understood to be polysemantic, motivating transcoders - sparse, learned approximations of MLP layers - as interpretable replacement models for mechanistic analysis. However, recent work shows that MLP neurons can be used to recover sparse, interpretable circuits (Arora et al. 2026, Ye et al. 2025), raising the question of whether transcoders are necessary for circuit analysis. We define neuron features as transcoder features whose weights are strongly aligned with individual MLP neurons, and show that they account for a substantial fraction of active features in late layers across 13 open-source transcoders. Only neuron features have a stable correspondence with something in the underlying MLP: other features vary more between training runs, do not match the MLP's Jacobian, and change less significantly from their random initialization. Our findings reframe MLP neurons as a stronger basis for interpretability than commonly assumed, and call for better methods of attributing the non-neuron structure that is used by the model.

Figures & tables

Figure 2: A summary of findings from our evaluations and case studies.

Figure 3: Reconstruction and replacement (a) and sort eval monosemanticity (b) for neurons vs transcoders. Top neuron activations explain most of the model’s computation and are similarly interpretable to transcoder feature activations.

Table 3

Figure 4

Figure 5

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

	Contents
Appendix A	Appendix A Model and transcoder details
Appendix B	Appendix B Evaluation methodology
Appendix C	Appendix C Full evaluation results
Appendix D	Appendix D Features not aligned to single neurons
Appendix E	Appendix E Neuron feature prevalence and coverage
Appendix F	Appendix F Panels of neuron and neuron feature co-activation

Appendix

Table 6

Model	HF slug	Layers	$d_{\text{model}}$	$d_{\text{mlp}}$	Activation
Dense
Gemma 2 2B	google/gemma-2-2b	26	2304	9,216	GeGLU
Gemma 3 1B	google/gemma-3-1b-pt	26	1152	6,912	GeGLU
Gemma 3 4B	google/gemma-3-4b-pt	34	2560	10,240	GeGLU
Gemma 3 12B	google/gemma-3-12b-pt	48	3840	15,360	GeGLU
Gemma 3 27B	google/gemma-3-27b-pt	62	5376	21,504	GeGLU

Appendix

Table 3: Model architectures used in this work. All models use gated MLPs ( up , gate , down ). GeGLU uses

\mathrm{GELU}_{\mathrm{tanh}}

[ Hendrycks and Gimpel, 2016 ] for the gate, SiGLU uses SiLU. Llama and Qwen use standard pre-norm around each sublayer. Gemma 3 is unusual in additionally applying a post_feedforward_layernorm after the MLP; this requires dividing by the LayerNorm weight to recover a faithful encoder/decoder geometry ( Section D.1 ) and shows up, among other things, as higher KL on Gemma 3 transcoders compared to the matched Llama/Qwen ones at similar

L_{0}

(Figure 12 ). For the two mixture-of-experts models,

d_{\text{mlp}}

is written as the shared expert’s width plus the number of routed experts times their width. Both models have 8 active experts per token. Gemma 4 26B-A4B is used for bigram probing ( Table 7 ) and for testing expert concentration in attribution graphs ( Appendix N ). Qwen 3.5 35B-A3B is the model we train mixture of experts transcoders on ( Appendix O ). Model families used: Gemma 2 [ Gemma Team, 2024 ] , Gemma 3 [ Gemma Team, 2025 ] , Gemma 4 [ Gemma Team, 2026 ] , Qwen 3 [ Qwen Team, 2025 ] , Qwen 3.5 [ Qwen Team, 2026a ] , Llama 3 [ Llama Team, 2024 ] .

Transcoder	Model	Act.	Width	$L_{0}$	Alive@66%	NF@66%	NF% acts@66%
GS-small	Gemma 3 1B	JumpReLU	262k	45	193k ( $74\%$ )	15k ( $5.9\%$ )	$57.9\%$
GS-big	Gemma 3 1B	JumpReLU	262k	148	212k ( $81\%$ )	24k ( $9.1\%$ )	$57.5\%$
GS-small	Gemma 3 4B	JumpReLU	262k	47	175k ( $67\%$ )	23k ( $8.9\%$ )	$63.8\%$
GS-big	Gemma 3 4B	JumpReLU	262k	154	200k ( $76\%$ )	33k ( $12.7\%$ )	$66.6\%$
GS-small	Gemma 3 12B	JumpReLU	262k	47	165k ( $63\%$ )	41k ( $15.6\%$ )	$73.7\%$
GS-small	Gemma 3 27B	JumpReLU	262k	41	194k ( $74\%$ )	97k ( $37.1\%$ )	$82.8\%$

Appendix

Table 4: Models used in the analysis of open source transcoders. GS is Gemmascope on Gemma 3 ( google/gemma-scope-2-{1,4,12,27}b-pt ); mwhanna is mwhanna/qwen3-{1.7,4,8,14}b-transcoders ; LXTC-8x and LXTC-32x are OpenMOSS-Team/Llama3_1-8B-Base-LXTC-{8,32}x ; mntss is mntss/clt-gemma-2-2b-426k and mntss/clt-llama-3.2-1b-524k (cross-layer transcoders; the width in the table is the per-layer width and the width in the name is the maximum width of the cross-layer decoder). All metrics measured on FineWeb at layer

\lfloor 2N/3\rfloor

(where

N

is the number of layers; at

66\%

of the way into the model).

L_{0}

= mean active features per token. Alive = features with

\mathrm{fire\_count}>0

; NF = features with encoder and decoder cosine similarity both above

0.6

to the same neuron; NF% acts = share of feature firings that come from neuron features. The last two rows are the skip transcoders mntss/skip-transcoders-gemma-2-2b and mntss/transcoder-Llama-3.2-1B , used for sparse probing ( Table 6 ). Neither release states its activation function; we measure TopK for the Llama one (FVU

0.15

against

0.99

for ReLU) and assume it for the Gemma one, which we do not measure further.

Figure 13: Gated-MLP neuron geometry. Per-neuron activations are heavy-tailed (left); high-magnitude activations split into two regimes that share the same gate-positive half-plane (centre); and within each regime the effective encoder direction concentrates along

W_{\mathrm{up}}^{(i)}\pm W_{\mathrm{gate}}^{(i)}

across neurons (right).

Model	Neurons	Transcoder	$\Delta$
Llama 3.1 8B	0.856	0.845	$+0.011\pm 0.007$
Gemma 3 4B	0.854	0.832	$+0.022\pm 0.008$
Qwen 3 8B	0.851	0.806	$+0.045\pm 0.011$
Qwen 3 4B	0.833	0.794	$+0.039\pm 0.011$
Qwen 3 1.7B	0.811	0.762	$+0.050\pm 0.010$
Gemma 3 1B	0.834	0.792	$+0.042\pm 0.011$

Appendix

Table 5: Sparse probing mean AUC at

k{=}1

across the

113

[ Kantamneni et al., 2025 ] tasks. Neurons : dense MLP neurons. Transcoder : best-available transcoder per model ( Table 4 ).

\Delta

shows the per-task mean difference. Neurons match or exceed the transcoder across all models. Per-task and full

k

-sweep tables in Appendix C.1 .

Model	Dictionary	Neurons	Dictionary	Neurons	Dictionary	Neurons	Dictionary	Neurons	Dictionary	Ties
	Name	AUC, $k\,{=}\,1$		AUC, $k\,{=}\,4$		AUC, $k\,{=}\,16$		Tasks won ( $k\,{=}\,1$ )
Llama 3.1 8B	LXTC-8 $\times$	0.883	0.843	0.908	0.888	0.925	0.911	79	15	19
Llama 3.1 8B	LXTC-32 $\times$	0.883	0.839	0.908	0.886	0.925	0.910	80	15	18
Gemma 3 4B	GS-big	0.877	0.849	0.899	0.888	0.919	0.907	70	18	25
Qwen 3 8B	mwhanna	0.876	0.829	0.902	0.877	0.921	0.900	83	13	17
Qwen 3 4B	mwhanna	0.864	0.819	0.891	0.866	0.909	0.890	79	15	19

Appendix

Table 6: Sparse probing results across nine (model, dictionary) pairs on the

113

-task suite of Kantamneni et al. [2025] . Neurons : post-GeGLU/SiLU sign-split neurons. Dictionary : canonical post-activation learned dictionary (Appendix B.1 ). Mean peak test AUC across the

113

tasks (peak = max over layers per task), then averaged across tasks; bold marks the higher of the two within each

(k,\text{row})

. Tasks won columns: number of tasks where one method beats the other by

{>}\,0.005

AUC at

k\,{=}\,1

; ties otherwise. Bold marks the larger of the two win counts per row. Win rates and AUC gaps narrow with

k

but the sign of the gap is preserved at every

k

on every model.

	Gemma 3 4B				Qwen 3 4B				Gemma 4 26B-A4B
	Neurons		Gemmascope		Neurons		Matryoshka TopK		Neurons		Neurons, routed
Layer	$k{=}1$	$k{=}8$	$k{=}1$	$k{=}8$	$k{=}1$	$k{=}8$	$k{=}1$	$k{=}8$	$k{=}1$	$k{=}8$	$k{=}1$	$k{=}8$
L0	.891	.991	.988	.996	.946	.995	.992	.999	.723	.895	.902	.983
L1	.886	.982	.977	.990	.955	.993	.987	.998	.844	.977	.957	.997
L2	.854	.987	.985	.995	.950	.992	.979	.997	.833	.983	.937	.986
L3	.875	.985	.984	.996	.949	.990	.977	.996	.783	.982	.929	.987

Appendix

Table 7: Per-layer mean test AUC on the

1{,}500

bigram probing tasks (Appendix B.2 ), at

k\,{\in}\,\{1,8\}

selected features per probe. Match reports the rate at which the chosen (best

k\,{=}\,1

across L0–L5) feature’s top-1 max-activating example contains the target bigram. The transcoder wins every AUC comparison on every model, and its features match the bigram in their top-1 example far more often than sparsified neurons do. The MoE model, Gemma 4, is close to the some of the transcoders’ level of performance; this is preliminary evidence that scaled up MoE training could lead to similar monosemanticity to that seen in transcoders.

Our TopK pipeline
	$32$ k	$65$ k	$131$ k	$262$ k	$524$ k	$1$ M
TopK $k{=}64$	$32$ k	$65$ k	$130$ k	$259$ k	—	$895$ k
TopK $k{=}128$	$32$ k	$65$ k	$131$ k	$261$ k	—	$939$ k
TopK $k{=}32$	$32$ k	$64$ k	$126$ k	$247$ k	—	$754$ k
JumpReLU pipeline
JumpReLU $L_{0}{\approx}30$	—	—	—	$202$ k	$263$ k	$262$ k

Appendix

Table 8: Effective dictionary size at width

M

, defined as the number of features with non-zero firing on a buffer of 100 steps (

\sim 200

k tokens).

Figure 14

Model + transcoder	early	late	early	late
	Dict.		Active
Gemma 3 1B L0-small	2.1	3.6	27	48 [ $1.8\times$ ]
Gemma 3 1B L0-big	4.9	6.7	46	52 [ $1.1\times$ ]
Gemma 3 4B L0-small	2.6	7.2	22	53 [ $2.4\times$ ]
Gemma 3 4B L0-big	6.7	11.2	41	59 [ $1.5\times$ ]
Gemma 3 12B small	2.5	10.7	12	43 [ $3.6\times$ ]

Appendix

Table 9: Neuron features in 13 open-source transcoders, defined by encoder

{>}\,0.7

and decoder

{>}\,0.7

to the same neuron. Numbers are % of the dictionary ( Dict. ) and % of active features ( Active ), averaged separately over layers in the first vs. second half of each model. Brackets in the rightmost column give the ratio between the share in late and early layers. See Appendix A for transcoder details.

Threshold	32k	65k	128k	256k	1M
$\|\cos\|\,{\geq}\,0.5$	$86.0$	$91.7$	$94.1$	$96.4$	$96.7$
$\|\cos\|\,{\geq}\,0.7$	$78.3$	$86.7$	$89.9$	$93.3$	$93.0$
$\|\cos\|\,{\geq}\,0.9$	$55.9$	$71.5$	$78.1$	$86.0$	$84.6$

Appendix

Table 10: Percentage of Gemma 3 4B L22 MLP neurons matched by at least one TopK transcoder feature across dictionary size (columns) and cosine threshold (rows).

Layer	$n$	med. AUC	enc bin	$n$	med. Gini
5	35	0.911	$[0.30,0.50)$	29	0.224
10	37	0.555	$[0.50,0.70)$	15	0.319
15	38	0.873	$[0.70,0.85)$	35	0.956
20	31	0.998	$[0.85,1.00]$	119	0.996
25	28	0.993
30	36	0.983

Appendix

Table 11: Left: per-layer median AUC. Right: median Gini (rescaled AUC) by encoder cosine similarity: only features we call neuron features have high median discriminability.

		$L_{0}\,{=}\,100$			$L_{0}\,{=}\,25$			$L_{0}\,{=}\,10$
Model + transcoder	RS	early	mid	late	early	mid	late	early	mid	late
Gemma 3 1B small	0.62	23 $\pm$ 0	37 $\pm$ 1	46 $\pm$ 1	16 $\pm$ 0	33 $\pm$ 1	45 $\pm$ 1	13 $\pm$ 0	25 $\pm$ 1	51 $\pm$ 1
Gemma 3 4B small	0.56	25 $\pm$ 0	39 $\pm$ 0	60 $\pm$ 1	15 $\pm$ 0	29 $\pm$ 1	56 $\pm$ 1	13 $\pm$ 0	24 $\pm$ 1	60 $\pm$ 1
Gemma 3 12B small	0.50	19 $\pm$ 0	36 $\pm$ 1	62 $\pm$ 1	16 $\pm$ 0	37 $\pm$ 1	66 $\pm$ 1	13 $\pm$ 0	37 $\pm$ 1	75 $\pm$ 2
Gemma 3 27B small	0.49	22 $\pm$ 0	41 $\pm$ 1	70 $\pm$ 1	15 $\pm$ 0	30 $\pm$ 1	72 $\pm$ 2	11 $\pm$ 0	26 $\pm$ 2	77 $\pm$ 3
Gemma 2 2B mntss	0.66	17 $\pm$ 0	44 $\pm$ 1	49 $\pm$ 1	11 $\pm$ 0	34 $\pm$ 1	45 $\pm$ 1	10 $\pm$ 0	27 $\pm$ 1	43 $\pm$ 1

Appendix

Table 12: Percentage of neuron features in attribution graphs across nine open-source transcoders, by layer group and pruning strength. RS : median replacement score over circuits; lower for only the larger Qwen 3 transcoders. early / mid / late : percentage of neuron features in each of the three groups of layers.

Figure 43: “Red” case study on Gemma 3 4B L22. Top: attribution graph for “Translation of the word красное :” (predicting “Red”/“red”), with N5039 highlighted. Bottom: max-activating contexts for selected MLP neurons, with activations N7405 +

=\,8.00

, N5039 +

=\,5.31

, N10102 +

=\,1.40

, N4137 +

=\,0.9

, N10100 +

=\,0.86

Figure 44: “Red” case study,

k=64

TopK transcoder. Attribution graph shows two mildly (light orange; encoder alignment, from 0 to 1, is represented as color, from white to orange) neuron-aligned features and one strongly neuron-aligned feature (F138722). N10102, the neuron it is aligned to, activates on “red”/“ted” (§ 45(d) ). The other two features are weakly aligned to neuron 5039, and both activate only on the token “red”. Figure 46 shows that they both cluster the neuron.

Figure 45: “Red” case study, Gemmascope transcoder. Attribution graph shows three neuron-aligned features in dark orange. One feature is aligned to a color neuron, 7405. Another is aligned to N10102, a “red”/“ted” neuron, the activations of which are shown in the bottom right. The third one is a neuron feature for N5039 and shows the same polysemanticity between “red” and “transparent”.

Our transcoders
Transcoder	$p_{1}$	$p_{2{:}4}$
Baseline $k{=}64$	—	—
Baseline $k{=}32$	0.64	0.77
NI $k{=}32$	0.66	0.70
NFT $k{=}32$ , $G{=}16,r{=}4$	1.00	0.66
SNRL $k{=}32$	0.71	0.57

Appendix

Table 13: Absorption evaluation for different transcoder-based clustering methods on Gemma 3 4B L22, with top

K{=}128

neurons considered active and

10

M SmolLM2 tokens.

p_{1}

is the probability the feature’s top-1 weight-aligned neuron is in the active set;

p_{2{:}4}

the same for the next three weight-aligned neurons.

Our transcoders
Transcoder	FVU $\downarrow$	KL $\downarrow$	Jac. faith. $\uparrow$
Baseline $k{=}64$	0.166	0.77	0.276
Baseline $k{=}32$	0.195	1.29	0.225
NI $k{=}32$	0.209	1.16	0.238
NFT $k{=}32$ , $G{=}16,r{=}4$	0.221	1.15	0.230
SNRL $k{=}32$	0.213	1.38	0.232

Appendix

Table 14: Reconstruction quality and Jacobian faithfulness on Gemma 3 4B layer 22. FVU and KL are over

1

M held-out SmolLM2 tokens. “Jac. faith.” is the cosine similarity between the MLP Jacobian and the transcoder Jacobian, divided by Layernorm for the Gemmascopes: the post-LN scaling factor

(1{+}w_{\text{ln}})

is removed so the comparison lives in pre-LN MLP-output space. FVU, however, is measured against a different target for our transcoders and Gemmascopes.

Our transcoders
Clustering	CM acc $\uparrow$	TNM acc $\uparrow$
Baseline $k{=}32$	$0.937\pm 0.005$	$0.731\pm 0.030$
NI $k{=}32$	$0.918\pm 0.008$	$0.629\pm 0.031$
NFT $k{=}32$ , $G{=}16,r{=}4$	$0.918\pm 0.010$	$0.598\pm 0.033$
SNRL $k{=}32$	$0.927\pm 0.008$	$0.657\pm 0.030$
Gemmascope

Appendix

Table 15: Within-cluster sort eval results on 50 random Gemma 3 4B L22 neurons (judge: Claude Sonnet 4.6, 60 trials per neuron). Up to 10 neurons per transcoder produced too few viable clusters and were dropped (more for Gemmascope, which has sparser per-panel cluster coverage). CM acc = Cluster match accuracy (SE over panels), TNM acc = Top-neuron match accuracy (binomial SE over panel-cluster pairs).

Tool	License
sparsify 32 32 32 Upstream for our TopK pipeline.	MIT
circuit-tracer [ Hanna et al., 2025 ]	MIT
sae-probes	MIT
nnsight	MIT
transformers , datasets , accelerate	Apache 2.0
vllm , safetensors , huggingface-hub , ftfy	Apache 2.0

Appendix

Table 16: Open source libraries and tools used.

Transformer Interpretability

Transformer interpretability studies how attention, feed-forward layers, and hidden representations support model predictions and behavior.

See all 334 papers

Oct 14, 2026

Sparse Transformer: a Controlled Testbed for Mechanistic Interpretability

Mechanistic interpretability aims to decompose neural network computations into human-understandable features, yet progress is limited by the absence of ground-truth units to evaluate against. We address this with a Sparse Transformer (ST): a 5-layer language model trained on TinyStories whose wide, TopK-sparse MLP forces neuron specialization. We show that ST neurons are highly interpretable: they outperform a loss-matched dense transformer by 43% on AutoInterp, and circuits of a median of 12 neurons suffice to reproduce individual model predictions. This testbed allows us to directly compare feature learning methods: Transcoder, which approximates the MLP input-output mapping, recovers ST's neuron vectors with 52.6% higher cosine similarity than Sparse Autoencoder, which only reconstructs outputs. Furthermore, the ST enables a controlled study of polysemanticity: distilling its sparse MLP into a dense counterpart reveals how compression leads to feature superposition. These results show how a known reference basis sharpens evaluation and understanding, a role we hope the ST can serve for the field.

Anton Korznikov, Alexey Dontsov, Elena Tutubalina

Mechanistic Interpretability Neuron-Level Interpretability Transformer Sparse Autoencoders Feature Superposition

Ask PDF

Jun 14, 2026

DifFRACT: Diffusion Feature Reconstruction and Attribution for Circuit Tracing

Mechanistic interpretability seeks to explain neural network behavior by decomposing model computations into interpretable features and circuits. While transcoder-based circuit tracing has recently enabled detailed causal analyses of large language models, multimodal diffusion transformers for image generation remain comparatively opaque. We still lack tools for understanding how semantic information propagates across denoising steps and how text and image representations interact within double-stream MM-DiT architectures. Existing methods provide only partial insight: attention maps expose a limited view of token interactions, while sparse autoencoders can discover interpretable features but do not directly reveal how these features are transformed and composed through nonlinear MLP layers. In this work, we extend transcoder-based circuit tracing to multimodal diffusion transformers. We train timestep-conditioned transcoders that faithfully approximate the input-output behavior of MLP sublayers in FLUX.1[schnell]. By replacing MLPs with transcoders and linearizing the remaining computation, we obtain exact feature-to-feature attribution and recover compact, interpretable circuits. Empirically, our transcoders match or slightly outperform sparse autoencoders on the sparsity-faithfulness tradeoff. The resulting circuits reveal mechanisms underlying attribute binding and cross-stream semantic propagation, and provide causal explanations for systematic generation errors. Moreover, circuit-guided interventions are substantially more precise and effective than standard SAE-based steering. Our results demonstrate that transcoder-based circuit analysis is feasible for state-of-the-art diffusion transformers and provides a powerful framework for understanding and controlling multimodal generative models. The code is available at https://github.com/Artalmaz31/DifFRACT

Artyom Mazur, Nina Konovalova, Aibek Alanov

HSE University · AXXX

Diffusion Transformer Multimodal Model Interpretability Multimodal Diffusion Models

Ask PDF

Jul 13, 2026

Sparse Inter-Layer Dependencies of Transformer FFN Neurons

Feedforward network (FFN) blocks account for a large fraction of the parameters and computation in Transformer architectures, yet their internal structure remains difficult to interpret due to the additive superposition induced by the residual stream. We examine whether the activation of an FFN neuron can be explained by a sparse set of preceding neuron activations and attention outputs. We introduce a training-free attribution method that estimates the relative influence of upstream neurons and attention outputs on a target neuron's activation. Empirically, across models and layers, we find that small subsets of preceding activations and attention outputs suffice to preserve neuron activations with high fidelity when all remaining inputs are masked with their average values. Effective sparsity is even greater when accounting for the inherent activation sparsity of upstream layers. Moreover, applying the neuron-specific masks in all layers simultaneously, such that the induced deviations propagate through the network, leaves model perplexity largely unchanged at moderate sparsity levels. These results demonstrate that, despite dense parameterization, FFNs exhibit sparse and structured inter-layer dependencies at the neuron level. Our method provides a practical, scalable tool for circuit-level interpretability and identifies candidate sparse pathways with potential implications for efficient inference.

Johannes Knittel, Hanspeter Pfister

Harvard University

Transformer Interpretability Transformer FFNs Neural Network Interpretability Neuron-Level Interpretability

Ask PDF

Abstract

Transformer Interpretability

Explore similar work

Sparse Transformer: a Controlled Testbed for Mechanistic Interpretability

DifFRACT: Diffusion Feature Reconstruction and Attribution for Circuit Tracing

Sparse Inter-Layer Dependencies of Transformer FFN Neurons