Disentangling MLP Neuron Weights in Vocabulary Space

Asaf Avrahamy; Yoav Gur-Arieh; Mor Geva

cs.CLApr 7, 2026

Disentangling MLP Neuron Weights in Vocabulary Space

Authors: Asaf Avrahamy, Yoav Gur-Arieh, Mor Geva

Organizations: Blavatnik School of Computer Science and AI, Tel Aviv University

Abstract

Interpreting the information encoded in language model weights remains a fundamental challenge in mechanistic interpretability. In this work, we introduce ROTATE (Rotation-Optimized Token Alignment in weighT spacE), a data-free method requiring no forward passes that disentangles MLP neurons directly in weight space. Our approach relies on a key statistical observation: neurons that encode coherent, monosemantic concepts exhibit high kurtosis when projected onto the model's vocabulary. By optimizing rotations of neuron weights to maximize their vocabulary-space kurtosis, our method recovers sparse, interpretable directions which we name vocabulary channels. Experiments on Llama-3.1-8B-Instruct and Gemma-2-2B-it demonstrate that ROTATE consistently recovers vocabulary channels that are faithful to the neuron's behavior; ablating individual channels selectively disables corresponding input activations or the promotion of specific concepts. Moreover, aggregating channel-level descriptions yields comprehensive neuron descriptions that outperform optimized activation-based baselines by 2-3x in head-to-head comparisons. By providing a data-free decomposition of neuron weights, ROTATE offers a scalable, fine-grained building block for interpreting language models.

Figures & tables

Figure 1: We propose to disentangle MLP neuron weights ( Left ) using ROTATE , a data-free method that learns rotations of a neuron’s weight vector

\mathbf{w}

to maximize kurtosis in the model’s vocabulary space, recovering sparse, interpretable directions we call vocabulary channels ( Middle ). Each channel isolates a distinct concept encoded in

\mathbf{w}

, allowing a fine-grained understanding of the neuron’s mechanism across diverse inputs ( Right ).

Figure 2: Vocabulary kurtosis of concept vectors in

\mathbf{W}_{\text{out}}

( Hong et al., 2025 ) vs. random neurons from the same layers.

	Faithfulness				Completeness
	Llama-3.1		Gemma-2		Llama-3.1		Gemma-2
Method	$\ell=18$	$\ell=22$	$\ell=4$	$\ell=18$	$\ell=18$	$\ell=22$	$\ell=4$	$\ell=18$
ROTATE (Ours)	0.71	0.58	0.46	0.47	0.55	0.49	0.55	0.60
SAE-Neuronpedia	0.45	0.41	0.33	0.35	0.44	0.41	0.42	0.49
SAE-TopK	0.49	0.46	0.34	0.37	0.40	0.40	0.36	0.42
Random	0.25	0.20	0.17	0.24	0.20	0.20	0.20	0.20

Table 1: Average Faithfulness and Completeness scores. ROTATE consistently outperforms SAE-based baselines across models and layers. Random reflects chance-level performance.

Figure 3: Input-side causal validity. Ablating the neuron’s top channel drives its activation toward 0; ablating other channels leaves it near 1.

Model	Layer	Self (%)	Cross (%)
Gemma-2	4	$-90\pm 34$	$+24\pm 55$
-2b-it	18	$-87\pm 40$	$+24\pm 57$
Llama-3.1	18	$-90\pm 35$	$+15\pm 60$
-8B	22	$-88\pm 37$	$+14\pm 60$

Table 2: Output-side causal validity via channel ablation. Mean (

\pm

std) % change in token frequency after self- or cross-channel ablation.

Figure 4: Head-to-head pairwise evaluation of ROTATE vocabulary channel descriptions against MaxAct+VocabProj and MaxAct++ baselines on Llama-3.1-8B-Instruct. Each bar shows the fraction of comparisons won by ROTATE , tied , or won by the baseline . Columns correspond to layers; rows to evaluation data sources and activation-rank ranges.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: A distribution with high kurtosis and positive skewness, concentrated around zero with few extreme outliers (left), compared to a Gaussian (right).

Figure 6: Median vocabulary kurtosis values of neuron weights in $\mathbf{W}_{\text{out}}$ across layers and checkpoints of OLMo-2-1124-7B ( Walsh et al., 2025 ) . We observe clear learning dynamics, rising sharply in early training and concentrating in middle and late layers. This temporal pattern confirms that vocabulary-aligned monosemantic structure is a learned property.

Figure 7: Per-layer vocabulary kurtosis distributions of

\mathbf{W}_{\text{out}}

neurons for one representative model per family.

Figure 8: Weight reconstruction analysis on Gemma-2-2B-it. Left: Per-channel cosine similarity with the original weight vector $w$ across iterations. Right: Cumulative explained norm ( $1-\|r_{t}\|/\|w\|$ ) over iterations. Lines show medians across 99 neurons; shaded regions indicate inter-quartile ranges. Channels collectively reconstruct nearly all of $w$ within 50 iterations across all layers and weight types.

	Gemma-2-2b-it		Llama-3.1-8B-Instruct
Layer	Faithfulness	Completeness	Faithfulness	Completeness
2	46.9	40.7	–	–
4	45.7	55.7	49.7	41.3
8	45.2	43.5	49.3	35.8
18	46.1	59.7	70.9	55.1
22	–	–	57.5	49.3

Appendix

Table 3: Layer sweep of input-side faithfulness and completeness (%). All layers exceed the random baselines (faithfulness 20–25%, completeness 20%), with the strongest gains in middle-to-late layers.

Figure 10: Effect of token masking on iterative decomposition quality. We compare ROTATE with token masking against independent optimization runs with no depletion over 20 iterations on 50 gate neurons from Layer 18, Gemma-2-2B-it. Left: Per-channel cosine similarity with

\mathbf{w}

. Right: Cumulative explained norm (

1-\|\mathbf{r}_{t}\|/\|\mathbf{w}\|

). Without masking, all iterations converge to the same dominant direction, yielding negligible reconstruction progress (mean explained norm

0.19

vs.

0.78

Figure 11: Ablation results evaluating weight reconstruction across optimization iterations for $W_{\text{gate}}$ (left) and $W_{\text{out}}$ (right). We compare the ROTATE baseline (token masking, $k=1$ Householder matrix) against two variants: utilizing a proper rotation via two Householder matrices ( $k=2$ ), and using residual subtraction instead of token masking. Top: Cumulative Explained Norm ( $1-\|\mathbf{r}\|/\|\mathbf{w}\|$ ). Bottom: Cosine similarity between the reconstructed vector and the original weight vector. The baseline ( $k=1$ ) matches the performance of the more complex $k=2$ parameterization and consistently outperforms residual subtraction.

$\lambda$	Best $\eta$	Best $\sigma$	Explained Norm	Final Orthogonality	Harmonic Mean
0.3	$2\times 10^{-3}$	4.0	0.72	0.78	0.749
0.5	$8\times 10^{-4}$	6.0	0.63	0.87	0.731
0.1	$8\times 10^{-4}$	4.0	0.86	0.54	0.663

Appendix

Table 4: Summary of hyperparameter grid search, reporting the best performing configuration (by harmonic mean) for each kurtosis regularization coefficient (

\lambda

\eta

: learning rate,

\sigma

: standard deviation threshold for token masking.

\mathbf{w}_{\text{gate}}

ch 2: “negative, Negative” (

\sigma=2.16

)

Detects contexts involving negation/polarity.

\mathbf{w}_{\text{in}}

ch 1: “negative, positive” (

\mathbf{w}_{\text{in}}{\cdot}\mathbf{x}={+}1.76

)

Polarity concept signal (93% of top examples).

Aligns with input

\Rightarrow

\sigma(\cdot)\times(+)>0

Predicted:

>0

True:

+2.76

Appendix

Figure 12: Complete mechanistic decomposition of Neuron 9005 (Layer 18, Gemma-2-2B-it) via vocabulary channels. Top: The neuron activates positively on technical text with negation/polarity concepts and negatively on temporal deferral. Middle: ROTATE ’s input-side

\mathbf{w}_{\text{gate}}

and

\mathbf{w}_{\text{in}}

channels explain the sign of the activation, the

\mathbf{w}_{\text{gate}}

detects relevant context, while the

\mathbf{w}_{\text{in}}

channel’s alignment or anti-alignment with the input determines the sign. Bottom: Output-side channels, organized by skewness sign, reveal what the neuron writes to the residual stream. Positive activation promotes polarity vocabulary ( “negative” , “positive” ); negative activation promotes temporal negation tokens ( “wasn’t” , “until” , “wait” ). All channels are discovered from weights alone.

Model	Neuron	MLP type	Ch	Top tokens	Description
Gemma-2 -2b-it	(18, 6528)	$W_{\mathrm{gate}}$	0	ride, Ride, riding, rides, ridden	Direct riding vocab.
			47	platform, Platform, platforms	Platform
			38	school, School	Dampens school ctx.
		$W_{\mathrm{in}}$	0	ride, riding, rides, bike, horseback	Riding / locomotion
			16	donkey, donkeys, horse, horses, mule	Animals / mounts
			22	gl, Gl, GL	gl- subtoken

Appendix

Table 5: Selected vocabulary channels for two example neurons, across

W_{\mathrm{gate}}

W_{\mathrm{in}}

, and

W_{\mathrm{out}}

weight matrices. Top tokens (up to 5) shown per channel.

Figure 13: Visualization of four gate channels of Neuron 9005 (Layer 18, Gemma-2-2B-it). Each row shows a channel’s top vocabulary tokens, its natural-language description, and three activating examples alongside one neutral example. Token color indicates activation polarity (red: positive, blue: negative) and opacity scales with magnitude. The channels capture distinct concepts: temporal markers, polarity/negation, GUI programming tokens, illustrating the fine-grained, interpretable structure recovered by ROTATE from a single neuron’s weight vector.

Figure 14: Per-channel faithfulness scores for representative gate channels of Neuron 9005 (Layer 18, Gemma-2-2B-it). Each panel shows the distribution of peak neuron activations on activating (blue) vs. neutral (orange) sentences generated from the channel description. Channels shown all pass the one-sided t-test at

p<0.05

(indicated in the title of each panel), confirming that their descriptions reliably distinguish activating from non-activating inputs.

Variant	$\mathbf{w}_{\text{gate}}$ channels	$\mathbf{w}_{\text{in}}$ channels
All gate, all in	all	all
Positive-skew gate, all in	positive-skew only	all
All gate, positive-skew in	all	positive-skew only
All gate, negative-skew in	all	negative-skew only

Appendix

Table 6: Four aggregation strategies for ROTATE neuron descriptions. The last two variants separate positive and negative activation regimes by filtering

\mathbf{w}_{\text{in}}

channels according to the sign of their vocabulary-projection skewness, while retaining all

\mathbf{w}_{\text{gate}}

channels.

Method	Polarity	Variant	Win rate
ROTATE	positive	all_gate_split_positive	78.3%
		all_gate_all_in	35.0%
		positive_gate_all_in	31.7%
ROTATE	negative	all_gate_split_negative	57.5%
ROTATE	negative	all_gate_all_in	37.5%
MaxAct+VocabProj	positive	v2	67.5%

Appendix

Table 7: Stage 1 within-method variant win rates on 20 neurons from Llama-3.1-8B-Instruct. Bold denotes the selected variant for each method and polarity. For ROTATE , we select all_gate_split_positive (positive) and all_gate_split_negative (negative). For MaxAct+VocabProj, we select v2 for as it enriches the activation-based evidence with vocabulary-projection tokens from both

\mathbf{w}_{\text{gate}}

and

\mathbf{w}_{\text{in}}

, providing the baseline with the strongest available signal and ensuring the most competitive comparison against ROTATE .

Layer, Neuron	Activating example	ROTATE description	Baseline description	Win/Loss
L22, N6946	/// Get the host name associated with the entry. template <class Allocator> std :: basic_string, std::char_traits<char>, Allocator> host_name( const Allocator& alloc Pile top [100-500]	This neuron activates on contexts related to sleep, rest, and altered states of consciousness (dreaming, falling asleep), alongside concepts of returning or restarting, often involving function words ( to, of, you ) and morphological elements. Additionally, it responds to notions of bursting/failure, central locations/functions, and suspension/hanging, and code snippets related to filtering operations on arrays .	This neuron activates on words related to sleep, sleeping, snoring, and waking up, as well as general personal pronouns and common function words like “to”, “or”, and “of”, possibly reflecting awareness of narrative context involving sleep. [ MaxAct+VocabProj ]	Win
L22, N1939	"thumbnail", "file", "fanart", "streamdetails" ], " player id": 1 ], "id": "VideoGetItem" Check this out} Pile top [0-100]	This neuron activates in contexts blending organizational systems, financial elements, and technical details, particularly those involving data processing and structured information. This includes: pipelines and routing of data, archives and architecture, financial assets and payments, macro/micro scale comparisons, lists/catalogs, letters/alphabets, notes/records, and measurements of volume. It is also sensitive to names and identifiers, particularly those containing the letter sequence ’ee’	This neuron activates on code snippets, particularly related to the VLC media player library (libVLC) or JSON- RPC calls for media players (like XBMC), often involving player control methods. It also activates on articles, ’the’ and ’to’ [ MaxAct+VocabProj ]	Loss
L12, N496	We just cruised on her to the Panama Canal last week! The Maitre’De in * the* Posh Dining Room Goran Gorigjewski is awesome!! Pile top [0-100]	This neuron activates positively in contexts involving the definite article ’the’ alongside varied semantic themes including: workplace interactions; self-reference; code overrides; strength/resilience; sending/transmission; philosophical concepts/proper nouns; authentication (’login’); geographical locations/cardinal directions; physical actions; and potentially female names. This suggests an emphasis on contextually defined entities within narrative or technical contexts	proper nouns; context indicating inquiry or explanation [MaxAct++]	Win
L18, N2241	The English prose poem is a verse form that is usually unrhymed and written in the… FineWeb top [0-100]	This neuron strongly activates on code snippets, configurations, and technical documentation, often featuring specific numerical identifiers, compound words, and elements related to authorship or provenance. It also demonstrates sensitivity to partial words and specific syllables (’an’, ’on’, ’ol’, ’ug’, ’ac’) and common suffixes. Addition-related terms, Slavic language fragments, and spoiler/coupon contexts can also trigger activation.	references to poetic forms , styles, or innovation [MaxAct++]	Loss

Appendix

Table 8: Example wins and losses of ROTATE in head-to-head comparisons against MaxAct++ and MaxAct+VocabProj.

Panel of Figure 15 — Figure 15: Polarity-split neuron description synthesis prompt (§ 6 ). $\mathbf{w}_{\text{gate}}$ and $\mathbf{w}_{\text{in}}$ channel descriptions are provided separately; the LLM produces a unified description of at most 50 words. Used with Gemini-2.0-Flash.

Panel of Figure 16 — Figure 16: MaxAct+VocabProj baseline description prompt (§ 6 ). Combines 20 top-activating examples with LogitLens vocabulary projections of the $\mathbf{w}_{\text{gate}}$ and $\mathbf{w}_{\text{in}}$ vectors. Used with Gemini-2.0-Flash.

Panel of Figure 17 — Figure 17: Head-to-head pairwise evaluation prompt (§ 6 ). Each comparison is run twice with swapped order; a winner is declared only when both orderings agree. Used with Gemini-3-Flash.

Panel of Figure 18 — Figure 18: Prompt used to describe a single vocabulary channel. Each channel is described independently before synthesis into a neuron-level description. Used with Gemini-2.0-Flash.

Panel of Figure 19 — Figure 19: Prompt used to generate activating and neutral examples for the input-side faithfulness evaluation (§ 5.2 ). Default: 40 positive + 40 negative examples. Used with Gemini-2.0-Flash.

Panel of Figure 20 — Figure 20: 5-way channel matching prompt used for the completeness evaluation (§ 5.4 ). The LLM judge (Gemini-3.1-Flash-Lite) selects which of five candidate descriptions best matches the activating input.

LLM Interpretability

Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.

See all 778 papers

Sep 10, 2026

Distribution-aware Language Neuron Identification in Multilingual Large Language Models

Multilingual large language models (mLLMs) contain a small fraction of feed-forward neurons that are sensitive to particular languages, commonly termed language-specific neurons. Existing work measures language specificity using the entropy of each neuron's language-wise probabilities of being active, where a neuron is considered active when its activation value is positive. However, this approach may not fully capture the multilingual nature of mLLMs, where language representations are distributional and mutually related. We propose Distribution-aware Language Neuron selection, which leverages pairwise relationships between per-language activation distributions over the full activation range, including negative values. Specifically, we quantify each neuron's language specificity by clustering languages using pairwise overlap coefficients between their activation distributions. Across two mLLMs and two held-out corpora, our identifier more effectively isolates language-specific causal effects, yielding up to 4.9

\times

higher on-target language damage per neuron while preserving off-target language performance.

Minjun Kim, Inho Won, Junghun Yuk +3

KAIST · KAIST InnoCORE PRISM-AI Center

Multilingual Language Models LLM Interpretability

Ask PDF

Jun 10, 2026

Non-Gaussianity Reveals Interpretable Directions in Language Models

Finding human-interpretable directions in language-model activations is important for understanding and controlling model behavior. These directions are usually discovered through indirect learning objectives such as sparse reconstruction, but it remains unclear what existing statistical structure distinguishes interpretable directions and whether that structure can be used directly for feature discovery. We find that non-Gaussianity provides such a signature: projection distributions of natural-text residual-stream activations along sparse autoencoder (SAE) decoder directions are substantially more non-Gaussian than those along random directions, with most contexts forming a concentrated background and recurring patterns often occupying one tail. This raises a natural question: if interpretable directions tend to have non-Gaussian projection distributions, can we discover them by searching for non-Gaussian directions directly in activation space, without training another sparse dictionary? To test this hypothesis, we introduce ICA Lens, a practical workflow for efficient discovery, reliable annotation, and faithful evaluation of non-Gaussian directions in LLM residual-stream activations. Across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 9B Base, we find that: (1) the recovered non-Gaussian directions capture coherent lexical, syntactic, and semantic patterns under both human and LLM-based evaluation; (2) non-Gaussianity is positively associated with both human-label confidence and automated interpretability; and (3) the resulting coordinates are competitive with SAEs across interpretability, sparse reconstruction, concept prediction, and targeted probe perturbation, while fitting in minutes per layer and using 7-43 times less memory. These results show that non-Gaussianity is a direct unsupervised signal for discovering interpretable and useful structure in activations.

Sida Liu, Feijiang Han, Sarah Wiegreffe +1

Independent Researcher · University of Maryland

Independent Component Analysis LLM Interpretability Language Model Introspection Representation Geometry in Language Models

Ask PDF

Sep 4, 2026

Some Tokens Behave like Magnets: Revealing Linguistic Organization in the Layers of Language Models

We identify a special group of token vectors inside large language models (LLMs), which we term magnetic vectors, that organize the surrounding tokens by either attracting or repelling them. Particularly, tokens pointing the same way as an attracting magnet are elongated; tokens pointing the same way as a repelling magnet are compressed. Just as physical magnets pull or push away the iron filings around them, these vectors organize their surroundings through two opposing polarities. Moreover, we identify a statistically significant pattern in linguistic category where function words consistently act as repelling magnets in early layers, and we also find magnets consistently reorganize their polarities in unique ways deeper in the model. In a further case study we find this observation may unveil a deliberate, layer-wise organization in how LLMs process language. This pattern is consistent across different LLM architectures, sizes, and layer configurations. It is also causally relevant. When the LLM is fine-tuned for a downstream task, the task-functional tokens emerge as magnets. E.g., in question answering, the answer-span tokens become uniquely repelling magnets in the final layer, geometrically carving the answer out of the surrounding context. Furthermore, removing early-layer repelling magnets devastates syntactic tasks (POS tagging accuracy drops from 91% to below 10%) while sparing semantic ones, and removing late-layer attracting magnets does the reverse. We believe this phenomenon warrants further investigation, as it opens the first probe-free path to understanding how language models geometrically organize linguistic computation across their layers.

Andrew Liu, Devan Srinivasan, Gerald Penn

University of Toronto Department of Computer Science

LLM Interpretability Representation Geometry in Language Models Language Modeling

Ask PDF

Abstract

LLM Interpretability

Explore similar work

Distribution-aware Language Neuron Identification in Multilingual Large Language Models

Non-Gaussianity Reveals Interpretable Directions in Language Models

Some Tokens Behave like Magnets: Revealing Linguistic Organization in the Layers of Language Models