Sparse autoencoders (SAEs) have emerged as a powerful technique for decomposing language model representations into interpretable features. Current interpretation pipelines infer feature semantics from activation patterns, implicitly assuming every feature has a semantic explanation while overlooking the computational roles features inherit from the SAE training objective. We introduce a weight-based interpretation framework that requires no activation data and grounds claims causally through targeted feature-ablation interventions. Across Gemma-2 and Llama-3.1, three depth-dependent signatures show that SAE features inherit the base model's geometry: semantic features form a U-shape under tied embeddings but bifurcate when untied, attention participation peaks mid-layer, and the two roles couple oppositely for input- vs. output-oriented features. This weight-based view supplies the causal half missing from activation-based interpretability.
Figures & tables
Figure 1: Two views of SAE features. ( 1(a) ) An activation-based explanation read off the feature’s top-activating contexts: tokens where the feature fires are highlighted and the highest activations are boxed, and GPT-4o-mini summarizes the co-occurring text. The summary does not specify what changes in the model’s computation when the feature is suppressed. ( 1(b) ) A uniform sample of features that pass all three weight-based metrics in Section 4 on Gemma-2-9B at layer 27, shown with their top-10 predicted tokens. “L” is Levenshtein similarity, “C” is cosine similarity, and “E” is top-100 entropy; boxed scores correspond to the range of displayed tokens.
Figure 2: Joint and per-metric pass rates for the decoder–unembedding classifier on Gemma-2-2B and 9B. Both sizes show a U-shaped distribution; ablating any single metric uniformly increases the pass rate, confirming the three are complementary.
Projection
Model
Calibration layer
ρmin , p∈[25,75]
ρmin , p∈[40,60]
Decoder–unembedding
Gemma-2-2B
16
0.971
0.992
Gemma-2-9B
27
0.953
0.993
Llama-3.1-8B
20
0.971
0.997
Encoder–embedding
Gemma-2-2B
3
0.720
0.878
Gemma-2-9B
6
0.905
0.978
Llama-3.1-8B
3
0.785
0.964
Table 1: Stability of the per-layer joint pass-rate profiles. Each ρmin is the minimum Spearman correlation with the same cell’s p=50 profile over the tested percentiles in the specified band. Calibration layers remain fixed throughout the sweep.
Figure 4: Main results for Experiment 2 on Gemma-2-9B and Llama-3.1-8B: Magnitude of attention participation across layers, with the total height at each layer representing the aggregate pre-softmax score. These scores are calculated as the mean across all keys for each query feature, then summed for each head. The stacking direction of the head values is determined based on whether they are positive or negative. Each color represents a rank of magnitude, rather than a specific head index, across all layers.
Figure 5: Per-layer query-side attention-weight densities for semantic vs. non-semantic features. Left: Gemma-2-9B layer 20, where the semantic distribution sits below the non-semantic one. Right: Llama-3.1-8B layer 4, in the input-semantic phase, where the relation inverts and semantic features carry the higher attention weights.
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Spearman rank correlation ( ρ ) matrix for 23 candidate semantic metrics, calculated on all features from the Gemma-2-9B layer 27 SAE.
Figure 7: Pairwise scatter samples illustrating the relationships between the three selected semantic metrics for the Gemma-2-9B layer 27 SAE. The plots demonstrate that while the metrics are not entirely independent, their correlations are moderate. This supports their selection as a complementary set, where each metric provides a distinct perspective for evaluating the semantic coherence of a feature.
Figure 8: Distributions of the three primary semantic metrics—Levenshtein similarity (higher is better), cosine similarity (higher is better), and top-100 entropy (lower is better)—for all features in the Gemma-2-9B layer 27 SAE. The distributions are continuous and generally unimodal, lacking a distinct bimodal structure or natural cutoff point. This characteristic motivates our use of a percentile-based thresholding approach in Experiment 1, as it provides a consistent relative standard for what constitutes a “semantic” feature, rather than relying on an arbitrary absolute value. The distribution plot applies normalization (RMSNorm) because, without it, the entropy is highly concentrated near the maximum value with extreme outliers, making visualization difficult.
Decoder–unembedding
Encoder–embedding
p
Gemma-2-2B
Gemma-2-9B
Llama-3.1-8B
Gemma-2-2B
Gemma-2-9B
Llama-3.1-8B
25
50.4
50.5
33.4
48.8
44.9
12.5
30
43.5
43.8
30.8
42.1
39.0
11.5
35
37.4
38.2
28.3
36.8
34.5
10.7
40
32.3
33.7
25.8
32.2
30.8
10.1
45
28.0
29.4
23.4
28.3
27.8
9.5
Appendix
Table 2: Threshold sensitivity of the three-criteria classifier. Entries are mean joint pass rates (%) across all layers. The joint strictness percentile p raises both similarity cutoffs and lowers the entropy cutoff; the bold row reproduces the paper’s calibration.
Projection
Model
Layers
Sensitivity
Specificity
Agreement
Cohen’s κ
Decoder– Unembedding
Gemma-2-2B
2, 5, 8, 11, 14, 17, 20, 23
0.916
0.908
0.912
0.824
Gemma-2-9B
5, 10, 15, 20, 25, 30, 35, 40
0.954
0.862
0.908
0.816
Llama-3.1-8B
16, 18, 20, 22, 24, 26, 28, 30
0.889
0.925
0.907
0.814
Encoder– Embedding
Gemma-2-2B
2, 5, 8, 11, 14, 17, 20, 23
0.916
0.952
0.934
0.868
Gemma-2-9B
5, 10, 15, 20, 25, 30, 35, 40
0.950
0.924
0.937
0.874
Llama-3.1-8B
1, 2, 3, 4, 5, 6, 8, 10
0.801
0.995
0.898
0.796
Appendix
Table 3: LLM-judge agreement with the three-criteria semantic classifier. For each (model, projection) cell we sample 1000 strict-pass and 1000 strict-fail features at every listed layer (top-5 logit-lens tokens shown to the judge, blinded to label). Sensitivity, specificity, agreement, and Cohen’s κ are computed per layer and averaged within the cell. Judge: Llama-3-70B-Instruct.
Model
Layer
nm
Median ∣Δlogit∣
Median KL
Win-rate
semantic
matched
semantic
matched
∣Δ∣
Gemma-2-9B
5
89
1.359
0.0391
0.0359
0.0001
0.92
10
99
0.988
0.0156
0.0156
0.0001
0.96
15
99
1.187
0.0312
0.0310
0.0001
0.92
20
93
0.818
0.0391
0.0230
0.0002
0.97
25
97
1.227
0.0469
0.0518
0.0003
0.97
Appendix
Table 4: Intervention validation for Experiment 1. For each sampled SAE layer we draw 100 flagged features and pair each with a matched-norm baseline: a strict-non-semantic feature whose decoder norm is within ±5% of the flagged feature’s and that fires on the same prompt. We apply do(z=0) to each and read the change in the next-token logit at the flagged feature’s top-1 logit-lens token. KL is over the full vocabulary. nm is the number of pairs for which a matched baseline was available.
Figure 10: Per-layer Spearman rank correlation between per-feature-normalized and unnormalized pre-softmax aggregate scores, across all attention heads in 8 sampled layers of Gemma-2-9B. Boxes show the per-head distribution; lines join the per-layer means. Both query (row-sum) and key (column-sum) aggregates remain above ρˉ≈0.88 at every depth, confirming that the per-feature normalization preserves the rank ordering on which our claims depend.
Figure 11: Per-layer cosine similarity between the encoder column and decoder row of the same SAE feature, for the Gemma Scope SAEs on Gemma-2-9B (left) and the Llama Scope SAEs on Llama-3.1-8B (right). Solid lines are the per-layer mean (red) and median (blue) across features; the shaded band is the inter-quartile range. Gemma sits around 0.8 on average with a dip in early layers; Llama sits substantially lower throughout, making the paired encoder–embedding and decoder–unembedding analyses non-redundant by an even wider margin.
Threshold τQ
0.95
0.90
0.75
0.50
0.25
Gemma-2-2B
39.54%
43.79%
50.94%
59.29%
68.98%
Gemma-2-9B
53.82%
58.49%
65.60%
72.84%
79.73%
Llama-3.1-8B
9.75%
11.71%
15.62%
21.14%
28.83%
Appendix
Table 5: Average percentage of features qualifying as query specialists across all layers for a range of thresholds ( τQ ). The criterion requires the sum of a feature’s top-10 post-softmax attention scores to exceed the threshold. The highlighted column corresponds to our primary choice.
Threshold τK
0.50
0.25
0.10
0.05
0.01
Gemma-2-2B
1.49%
2.90%
8.88%
14.16%
25.82%
Gemma-2-9B
3.16%
6.15%
17.52%
27.31%
43.59%
Llama-3.1-8B
0.42%
0.85%
2.15%
3.57%
8.03%
Appendix
Table 6: Average percentage of features qualifying as key hubs across all layers for a range of thresholds ( τK ). The criterion requires the mean of a feature’s top-10 incoming post-softmax attention scores to exceed the threshold. The highlighted column corresponds to our primary choice.
Model
Layer
nm
Median KL
Win-rate
flagged
matched
flagged > matched
Gemma-2-9B
5
96
0.2456
8.21×10−5
1.00
10
99
0.05559
5.67×10−5
0.98
15
100
0.05594
8.50×10−5
1.00
20
95
0.04033
2.94×10−4
0.99
25
96
0.03372
1.35×10−4
1.00
Appendix
Table 7: Intervention validation for Experiment 2, query specialists. For each sampled SAE layer we draw 100 flagged query specialists and pair each with a matched-norm baseline: a strict-non-specialist feature whose decoder norm is within ±5% of the flagged feature’s and that fires on the same prompt at the same position. We apply do(z=0) to each and read the KL divergence between clean and intervened post-softmax attention rows of the flagged head at the firing position. nm is the number of pairs for which a matched baseline was available.
Model
Layer
nm
Median ∣ drop ∣
Win-rate
flagged
matched
flagged > matched
Gemma-2-9B
5
98
1.705×10−3
1.02×10−4
0.94
10
93
2.504×10−3
6.22×10−5
0.98
15
99
1.058×10−3
4.21×10−5
0.97
20
93
2.026×10−3
2.33×10−4
0.94
25
92
1.570×10−3
1.29×10−4
0.92
Appendix
Table 8: Intervention validation for Experiment 2, key hubs. For each sampled SAE layer we draw 100 flagged key hubs and pair each with a matched-norm baseline. We apply do(z=0) at the hub’s firing position and measure the change in mean incoming attention from later queries ( q>p ) at the flagged head. Magnitude columns report ∣Δ∣ of the drop.
Analysis Type
Model
Layer
Levenshtein Sim. ↑
Cosine Sim. ↑
Top-100 Entropy ↓
Decoder- Unembedding
Gemma-2-2B
16
0.157385
0.292363
0.612010
Gemma-2-9B
27
0.194815
0.332252
0.550316
Llama-3.1-8B
20
0.154048
0.092486
2.911260
Encoder- Embedding
Gemma-2-2B
3
0.149182
0.282485
4.604819
Gemma-2-9B
6
0.153151
0.348118
4.604956
Llama-3.1-8B
3
0.143704
0.028160
4.597589
Appendix
Table 9: Thresholds for semantic feature classification, set at the 50th percentile of scores from a representative layer. The arrows indicate the desired direction for each metric.
Figure 12: Supplementary results for Experiment 1 on Gemma-2-2B and 9B, using encoder–embedding alignment. Semantic features display a U-shaped distribution, with an average joint pass rate of 24.66% and 25.16%, respectively.
Figure 13: Main results for Experiment 2 on Gemma-2-2B: Magnitude of attention participation across layers, with the total height at each layer representing the aggregate pre-softmax score. These scores are calculated as the mean across all keys for each query feature, then summed for each head. The stacking direction of the head values is determined based on whether they are positive or negative. Each color represents a rank of magnitude, rather than a specific head index, across all layers.
Figure 17: Attention weight distributions in Gemma-2-2B layer 12. White diamonds mark means and white lines mark medians.
Figure 18: Attention weight distributions in Gemma-2-9B layer 20. White diamonds mark means and white lines mark medians.
Figure 19: Attention weight distributions in Llama-3.1-8B layer 28. White diamonds mark means and white lines mark medians.
Figure 20: Attention weight distributions in Llama-3.1-8B layer 4. White diamonds mark means and white lines mark medians.
Figure 21: Probability densities of attention weights in Gemma-2-2B layer 12. Dotted lines mark means.
Figure 22: Probability densities of attention weights in Gemma-2-9B layer 20. Dotted lines mark means.
Figure 23: Probability densities of attention weights in Llama-3.1-8B layer 28. Dotted lines mark means.
Figure 24: Probability densities of attention weights in Llama-3.1-8B layer 4. Dotted lines mark means.
Figure 25: Mean attention weights across layers in Gemma-2-2B.
Figure 26: Mean attention weights across layers in Gemma-2-9B.
Figure 27: Mean attention weights across layers in Llama-3.1-8B, decoder–unembedding phase (layers 12–31).
Figure 28: Mean attention weights across layers in Llama-3.1-8B, encoder–embedding phase (layers 0–10).
Figure 29: Median attention weights across layers in Gemma-2-2B.
Figure 30: Median attention weights across layers in Gemma-2-9B.
Figure 31: Median attention weights across layers in Llama-3.1-8B, decoder–unembedding phase (layers 12–31).
Figure 32: Median attention weights across layers in Llama-3.1-8B, encoder–embedding phase (layers 0–10).
Figure 33: Percentage of features exceeding global P75 threshold across layers in Gemma-2-2B.
Figure 34: Percentage of features exceeding global P75 threshold across layers in Gemma-2-9B.
Figure 35: Percentage of features exceeding global P75 threshold across layers in Llama-3.1-8B, decoder–unembedding phase (layers 12–31).
Figure 36: Percentage of features exceeding global P75 threshold across layers in Llama-3.1-8B, encoder–embedding phase (layers 0–10).
Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.
Sparse autoencoder (SAE) features are used to interpret and steer large language models, yet whether a feature's causal role is stable across SAE families remains untested. Single-token features that activate on one vocabulary item provide the diagnostic case where ground truth permits direct comparison. We analyze 3.9M features across six models and three SAE families using zero-ablation at full layer depth. Single-token features cluster 4.7x tighter in decoder space and concentrate in early layers (Layer 0 in GPT2-Small; L0-L4 in Gemma). Ablating them yields Benjamini-Hochberg-significant logit reductions in 178 of 208 full-layer conditions, with depth controlling whether damage cascades downstream or shapes the output directly. Cross-family causal differences exceed within-family scale effects: on the same base model, GemmaScope and BatchTopK features remain causally anchored, while LlamaScope features are locally redundant. The target token's rank recovers to within 2x baseline 96-98% of the time after the same ablation, and a controlled activation-function comparison reverses sign within the same model, leaving training recipe as the residual candidate. Cross-family interpretability claims are therefore sensitive to training methodology, not just activation function or scale.
The wide-scale use of sparse autoencoders (SAEs) as interpretability tools is limited by inconsistent links between SAE features and model behavior. Features with clear activation descriptions may have weak or unexpected causal effects; steering can vary across prompts or oppose the intended direction; and activation-based feature selection can miss features that produce the desired output change. Prior work has studied feature geometry inside the model, where features are computed. We instead study the geometry of changes in model logits caused by feature interventions. We introduce Feature-Effect Geometry Analysis (FEGA), an unsupervised framework that removes the same active SAE feature across contexts and analyzes the resulting cloud of logit changes. Across SAE variants, consistent one-dimensional effects are rare: few features behave like reusable directions. To interpret this variation, we distinguish value-like features, tied to static information such as factual attributes, from pointer-like features, associated with context-dependent operations. Value-like features more often exhibit structured, low-dimensional effects, although these effects typically span several directions. Pointer-like features, by contrast, predominantly exhibit diffuse effects. Our results show that a feature can be interpretable and causally relevant without providing a stable direction for steering.
Phu Gia Hoang, Anwoy Chatterjee, Tanmoy Chakraborty +2
UKP Lab, Technical University of Darmstadt · Department of Electrical Engineering Indian Institute of Technology Delhi, India
Sparse Autoencoders (SAEs) are widely used for mechanistic interpretability in large language models, yet their standard formulation assigns each latent feature a single decoder direction, implicitly assuming features to be one-dimensional. We show that this assumption can induce feature splitting when model features are multi-dimensional. Geometrically, accurately reconstructing a feature under a fixed co-activation budget can require exponentially many decoder directions as its dimension grows. From an optimization perspective, adding decoder directions can strictly lower the ℓ1-regularized population risk, even when a basis can already reconstruct the feature exactly. Motivated by these results, we introduce Subspace-Aware Sparse Autoencoders (SASA), which replace single-vector decoders with learned subspaces, enforce sparsity through Top-s group gating, and adapt group rank with nuclear-norm regularization. Under an idealized single-feature model, we prove that the SASA objective is minimized by a single group whose reconstruction map recovers the feature subspace. We also derive polynomial sample complexity bounds for subspace estimation within a group and a lower bound on the samples needed to observe every atom in a fixed fragmented dictionary. Experiments on GPT-2 and Mistral-7B show reduced absorption compared with SAEs. Matched-budget GPT-2 comparisons show lower absorption for concept families and more accurate prediction of their members using a small number of latent coordinates, at a reconstruction cost. The code is available at https://github.com/arshandalili/sasa.