Sparse autoencoders (SAEs) have emerged as a powerful technique for decomposing language model representations into interpretable features. Current interpretation pipelines infer feature semantics from activation patterns, implicitly assuming every feature has a semantic explanation while overlooking the computational roles features inherit from the SAE training objective. We introduce a weight-based interpretation framework that requires no activation data and grounds claims causally through targeted feature-ablation interventions. Across Gemma-2 and Llama-3.1, three depth-dependent signatures show that SAE features inherit the base model's geometry: semantic features form a U-shape under tied embeddings but bifurcate when untied, attention participation peaks mid-layer, and the two roles couple oppositely for input- vs. output-oriented features. This weight-based view supplies the causal half missing from activation-based interpretability.
Figures & tables
Figure 1: Two views of SAE features. ( 1(a) ) An activation-based explanation read off the feature’s top-activating contexts: tokens where the feature fires are highlighted and the highest activations are boxed, and GPT-4o-mini summarizes the co-occurring text. The summary does not specify what changes in the model’s computation when the feature is suppressed. ( 1(b) ) A uniform sample of features that pass all three weight-based metrics in Section 4 on Gemma-2-9B at layer 27, shown with their top-10 predicted tokens. “L” is Levenshtein similarity, “C” is cosine similarity, and “E” is top-100 entropy; boxed scores correspond to the range of displayed tokens.
Figure 2: Joint and per-metric pass rates for the decoder–unembedding classifier on Gemma-2-2B and 9B. Both sizes show a U-shaped distribution; ablating any single metric uniformly increases the pass rate, confirming the three are complementary.
Projection
Model
Calibration layer
ρmin , p∈[25,75]
ρmin , p∈[40,60]
Decoder–unembedding
Gemma-2-2B
16
0.971
0.992
Gemma-2-9B
27
0.953
0.993
Llama-3.1-8B
20
0.971
0.997
Encoder–embedding
Gemma-2-2B
3
0.720
0.878
Gemma-2-9B
6
0.905
0.978
Llama-3.1-8B
3
0.785
0.964
Table 1: Stability of the per-layer joint pass-rate profiles. Each ρmin is the minimum Spearman correlation with the same cell’s p=50 profile over the tested percentiles in the specified band. Calibration layers remain fixed throughout the sweep.
Figure 4: Main results for Experiment 2 on Gemma-2-9B and Llama-3.1-8B: Magnitude of attention participation across layers, with the total height at each layer representing the aggregate pre-softmax score. These scores are calculated as the mean across all keys for each query feature, then summed for each head. The stacking direction of the head values is determined based on whether they are positive or negative. Each color represents a rank of magnitude, rather than a specific head index, across all layers.
Figure 5: Per-layer query-side attention-weight densities for semantic vs. non-semantic features. Left: Gemma-2-9B layer 20, where the semantic distribution sits below the non-semantic one. Right: Llama-3.1-8B layer 4, in the input-semantic phase, where the relation inverts and semantic features carry the higher attention weights.
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Spearman rank correlation ( ρ ) matrix for 23 candidate semantic metrics, calculated on all features from the Gemma-2-9B layer 27 SAE.
Figure 7: Pairwise scatter samples illustrating the relationships between the three selected semantic metrics for the Gemma-2-9B layer 27 SAE. The plots demonstrate that while the metrics are not entirely independent, their correlations are moderate. This supports their selection as a complementary set, where each metric provides a distinct perspective for evaluating the semantic coherence of a feature.
Figure 8: Distributions of the three primary semantic metrics—Levenshtein similarity (higher is better), cosine similarity (higher is better), and top-100 entropy (lower is better)—for all features in the Gemma-2-9B layer 27 SAE. The distributions are continuous and generally unimodal, lacking a distinct bimodal structure or natural cutoff point. This characteristic motivates our use of a percentile-based thresholding approach in Experiment 1, as it provides a consistent relative standard for what constitutes a “semantic” feature, rather than relying on an arbitrary absolute value. The distribution plot applies normalization (RMSNorm) because, without it, the entropy is highly concentrated near the maximum value with extreme outliers, making visualization difficult.
Decoder–unembedding
Encoder–embedding
p
Gemma-2-2B
Gemma-2-9B
Llama-3.1-8B
Gemma-2-2B
Gemma-2-9B
Llama-3.1-8B
25
50.4
50.5
33.4
48.8
44.9
12.5
30
43.5
43.8
30.8
42.1
39.0
11.5
35
37.4
38.2
28.3
36.8
34.5
10.7
40
32.3
33.7
25.8
32.2
30.8
10.1
45
28.0
29.4
23.4
28.3
27.8
9.5
Appendix
Table 2: Threshold sensitivity of the three-criteria classifier. Entries are mean joint pass rates (%) across all layers. The joint strictness percentile p raises both similarity cutoffs and lowers the entropy cutoff; the bold row reproduces the paper’s calibration.
Projection
Model
Layers
Sensitivity
Specificity
Agreement
Cohen’s κ
Decoder– Unembedding
Gemma-2-2B
2, 5, 8, 11, 14, 17, 20, 23
0.916
0.908
0.912
0.824
Gemma-2-9B
5, 10, 15, 20, 25, 30, 35, 40
0.954
0.862
0.908
0.816
Llama-3.1-8B
16, 18, 20, 22, 24, 26, 28, 30
0.889
0.925
0.907
0.814
Encoder– Embedding
Gemma-2-2B
2, 5, 8, 11, 14, 17, 20, 23
0.916
0.952
0.934
0.868
Gemma-2-9B
5, 10, 15, 20, 25, 30, 35, 40
0.950
0.924
0.937
0.874
Llama-3.1-8B
1, 2, 3, 4, 5, 6, 8, 10
0.801
0.995
0.898
0.796
Appendix
Table 3: LLM-judge agreement with the three-criteria semantic classifier. For each (model, projection) cell we sample 1000 strict-pass and 1000 strict-fail features at every listed layer (top-5 logit-lens tokens shown to the judge, blinded to label). Sensitivity, specificity, agreement, and Cohen’s κ are computed per layer and averaged within the cell. Judge: Llama-3-70B-Instruct.
Model
Layer
nm
Median ∣Δlogit∣
Median KL
Win-rate
semantic
matched
semantic
matched
∣Δ∣
Gemma-2-9B
5
89
1.359
0.0391
0.0359
0.0001
0.92
10
99
0.988
0.0156
0.0156
0.0001
0.96
15
99
1.187
0.0312
0.0310
0.0001
0.92
20
93
0.818
0.0391
0.0230
0.0002
0.97
25
97
1.227
0.0469
0.0518
0.0003
0.97
Appendix
Table 4: Intervention validation for Experiment 1. For each sampled SAE layer we draw 100 flagged features and pair each with a matched-norm baseline: a strict-non-semantic feature whose decoder norm is within ±5% of the flagged feature’s and that fires on the same prompt. We apply do(z=0) to each and read the change in the next-token logit at the flagged feature’s top-1 logit-lens token. KL is over the full vocabulary. nm is the number of pairs for which a matched baseline was available.
Figure 10: Per-layer Spearman rank correlation between per-feature-normalized and unnormalized pre-softmax aggregate scores, across all attention heads in 8 sampled layers of Gemma-2-9B. Boxes show the per-head distribution; lines join the per-layer means. Both query (row-sum) and key (column-sum) aggregates remain above ρˉ≈0.88 at every depth, confirming that the per-feature normalization preserves the rank ordering on which our claims depend.
Figure 11: Per-layer cosine similarity between the encoder column and decoder row of the same SAE feature, for the Gemma Scope SAEs on Gemma-2-9B (left) and the Llama Scope SAEs on Llama-3.1-8B (right). Solid lines are the per-layer mean (red) and median (blue) across features; the shaded band is the inter-quartile range. Gemma sits around 0.8 on average with a dip in early layers; Llama sits substantially lower throughout, making the paired encoder–embedding and decoder–unembedding analyses non-redundant by an even wider margin.
Threshold τQ
0.95
0.90
0.75
0.50
0.25
Gemma-2-2B
39.54%
43.79%
50.94%
59.29%
68.98%
Gemma-2-9B
53.82%
58.49%
65.60%
72.84%
79.73%
Llama-3.1-8B
9.75%
11.71%
15.62%
21.14%
28.83%
Appendix
Table 5: Average percentage of features qualifying as query specialists across all layers for a range of thresholds ( τQ ). The criterion requires the sum of a feature’s top-10 post-softmax attention scores to exceed the threshold. The highlighted column corresponds to our primary choice.
Threshold τK
0.50
0.25
0.10
0.05
0.01
Gemma-2-2B
1.49%
2.90%
8.88%
14.16%
25.82%
Gemma-2-9B
3.16%
6.15%
17.52%
27.31%
43.59%
Llama-3.1-8B
0.42%
0.85%
2.15%
3.57%
8.03%
Appendix
Table 6: Average percentage of features qualifying as key hubs across all layers for a range of thresholds ( τK ). The criterion requires the mean of a feature’s top-10 incoming post-softmax attention scores to exceed the threshold. The highlighted column corresponds to our primary choice.
Model
Layer
nm
Median KL
Win-rate
flagged
matched
flagged > matched
Gemma-2-9B
5
96
0.2456
8.21×10−5
1.00
10
99
0.05559
5.67×10−5
0.98
15
100
0.05594
8.50×10−5
1.00
20
95
0.04033
2.94×10−4
0.99
25
96
0.03372
1.35×10−4
1.00
Appendix
Table 7: Intervention validation for Experiment 2, query specialists. For each sampled SAE layer we draw 100 flagged query specialists and pair each with a matched-norm baseline: a strict-non-specialist feature whose decoder norm is within ±5% of the flagged feature’s and that fires on the same prompt at the same position. We apply do(z=0) to each and read the KL divergence between clean and intervened post-softmax attention rows of the flagged head at the firing position. nm is the number of pairs for which a matched baseline was available.
Model
Layer
nm
Median ∣ drop ∣
Win-rate
flagged
matched
flagged > matched
Gemma-2-9B
5
98
1.705×10−3
1.02×10−4
0.94
10
93
2.504×10−3
6.22×10−5
0.98
15
99
1.058×10−3
4.21×10−5
0.97
20
93
2.026×10−3
2.33×10−4
0.94
25
92
1.570×10−3
1.29×10−4
0.92
Appendix
Table 8: Intervention validation for Experiment 2, key hubs. For each sampled SAE layer we draw 100 flagged key hubs and pair each with a matched-norm baseline. We apply do(z=0) at the hub’s firing position and measure the change in mean incoming attention from later queries ( q>p ) at the flagged head. Magnitude columns report ∣Δ∣ of the drop.
Analysis Type
Model
Layer
Levenshtein Sim. ↑
Cosine Sim. ↑
Top-100 Entropy ↓
Decoder- Unembedding
Gemma-2-2B
16
0.157385
0.292363
0.612010
Gemma-2-9B
27
0.194815
0.332252
0.550316
Llama-3.1-8B
20
0.154048
0.092486
2.911260
Encoder- Embedding
Gemma-2-2B
3
0.149182
0.282485
4.604819
Gemma-2-9B
6
0.153151
0.348118
4.604956
Llama-3.1-8B
3
0.143704
0.028160
4.597589
Appendix
Table 9: Thresholds for semantic feature classification, set at the 50th percentile of scores from a representative layer. The arrows indicate the desired direction for each metric.
Figure 12: Supplementary results for Experiment 1 on Gemma-2-2B and 9B, using encoder–embedding alignment. Semantic features display a U-shaped distribution, with an average joint pass rate of 24.66% and 25.16%, respectively.
Figure 13: Main results for Experiment 2 on Gemma-2-2B: Magnitude of attention participation across layers, with the total height at each layer representing the aggregate pre-softmax score. These scores are calculated as the mean across all keys for each query feature, then summed for each head. The stacking direction of the head values is determined based on whether they are positive or negative. Each color represents a rank of magnitude, rather than a specific head index, across all layers.
Figure 17: Attention weight distributions in Gemma-2-2B layer 12. White diamonds mark means and white lines mark medians.
Figure 18: Attention weight distributions in Gemma-2-9B layer 20. White diamonds mark means and white lines mark medians.
Figure 19: Attention weight distributions in Llama-3.1-8B layer 28. White diamonds mark means and white lines mark medians.
Figure 20: Attention weight distributions in Llama-3.1-8B layer 4. White diamonds mark means and white lines mark medians.
Figure 21: Probability densities of attention weights in Gemma-2-2B layer 12. Dotted lines mark means.
Figure 22: Probability densities of attention weights in Gemma-2-9B layer 20. Dotted lines mark means.
Figure 23: Probability densities of attention weights in Llama-3.1-8B layer 28. Dotted lines mark means.
Figure 24: Probability densities of attention weights in Llama-3.1-8B layer 4. Dotted lines mark means.
Figure 25: Mean attention weights across layers in Gemma-2-2B.
Figure 26: Mean attention weights across layers in Gemma-2-9B.
Figure 27: Mean attention weights across layers in Llama-3.1-8B, decoder–unembedding phase (layers 12–31).
Figure 28: Mean attention weights across layers in Llama-3.1-8B, encoder–embedding phase (layers 0–10).
Figure 29: Median attention weights across layers in Gemma-2-2B.
Figure 30: Median attention weights across layers in Gemma-2-9B.
Figure 31: Median attention weights across layers in Llama-3.1-8B, decoder–unembedding phase (layers 12–31).
Figure 32: Median attention weights across layers in Llama-3.1-8B, encoder–embedding phase (layers 0–10).
Figure 33: Percentage of features exceeding global P75 threshold across layers in Gemma-2-2B.
Figure 34: Percentage of features exceeding global P75 threshold across layers in Gemma-2-9B.
Figure 35: Percentage of features exceeding global P75 threshold across layers in Llama-3.1-8B, decoder–unembedding phase (layers 12–31).
Figure 36: Percentage of features exceeding global P75 threshold across layers in Llama-3.1-8B, encoder–embedding phase (layers 0–10).
Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.