Organizations: VRAIN – Valencian Research Institute for Artificial Intelligence, Universitat Politècnica de València, Valencia, Spain · Department of Economics and Social Sciences, Universitat Politècnica de València, Valencia, Spain · Department of Business Organisation, Universitat Politècnica de València, Valencia, Spain
Weight pruning is a standard technique for compressing large language models, yet its effect on learned internal representations remains poorly understood. We present the first systematic study of how unstructured pruning reshapes the feature geometry of language models, using Sparse Autoencoders (SAEs) as interpretability probes. Across three model families (Gemma 3 1B, Gemma 2 2B, Llama 3.2 1B), two pruning methods (magnitude and Wanda), and six sparsity levels (0--60%), we investigate five research questions spanning seed stability, feature survival, SAE transferability, feature fragility, and causal relevance. Our most striking finding is that rare SAE features--those with low firing rates--survive pruning far better than frequent ones, with within-condition Spearman correlations of rho = -1.0 in 11 of 17 experimental conditions. This counter-intuitive result suggests that pruning acts as implicit feature selection, preferentially destroying high-frequency generic features while preserving specialized rare ones. We further show that Wanda pruning preserves feature structure up to 3.7x better than magnitude pruning, that pre-trained SAEs remain viable on Wanda-pruned models up to 50% sparsity, and that geometric feature survival does not predict causal importance--a dissociation with implications for interpretability under compression.
Figures & tables
Model
Method
0%
20%
30%
40%
50%
60%
Gemma 3 1B
Magnitude
64.2
93.8
333
4,472
> 10 6
> 10 6
Wanda
—
286
822
9,315
34,020
110,751
Gemma 2 2B
Magnitude
410
—
2,359
—
303,290
—
Wanda
—
—
489
—
904
—
Llama 3.2 1B
Magnitude
17.7
—
—
—
2,583
—
Wanda
—
—
20.9
—
53.1
—
Table 1: WikiText-2 perplexity across models, pruning methods, and sparsity levels. Lower is better. Gemma 2 dense perplexity is anomalously high (likely due to context length mismatch); relative differences remain informative. Catastrophic values ( >105 ) are abbreviated.
Figure 1: Seed stability (MNN rate at τ=0.7 ) as a function of sparsity. Error bars show standard deviation across seed pairs. While absolute MNN rates are low (2–4%), the pattern of degradation with sparsity is consistent.
Figure 2: Dense → pruned feature survival (MNN at τ=0.7 ). Wanda consistently preserves more features than magnitude pruning across all models and sparsity levels.
Model
Method
Sparsity
FVU
L0
Gemma 3 1B
Dense
0%
0.011
11.4
Magnitude
20%
0.015
10.2
Magnitude
30%
0.024
10.0
Magnitude
50%
0.077
4.5
Wanda
30%
0.016
10.4
Wanda
50%
0.024
11.0
Table 2: Transferability: FVU of official (dense-trained) SAEs applied to pruned model activations. Lower FVU indicates better reconstruction. Wanda-pruned activations remain well-reconstructable up to 50% sparsity.
Figure 3: Transferability of pre-trained (official) SAEs to pruned model activations. FVU is plotted against sparsity for Gemma 3 1B and Gemma 2 2B, separated by pruning method. Wanda-pruned activations remain well within the reconstructable range.
Figure 4: Feature survival rate by firing rate quintile ( τ=0.7 ). Across all models and pruning methods, rare features (Q1) survive substantially better than frequent ones (Q5). Lines represent different pruning method–sparsity combinations.
Model
Condition
Q1 (rare)
Q5 (freq)
Ratio
Gemma 3 1B
Mag. s=0.3
0.760
0.146
5.2 ×
Wanda s=0.3
0.852
0.351
2.4 ×
Wanda s=0.5
0.652
0.006
109 ×
Mag. s=0.5
0.225
0.001
225 ×
Gemma 2 2B
Wanda s=0.3
0.867
0.550
1.6 ×
Mag. s=0.3
0.265
0.130
2.0 ×
Table 3: Feature survival rate by firing rate quintile ( τ=0.7 ). In all conditions, rare features (Q1) survive at substantially higher rates than frequent features (Q5). The ratio column shows the Q1/Q5 survival advantage.
Figure 5: Feature fragility by firing rate, separated by pruning method. Blue: magnitude pruning; orange: Wanda. The monotonic decline from Q1 (rare) to Q5 (frequent) is robust across all models, methods, and sparsity levels.
Figure 6: Mean KL divergence upon single-feature ablation for robust versus fragile features. Across all models and conditions, robust and fragile features show nearly identical causal effects, indicating a dissociation between geometric survival and causal importance.
Model
Condition
Robust KL
Fragile KL
Δ
Gemma 3 1B
Dense
0.695
0.696
0.1%
Pruned (Wanda 30%)
0.268
0.266
0.6%
Gemma 2 2B
Dense
0.907
0.896
1.2%
Pruned (Wanda 30%)
0.684
0.686
0.2%
Llama 3.2 1B
Dense
0.497
0.498
0.1%
Pruned (Wanda 30%)
0.565
0.566
0.2%
Table 4: Mean KL divergence upon single-feature ablation. Robust and fragile features show nearly identical causal effects across all models and conditions. Δ is the absolute percent difference.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Tier
Model
Layer
Purpose
Runs
Seeds
Steps
A1
Gemma 3 1B
L12
Primary grid
11
3–5
20,000
A3
Gemma 3 1B
L6, L18
Layer depth
4
3
12,000
B1
Gemma 2 2B
L12
Anchor model
5
3–5
20,000
C1
Llama 3.2 1B
L8
Cross-family
4
3–5
12,000
Appendix
Table A.1 : Summary of experimental runs by tier. Seeds column shows number of SAE random seeds per run.
Model
Method
Sparsity
FVU
Alive
n
Gemma 3 1B
Magnitude
0.0
0.367
9,216
5
Magnitude
0.2
0.355
9,216
3
Magnitude
0.3
0.335
9,216
3
Magnitude
0.4
0.293
9,216
3
Magnitude
0.5
0.285
9,216
3
Magnitude
0.6
0.351
9,216
3
Appendix
Table B.1 : SAE reconstruction quality (FVU) averaged over seeds. The large majority of SAEs achieved near-complete feature utilization with our training procedure.
Sparse autoencoders (SAEs) are widely used to interpret the internal representations of large language models (LLMs), yet their reliability under post-hoc model compression remains poorly understood. We present a systematic study of how pruning affects SAE behavior and theoretically show that, for a fixed SAE, its impact is governed by perturbation energy, a covariance-weighted norm. This perspective exposes a key limitation of magnitude pruning: by ignoring activation geometry, it distorts the learned representation space and degrades SAE functionality. Activation-aware methods such as Wanda and SparseGPT, in contrast, implicitly control perturbation energy and are therefore substantially more robust at preserving SAE behavior. We further reveal a consistent structural vulnerability across all pruning methods: middle layers are significantly more sensitive to pruning than early or late layers. Guided by this insight, we propose a layer-wise sparsity allocation strategy, achieving lower perplexity under the same average pruning sparsity. Experiments across four model architectures validate our theoretical findings. Code is publicly available at https://github.com/osu-srml/sae-robustness-under-pruning/tree/main.
Suchit Gupte, Xueru Zhang, Mohammad Mahdi Khalili
Department of Computer Science The Ohio State University Columbus, OH 43210, USA
Sparse Autoencoders (SAEs) extract features from LLM internal activations, meant to correspond to interpretable concepts. A core SAE training hyperparameter is L0: how many SAE features should fire per token on average. Existing work compares SAE algorithms using sparsity-reconstruction tradeoff plots, implying L0 is a free parameter with no inherently correct value aside from its effect on reconstruction. In this work we study the effect of L0 on SAEs, and show that if L0 is not set correctly, the SAE fails to disentangle the underlying features of the LLM. If L0 is too low, the SAE will mix correlated features to improve reconstruction. If L0 is too high, the SAE finds degenerate solutions that also mix features. Further, we present a proxy metric that can help guide the search for the correct L0 for an SAE on a given training distribution. We show that our method finds the correct L0 in toy models and coincides with peak sparse probing performance in LLM SAEs. We find that most commonly used SAEs have an L0 that is too low. Our work shows that practitioners must set L0 correctly to train SAEs with monosemantic features.
Sparse autoencoder (SAE) features are used to interpret and steer large language models, yet whether a feature's causal role is stable across SAE families remains untested. Single-token features that activate on one vocabulary item provide the diagnostic case where ground truth permits direct comparison. We analyze 3.9M features across six models and three SAE families using zero-ablation at full layer depth. Single-token features cluster 4.7x tighter in decoder space and concentrate in early layers (Layer 0 in GPT2-Small; L0-L4 in Gemma). Ablating them yields Benjamini-Hochberg-significant logit reductions in 178 of 208 full-layer conditions, with depth controlling whether damage cascades downstream or shapes the output directly. Cross-family causal differences exceed within-family scale effects: on the same base model, GemmaScope and BatchTopK features remain causally anchored, while LlamaScope features are locally redundant. The target token's rank recovers to within 2x baseline 96-98% of the time after the same ablation, and a controlled activation-function comparison reverses sign within the same model, leaving training recipe as the residual candidate. Cross-family interpretability claims are therefore sensitive to training methodology, not just activation function or scale.