Sparse autoencoders decompose language model representations into interpretable features, but feature interpretation in this literature remains predominantly correlational. Attribution methods offer more direct estimates of feature influence, yet their faithfulness to actual causal effects has not been systematically evaluated in the SAE setting. We apply SHAP-based attribution to SAE latent features and evaluate it against direct causal intervention, using sentiment classification on SST-2 with GPT-2 Small and a 32,768-feature layer-7 SAE as a testbed. We compare SHAP to Integrated Gradients, Gradient×Input, and probe-weight baselines across linear and MLP probe arms, scoring each against feature ablation and activation steering measured through the language model's own logit difference. Most faithfulness associations we measure do not survive multiple-comparison correction, and attribution methods that agree strongly with one another (ρ=0.42--0.80) agree poorly with measured causal effects, indicating that inter-method agreement is not evidence of causal faithfulness. Two effects hold under control. First, local per-example attribution retains a directional advantage over global averaged attribution, while its apparent importance advantage is explained by feature activity rather than superior causal ranking. Second, under known-ground-truth minimal pairs, attribution recovers inserted sentiment signal well above chance (AUROC 0.792) but fails to localize deletions (AUROC 0.520), suggesting that these methods detect the presence of a feature's influence more reliably than its absence. We argue that importance faithfulness and directional faithfulness are separable properties requiring separate evaluation, and that attribution agreement should not substitute for causal validation.
Figures & tables
Arm
Input dim
Val. Accuracy
Val. AUROC
Linear (logistic)
32,768
0.761
0.84
MLP (hidden=32)
116 (filtered)
0.735
0.808
Table 1: Probe architectures and performance.
Figure 1: MLP-arm attribution agreement (Spearman ρ on ∣score∣ , n=116 filtered features). IG, GA, and SHAP form a correlated cluster ( ρ=0.42 – 0.80 ); the probe weight is essentially uncorrelated with all three ( ρ≤0.04 ).
Arm
Method
ρimp ( q )
ρdir ( q )
Linear
SHAP/IG/GA
+0.256 ( q =0.0111)*
−0.054 ( q =0.563)
Linear
Probe
+0.096 ( q =0.487)
+0.410 ( q <0.0001)*
MLP
SHAP
+0.279 ( q =0.0088)*
−0.157 ( q =0.162)
MLP
Probe saliency
−0.011 ( q =0.911)
n/a
MLP
IG
+0.278 ( q =0.0088)*
−0.046 ( q =0.862)
MLP
GA
+0.197 ( q =0.0797)
−0.031 ( q =0.862)
Table 2: Powered global faithfulness replication ( k=116 , steering, all filtered features, BH-corrected within arm).
Figure 2: Local vs. global attribution, per-example faithfulness ( n=100 sentences, ablation). Bars are mean Spearman ρ across examples ± 95% CI; asterisks mark the paired Wilcoxon signed-rank test on the per-example gap (*** p<10−4 , ** p<10−2 , * p<0.05 , n.s. otherwise). Block A = all top-20-local ∪ top-20-global candidates; Block B = active-only control.
Arm
Block : kind
Local
Global
Gap
Wilcoxon p
Win frac.
Linear
A : importance
+0.414
−0.269
+0.682
3.9×10−18
1.000
Linear
A : directional
+0.135
−0.158
+0.294
4.1×10−15
0.860
Linear
B : importance
+0.202
+0.285
−0.083
6.6×10−2
0.420
Linear
B : directional
+0.144
−0.311
+0.455
9.1×10−16
0.880
MLP
A : importance
+0.345
−0.223
+0.568
3.9×10−18
1.000
MLP
A : directional
+0.102
−0.239
+0.341
1.2×10−15
0.880
Table 3: Local vs. global attribution, per-example faithfulness ( n=100 sentences, ablation): numeric detail. Mean ρ across examples with a paired Wilcoxon signed-rank test on the per-example gap.
Edit
Sign acc.
Method
AUROC
ρ (causal recovery)
remove
0.770
local SHAP
0.520
0.254
global SHAP/IG/GA
0.509
0.163
Probe weights
0.503
−0.099
insert
0.700
local SHAP
0.792
0.536
global SHAP/IG/GA
0.792
0.458
Probe weights
0.610
0.167
Table 4: Minimal-pair ground-truth validation, n=100 pairs, random distractors.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Split
n
Used for
Probe train
47,144
Fitting probes; SHAP/IG/GA background and baseline; steering α calibration
Probe val
10,102
Probe hyperparameters; the activation-frequency feature mask; reported probe accuracy
Held-out (“shap”)
10,103
All attribution (SHAP/IG/GA) and all causal evaluation (ablation/steering)
Appendix
Table 5: Data split sizes and purposes.
Arm
Explainer
Target function
Gradient used by IG/GA
Linear (logistic)
shap.LinearExplainer (exact)
Log-odds margin, w⋅x+b
coef_ (constant)
MLP
shap.DeepExplainer
Pre-sigmoid logit
Input-dependent, through ReLU
Appendix
Table 6: Target function attributed by each arm.
α
mean ∣ΔP∣
mean signed ΔP
saturation frac.
0.3316
0.0430
−0.0426
0.085
0.6632
0.0866
−0.0859
0.090
1.3264
0.1705
−0.1690
0.120
2.6528
0.2983
−0.2957
0.263
Appendix
Table 7: Raw α -calibration pilot sweep (probe-readout, 3 pilot features, 200 train examples).
Table 9: SHAP faithfulness across candidate-selection strategies (steering), k=20 except all ( k=116 ). top biases SHAP’s ρ downward relative to other methods (Appendix C.4 ); top-pos / top-neg test the sign-inversion hypothesis without mixed-sign cancellation.
Arm
Method
ρimp (scaled / constant)
ρdir (scaled / constant)
Linear
SHAP/IG/GA
−0.083 / −0.114
+0.119 / +0.202
Linear
Probe
−0.186 / −0.081
−0.011 / −0.089
MLP
SHAP
−0.298 / −0.377
+0.003 / +0.081
MLP
Probe saliency
−0.146 / +0.003
n/a
Appendix
Table 10: Alpha-mode sensitivity: scaled vs. constant per-feature α ( steering , random , k=20 ).
Arm
Test
Mean ρ
Std
95% CI
Wilcoxon p
Ref p (seed 0)
Linear
SHAP/IG/GA : imp
+0.264
0.184
[0.149, 0.382]
0.0039
1.000
Linear
SHAP/IG/GA : dir
−0.022
0.227
[ − 0.155, 0.125]
0.770
0.700
Linear
Probe : imp
+0.140
0.197
[0.021, 0.262]
0.0488
0.500
Linear
Probe : dir
+0.320
0.188
[0.203, 0.433]
0.0039
1.000
MLP
SHAP : imp
+0.189
0.254
[0.024, 0.342]
0.0625
0.500
MLP
SHAP : dir
−0.064
0.140
[ − 0.154, 0.020]
0.322
1.000
Appendix
Table 11: Multi-seed faithfulness sweep (10 seeds, k=20 , scaled α ). “Ref p ” is the fraction of seeds with ∣ρ∣ at least the published seed-0 draw’s.
Arm
Mode
Method
ρimp ( p )
ρdir ( p )
Linear
Steering
SHAP/IG/GA
−0.083 (0.729)
+0.119 (0.618)
Linear
Steering
Probe
−0.186 (0.431)
−0.011 (0.965)
Linear
Ablation
SHAP/IG/GA
−0.045 (0.850)
−0.164 (0.490)
Linear
Ablation
Probe
−0.183 (0.439)
+0.448 (0.048)
MLP
Steering
SHAP
−0.298 (0.202)
+0.003 (0.990)
MLP
Steering
Probe saliency
−0.146 (0.540)
n/a
Appendix
Table 12: Underpowered causal faithfulness ( k=20 , steering, random selection, neval=1000 ). For the linear arm, IG and GA are analytically identical to SHAP and omitted as separate rows.
Figure 3: Full group-steering effect-vs-cost grid ( k×α ), one panel per method per arm. This experiment’s baseline is a clean forward pass (no SAE encode/decode), unlike the per-feature steering/ablation experiments’ encode → decode-reconstruction baseline (Appendix E ); we compare curve shapes across arms, never magnitudes.
Quantity
Value
Mean active filtered features / example (of 116)
23.06
Mean pairwise Jaccard, top-20 ∣Φ∣ features
0.211
Mean ∣r(activation,has-lexicon-word)∣ , top-20
0.063
Lexicon bag-of-words accuracy / AUROC
0.629 / 0.676
SAE probe accuracy / AUROC (held-out)
0.761 / 0.838
Accuracy gap (SAE probe − lexicon)
+0.132
Appendix
Table 13: Lexical simplicity control (full held-out split, n=10,103 ).
Arm
Method
ρimp ( q )
ρdir ( q )
Linear
SHAP/IG/GA
+0.241 ( q =0.019)*
+0.034 ( q =0.719)
Linear
Probe
+0.134 ( q =0.241)
+0.356 ( q =0.0007)*
MLP
SHAP
+0.210 ( q =0.126)
−0.005 ( q =0.958)
MLP
Probe saliency
+0.018 ( q =0.958)
n/a
MLP
IG
+0.152 ( q =0.242)
+0.019 ( q =0.958)
MLP
GA
+0.195 ( q =0.126)
+0.030 ( q =0.958)
Appendix
Table 14: Powered global faithfulness replication ( k=116 , steering, all filtered features, BH-corrected within arm).
Sparse autoencoder (SAE) features are used to interpret and steer large language models, yet whether a feature's causal role is stable across SAE families remains untested. Single-token features that activate on one vocabulary item provide the diagnostic case where ground truth permits direct comparison. We analyze 3.9M features across six models and three SAE families using zero-ablation at full layer depth. Single-token features cluster 4.7x tighter in decoder space and concentrate in early layers (Layer 0 in GPT2-Small; L0-L4 in Gemma). Ablating them yields Benjamini-Hochberg-significant logit reductions in 178 of 208 full-layer conditions, with depth controlling whether damage cascades downstream or shapes the output directly. Cross-family causal differences exceed within-family scale effects: on the same base model, GemmaScope and BatchTopK features remain causally anchored, while LlamaScope features are locally redundant. The target token's rank recovers to within 2x baseline 96-98% of the time after the same ablation, and a controlled activation-function comparison reverses sign within the same model, leaving training recipe as the residual candidate. Cross-family interpretability claims are therefore sensitive to training methodology, not just activation function or scale.
Sparse autoencoder (SAE) features are typically discovered from how their activations co-vary with behaviour, yet they are often used through interventions such as steering or ablation. We take intervention attribution, the first-order effect of a specified intervention on the model output, as our estimand. For a broad class of response functions, we show that observational feature-discovery evidence generally does not identify this attribution, and derive an exact gradient-based decomposition of the gap between linear-probe predictions and intervention attribution. Across four model-SAE pairs, median probe-to-attribution magnitude ratios range from tens to hundreds, while attributions closely track small-step steering. For single-latent ablations with nonzero effects, gradient predictions match the realized sign in 91-97% of cases, compared with 17-35% for probe-implied changes. Attribution-selected sets also produce larger absolute mean ablation effects on held-out prompts than sets selected by correlation, rank tests or probe coefficients. Together, these results establish intervention attribution as a principled, gradient-estimable target for selecting SAE features for intervention.
Yifeng Guo, Yanran Li, Xiang Chen
St. Jude Children’s Research Hospital · Columbia University
The wide-scale use of sparse autoencoders (SAEs) as interpretability tools is limited by inconsistent links between SAE features and model behavior. Features with clear activation descriptions may have weak or unexpected causal effects; steering can vary across prompts or oppose the intended direction; and activation-based feature selection can miss features that produce the desired output change. Prior work has studied feature geometry inside the model, where features are computed. We instead study the geometry of changes in model logits caused by feature interventions. We introduce Feature-Effect Geometry Analysis (FEGA), an unsupervised framework that removes the same active SAE feature across contexts and analyzes the resulting cloud of logit changes. Across SAE variants, consistent one-dimensional effects are rare: few features behave like reusable directions. To interpret this variation, we distinguish value-like features, tied to static information such as factual attributes, from pointer-like features, associated with context-dependent operations. Value-like features more often exhibit structured, low-dimensional effects, although these effects typically span several directions. Pointer-like features, by contrast, predominantly exhibit diffuse effects. Our results show that a feature can be interpretable and causally relevant without providing a stable direction for steering.
Phu Gia Hoang, Anwoy Chatterjee, Tanmoy Chakraborty +2
UKP Lab, Technical University of Darmstadt · Department of Electrical Engineering Indian Institute of Technology Delhi, India