What Do Sparse Autoencoder Feature Discovery Scores Identify? From Activations to Interventions

Yifeng Guo; Yanran Li; Xiang Chen

cs.LGOct 11, 2026

What Do Sparse Autoencoder Feature Discovery Scores Identify? From Activations to Interventions

Authors: Yifeng Guo, Yanran Li, Xiang Chen

Organizations: St. Jude Children’s Research Hospital · Columbia University

Abstract

Sparse autoencoder (SAE) features are typically discovered from how their activations co-vary with behaviour, yet they are often used through interventions such as steering or ablation. We take intervention attribution, the first-order effect of a specified intervention on the model output, as our estimand. For a broad class of response functions, we show that observational feature-discovery evidence generally does not identify this attribution, and derive an exact gradient-based decomposition of the gap between linear-probe predictions and intervention attribution. Across four model-SAE pairs, median probe-to-attribution magnitude ratios range from tens to hundreds, while attributions closely track small-step steering. For single-latent ablations with nonzero effects, gradient predictions match the realized sign in 91-97% of cases, compared with 17-35% for probe-implied changes. Attribution-selected sets also produce larger absolute mean ablation effects on held-out prompts than sets selected by correlation, rank tests or probe coefficients. Together, these results establish intervention attribution as a principled, gradient-estimable target for selecting SAE features for intervention.

Figures & tables

Figure 1: Association and intervention for SAE latents. (A) Gemma-2-9B completes “In Turkey, people commonly speak” with

p(\text{Turkish})=0.349

. FDFI ranks latent 102779 first and latent 99439 23rd among the

39

latents it selects for this task. Zero-ablating each latent at every position but the first gives the realized intervention effect (bars) and the prompt-specific gradient prediction (ticks) in standardized log-probability units. The illustrative example is post hoc. It is the sole eligible unit under a selection rule specified after the aggregate analyses with a required positive attribution-set versus FDFI-set contrast ( Appendix H ). (B) A separately fitted probe on the final-position score

d_{j}^{\top}h

implies a change

\rho

, whereas an intervention that edits along

d_{j}

has first-order effect

\alpha

, its attribution. The law of

(H,Y)

fixes

F

only on the support of

H

and need not identify

\alpha

( Proposition 1 ). (C) With gradients the gap

\rho-\alpha

is a Stein residual plus an omitted-variable term minus an edit-gradient covariance ( Theorem 1 ). The secant illustration assumes affinely independent support.

Term	Compares	Zero when	SAE instance
Stein residual $\mathsf{s}_{\bar{u}^{\prime}}$	observational regression slope with mean gradient	$F$ globally affine, or $H$ Gaussian with $\Sigma_{H}\succ 0$	a response curved across prompts
Omitted variable $\mathsf{c}(\Delta)$	probe’s activation pattern with mean edit	$(v_{A}(\bar{u})-\bar{u})\perp\gamma$ , e.g. $v_{A}(\bar{u})=\bar{u}$	an edit at positions the probe does not read
Edit-gradient covariance $\mathsf{e}(\Delta)$	prompt-varying edit with model sensitivity	fixed edit, as for steering	ablation of a rarely firing latent

Table 1: The three terms of ( 5 ), what each compares, when it vanishes and where it arises in SAE pipelines. The empirical stratum laws are finitely supported. The Gaussian case is a benchmark ( Lemma B.1 ).

	Size of the gap						Realized interventions				Choosing latents
							Steering		Ablation			Sets, attr. $-$
Model	Pairs	$\|\alpha/\beta\|$ (%) [CI]	$<0.1$	$\|\mathsf{s}/\beta\|$	$\|\mathsf{c}/\beta\|$	$\rho\approx 0$	$\|T/\alpha\|$	$\|T/\beta\|$	MAE red.	Sign (%)	Top-1	$\|\beta\|$	FDFI
GPT-2	46	4.6 [3.0, 5.9]	0.739	0.614	0.514	0.370	1.000	7.60	0.303	92/21	1/0/4 (6)	0.454	0.290
Gemma-2-2B	230	3.2 [2.2, 3.6]	0.770	0.922	0.593	0.296	1.000	3.68	0.119	91/22	0/0/4 (9)	0.506	$-0.209$
Gemma-2-9B	457	2.8 [1.9, 3.5]	0.751	0.728	0.463	0.287	1.001	2.50	0.114	93/17	0/0/4 (9)	2.137	1.380
Llama-3.1-8B	23 ∗	3.2 [2.2, 5.1]	0.826	0.583	0.406	0.043	1.000	3.36	0.300	97/35	2/1/4 (4)	0.309	0.206

Table 2: Main results on held-out prompts. Columns Pairs through

|\mathsf{c}/\beta|

use FDFI-selected directions under unit steering. Ratios are medians over pairs, with intervals that resample strata within task. Column

<0.1

is the share with

|\alpha/\beta|<0.1

and

\rho\approx 0

is the post hoc share of single-latent zero-ablations with

|\rho|\leq 10^{-3}<|\alpha|

. Steering is descriptive, on coefficient-selected directions (

90

100

100

and

90

pairs), with

T

the central difference quotient of the realized steering interventions and

|T/\beta|

in units of

10^{-3}

. Ablation gives the reduction in mean absolute error from the probe to the gradient prediction. Sign gives the gradient/probe sign agreement among rows with nonzero realized effects, averaged equally over tasks, with a zero prediction counted as a disagreement. Choosing latents is descriptive. Top-1 counts the units, out of the number in parentheses, in which the FDFI statistic,

|\beta|

|\alpha|

ranks first the latent with the largest

|T|

at one maximum activation. The last columns give model means of the attribution-set minus comparator contrast in absolute held-out mean set-ablation effects, in standardized units, with all five comparators in Appendix G . ∗ Fewer than thirty pairs.

Figure 2: Probe-implied changes and intervention attributions on held-out prompts, in standardized units. (A) Magnitudes of the unit-direction attribution

\alpha

(filled) and the probe coefficient

\beta

(open, crosses for the sign opposite to

T

) against the central difference quotient

|T|

of steering at

\pm 0.25

within-stratum standard deviations for the

380

coefficient-selected pairs. (B) Probe-implied change

\rho

against attribution

\alpha

on symmetric-log axes under zero-ablation of the

756

FDFI-selected pairs. Shading marks opposite signs and negligible predictions (

|\rho|<10^{-3}

). Dashed diagonals mark equality. (C) Shares of the summed absolute terms of ( 5 ) for the final-position probe, with the omitted-variable (OV) term split by position and zero- and mean-ablation as upper and lower bars for each model. Panel A is descriptive. Panels B and C are post hoc and descriptive ( Sections E.2 and E.4 ).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Law of $H$	$F(x)$	Edit	$\rho$	$\alpha$	$\mathsf{s}_{\bar{u}^{\prime}}$	$\mathsf{c}$	$\mathsf{e}$
uniform on $\{-1,1\}$	$x^{3}$	$1$	$1$	$3$	$-2$	$0$	$0$
$(X,X)$ , $\operatorname{Var}(X)>0$	$x_{2}$	$e_{1}$	$1$	$0$	$0$	$1$	$0$
uniform on $\{0,2\}$	$x^{2}/4$	$-H$	$-\frac{1}{2}$	$-1$	$0$	$0$	$-\frac{1}{2}$

Appendix

Table B.1: Closed-form examples of the three terms of ( 5 ), with

A=1

A=(1,0)

. Each row satisfies

\rho-\alpha=\mathsf{s}_{\bar{u}^{\prime}}+\mathsf{c}-\mathsf{e}

	Gaussian				$t_{5}$				JumpReLU code
	GPT-2	2B	9B	8B	GPT-2	2B	9B	8B	GPT-2	2B	9B	8B
Tasks	8	10	10	7	8	10	10	7	8	10	10	7
Stein, mean	0.961	0.971	0.970	0.967	0.918	0.925	0.927	0.925	0.835	0.821	0.824	0.820
Stein, test-weighted	0.955	0.971	0.970	0.964	0.915	0.929	0.925	0.921	0.863	0.814	0.829	0.830
Stein, $K_{s}\leq 9$	0.974	0.981	0.978	0.978	0.925	0.934	0.939	0.936	0.804	0.792	0.796	0.777
Stein, $K_{s}\geq 30$	0.946	0.952	0.960	0.950	0.908	0.914	0.915	0.914	0.882	0.886	0.889	0.880

Appendix

Table C.1: Coverage of nominal

95\%

stratum-level

t

intervals at the empirical stratum sizes, averaged over the reproduced tasks (Tasks). 2B, 9B and 8B are Gemma-2-2B, Gemma-2-9B and Llama-3.1-8B. Test-weighted weights tasks by their number of tests. The

K_{s}

rows keep tasks with at most nine or at least thirty strata.

Model	Alternate SAE (primary)	FVU alt. / prim. (thr.)	Pool prim. / alt.	Primary (i) / (ii) / (iii)	Alternate (i) / (ii) / (iii)
GPT-2	res_scefr-ajt ( res-jb )	0.729 / 0.139 (0.278)	N/A	not eligible	N/A
Gemma-2-2B	32k, $L_{0}$ 40 (16k, $L_{0}$ 41)	0.084 / 0.092 (0.184)	86 / 62	$4.97\times 10^{-3}$ / 0.420 / 0.040	$5.02\times 10^{-3}$ / 0.390 / 0.040
Gemma-2-9B	16k, $L_{0}$ 35 (131k, $L_{0}$ 32)	0.056 / 0.045 (0.090)	46 / 71	$9.51\times 10^{-3}$ / 0.300 / 0.030	$5.95\times 10^{-3}$ / 0.390 / 0.040
Llama-3.1-8B	l16r_32x ( l16r_8x )	0.243 / 0.275 (0.500)	409 / 278	$4.75\times 10^{-3}$ / 0.389 / 0.111	$4.07\times 10^{-3}$ / 0.356 / 0.111

Appendix

Table E.3: A second dictionary at the same hook, with the candidate pool and the criteria (i) to (iii) of Section E.2 . The FVU threshold is

\min\{0.5,\,2\,\mathrm{FVU}_{\rm prim}\}

Pairs	Model	$N$ (tasks)	Median $\|\alpha/\beta\|$	$\sum\|\alpha\|/\sum\|\beta\|$ [CI]	all	$K_{s}\geq 10$	median [q5]	opposite sign
					Rejections of $G=0$		Noise-only null	Both significant,
FDFI-selected	GPT-2	46 (8)	0.046	0.036 [0.019, 0.064]	35	34/38	0.226 [0.156]	5/20
	Gemma-2-2B	230 (10)	0.032	0.032 [0.022, 0.048]	65	50/92	0.080 [0.066]	11/42
	Gemma-2-9B	457 (10)	0.028	0.038 [0.021, 0.071]	110	99/225	0.067 [0.059]	9/67
	Llama-3.1-8B	23 (7) ∗	0.032	0.033 [0.023, 0.049]	17	13/14	0.170 [0.103]	2/12
Coefficient-selected	GPT-2	90 (9)	$7.61\times 10^{-3}$	$9.35\times 10^{-3}$ [ $6.07\times 10^{-3}$ , 0.013]	43	36/40	0.030 [0.023]	6/16

Appendix

Table E.4: Per-pair tests of the gap

G=\beta-\alpha

(BH at

0.05

within model) and the noise-only null of Section E.3 . The pooled ratio has a task-resampled interval. Column

K_{s}\geq 10

counts rejections among pairs estimated from at least ten strata. The null columns give the median and fifth percentile of the simulated median ratio. The last column counts opposite signs among the pairs whose

\alpha

and

\beta

are both significant.

		Probe-blind share			Opposite sign of $\rho$ and $\alpha$			Median $\|\alpha/\rho\|$
Model	Pairs	final	avg. decoder	avg. code	final	avg. decoder	avg. code	avg. decoder / code
GPT-2	46	0.391	0	0	0.414	0.403	0.336	0.294 / 0.167
Gemma-2-2B	230	0.452	0.013	0	0.268	0.487	0.456	0.100 / 0.119
Gemma-2-9B	457	0.534	0.020	0	0.436	0.542	0.516	0.103 / 0.056
Llama-3.1-8B	23	0.087	0	0	0.141	0.165	0.160	0.298 / 0.127

Appendix

Table E.6: Probes on FDFI’s position-averaged summaries under zero-ablation, against the final-position probe with the same weights (final). Blind shares are over pairs. The descriptive sign shares and the ratio column are equal-task averages of within-task shares and medians. The code summary excludes

4

and

10

pairs on the Gemma models whose averaged code has negligible pooled within-stratum variance.

Figure E.1: Probes on FDFI’s position-averaged summaries under zero-ablation, a post hoc descriptive analysis ( Table E.6 ). The left panel gives the probe-blind share pooled over pairs and the right panel the equal-task average of within-task median

|\alpha/\rho|

over pairs with

\rho\neq 0

on a logarithmic scale. Dashed lines mark a zero blind share and equal magnitudes. The Gemma-2-2B final-position ratio of

2.916

is driven by one task with a single nonzero-probe pair.

Zero-ablation
	GPT-2	Gemma-2-2B	Gemma-2-9B	Llama-3.1-8B
Estimates (tasks)	46 (8)	230 (10)	457 (10)	23 (7) ∗
$\|C_{j}\|>\|M_{j}\|$	0.522	0.339	0.385	0.174
equal task weights	0.317	0.287	0.335	0.097
strata with $n_{k}\geq 2$	0.500	0.344	0.360	0.208
$\operatorname{sign}(M_{j}+C_{j})\neq\operatorname{sign}(M_{j})$	0.261	0.174	0.182	0.043

Appendix

Table F.1: The two terms of ( 7 ) per direction-task estimate, in strata of at least three prompts unless stated otherwise. The equal-task-weights row averages the within-task fraction over tasks. The share of

C_{j}

is the median of

|C_{j}|/(|M_{j}|+|C_{j}|)

, with its tenth and ninetieth percentiles. BH counts the Benjamini-Hochberg rejections of

C_{j}=0

within model among the estimates with a defined standard error, based on stratum-level

t

intervals ( Section C.2 ).

Single-latent zero-ablations	GPT-2	Gemma-2-2B	Gemma-2-9B	Llama-3.1-8B
Prompt-latent rows (tasks)	1,795 (7)	10,727 (9)	29,180 (9)	1,782 (7)
MAE of $P_{ij}$	0.026	0.097	0.082	$8.52\times 10^{-3}$
MAE of $M_{ij}$	0.054	0.115	0.091	0.032
MAE of ${\rm probe}_{ij}$	0.329	0.216	0.196	0.308
Gain over the probe	0.303	0.119	0.114	0.300
95% CI	[0.037, 0.799]	[0.078, 0.167]	[0.064, 0.171]	[0.114, 0.611]

Appendix

Table F.2: Single-latent zero-ablations, with the predictions

P_{ij}

M_{ij}

and

{\rm probe}_{ij}

of Section F.2 . MAE weights tasks equally. Gains are differences in MAE, positive when

P_{ij}

is more accurate. Their intervals resample tasks within model. Sign agreement is over rows with

D_{ij}\neq 0

, with a zero prediction counted as a disagreement. Sign agreement averages within-task rates equally over tasks. The correlation

r

is the mean over tasks of the within-task Pearson correlation.

		Within-stratum standard deviations				Maximum activations
Target	Ranked by	1	2	5	10	1	2	5
$\|T\|$	FDFI statistic	4	4	3	3	3	4	2
	$\|\beta\|$	2	2	2	2	1	3	3
	$\|\alpha\|$	28	27	19	17	16	10	5
$\|\varepsilon T\|$	FDFI statistic	4	4	3	3	5	4	2
	$\|\varepsilon\beta\|$	7	7	5	4	4	6	8

Appendix

Table G.1: Number of the

28

model-task units in which the latent with the largest steering response is ranked first. Uniformly random choice gives

3.31

in expectation. Post hoc and descriptive.

Comparator $c$	$d_{c}$	Stratum-resampled 95% CI	Two-stage 95% CI	GPT-2	Gemma-2-2B	Gemma-2-9B	Llama-3.1-8B	Units with $d_{c}>0$
Correlation	0.949	[0.903, 1.001]	[0.292, 2.147]	0.396	0.370	2.033	0.185	28/28
Rank test	0.933	[0.885, 0.984]	[0.277, 2.121]	0.415	0.276	2.044	0.278	28/28
Class-mean difference	0.724	[0.640, 0.798]	[0.061, 1.795]	0.331	$-4.91\times 10^{-3}$	1.795	0.236	24/28
$\|\beta\|$	1.058	[1.013, 1.112]	[0.371, 2.304]	0.454	0.506	2.137	0.309	28/28
FDFI	0.492	[0.357, 0.602]	[ $-0.179$ , 1.441]	0.290	$-0.209$	1.380	0.206	23/28

Appendix

Table G.2: Held-out set ablations under six selection rules. The contrast

d_{c}=A_{\rm attr}-A_{c}

is in standardized units, equal over units. Model columns give the mean contrast over the model’s units. The mean attribution-set effect is

0.467

0.511

2.165

and

0.310

against

0.013

5.55\times 10^{-3}

0.028

and

7.68\times 10^{-4}

for coefficient sets.

Figure H.1: The worked example (Gemma-2-9B, country to language). (A) Three held-out prompts chosen by baseline log-probability quantiles, with

p(\text{answer})

unedited and after zero-ablating latent 102779 or 99439 at every position but the first (change in parentheses). Headers give each latent’s FDFI rank and statistic. (B) Realized change of each ablation (bars) against the prompt-specific gradient prediction (ticks), in standardized units. The example is illustrative, selected post hoc as the sole eligible unit of the deterministic rule of this section.

Prompt			Realized	First-order	Edit
answer (unedited $p$ )	Zero-ablated	$p(\text{answer})$	change	prediction	norm
In Turkey, people commonly speak	FDFI repr.	0.353	0.015	0.012	97.006
Turkish (0.349)	attribution repr.	0.403	0.201	0.224	24.246
	FDFI set	0.334	$-0.062$	$-0.043$	120.406
	attribution set	0.457	0.381	0.479	85.688
Most people in Lebanon speak	FDFI repr.	0.448	$3.04\times 10^{-4}$	$3.08\times 10^{-4}$	18.565

Appendix

Table H.1: The worked example of Figure H.1 (Gemma-2-9B country_language , first fold). Changes and the prompt-specific gradient predictions, summed over the ten latents for set ablations, are in standardized log-probability units. The edit norm is

(\sum_{\tau}\|\Delta h_{\tau}\|^{2})^{1/2}

and repr. is the representative latent.

Sparse Autoencoders

Sparse autoencoders decompose neural network activations into sparse features to study, interpret, and control learned representations.

See all 218 papers

Sparse Autoencoders Encode Both Concepts and Functions: The Downstream Geometry of Feature Effects
Jul 27, 2026Phu Gia Hoang, Anwoy Chatterjee, Tanmoy Chakraborty +2Sparse Autoencoders Mechanistic Interpretability
UKP Lab, Technical University of Darmstadt · Department of Electrical Engineering Indian Institute of Technology Delhi, India
ScopeSAE: Model-Scope Feature Discovery with Interpretable Layer Selection
Oct 4, 2026Qingwen Zeng, Zehao Fu, Shuyu Meng +5Transformer Interpretability Sparse Autoencoders
The University of Sydney · The Hong Kong University of Science and Technology (Guangzhou) · DeepWisdom +1
From Input to Output: A Flexible Agent for Dual-End Interpretation of Sparse Autoencoder Features
Sep 28, 2026Dewen Liu, Zixuan Li, Jonathan Pan +4Sparse Autoencoders Mechanistic Interpretability
Tsinghua University · Fudan University · University of Edinburgh

Jul 27, 2026J/K move · Enter open · S save

Sparse Autoencoders Encode Both Concepts and Functions: The Downstream Geometry of Feature Effects

Phu Gia Hoang, Anwoy Chatterjee, Tanmoy Chakraborty, Iryna Gurevych+1

UKP Lab, Technical University of Darmstadt · Department of Electrical Engineering Indian Institute of Technology Delhi, India

Sparse Autoencoders Mechanistic Interpretability Causal Interventions in Language Models

Open paper AskPDF arXiv

The wide-scale use of sparse autoencoders (SAEs) as interpretability tools is limited by inconsistent links between SAE features and model behavior. Features with clear activation descriptions may have weak or unexpected causal effects; steering can vary across prompts or oppose the intended direction; and activation-based feature selection can miss features that produce the desired output change. Prior work has studied feature geometry inside the model, where features are computed. We instead study the geometry of changes in model logits caused by feature interventions. We introduce Feature-Effect Geometry Analysis (FEGA), an unsupervised framework that removes the same active SAE feature across contexts and analyzes the resulting cloud of logit changes. Across SAE variants, consistent one-dimensional effects are rare: few features behave like reusable directions. To interpret this variation, we distinguish value-like features, tied to static information such as factual attributes, from pointer-like features, associated with context-dependent operations. Value-like features more often exhibit structured, low-dimensional effects, although these effects typically span several directions. Pointer-like features, by contrast, predominantly exhibit diffuse effects. Our results show that a feature can be interpretable and causally relevant without providing a stable direction for steering.

Abstract

Sparse Autoencoders

Explore similar work

Sparse Autoencoders Encode Both Concepts and Functions: The Downstream Geometry of Feature Effects

ScopeSAE: Model-Scope Feature Discovery with Interpretable Layer Selection

From Input to Output: A Flexible Agent for Dual-End Interpretation of Sparse Autoencoder Features