Look Before You Steer: Geometry Predicts SAE Feature Steerability

Muhammad Khan; Shlok Channawar; Akshaj Gurugubelli; Girish Gupta; Aditya Shah

cs.LGSep 19, 2026

Look Before You Steer: Geometry Predicts SAE Feature Steerability

Authors: Muhammad Khan, Shlok Channawar, Akshaj Gurugubelli, Girish Gupta, Aditya Shah

Organizations: Algoverse AI Research.

Abstract

Steering with SAE features requires per-feature coefficient tuning, which currently demands intervention sweeps. We ask whether properties of the SAE itself, computable before any forward pass, predict which features will be cheap or expensive to steer. We show that variation in SAE feature steerability is partially predicted by decoder-space geometry: neighbor density and maximum cosine similarity to nearby decoder directions, both computable from the SAE weight matrix before any intervention, rank features by how much steering they require for a fixed behavioral effect ( $ρ$ up to $-0.546$ , $p < 10^{-6}$ , AUROC 0.610-0.822 across conditions; the signal is rank-based, consistent with grid discreteness). This geometry-steerability relationship replicates across two Gemma-2 model scales (2B and 9B), two SAE widths (16K and 65K), and is detectable cross-architecturally on Llama-3.1-8B-Instruct ( $ρ= -0.266$ , $n = 300$ ). On Qwen3-8B with BatchTopK SAEs, geometry predicts whether a feature is steerable at all but not the continuous ordering among responsive features, revealing a boundary condition tied to SAE training regime. The signal weakens at deep proportional layer depth in both models, where the cost of steering exceeds our intervention budget, a consistent depth boundary. These results provide preliminary evidence that pre-steering geometry can partially inform coefficient selection, offering a path toward screening features for controllability before deployment.

Figures & tables

Figure 1: Illustration of the geometry–steerability relationship. (a) Two SAE features in decoder space. The isolated feature (

v_{f}

, blue) has few nearby decoder directions; the crowded feature (

v_{g}

, red) sits in a dense cluster. Grey arrows are other features in the same dictionary. (b) Steering each feature: the crowded one crosses the refusal threshold

T

at a much smaller

\alpha^{*}

than the isolated one. weight matrix before any steering — predicts this gap. Curves are schematic; empirical results are shown in Figure 2 .

Model	SAE	Layer(s)	Dict.	Rel. depth
Gemma-2-2B	GemmaScope 16K	20, 22, 24	16K	77%, 85%, 92%
Gemma-2-2B	GemmaScope 65K	20	65K	77%
Gemma-2-9B	GemmaScope 16K	20, 36	16K	48%, 86%
Llama-3.1-8B	BatchTopK	15	131K	47%
Llama-3.1-8B	BatchTopK	23	131K	72%
Qwen3-8B	BatchTopK	18	32K	50%

Table 1: Experimental conditions.

Model	Condition	Rel. depth	Runs	$N$	$N_{\text{cens}}$ (%)	Mean $\alpha^{*}$	Median $\alpha^{*}$
2B	L20 16K	77%	3	100	12 (12%)	3.39	3.0
	L22 16K	85%	3	100	27 (27%)	3.84	3.0
	L24 16K †	92%	3	100	—	—	—
	L20 65K	77%	4	100	11 (11%)	3.39	3.0
9B	L20 16K	48%	3	75	2 (3%)	4.25	3.0
9B	L36 16K	86%	2	100	46 (46%)	4.83	5.0

Table 2: Steerability distribution across experimental conditions.

N_{\text{cens}}

is the number of right-censored features (

\alpha^{*}>\alpha_{\max}

). Mean and median

\alpha^{*}

are computed over uncensored features only. Relative depth is layer index divided by total layers. Runs is the number of independent pipeline runs completed for each condition. † Distribution collapsed due to coherence collapse at high

\alpha_{\max}

; excluded from primary inference per Section 3.2 .

Condition	Metric	$r_{\text{full}}$	$p_{\text{full}}$	$r_{\text{nocens}}$	$p_{\text{nocens}}$	$n$
2B L20 16K (3 runs)	max_cosine	$-0.192$	$0.055$	$-0.299$	$0.005$	88
	neighbor_density	$-0.206$	$0.039$	$-0.288$	$0.006$	88
	coactivation	$+0.030$	$0.765$	$+0.184$	$0.087$	88
2B L22 16K (3 runs)	max_cosine	$-0.293$	$0.003$	$-0.425$	$0.0002$	73
	neighbor_density	$-0.222$	$0.026$	$-0.512$	$3.6\times 10^{-6}$	73
	coactivation	$+0.147$	$0.145$	$+0.147$	$0.214$	73

Table 3: Spearman correlations between pre-steering metrics and

\log\alpha^{*}(f)

r_{\text{full}}

includes censored features (rank-based handling of ties at boundary);

r_{\text{nocens}}

excludes them.

n

is the number of uncensored features used for

r_{\text{nocens}}

. For Llama-3.1-8B,

r_{\text{full}}=r_{\text{nocens}}

since censoring is zero. Qwen3-8B results are reported in Appendix F .

Condition	$n$	CV $R^{2}$	AUROC	Perm. null $\rho_{95}$	LOO: drop density	LOO: drop max cos
2B L20 16K	88	0.068	0.610	0.168	0.054	0.075
2B L22 16K	73	0.047	0.736	0.194	$-$ 0.045	0.109
2B L20 65K	89	$-$ 0.005	0.686	0.172	$-$ 0.048	0.022
9B L20 16K	73	0.025	0.822	0.197	$-$ 0.028	0.053
9B L36 16K †	54	$-$ 0.698	0.723	0.225	$-$ 0.607	$-$ 0.742
Llama L15 131K	300	—	0.653	—	—	—

Table 4: Predictability analysis: cross-validated

R^{2}

(ridge regression on all three metrics), AUROC (steerable

\alpha^{*}\leq 3

vs. hard-to-steer), and permutation-null 95th percentile

\rho

across well-identified conditions. Censored features excluded from

R^{2}

and AUROC; included (rank-tied) in Spearman. Llama-3.1-8B L23 is omitted; cross-architecture Spearman correlations for that condition are reported in Table 3 .

Figure 2: Pre-steering metrics versus log steerability for the 9B model at layer 20 (strongest signal condition). Each point is one SAE feature; red triangles indicate right-censored features (

\alpha^{*}=20

). Trend line shows LOWESS fit; shaded band is bootstrap 95% CI. Neighbor density:

\rho=-0.546

p=5.75\times 10^{-7}

. Max cosine similarity:

\rho=-0.490

p=1.10\times 10^{-5}

. Co-activation correlation: zero-inflated; see text.

Figure 3: Spearman

\rho

(neighbor density vs.

\log\alpha^{*}

) as a function of relative layer depth across all conditions. The geometry–steerability signal is strongest at shallow-to-mid depth and weakens consistently as the intervention layer approaches the output. Open marker indicates a not-well-identified condition (Section 3.2 ). Error bars show approximate 95% CIs via Fisher

z

-transform. Shaded region marks the deep-depth boundary.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Condition	ND mean	ND std	MC mean	MC std
2B L20 16K	0.194	0.053	0.392	0.143
2B L22 16K	0.178	0.049	0.368	0.141
2B L20 65K	0.323	0.078	0.547	0.139
9B L20 16K	0.212	0.045	0.400	0.110
9B L36 16K	0.180	0.045	0.373	0.127

Appendix

Table 5: Pre-steering geometry statistics per condition.

Sparse Autoencoders

Sparse autoencoders decompose neural network activations into sparse features to study, interpret, and control learned representations.

See all 218 papers

Enhancing SAE-based Steering via Neighbor Integrated Feature Selection
Aug 28, 2026Yutian Liu, Xu Wang, Difan ZouLanguage Model Steering Sparse Autoencoders
The University of Hong Kong · University of Science and Technology of China
Pre-Intervention Prediction of Sparse Autoencoder Steering Side Effects
Jun 6, 2026Evan DuanLanguage Model Steering LLM Interpretability
Sparse Autoencoders Encode Both Concepts and Functions: The Downstream Geometry of Feature Effects
Jul 27, 2026Phu Gia Hoang, Anwoy Chatterjee, Tanmoy Chakraborty +2Sparse Autoencoders Mechanistic Interpretability
UKP Lab, Technical University of Darmstadt · Department of Electrical Engineering Indian Institute of Technology Delhi, India

Aug 28, 2026J/K move · Enter open · S save

Enhancing SAE-based Steering via Neighbor Integrated Feature Selection

Yutian Liu, Xu Wang, Difan Zou

The University of Hong Kong · University of Science and Technology of China

Language Model Steering Sparse Autoencoders

Open paper AskPDF arXiv

Sparse autoencoders (SAEs) disentangle model activations into interpretable features and are widely used for steering large language models. Most existing SAE-based steering methods select features by applying a top- filter based on statistical scores, assuming that higher-scoring features yield stronger steering effects. In this paper, we show that this assumption is often invalid, leading to suboptimal feature selection. Our analysis reveals that effective steering features may be distributed among representationally adjacent, semantically similar groups induced by feature splitting in SAEs. Within such groups, features may exhibit disparate statistical scores despite having comparable steering influence, causing score-based selection to overlook important features. Based on these observations, we propose Neighbor Integrated Feature Selection (NIFS), a plug-and-play strategy that leverages representation similarity to improve feature selection for steering. We evaluate NIFS across multiple SAE-based steering methods and tasks, and demonstrate consistent performance gains over conventional top-

k

selection.

Abstract

Sparse Autoencoders

Explore similar work

Enhancing SAE-based Steering via Neighbor Integrated Feature Selection

Pre-Intervention Prediction of Sparse Autoencoder Steering Side Effects

Sparse Autoencoders Encode Both Concepts and Functions: The Downstream Geometry of Feature Effects