Changing sparse autoencoder (SAE) codes can alter both intervention direction and magnitude, so comparing intervention effects requires a strength control specifying what is held equal. Norm matching and output calibration provide different controls, but whether interventions yield different effects across codes after strength matching remains unclear. We compare intervention effects under these controls for fixed feature groups in Gemma and Llama and observe: (1) Whole-patch scaling. With one coefficient per input, full-output Kullback-Leibler (KL) calibration leaves smaller mean effect gaps from the encoder-based reference intervention than total-norm matching for nonnegative least-squares (NNLS) refits. On new Llama inputs, total-norm matching loses 11 reference transfers, whereas full-output calibration preserves every transfer decision. (2) Positionwise scaling. On a small matched gender-agreement subset, positionwise norm matching leaves smaller observed mean gaps than full-output calibration. (3) Answer exclusion. Compared with unscaled NNLS, answer-excluded calibration reduces gaps in the measured subject-verb agreement comparisons but increases them for gender refits. Full-output KL calibration also reduces larger gaps for norm-matched constructed candidates, although matching coverage is incomplete in several settings. Since calibration uses the evaluated input's outputs, it serves as a diagnostic of residual disagreement.
Figures & tables
Figure 1: Strength control can change a transfer decision. A Llama-128K example selected post hoc. (1) Prompts with task answers appended. (2) E is the reference intervention built from the released encoder’s codes. Separate patches on the same native recipient. Rankings ( > ) compare only the two answer logits. Ticks indicate donor-answer transfer, crosses its absence. (3) Transfers lost or gained relative to E across all 304 inputs. Agreement with E is not a faithfulness standard.
Figure 2: Four-step comparison with a fixed model, dictionary, feature group, and sentence pair. The reference intervention E , built from the released encoder’s codes, supplies the reference targets (gray) for rescaling the original alternative patch (color). Norm segments show squared patch norms, matched in sum or per position. KL controls use trial model outputs to choose one scaling coefficient (knob) per input for matching scalar KL. Q2 excludes both answer tokens (crossed) and renormalizes before matching conditional KL. Step 4 keeps beginning-of-sequence (BOS) and last prompt tokens, comparing native-to-patched KL (nats), donor-oriented margin change (logits) and transfer with E . Matching can fail.
Configuration
Width
Layer
∣G∣
NNLS
Cand.
Gemma-9B-it SVA
131,072
20
6
166
504
Gemma-2B SVA
131,072
12
6
62
64
Llama-8B SVA
131,072
15
12
188
124
New material
131,072
15
12
304
—
Llama-8B SVA
32,768
15
12
112
106
Gemma-2B gender
131,072
12
12
42
40
Table 1: Configurations and eligible inputs before matching. Width denotes dictionary size, Layer is zero-indexed, and ∣G∣ is group size. SVA denotes subject–verb agreement. Llama-128K/32K indicate dictionary widths. The 9B NNLS and candidate panels are separate. New Llama material has no candidate comparison.
Figure 3: NNLS gap reductions relative to total-norm matching. Full-output KL calibration (Q1, panel a) reduces mean gaps in every cohort. Positionwise matching (b) improves on total-norm matching in most observed means. Points show mean paired differences in absolute effect gaps from the encoder-based reference intervention E (total-norm minus alternative), so positive values favor the alternative. Each comparison uses its n common matched inputs. Means are case/block-balanced (pooled for 9B). Bars show pointwise 95% paired block-bootstrap intervals, and “CI n/a” marks an unavailable interval. Transfer counts report lost/gained relative to E .
Figure 4: NNLS control orderings differ across settings. Q1 denotes full-output KL, Q2 answer-excluded KL. (a) Case/block-balanced mean gaps from the encoder-based reference intervention E on inputs matched by all three controls in each row, shown descriptively without intervals. (b) Case/block-balanced mean paired gap differences (Q2 minus unscaled NNLS), with pointwise 95% block-bootstrap intervals. Parentheses give input counts. Gender subsets differ between panels. Llama rows use the new cohort, where Q2 is a follow-up after inspection.
Figure 5: Constructed candidates under full-output KL calibration (Q1). (a) Pooled gaps from the encoder-based reference intervention E before (open circles) and after (filled circles) search within α≤2 . All-attempt rows retain failed searches at their best measured point. Diamonds show post hoc continuation to α≤4 (Appendix C.5 ). (b) Paired reductions on original matched inputs, case-balanced for 9B and case/block-balanced otherwise. Pointwise 95% intervals use lexical-block Student- t for 9B and paired block bootstrap elsewhere. The 9B primary interval’s half-width exceeds the planned 0.25 logit. Gender lacks a balanced estimate and interval because one case has no matches. Coverage follows the original per-case rule. Llama intervals describe selected subsets.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Mean gap (logit)
Paired difference (logit)
Lost/gained
Cohort
n
Weighting
Total norm
Q1
Estimate
95% interval
Total norm
Q1
Earlier panels (follow-up controls)
9B SVA
165
pooled
0.056
0.015
0.041
unavailable
1/2
1/0
2B SVA
62
balanced
0.033
0.008
0.024
[0.011,0.042]
0/0
0/0
Llama-128K
188
balanced
0.160
0.017
0.142
[0.117,0.168]
10/1
0/1
Llama-32K
112
balanced
0.127
0.012
0.115
[0.077,0.159]
3/1
0/1
Appendix
Table 3: Coefficient-count matched NNLS controls. Q1 leaves smaller mean effect gaps on inputs matched by both controls. Differences are total norm minus Q1 (logits). Balanced means equally weight cases and blocks within case. Pointwise 95% paired block-bootstrap intervals assume independent blocks; the 9B interval is unavailable. Lost/gained transfers are unweighted counts relative to E . The prospective cohort uses new material in the same Llama-128K configuration and is estimated separately from the earlier panel. These are the full values underlying Figure 3 a.
Cohort
n
Estimate
95% interval
9B SVA
150
−0.024
unavailable
2B SVA
62
0.001
[−0.004,0.006]
Llama-128K
188
−0.048
[−0.067,−0.030]
Llama-32K
109
−0.029
[−0.047,−0.012]
2B gender
24
−0.073
unavailable
Prospective confirmation
Appendix
Table 4: Positionwise minus total-norm effect gap. Negative values favor positionwise norm matching. Each row uses its own common matched inputs; Figure 3 b plots the negative of these values. Values are in logits, with case/block balancing except for pooled 9B. Intervals use the assumptions in Table 3 . The gender interval is unavailable because too few bootstrap draws preserve the required cells.
Cohort
n
Median ∣ei(E)∣
Encoder transfers
9B SVA
165
3.92
85
2B SVA
62
4.61
55
Llama-128K
188
3.33
126
Llama-32K
112
4.05
96
2B gender
42
4.47
42
Appendix
Table 5: Effect magnitude of the encoder-based reference intervention E on the five earlier total-norm/Q1 common-input cohorts. Absolute-effect medians are in logits; transfer counts are unweighted. These background quantities describe the same inputs as the paired comparison, separately for each cohort.
Code
Inputs
Q2 matches
Before
Final
Reduction and 95% interval
NNLS
40
40
0.089
0.139
−0.050[−0.091,−0.008]
Candidate
40
13
0.754
0.401
0.353[0.265,0.447]
Appendix
Table 6: Answer exclusion is a boundary, not a uniform robustness result. Q2 worsens NNLS effect agreement on the gender common-input cohort despite complete matching. Mean candidate gaps decrease across all attempts, with failed coverage. Means and reductions are case/block-balanced logits; final means retain unmatched attempts. Intervals are pointwise conditional 95% block-bootstrap intervals. A negative reduction denotes deterioration.
Configuration
Code
Quantity
n
Estimate
Lower
Upper
9B SVA
Candidate
Q1 reduction
478
0.7239
0.3241
1.1236
9B SVA
Candidate
Q1 gap
478
0.0284
0.0223
0.0344
9B SVA
Candidate
Q2 gap − Q1 gap
504
0.0071
0.0042
0.0100
2B SVA
Candidate
Q1 reduction
64
0.5459
0.3637
0.7485
2B SVA
Candidate
Q1 gap
64
0.0149
0.0106
0.0201
2B SVA
Candidate
Q2 reduction
64
0.5410
0.3590
0.7446
Appendix
Table 7: Primary and supporting calibration estimates with pointwise 95% intervals. All quantities are in logits. A reduction is the original gap minus the selected gap, so a negative reduction denotes deterioration. The 9B rows use case-balanced means and block-based Student- t intervals. Other rows use case/block-balanced means and lexical-block bootstrap intervals. Candidate Q1 rows use matched inputs, including the prespecified 9B primary reduction. The 9B Q2-minus-Q1 row retains all attempts, and the gender row uses the complete 42-input NNLS cohort, distinct from the common-40 comparison in Table 6 . Each row preserves its own input set and inferential assumptions.
Balanced mean absolute gap (logit)
Panel
n
Before
After
9B: all attempts
504
0.816
0.076
9B: matched
478
0.752
0.028
2B: all, matched
64
0.561
0.015
Appendix
Table 8: Candidate absolute effect gaps from the encoder-based reference intervention E , in logits, before and after Q1. Means use each setting’s specified balancing; n counts inputs. The 9B all-attempt row includes 26 failed matches. The 9B matched row uses the inputs for the prespecified primary comparison, whose estimand is the paired reduction. All 2B attempts match.
Subject–attractor
Matched
Before
After
Plural–plural
118/124
0.643
0.020
Plural–singular
117/124
0.702
0.036
Singular–plural
130/136
0.740
0.025
Singular–singular
113/120
0.923
0.033
Appendix
Table 9: Matching coverage and mean absolute effect gaps (logit) before and after Q1, by original subject and attractor number. Counts are matched/attempted inputs. Both means exclude unmatched inputs; the case label is retained when donor and recipient are reversed.
Matched inputs
All-attempt gap
Matched gap
Rescued
Lost
Setting
Sel.
α≤2
α≤4
α≤2
α≤4
original
rescued
median α
α≤2→4
9B SVA
Q1
478/504
498/504
0.076
0.047
0.028
0.392
3.03
11 → 6
Llama-128K
Q1
92/124 †
116/124
0.270
0.193
0.130
0.241
2.53
13 → 7
Llama-128K
Q2
91/124 †
117/124
0.302
0.234
0.174
0.274
2.39
14 → 9
Llama-32K
Q1
75/106 †
92/106 †
0.197
0.144
0.041
0.087
2.38
10 → 10
Llama-32K
Q2
76/106 †
94/106
0.205
0.173
0.051
0.112
2.47
10 → 10
Appendix
Table 10: Post hoc search-range sensitivity for unmatched candidate searches. Matched counts use the original denominators; † marks matched subsets that do not meet the original per-case coverage rule. Gaps are pooled mean absolute effect gaps from the encoder-based reference intervention E (logits). All-attempt gaps retain each unmatched search at its best attempt. The matched-gap columns compare inputs matched within α≤2 with inputs first matched by the continuation. Lost counts are encoder transfers lost over the full cohort. Search records overlap across Q1 and Q2 and are not independent inputs.
Figure 6: Candidate residual gaps across answer-excluded KL budgets. Small mean gaps persist on both changing and fixed matched subsets. E is the encoder-based reference intervention. Both reference and candidate patches are rescaled to each budget. Filled points use each budget’s joint E /candidate matches (counts out of 346 in 9B, 64 in 2B). Open rings and dashed lines use the all-six-budget intersection; superposed symbols indicate overlapping means. Both axes are shared, but weighting differs: pooled in 9B, case/block-balanced in 2B. The 2B coverage failures are labeled; 9B counts are descriptive. Lines connect measured means, without uncertainty bands or an equivalence claim. Exact values and coverage are in Appendix D.3 .
Figure 7: Candidate effect-gap reductions under shared-coefficient and per-input calibration. Gaps are measured from the encoder-based reference intervention E . Shared coefficients improve means while worsening many individual comparisons. Negative reductions denote larger gaps. Dots at zero mark the shared-coefficient worsening fraction. The curves weight inputs equally, retaining 11 failed per-input Q1 attempts in 9B. Table 13 reports balanced means. Beyond the displayed 3-logit limit lie 2 shared-coefficient and 40 per-input values in 9B, none in 2B (full curves: Appendix D.4 , Figure 8 ). Selection protocols differ between panels. Shared-coefficient selection holds out evaluation inputs, which were exposed in earlier analyses. Per-input Q1 uses each evaluation input’s outputs. Curves are descriptive, without confidence intervals.
Figure 8: Full-range counterpart to Figure 7 . The wider axis reveals the positive tail clipped in that figure, with unchanged cohorts and equal input weights. Reductions are in logits; negative values denote larger gaps from the encoder-based reference intervention E , and dots at zero mark the shared-coefficient worsening fraction. All 11 failed 9B Q1 searches remain included. The panels retain their distinct selection protocols, and Q1 uses additional evaluation-input information. Curves are descriptive, without confidence intervals.
9B: pooled
2B: balanced
q
Matched
Gap
Matched
Gap
0.25
339/346
0.074
64/64
0.019
0.50
343/346
0.056
64/64
0.017
0.75
343/346
0.043
64/64
0.019
1.00
334/346
0.037
64/64
0.020
1.25
303/346
0.038
56/64
0.027
Appendix
Table 11: Candidate joint matching coverage and absolute effect gaps (logit) at each answer-excluded KL budget. Means condition on both patches matching: pooled inputs in 9B and case/block-balanced in 2B. The 9B panel is the global evaluation subset, not all 504 lexical inputs. High-budget coverage failures prevent extending these subset means to the full panels.
q
NNLS matched
Gap
Candidate
0.25
62/62
0.00782
0.01927
0.50
62/62
0.00798
0.01679
0.75
62/62
0.00986
0.01902
1.00
62/62
0.00821
0.01977
1.25
62/62
0.01217
0.04455
1.50
50/62
0.01225
0.13646
Appendix
Table 12: Additional 2B budget accounting. NNLS gaps condition on joint matching. The candidate column retains all 64 inputs, using nearest measured attempts for failed matches; its larger-budget entries are not matched comparisons. All gaps are balanced mean absolute differences in logits.
Balanced mean absolute gap (logit)
Setting
Before
Shared
Per-input
9B ( n=346 )
0.876
0.704
0.046
2B ( n=64 )
0.561
0.313
0.015
Appendix
Table 13: Balanced candidate effect gaps (logit) on the same inputs within each setting. Both selectors return α=1.25 ; every 2B fold selects that value. Per-input Q1 uses each evaluation input’s reference and separate search, including 11 failed attempts in 9B. It is an information-richer diagnostic, not a same-budget comparator.
Pooled mean absolute gap (logit)
Config. / code
Selector
Matched / attempted
Matched only
All attempts
Coverage
Before
After
Before
Final
128K / NNLS
Q1
188/188
0.166
0.016
0.166
0.016
Pass
Q2
188/188
0.166
0.023
0.166
0.023
Pass
32K / NNLS
Q1
112/112
0.159
0.012
0.159
0.012
Pass
Q2
112/112
0.159
0.014
0.159
0.014
Pass
Appendix
Table 14: Earlier Llama cohorts: pooled mean absolute effect gaps from the encoder-based reference intervention E in logits. Matching counts use each code’s own qualified cohort. Before/after columns compare identical inputs within a row; matched membership can differ between selectors. All attempts retain failed searches at their best measured point, without calling them calibrated. Coverage requires the prespecified per-case retention and support conditions. These descriptive means are distinct from the original case/block-balanced estimates.
Intervention
Inputs
Balanced gap
Pooled gap
Transfers
Lost
Gained
Reference E
304
0.000
0.000
175
0
0
Unscaled NNLS
304
0.213
0.207
181
3
9
Total-norm matching
304
0.181
0.176
165
11
1
Positionwise norm matching
304
0.134
0.130
167
10
2
Full-output KL (Q1)
304
0.021
0.020
175
0
0
Follow-up after inspection of the confirmation results
Appendix
Table 15: New-material Llama-128K controls on common inputs. Q1 preserves the transfer decisions of the encoder-based reference intervention E and leaves the smallest observed mean gap among these controls. Every measured matching control matches all 304 inputs. Gaps are in logits; balanced values equally weight cases and blocks within case, whereas pooled values weight inputs equally. Transfers and losses/gains relative to E are unweighted counts. The separated Q2 row is a follow-up after inspection of the confirmation, on the same inputs. This cohort remains separate from the earlier 188-input panel.
Case
n
Contrast
Singular / singular
65
0.195
Singular / plural
87
0.122
Plural / singular
65
0.185
Plural / plural
87
0.138
Appendix
Table 16: New-material confirmation by recipient subject/attractor number; n counts inputs, all matched by both primary controls. The mean total-norm minus Q1 gap is positive in every case. Contrasts are in logits, with equal block weights within each case.
Effect-gap contrast
Mean
95% interval
Unscaled NNLS − Q2
0.184
[0.149,0.223]
Q2 − Q1
0.008
[0.005,0.011]
Total norm − Q2
0.152
[0.124,0.183]
Appendix
Table 17: Q2 follow-up contrasts on common Llama inputs. Answer exclusion retains a reduction but leaves a larger residual gap than Q1. All 304 inputs match the compared controls. Estimates and pointwise 95% block-bootstrap intervals are in logits, with equal case weights and equal block weights within case.
Comparison
Evidence status
Inferential scope
9B lexical candidate
Prespecified primary; effects unmeasured when specified
Coverage passes; planned precision unmet
2B and Llama SVA extensions
Specified after group qualification, before alternative effects
2B matches fully; Llama candidate coverage fails
Gender
Selected group validated before code comparison
Partial extension; Q2 reverses the NNLS pattern
Single-scale controls
Prespecified follow-ups on exposed panels
Paired comparisons, with unavailable intervals reported
New Llama-128K NNLS panel
Materials and analysis specified before tokenization or group effects
Prospective material confirmation; all support, coverage and precision requirements pass
Q2 on the new Llama panel
Specified after inspection of the original confirmation results
Follow-up reduction; larger residual gap and transfer losses than Q1 on the same inputs
Appendix
Table 18: Evidence status and scope. Exposition starts with practical NNLS comparisons without replacing the prespecified primary endpoint. The new Llama panel provides prospective material confirmation separately from exposed-panel follow-ups.
Sparse autoencoder (SAE) features are increasingly used to steer language models, but feature steering is rarely clean: the same intervention can behave inconsistently across contexts and perturb unrelated features. We introduce a pre-intervention screening framework for forecasting SAE steering side effects from feature statistics computed before steering. We operationalize side effects along two axes of steering modularity, effect stability and collateral spread, and evaluate GPT-2-small, Pythia-70M-deduped, Gemma-2-2B, and Llama-3.1-8B across ReLU, JumpReLU, and TopK SAE dictionaries. Across these settings, decoder geometry, activation statistics, co-activation structure, and direct-logit footprint predict steering modularity better than frequency-only and activation-magnitude baselines. The signal is strongest in GPT-2-small, Pythia-70M, and Llama-3.1-8B, where it survives residualization against magnitude-related confounds, and weaker in Gemma-2-2B. Held-out screening shows that ranking unseen features by predicted cleanliness can select features that steer more cleanly on fresh contexts, but the successful axis varies by setting: GPT-2 improves most cleanly, Pythia improves mainly on stability, Llama mainly on collateral, and Gemma only partially. A controlled Llama Scope width comparison shows that the predictive signal persists under a 32K-to-128K dictionary-width change, although the screening payoff becomes less stable. Overall, SAE steering side effects are predictable in advance, but the useful predictor signature and transferred modularity axis are model- and dictionary-setting dependent.
Sparse Autoencoders (SAEs) have been seen as a promising avenue for exploring the internals of Large Language Models (LLMs) and for steering model output generation. When AxBench - a model steering benchmark - was introduced in Wu et al. (2025), SAEs did not seem to live up to their original hype due to poor steering performance relative to a set of simple baselines. This work serves as a partial rebuttal for Sparse Autoencoders and suggests that the results of Wu et al. (2025) did not do them full justice. We find that Sparse Autoencoders can, in fact, perform close to on par with the reference LoRA performance on the AxBench benchmark, when features are selected and labelled with our supervised pipeline. We also find that our pipeline selects features that are surprisingly causal of their identified labels when using only its interpretability-based components. Lastly, we present evidence that high sparsity (low l0) may not be crucial for successful steering based on interpretability, which is in contrast to the earlier findings in Wang et al. (2025).
Sparse autoencoders are meant to name the things a language model computes, and the usual way to check that a latent matters is to switch it off and see what changes. But a latent fires at many tokens, and the effect has to be measured at one of them. The convention is to measure where the latent fires hardest. That choice is almost never reported, and it is not made by the experimenter: it is made by the dictionary under evaluation. Change the dictionary and the measurement moves to a different token. We show this is not a detail. Take two sparse autoencoders released by Google for the same model and match their latents by decoder similarity: even among the pairs the two dictionaries encode almost identically, they pick different tokens for a large share of them. Two dictionaries compared under the usual protocol are therefore very often compared at different places. To separate the convention from the dictionaries we train six autoencoders from one initialisation, differing only in fitting choices, so that a latent means the same thing in each. Most of the variance such a comparison reads as "these dictionaries disagree about this latent" turns out to be the position instead: it falls from 7.6% and 11.9% of variance to near zero once every dictionary is measured at the same token. More evaluation data does not rescue it. Across a sixteenfold range of corpus sizes the dictionaries agree less about where to measure, not more, so the problem grows with scale. The correction is one line of evaluation code. We give the protocol an ablation-based causal number must report to be comparable across papers, and an audit of five published papers against it. In short: a causal number reported without its position describes the token it was taken at as much as the latent it was taken from.