Sparse Autoencoders (SAEs) are effective tools for mechanistic interpretability by decomposing neural representations into sparse and interpretable features. However, a single sparse representation in traditional SAEs is responsible for both reconstruction fidelity and interpretability, which are not guaranteed to align. To separate these objectives, we propose DualSAE, a modular framework that augments standard SAEs with a plug-in Residual Branch. A standard SAE serves as the Sparse Branch, learning a sparse code that contains the dominant semantics of the input. The added Residual Branch models the non-semantic information omitted by the Sparse Branch. We ensure that the contributions of the two branches towards the overall reconstruction have minimal overlap through a decorrelation loss. We also adopt a two-stage training strategy that first stabilizes sparse representation learning before coupling with the Residual Branch. Across diverse encoder-SAE configurations on two vision datasets and two language corpora, DualSAE achieves mean relative improvements of approximately 29% in reconstruction fidelity and 34% in interpretability (measured by monosemanticity score). Moreover, with more interpretable representations, DualSAE yields consistent performance gains in the downstream applications of topic modeling, hypothesis generation, and LLM steering. Qualitative analysis further suggests that the Sparse Branch indeed preserves dominant semantic content, whereas the Residual Branch captures non-semantic information to improve reconstruction fidelity.
Figures & tables
Figure 1: Standard SAE and DualSAE comparison . (a) A standard SAE encodes both dominant semantic concepts and non-semantic information into a single sparse code, entangling interpretability with reconstruction. (b) Our DualSAE disentangles these objectives by adding a non-semantic Residual Branch, which separates interpretability from the overall reconstruction objective.
Figure 2: Left: Distribution of class purity for nearest neighbors retrieved using reconstruction x^ and residual r from BatchTopK SAE. Retrievals from x^ are substantially purer than those from r . Right: Example retrievals for the same query image using x^ (top) and r (bottom). The reconstruction x^ shows consistent semantics, while the residual is non-semantic.
Figure 3: DualSAE architecture . The Sparse Branch , modeled as a standard SAE, captures semantic information of the input feature x in the sparse latent z , then projects it back to the original dimension as xM . The augmented Residual Branch captures the non-semantic information r^ . The final reconstructed feature combines the outputs of both branches: x^=xM+r^ .
Figure 4: Effect of warm-up length , showing reconstruction R2 (left) and MonoScore (right) during ImageNet-1K training. Shorter warm-up accelerates reconstruction gains; moderate warm-up improves monosemanticity. Excessive warm-up delays reconstruction gains without further improving MonoScore.
Figure 5: Comparison of SAE and DualSAE on ImageNet-1K. Radar plots compare each SAE variant with its DualSAE counterpart across three encoders and six SAE variants using reconstruction R2 , MonoScore, class purity, KNN accuracy, and LP accuracy. Across 18 settings, DualSAE generally improves reconstruction fidelity and interpretability. Similar results on OpenImagesV7 are presented in Figure 15 .
Figure 6: Improved interpretability of sparse latents with DualSAE. Left: For two latent dimensions selected at the same MonoScore rank, Dual-BatchTopK shows more coherent activating samples from high to low activations compared to BatchTopK. Right: MonoScore (top) and class purity (bottom) across all latents, showing consistently higher interpretability for DualSAE.
Figure 7: t-SNE visualization of sparse representations. Sparse codes of 20 randomly selected ImageNet-1K classes show that Dual-JumpReLU (left) forms better-separated local semantic neighborhoods than JumpReLU (right).
Encoder
Activation
Par.
ImageNet-1K
OpenImagesV7
ΔR2
ΔMS
ΔPur.
ΔKNN
ΔLP
ΔR2
ΔMS
ΔPur.
ΔKNN
ΔLP
ViT
TopK
18M
-0.002
-0.004
-0.004
-0.003
-0.007
-0.006
-0.001
-0.001
-0.009
-0.001
BatchTopK
18M
-0.004
-0.003
-0.006
-0.009
-0.001
-0.003
-0.001
-0.002
-0.002
-0.008
JumpReLU
18M
-0.113
-0.246
-0.235
-0.003
-0.002
-0.251
-0.059
-0.086
-0.019
-0.034
CLIP
TopK
13M
-0.003
-0.003
-0.002
-0.001
-0.005
-0.010
-0.009
-0.001
-0.011
-0.008
BatchTopK
13M
-0.009
-0.001
-0.001
-0.001
-0.001
-0.001
-0.002
-0.007
-0.008
-0.001
Table 1: Parameter-matched SAE comparison. Each baseline uses a wider sparse dictionary so that its number of trainable parameters matches that of DualSAE. We report Δmetric=metricmatched SAE−metricDualSAE ; negative values favor DualSAE. DualSAE outperforms the parameter-matched baselines in every setting, indicating its gains are not due to additional parameters.
Figure 8: DualSAE vs. SAE with Residual Branch (BatchTopK on ImageNet-1K). SAE + Residual denotes a sequential baseline that freezes a pretrained SAE and trains only the Residual Branch, whereas DualSAE jointly optimizes both branches in its second training phase. Adding a Residual Branch improves R2 by 15.4–24.8% over the standard SAE. In contrast, DualSAE further improves MonoScore by 10.5–43.3% over the sequential baseline while maintaining comparable or higher R2 , showing the benefits of joint optimization beyond additional residual capacity.
Figure 9: Branch decomposition in DualSAE. (a) Joint t-SNE of the Sparse Branch output xM , Residual Branch output r^ , and reconstruction x^ for three ImageNet-1K classes: xM forms class-separated clusters, while r^ of all classes overlap. (b) KNN and LP top-1 accuracy on ImageNet-1K: xM far exceeds r^ in every setting, i.e . , the Sparse Branch carries the class semantics.
Figure 10: Effect of decorrelation loss on branch separation. For the same query image , we compare nearest-neighbor retrievals from the sparse reconstruction xM and residual r^ , with and without the decorrelation loss. The loss preserves coherent retrievals from xM while making r^ retrievals less similar to the query.
Figure 11: Effect of the expansion ratio (BatchTopK, ImageNet-1K). R2 (top) and MonoScore (bottom) of a standard SAE and DualSAE with expansion ratio of 4, 8, and 16: DualSAE is higher in both metrics at every ratio for all three encoders. Please refer to Figures 18 and 19 for the results of other settings.
Figure 12: Qualitative analysis on language representations. (a) “Time-related” latents learned by Dual-TopK and TopK. Dual-TopK responds more selectively to genuine “time-related” concepts , whereas TopK exhibits missed temporal concepts and activations on unrelated tokens . (b) Positive-sentiment steering. Compared with the unsteered generation and TopK steering, Dual-TopK produces a coherent flow containing clear positive-sentiment expressions.
Corpus
Method
Δ CE ↓
KL ↓
R2↑
MS ↑
OWT
JumpReLU
0.7412 0.0201
0.5498 0.0011
0.8994 0.1209
0.4995 0.1225
Dual-JumpReLU
0.1423 0.0097
0.0532 0.0101
0.9732 0.0464
0.6882 0.0743
TopK
0.5538 0.0193
0.5608 0.0201
0.9677 0.0559
0.4317 0.0541
Dual-TopK
0.0105 0.0022
0.0131 0.0016
0.9896 0.0180
0.6247 0.0236
BatchTopK
0.6587 0.0442
0.6661 0.0441
0.9432 0.0984
0.4982 0.1455
Dual-BatchTopK
0.2118 0.0229
0.3129 0.0206
0.9887 0.0196
0.9695 0.0078
Table 2: Language-model evaluation on OpenWebText (OWT) and GitHub code (GHC) at layer 12 of Pythia. Lower Δ CE and KL indicate better preservation of the original language-model behavior; higher R2 and MS indicate better reconstruction and monosemanticity. DualSAE improves all four metrics over its standard counterpart for every SAE variant on both corpora.
Figure 13: Image–topic assignment comparison (TopK on SigLIP2). Dominant topics from DualSAE and SAE : DualSAE’s are visually grounded, while SAE’s are unrelated.
Figure 14: Hypothesis generation for bull mastiff . Top latent of each model with its VLM hypothesis and five highest-activating images: JumpReLU’s mixes unrelated scenes, whereas Dual-JumpReLU’s is coherent and class-relevant.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Number of latents
8192
Training epochs
50
Batch size
2048
Learning rate
10−4
Adam ϵ
6.25×10−10
Random seeds
42, 43, 44
Appendix
Table 3: Common training hyperparameters. Unless otherwise specified, these settings are shared across SAE and DualSAE variants. TK = TopK.
BatchTopK activation, group fractions of 1/32, 1/16, 1/8, 1/4, and 17/32.
GATE
Learning rate of 5e–5, and no grad-norm clip.
CB
Concept set of VLG–CBM ImageNet-filtered, retain of 1024 latents, kcb of 64, and CB epochs/batch/lr: 25/2048/1e–4.
Appendix
Table 4: Model-specific hyperparameters. DualSAE uses the same Sparse Branch configuration as the corresponding SAE baseline, with an additional Residual Branch and decorrelation loss. TK = TopK, BTK = BatchTopK, JR = JumpReLU, MAT = MatryoshkaSAE, GATE = GatedSAE, CB = CB-SAE.
IN
OI
Enc.
SAE
R2
MS
Pur.
KNN
LP
R2
MS
Pur.
KNN
LP
ViT
TK
0.8081 ± 0.0008
0.2375 ± 0.0011
0.1271 ± 0.0007
0.8203 ± 0.0011
0.8230 ± 0.0006
0.7037 ± 0.0011
0.1283 ± 0.0007
0.2259 ± 0.0003
0.4675 ± 0.0011
0.4022 ± 0.0008
D-TK
0.9661 ± 0.0003
0.4874 ± 0.0008
0.5666 ± 0.0002
0.8226 ± 0.0008
0.8239 ± 0.0004
0.9611 ± 0.0006
0.2446 ± 0.0008
0.5028 ± 0.0001
0.4701 ± 0.0010
0.4049 ± 0.0005
BTK
0.7879 ± 0.0007
0.2450 ± 0.0009
0.1513 ± 0.0004
0.8207 ± 0.0005
0.8230 ± 0.0005
0.6782 ± 0.0012
0.1369 ± 0.0008
0.2569 ± 0.0004
0.4672 ± 0.0013
0.4050 ± 0.0009
D-BTK
0.9725 ± 0.0004
0.4775 ± 0.0005
0.5481 ± 0.0002
0.8228 ± 0.0003
0.8242 ± 0.0005
0.9593 ± 0.0006
0.2708 ± 0.0003
0.5284 ± 0.0002
0.4700 ± 0.0006
0.4035 ± 0.0003
JR
0.8020 ± 0.0007
0.2840 ± 0.0008
0.2391 ± 0.0006
0.8201 ± 0.0008
0.8221 ± 0.0007
0.7238 ± 0.0008
0.1075 ± 0.0009
0.2273 ± 0.0004
0.4675 ± 0.0006
0.4041 ± 0.0007
Appendix
Table 5: Comparison of SAE variants across vision encoders on ImageNet-1K (IN) and OpenImagesV7 (OI). DualSAE generally improves reconstruction ( R2 ), interpretability (MS, Pur.), and KNN over its standard counterpart across encoders, activations, and both datasets, with comparable or better LP. SL2 = SigLIP2. SAE abbreviations: TK = TopK, BTK = BatchTopK, JR = JumpReLU, MAT = MatryoshkaSAE, GATE = GatedSAE, and CB = CB-SAE. Prefix D- denotes the corresponding DualSAE variant; CB-DUAL denotes CB-SAE augmented with the DualSAE Residual Branch. MS denotes MonoScore; Pur. denotes binary purity; KNN and LP denote KNN- and linear-probe-based downstream performance, respectively. For IN, both are reported as top-1 accuracy; for OI, both are reported as micro-F1.
Figure 15: Comparison of SAE and DualSAE on OpenImagesV7. Radar plots compare each SAE variant with its DualSAE counterpart across three encoders and six SAE variants using reconstruction R2 , MonoScore, class purity, KNN accuracy, and LP accuracy. Across 18 settings, DualSAE generally improves reconstruction fidelity, monosemanticity, and downstream performance.
Figure 16: Input–reconstruction alignment. Dual-BatchTopK preserves the original embedding geometry better than BatchTopK, shown by the overlap between input x and reconstructed x^ .
Figure 17: Class information in each DualSAE branch on OpenImagesV7. KNN and LP micro-F1 of the Sparse Branch output xM and the Residual Branch output r^ (mean and standard deviation over three seeds). xM far exceeds r^ in all 18 settings, so the class information resides in the Sparse Branch. SL2: SigLIP2; TK, BTK, JR: TopK, BatchTopK, JumpReLU.
Enc.
SAE
IN
OI
ΔR2
ΔMS
ΔPur.
ΔKNN
ΔLP
ΔR2
ΔMS
ΔPur.
ΔKNN
ΔLP
ViT
TK
-0.014
-0.098
-0.230
-0.029
-0.175
-0.028
-0.033
-0.042
-0.019
-0.005
BTK
-0.021
-0.072
-0.179
-0.045
-0.001
-0.027
-0.029
-0.041
-0.077
-0.004
JR
-0.013
-0.232
-0.362
-0.004
-0.015
-0.016
-0.045
-0.040
-0.001
-0.006
CLIP
TK
-0.022
-0.051
-0.015
-0.022
-0.001
-0.037
-0.023
-0.038
-0.004
-0.001
BTK
-0.027
-0.075
-0.007
-0.056
-0.001
-0.046
-0.052
-0.021
-0.003
-0.001
Appendix
Table 8: Ablation of the decorrelation loss. We report the metric changes after removing Ldcor . Every entry is negative, indicating that removing the decorrelation loss degrades performance. In other words, Ldcor allows the Sparse Branch to learn more interpretable representations (higher MS and Purity) and better reconstruction (higher R2 , KNN, and LP). IN = ImageNet-1K, OI = OpenImagesV7, SL2 = SigLIP2, TK = TopK, BTK = BatchTopK, JR = JumpReLU.
Data
Enc.
SAE
Projector
R2
MS
Pur.
KNN
LP
IN
CLIP
TK
Default
0.9963 ± 0.0003
0.5604 ± 0.0015
0.4212 ± 0.0078
0.8157 ± 0.0002
0.8449 ± 0.0003
B256
0.9440 ± 0.0010
0.5612 ± 0.0038
0.2808 ± 0.0019
0.8153 ± 0.0004
0.8437 ± 0.0002
B512
0.9886 ± 0.0001
0.5851 ± 0.0028
0.3453 ± 0.0006
0.8157 ± 0.0001
0.8449 ± 0.0002
D3
0.9514 ± 0.0023
0.5622 ± 0.0048
0.2853 ± 0.0051
0.8148 ± 0.0002
0.8436 ± 0.0007
D4
0.9290 ± 0.0013
0.5585 ± 0.0039
0.2718 ± 0.0071
0.8141 ± 0.0009
0.8427 ± 0.0004
BTK
Default
0.9969 ± 0.0002
0.5821 ± 0.0055
0.4173 ± 0.0130
0.8157 ± 0.0003
0.8451 ± 0.0001
Appendix
Table 9: Projector ablation of the Residual Branch. Default: Linear(A,A)→GELU ; B h : Linear(A,h)→GELU→Linear(h,A) ; D3/D4: three-/four-layer MLP projectors with hidden width 512. The default single-layer projector gives the highest R2 and purity in every setting, while KNN and LP vary by less than 0.005 across projectors. IN = ImageNet-1K, OI = OpenImagesV7, SL2 = SigLIP2, TK = TopK, BTK = BatchTopK. MS denotes MonoScore; Pur. denotes binary purity; KNN and LP denote KNN- and linear-probe-based downstream performance. Bold marks the best mean per setting. Results are reported as mean ± standard deviation across three seeds.
Data
Enc.
SAE
Method
R2
MS
Pur.
KNN
LP
IN
CLIP
TK
Sequential Training
0.9983 ± 0.0001
0.5563 ± 0.0025
0.2746 ± 0.0049
0.8155 ± 0.0002
0.8452 ± 0.0000
Joint Training
0.9964 ± 0.0002
0.5603 ± 0.0007
0.4256 ± 0.0002
0.8155 ± 0.0002
0.8450 ± 0.0004
BTK
Sequential Training
0.9987 ± 0.0000
0.5206 ± 0.0149
0.2822 ± 0.0010
0.8155 ± 0.0002
0.8451 ± 0.0001
Joint Training
0.9972 ± 0.0004
0.5750 ± 0.0003
0.4287 ± 0.0004
0.8156 ± 0.0004
0.8451 ± 0.0003
SL2
TK
Sequential Training
0.9969 ± 0.0001
0.5596 ± 0.0702
0.2742 ± 0.0098
0.8517 ± 0.0000
0.8837 ± 0.0001
Joint Training
0.9944 ± 0.0005
0.6354 ± 0.0008
0.4036 ± 0.0008
0.8517 ± 0.0008
0.8830 ± 0.0005
Appendix
Table 10: Sequential Training vs. Joint Training. Sequential Training: a pre-trained SAE whose Sparse Branch is frozen while a Residual Branch is trained afterwards; Joint Training: both branches trained jointly. Sequential training recovers most of DualSAE’s reconstruction fidelity and downstream performance, but joint training gives higher MonoScore and purity in all 12 settings. IN = ImageNet-1K, OI = OpenImagesV7, SL2 = SigLIP2, TK = TopK, BTK = BatchTopK. MS denotes MonoScore; Pur. denotes binary purity; KNN and LP denote KNN- and linear-probe-based downstream performance (top-1 accuracy on ImageNet-1K, micro-F1 on OpenImagesV7). Bold marks the higher mean in each setting. Results are reported as mean ± standard deviation across three seeds.
Figure 18: Expansion-ratio ablation on ImageNet-1K. DualSAE has higher R2 , MS, purity, KNN, and LP than the standard SAE at every expansion ratio in all six settings. Each panel plots one metric (rows) against the expansion ratio m/d (number of latents over the encoder’s embedding dimension, log scale) for one encoder and activation (columns). Standard SAE: DualSAE’s Sparse Branch trained without the Residual Branch (as in Supplementary C.4 ), not the baselines of Table 5 . All panels in a row share the same y-axis. MS denotes MonoScore and Purity binary purity; KNN and LP are top-1 accuracy. All runs use k=64 and a single seed (42).
Figure 19: Expansion-ratio ablation on OpenImagesV7. DualSAE has higher R2 , purity, and KNN than the standard SAE in all 18 pairs and higher MS in 17; its LP is higher in 10 and at most 0.019 lower in the others. Each panel plots one metric (rows) against the expansion ratio m/d (number of latents over the encoder’s embedding dimension, log scale) for one encoder and activation (columns). Standard SAE: DualSAE’s Sparse Branch trained without the Residual Branch (as in Supplementary C.4 ), not the baselines of Table 5 . All panels in a row share the same y-axis. MS denotes MonoScore and Purity binary purity; KNN and LP are micro-F1. All runs use k=64 and a single seed (42).
Figure 20: Qualitative comparison of “person name” latents learned by Dual-BatchTopK and standard BatchTopK. Both latents respond strongly to person names . Dual-BatchTopK exhibits more consistent activation on complete person-name spans, whereas the standard BatchTopK latent also activates on unrelated or only partially relevant tokens , as highlighted in gray.
Figure 21: Qualitative comparison of “professional title” latents learned by Dual-JumpReLU and standard JumpReLU. Both latents respond to professional and organizational titles such as Chief , CEO , Chairman , and editor . Dual-JumpReLU activates more consistently on complete title-related spans, whereas the standard JumpReLU latent also responds to surrounding or semantically different tokens, highlighted in gray. The moderately activating examples further reveal this secondary behavior, suggesting that DualSAE yields a more semantically focused and less entangled latent representation.
Figure 22: Qualitative comparison of “end-of-text” latents learned by Dual-GatedSAE and standard GatedSAE. Both latents respond strongly to the <|endoftext|> token at boundaries between code fragments or files. Dual-GatedSAE exhibits a cleaner and more consistent association with the boundary token, whereas the standard GatedSAE latent also activates on surrounding punctuation and formatting-related tokens , highlighted in gray. The moderately activating examples further reveal this secondary behavior, suggesting that DualSAE yields a more semantically focused and less entangled boundary feature.
Figure 23: Qualitative comparison of “class attribute” latents learned by Dual-MatryoshkaSAE and standard MatryoshkaSAE. Both latents respond to HTML attributes, but Dual-MatryoshkaSAE is more selectively aligned with the class attribute across diverse contexts. In contrast, the standard MatryoshkaSAE latent also activates on other attributes such as href , rel , and type , highlighted in gray. The moderately activating examples show the same pattern, with Dual-MatryoshkaSAE continuing to focus on class while the baseline shifts toward other attribute types, suggesting a cleaner and less entangled semantic feature.
SAE / Enc.
IN
OI
K=50
K=100
K=200
K=300
K=500
K=50
K=100
K=200
K=300
K=500
Diversity D
TK / CLIP
+0.11
+0.10
+0.10
+0.11
+0.10
+0.05
+0.03
+0.02
+0.04
+0.05
TK / SL2
-0.01
+0.00
+0.03
+0.06
+0.12
+0.04
+0.03
+0.03
+0.05
+0.08
TK / ViT
+0.10
+0.08
+0.06
+0.06
+0.03
+0.05
+0.04
+0.03
+0.02
+0.03
BTK / CLIP
+0.17
+0.16
+0.16
+0.16
+0.17
+0.12
+0.09
+0.10
+0.10
+0.13
Appendix
Table 11: Topic modeling results on ImageNet-1K and OpenImagesV7. We compare SAE variants with different activation functions and encoders across topic counts K∈{50,100,200,300,500} . Diversity D evaluates topic non-redundancy, and intruder accuracy CI measures whether unrelated concepts can be identified from each topic. DualSAE increases topic diversity in 89 of the 90 configurations and improves intruder accuracy in most of them. IN = ImageNet-1K, OI = OpenImagesV7, SL2 = SigLIP2, TK = TopK, BTK = BatchTopK, JR = JumpReLU.
Figure 24: Image–topic assignments between DualSAE and traditional SAEs (one row per encoder/activation, K=100 ). Labels show each image’s dominant topic under DualSAE ( red ) and the activation-matched baseline ( blue ); topic names are LLM summaries of top topic words. DualSAE yields concrete, object-specific topics, while the baseline entangles objects with context or abstract style.
Enc.
SAE
IN
OI
Δ AUROC
Δ #Significant
Δ AUROC
Δ #Significant
CLIP
TK
+0.001
+0.11
+0.002
+0.31
BTK
-0.003
+0.12
-0.001
+0.07
JR
+0.011
+0.31
+0.010
+0.15
SL2
TK
+0.001
+0.01
+0.000
+0.39
BTK
+0.005
+0.52
-0.005
+0.25
Appendix
Table 12: Hypothesis-generation results for SAE variants across ImageNet-1K and OpenImagesV7. We report only the performance differences between the DualSAE and baseline models. Δ AUROC measures the change in predictive quality of generated hypotheses, while Δ #Significant measures the change in the average number of statistically significant hypotheses. DualSAE yields more significant hypotheses in 17 of the 18 configurations and higher AUROC in all six JumpReLU configurations; every AUROC deficit is at most 0.005. IN = ImageNet-1K, OI = OpenImagesV7, SL2 = SigLIP2, TK = TopK, BTK = BatchTopK, JR = JumpReLU.
Figure 25: Class-level hypotheses for the prayer rug and bull mastiff classes. We show the top-2 selected latents for the baseline SAE (red) and DualSAE (blue), each accompanied by its VLM-generated hypothesis and five highest-activating images. Baseline latents are often polysemantic or off-topic, whereas DualSAE produces coherent, class-relevant concepts.
Block 6, TK
Block 12, TK
Block 18, TK
Block 6, BTK
Data
Concept
Dual
Standard
Dual
Standard
Dual
Standard
Dual
Standard
AxBench score (0–2) ↑
AMZ
Positive
0.194 ± 0.085
0.118 ± 0.019
0.249 ± 0.031
0.139 ± 0.019
0.218 ± 0.013
0.136 ± 0.050
0.228 ± 0.031
0.107 ± 0.022
Negative
0.062 ± 0.023
0.041 ± 0.034
0.091 ± 0.026
0.053 ± 0.042
0.072 ± 0.010
0.037 ± 0.016
0.050 ± 0.015
0.028 ± 0.016
AGN
World
0.219 ± 0.037
0.083 ± 0.025
0.327 ± 0.032
0.088 ± 0.028
0.176 ± 0.063
0.085 ± 0.065
0.236 ± 0.035
0.158 ± 0.027
Sports
0.170 ± 0.037
0.040 ± 0.011
0.278 ± 0.118
0.167 ± 0.123
0.279 ± 0.073
0.114 ± 0.117
0.303 ± 0.013
0.185 ± 0.097
Appendix
Table 13: Quantitative LLM steering on Pythia (SAEs trained on OpenWebText). The unit-norm decoder direction of each selected latent is added to the residual stream at the block of its SAE, with a strength calibrated to a next-token KL of 0.2 nats. AxBench score: harmonic mean of the phi-4 concept, instruction, and fluency ratings; P(concept) : probability of the target concept from a BGE-based classifier. The standard SAE has a lower mean than DualSAE in all 80 comparisons and, after Benjamini–Hochberg correction, is significantly worse in 33 and significantly better in none. Std. = standard SAE, Dual = DualSAE, TK = TopK, BTK = BatchTopK, AMZ = Amazon reviews, AGN = AG News, GoE = GoEmotions. Bold marks the higher mean in each setting. Results are reported as mean ± standard deviation across three seeds.
Figure 26: Qualitative steering example for the “sadness” concept. Given the same initial prompt, the unsteered generation remains largely neutral, while steering with BatchTopK SAE does not reliably induce the target concept and instead shifts the continuation toward unrelated content. In contrast, steering with Dual-BatchTopK SAE produces a continuation containing multiple semantically consistent expressions of sadness , including sad , upset , and cry , demonstrating stronger concept-specific control.
Figure 27: Qualitative steering example for the “fear” concept. Given the same prompt, the unsteered continuation develops an unrelated narrative, while JumpReLU SAE steering does not clearly induce the target emotion and instead produces a largely neutral continuation. In contrast, steering with Dual-JumpReLU SAE introduces multiple fear-related expressions , including afraid and worried , producing a continuation more consistently aligned with the intended concept.
Figure 28: Qualitative steering example for the “sports” concept. Given the same prompt, the unsteered and GatedSAE continuations do not introduce sports-related content. In contrast, Dual-GatedSAE steering yields a fluent continuation containing multiple sport-related concepts , including the Yankees , series , homers , RBIs , and Red Sox , demonstrating more effective concept-specific steering while preserving generation quality.
Figure 29: Qualitative steering example for the “anger” concept. Given the same prompt, the unsteered generation remains positive and neutral, while MatryoshkaSAE steering produces a repetitive continuation without expressing the target emotion. In contrast, Dual-MatryoshkaSAE steering shifts the generation toward anger-related language , including sick , terrible , and hate , while maintaining a coherent continuation.