Readout Scaling Shapes Sparse Feature Recovery

Brian Xiao; Sarah Wilson

cs.LGOct 13, 2026

Readout Scaling Shapes Sparse Feature Recovery

Authors: Brian Xiao, Sarah Wilson

Organizations: University of Waterloo · Columbia University

Abstract

Readout scaling changes what a sparse autoencoder (SAE) learns, but the benefit of removing it depends on the data. We use readout-canonicalized decision-space (RC-DS) coordinates to remove singular-value gains while preserving retained decision information. At a fixed readout condition number, changing the generating distribution reverses which training coordinates recover known factors best. In our controlled coordinate-matching study, recovery is highest when training restores the space in which the factors were generated. When factors are generated before readout scaling, RC-DS increases mean sparse-factor recovery from 0.506 to 0.913 at condition number 100, compared with centered-logit training under the same budget. An exact wrapper reproduces RC-DS training to numerical precision, isolating training geometry rather than additional information. In neural policies, RC-DS improves mean logit reconstruction under the tested budgets. On three fixed language-model checkpoints, RC-DS has higher mean gain-free reconstruction in all 18 tested task-sparsity settings, while centered-logit SAEs reconstruct output logits better in 17. Small-label-head controls show that longer training reduces the reconstruction gap from 0.0198 to 0.0016. Together, these results show how readout scaling, the data distribution, and training shape sparse learning. RC-DS provides an information-preserving method for studying and controlling the readout's contribution. Separate label-detection and steering tests distinguish these reconstruction effects from task-specific feature usefulness.

Figures & tables

Panel of Figure 1 — Figure 1: The same retained information, with different reconstruction weights. In a two-dimensional retained singular basis, the readout multiplies one direction by 100 and the other by 1. An error $e=(e_{1},e_{2})$ is consequently weighted by $10{,}000:1$ in logit squared error. RC-DS removes these gains. The orthonormal map into output space is omitted because it preserves these norms.

Input or training control	Explanation tested
Centered logits, $U\Sigma q$	Keep the readout’s scaling.
Hidden states, $h-\mu$	Keep all hidden directions, including those invisible to the readout.
Compact logits, $\Sigma q$	Keep the gains but omit the map $U$ into output space.
Compact RC-DS, $q$	Remove the gains and omit $U$ .
Coordinate standardization	Rescale each input coordinate by its standard deviation.
PCA / ZCA whitening	Rescale using both variances and correlations in the data.

Table 1: Baselines and what they test. Reconstruction comparisons use common logit and gain-free metrics. The

q

coordinates avoid storing a vocabulary-wide input in the full-readout study. The exact wrapper is tested on synthetic data.

Figure 2: Changing the generating geometry changes which training coordinates recover factors best, with the readout condition number fixed at 100. Rows apply $\Sigma^{g}$ to the base sparse signal, and columns remove $\Sigma^{t}$ before training. Scores average twelve paired seeds after 40 epochs. The diagonal returns the same training inputs, so its equal scores are a construction check, not independent replications. The off-diagonal scores measure the effect of mismatched scaling.

Figure 3: The case where factors are generated in the unscaled hidden space. Unequal readout scaling reduces mean recovery under ordinary logit training in this generator. Results average seeds 200–219 after 60 epochs. RC-DS (

\alpha=0

) receives the same inputs at every condition number. Noise added before scaling is transformed with the signal, so the methods still receive the same information.

Figure 4: Training controls for the case with factors generated before readout scaling. RC-DS and the exact wrapper both score 0.8176, versus 0.6284 when only the loss is reweighted and 0.4841 with ordinary scaled-input MSE. The exact wrapper changes the input, output, and loss together. Means use seeds 400–411, 2,048 samples, 40 epochs, and

\kappa=100

Figure 5: Fitting-data geometry predicts opposite recovery effects at the same readout condition number. Left: all 36 frozen forecasts, covering twelve settings and three seeds. Right: two matched settings with condition number 100, two active factors, and zero construction correlation. Bars average three seeds, shown as dots. Moving the generating factors reverses the RC-DS–logit ordering. Whitening is a separate, strong control.

Figure 6: Changing the reconstruction target changes the preferred coordinates. The same fitted SAEs are scored before scaling (left) and on output logits (right). Positive values favor RC-DS over centered-logit training. Each point averages three paired SAE seeds for one dataset and sparsity level. Vertical lines span the three paired differences, not confidence intervals.

K=8

is to the left and

K=32

to the right of each model label. The 18 points per panel summarize the nine model–dataset pairs at two sparsity levels, not independent model replications. The right panel shows the corresponding output-score trade-off.

Model	Gain-free EV vs. logits	Gain-free EV vs. hidden	Logit EV vs. logits	Logit EV vs. hidden
Qwen2.5	$+0.0216\pm 0.0008$	$+0.0043\pm 0.0019$	$-0.0058\pm 0.0006$	$+0.0052\pm 0.0019$
Qwen3	$+0.0090\pm 0.0012$	$+0.0073\pm 0.0011$	$-0.0033\pm 0.0010$	$+0.0068\pm 0.0008$
SmolLM2	$+0.0566\pm 0.0025$	$+0.0022\pm 0.0006$	$-0.0272\pm 0.0011$	$+0.0090\pm 0.0017$

Table 2: RC-DS reconstruction differences from two baselines, reported as mean

\pm

sample standard deviation across three paired seed summaries. For each checkpoint and seed, we first average its six task–sparsity differences. We then summarize the three seed averages, rather than treating 18 fits as independent observations. Positive values favor RC-DS. “Hidden” is the ordinary hidden-state SAE. The spread is conditional on the fixed checkpoint, texts, and training budget, not a confidence interval over models. Qwen2.5 supplied development data. Qwen3 and SmolLM2 were reserved for the semantic study.

Appendix figures & tables37 assets

Supplementary material from the paper’s appendix.

Appendix

Study	Models and data	Budget and seeds	Question tested	Details
Gain and noise sweep	Synthetic, 16 factors in 8 dimensions, 4,096 samples	60 epochs, 20 seeds (200–219)	How does unequal scaling affect recovery when factors are generated before scaling?	B , C
Approximate inverse loss	Separate synthetic samples, same dimensions and sample count as the sweep	60 epochs, 20 different seeds (300–319)	How much of the recovery difference does changing only the loss remove?	B , C
Coordinate matching and attribution	Synthetic, 2,048 samples per run	40 epochs, 12 seeds per study, separate seed sets for four controls	Separate generating coordinates, exact wrapping, alternative training, and noise effects.	B , C
Exact-wrapper check	Synthetic, condition numbers 3, 30, and 300	Short optimizer trajectories, 3 seeds per condition (9 cases)	Do wrapped activations, gradients, parameters, and reconstructions match numerically?	A
Bargaining	OpenSpiel, 800 fitting and 400 evaluation episodes per policy	Policy: 3,000 episodes. SAE: 15 epochs. 3 development and 3 reused evaluation policy seeds	Compare logit reconstruction and decoder-family steering with direct and random controls.	B , D , H
Goofspiel	OpenSpiel, 7 cards, 400 fitting and 200 evaluation episodes per policy	Policy: 5,000 episodes. SAE: 125 epochs. 3 development and 3 evaluation policy seeds	Test reconstruction and decoder-family steering on a second game.	B , D , H

Appendix

Table 3: Study overview. Data sizes describe fitting and evaluation roles, not independent model replications. The final column links to supporting appendices.

Setting	Bargaining	Goofspiel
Development policy seeds	710–712	920–922
Evaluation policy seeds	810–812	1120–1122
Policy-training episodes	3,000	5,000
Hidden layers	$128,128$	$128,128$
SAE fit/evaluation episodes	$800/400$	$400/200$
Dictionary size / Top- $K$	$192/8$	$192/8$

Appendix

Table 4: Policy settings fixed before evaluation. Each policy is tested with no readout scaling, square-root scaling, and full scaling.

Noise	$\alpha=0$	$\alpha=1/2$	$\alpha=1$	Difference [95% CI]
None	0.9134	0.7435	0.5063	0.4070 [0.3702, 0.4446]
Before gain	0.9135	0.7464	0.5101	0.4034 [0.3627, 0.4445]
After gain	0.9135	0.7342	0.4876	0.4259 [0.3958, 0.4556]

Appendix

Table 5: Factor recovery at condition number 100 under the invertibility and noise-placement controls, using seeds 200–219, 4,096 samples, and 60 epochs. Intervals are paired 95% bootstrap intervals for

\alpha=0

minus

\alpha=1

. Means and differences are computed at full precision and rounded separately.

Policy	$\alpha=0$	$\alpha=1/2$	$\alpha=1$
810	0.9108	0.9107	0.8739
811	0.9207	0.9167	0.9009
812	0.9116	0.8774	0.8657
Macro mean	0.9144	0.9016	0.8802

Appendix

Table 6: Bargaining reconstruction by policy. All values are evaluation centered-logit EV.

Coordinates	EV, 60 epochs	EV, 1,000	Excess, 1,000	Stable fits
Hidden states	0.7735	0.8025	$+8.42\%$	5/5
Centered logits	0.6066	0.9639	$+7.14\%$	0/5
RC-DS	0.6264	0.9655	$+0.15\%$	0/5
ZCA-whitened logits	0.7041	0.9690	$+5.07\%$	0/5

Appendix

Table 7: Qwen coordinate comparison on reused development data. EV is measured in common centered-logit space. Excess is learned minus strongest control gain, divided by oracle gain, after 1,000 epochs. All entries average the same five seeds.

Figure 7: Training-budget sensitivity on the Qwen2.5 label head, using the same five paired seeds and three parameter-matched action-space arms. Left: fitting histories normalized within each fit to epoch 60. Thin curves show individual seeds and thick curves their means. Right: common-logit development EV at the two measured endpoints. Faint points show every seed and large markers show means. Connecting lines do not imply intermediate measurements. The extension reaches its cap, not convergence.

Encoder / learning rate	Stable fits	Accurate fits	Both
Copy / constant	2/12	12/12	2/12
Copy / decay	10/12	3/12	3/12
Independent / constant	9/12	12/12	9/12
Independent / decay	11/12	11/12	10/12

Appendix

Table 8: The 48-fit training audit. No recipe satisfies both conditions for all twelve fits, so no recipe is selected and the twenty reserved-seed confirmation fits are not run.

Coordinates	Copy MSE	Independent MSE	Copy EV	Independent EV
Hidden states	0.3774	0.3058	0.8002	0.8656
Centered logits	0.01069	0.006251	0.9894	0.9937
RC-DS	0.01425	0.006699	0.9865	0.9935
Whitened logits	0.01463	0.01246	0.9852	0.9873

Appendix

Table 9: Effect of learning encoder weights independently at a constant learning rate. MSE is measured on training data in each method’s normalized input coordinates. Compare MSE within rows, not between coordinate systems. EV uses the common logit space, also on fitting data.

Model	$K$	Logits	RC-DS	Std.	ZCA	PCA	Hidden
Qwen2.5-0.5B	1	0.9203	0.9175	0.9178	0.8560	0.8425	0.7010
Qwen2.5-0.5B	2	0.9437	0.9398	0.9386	0.9175	0.9210	0.7491
Qwen2.5-0.5B	4	0.9920	0.9926	0.9917	0.9866	0.9839	0.8120
Qwen3-0.6B-Base	1	0.9228	0.9190	0.9161	0.8540	0.8397	0.7705
Qwen3-0.6B-Base	2	0.9409	0.9344	0.9323	0.9142	0.9157	0.7987
Qwen3-0.6B-Base	4	0.9918	0.9919	0.9916	0.9838	0.9819	0.8440

Appendix

Table 10: Centered-logit EV on 360 fresh prompts at 6,000 updates. Means of all five SAE seeds. Hidden is a larger, unmatched reference.

Model	$K$	Logits	RC-DS	Std.	ZCA	PCA	Hidden
Qwen2.5-0.5B	1	0.9090	0.9126	0.9055	0.8537	0.8410	0.6951
Qwen2.5-0.5B	2	0.9372	0.9373	0.9313	0.9169	0.9201	0.7448
Qwen2.5-0.5B	4	0.9910	0.9923	0.9905	0.9865	0.9838	0.8084
Qwen3-0.6B-Base	1	0.9176	0.9182	0.9070	0.8539	0.8401	0.7691
Qwen3-0.6B-Base	2	0.9371	0.9335	0.9262	0.9144	0.9158	0.7985
Qwen3-0.6B-Base	4	0.9910	0.9916	0.9907	0.9837	0.9818	0.8446

Appendix

Table 11: Gain-free EV on 360 fresh prompts at 6,000 updates. Means of all five SAE seeds. Hidden is a larger, unmatched reference.

Figure 8: Logit EV on new prompts versus SAE training time at

K=4

. Points show the four checkpoints selected before evaluation. Thin lines show each seed and thick lines show means. The hidden-state SAE is larger. Tuning adds to the reported time, and Qwen3 reuses Qwen2.5’s rates. Lines connect measurements without implying scores at intermediate times.

Figure 9: Gain-free reconstruction for the same fresh data, fitted models, and sparsity levels as the common-logit comparison. These metrics score the same residuals with different weighting. Neither identifies ground-truth sparse factors in the language models.

Figure 10: Common fitting-quality curves for every method at

K=4

. Upper panels score logit reconstruction and lower panels score gain-free reconstruction. Thin lines show individual seeds and thick lines show their means. Only the four saved checkpoints are measured. Joining them does not supply intermediate observations. The hidden-state reference has more parameters, so an update is not equal compute across all methods.

Model	Target	Hits	$n$	$\Delta T$	$\Delta F$	$\Delta E$	$\Delta$ AUC
Qwen2.5	0.90	5/5	5	+80	+0.0127	+0.0135	+0.0122
Qwen2.5	0.95	5/5	5	+0	+0.0004	+0.0014	+0.0029
Qwen2.5	0.98	5/5	5	+0	+0.0007	+0.0017	-0.0014
Qwen2.5	0.99	5/5	5	+800	+0.0004	+0.0012	+0.0036
Qwen3	0.90	5/5	5	+0	-0.0002	-0.0011	+0.0055
Qwen3	0.95	5/5	5	+0	-0.0002	-0.0011	+0.0055

Appendix

Table 12: Logit fitting-quality thresholds at

K=4

. Hits count RC-DS/logit fits out of five seeds.

n

counts seeds for which both methods reach the target at a saved checkpoint. Differences are RC-DS minus logits over those pairs only.

\Delta T

is updates,

\Delta F

is attained fitting EV, and

\Delta E

is held-out EV in the same metric. AUC is held-out single-feature macro AUROC. Nonzero

\Delta F

shows that threshold matching is not exact loss matching.

Model	Target	Hits	$n$	$\Delta T$	$\Delta F$	$\Delta E$	$\Delta$ AUC
Qwen2.5	0.90	5/5	5	+0	+0.0025	+0.0037	+0.0029
Qwen2.5	0.95	5/5	5	+0	+0.0025	+0.0037	+0.0029
Qwen2.5	0.98	5/5	5	+0	+0.0018	+0.0027	-0.0014
Qwen2.5	0.99	5/5	5	+0	+0.0007	+0.0014	+0.0024
Qwen3	0.90	5/5	5	+0	+0.0013	+0.0002	+0.0055
Qwen3	0.95	5/5	5	+0	+0.0013	+0.0002	+0.0055

Appendix

Table 13: Gain-free fitting-quality thresholds at

K=4

. Hits count RC-DS/logit fits out of five seeds.

n

counts seeds for which both methods reach the target at a saved checkpoint. Differences are RC-DS minus logits over those pairs only.

\Delta T

is updates,

\Delta F

is attained fitting EV, and

\Delta E

is held-out EV in the same metric. AUC is held-out single-feature macro AUROC. Nonzero

\Delta F

shows that threshold matching is not exact loss matching.

Method	Updates	Training seconds	Dense work ( $10^{9}$ FLOPs)
Centered logits	36,000	18.21	42.47
RC-DS	36,000	18.11	42.47
Standardized	36,000	18.21	42.47
ZCA	36,000	18.48	42.47
PCA whitening	36,000	18.36	42.47
Hidden (larger)	36,000	67.19	6342

Appendix

Table 14: Original Qwen2.5 learning-rate search cost at

K=4

, six complete fits per method (three rates, two seeds). Qwen3 reuses these choices. Training-only seconds exclude extraction, preprocessing, evaluation, and serialization.

Figure 11: Factor recovery at condition number 100 as correlation and simultaneous activity change. Both learning rates are shown. Thin lines show individual seeds and thick lines show means. Recovery is signed cosine similarity after mapping decoder directions back to the known generating space.

Training method	Logit EV	Gain-free EV
RC-DS	0.7741	0.7928
Half gain	0.7816	0.7922
Centered logits	0.7835	0.7882
Hidden states	0.7725	0.7908
Standardized	0.7765	0.7802
PCA whitening	0.5138	0.5021

Appendix

Table 15: Qwen full-readout scoring reconstruction. Means use both SAE seeds at each arm’s tuning-selected learning rate. The metrics use the same predictions in different spaces. Neither is known-factor recovery.

Model	Mean excess	95% seed interval	Seeds above 5%
Qwen2.5-0.5B	$+7.73\%$	$[+4.37\%,+11.18\%]$	4/5

Appendix

Table 16: Frozen Qwen Emotion test. The seed-bootstrap interval and practical criterion count retain every seed. A positive learned margin does not isolate the coordinate choice.

Model	$K$	Mean difference	Seed range
Qwen2.5	1	$+0.00095$	$[-0.00455,+0.00688]$
Qwen2.5	4	$-0.00603$	$[-0.00834,-0.00170]$
Qwen3	1	$+0.00289$	$[+0.00162,+0.00357]$
Qwen3	4	$+0.00142$	$[-0.00519,+0.01019]$

Appendix

Table 17: RC-DS minus centered-logit semantic AUROC on 480 new texts. Seed ranges are conditional on each fixed checkpoint and text panel, not confidence intervals over models.

Setting	Centered logits	RC-DS	Difference
Bargaining	0.8802	0.9144	$+0.0342$
Goofspiel	0.7654	0.8637	$+0.0983$
Qwen label head, 60 epochs	0.6066	0.6264	$+0.0198$
Qwen label head, 1,000 epochs	0.9639	0.9655	$+0.0016$

Appendix

Table 18: Shared centered-logit EV. Game rows average three policies under their fixed budgets. Qwen rows average five paired SAE seeds on reused development data.

Checkpoint	Logit EV	RC-DS EV	Difference	95% seed interval
Qwen2.5-0.5B	0.9920	0.9926	$+0.00058$	$[-0.00011,+0.00128]$
Qwen3-0.6B-Base	0.9918	0.9919	$+0.00008$	$[-0.00106,+0.00141]$

Appendix

Table 19: RC-DS and centered-logit SAEs are nearly tied on new prompts at

K=4

. Intervals resample the five paired SAE seeds for each fixed model and the same 360 prompts. Both intervals include zero. They do not measure uncertainty across language models.

Figure 12: Reconstruction on new prompts with one, two, or four active features after 6,000 updates. Thin lines show all five seeds and thick lines show means. The larger hidden-state arm is a reference, not a parameter-matched baseline. Rates are tuned per method and

K

on Qwen2.5 and transferred unchanged to Qwen3.

Figure 13: All twelve reserved full-readout settings. Dots show paired seeds, diamonds their means, and crosses forecasts frozen before confirmation training. Horizontal lines span the three seed values and are not confidence intervals. Gray shading marks the declared

\pm 0.01

AUROC band. Each panel is one checkpoint, not six independent models.

Model	Task	$K$	Mean	Seed 1	Seed 2	Seed 3
Qwen2.5	Emotion	8	$+0.00047$	$-0.01294$	$+0.01430$	$+0.00004$
Qwen2.5	Emotion	32	$-0.00265$	$+0.00059$	$-0.00643$	$-0.00210$
Qwen2.5	AG News	8	$+0.03900$	$+0.04816$	$-0.00584$	$+0.07467$
Qwen2.5	AG News	32	$+0.04761$	$+0.05308$	$+0.04760$	$+0.04215$
Qwen2.5	IMDb	8	$-0.03294$	$-0.05240$	$-0.01532$	$-0.03109$
Qwen2.5	IMDb	32	$+0.01331$	$+0.02103$	$+0.01708$	$+0.00183$

Appendix

Table 20: All trained RC-DS minus centered-logit AUROC differences. Seeds 1–3 are 730201–730203. Qwen2.5 is development, not an additional reserved-model replication.

Model	Task	Raw	RC-DS, $K=8$	RC-DS, $K=32$
Qwen2.5	Emotion	0.5669	$0.5223\ [0.5128,0.5296]$	$0.5372\ [0.5349,0.5404]$
Qwen2.5	AG News	0.8493	$0.7053\ [0.6918,0.7130]$	$0.8273\ [0.8160,0.8351]$
Qwen2.5	IMDb	0.5650	$0.5143\ [0.5045,0.5326]$	$0.5648\ [0.5462,0.5792]$
Qwen3	Emotion	0.5958	$0.5236\ [0.5152,0.5289]$	$0.5428\ [0.5392,0.5501]$
Qwen3	AG News	0.8471	$0.6721\ [0.6585,0.6965]$	$0.7982\ [0.7862,0.8157]$
Qwen3	IMDb	0.5564	$0.5385\ [0.5227,0.5473]$	$0.5421\ [0.5330,0.5480]$

Appendix

Table 21: Raw singular coordinates versus trained RC-DS features on label detection. Each entry is macro-AUROC on the same scoring texts after feature and sign selection on separate texts. Trained entries show the mean and [minimum, maximum] over three SAE seeds. These are observed ranges, not confidence intervals. The same raw score is compared with both sparsity levels. Qwen2.5 supplies development data. The following tables retain all other coordinate methods and untrained controls.

Model	Task	$K$	Logits	RC-DS	Hidden	Std.	ZCA
Qwen2.5	Emotion	8	0.5219	0.5223	0.5128	0.5226	0.5072
Qwen2.5	Emotion	32	0.5398	0.5372	0.5408	0.5569	0.5133
Qwen2.5	AG News	8	0.6663	0.7053	0.7138	0.6235	0.5167
Qwen2.5	AG News	32	0.7797	0.8273	0.8313	0.7513	0.5346
Qwen2.5	IMDb	8	0.5472	0.5143	0.5227	0.5464	0.5008
Qwen2.5	IMDb	32	0.5515	0.5648	0.5589	0.5673	0.5106

Appendix

Table 22: Trained single-feature macro-AUROC, averaged over three selected SAE seeds. Std. denotes per-singular-coordinate standardization. Qwen2.5 is development data.

Model	Task	$K$	Logits	RC-DS	Hidden	Std.	ZCA
Qwen2.5	Emotion	8	0.5079	0.5062	0.5062	0.5105	0.5040
Qwen2.5	Emotion	32	0.5253	0.5227	0.5227	0.5297	0.5091
Qwen2.5	AG News	8	0.5384	0.5442	0.5442	0.5361	0.5183
Qwen2.5	AG News	32	0.6344	0.6472	0.6472	0.6293	0.5261
Qwen2.5	IMDb	8	0.5070	0.5084	0.5084	0.5143	0.5011
Qwen2.5	IMDb	32	0.5236	0.5212	0.5212	0.5333	0.5083

Appendix

Table 23: Untrained SAE control macro-AUROC. Feature selection uses the same separate texts as for trained SAEs. Each value averages the three paired initializations.

Model	Task	$K$	Logits	RC-DS	Hidden	Std.	ZCA
Qwen2.5	Emotion	–	0.5669	0.5669	0.5815	0.5669	0.5268
Qwen2.5	AG News	–	0.8493	0.8493	0.8580	0.8493	0.5324
Qwen2.5	IMDb	–	0.5650	0.5650	0.6886	0.5650	0.5354
Qwen3	Emotion	–	0.5958	0.5958	0.5841	0.5958	0.5165
Qwen3	AG News	–	0.8471	0.8471	0.8664	0.8471	0.5449
Qwen3	IMDb	–	0.5564	0.5564	0.6233	0.5564	0.5094

Appendix

Table 24: Raw-coordinate macro-AUROC with the same feature-selection and scoring procedure. These probes do not have an SAE seed or a Top-

K

setting. Logits, RC-DS, and Std. tie because positive per-coordinate scaling and constant shifts preserve rankings and AUROC.

Model	Task	$K$	Gain-free EV	Logit EV
Qwen2.5	Emotion	8	$+0.0353\ [+0.0318,+0.0398]$	$-0.0041\ [-0.0069,-0.0025]$
Qwen2.5	Emotion	32	$+0.0305\ [+0.0265,+0.0349]$	$-0.0055\ [-0.0083,-0.0027]$
Qwen2.5	AG News	8	$+0.0108\ [+0.0092,+0.0131]$	$-0.0046\ [-0.0058,-0.0039]$
Qwen2.5	AG News	32	$+0.0114\ [+0.0104,+0.0122]$	$-0.0067\ [-0.0072,-0.0065]$
Qwen2.5	IMDb	8	$+0.0242\ [+0.0216,+0.0261]$	$-0.0036\ [-0.0042,-0.0032]$
Qwen2.5	IMDb	32	$+0.0177\ [+0.0160,+0.0189]$	$-0.0101\ [-0.0106,-0.0095]$

Appendix

Table 25: Paired RC-DS minus centered-logit reconstruction differences. Each metric gives the mean and [minimum, maximum] across seeds 730201–730203, not a confidence interval. Positive values favor RC-DS. Tasks and sparsity levels repeat within each fixed checkpoint.

Model	Task	$K$	Gain-free EV	Logit EV
Qwen2.5	Emotion	8	$+0.0013\ [-0.0003,+0.0027]$	$+0.0018\ [-0.0017,+0.0041]$
Qwen2.5	Emotion	32	$+0.0048\ [+0.0003,+0.0098]$	$+0.0054\ [+0.0006,+0.0100]$
Qwen2.5	AG News	8	$+0.0064\ [+0.0050,+0.0078]$	$+0.0081\ [+0.0074,+0.0089]$
Qwen2.5	AG News	32	$+0.0080\ [+0.0062,+0.0090]$	$+0.0095\ [+0.0067,+0.0113]$
Qwen2.5	IMDb	8	$+0.0004\ [-0.0012,+0.0027]$	$+0.0006\ [-0.0001,+0.0013]$
Qwen2.5	IMDb	32	$+0.0050\ [+0.0011,+0.0075]$	$+0.0061\ [+0.0022,+0.0083]$

Appendix

Table 26: Paired RC-DS minus hidden-state reconstruction differences, with the same mean and observed seed-range convention as Table 25 .

Model	Task	$K$	Gain-free EV	Logit EV
Qwen2.5	Emotion	8	$+0.0644\ [+0.0582,+0.0726]$	$+0.0127\ [+0.0095,+0.0154]$
Qwen2.5	Emotion	32	$+0.0613\ [+0.0572,+0.0686]$	$+0.0143\ [+0.0108,+0.0200]$
Qwen2.5	AG News	8	$+0.0252\ [+0.0235,+0.0280]$	$-0.0007\ [-0.0017,+0.0001]$
Qwen2.5	AG News	32	$+0.0277\ [+0.0258,+0.0288]$	$+0.0022\ [+0.0009,+0.0036]$
Qwen2.5	IMDb	8	$+0.0777\ [+0.0685,+0.0865]$	$+0.0197\ [+0.0166,+0.0240]$
Qwen2.5	IMDb	32	$+0.0496\ [+0.0478,+0.0508]$	$+0.0093\ [+0.0084,+0.0105]$

Appendix

Table 27: Paired RC-DS minus standardized-coordinate reconstruction differences. Entries give the mean and observed range over three matched seeds.

Model	Task	$K$	Gain-free EV	Logit EV
Qwen2.5	Emotion	8	$+0.3025\ [+0.2962,+0.3110]$	$+0.2570\ [+0.2559,+0.2589]$
Qwen2.5	Emotion	32	$+0.4759\ [+0.4687,+0.4796]$	$+0.4301\ [+0.4240,+0.4366]$
Qwen2.5	AG News	8	$+0.3112\ [+0.3029,+0.3238]$	$+0.2811\ [+0.2720,+0.2915]$
Qwen2.5	AG News	32	$+0.3696\ [+0.3651,+0.3777]$	$+0.3429\ [+0.3350,+0.3483]$
Qwen2.5	IMDb	8	$+0.3131\ [+0.2944,+0.3226]$	$+0.2670\ [+0.2462,+0.2779]$
Qwen2.5	IMDb	32	$+0.4231\ [+0.4179,+0.4261]$	$+0.3855\ [+0.3806,+0.3901]$

Appendix

Table 28: Paired RC-DS minus ZCA reconstruction differences. Entries give the mean and observed range over three matched seeds.

Model	Task	$K$	Logits	RC-DS	Hidden	Std.	ZCA
Qwen2.5	Emotion	8	0.3930	0.3889	0.3870	0.3762	0.1319
Qwen2.5	Emotion	32	0.5207	0.5152	0.5098	0.5008	0.0851
Qwen2.5	AG News	8	0.6325	0.6279	0.6198	0.6285	0.3467
Qwen2.5	AG News	32	0.7002	0.6935	0.6840	0.6912	0.3506
Qwen2.5	IMDb	8	0.5403	0.5367	0.5361	0.5170	0.2697
Qwen2.5	IMDb	32	0.6167	0.6066	0.6006	0.5973	0.2212

Appendix

Table 29: Scoring-set explained variance in the same full centered-logit metric, averaging three selected SAE seeds. Higher is better.

Model	Task	$K$	Logits	RC-DS	Hidden	Std.	ZCA
Qwen2.5	Emotion	8	0.4269	0.4622	0.4609	0.3977	0.1596
Qwen2.5	Emotion	32	0.5700	0.6006	0.5958	0.5392	0.1247
Qwen2.5	AG News	8	0.6673	0.6781	0.6717	0.6529	0.3668
Qwen2.5	AG News	32	0.7383	0.7497	0.7416	0.7220	0.3800
Qwen2.5	IMDb	8	0.6086	0.6328	0.6324	0.5551	0.3196
Qwen2.5	IMDb	32	0.6916	0.7092	0.7042	0.6596	0.2861

Appendix

Table 30: Scoring-set explained variance after removing readout gains, averaging three selected SAE seeds. Higher is better. This weights errors differently from logit EV. Entries are rounded to four decimal places. Differences reported in the text are computed from unrounded means, then rounded separately.

Figure 14: Fitting curves for

K=8

. Lines average three selected learning-rate fits, one per SAE seed. Shading spans the seeds, not a confidence interval. Loss is native normalized fitting MSE, with a logarithmic vertical axis. Qwen2.5 is development data.

Figure 15: Fitting curves for

K=32

, using the same conventions as Figure 14 . Both candidate learning rates and all histories remain in the internal study records, including rates not selected for scoring.

Explore similar work

Decoder-Preserving Sparse Autoencoders: Which Readouts Survive Sparse Compression?
Jul 19, 2026Aniket DeshpandeModel Compression Activation Sparsity
Department of Physics, University of Illinois Urbana-Champaign
Expander Sparse Autoencoders: Parameter-Efficient Dictionaries for Mechanistic Interpretability
Jul 2, 2026Rodrigo Mendoza-SmithSparse Recovery Activation Sparsity
Independent Researcher.
From Geometric Recovery to Causal Validation: A Reproducible Audit of Sparse Autoencoder Features, from Superposition Geometry to Causal Inertness
Jul 13, 2026Mohamed Abdessalem BalFeature Superposition Neural Representation Geometry
Independent Researcher, Algiers, Algeria

Jul 19, 2026J/K move · Enter open · S save

Decoder-Preserving Sparse Autoencoders: Which Readouts Survive Sparse Compression?

Aniket Deshpande

Department of Physics, University of Illinois Urbana-Champaign

Model Compression Activation Sparsity Sparse Autoencoders

Open paper AskPDF arXiv

Sparse autoencoders (SAEs) compress model activations into sparse codes, but equal reconstruction error and sparsity can preserve different linearly decodable signals. We formalize this ambiguity as a matrix-valued distortion between optimal ridge-prediction operators and train decoder-preserving SAEs by combining this distortion with reconstruction loss. In a rank relaxation, an isotropic task prior saturates per-mode omission costs without changing PCA's ordering, whereas a structured prior can change which modes are retained. A controlled sparse experiment shows that a declared prior protects held-out combinations from its task subspace. On GPT-2 small block 8, DPSAE reduces held-out decoder distortion by 10.6--11.4% across three paired runs while matching reconstruction NMSE. The same checkpoints pass an average natural-text output-KL noninferiority test, but one matched Pythia pair shows no improvement in probes restricted to a few sparse features. These results show that reconstruction quality does not determine which refitted linear readouts survive sparse compression, and that readout preservation is distinct from learning cleaner benchmark concepts or preserving every frozen-model behavior.

Abstract

Explore similar work

Decoder-Preserving Sparse Autoencoders: Which Readouts Survive Sparse Compression?

Expander Sparse Autoencoders: Parameter-Efficient Dictionaries for Mechanistic Interpretability

From Geometric Recovery to Causal Validation: A Reproducible Audit of Sparse Autoencoder Features, from Superposition Geometry to Causal Inertness