Diminishing Returns of Data in Sparse Autoencoders: Functional Behavior does not Guarantee Feature Reproducibility

Sambit Ghosh; Madelyn Mathai; R. Venkatesh Babu; Chirag Agarwal

cs.LGOct 14, 2026

Diminishing Returns of Data in Sparse Autoencoders: Functional Behavior does not Guarantee Feature Reproducibility

Authors: Sambit Ghosh, Madelyn Mathai, R. Venkatesh Babu, Chirag Agarwal

Organizations: IBM · Indian Institute of Science · University of Virginia

Abstract

Sparse autoencoders (SAEs) are typically trained on hundreds of millions to billions of activation tokens, yet little work has tested whether this scale is necessary. We study how much activation data SAEs need by training four SAE architectures on three language models with budgets ranging from 5M tokens to a 500M-token reference. We measure two notions of convergence: functional behavior on standard evaluations and one-to-one reproducibility of learned features. Functional data requirements are strongly metric-dependent: reconstruction often approaches the full-data reference at substantially smaller budgets, whereas feature reproducibility continues to improve with data. For Gemma-2-2B BatchTopK, the fraction of strictly matched features rises from 2% at 5M tokens to 29% at 50M. Data selection also matters: random sampling better preserves the original activation distribution and generally favors reconstruction, whereas diversity-based sampling frequently improves AutoInterp. A theoretical analysis separates statistical consistency of subsampled training from the stronger identifiability and optimization conditions required for one-to-one feature recovery. Our results show that SAE training can often be made far more data-efficient when budgets are matched to the goal, and feature-level claims are backed by explicit reproducibility checks, paving the way toward cheaper and more reliable SAE research.

Figures & tables

Figure 1: Functional data scaling for a BatchTopK SAE with

16

k latents. SAEBench metrics are reported for SAEs trained with {5M, 25M, 50M, 500M} activation tokens. While smaller activation budgets approaches the 500M reference for reconstruction, probing, AutoInterp, and intervention-metrics (e, h) exhibit heterogeneous, non-monotonic dependence on activation exposure.

Figure 2: Functional data scaling for a standard ReLU SAE with

65

k latents. SAEBench metrics are reported for SAEs trained with {5M, 25M, 50M, 500M} activation tokens. Reconstruction remains close to the

500

M reference at smaller activation budgets, whereas feature-level metrics show metric-dependent and frequently non-monotonic responses to increasing activation exposure.

Figure 3: Random sampling (RS) vs. FPS for a BatchTopK SAE with

16

k latents. At each activation budget, paired bars compare FPS and RS, while the green dashed line denotes the corresponding

500

M reference. Selection strategy induces metric-dependent tradeoffs: RS better preserves reconstruction and several intervention metrics, whereas FPS often improves AutoInterp.

Model	Selection	Coreset	$\boldsymbol{\Delta}$ mean norm (%)	KL (bits)	Mean-dir. cosine
			Selected vs. 500M stream
Gemma-2-2B	FPS	5M	$+28.8\pm 0.9$	2.28	$0.941\pm 0.003$
		25M	$+22.7\pm 0.3$	1.51	$0.954\pm 0.001$
		50M	$+18.9\pm 0.2$	1.11	$0.961\pm 0.001$
	Random	5M	$0.0\pm 0.4$	0.000	$1.000\pm 0.000$
		25M	$0.0\pm 0.2$	0.000	$1.000\pm 0.000$

Table 1: FPS reshapes the training-data distribution; random selection does not. * Random selection is distribution-neutral at every budget (KL

\leq 0.0003

bits); FPS-induced divergence grows with subset selectivity. Results for Qwen3-8B and Pythia-160M are in Table 24 in the appendix

Model	SAE	Budget	Counterpart (%)	Shared, $\tau\geq 0.7$ (%)
Gemma-2-2B	ReLU	5M	76.15	28.42
		25M	83.06	41.13
		50M	85.28	47.26
	BatchTopK	5M	66.61	2.20
		25M	80.15	16.94
		50M	83.47	28.86

Table 2: Counterpart agreement and strict feature correspondence. Enc.-dec. agree is the fraction of features for which independently computed encoder and decoder Hungarian matchings select the same

500

M reference latent. Shared additionally requires both encoder and decoder cosine similarities to exceed

\tau=0.7

. We find a similar observation for both Qwen3-8b and Pythia-160m, shown in Table 25 in the appendix.

Regime	Mean alignment	5M SAE feature	500M matched feature
low	$0.0\text{--}0.2$	The word average and surrounding mathematical or statistical expressions	Mathematical formatting, LaTeX code, variable subscripts, or mathematical expressions
Medium	$0.4\text{--}0.6$	Words related to medical topics, particularly cancer, diseases, and research	Mathematical expressions, code snippets, formatting symbols, or sequential lists
high	$0.8\text{--}1.0$	Comparative adjectives and adverbs, often followed by than or as	Comparative adjectives and adverbs, often followed by words such as than or more

Table 3: Representative matched Gemma-2-2B ReLU features between the

5

M SAE and the

500

M reference across increasing mean encoder-decoder alignment. The matched latents are (

2782\leftrightarrow 2782

), (

411\leftrightarrow 2629

),(

1507\leftrightarrow 1507

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Trainer	Budget	Selection	Loss Recovered $\uparrow$	$1-$ Abs. Fraction $\uparrow$	AutoInterp $\uparrow$	SCR Top-20 $\uparrow$	TPP Top-20 $\uparrow$	Sparse Probe Top-1 $\uparrow$	RAVEL Disent. $\uparrow$	Unlearning $\uparrow$	Sparsity (L0) $\downarrow$
0	5M	Random	0.993	0.959	0.703	0.253	0.029	0.756	0.657	0.006	1400.2
	5M	FPS	0.985	0.974	0.703	0.194	0.021	0.750	0.639	0.009	1298.8
	25M	Random	0.995	0.974	0.704	0.261	0.031	0.787	0.653	0.002	1353.2
	25M	FPS	0.992	0.983	0.718	0.223	0.022	0.802	0.634	0.004	1265.1
	50M	Random	0.995	0.979	0.715	0.264	0.030	0.797	0.656	0.009	1346.5
	50M	FPS	0.993	0.955	0.712	0.230	0.024	0.772	0.625	0.004	1204.0

Appendix

Table 4: SAE evaluation across activation budgets and selection strategies for Gemma-2-2B, ReLU (16k). Random and FPS are compared at 5M,25M, and 50M activation budgets against the corresponding 500M baseline.

Trainer	Budget	Selection	Loss Recovered $\uparrow$	$1-$ Abs. Fraction $\uparrow$	AutoInterp $\uparrow$	SCR Top-20 $\uparrow$	TPP Top-20 $\uparrow$	Sparse Probe Top-1 $\uparrow$	RAVEL Disent. $\uparrow$	Unlearning $\uparrow$	Sparsity (L0) $\downarrow$
6	5M	Random	0.995	0.902	0.648	0.191	0.011	0.775	0.690	0.024	1700.5
	5M	FPS	0.988	0.894	0.663	0.125	0.008	0.792	0.664	0.013	1439.5
	25M	Random	0.995	0.865	0.669	0.192	0.012	0.811	0.685	0.011	1932.7
	25M	FPS	0.993	0.758	0.695	0.155	0.009	0.805	0.665	0.009	1686.4
	50M	Random	0.997	0.819	0.673	0.183	0.012	0.813	0.685	0.021	1987.0
	50M	FPS	0.995	0.729	0.702	0.159	0.009	0.812	0.681	0.051	1678.4

Appendix

Table 5: Per-trainer SAE evaluation — Gemma-2-2B, ReLU (64K). Each row is one trainer run. Missing evaluations shown as “–”.

Trainer	Budget	Selection	Loss Recovered $\uparrow$	$1-$ Abs. Fraction $\uparrow$	AutoInterp $\uparrow$	SCR Top-20 $\uparrow$	TPP Top-20 $\uparrow$	Sparse Probe Top-1 $\uparrow$	RAVEL Disent. $\uparrow$	Unlearning $\uparrow$	Sparsity (L0) $\downarrow$
0	5M	Random	0.979	0.947	0.815	0.352	0.047	0.732	0.555	0.041	79.8
	5M	FPS	0.951	0.928	0.819	0.243	0.025	0.733	0.517	0.063	50.9
	25M	Random	0.985	0.928	0.833	0.366	0.043	0.787	0.540	0.101	80.6
	25M	FPS	0.974	0.891	0.845	0.268	0.025	0.724	0.535	0.024	55.2
	50M	Random	0.985	0.950	0.835	0.356	0.050	0.729	0.578	0.024	79.7
	50M	FPS	0.979	0.844	0.853	0.285	0.032	0.739	0.573	0.045	57.6

Appendix

Table 6: Per-trainer SAE evaluation for Gemma-2-2B, BatchTopK (16K).

Trainer	Budget	Selection	Loss Recovered $\uparrow$	$1-$ Abs. Fraction $\uparrow$	AutoInterp $\uparrow$	SCR Top-20 $\uparrow$	TPP Top-20 $\uparrow$	Sparse Probe Top-1 $\uparrow$	RAVEL Disent. $\uparrow$	Unlearning $\uparrow$	Sparsity (L0) $\downarrow$
2	5M	Random	0.980	0.899	0.785	0.330	0.052	0.793	0.556	0.048	79.6
	5M	FPS	0.954	0.900	0.848	0.255	0.022	0.769	0.536	0.095	53.4
	25M	Random	0.987	0.896	0.826	0.327	0.038	0.789	0.589	0.056	80.7
	25M	FPS	0.977	0.666	0.863	0.231	0.028	0.721	0.585	0.102	57.4
	50M	Random	0.987	0.883	0.838	0.339	0.028	0.739	0.566	0.102	80.6
	50M	FPS	0.984	0.660	0.857	0.242	0.019	0.738	0.576	0.091	59.9

Appendix

Table 7: Per-trainer SAE evaluation for Gemma-2-2B, BatchTopK (64K).

Trainer	Budget	Selection	Loss Recovered $\uparrow$	$1-$ Abs. Fraction $\uparrow$	AutoInterp $\uparrow$	SCR Top-20 $\uparrow$	TPP Top-20 $\uparrow$	Sparse Probe Top-1 $\uparrow$	RAVEL Disent. $\uparrow$	Unlearning $\uparrow$	Sparsity (L0) $\downarrow$
0	5M	Random	0.977	0.910	0.825	0.365	0.047	0.736	0.523	0.015	83.4
	5M	FPS	0.949	0.910	0.828	0.232	0.034	0.695	0.511	0.094	54.3
	25M	Random	0.982	0.895	0.811	0.383	0.043	0.715	0.577	0.043	83.0
	25M	FPS	0.972	0.842	0.844	0.290	0.029	0.691	0.542	0.079	57.3
	50M	Random	0.985	0.914	0.813	0.374	0.053	0.733	0.573	0.030	82.7
	50M	FPS	0.979	0.802	0.849	0.277	0.031	0.729	0.567	0.026	60.3

Appendix

Table 8: Per-trainer SAE evaluation for Gemma-2-2B, JumpReLU (16K).

Trainer	Budget	Selection	Loss Recovered $\uparrow$	$1-$ Abs. Fraction $\uparrow$	AutoInterp $\uparrow$	SCR Top-20 $\uparrow$	TPP Top-20 $\uparrow$	Sparse Probe Top-1 $\uparrow$	RAVEL Disent. $\uparrow$	Unlearning $\uparrow$	Sparsity (L0) $\downarrow$
2	5M	Random	0.977	0.873	0.809	0.369	0.044	0.733	0.561	0.069	83.5
	5M	FPS	0.956	0.757	0.835	0.196	0.014	0.694	0.539	0.071	55.5
	25M	Random	0.987	0.827	0.835	0.328	0.029	0.765	0.567	0.081	82.9
	25M	FPS	0.975	0.717	0.867	0.246	0.027	0.725	0.581	0.092	59.0
	50M	Random	0.987	0.834	0.825	0.308	0.039	0.737	0.572	0.084	82.8
	50M	FPS	0.984	0.663	0.852	0.264	0.020	0.741	0.582	0.221	62.0

Appendix

Table 9: Per-trainer SAE evaluation for Gemma-2-2B, JumpReLU (64K).

Trainer	Budget	Selection	Loss Recovered $\uparrow$	$1-$ Abs. Fraction $\uparrow$	AutoInterp $\uparrow$	SCR Top-20 $\uparrow$	TPP Top-20 $\uparrow$	Sparse Probe Top-1 $\uparrow$	RAVEL Disent. $\uparrow$	Unlearning $\uparrow$	Sparsity (L0) $\downarrow$
0	5M	Random	0.979	0.948	0.845	0.404	0.107	0.755	0.561	0.028	79.5
	5M	FPS	0.941	0.967	0.863	0.295	0.053	0.724	0.526	0.015	49.1
	25M	Random	0.984	0.966	0.844	0.455	0.104	0.743	0.549	0.006	80.0
	25M	FPS	0.969	0.953	0.884	0.340	0.092	0.760	0.545	0.090	54.4
	50M	Random	0.984	0.982	0.851	0.460	0.130	0.776	0.562	0.041	79.5
	50M	FPS	0.975	0.975	0.863	0.373	0.106	0.734	0.569	0.056	57.4

Appendix

Table 10: Per-trainer SAE evaluation for Gemma-2-2B, Matryoshka (16K).

Trainer	Budget	Selection	Loss Recovered $\uparrow$	$1-$ Abs. Fraction $\uparrow$	AutoInterp $\uparrow$	SCR Top-20 $\uparrow$	TPP Top-20 $\uparrow$	Sparse Probe Top-1 $\uparrow$	RAVEL Disent. $\uparrow$	Unlearning $\uparrow$	Sparsity (L0) $\downarrow$
2	5M	Random	0.975	0.925	0.772	0.399	0.086	0.756	0.543	0.026	79.5
	5M	FPS	0.949	0.967	0.873	0.266	0.074	0.760	0.540	0.086	50.6
	25M	Random	0.985	0.992	0.832	0.393	0.106	0.783	0.602	0.017	80.1
	25M	FPS	0.975	0.958	0.885	0.339	0.048	0.756	0.612	0.038	55.1
	50M	Random	0.987	0.974	0.829	0.428	0.101	0.767	0.610	0.043	79.9
	50M	FPS	0.980	0.961	0.881	0.325	0.086	0.772	0.581	0.079	57.9

Appendix

Table 11: Per-trainer SAE evaluation for Gemma-2-2B, Matryoshka (64K).

Trainer	Budget	Selection	Loss Recovered $\uparrow$	$1-$ Abs. Fraction $\uparrow$	AutoInterp $\uparrow$	SCR Top-20 $\uparrow$	TPP Top-20 $\uparrow$	Sparse Probe Top-1 $\uparrow$	RAVEL Disent. $\uparrow$	Sparsity (L0) $\downarrow$
0	5M	Random	0.975	0.944	0.676	0.123	0.035	0.725	0.499	1208.2
	5M	FPS	0.955	0.942	0.680	0.105	0.021	0.735	0.499	1000.0
	25M	Random	0.980	0.907	0.688	0.142	0.033	0.712	0.500	1200.3
	25M	FPS	0.960	0.942	0.689	0.120	0.029	0.727	0.501	1074.3
	50M	Random	0.978	0.913	0.697	0.146	0.034	0.719	0.501	1180.7
	50M	FPS	0.962	0.918	0.702	0.127	0.030	0.724	0.502	1117.9

Appendix

Table 12: Per-trainer SAE evaluation for Pythia-160M, ReLU (16K).

Trainer	Budget	Selection	Loss Recovered $\uparrow$	$1-$ Abs. Fraction $\uparrow$	AutoInterp $\uparrow$	SCR Top-20 $\uparrow$	TPP Top-20 $\uparrow$	Sparse Probe Top-1 $\uparrow$	RAVEL Disent. $\uparrow$	Sparsity (L0) $\downarrow$
6	5M	Random	0.972	0.892	0.659	0.099	0.012	0.719	0.499	1861.4
	5M	FPS	0.944	0.878	0.666	0.076	0.008	0.719	0.495	1371.2
	25M	Random	0.976	0.800	0.671	0.088	0.014	0.714	0.500	2002.8
	25M	FPS	0.964	0.784	0.679	0.090	0.010	0.706	0.499	1646.3
	50M	Random	0.977	0.809	0.671	0.093	0.014	0.730	0.499	2012.6
	50M	FPS	0.967	0.728	0.674	0.083	0.012	0.718	0.498	1758.5

Appendix

Table 13: Per-trainer SAE evaluation for Pythia-160M, ReLU (64K).

Trainer	Budget	Selection	Loss Recovered $\uparrow$	$1-$ Abs. Fraction $\uparrow$	AutoInterp $\uparrow$	SCR Top-20 $\uparrow$	TPP Top-20 $\uparrow$	Sparse Probe Top-1 $\uparrow$	RAVEL Disent. $\uparrow$	Sparsity (L0) $\downarrow$
0	5M	Random	0.946	0.902	0.733	0.189	0.245	0.707	0.492	142.6
	5M	FPS	0.916	0.845	0.744	0.216	0.163	0.707	0.490	106.3
	25M	Random	0.954	0.911	0.755	0.183	0.222	0.704	0.492	137.5
	25M	FPS	0.940	0.852	0.776	0.196	0.163	0.697	0.489	108.2
	50M	Random	0.953	0.914	0.774	0.027	0.224	0.697	0.491	135.3
	50M	FPS	0.949	0.840	0.768	0.175	0.183	0.698	0.488	118.1

Appendix

Table 14: Per-trainer SAE evaluation for Pythia-160M, BatchTopK (16K).

Trainer	Budget	Selection	Loss Recovered $\uparrow$	$1-$ Abs. Fraction $\uparrow$	AutoInterp $\uparrow$	SCR Top-20 $\uparrow$	TPP Top-20 $\uparrow$	Sparse Probe Top-1 $\uparrow$	RAVEL Disent. $\uparrow$	Sparsity (L0) $\downarrow$
2	5M	Random	0.945	0.886	0.692	0.229	0.248	0.708	0.492	352.4
	5M	FPS	0.916	0.836	0.703	0.189	0.210	0.704	0.488	294.7
	25M	Random	0.961	0.862	0.717	0.200	0.260	0.697	0.492	330.6
	25M	FPS	0.946	0.769	0.763	0.196	0.215	0.710	0.483	276.4
	50M	Random	0.961	0.857	0.736	-0.084	0.235	0.709	0.490	315.7
	50M	FPS	0.952	0.635	0.746	0.180	0.145	0.691	0.488	282.9

Appendix

Table 15: Per-trainer SAE evaluation for Pythia-160M, BatchTopK (64K).

Trainer	Budget	Selection	Loss Recovered $\uparrow$	$1-$ Abs. Fraction $\uparrow$	AutoInterp $\uparrow$	SCR Top-20 $\uparrow$	TPP Top-20 $\uparrow$	Sparse Probe Top-1 $\uparrow$	RAVEL Disent. $\uparrow$	Sparsity (L0) $\downarrow$
0	5M	Random	0.946	0.857	0.732	0.240	0.229	0.696	0.491	139.5
	5M	FPS	0.909	0.809	0.747	0.131	0.210	0.691	0.491	106.7
	25M	Random	0.955	0.907	0.742	0.246	0.213	0.686	0.486	141.6
	25M	FPS	0.942	0.832	0.763	0.161	0.175	0.686	0.490	111.9
	50M	Random	0.957	0.908	0.753	0.208	0.246	0.695	0.492	141.5
	50M	FPS	0.948	0.828	0.766	0.214	0.186	0.680	0.488	115.6

Appendix

Table 16: Per-trainer SAE evaluation for Pythia-160M, JumpReLU (16K).

Trainer	Budget	Selection	Loss Recovered $\uparrow$	$1-$ Abs. Fraction $\uparrow$	AutoInterp $\uparrow$	SCR Top-20 $\uparrow$	TPP Top-20 $\uparrow$	Sparse Probe Top-1 $\uparrow$	RAVEL Disent. $\uparrow$	Sparsity (L0) $\downarrow$
2	5M	Random	0.950	0.833	0.703	0.147	0.211	0.698	0.494	297.4
	5M	FPS	0.909	0.739	0.714	0.125	0.165	0.665	0.490	255.9
	25M	Random	0.958	0.818	0.727	0.244	0.238	0.698	0.494	309.8
	25M	FPS	0.944	0.654	0.733	0.163	0.168	0.722	0.487	266.2
	50M	Random	0.961	0.780	0.725	0.218	0.205	0.676	0.492	315.0
	50M	FPS	0.957	0.635	0.747	0.207	0.184	0.690	0.494	272.5

Appendix

Table 17: Per-trainer SAE evaluation for Pythia-160M, JumpReLU (64K).

Trainer	Budget	Selection	Loss Recovered $\uparrow$	$1-$ Abs. Fraction $\uparrow$	AutoInterp $\uparrow$	SCR Top-20 $\uparrow$	TPP Top-20 $\uparrow$	Sparse Probe Top-1 $\uparrow$	RAVEL Disent. $\uparrow$	Sparsity (L0) $\downarrow$
0	5M	Random	0.934	0.902	0.758	0.136	0.244	0.727	0.486	132.5
	5M	FPS	0.914	0.892	0.755	0.182	0.223	0.693	0.490	107.5
	25M	Random	0.952	0.907	0.791	-0.040	0.277	0.720	0.493	126.6
	25M	FPS	0.931	0.938	0.795	-0.086	0.238	0.699	0.493	101.8
	50M	Random	0.949	0.890	0.803	-0.302	0.213	0.697	0.489	120.6
	50M	FPS	0.941	0.952	0.800	0.037	0.232	0.714	0.490	107.5

Appendix

Table 18: Per-trainer SAE evaluation for Pythia-160M, Matryoshka (16K).

Trainer	Budget	Selection	Loss Recovered $\uparrow$	$1-$ Abs. Fraction $\uparrow$	AutoInterp $\uparrow$	SCR Top-20 $\uparrow$	TPP Top-20 $\uparrow$	Sparse Probe Top-1 $\uparrow$	RAVEL Disent. $\uparrow$	Sparsity (L0) $\downarrow$
2	5M	Random	0.948	0.891	0.687	0.120	0.214	0.717	0.490	340.3
	5M	FPS	0.923	0.877	0.690	0.209	0.214	0.717	0.490	314.7
	25M	Random	0.954	0.939	0.712	-0.259	0.251	0.714	0.492	330.7
	25M	FPS	0.945	0.941	0.734	-0.035	0.192	0.704	0.489	283.3
	50M	Random	0.954	0.967	0.744	-0.085	0.219	0.694	0.489	300.2
	50M	FPS	0.951	0.942	0.747	0.011	0.173	0.710	0.489	270.7

Appendix

Table 19: Per-trainer SAE evaluation for Pythia-160M, Matryoshka (64K).

Trainer	Budget	Selection	Loss Recovered $\uparrow$	$1-$ Abs. Fraction $\uparrow$	AutoInterp $\uparrow$	SCR Top-20 $\uparrow$	TPP Top-20 $\uparrow$	Sparse Probe Top-1 $\uparrow$	Sparsity (L0) $\downarrow$
0	5M	Random	0.993	0.959	0.823	0.178	0.155	0.732	1337.0
	5M	FPS	0.984	0.970	0.808	0.116	0.057	0.732	1199.0
	25M	Random	0.993	0.894	0.833	0.147	0.101	0.726	1835.8
	25M	FPS	0.991	0.860	0.812	0.137	0.065	0.732	1638.7
	50M	Random	0.995	0.815	0.815	0.151	0.090	0.780	1944.7
	50M	FPS	0.991	0.755	0.793	0.125	0.054	0.737	1703.2

Appendix

Table 20: Per-trainer SAE evaluation for Qwen3-8B, ReLU (64K).

Trainer	Budget	Selection	Loss Recovered $\uparrow$	$1-$ Abs. Fraction $\uparrow$	AutoInterp $\uparrow$	SCR Top-20 $\uparrow$	TPP Top-20 $\uparrow$	Sparse Probe Top-1 $\uparrow$	Sparsity (L0) $\downarrow$
2	5M	Random	0.984	0.974	0.852	0.282	0.303	0.703	85.6
	5M	FPS	0.968	0.962	0.878	0.189	0.258	0.679	63.4
	25M	Random	0.986	0.951	0.867	0.264	0.303	0.692	86.1
	25M	FPS	0.977	0.983	0.882	0.202	0.296	0.671	66.9
	50M	Random	0.986	0.944	0.867	0.258	0.289	0.692	85.8
	50M	FPS	0.982	0.943	0.877	0.236	0.313	0.698	69.6

Appendix

Table 21: Per-trainer SAE evaluation for Qwen3-8B, BatchTopK (64K).

Trainer	Budget	Selection	Loss Recovered $\uparrow$	$1-$ Abs. Fraction $\uparrow$	AutoInterp $\uparrow$	SCR Top-20 $\uparrow$	TPP Top-20 $\uparrow$	Sparse Probe Top-1 $\uparrow$	Sparsity (L0) $\downarrow$
2	5M	Random	0.980	0.990	0.857	0.336	0.226	0.652	87.4
	5M	FPS	0.968	0.943	0.860	0.192	0.106	0.700	68.5
	25M	Random	0.984	0.932	0.869	0.270	0.229	0.664	87.3
	25M	FPS	0.982	0.965	0.884	0.212	0.098	0.677	67.7
	50M	Random	0.986	0.962	0.876	0.257	0.198	0.671	87.4
	50M	FPS	0.982	0.909	0.868	0.242	0.151	0.662	70.1

Appendix

Table 22: Per-trainer SAE evaluation for Qwen3-8B, JumpReLU (64K).

Trainer	Budget	Selection	Loss Recovered $\uparrow$	$1-$ Abs. Fraction $\uparrow$	AutoInterp $\uparrow$	SCR Top-20 $\uparrow$	TPP Top-20 $\uparrow$	Sparse Probe Top-1 $\uparrow$	Sparsity (L0) $\downarrow$
2	5M	Random	0.980	0.982	0.848	0.290	0.342	0.662	85.3
	5M	FPS	0.962	0.998	0.892	0.238	0.286	0.726	58.3
	25M	Random	0.984	0.990	0.863	0.268	0.325	0.708	85.5
	25M	FPS	0.978	0.986	0.894	0.296	0.298	0.672	62.6
	50M	Random	0.986	0.986	0.857	0.365	0.311	0.691	85.9
	50M	FPS	0.980	0.978	0.894	0.312	0.299	0.704	65.7

Appendix

Table 23: Per-trainer SAE evaluation for Qwen3-8B, Matryoshka (64K).

Model	Selection	Coreset	$\boldsymbol{\Delta}$ mean norm (%)	KL (bits)	Mean-dir. cosine
			Selected vs. 500M stream
Qwen3-8B	FPS	5M	$+33.8\pm 1.3$	4.02	$0.889\pm 0.007$
		25M	$+21.4\pm 0.4$	2.08	$0.952\pm 0.002$
		50M	$+16.4\pm 0.3$	1.35	$0.970\pm 0.001$
	Random	5M	$+0.1\pm 0.2$	0.000	$1.000\pm 0.000$
		25M	$0.0\pm 0.1$	0.000	$1.000\pm 0.000$

Appendix

Table 24: FPS distribution shift for Qwen3-8B and Pythia-160M. * Metrics are defined as in Table 1 . The Gemma-2-2B pattern holds across model families: random selection is distribution-neutral at every budget, while FPS shifts the selected subset toward higher-norm activations, with divergence growing as the subset shrinks.

			Counterpart (%)	Shared features (%)
Model	SAE	Budget	Enc.–dec. agree	$\boldsymbol{\tau\geq 0.3}$	$\boldsymbol{\tau\geq 0.5}$	$\boldsymbol{\tau\geq 0.7}$
Qwen3-8B	ReLU	5M	69.85	33.07	22.90	18.58
		25M	79.47	51.16	36.23	25.19
		50M	82.49	61.70	41.41	29.51
	BatchTopK	5M	66.06	34.16	7.34	1.14
		25M	81.05	65.56	35.63	8.83

Appendix

Table 25: Feature correspondence under increasingly strict alignment criteria. Encoder-decoder agreement is the fraction of features for which independently computed encoder and decoder Hungarian matchings select the same reference latent. Shared features additionally require both encoder and decoder cosine similarities to exceed the specified threshold

\tau

. Results are reported for Qwen3-8B, and Pythia-160M.

A. 500M vs. Random
	ReLU			BatchTopK
Metric	5M	25M	50M	5M	25M	50M
Loss Rec.	$+.005/<.001^{***}$	$+.002/.002^{**}$	$+.002/.014^{*}$	$+.007/.006^{**}$	$+.002/.015^{*}$	$+.002/.092$
Sparse Probe	$+.018/.008^{**}$	$+.016/.005^{**}$	$+.004/.363$	$-.022/.355$	$-.036/.050$	$-.006/.769$
1-Abs.	$-.098/<.001^{***}$	$-.053/<.001^{***}$	$-.030/<.001^{***}$	$-.032/.516$	$-.047/.252$	$-.055/.135$
AutoInterp	$+.020/<.001^{***}$	$+.014/<.001^{***}$	$+.006/.085$	$+.032/.039^{*}$	$+.008/.365$	$+.003/.734$

Appendix

Table 26: Paired statistical tests for Gemma-2-2B ReLU and BatchTopK. Panel A compares the

500

M reference with Random subsets (

\Delta=\mathrm{500M}-\mathrm{Random}

); Panel B compares FPS with Random at matched budgets (

\Delta=\mathrm{FPS}-\mathrm{Random}

). Each entry reports

\Delta/p

from a paired two-sided

t

-test across matched SAE configurations.

Sparse Autoencoders

Sparse autoencoders decompose neural network activations into sparse features to study, interpret, and control learned representations.

See all 218 papers

Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders
Jun 10, 2026Gleb Gerasimov, Timofei Rusalev, Nikita Balagansky +3Representation Identifiability Sparse Autoencoders
T-Tech
Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoders
Aug 22, 2025David Chanin, Adrià Garriga-AlonsoActivation Sparsity LLM Interpretability
University College London · MATS Research
C $^{2}$ R: Cross-sample Consistency Regularization Mitigates Feature Splitting and Absorption in Sparse Autoencoders
Jun 29, 2026Haoran Jin, Xiting Wang, Shijie Ren +2LLM Interpretability Representation Disentanglement
University of Science and Technology of China · State Key Laboratory of Cognitive Intelligence · Gaoling School of Artificial Intelligence, Renmin University of China

Jun 10, 2026J/K move · Enter open · S save

Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders

Gleb Gerasimov, Timofei Rusalev, Nikita Balagansky, Daniil Laptev+2

T-Tech

Representation Identifiability Sparse Autoencoders Mechanistic Interpretability

Open paper AskPDF arXiv

Sparse autoencoders (SAEs) are widely used to interpret neural network representations, but their utility depends on whether the learned features are reproducible across training runs. We study this question through feature stability: for each SAE feature, we estimate the probability that a similar feature reappears in an independently trained SAE. This yields a scalable per-feature signal that separates stable from unstable features. In a large-scale study across seeds, models, layers, dictionary sizes, and SAE variants, we find a pronounced functional asymmetry: stable features carry most of the reconstruction- and prediction-relevant signal, while unstable features have weak marginal impact and are dominated by low-frequency surface-form triggers in both activation statistics and automatic explanations. Geometrically, unstable features are individually non-reproducible but concentrate in reproducible lower-rank subspaces, suggesting that seed dependence often reflects basis ambiguity within a shared region of activation space rather than pure noise. A controlled synthetic model makes this mechanism explicit, showing that low-rank ground-truth features can be recovered at the subspace level while remaining non-identifiable as individual SAE latents across seeds. Finally, by pooling unique cross-seed features, we construct more stable SAEs while preserving explained variance in this setting. Together, these results show that unstable features are not merely failed or noisy latents: they have weak individual functional impact, but reflect reproducible low-dimensional structure that standard SAEs resolve differently across seeds.

Abstract

Sparse Autoencoders

Explore similar work

Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders

Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoders

C2^{2}2R: Cross-sample Consistency Regularization Mitigates Feature Splitting and Absorption in Sparse Autoencoders

C $^{2}$ R: Cross-sample Consistency Regularization Mitigates Feature Splitting and Absorption in Sparse Autoencoders