Large language models are increasingly exposed to model-generated text, raising the possibility that future models will be trained recursively on the outputs of their predecessors. Prior work on such self-consuming training loops has largely focused on output quality and downstream performance, leaving their effects on internal representations poorly understood. We study how representations evolve across five generations of self-consumption and whether sparse autoencoders (SAEs) remain reliable under this shift. At each generation, we compare a lineage trained on synthetic data with a matched control trained on real data under the same training settings. We evaluate how well SAEs trained before self-consumption transfer to later generations, measuring reconstruction and sparsity across SAE widths, architectures, and model layers. We then fine-tune the original SAE on each generation and use sparse probing to test whether task-relevant information remains concentrated in a small number of latents or becomes more distributed across the representation. We find that zero-shot SAE performance degrades progressively under self-consumption, with growing reconstruction and sparsity gaps that vary substantially across layers. Fine-tuning largely restores reconstruction, but not feature-level usefulness: task-relevant information remains accessible in the full latent space while becoming less concentrated in individual features. These results show that self-consumption induces representational drift and that reconstruction fidelity alone is insufficient to establish that an adapted SAE remains equally useful for interpretation.
Figures & tables
Figure 1: Zero-shot damage to Pythia-160M SAEs grows with self-consumption; fine-tuning removes most of it. Top: BatchTopK SAEs with widths 212 , 214 , and 216 . Bottom: BatchTopK, JumpReLU, and Matryoshka-BatchTopK SAEs at width 214 . Solid lines denote zero-shot evaluation with the original SAE held fixed; dashed lines denote SAEs fine-tuned separately on each generation (error bars show variation across fine-tuning seeds). All quantities are damage relative to the matched real-data lineage; higher values indicate greater degradation under self-consumption.
Figure 2: Fine-tuning does not restore sparse localization in Pythia-160M . Left: sparse-probing damage using the top- k latents or the full SAE representation. Damage remains high for small k after fine-tuning, while full-SAE probing stays near the real-data control, indicating that task-relevant information persists but is less concentrated in individual latents. Right: SCR/TPP damage at gen. 5 across intervention budgets, providing complementary feature-level evaluations.
Figure 3: Self-consumption damage in Gemma-2-2B SAEs at an early (5), middle (12), and late (19) layer. Top: core metrics (solid: zero-shot; dashed: fine-tuned). Zero-shot damage grows with depth and generation; fine-tuning removes nearly all of it, leaving a small residual L0 gap at layer 19. Bottom: sparse-probing, SCR, and TPP damage; color denotes the regime (orange: zero-shot; blue: fine-tuned) and line style the layer. Higher is worse.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Pythia-160M
Gemma-2-2B
Checkpoint
pythia-160m-deduped
gemma-2-2b
Blocks per generation
10,000
10,000
Block length (tokens)
512
512
Tokens per generation
≈ 5.1M
≈ 5.1M
Epochs per generation
1
1
Optimizer
AdamW
AdamW
Appendix
Table 1: Fine-tuning and sampling settings. Each setting is shared by the real and synthetic lineages of a given model.
Figure 4: Self-consumption degrades language modeling. (a) Perplexity on held-out real data for the synthetic lineage and the matched real-data control. (b) Log-perplexity gap between the two lineages. By generation 5, the gap reaches 0.75 nats ( 2.13× perplexity) for Pythia-160M and 0.38 nats ( 1.46× ) for Gemma-2-2B . The control plateaus after the first generation for Pythia-160M and improves for Gemma-2-2B .
Self-training for language models uses their own generations, feedback, and interaction traces as supervision to improve capabilities with less labeled data.
Sparse Autoencoders (SAEs) extract features from LLM internal activations, meant to correspond to interpretable concepts. A core SAE training hyperparameter is L0: how many SAE features should fire per token on average. Existing work compares SAE algorithms using sparsity-reconstruction tradeoff plots, implying L0 is a free parameter with no inherently correct value aside from its effect on reconstruction. In this work we study the effect of L0 on SAEs, and show that if L0 is not set correctly, the SAE fails to disentangle the underlying features of the LLM. If L0 is too low, the SAE will mix correlated features to improve reconstruction. If L0 is too high, the SAE finds degenerate solutions that also mix features. Further, we present a proxy metric that can help guide the search for the correct L0 for an SAE on a given training distribution. We show that our method finds the correct L0 in toy models and coincides with peak sparse probing performance in LLM SAEs. We find that most commonly used SAEs have an L0 that is too low. Our work shows that practitioners must set L0 correctly to train SAEs with monosemantic features.
Sparse autoencoders (SAEs) are typically trained on hundreds of millions to billions of activation tokens, yet little work has tested whether this scale is necessary. We study how much activation data SAEs need by training four SAE architectures on three language models with budgets ranging from 5M tokens to a 500M-token reference. We measure two notions of convergence: functional behavior on standard evaluations and one-to-one reproducibility of learned features. Functional data requirements are strongly metric-dependent: reconstruction often approaches the full-data reference at substantially smaller budgets, whereas feature reproducibility continues to improve with data. For Gemma-2-2B BatchTopK, the fraction of strictly matched features rises from 2% at 5M tokens to 29% at 50M. Data selection also matters: random sampling better preserves the original activation distribution and generally favors reconstruction, whereas diversity-based sampling frequently improves AutoInterp. A theoretical analysis separates statistical consistency of subsampled training from the stronger identifiability and optimization conditions required for one-to-one feature recovery. Our results show that SAE training can often be made far more data-efficient when budgets are matched to the goal, and feature-level claims are backed by explicit reproducibility checks, paving the way toward cheaper and more reliable SAE research.
Sambit Ghosh, Madelyn Mathai, R. Venkatesh Babu +1
IBM · Indian Institute of Science · University of Virginia
Sparse autoencoders (SAEs) decompose language model activations into sparse features, but prompt transformations designed to preserve the underlying request can redistribute activity across them. A feature change observed on one transformed prompt need not recur, and even a recurring change need not affect a downstream computation. We introduce the redistribution sensitivity score (RSS), which identifies large feature responses that recur in a consistent direction, and evaluate the resulting supports on variants excluded from selection. Across 27 pretrained SAEs spanning eight language models, the tested prompt transformations produce a reproducible redistribution profile on these excluded variants. We then separate recurring representation change from signed readout contribution and decision failure. For a fixed linear readout, an exact decomposition shows how response magnitude, readout coefficients, directional alignment, and the clean margin determine downstream consequences. In two frozen safety readouts, combining RSS support erosion with the full clean margin yields recurrent failure AUROCs of 0.935 and 0.958 using median supports of 20 and 19 features, compared with 0.965 and 0.964 using full erosion. RSS therefore provides a way to identify recurring sparse representation changes and quantify their contribution to a downstream readout without equating representation change with failure. Code is available at https://github.com/OpenSAE/Redistribution-Sensitivity-Score
Ziye Mei, Haotian Ma
Department of Computer Sciences, University of Wisconsin–Madison