Representational Change Under Self-Consuming Training: A Sparse Autoencoder Analysis
Organizations: Department of Computer Science, The Ohio State University, Columbus, OH 43210, USA
Abstract
Large language models are increasingly exposed to model-generated text, raising the possibility that future models will be trained recursively on the outputs of their predecessors. Prior work on such self-consuming training loops has largely focused on output quality and downstream performance, leaving their effects on internal representations poorly understood. We study how representations evolve across five generations of self-consumption and whether sparse autoencoders (SAEs) remain reliable under this shift. At each generation, we compare a lineage trained on synthetic data with a matched control trained on real data under the same training settings. We evaluate how well SAEs trained before self-consumption transfer to later generations, measuring reconstruction and sparsity across SAE widths, architectures, and model layers. We then fine-tune the original SAE on each generation and use sparse probing to test whether task-relevant information remains concentrated in a small number of latents or becomes more distributed across the representation. We find that zero-shot SAE performance degrades progressively under self-consumption, with growing reconstruction and sparsity gaps that vary substantially across layers. Fine-tuning largely restores reconstruction, but not feature-level usefulness: task-relevant information remains accessible in the full latent space while becoming less concentrated in individual features. These results show that self-consumption induces representational drift and that reconstruction fidelity alone is insufficient to establish that an adapted SAE remains equally useful for interpretation.
Figures & tables
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Pythia-160M | Gemma-2-2B | |
| Checkpoint | pythia-160m-deduped | gemma-2-2b |
| Blocks per generation | 10,000 | 10,000 |
| Block length (tokens) | 512 | 512 |
| Tokens per generation | 5.1M | 5.1M |
| Epochs per generation | 1 | 1 |
| Optimizer | AdamW | AdamW |
Self-Training for Language Models
Self-training for language models uses their own generations, feedback, and interaction traces as supervision to improve capabilities with less labeled data.
See all 95 papers