Organizations: DIAG, Sapienza University of Rome, Italy · Sony CSL Rome, CREF, Rome, Italy · Copenhagen Center for Social Data Science (SODAS), University of Copenhagen, Denmark · Department of Physics, Sapienza University of Rome, Italy · Complexity Science Hub, Vienna, Austria
Training generative models on synthetic data causes loss of diversity and a decrease in performance, a phenomenon known as model collapse. We study this degeneration spectrally in a controlled Markov setting. The eigenvalue spectrum of the learned transition operator offers a direct view of the process, since the subdominant eigenvalues set how fast the chain mixes and therefore how much of the state space it continues to explore. We find that sampling at temperature Ts<1 acts on the operator like repeated sharpening, pushing it toward a deterministic system: the subdominant eigenvalues approach the unit circle and the spectral gap narrows. When Ts<1, even data accumulation does not appear to arrest this drift. We then analyze Dreaming Learning, a regulariser that, in parallel with regular training, adds a loss term on the model's own samples drawn at a raised temperature Td>1. Because high-temperature samples flatten the predicted distribution, training on them keeps the entries of the learned operator from becoming vanishingly small, which in turn keeps the eigenvalues bounded away from the unit circle. Measurements of the spectra over 5000 generations are consistent with this picture, and the mitigation effect persists in more complex settings such as LSTM language models and a 124M-parameter GPT-2.
Figures & tables
Figure 1: Eigenvalue magnitudes ∣λ2∣,…,∣λ5∣ of the reconstructed transition matrix across 5000 generations and 10 seeds, moving average ( w=100 ), shaded region represents 25th to 75th percentile. Dashed lines mark the true spectrum of M0 . Left: vanilla recursion. The subdominant eigenvalues climb toward 1 and the spectral gap narrows. Right: with Dreaming Learning ( Td=1.5 , α=0.8 ) the spectrum is flat in t . The dreaming spectrum stabilises below the true one, indicating a residual bias toward faster mixing that is controlled by Td and α .
Figure 2: GPT-2 (124M) under the generate-and-retrain loop, with synthetic corpora sampled at Ts=0.8 . WikiText-2 validation perplexity against generation count, vanilla and dreaming ( Td=1.1 , α=0.5 ), over five seeds; shaded regions represent 25th to 75th percentile.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: DKL(p∥q) between the true and reconstructed stationary distributions after 5000 generations over 10 seeds. Left: varying Td at α=0.8 . Right: varying α at Td=1.5 . Vanilla denotes no dreaming term. Shaded regions represent 25th to 75th percentile.
Figure 4: DKL between true and reconstructed stationary distributions after 5000 generations across the (Td,α) grid.
Figure 5: Left, right: LSTM validation loss against generation count for vanilla and dreaming ( Td=3 ) models at Ts=0.7 and Ts=1 over 10 seeds each. Shaded region represents 25th to 75th percentile.
Figure 6: WikiText-2 validation perplexity at the best primary-validation epoch, for each (Td,α) configuration tried. Only vanilla and Td=1.1 , α=0.5 are averaged over seeds, with shaded regions spanning the 25th to 75th percentile; the remaining curves are single runs and some end at generation 10.
As scaling laws push the training of frontier large language models (LLMs) toward ever-growing data requirements, training pipelines are approaching a regime where much of the publicly available online text may be consumed. At the same time, widespread LLM usage increases the volume of machine-generated content on the web; together, these trends raise the likelihood of generated text re-entering future training corpora, increasing the associated risk of performance degradation often called model collapse. In practice, model developers address this concern through data cleaning, watermarking, synthetic-data policies, or, in some cases, blissful ignorance. However, the problem of model collapse in generative models has not been examined from a learning-theoretic perspective: we study it through the theoretical lens of the language generation in the limit framework, introducing a replay adversary that augments the example stream with the generator's own past outputs. Our main contribution is a fine-grained learning-theoretic characterization of when replay fundamentally limits generation: while replay is benign for the strongest notion of uniform generation, it provably creates separations for the weaker notions of non-uniform generation and generation in the limit. Interestingly, our positive results mirror heuristics widely used in practice, such as data cleaning, watermarking, and output filtering, while our separations show when these ideas can fail.
Mode collapse is a persistent challenge in generative modeling and appears in autoregressive text generation as behaviors ranging from explicit looping to gradual loss of diversity and premature trajectory convergence. We take a dynamical-systems view and reinterpret mode collapse as reduced state-space accessibility caused by geometric collapse: during generation, the model's internal trajectory becomes confined to a low-dimensional region of its representation space. This implies mode collapse is not purely a token-level phenomenon and cannot be reliably solved by symbolic constraints or probability-only decoding heuristics. Guided by this perspective, we propose Reinforced Mode Regulation (RMR), a lightweight, online state-space intervention that regulates dominant self-reinforcing directions in the Transformer value cache (implemented as low-rank damping). Across multiple large language models, RMR substantially reduces mode collapse and enables stable generation at extremely low entropy rates (down to 0.8 nats/step), whereas standard decoding typically collapses near 2.0 nats/step.
Xin Du, Kumiko Tanaka-Ishii
Department of Communications and Computer Engineering, Waseda University, Tokyo, Japan · Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University, Shanghai, China · Department of Computer Science and Engineering, Waseda University, Tokyo, Japan
Model collapse arises when generative models are trained on synthetic data produced by earlier models. The phenomenon has attracted considerable attention because of its societal and technical implications. However, previous studies have reached seemingly contradictory conclusions: replacing real data with synthetic data causes collapse (Shumailov et al.), yet accumulating real data alongside synthetic data can prevent it. For diffusion models, we study an intermediate regime typical of finite-budget pipelines: all past datasets and the real data are kept, but each new model is trained on a fixed-size sample from this growing pool, so the real fraction vanishes without any data being removed. Experiments on a 2D spiral dataset as well as the image benchmarks (MNIST, Fashion-MNIST, and CIFAR-10) show that replacement protocol degrades dataset rapidly as in the literature, whereas the fixed budget degrades only partially, sparing some features. A linear-response model of the multi-generational parameter dynamics, analyzed by stochastic recursion, confirms that the two protocols differ: some features will be fragile and lost within a few generations for both protocols, while some will be robust and preserved over practically unbounded horizons under the fixed budget protocol.
Hanna Malet, Gabriel Turinici
CEREMADE CNRS Université Paris Dauphine - PSL Paris 75116 FRANCE · Cornell University Ithaca, NY, USA