Organizations: DIAG, Sapienza University of Rome, Italy · Sony CSL Rome, CREF, Rome, Italy · Copenhagen Center for Social Data Science (SODAS), University of Copenhagen, Denmark · Department of Physics, Sapienza University of Rome, Italy · Complexity Science Hub, Vienna, Austria
Training generative models on synthetic data causes loss of diversity and a decrease in performance, a phenomenon known as model collapse. We study this degeneration spectrally in a controlled Markov setting. The eigenvalue spectrum of the learned transition operator offers a direct view of the process, since the subdominant eigenvalues set how fast the chain mixes and therefore how much of the state space it continues to explore. We find that sampling at temperature Ts<1 acts on the operator like repeated sharpening, pushing it toward a deterministic system: the subdominant eigenvalues approach the unit circle and the spectral gap narrows. When Ts<1, even data accumulation does not appear to arrest this drift. We then analyze Dreaming Learning, a regulariser that, in parallel with regular training, adds a loss term on the model's own samples drawn at a raised temperature Td>1. Because high-temperature samples flatten the predicted distribution, training on them keeps the entries of the learned operator from becoming vanishingly small, which in turn keeps the eigenvalues bounded away from the unit circle. Measurements of the spectra over 5000 generations are consistent with this picture, and the mitigation effect persists in more complex settings such as LSTM language models and a 124M-parameter GPT-2.
Figures & tables
Figure 1: Eigenvalue magnitudes ∣λ2∣,…,∣λ5∣ of the reconstructed transition matrix across 5000 generations and 10 seeds, moving average ( w=100 ), shaded region represents 25th to 75th percentile. Dashed lines mark the true spectrum of M0 . Left: vanilla recursion. The subdominant eigenvalues climb toward 1 and the spectral gap narrows. Right: with Dreaming Learning ( Td=1.5 , α=0.8 ) the spectrum is flat in t . The dreaming spectrum stabilises below the true one, indicating a residual bias toward faster mixing that is controlled by Td and α .
Figure 2: GPT-2 (124M) under the generate-and-retrain loop, with synthetic corpora sampled at Ts=0.8 . WikiText-2 validation perplexity against generation count, vanilla and dreaming ( Td=1.1 , α=0.5 ), over five seeds; shaded regions represent 25th to 75th percentile.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: DKL(p∥q) between the true and reconstructed stationary distributions after 5000 generations over 10 seeds. Left: varying Td at α=0.8 . Right: varying α at Td=1.5 . Vanilla denotes no dreaming term. Shaded regions represent 25th to 75th percentile.
Figure 4: DKL between true and reconstructed stationary distributions after 5000 generations across the (Td,α) grid.
Figure 5: Left, right: LSTM validation loss against generation count for vanilla and dreaming ( Td=3 ) models at Ts=0.7 and Ts=1 over 10 seeds each. Shaded region represents 25th to 75th percentile.
Figure 6: WikiText-2 validation perplexity at the best primary-validation epoch, for each (Td,α) configuration tried. Only vanilla and Td=1.1 , α=0.5 are averaged over seeds, with shaded regions spanning the 25th to 75th percentile; the remaining curves are single runs and some end at generation 10.
Department of Communications and Computer Engineering, Waseda University, Tokyo, Japan · Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University, Shanghai, China · Department of Computer Science and Engineering, Waseda University, Tokyo, Japan