Large generative models are commonly trained on data collected from the open web, which may contain data produced by previous generations of the same model. Previous work has shown that recursively training on model-generated data can progressively degrade model performance, a phenomenon known as model collapse. Yet, generative models continue to improve dramatically over time. Existing theory suggests that this is enabled by the availability of new real data at each iteration, kept separate from generated data. However, exact separation is not realistic, and uncurated data is still commonly used. We bridge this gap by considering a more realistic setup in which real and generated data are not handled differently during training. Specifically, we present a comprehensive theoretical study of iterative estimation of the mean and covariance of a multivariate Gaussian distribution. At each iteration, samples are drawn from a mixture of the real data distribution and the estimated distribution from the previous iteration, but are used indistinguishably. We provide a single equivalent condition for consistency of the mean and covariance estimators in Euclidean norms, and convergence to zero of the expected squared 2-Wasserstein distance between the estimated and true distributions. Using this result, we show that consistency holds, i.e., model collapse is avoided, for any fixed real data fraction above zero, and even for a fraction that vanishes across iterations, provided it does not vanish too quickly. However, more real data (or data curation) remains important at practical finite times: the convergence rate of the mean estimator deteriorates as the real data fraction approaches zero. We support our results with an empirical investigation of the theoretical setup, and with experiments on iterative training of a diffusion model on CIFAR-10 that correlate with our theory.
Figures & tables
Figure 1: Convergence under constant and time-varying synthetic fractions. Top: constant α , from 0 (dark) to 1 (light, see color bar): (a) E[∥μ(t)−μ∥22] , (b) E[∥Σ(t)−Σ∥F] , and (c) E[W22(Dt,Dreal)] . Bottom: (d)–(f) the same quantities for αt=1−t1 , 1−t1 , 1−ln(t+2)1 and 1−lnln(t+16)1 , with α≡1 (dashed) for reference.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Exact E[∥μ(t)−μ∥22] against simulation. (a) Constant α (color bar) and (b) the four time-varying sequences of Figure 2 , with α≡1 (dashed). Lines are the recursion of Theorem A.11 , and circles are Monte-Carlo averages.
Figure 4: Unbiasedness of μ(t) for four constant values of α . For each coordinate i , the colored line is the Monte-Carlo average of μi(t) , the wide gray line underneath is the true value μi , and the shaded band is ±1 standard deviation of a single run’s μi(t) .
Figure 5: The bias of Σ(t) for four constant values of α . Lines are the predicted downward bias of the trace, \Tr(Ct) , computed from the recursion of Theorem A.11 ; circles are the measured \Tr(Σ)−\Tr(E[Σ(t)]) , averaged over 2×106 Monte-Carlo runs.
Figure 6: Test FID of DDPM on CIFAR-10 as a function of the pool size Nt , for mixing fractions α∈{0,0.25,0.75,1} (single seed). At each iteration, the checkpoint with the lowest validation FID is selected and its test FID is reported. The dashed line is the α=0 model at t=9 , trained on all 45,000 training images (FID 9.7 on the test set, and 5.1 with 50,000 samples against the training set; see text). Differences below ≈0.5 FID between consecutive iterations are within run-to-run variation.
Recent studies identified an intriguing phenomenon in recursive generative model training known as model collapse, where models trained on data generated by previous models exhibit severe performance degradation. Addressing this issue and developing more effective training strategies have become central challenges in generative model research. In this paper, we investigate this phenomenon within a novel framework, where generative models are iteratively trained on a combination of newly collected real data and synthetic data from the previous training step. To develop an optimal training strategy for integrating real and synthetic data, we evaluate the performance of a weighted training scheme in various scenarios, including Gaussian distribution estimation, generalized linear models, and nonparametric estimation. We theoretically characterize the impact of the mixing proportion and weighting scheme of synthetic data on the final model's performance. Our key finding is that, across different settings, the optimal weighting scheme under different proportions of synthetic data asymptotically follows a unified expression, revealing a fundamental trade-off between leveraging synthetic data and model performance. In some cases, the optimal weight assigned to real data corresponds to the reciprocal of the golden ratio. Finally, we validate our theoretical results on extensive simulated datasets and a real tabular dataset.
Hengzhi He, Shirong Xu, Guang Cheng
Department of Statistics and Data Science, University of California, Los Angeles · Wang Yanan Institute for Studies in Economics, Xiamen University
As artificial intelligence (AI)-generated content proliferates, models are increasingly trained on their own outputs, risking progressive degradation or collapse. In this article, we provide the first positive, rigorous theoretical results, to the best of our knowledge, showing that under model-agnostic mild conditions, the model converges to the true data-generating distribution. The convergence rate is the minimum of the model's intrinsic rate and the fraction of real data at each training iteration, revealing a phase transition between data-limited and model-limited regimes. We further show that, for biased real data, correcting the bias prevents the persistence and amplification of early bias over training iteration. Extensive experiments across simulations, real images and texts validate our theoretical framework, establishing quantitative conditions for long-term AI stability in contaminated environments.
Kevin Wang, Hongqian Niu, Didong Li
Department of Biostatistics, University of North Carolina at Chapel Hill
Recursive training of generative models on their own outputs can lead to model collapse, a compounding drift away from the true data distribution. Existing theoretical works bound finite-round error accumulation in the context of diffusion models, but two questions remain open:~what distribution does the recursion converge to, and how fast? We answer both, isolating a mechanism distinct from imperfect learning: even with perfect score estimation and exact sampling, the early stopping of the reverse diffusion (required for numerical stability) drives a progressive drift away from the data distribution. We prove that this recursion converges geometrically to a unique limiting distribution, which admits a closed-form characterization as an infinite mixture of increasingly Gaussian-smoothed versions of the data distribution. A Hermite spectral decomposition of this limit reveals that recursive training acts as a low-pass filter: higher-order modes, which encode fine non-Gaussian structure, are attenuated much more strongly than coarse modes. This spectral picture motivates annealed truncation schedules that progressively shrink truncation times across retraining rounds; we prove that any schedule converging to 0 asymptotically eliminates recursive compounding. Finally, we show our idealized characterization is robust: in the presence of discretization and score estimation errors, the learned distribution remains in a Wasserstein-2 ball around the ideal limit, with mode-dependent contraction rates that contract high-order errors faster than low-order ones. We validate the theory on synthetic Gaussian mixtures and CIFAR-10.
Naïl B. Khelifa, Richard E. Turner, Ramji Venkataramanan