Generative models are increasingly trained in self-consuming iterative loops, where users curate preferred samples from model-generated candidates and the curated samples are used to train future generations of the model. Prior work has largely assumed fixed user preferences, but in practice exposure to model outputs gradually reshapes what users perceive as desirable, creating a feedback loop in which model distributions and user preferences co-evolve. We take a first step toward understanding the long-term behavior of such coupled dynamics. We show that when training relies entirely on user-curated synthetic data, iterative curation amplifies initial biases and drives the system toward one of multiple singleton equilibria in which the instance holding an initial advantage eventually dominates. In contrast, injecting reference data into training at a sufficiently large rate fundamentally changes the dynamics and yields a unique globally attracting equilibrium. Building on this insight, we study how reference-data injection can be used to control long-term outcomes, and propose an efficient algorithm that jointly selects a reference distribution and its mixing weight to steer the coupled system toward equilibria that preserve desired attributes while minimizing data collection costs.
Figures & tables
Figure 1 : Overview of the coupled model-user dynamics. Top row (model update): the current model pt generates K candidates, of which one is retained via preference-guided K -way Luce choice; the curated set Dcur is mixed with a reference dataset Dref at rate η to form Dt , on which pt+1 is trained. Bottom row (preference update): the user assesses how each candidate aligns with the current preference wt via the reward rt(x)=⟨wt,φ(x)⟩ , where φ is the feature map, and the preference vector then drifts toward the mean feature direction φˉ(pt+1) of the updated model.
Work
Preference
# Models
Core conclusion
Risk
Role of real data
Ferbach et al. [11]
Fixed
Single
Curation implicitly optimizes a fixed preference signal.
Bias amplification
Stabilizer
Wei and Zhang [46]
Fixed
Single
Adversarial curation can systematically misalign the model.
Malicious feedback
Improves robustness
Zhao et al. [59]
Heterogeneous fixed
Single
Reference mixing yields convergence and stability under noisy heterogeneous curation.
Reward perturbation instability
Stabilizer and regularizer
Zhang et al. [58]
Fixed
Multi
Benign curation may backfire under model interaction.
Cross-model coupling
Convergence aid
This work
Co-evolving
Single
Preferences co-evolve with the model, altering equilibrium structure.
Preference drift; diversity collapse
Stabilizer; Rreference data design
Table 1 : Comparison with prior work on self-consuming generative models with curation.
Figure 2 : Co-evolution dynamics under different initializations and mixing ratios. Top : per-class generation proportions. Bottom : preference vector wt .(a) The model locks into airplane, reaching > 95% by round 20. (b) Uniform reference data stabilizes the distribution near equilibrium. (c) A different initialization leads to a different outcome. (d) The system converges to a similar equilibrium, independent of initialization.
Figure 4
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5
η
air
auto
bird
cat
deer
dog
frog
horse
ship
truck
C∗
J
status
0.20
.000
.600
.000
.000
.000
.000
.100
.100
.100
.100
11.59
46,378
✓
0.30
.000
.543
.000
.000
.000
.000
.114
.114
.114
.114
13.11
45,883
✓
0.40
.000
.467
.000
.000
.000
.000
.133
.133
.133
.133
15.13
45,388
✓
0.50
.000
.360
.000
.000
.000
.000
.160
.160
.160
.160
17.96
44,892
✓
0.60
.000
.200
.000
.000
.000
.000
.200
.200
.200
.200
22.20
44,397
✓
0.67
.000
.026
.000
.000
.000
.000
.243
.243
.243
.243
26.81
44,043
η∗
Appendix
Table 3: Partial results of searching over 50 evenly spaced points in η∈[0.2,0.9] . (Setting A: protect frog/horse/ship/truck, θℓ=0.08 , c=(1,1,11,12,13,16,30,28,27,25) )
η
air
auto
bird
cat
deer
dog
frog
horse
ship
truck
C∗
J
status
0.20
.000
.525
.000
.000
.000
.000
.062
.100
.125
.187
13.26
53,030
✓
0.30
.000
.457
.000
.000
.000
.000
.071
.114
.143
.214
15.01
52,534
✓
0.40
.000
.367
.000
.000
.000
.000
.083
.133
.167
.250
17.35
52,039
✓
0.50
.000
.240
.000
.000
.000
.000
.100
.160
.200
.300
20.62
51,544
✓
0.60
.000
.050
.000
.000
.000
.000
.125
.200
.250
.375
25.52
51,049
✓
0.61
.000
.015
.000
.000
.000
.000
.130
.207
.259
.389
26.43
50,978
η∗
Appendix
Table 4: Partial results of searching over 50 evenly spaced points in η∈[0.2,0.9] (Setting B: protect frog/horse/ship/truck, θ=(0.05,0.08,0.10,0.15) , c=(1,1,11,12,13,16,30,28,27,25) ).
Figure 8
Figure 9: Generated samples from OT-CFM under w0=[1,0,0,0,…] (airplane preference), τ=5.0 , β=0.01 . Each row corresponds to a different reference data strategy; each column shows samples at the indicated round. Under pure synthetic training ( η=1.0 ), the model collapses to airplane images. Mixing with uniform reference data increasingly preserves diversity. The LP-designed reference with θℓ=0.08 for last four classes (bottom row) maintains diversity at η=0.67 .
Figure 10: Generated samples from OT-CFM under w0=[0,0,0,1,…] (cat preference), τ=5.0 , β=0.01 . Each row corresponds to a different reference data strategy; each column shows samples at the indicated round. Under pure synthetic training ( η=1.0 ), the model collapses to airplane images. Mixing with uniform reference data increasingly preserves diversity.
Figure 11: Co-evolution dynamics under different initializations and mixing ratios for DDPM model. Top : per-class generation proportions. Bottom : preference vector wt .
Figure 12: Reproducibility of DDPM co-evolution across three independent seeds ( η=1.0 , w0=[1,0,0,0,…] , τ=5.0 , K=∞ , β=0.01 ).
Figure 13: Generated samples from DDPM under w0=[1,0,0,0,…] (airplane preference), τ=5.0 , β=0.01 . Each row corresponds to a different reference data strategy; each column shows samples at the indicated round. Under pure synthetic training ( η=1.0 ), the model collapses to airplane images. Mixing with uniform reference data increasingly preserves diversity.
Recursive retraining of generative models poses a critical representation challenge: when synthetic outputs are curated based on a fixed reward signal, the model tends to collapse onto a narrow set of outputs that over-optimize that objective. Prior work suggests that such collapse is unavoidable without adding real data into the mix. We revisit this conclusion from an alignment perspective and show that collapse can be mitigated through curation based on multiple reward functions. We formalize the dynamics of recursive training under heterogeneous preferences and prove that, under certain conditions, the model converges to a stable distribution that allocates probability mass across competing high-reward regions. The limiting distribution preserves diversity and provably satisfies a weighted Nash bargaining solution, offering a formal interpretation of value aggregation in synthetic retraining loops.
Ali Falahati, Mohammad Mohammadi Amiri, Kate Larson +1
University of Waterloo · 2Rensselaer Polytechnic Institute.
Foundation models are increasingly trained on synthetic data generated by prior model iterations rather than exclusively on real data. This self-consuming training paradigm can lead to model collapse, divergence, or bias amplification. Recent work (Ferbach et al., 2024) shows that incorporating human curation into the loop can steer a self-consuming model toward human-aligned behavior, but these analyses focus on a single, isolated model that solely consumes its own outputs. In practice, however, models often interact and train on input-output pairs produced by other models. This paper studies self-consuming training in the multi-model regime. We first formalize a framework for interacting self-consuming models and characterize when the resulting dynamical system converges to a stable point. We then examine how human curation of one model affects its own alignment (self-influence) and how such effects propagate to other models (cross-influence). Unlike isolated settings where human curation always enhances model alignment, we show that cross-model interactions can dampen or even invert this effect, ultimately degrading long-term alignment.
Yang Zhang, Xiukun Wei, Xueru Zhang
Department of Computer Science and Engineering, The Ohio State University, Columbus, Ohio.
The widespread deployment of generative AI has made it increasingly difficult to distinguish synthetic content from real data. Consequently, synthetic data is inevitably incorporated into the training pipelines of future model generations, forming a self-consuming training loop. Prior work has studied the effects of such recursive self-consuming training, but analyses have largely been limited to isolated models, where a model consumes only its own synthetic data, or to simplified interactions between two models. This paper takes a first step toward understanding networked self-consuming generative models, in which multiple models consume synthetic data generated by one another through complex interaction pathways. We introduce a theoretical framework representing models as nodes in a directed, weighted graph, with edge weights governing the flow of synthetic data among models. Using this framework, we analyze the long-term behavior of networked models under retraining dynamics, establishing conditions for convergence and characterizing the resulting fixed points. We further investigate how the system's long-term stability and diversity are shaped by each model's access to real data, cross-model data consumption, and the structure of the interaction graph.