Self-Consuming Generative Models with Co-Evolving Human Preferences

Xiukun Wei; Tian Xie; Ding Zhu; Xueru Zhang

cs.LGOct 7, 2026

Self-Consuming Generative Models with Co-Evolving Human Preferences

Authors: Xiukun Wei, Tian Xie, Ding Zhu, Xueru Zhang

Organizations: The Ohio State University

Abstract

Generative models are increasingly trained in self-consuming iterative loops, where users curate preferred samples from model-generated candidates and the curated samples are used to train future generations of the model. Prior work has largely assumed fixed user preferences, but in practice exposure to model outputs gradually reshapes what users perceive as desirable, creating a feedback loop in which model distributions and user preferences co-evolve. We take a first step toward understanding the long-term behavior of such coupled dynamics. We show that when training relies entirely on user-curated synthetic data, iterative curation amplifies initial biases and drives the system toward one of multiple singleton equilibria in which the instance holding an initial advantage eventually dominates. In contrast, injecting reference data into training at a sufficiently large rate fundamentally changes the dynamics and yields a unique globally attracting equilibrium. Building on this insight, we study how reference-data injection can be used to control long-term outcomes, and propose an efficient algorithm that jointly selects a reference distribution and its mixing weight to steer the coupled system toward equilibria that preserve desired attributes while minimizing data collection costs.

Figures & tables

Figure 1 : Overview of the coupled model-user dynamics. Top row (model update): the current model $p_{t}$ generates $K$ candidates, of which one is retained via preference-guided $K$ -way Luce choice; the curated set $\mathcal{D}_{\mathrm{cur}}$ is mixed with a reference dataset $\mathcal{D}_{\mathrm{ref}}$ at rate $\eta$ to form $\mathcal{D}_{t}$ , on which $p_{t+1}$ is trained. Bottom row (preference update): the user assesses how each candidate aligns with the current preference $w_{t}$ via the reward $r_{t}(x)=\langle w_{t},\varphi(x)\rangle$ , where $\varphi$ is the feature map, and the preference vector then drifts toward the mean feature direction $\bar{\varphi}(p_{t+1})$ of the updated model.

Work	Preference	# Models	Core conclusion	Risk	Role of real data
Ferbach et al. [11]	Fixed	Single	Curation implicitly optimizes a fixed preference signal.	Bias amplification	Stabilizer
Wei and Zhang [46]	Fixed	Single	Adversarial curation can systematically misalign the model.	Malicious feedback	Improves robustness
Zhao et al. [59]	Heterogeneous fixed	Single	Reference mixing yields convergence and stability under noisy heterogeneous curation.	Reward perturbation instability	Stabilizer and regularizer
Zhang et al. [58]	Fixed	Multi	Benign curation may backfire under model interaction.	Cross-model coupling	Convergence aid
This work	Co-evolving	Single	Preferences co-evolve with the model, altering equilibrium structure.	Preference drift; diversity collapse	Stabilizer; Rreference data design

Table 1 : Comparison with prior work on self-consuming generative models with curation.

Figure 2 : Co-evolution dynamics under different initializations and mixing ratios. Top : per-class generation proportions. Bottom : preference vector

w_{t}

.(a) The model locks into airplane, reaching

>

95% by round 20. (b) Uniform reference data stabilizes the distribution near equilibrium. (c) A different initialization leads to a different outcome. (d) The system converges to a similar equilibrium, independent of initialization.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

$\eta$	air	auto	bird	cat	deer	dog	frog	horse	ship	truck	$C^{*}$	$J$	status
0.20	.000	.600	.000	.000	.000	.000	.100	.100	.100	.100	11.59	46,378	✓
0.30	.000	.543	.000	.000	.000	.000	.114	.114	.114	.114	13.11	45,883	✓
0.40	.000	.467	.000	.000	.000	.000	.133	.133	.133	.133	15.13	45,388	✓
0.50	.000	.360	.000	.000	.000	.000	.160	.160	.160	.160	17.96	44,892	✓
0.60	.000	.200	.000	.000	.000	.000	.200	.200	.200	.200	22.20	44,397	✓
0.67	.000	.026	.000	.000	.000	.000	.243	.243	.243	.243	26.81	44,043	${\eta^{*}}$

Appendix

Table 3: Partial results of searching over 50 evenly spaced points in

\eta\in[0.2,0.9]

. (Setting A: protect frog/horse/ship/truck,

\theta_{\ell}=0.08

c=(1,1,11,12,13,16,30,28,27,25)

)

$\eta$	air	auto	bird	cat	deer	dog	frog	horse	ship	truck	$C^{*}$	$J$	status
0.20	.000	.525	.000	.000	.000	.000	.062	.100	.125	.187	13.26	53,030	✓
0.30	.000	.457	.000	.000	.000	.000	.071	.114	.143	.214	15.01	52,534	✓
0.40	.000	.367	.000	.000	.000	.000	.083	.133	.167	.250	17.35	52,039	✓
0.50	.000	.240	.000	.000	.000	.000	.100	.160	.200	.300	20.62	51,544	✓
0.60	.000	.050	.000	.000	.000	.000	.125	.200	.250	.375	25.52	51,049	✓
0.61	.000	.015	.000	.000	.000	.000	.130	.207	.259	.389	26.43	50,978	$\eta^{*}$

Appendix

Table 4: Partial results of searching over 50 evenly spaced points in

\eta\in[0.2,0.9]

(Setting B: protect frog/horse/ship/truck,

\theta=(0.05,0.08,0.10,0.15)

c=(1,1,11,12,13,16,30,28,27,25)

Panel of Figure 9 — Figure 9: Generated samples from OT-CFM under $w_{0}=[1,0,0,0,\dots]$ (airplane preference), $\tau=5.0$ , $\beta=0.01$ . Each row corresponds to a different reference data strategy; each column shows samples at the indicated round. Under pure synthetic training ( $\eta=1.0$ ), the model collapses to airplane images. Mixing with uniform reference data increasingly preserves diversity. The LP-designed reference with $\theta_{\ell}=0.08$ for last four classes (bottom row) maintains diversity at $\eta=0.67$ .

Panel of Figure 10 — Figure 10: Generated samples from OT-CFM under $w_{0}=[0,0,0,1,\dots]$ (cat preference), $\tau=5.0$ , $\beta=0.01$ . Each row corresponds to a different reference data strategy; each column shows samples at the indicated round. Under pure synthetic training ( $\eta=1.0$ ), the model collapses to airplane images. Mixing with uniform reference data increasingly preserves diversity.

Figure 11: Co-evolution dynamics under different initializations and mixing ratios for DDPM model. Top : per-class generation proportions. Bottom : preference vector

w_{t}

Figure 12: Reproducibility of DDPM co-evolution across three independent seeds (

\eta=1.0

w_{0}=[1,0,0,0,\dots]

\tau=5.0

K=\infty

\beta=0.01

Panel of Figure 13 — Figure 13: Generated samples from DDPM under $w_{0}=[1,0,0,0,\dots]$ (airplane preference), $\tau=5.0$ , $\beta=0.01$ . Each row corresponds to a different reference data strategy; each column shows samples at the indicated round. Under pure synthetic training ( $\eta=1.0$ ), the model collapses to airplane images. Mixing with uniform reference data increasingly preserves diversity.

Explore similar work

Curated Synthetic Data Doesn't Have to Collapse: A Theoretical Study of Generative Retraining with Pluralistic Preferences
May 8, 2026Ali Falahati, Mohammad Mohammadi Amiri, Kate Larson +1Synthetic Data Generative Models
University of Waterloo · 2Rensselaer Polytechnic Institute.
When and How Human Curation Backfires: Preference Alignment under Multi-Model Self-Consuming Loop
May 28, 2026Yang Zhang, Xiukun Wei, Xueru ZhangPreference Alignment Reinforcement Learning From Human Feedback
Department of Computer Science and Engineering, The Ohio State University, Columbus, Ohio.
Stability and Diversity of Networked Self-Consuming Generative Ecosystems
Oct 7, 2026Xiukun Wei, Yang Zhang, Xueru Zhang
The Ohio State University

May 8, 2026, cs.LGJ/K move · Enter open · S save

Curated Synthetic Data Doesn't Have to Collapse: A Theoretical Study of Generative Retraining with Pluralistic Preferences

Ali Falahati, Mohammad Mohammadi Amiri, Kate Larson, Lukasz Golab

University of Waterloo · 2Rensselaer Polytechnic Institute.

Synthetic Data Generative Models Retraining Collapse Theory

Open paper AskPDF arXiv

Recursive retraining of generative models poses a critical representation challenge: when synthetic outputs are curated based on a fixed reward signal, the model tends to collapse onto a narrow set of outputs that over-optimize that objective. Prior work suggests that such collapse is unavoidable without adding real data into the mix. We revisit this conclusion from an alignment perspective and show that collapse can be mitigated through curation based on multiple reward functions. We formalize the dynamics of recursive training under heterogeneous preferences and prove that, under certain conditions, the model converges to a stable distribution that allocates probability mass across competing high-reward regions. The limiting distribution preserves diversity and provably satisfies a weighted Nash bargaining solution, offering a formal interpretation of value aggregation in synthetic retraining loops.

Abstract

Explore similar work

Curated Synthetic Data Doesn't Have to Collapse: A Theoretical Study of Generative Retraining with Pluralistic Preferences

When and How Human Curation Backfires: Preference Alignment under Multi-Model Self-Consuming Loop

Stability and Diversity of Networked Self-Consuming Generative Ecosystems