Epidemiology of Model Collapse: Modeling Synthetic Data Contamination via Bilayer SIR Dynamics

Xiangyu Wang

cs.CLApr 14, 2026

Epidemiology of Model Collapse: Modeling Synthetic Data Contamination via Bilayer SIR Dynamics

Authors: Xiangyu Wang

Abstract

Training on synthetic data causes model collapse, but existing analyses treat this as single-chain degradation. In reality, the AI ecosystem involves cross-contamination: models ingest synthetic data from other models, produce new synthetic text, and contaminate shared corpora. We propose a bilayer coupled SIR/SIRS framework -- a phenomenological mean-field model treating data corpora and AI models as two interacting populations, each with susceptible, infected, and recovered compartments linked by cross-layer transmission. The SIRS variant (our primary recommendation) incorporates immunity waning, reflecting that filtered corpora and retrained models remain susceptible to re-contamination. We derive the basic reproduction number $R_0 = \sqrt{β_D β_M / [(γ_D+μ_D)(γ_M+μ_M)]}$ via the Next Generation Matrix and apply standard epidemic threshold results to the bilayer system. Illustrative scenario-based calibration from public AI text prevalence data yields supercritical dynamics ( $R_0 > 1$ ) across three scenarios; Sobol sensitivity analysis identifies synthetic-text detection as the highest-leverage parameter. A bipartite-network agent-based model confirms mean-field consistency ( $R^2 > 0.96$ ) for dense networks but degrades under heterogeneity. GPT-2 contamination chain experiments (192 runs across WikiText and Shakespeare) show dose-response degradation and diversity loss qualitatively consistent with the threshold picture. Matched-budget source-diversity experiments (1,088 runs) provide suggestive evidence that multi-source mixing modestly attenuates collapse, but the effect vanishes at lower contamination fractions. Intervention analysis identifies detection-based filtering and herd immunity as the highest-leverage strategies.

Figures & tables

Layer	Compartment	Ecosystem Interpretation	Key Parameter
Data ( $D$ )	$S_{D}$	Clean data corpora	$\Lambda_{D}$ : new data creation rate
	$I_{D}$	Contaminated corpora	$\beta_{D}$ : contamination ingestion rate
	$R_{D}$	Cleaned/filtered corpora	$\gamma_{D}$ : detection & removal rate
Model ( $M$ )	$S_{M}$	Clean-trained models	$\Lambda_{M}$ : new model deployment rate
	$I_{M}$	Contaminated models	$\beta_{M}$ : contaminated training rate
	$R_{M}$	Retrained/clean models	$\gamma_{M}$ : retraining rate

Table 1: Compartment mapping between the bilayer SIR model and the AI ecosystem.

Panel of Figure 1 — Figure 1: Bilayer SIR schematic. Data corpora (top, blue) and AI models (bottom, orange) form two coupled epidemic layers. Solid arrows denote within-layer transitions; dashed red arrows denote cross-layer infection. Contaminated models ( $I_{M}$ ) inject synthetic content into data corpora, while contaminated data ( $I_{D}$ ) infects models during training.

Figure 2: ODE trajectory under baseline parameters (

R_{0}=2.62

). Both data and model infection fractions converge to a positive endemic equilibrium, consistent with Proposition 3 . The initial transient captures the early growth phase; damped oscillations are visible before settling.

Scenario	$\beta_{D}$	$\gamma_{D}$	$\beta_{M}$	$\gamma_{M}$	$R_{0}$
Optimistic	0.201	0.200	0.240	0.150	1.10
Baseline	0.216	0.099	0.340	0.060	2.62
Pessimistic	0.231	0.020	0.380	0.020	6.63

Table 2: Calibration scenarios. All scenarios yield

R_{0}>1

. These are illustrative estimates, not precise ecosystem measurements. Data sources: AI text prevalence [ 22 , 11 ] ; detection accuracy [ 21 ] ; training practices from public model documentation.

Figure 3:

R_{0}

as a function of data infection rate (

\beta_{D}

) and model infection rate (

\beta_{M}

), with other parameters at baseline values. The thick contour marks

R_{0}=1

. All three calibration scenarios (markers) lie in the supercritical region, though the optimistic scenario is near the boundary.

Figure 4: ODE (solid) vs. ABM ensemble mean (dashed,

\pm 1

std shaded) for infection fractions at

p_{\text{edge}}=0.8

R^{2}=0.976

(data),

0.968

(model).

Domain	$\alpha$	G0 PPL	G7 PPL	Excess G1	Excess G7	$r$	95% CI	$\Delta\mathrm{AIC}$
WikiText	0.00	33.52	33.47	—	0.00	0.000	$[0.000,0.000]$	$+2.0$
	0.25	33.52	34.40	$+0.80$	$+0.93$	0.003	$[0.0022,0.0029]$	$-12.0$
	0.50	33.52	35.91	$+1.87$	$+2.44$	0.007	$[0.0065,0.0074]$	$-16.8$
	0.75	33.52	38.55	$+3.68$	$+5.08$	0.014	$[0.014,0.015]$	$-19.2$
	1.00	33.52	126.92	$+15.49$	$+93.45$	0.183	$[0.180,0.186]$	$-60.2$
Shakespeare	0.00	33.66	33.63	—	0.00	—	—	—

Table 3: Empirical results: excess perplexity over the

\alpha=0

control (WikiText and Shakespeare). Growth rate

r

is the slope of excess log-perplexity ratio

\log(\mathrm{PPL}_{\alpha}/\mathrm{PPL}_{\text{control}})

vs. generation.

\Delta\mathrm{AIC}

compares linear growth vs. plateau models (negative favors growth). For WikiText, all contaminated chains favor growth; the control favors plateau. Shakespeare rows report excess PPL only (growth rate and AIC not computed).

Figure 5: Matched-budget source-diversity experiment (

\alpha{=}1.0

, pool size fixed at

n{=}2{,}000

). Left : perplexity trajectories for

K\in\{1,3,5\}

and the

\alpha{=}0

control (mean

\pm

std, 8 seeds). Right : G7 excess PPL vs.

K

. Self-training (

K{=}1

) degrades most; multi-source (

K{>}1

) shows modest attenuation (

{\sim}2

PPL,

d\approx 0.8

) that saturates by

K{=}3

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: SIRS oscillatory dynamics for a representative configuration (

\delta=0.1

R_{0}=2.62

). Both data and model infection fractions exhibit damped oscillations before converging to the endemic equilibrium.

Parameter	Description	Baseline	Unit
$\Lambda_{D}$	Data creation rate	5.0	corpora/month
$\Lambda_{M}$	Model deployment rate	3.0	models/month
$\beta_{D}$	Data contamination rate	0.216	month -1
$\beta_{M}$	Model infection rate	0.340	month -1
$\gamma_{D}$	Data recovery (detection) rate	0.099	month -1
$\gamma_{M}$	Model recovery (retraining) rate	0.060	month -1

Appendix

Table 4: Complete parameter table with units and baseline values.

Parameter	$S_{1}$	$S_{T}$	$S_{T}-S_{1}$ (interactions)
$\gamma_{D}$	0.272	0.324	0.053
$\beta_{D}$	0.248	0.280	0.032
$\gamma_{M}$	0.220	0.277	0.057
$\beta_{M}$	0.175	0.204	0.029
$\mu_{M}$	0.023	0.032	0.009
$\mu_{D}$	0.014	0.022	0.008

Appendix

Table 11

Figure 7: Sobol total-order sensitivity indices for

R_{0}

. Data recovery rate (

\gamma_{D}

) has the highest leverage.

Figure 8: ABM threshold verification: sub-critical vs. super-critical classification across 20 parameter configurations. 18/20 are correctly identified; the two errors occur near

R_{0}\approx 1

Figure 9: Perplexity over 8 generations for WikiText contamination chains at five contamination fractions

\alpha

. Lines show means across 3 seeds; shaded regions are

\pm 1

std. The

\alpha=0

control (gray) remains flat. At

\alpha=1.0

, perplexity grows from 33.52 to 126.92 (

+93.45

excess). At

\alpha<1

, near-plateau dynamics emerge (

r<0.015

Hyperparameter	Value
Base model	GPT-2 (124M parameters)
Training samples per generation	2000 (WikiText), 1500 (Shakespeare)
Max sequence length	128 tokens
Epochs	3
Batch size	8
Learning rate	$5\times 10^{-5}$

Appendix

Table 15

Seed	G0	G1	G3	G5	G7
42	33.53	49.08	81.21	105.83	128.23
123	33.47	48.75	81.49	106.46	121.94
456	33.56	49.13	82.41	108.61	130.59
Mean $\pm$ std	33.52 $\pm$ 0.04	48.99 $\pm$ 0.17	81.70 $\pm$ 0.51	106.97 $\pm$ 1.19	126.92 $\pm$ 3.65

Appendix

Table 5: WikiText perplexity per seed at selected generations for

\alpha=1.0

Figure 10: Cross-domain comparison of contamination chain dynamics. WikiText (left) and Shakespeare (right) show qualitatively identical patterns: flat control, mild degradation at

\alpha=0.5

, and strong supercritical growth at

\alpha=1.0

. Shakespeare exhibits even stronger degradation under pure contamination (6.6

\times

vs. 3.8

\times

), likely due to lower redundancy in literary text.

Figure 11: Distinct-2 diversity across generations for

\alpha=1.0

(WikiText). Diversity drops from 0.68 to 0.38, providing an independent degradation signal correlated with perplexity increase.

Figure 12: Training loss convergence diagnostics across generations and contamination fractions, confirming that all models converge within the 3-epoch training budget.

Strategy	Mechanism	Parameter	Best $R_{0}$	$R_{0}<1$ ?
Watermark + Filtering	Detect & remove synthetic text	$\gamma_{D}\!\uparrow,\,\gamma_{M}\!\uparrow$	${<}1$ at 63%	Yes
Herd Immunity	Fraction trained on clean data	$\beta_{D},\beta_{M}\!\downarrow$	${<}1$ at 63% cov.	Yes
Superspreader Control	Limit high-traffic model output	$\beta_{M}\!\downarrow$	1.015	No
Open Detection Stds	Standardize detection tools	$\gamma_{D}\!\uparrow,\,\gamma_{M}\!\uparrow$	1.142	No
Output Restriction	Rate-limit synthetic generation	$\beta_{D}\!\downarrow$	1.172	No
Provenance Tracking	Data lineage metadata	$\gamma_{D}\!\uparrow,\,\beta_{D}\!\downarrow$	1.340	No

Appendix

Table 6: Intervention strategies and their effects on

R_{0}

(from baseline

R_{0}=2.62

). Only watermark-based filtering and herd immunity achieve

R_{0}<1

as single strategies.

Figure 13:

R_{0}

as a function of intervention intensity for all six strategies. Only watermark-based filtering and herd immunity cross the

R_{0}=1

threshold.

Figure 14: Pareto frontier: illustrative cost vs.

R_{0}

reduction across single-strategy sweeps (6 strategies

\times

20 intensity levels

=

120 points). Only watermark-based filtering and herd immunity achieve

R_{0}<1

alone. Combined interventions (15 pairs

\times

9 intensity combinations

=

135 evaluations) are analyzed separately in Section 7 .

Figure 15: ODE infection trajectories under the three calibration scenarios. The pessimistic scenario (

R_{0}=6.63

) converges rapidly to a high endemic level; the optimistic scenario (

R_{0}=1.10

) shows slow, marginal growth.

Figure 16: Bifurcation diagram showing endemic equilibrium infection level as a function of

R_{0}

. The transcritical bifurcation at

R_{0}=1

is clearly visible: the DFE (zero infection) is stable for

R_{0}<1

and unstable for

R_{0}>1

, where the endemic branch emerges.

Model Collapse

Model collapse studies how recursive training or feedback can erode a model’s diversity, accuracy, or fidelity to the target distribution.

See all 41 papers

Sep 11, 2026

SynthSentry: Detecting Synthetic Data Contamination in Language Model Training Data

Large language models trained recursively on their own or other models' outputs undergo model collapse, in which distributional tails and factual accuracy deteriorate while fluency survives. Prior work diagnoses collapse after training; the actionable problem is screening a corpus of unknown provenance before training. We introduce SynthSentry, a corpus-level, model-agnostic contamination signal requiring no access to the generating model, no generation history, and no synthetic labels. The score is a distributional divergence over three statistics: lexical diversity collapse, n-gram tail truncation, and perplexity variance across reference models. We evaluate on corpora contaminated by small open-weight generators and an instruction-tuned open-weight model under a leave-one-generator-out protocol. A domain-stratified study measures false positives on naturally repetitive human text (legal, clinical, source code). The score ranks corpora by severity with little loss when whole generator families are held out. Per-domain calibration holds near its nominal false-positive budget once covariance shrinkage and a bootstrap threshold replace a naive quantile, which runs four times over budget. A downstream fine-tuning check showed no contamination-driven accuracy deficit at our scale, so whether pruning recovers one remains open; the same run shows over-pruning risk once pruning exceeds the true contamination fraction. We frame screening as a data-curation defense rather than a post-hoc diagnosis and release the scoring toolkit. All results are small-scale; scope is English-language, batch-mode corpus screening. Contamination sources are single-generation or hand-authored rather than recursively generated, so results speak to synthetic contamination generally and not to recursion depth.

Praveen Kumar Myakala, Ravichandra Namburi, Sowmya Keragodu Jayaramu +1

Independent AI Researcher Melissa, USA · Independent Researcher Frisco, USA · Independent Researcher Austin, USA

AI-Generated Text Detection Training Data Curation Data Contamination in Language Models

Ask PDF

May 19, 2026

The Economics of Model Collapse: Equilibrium, Welfare, and Optimal Provenance Subsidies in Synthetic Data Markets

Generative artificial intelligence is rapidly transforming the supply side of training data: an increasing share of new tokens, images, and structured records is produced by previous-generation models rather than by human originators. Recursive training on such synthetic content induces a measurable and often irreversible loss of distributional fidelity, a phenomenon known as model collapse. We develop the first unified microeconomic theory of synthetic data markets under model collapse. We introduce the Synthetic Data Contamination Equilibrium (SDCE), prove existence and generic uniqueness, derive a welfare decomposition W = W_prod + W_cons - L_coll - L_info, establish a Wasserstein-gradient-flow mean-field collapse limit, prove an impossibility of information-constrained implementation, and obtain closed-form expressions for the welfare-maximizing provenance subsidy s* = KL(q||p)/(2 kappa) and the welfare-maximizing watermark strength w* = (1 - psi) KL(q||p)/(2 kappa psi). We prove an information-theoretic Cramer-Rao lower bound on any provenance estimator using only producer-side observations and show that the Provenance-Market Iterative Retraining (PMIR) algorithm attains this bound up to constants while converging to an epsilon-SDCE in O(epsilon^-2 log T) iterations. A reduced-form OLS estimation on a C4-synthetic benchmark over ten retraining generations yields a collapse-rate coefficient b-hat = 0.181 (HAC s.e. 0.024), within one standard error of the structural prediction 0.183. Calibrated experiments raise generation-ten model quality by 23.1 percent over the unregulated benchmark while lowering the 2-Wasserstein drift on a held-out diversity probe from 0.318 to 0.142. Scaling experiments over generations t in {1,...,10} recover a logarithmic-in-t collapse law log Q_t = log Q_0 - 0.183 t rho^2 with R^2 = 0.962.

Gustav Olaf Yunus Laitinen-Fredriksson Lundström-Imanov

Department of Economics Stockholm University Stockholm, Sweden

Data Provenance Dataset Watermarking Synthetic Data Generation Model Collapse

Ask PDF

Sep 10, 2026

A Dominant Supplier Slows Recursive Drift More Than It Steers It

More and more of the text future language models learn from is written by a few of today's models. If one supplier writes most of a shared corpus, does it pull the models trained on it toward its own writing, or change how fast they drift? We retrain eight open models from their base weights on a shared pool of each other's text for five generations, varying the part written by one model, Phi-2, from an equal share to 90%. The models drift together toward a style with fewer function words, and none starts repeating itself. No share of Phi-2 brings the other models closer to its text than the equal share does. We split each ecosystem's separation from the equal-share one into a delay along its route and a departure from that route, both counted beyond the difference between two equal-share runs. With Phi-2 at 90%, delay outweighs departure 72 to 28 and 64 to 36 in two runs, and the ecosystem falls 2.7 and 2.5 generations behind. With Phi-2 at half the pool the two parts are about equal. When SmolLM2 or Qwen3-1.7B writes half instead, the ecosystem slows less or not at all. The departure leans toward Phi-2 more as its share grows, but more than toward every other model only at 90%. Human text filling a quarter or half of the pool slows the models along the same route.

Yangze Liu, Zhongyi Han

Shandong University

Drifting Models Recursive Language Models

Ask PDF

Abstract

Model Collapse

Explore similar work

SynthSentry: Detecting Synthetic Data Contamination in Language Model Training Data

The Economics of Model Collapse: Equilibrium, Welfare, and Optimal Provenance Subsidies in Synthetic Data Markets

A Dominant Supplier Slows Recursive Drift More Than It Steers It