Statistical Analysis of Synthetic Data with Verification: A Linear Regression Framework

Heejune Sheen; Harrison H. Zhou

stat.MLOct 29, 2026

Statistical Analysis of Synthetic Data with Verification: A Linear Regression Framework

Authors: Heejune Sheen, Harrison H. Zhou

Organizations: Department of Statistics and Data Science Yale University New Haven, CT 06511 USA

Abstract

Synthetic data are widely used for training modern AI systems. However, their statistical understanding remains limited. It is well recognized in statistics that repeated self-training on synthetic data can amplify errors inherited from pre-trained models, leading to model collapse. Verification plays a crucial role in selecting high-quality synthetic data to help prevent model collapse and improve model performance. We study verified self-training in a linear regression framework, where a pre-trained estimator generates synthetic data with injected generation noise and a verifier selects samples according to a verification threshold. Post-training on the accepted samples updates either all coordinates (full fine-tuning) or only an active subset (partial fine-tuning). Our analysis reveals three potentially counterintuitive effects. We establish that tighter verification does not always lead to lower prediction risk. We also find that a suitable level of injected generation noise can reduce prediction risk, while excessive noise can increase it. Moreover, full and partial fine-tuning can respond differently to the same verification rule, since pre-training error in the frozen coordinates affects both sample selection and post-training risk. We analyze these effects by deriving finite sample risk bounds for four separate regimes determined by the relative scales of the verification threshold, generation noise, and pre-training error. The bounds precisely characterize how verification affects prediction risk across these regimes. In particular, we quantify how the generation noise, the effective accepted sample size, and the covariance of the accepted covariates affect prediction risk. The analysis extends to iterative verified full fine-tuning, where sufficiently tight verification reduces the initial error contribution at a geometric rate.

Figures & tables

Figure 1: Excess prediction risk under verified post-training. Panel (a) displays risk against the verification threshold under partial fine-tuning, panel (b) shows risk against generation noise under full fine-tuning. In panel (c), we compare full and partial fine-tuning across verification thresholds when generation noise is small. For partial fine-tuning, the excess risk excludes the direct contribution from the frozen coordinates.

Panel of Figure 2 — Figure 2: Post-training with verified synthetic data.

Figure 3: Full fine-tuning with

s=d=20

n=2000

, and

m=3\times 10^{6}

. Simulation estimates of

\mathcal{V}_{1}+\mathcal{V}_{2}

are plotted against the verification threshold

\delta

, with

\tilde{\sigma}

fixed in each panel. Tighter verification reduces the variance in Regime I even when

\delta

exceeds

\sqrt{d/n}

. The variance is smallest in Regime II, increases in Regime III, and remains near

d/n

in Regime IV, where tight verification provides little improvement when generation noise is too small.

Figure 4: Full fine-tuning with

s=d=20

n=2000

, and

m=3\times 10^{6}

. Simulation estimates of

\mathcal{V}_{1}+\mathcal{V}_{2}

are plotted against the generation noise

\tilde{\sigma}

under (a) tight and (b) loose verification. In both cases, the variance is minimized at an intermediate generation noise level. Nearly deterministic generation leaves the variance near

d/n

, whereas excessive generation noise increases it.

Figure 5: Partial fine-tuning with

s=10

d=20

n=2000

, and

m=3\times 10^{6}

. Simulation estimates of

\mathcal{V}_{1}+\mathcal{V}_{2}

are plotted against

\delta

, with

\tilde{\sigma}

fixed in each panel. Regime I exhibits the same reduction as under full fine-tuning, while the frozen coordinates change the behavior in the Regimes II and III. The U-shaped graph in Regime II reflects the tradeoff between the error in the accepted responses and the frozen coordinate contribution, and the variance remains near

s/n

in Regimes III and IV.

Figure 6: Partial fine-tuning with

s=10

d=20

n=2000

, and

m=3\times 10^{6}

. Simulation estimates of

\mathcal{V}_{1}+\mathcal{V}_{2}

are plotted against

\tilde{\sigma}

under (a) tight and (b) loose verification. In both cases, sufficiently large generation noise is needed to reduce the variance below

s/n

, while excessive generation noise increases it. The variance is therefore minimized at an intermediate generation noise level.

Regime	Scales	$\mathcal{V}_{1}$ terms	$\mathcal{V}_{2}$ terms
No verification	all generated samples	$\displaystyle s/n$	$\displaystyle s\tilde{\sigma}^{2}/m$
I	$\sqrt{d/n}\lesssim\delta$	$\displaystyle\frac{s}{n}\min\left(1,(\delta/\tilde{\sigma})^{4}\right)$	$\displaystyle\frac{s\min(\delta^{2},\tilde{\sigma}^{2})}{mp_{\mathrm{I}}}$
II	$\delta\lesssim\sqrt{d/n}\lesssim\tilde{\sigma}$	$\displaystyle\frac{s}{n}\left(\left(\frac{d-s}{n\tilde{\sigma}^{2}+d-s}\right)^{2}+(\delta/\tilde{\sigma})^{4}\right)$	$\displaystyle\frac{s}{mp_{\mathrm{II}}}\left(\delta^{2}+\frac{\tilde{\sigma}^{2}(d-s)}{n\tilde{\sigma}^{2}+d-s}\right)$
III	$\delta\lesssim\tilde{\sigma}\lesssim\sqrt{d/n}$	$\displaystyle\frac{s}{n}\left(\left(\frac{d-s}{n\tilde{\sigma}^{2}+d-s}\right)^{2}+(\delta/\tilde{\sigma})^{4}\right)$	$\displaystyle\frac{s}{mp_{\mathrm{III}}}\left(\delta^{2}+\frac{\tilde{\sigma}^{2}(d-s)}{n\tilde{\sigma}^{2}+d-s}\right)g_{d-s,d}(\sqrt{d/n}/\tilde{\sigma})$
IV	$\tilde{\sigma}\lesssim\delta\lesssim\sqrt{d/n}$	$\displaystyle s/n$	$\displaystyle\frac{s\tilde{\sigma}^{2}}{mp_{\mathrm{IV}}}g_{d-s,d}(\sqrt{d/n}/\delta)$

Table 1: Leading active coordinate variance terms for partial fine-tuning

(s<d)

Figure 7: Test risk across verification thresholds and generation noise scales, with

n=2048

d=762

, and

m=32{,}000

. Panel (a) varies the threshold at

\tilde{\sigma}\approx 1.58\sqrt{R_{\mathrm{pre}}}

, while panel (b) presents the best average test risk ratio among the 11 examined thresholds for each generation noise scale. Each setting averages 40 runs. The direct fitting reference fits the linear model directly on all

7229

target human labels.

Figure 8: Test risk across verification thresholds and generation noise scales at a fixed accepted sample size, with $n=2048$ and $d=762$ . Across 16 attempted runs per setting, every fitted estimator uses 3048 accepted samples, and the star marks the minimum test risk ratio in each panel.

Synthetic Data Selection

Synthetic Data Selection studies how to choose generated examples that improve downstream model training while limiting noise, redundancy, and cost.

See all 33 papers

Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence
Oct 18, 2025Bingji Yi, Qiyuan Liu, Yuwei Cheng +1Synthetic Data Pretraining Model Collapse
Independent Researcher. Work done while visiting UChicago CS. · Work done while visiting UChicago CS. · Department of Statistics, University of Chicago. +1
Learning with Synthetic Data via SGD in High-Dimensional Linear Regression
Sep 9, 2026Jichu li, Difan ZouSynthetic Data Pretraining High-Dimensional Linear Regression
Renmin University of China · The University of Hong Kong
Training-Aware Target Coverage for Synthetic Data Selection
Sep 30, 2026Yang Ba, Michelle V. Mancenido, Rong PanLLM Fine-Tuning Data Selection
School of Computing and Augmented Intelligence, Arizona State University · School of Mathematical and Natural Sciences, Arizona State University

Oct 18, 2025J/K move · Enter open · S save

Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence

Bingji Yi, Qiyuan Liu, Yuwei Cheng, Haifeng Xu

Independent Researcher. Work done while visiting UChicago CS. · Work done while visiting UChicago CS. · Department of Statistics, University of Chicago.+1

Synthetic Data Pretraining Model Collapse Synthetic Data Generation for Language Models Synthetic Data Evaluation

Open paper AskPDF arXiv

Synthetic data has been increasingly used to train frontier generative models. However, recent studies raise key concerns that iteratively retraining a generative model on its self-generated synthetic data may keep deteriorating model performance, a phenomenon often coined model collapse. In this paper, we investigate ways to modify the synthetic retraining process to avoid model collapse, and even possibly help reverse the trend from collapse to improvement. Our key finding is that by injecting information through an external synthetic data verifier, whether a human or a better model, synthetic retraining will not cause model collapse. Specifically, we situate our theoretical analysis in the fundamental linear regression setting, showing that verifier-guided retraining can yield near-term improvements, but ultimately drives the parameter estimate to the verifier's "knowledge center" in the long run. Our theory further predicts that, unless the verifier is perfectly reliable, these early gains will plateau and may even reverse. Indeed, our experiments across linear regression, Variational Autoencoders (VAEs) trained on MNIST, and fining-tuning SmolLM2-135M on the XSUM task confirm these theoretical insights.

Abstract

Synthetic Data Selection

Explore similar work

Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence

Learning with Synthetic Data via SGD in High-Dimensional Linear Regression

Training-Aware Target Coverage for Synthetic Data Selection