Training-Aware Target Coverage for Synthetic Data Selection

Yang Ba; Michelle V. Mancenido; Rong Pan

cs.LGSep 30, 2026

Training-Aware Target Coverage for Synthetic Data Selection

Authors: Yang Ba, Michelle V. Mancenido, Rong Pan

Organizations: School of Computing and Augmented Intelligence, Arizona State University · School of Mathematical and Natural Sciences, Arizona State University

Abstract

Synthetic data are increasingly used to scale LLM training, yet more synthetic data do not necessarily produce better models. Useful synthetic data must add information relevant to the target task without introducing errors that offset their benefit, and the value of an example can change as the training set grows. We develop a linear theory that characterizes this tradeoff and determines where synthetic data are useful, how much should be added, and the marginal value of adding one example to an existing set. The analysis shows the conditions when input coverage alone is sufficient and when synthetic errors must also be considered. Guided by these results, we introduce \emph{Training-Aware Target Coverage} (TATC), a synthetic data selection method for LLM fine-tuning. TATC identifies candidates whose training effects are beneficial to the target task and selects among them to expand coverage of target-relevant directions not already represented by the available data. Experiments on text and image data verify the linear theory. With mathematical reasoning tasks, TATC selects synthetic solutions for fine-tuning Qwen2.5-Math-1.5B-Instruct and outperforms alternative synthetic-data selection methods on GSM8K across selection budgets. In summary, we provide a principled approach to synthetic data selection by quantifying and maximizing its value to the target task.

Figures & tables

Figure 1: Risk and value of synthetic data. The theory in one view. (a) A synthetic set is valuable when target-relevant input information outweighs random label noise and systematic label error. (b) This tradeoff determines whether the optimal amount along a direction is zero, finite, or unbounded. (c) Candidate value depends on both input coverage relative to the current set and label error.

Figure 2: Candidate value and synthetic-data amount with real inputs and controlled targets. (a) Candidate-level test on SetFit/subj. Coverage alone does not distinguish beneficial from harmful candidates, whereas the complete marginal closely predicts the risk reduction from explicit refitting. (b) Amount selection with ResNet-50 features on CIFAR-10. The amount predicted from a labeled real probe closely tracks the test-optimal amount and decreases as pseudo-label reliability falls, while the input-only rule selects the full candidate pool. Gray segments indicate amounts within 1% of the test minimum.

Figure 3: (a) Exact match at

B=128

on the GSM8K holdout. The dashed line marks the checkpoint trained on real data. (b) Exact match across selection budgets. Each method selects nested subsets, with each subset trained independently from the same checkpoint. Error bars show one standard deviation over three independent training runs.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Partition	Examples	Role
$D_{\mathrm{rep}}$	1,500	Fit PCA and the reference linear target
$D_{\mathrm{cal}}$	500	Estimate the residual variance scale
$D_{r}$	1,200	Define the real training information
$D_{q}$	1,000	Estimate the target moment $\widehat{\bm{Q}}$
$\mathcal{P}_{s}$	3,800	Supply candidate inputs

Appendix

Table 1: Data partitions for the text experiment.

Score	Metric	Value
Coverage	AUROC	0.519
Complete marginal	AUROC	0.99
Complete marginal	Correlation	0.99
Complete marginal	Sign agreement	99.2%

Appendix

Table 2: Agreement with observed risk reductions in the text experiment.

Label model	Label accuracy	Predicted ratio	Test optimum ratio
12,000 training examples	0.889	7.0	8.0
400 training examples	0.836	0.90	0.95
12 coordinates removed	0.604	0	0

Appendix

Table 3: Mean label accuracy and ratios of synthetic to real data for the three conditions in Figure 2 (b).

Linear quantity or role	TATC quantity	Status	Correct interpretation
Real information $\bm{A}_{r}$	$\widehat{\bm{A}}_{0}$	Approximation	Summarizes prompt directions already represented by real data; $\rho\bm{I}$ is numerical stabilization.
Target moment $\bm{Q}$	$\widehat{\bm{Q}}$	Sample estimate	Estimates target relevance only in the checkpoint prompt representation.
Selected information $\bm{A}_{S}$	$\sum_{i\in S}\bm{\phi}_{i}\bm{\phi}_{i}^{\top}$	Exact in representation	Each selected input adds a rank-one prompt-feature contribution.
Clean coverage marginal $q_{j}/c_{j}$	$a_{j}(S)$	Exact in representation	Exact decrease in the downstream-weighted inverse-trace criterion.
Systematic label error $(\bm{d}_{S},\bm{u})$	Response-dependent gradient and AdamW update	Approximation	For squared loss, response error changes the gradient by $-(\bm{x}^{\top}\bm{\delta})\bm{x}$ ; in an LLM the full response changes a nonlinear gradient.
Exact marginal value $\Delta_{j}(S)$	Quadratic training score $h_{j}$	Surrogate	Ranks update differences using a quadratic model of probe loss at the shared checkpoint.

Appendix

Table 4: Correspondence between the linear analysis and TATC.

Explore similar work

Synthetic Data Characterization via Training Dynamics
Sep 30, 2026Irene Lago, Ana Ezquerro, David VilaresSynthetic Data Learnability
Universidade da Coruña, CITIC, Spain · Graz University of Technology, IML, Austria
LiBaGS: Lightweight Boundary Gap Synthesis for Targeted Synthetic Data Selection
May 11, 2026Abhishek Moturu, Anna Goldenberg, Babak TaatiSynthetic Data Synthetic Training Data
Department of Computer Science University of Toronto The Hospital for Sick Children UHN KITE Research Institute T-CAIREM Vector Institute · Department of Computer Science Department of Laboratory Medicine and Pathobiology University of Toronto The Hospital for Sick Children T-CAIREM Vector Institute · Department of Computer Science Institute of Biomedical Engineering University of Toronto Rehabilitation Sciences Institute UHN KITE Research Institute Vector Institute
Effective Synthetic Data Curation Requires Group-Level Signals
Sep 30, 2026Cathy Jiao, Chenyan XiongSynthetic Data Data-Curation
Language Technologies Institute, Carnegie Mellon University

Sep 30, 2026, cs.CLJ/K move · Enter open · S save

Synthetic Data Characterization via Training Dynamics

Irene Lago, Ana Ezquerro, David Vilares

Universidade da Coruña, CITIC, Spain · Graz University of Technology, IML, Austria

Synthetic Data Learnability Distributional Information Multi-Label Classification

Open paper AskPDF arXiv

Interpreting properties of LLM-generated data is important for understanding its utility and limitations across learning tasks. In this work, we characterize synthetic data through sample-level learnability, studying variation among LLM families and scales, alongside human-written data as a reference. We first generate synthetic datasets spanning single- and multi-label classification, labeling, and tree prediction tasks. We then derive empirical data distributions from encoder training dynamics for both machine and organic data, and estimate the robustness of these distributions across encoders. Finally, we evaluate how data selection strategies based on these learnability signals affect both data sources differently.

Abstract

Explore similar work

Synthetic Data Characterization via Training Dynamics

LiBaGS: Lightweight Boundary Gap Synthesis for Targeted Synthetic Data Selection

Effective Synthetic Data Curation Requires Group-Level Signals