Effective Synthetic Data Curation Requires Group-Level Signals

Cathy Jiao; Chenyan Xiong

cs.CLSep 30, 2026

Effective Synthetic Data Curation Requires Group-Level Signals

Authors: Cathy Jiao, Chenyan Xiong

Organizations: Language Technologies Institute, Carnegie Mellon University

Abstract

Synthetic data now is essential to LLM training, used to strengthen advanced capabilities such as autonomous and long-horizon task execution. Yet recent work shows that training on it at scale can degrade model generation, making it important to decide what synthetic data is worth training on. While current data curation practices do so with individual-level signals (i.e., estimates of each data sample's training utility in isolation), across pre-training and post-training settings we show that this is insufficient for synthetic data, and that group-level signals (i.e., estimates of utility that account for interactions among data samples) are necessary for effective data curation. First, we show that individual-level signals are blind to how samples jointly affect training: synthetic datasets with different compositions can be indistinguishable under individual-level influence yet differ sharply under group-level influence, and curating by the latter yields better downstream performance, particularly in generative capability. Second, we find that group-level signals matter more as training pipelines become increasingly synthetic: among widely used data curation methods, only those incorporating them improve over baseline, with gains increasing when weights capturing relations among samples are amplified. Finally, we translate these findings into practice -- for model developers under a compute budget, we offer a cheap diagnostic that prioritizes which groups of synthetic data most need group-level estimation, recovering much of the benefit of full group-level scoring at a fraction of the compute cost.

Figures & tables

Panel of Figure 2 — Figure 2: Semantic/topic diversity of our candidate pool on the pre-training corpus.

Panel of Figure 3 — Figure 3: Range of group oracle and individual proxy oracle scores across our constructed groups (on groups of $5120$ and $128$ samples). See Appendix D.4 for full sweep over group sizes.

Panel of Figure 4 — Figure 4: (a) Relative improvement over the base model by task type in pre-training. (b) Pass@8 over GRPO training steps in post-training.

	Method	Representation
Individual	GradSim ( Pruthi et al., 2020 )	Gradient
	FineWeb-Edu ( Penedo et al., 2024 )	Classifier score
	BM25 ( Trotman et al., 2014 )	Lexical (tokens)
Group	GREATS ( Wang et al., 2024 )	Gradient
Group	GMRel ( Yu et al., 2025 )	Learned influence

Table 2: Taxonomy of individual-and group-level data selection methods. Details in Appendix B.5

Panel of Figure 5 — Figure 5: (a, b) Performance of individual-/group- level selection methods as rephrase factor increases. (c) Group-oracle value and diversity trade-off. (d) Increasing GMRel’s relational weight at $r=50$ .

Panel of Figure 6 — Figure 6: Gap between the group and individual proxy oracle diverges as within-group diversity increases.

Panel of Figure 7 — Figure 7: Allocating a limited group-oracle budget with cheap diagnostics. See Appendix D.5 for additional analysis plots.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Pre-Training Setting	Post-training Setting
Probe model	Repro-400M (trained on 6.59B Repro tokens ( Yu et al., 2025 ) )	Qwen2.5-1.5B-Instruct
Tokenizer	Pythia-410m	Qwen2.5-1.5B-Instruct
Optimizer	AdamW, 1 step	AdamW, 1 step
Learning rate	2e-5	1e-5
Precision	bf16	bf16
Grad. accumulation	Yes	Yes

Appendix

Table 3: Oracle scoring configurations for the pre-training and post-training settings.

Setting	size	Quantity	$\max_{j}\|\mathcal{I}_{\mathrm{proxy}}(G_{j},\theta)-\tau_{j}\|$	$\epsilon$	diversity range
Pretraining	$5120$	$1000$	$5.0\times 10^{-5}$	$5\times 10^{-5}$	$0.095$ – $0.212$
Pretraining	$10240$	$500$	$5.0\times 10^{-5}$	$5\times 10^{-5}$	$0.096$ – $0.200$
Post-training	$128$	$1000$	$5.0\times 10^{-6}$	$5\times 10^{-6}$	$0.156$ – $0.435$
Post-training	$512$	$500$	$4.8\times 10^{-6}$	$5\times 10^{-6}$	$0.157$ – $0.425$

Appendix

Table 4: Diversity-swept candidate pools. Diversity is

1-\overline{\cos}(G)

. Every group’s individual proxy oracle lands within

\epsilon

of its assigned band target, so the pools vary in diversity at (near-)fixed individual proxy oracle.

	Pre-training	Post-training
Phase 1: Candidate pool construction
Source dataset	Repro-Rephrased-72B	GooseReason
Semantic clusters	20	20
Group sizes $s$	$5120$ (small), $10240$ (large)	$128$ (small), $512$ (large)
Candidate pool $H$	$1000$ (small), $500$ (large)	$1000$ (small), $500$ (large)
Phase 2: Data selection

Appendix

Table 5: Task settings for the pre-training and post-training experiments, organized by the three phases in Section 3.2 . Each training setting is run under two configurations: a small -group and a large -group setting.

Phase 1: Synthetic pool construction
Source documents	Organic web text (DCLM-RefinedWeb)
Rephraser	Repro rephraser-4B ( Yu and Xiong, 2025 )
Sampling	temperature $1.0$ , top- $p$ $0.9$
Pool size $N$	$716{,}800$ sequences
Rephrase factor $r$	$\{1,10,50\}$
Unique source docs ( $N/r$ )	$716{,}800\,/\,71{,}680\,/\,14{,}336$

Appendix

Table 6: Experimental settings for the data curation experiments, organized by the three phases in Section 4.1 . The pool is held at a fixed size while the rephrase factor

r

controls the degree of near-duplicate redundancy.

Parameter	Pre-Training Setting	Post-training Setting
Rollout generation (supervision)
Base model	Repro-400M (as in Table 3 )	Qwen2.5-1.5B-Instruct
Candidate pool	Repro-Rephrased-72B	GooseReason
Source tokenizer	Pythia-410m	Qwen2.5-1.5B-Instruct
Reference set	FLAN (128 samples)	GPQA-extend (128 samples)
Candidates per rollout $K$	$10$	$10$

Appendix

Table 7: GMRel relational model configurations: oracle rollout generation (which produces the supervision) and fitting of the relational influence model.

Allocation	$B$	$q$	Ratio (Eq. 8 )	Calculated Ratio
Group oracle	$H$	$0$	$1$	$1.000$
GradDiv full	$B$	$1$	$\approx B/H+\kappa$	$0.790$
GradDiv $q$	$B$	$0.5$	$\approx B/H+\kappa q$	$0.495$
GradDiv $q$	$B$	$0.1$		$0.259$
GradDiv $q$	$B$	$0.05$		$0.229$
GradDiv $q$	$B$	$0.01$		$0.206$

Appendix

Table 8: Cumulative GPU memory of each allocation relative to the full group oracle. The last column is the exact ratio of Equation 8 on the pre-training pool of Figure 7(d) (

H=1000

|G|=5120

R=128

B/H=0.2

\kappa=0.593

Method	HellaSwag	ARC-C	PIQA	MMLU	CSQA	LAMBADA	SQuAD	CoQA	Avg
Rephrase factor $r=1$
Random	30.95	26.96	66.16	23.19	19.41	41.04	9.82	25.52	30.38
GradSim	30.81	26.28	65.34	22.98	21.21	36.37	3.65	24.69	28.92
FineWeb-Edu	30.89	25.51	64.53	22.87	19.16	38.09	17.79	23.60	30.31
BM25	31.14	26.45	65.34	23.21	19.66	38.15	5.40	26.68	29.50
GMRel ( $\tau\!\approx\!1.01$ )	30.41	25.94	65.02	23.15	19.82	36.76	25.17	24.62	31.36

Appendix

Table 9: Per-task accuracy at the final step (200 steps) for the Repro-400M continuation, by selection method and rephrase factor

r

. Best average within each

r

is in bold , and second best is underlined .

Panel of Figure 8 — Figure 8: Discrepancy between the group oracle $\mathcal{I}_{\mathrm{orcl}}(G,\theta)$ and individual proxy oracle $\mathcal{I}_{\mathrm{proxy}}(G_{j},\theta)$ scores across varying sizes of both the synthetic pretraining and the RLVR data groups.

Panel of Figure 9 — Figure 9: Range of group oracle and individual proxy oracle scores across our constructed groups, for each group size in the pre-training and post-training settings.

Panel of Figure 10 — Figure 10: Topic composition of the $k=20$ semantic clusters used for proxy-matched diversity sampling. Each slice is one cluster, labeled with its topic and share of the corpus, and colored by topic family.

Panel of Figure 11 — Figure 11: Diversity metrics on our constructed groups versus randomly formed groups in the pre-training setting.

Panel of Figure 12 — Figure 12: Diversity metrics on our constructed groups versus randomly formed data groups in the post-training setting.

Panel of Figure 13 — Figure 13: Choice of diagnostic for group-oracle budget allocation, for pre-train and post-train data (group sizes of 5120 and 512 samples, respectively).

Panel of Figure 14 — Figure 14: Effect of gradient subsampling on group-oracle budget allocation, comparing the full GradDiv diagnostic against subsampled variants ( $q=0.1$ , $q=0.01$ ) for pre-train and post-train data (group sizes of 5120 and 512 samples, respectively).

Explore similar work

Sep 30, 2026cs.CL

Synthetic Data Characterization via Training Dynamics

Interpreting properties of LLM-generated data is important for understanding its utility and limitations across learning tasks. In this work, we characterize synthetic data through sample-level learnability, studying variation among LLM families and scales, alongside human-written data as a reference. We first generate synthetic datasets spanning single- and multi-label classification, labeling, and tree prediction tasks. We then derive empirical data distributions from encoder training dynamics for both machine and organic data, and estimate the robustness of these distributions across encoders. Finally, we evaluate how data selection strategies based on these learnability signals affect both data sources differently.

Irene Lago, Ana Ezquerro, David Vilares

Universidade da Coruña, CITIC, Spain · Graz University of Technology, IML, Austria

Synthetic Data Learnability Distributional Information Multi-Label Classification

Ask PDF

May 11, 2026cs.LG

An Information-Theoretic Criterion for Efficient Data Synthesis

Synthetic data becomes crucial for large language model training, but its effectiveness is highly inconsistent. We provide an information-theoretic account of this inconsistency: synthetic data improves a model only when the generation-training loop is information-open, i.e., shaped by external signals (verifiers, environments, or rubrics) that inject task-relevant information beyond the model's current distribution. When the loop is information-closed (relying on the model's own outputs without such signals), the data processing inequality ensures that task-relevant information can only decrease, making collapse a predicted outcome. Among information-open pipelines, both efficiency and generalization hinge on the meta-level of supervision: a coarser signal such as binary correctness treats all acceptable outputs as equivalent, so the behavior it teaches is not tied to any particular domain or surface form and generalizes naturally across tasks and domains. These observations lead to a guiding thesis: learning preferentially converges to the most information-efficient signal component available, which accelerates learning when that component is the intended one, but causes reward hacking when a spurious pattern happens to be simpler.

Hanyu Li, Zhengqi Sun, Xiaotie Deng

CFCS, School of Computer Science, Peking University, Beijing, China · Department of Information Management, Peking University, Beijing, China

Synthetic Data Information Bottleneck Large Language Model Training Information-Theoretic Limits Task-Relevant Information

Ask PDF

Jun 19, 2026cs.CL

CuratorKIT : Data Curation and Synthetic Data Generation for LLM Post-Training

Data curation is a critical part of post-training pipelines for large language models, yet existing tools often treat ingestion, deduplication, synthetic generation, and quality filtering as separate stages. This fragmentation makes it difficult to audit pipeline decisions or understand why individual samples are rejected. CuratorKIT is an open-source Python library that covers this full lifecycle in a single configurable pipeline. The framework is composed of six source format readers and automatic schema detection, a pre-generation data hygiene layer for credentials, PII, and toxic content, eight LLM-powered generation tasks, three complementary quality gates with provenance-exact hallucination verification, structured adaptive recovery, and five training-ready export formats compatible with TRL, Unsloth, and AlignTune. Every pipeline decision is recorded in an append-only per-sample provenance chain, and rejected samples carry structured failure reasons rather than being silently discarded. CuratorKIT supports 100+ LLM providers through LiteLLM, exposes both a Python API and a YAML-driven CLI, and is designed for practitioners who need reproducible, auditable data pipelines at scale .

Soham Bhattacharjee, Karun Sharma, Vinay Kumar Sankarapu +1

Lexsi Labs

Data-Curation Large Language Model Pipelines Synthetic Data

Ask PDF