Advancement or Acceleration? How Data Curation Improves LLM Training

Jiaxing Wang; Zicheng Zhang; Deping Xiang; Jin Xu; Yong Li; Chao Liu; Pengzhang Liu; Qixia Jiang

cs.LGOct 27, 2026

Advancement or Acceleration? How Data Curation Improves LLM Training

Authors: Jiaxing Wang, Zicheng Zhang, Deping Xiang, Jin Xu, Yong Li, Chao Liu, Pengzhang Liu, Qixia Jiang

Organizations: University of Oxford

Abstract

While data curation is widely credited with improving large language model training, a basic question remains unresolved: does curation expand what a model can ultimately learn, or merely reach the same solution faster? In this paper, we formalize this distinction between advancement (improving the attainable performance) and acceleration (reaching the same performance with less data). Through an analytic characterization and real-world LLM experiments, we show that advancement requires a correctable train-target gap. When the baseline training and target objectives coincide, no curation policy improves the endpoint, although curation may still accelerate training. This reframes advancement as a property of the curation task rather than the curation method. Under a common gap-centric view of data mixture and selection, we operationalize this condition as the $Δ$ -test, an observable discrepancy between the uncurated baseline's training and target losses that is computable prior to expensive curation. Across extensive experiments, an appreciably nonzero $Δ$ predicts persistent gains, whereas gains near $Δ=0$ tend to diminish as the training budget grows. Beyond individual methods, our findings recenter data curation on the task itself and point to target-set construction as a promising direction for shaping what curation can achieve.

Figures & tables

Panel of Figure 1 — Figure 1: An analytic account of data curation. Theory (solid lines) closely matches $N=200$ perceptron simulations (circles). Top row—mixture. (a) The setup. Two-domain training and target mixtures with compositions $\bm{w}$ and $\widehat{\bm{w}}$ . (b) Acceleration. Tilting the training mixture toward the harder domain ( $w_{1}<\widehat{w}_{1}=0.5$ ) lowers target loss at small budgets, but the gain vanishes as $\alpha$ grows. (c) Advancement. At a large budget ( $\alpha=100$ ), each target loss vs. mixture curve is minimized by target matching, $\bm{w}^{\star}=\widehat{\bm{w}}$ . Appendix A.1 shows that this holds as $\alpha\to\infty$ . Bottom row—selection. (d) The setup. A selection policy favors examples near the target boundary and filters those whose labels conflict with the target. (e) Acceleration. At zero gap ( $\phi=0$ ), moderate selection preserves accuracy, but never exceeds full retention ( $f=1$ ). Aggressive selection degrades performance. (f) Advancement. With $\phi>0$ , target accuracy initially rises as $f$ decreases: improved train–target alignment outweighs the reduction in sample size. Reaches an interior optimum $f^{\star}<1$ , and the gain increases with $\phi$ . Appendix A.2 shows that an interior selection policy remains better than full retention as $\alpha\to\infty$ .

Methods	Arxiv	Book	C4	CommonCrawl	Github	Stackexchange	Wikipedia	Average
Uniform	10.02	39.61	49.67	42.70	5.16	12.25	20.00	19.71 $\pm_{\text{0.17}}$
RegMix	10.53	41.90	48.67	41.65	5.75	11.73	19.86	19.80 $\pm_{\text{0.12}}$
DoGE	10.96	35.49	42.42	36.81	6.70	13.97	23.92	20.17 $\pm_{\text{0.16}}$
Skill-It	10.64	38.32	47.99	40.62	5.73	12.07	20.78	19.68 $\pm_{\text{0.16}}$
Aioli	10.09	39.86	49.81	42.83	5.66	12.36	19.67	19.77 $\pm_{\text{0.15}}$
TANDEM	10.18	37.89	46.77	40.41	5.89	12.61	20.96	19.66 $\pm_{\text{0.15}}$

Table 1: Perplexity in the no-gap (

\Delta=0.07

) mixture setting on SlimPajama. “Average” represents the exponential of the average loss across all domains as in ( Fan et al., 2024 ) .

Figure 2: Left: With a negligible gap (

\Delta=0.07

), the perplexity difference relative to Uniform (method

-

Uniform, lower is better) can be negative early but approaches zero with training. Bands show the standard deviation over five runs. Right: Final gain over Uniform (higher is better) is negligible in the no-gap setting and becomes substantial under endogenous and exogenous gaps.

Endogenous Gap									Exogenous Gap
Methods	Arxiv	Book	C4	CC.	Github	Stackexchange	Wikipedia	Average	CC.
Uniform	13.14	57.44	65.57	57.93	8.21	19.59	33.90	28.60	49.19
RegMix	13.33	54.12	63.74	56.63	8.23	20.03	33.34	28.24	40.63
DoGE	12.95	54.82	62.84	55.52	8.13	18.61	32.72	27.62	40.67
Skill-It	14.54	44.24	48.99	42.10	8.33	21.10	49.46	27.38	41.08
Aioli	12.80	51.01	61.13	53.51	8.14	18.97	32.63	27.11	40.17

Table 2: Final target perplexity under endogenous (

\Delta=0.64

) and exogenous (

\Delta=1.26

) mixture gaps. The endogenous block reports all target domains and their ”Average”. The exogenous block reports the perplexity of the held-out CommonCrawl target (CC).

Scenario	Model	Uniform	TANDEM	Gain
No gap $\Delta=0.07$	160M	19.71	19.66	$+0.05$
No gap $\Delta=0.07$	1.1B	17.16	17.05	$+0.11$
Endogenous gap $\Delta=0.64$	160M	28.60	26.02	$+2.58$
Endogenous gap $\Delta=0.64$	1.1B	28.14	25.85	$+2.29$
Exogenous gap $\Delta=1.26$	160M	49.19	40.24	$+8.95$
Exogenous gap $\Delta=1.26$	1.1B	40.03	34.72	$+5.31$

Table 3: Scale study comparing Uniform and TANDEM. Gain is Uniform

-

TANDEM.

Selection without Gap
	Retained	GSM8K	MATH	SVAMP	ASDiv	MAWPS	TAB	MQA	MMLU-	Avg.	Target
	Toks.	GSM8K	MATH	SVAMP	ASDiv	MAWPS	TAB	MQA	STEM	Avg.	Loss
Base	-	2.9	3.2	11.0	18.1	20.4	12.5	14.6	16.1	12.4	1.97
Base-CT	5B	4.3	4.3	17.1	29.4	39.4	14.5	8.3	20.4	17.21	1.70
Random	3B	4.0	3.9	15.8	29.1	37.2	13.8	9.4	17.8	16.38	1.73
EL2N	3B	3.0	3.2	15.5	25.0	31.2	14.4	8.9	18.3	14.94	1.76

Table 4: Data-selection results for an in-distribution OpenWebMath target (no gap,

\Delta=0.016

, upper) and a shifted MetaMath–Mammoth target (with gap,

\Delta=0.549

, lower). We report accuracy on eight math benchmarks, their average, and held-out target loss. All methods operate on the same 5B-token candidate pool with a retained fraction of

f=0.6

. Target-aware RHO-1 and BLADE improve substantially only when a gap is present.

Figure 3: Left: Held-out target loss across retained fractions

f\in[0.6,1.0]

for an in-distribution OpenWebMath target (

\Delta=0.016

). Target loss directly evaluates the distribution used to define

\Delta

. No selector consistently improves over Base-CT and Random, and more aggressive pruning generally increases loss. Right: Average accuracy change relative to Base-CT in the no-gap and shifted-target settings. The comparison shows little optimization headroom in the no-gap setting, whereas substantial positive gains become possible after the target shift (

\Delta=0.549

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Curation gain versus the $\Delta$ diagnostic in the solvable models. Each point varies the parameter that generates the train–target gap and reports the improvement of the optimized policy over the uncurated baseline. (a) Mixture: target-composition sweep. Gain is the reduction in target logistic loss. (b) Selection: teacher-angle sweep. Gain is the reduction in target classification error.

Configuration	160M	1.1B
Training steps	40K	40K
Batch size	8	8
Context length	2048	2048
Peak learning rate	$5\times 10^{-4}$	$2\times 10^{-4}$
Warmup steps	–	500
Weight decay	$0.01$	$0.01$

Appendix

Table 5: Shared configuration for the data-mixture experiments.

Configuration	TinyLlama-1.1B
Candidate-training steps	5000
Batch size	512
Context length	2048
Learning rate	$8\times 10^{-5}$
Learning-rate schedule	Constant
Weight decay	0.01

Appendix

Table 6: Shared configuration for the data-selection experiments.

Symbol	Meaning
Data and dimensions
$N$	input dimension.
$P$	number of training examples.
$\alpha=P/N$	total sample budget (samples per dimension). In high-dim limit $N,P\to\infty$ , $\alpha$ fixed.
$\alpha_{k}=w_{k}\alpha$	sample budget contributed by domain $k$ .
$\bm{x}\in\mathbb{R}^{N}$	input data.

Appendix

Table 10: Summary of the notations used throughout this paper. Variables only used in theoretical analysis are grayed for better readability.

Training Data Curation

Training data curation improves the quality, diversity, organization, and reliability of datasets used to train machine-learning models.

See all 156 papers

CurateEvo: Data-Curation Evolving for Agentic Post-Training
Jul 7, 2026Dingzirui Wang, Xuanliang Zhang, Keyan Xu +2Training Data Curation LLM Agent Training
Harbin Institue of Technology
20/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone
May 12, 2026DatologyAI, :, Siddharth Joshi +32Training Data Curation Large Vision-Language Models
DatologyAI Team1
Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods
Apr 19, 2026Wanru Zhao, Yihong Chen, Yuzhi Tang +6Language Model Pretraining LLM Fine-Tuning
University of Cambridge · OATML, University of Oxford · University of Toronto +2

Jul 7, 2026J/K move · Enter open · S save

CurateEvo: Data-Curation Evolving for Agentic Post-Training

Dingzirui Wang, Xuanliang Zhang, Keyan Xu, Qingfu Zhu+1

Harbin Institue of Technology

Training Data Curation LLM Agent Training LLM Post-Training

Open paper AskPDF arXiv

Large language model (LLM) agents require post-training methods that can improve long-horizon decision making from environment feedback. However, existing agentic post-training pipelines often treat data curation as a fixed preprocessing step, focusing mainly on data augmentation while neglecting filtering, refinement, and adaptation to downstream failures. We propose CurateEvo, a failure-driven dynamic evolution framework for agentic post-training data curation. CurateEvo represents the curation strategy as executable code and iteratively rewrites it using failed trajectories from a held-out development set. At each epoch, the evolved strategy transforms a fixed raw corpus into supervised fine-tuning data, reinforcement learning data, and an inference-time memory bank. The evolution process first improves effectiveness by diagnosing recurring failure modes and augmenting, filtering, or refining data accordingly, and then improves efficiency by pruning redundant or low-utility training turns under a cost-aware objective. Experiments on ACEBench-Agent, BFCL-V4, and τ^2-Bench under both labeled and wild-data settings show that CurateEvo consistently outperforms prior curation methods, improving average scores by 3.2 and 2.7 points, respectively. Further analyses demonstrate that CurateEvo is compatible with different post-training recipes and substantially reduces curation overhead.

Abstract

Training Data Curation

Explore similar work

CurateEvo: Data-Curation Evolving for Agentic Post-Training

20/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone

Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods