Data Strategies for Large Language Models: A Unified Perspective on the Training Lifecycle

Yuanjian Xu; Jianing Hao; Tianze Sun; Yongyao Duan; Sam Chen; Jinyi Han; Wanbo Zhang; Yuxuan Sun

cs.CLOct 15, 2026

Data Strategies for Large Language Models: A Unified Perspective on the Training Lifecycle

Authors: Yuanjian Xu, Jianing Hao, Tianze Sun, Yongyao Duan, Sam Chen, Jinyi Han, Wanbo Zhang, Yuxuan Sun,

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Harbin Institute of Technology · OpenCSG · East China Normal University ·

Abstract

Data strategy plays a central role in training large language models (LLMs). It governs the selection, weighting, and scheduling of training data, thereby determining the capabilities a model can acquire from a fixed data budget. We treat high-quality data as having high training-dependent utility, which depends on the target objective, the model, and its training state rather than on the text alone. Research in this area now spans pretraining and post-training, but related ideas use inconsistent terminology and evaluation protocols. This work reviews data strategies across the training lifecycle and formulates them as policies that allocate a training budget over candidate examples. We organise methods by their decision granularity, model dependence, and temporal adaptivity, then compare their computational requirements and empirical support. Fine-grained or state-dependent policies offer more control but incur additional scoring, search, or rollout costs. Evidence for transfer across model scales and comparable end-to-end cost remains limited. A continuously updated list of papers, along with their public implementations, is available at https://github.com/xuyj233/data_strategy_survey.

Figures & tables

Panel of Figure 2 — Figure 2: Decision granularity in LLM data strategies. Dataset-level selection retains whole corpora, domain reweighting assigns continuous weights to predefined groups, and example-level selection retains individual examples. Their feasible sets need not be nested.

Rule type	Example rule	Source
Metadata only
URL / domain	Domain not on a blocklist	RefinedWeb
Single document: statistic + threshold
Size	$50$ – $100$ K words	Gopher
Composition	Mean word length $3$ – $10$	Gopher
Line structure	Punctuated lines ${>}12\%$	FineWeb

Table 1: Rule-based filters. Filters are grouped by the information each rule requires. Thresholds follow the cited source. FineWeb2 re-derives them for each language.

Dimension	Scorer	Source
Quality from a reference signal
Similarity	Curated vs. web	DCLM
Predictiveness	Loss–benchmark fit	PreSelect
Quality defined by an LLM
Education	Distilled LLM labels	FineWeb-Edu
Style / expertise	Pairwise LLM ratings	QuRating

Table 2: External-model scorers. Scorers are grouped by how the target property is defined. PRRC denotes professionalism, readability, reasoning, and cleanliness; SCPI denotes sensitive personal information.

Method	Class	Surrogate $\widehat{{\color[rgb]{0.6914,0.2266,0.1797}\mathcal{J}}}(w)$	Solver
RegMix	Linear	$b_{0}+b^{\top}w$	Random search
MoDoMoDo	Quadratic	$a+b^{\top}w+w^{\top}Cw$	Grid search
Data Mixing Laws	Log-linear	$c_{i}+k_{i}\exp(t_{i}^{\top}w)$	Grid search
DUET	Gaussian process	$\sim\mathcal{GP}(\mu,\kappa)$	Bayesian opt.
Data Mixing Agent	Neural policy	$\mathbf{w}_{t}={\color[rgb]{0.1797,0.4883,0.1953}\pi_{\phi}}(s_{t})$	Offline RL

Table 3: Performance modelling for domain reweighting. Surrogate class and solver are separate design choices. The first four methods fit

\widehat{{\color[rgb]{0.6914,0.2266,0.1797}\mathcal{J}}}

over static mixtures, whereas Data Mixing Agent learns a state-dependent policy. MoDoMoDo is included for its surrogate class.

Category	Signal or criterion	Example
Model-derived signal
Distribution-based	Generated domain profile	VersaTune
Gradient-based	Gradient direction or magnitude	LESS, OASIS
Outer objective or proxy
Average	Fixed-mixture target risk	LESS
Worst-group	Maximum group risk	AutoMixAlign

Table 4: Classification axes for post-training data strategies. Categories within each group are alternatives along one design axis. Representative examples are shown, with citations in Table 5 . ADG denotes answer-divergence-guided selection.

Figure 3: Extracting model-derived signals. Post-training policies query a model checkpoint to estimate data value from generated domain distributions or local gradient information.

Method	Average	Worst-group	Base preservation
LESS ( Xia et al., 2024 )	✓
DynamixSFT ( Shin et al., 2025 )	✓
OASIS ( Lee et al., 2026 )	✓
EVIC ( Liang et al., 2025a )	✓
AutoMixAlign ( Corrado et al., 2025 )		✓
VersaTune ( Lu et al., 2025 )			✓

Table 5: Objectives and proxies for post-training data value. Average, worst-group, and base preservation denote their corresponding training objectives; Proxy denotes a criterion that constructs the data policy without evaluating the final model. IFD denotes instruction-following difficulty.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Survey	Training stages covered
	PT	AN	SFT	PO
Wang et al. (2023)	✓	✗	✓	✗
Albalak et al. (2024)	✓	✗	✓	✓
Qin et al. (2025a)	✗	✗	✓	✗
Zhang et al. (2025a)	✗	✗	✓	✗
Ours	✓	✓	✓	✓

Appendix

Table 6: Training-stage coverage of related surveys. PT, AN, SFT, and PO denote stable pretraining, annealing, supervised fine-tuning, and preference optimisation, respectively.

Figure 4: Different signals induce different domain structures on the same SlimPajama sample. (a–b) Principal component analysis (PCA) views of semantic and L2-normalised gradient representations, coloured by source. Crosses mark source medians. A stratified subset is shown for legibility, while clustering and agreement metrics use the full sample. (c) Agreement among source labels and the two induced partitions. Semantic clusters follow source boundaries more closely than gradient clusters, while semantic and gradient partitions have the lowest agreement.

Figure 5: Estimated auxiliary compute. (a) Auxiliary-training FLOPs relative to a shared target run. (b) Logged component shares for a DIDS-inspired adaptation, using a projected target run as the denominator.

Method	Proxy size	Runs	Tokens/run	Auxiliary FLOPs	Overhead
DoGE ( Fan et al., 2024 )	42.5M	1	2.911B	$7.423{\times}10^{17}$	$2.05\%$
RegMix ( Liu et al., 2025b )	1M	512	1.050B	$3.226{\times}10^{18}$	$8.92\%$
DoReMi ( Xie et al., 2023 )	84.9M	2	5.822B	$6.920{\times}10^{18}$	$19.1\%$
DDO diagnostic ( Kang et al., 2025 )	1.035B	30	0.3B / 1.2B	$1.397{\times}10^{20}$	$386\%$
Data Mixing Laws ( Ye et al., 2025 )	18.9–302.3M	103	1.747B	$1.326{\times}10^{20}$	$367\%$

Appendix

Table 7: Auxiliary compute under a shared target configuration. FLOPs use non-embedding parameter counts; method-specific tokens/run and overhead are measured against the shared target.

LLM Training

Large language model training studies the data, objectives, optimizers, and systems used to build and adapt language models.

See all 344 papers

How Should LLMs Consume High-Quality Data? Optimal Data Scheduling via Quality-Aware Functional Scaling Laws
May 25, 2026Zhitao Zhu, Xili Wang, Shizhe Wu +2Language Model Scaling Laws LLM Training
Peking University · Meituan
Data Mixing for Large Language Models Pretraining: A Survey and Outlook
Mar 25, 2026Zhuo Chen, Yuxuan Miao, Supryadi +1Data Mixture Optimization Language Model Pretraining
The International Joint Institute of Tianjin University, Tianjin University, Tianjin 300072, China · TJUNLP Lab, School of Computer Science and Technology, Tianjin University, Tianjin 300350, China
Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning
Apr 9, 2026Shiwan Zhao, Zhihu Wang, Xuyang Zhao +10LLM Post-Training RL for Language Models
Nankai University, China · Huawei Technologies Ltd., China

May 25, 2026J/K move · Enter open · S save

How Should LLMs Consume High-Quality Data? Optimal Data Scheduling via Quality-Aware Functional Scaling Laws

Zhitao Zhu, Xili Wang, Shizhe Wu, Jiawei Fu+1

Peking University · Meituan

Language Model Scaling Laws LLM Training

Open paper AskPDF arXiv

High-quality data is scarce in large language model (LLM) training, yet how to schedule its use jointly with training dynamics lacks theoretical guidance. We extend functional scaling laws by incorporating a data-quality dimension, and solve the joint data-quality and batch-size scheduling problem in asymptotic closed form. The solution reveals two regimes and a dual role of high-quality data. In the noise-limited regime, high-quality data should be used as a signal amplifier: lowering the batch size converts cleaner data into more signal without amplifying noise. In the signal-limited regime, it should be used as a noise suppressor: late placement reduces terminal noise without sacrificing signal accumulation. Existing curriculum-style pipelines primarily exploit the second role by placing cleaner data late, but miss the first role because conventional decay schedules reduce update intensity exactly when high-quality data becomes available. Guided by this, we propose Drop-Stable-Rampup for LLM midtraining: upon the quality transition, drop the batch size, hold it stable to accumulate signal, then ramp up to suppress terminal noise. On a 15B Mixture-of-Experts model midtrained on 108B tokens, Drop-Stable-Rampup improves average accuracy over Warmup-Stable-Decay (WSD) by +1.70 and over Cosine-decay by +2.98, with particularly large gains on mathematical reasoning benchmarks such as GSM8K (+4.23) and MATH (+2.80).

Abstract

LLM Training

Explore similar work

How Should LLMs Consume High-Quality Data? Optimal Data Scheduling via Quality-Aware Functional Scaling Laws

Data Mixing for Large Language Models Pretraining: A Survey and Outlook

Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning