A Comprehensive Framework for All Stages of Large-Model Training: From Data and Tokenization, through Pretraining, to Post-Training Alignment

Xiangjing An; Zhibin Jiao; Aiting Chen

cs.AIOct 24, 2026

A Comprehensive Framework for All Stages of Large-Model Training: From Data and Tokenization, through Pretraining, to Post-Training Alignment

Authors: Xiangjing An, Zhibin Jiao, Aiting Chen

Organizations: Independent Researcher Team

Abstract

The capabilities of large language models (LLMs) and multimodal LLMs (MLLMs) arise from the synergy of the full training pipeline, including data and tokenization, pretraining, and post-training (supervised fine-tuning, preference alignment, reinforcement learning, and continual optimization), rather than any individual stage. Existing surveys mostly adopt a single-stage or single-technique perspective, treating related research as isolated systems without unified stage division or a horizontally comparable basis. This paper proposes a unified stage division and a six-dimensional analytical framework covering data, objective function, optimization strategy, system constraints, evaluation, and risk and governance. We systematically review 395 core references from 2017 to 2026, and present a conceptual map of cross-stage effects: forward constraints of pretraining on post-training, and reverse empowerment of post-training on data and pretraining. We derive three core findings. First, the comparability of scaling laws depends on data quality and mixture, a long-neglected variable. Second, preference alignment methods (RLHF, DPO) and reasoning reinforcement learning with verifiable rewards redefine post-training boundaries, while the Pass@1 versus Pass@K debate remains unresolved. Third, multimodal training poses structural challenges to the pure-text paradigm in tokenizer design, training objectives, and evaluation. The paper provides a cross-stage effects diagram, a method-family map, a practical decision tree, and at least 10 verifiable open problems, with an extensible skeleton for focused sub-surveys per stage. As of the retrieval date, peer-reviewed evidence (A+B) accounts for approximately 79% of the 395 references.

Figures & tables

Figure 1: The full training pipeline—a linear backbone plus vertical six-dimensional labels (applied across all stages)

Stage \ Dimension	Data	Objective Function	Optimization Strategy	System Constraints	Evaluation	Risk and Governance
Data and tokenization	Collection/cleaning/deduplication/mixing	—	—	Vocabulary size vs throughput	Compression rate, coverage, fairness	Copyright, privacy, toxicity
Pretraining	Trillions of tokens, mixture and curriculum	NTP/MLM/denoising/contrastive	Scaling laws, learning rate, numerical precision	3D parallelism, ZeRO/FSDP	Perplexity, downstream zero/few-shot	Data provenance, energy consumption
SFT	Instruction–response pairs, templates	Conditional LM + loss masking	Full-parameter / PEFT	Multi-turn dialogue GPU memory	Instruction following, format	Synthetic data bias
Preference alignment / RL	Preference pairs, rewards, verifiable environments	BT loss, KL constraint, GRPO	PPO stability, rollout cost	Sampling throughput	Win rate, RewardBench	Jailbreak, sycophancy, reward hacking
Continual optimization	Domain/long-context data	Continued training, distillation, quantization objectives	Low-rank, merging algorithms	Deployment latency and GPU memory	Retention, catastrophic forgetting	Unlearning ethics, licensing

Table 2

Concept	One-sentence definition	Key chapters
Five stages	Data and tokenization $\rightarrow$ pretraining $\rightarrow$ SFT $\rightarrow$ preference alignment/RL $\rightarrow$ continual optimization	Chapters 5–9
Six dimensions	Data, objective, optimization, system, evaluation, governance	Sections 2.4 and 4.2
Capability vs behavior	Capability = elicitable knowledge and skills; behavior = output style and compliance	Sections 3.2 and Chapter 11
Forward constraint	Earlier stages limit the attainable upper bound of later stages	Sections 3.2 and 11.5
Reverse empowerment	Post-training/evaluation failures feed back into data and pretraining design	Chapters 11 and 13
Scaling laws	Empirical regularities of loss with respect to $N,D,C$ (Kaplan/Chinchilla)	Section 6.3

Table 3

Stage	Input	Output	Objective	Key Methods	Principal Risks	Principal Evaluation
Data and Tokenization (Chapter 5)	Raw corpus / multimodal data	Cleaned dataset + tokenizer	High quality, leak-free, well-proportioned	Deduplication, filtering [392], mixing, BPE/Unigram [1][5]	Contamination, toxicity, copyright, vocabulary bias	Deduplication rate, coverage, downstream gains
Pretraining (Chapter 6)	token sequences	Base model $\theta_{0}$	Acquisition of general representations and knowledge	NTP/MLM [47], scaling laws [119][16], MoE [9]	loss spike, data-constrained, compute imbalance	Perplexity, downstream zero/few-shot [121]
SFT (Chapter 7)	instruction–response pairs	Instruct model $\theta_{1}$	Instruction following, formatting	Full-parameter / LoRA [12] / QLoRA [30]	Overfitting, capability forgetting, template leakage	Instruction following, human / model evaluation
Preference Alignment / RL (Chapter 8)	preference pairs / verifiable environments	Aligned model $\theta_{2}$	Preference alignment, suppression of harmful outputs	RLHF [15], DPO [32], RLVR	Reward hacking, sycophancy, capability narrowing	Win rate, safety, reasoning Pass@1/K
Continual Optimization (Chapter 9)	base / aligned model + domain data	Deployment-optimized model $\theta_{3}$	Domain adaptation, efficiency, long context	Continued pretraining, long-context extension [11][129], distillation/pruning/quantization [21]	Alignment tax, forgetting, quantization damage	Domain metrics, throughput, safety regression

Table 4

Figure 3: Method-family map—maturity annotated in four Morandi tones (sage = classic, misty blue = mainstream, clay = frontier, dusty rose = disputed)

Direction	Representative work	Core idea	Evidence	Notes
Deduplication	Lee et al. [13]	Exact/approximate deduplication reduces memorization and improves performance	A	Widely reproduced
Data selection	DSIR [67]	Importance resampling to match the target distribution	A	Requires target distribution
Learnability selection	RHO-LOSS-type [63]	Sampling prioritized by learnability	A	Requires a reference model
Mixing optimization	DoReMi [70]	Minimax domain-weight optimization	A	Requires domain labels
Mixing laws	Data mixing laws [86]	Predict the mixing–loss relation	A	Extrapolate with caution
Filtering compute	Filtering scaling law [87]	Filtering optimality depends on the compute budget	A	Against "one-size-fits-all"

Table 6

Direction	Representative Works	Core Idea	Evidence	Scope Note
NTP + scale	GPT-3 [121]	few-shot emergence, parameter scaling	A	175B, ~300B tokens
Scaling laws	Kaplan [119] / Chinchilla [16]	parameter-first vs. data-first	A/B	conclusion depends on setting
Data $\times$ scale	filtering scaling law [87] / mixture laws [86]	quality/mixture enter the law	A	compute-dependent
Optimizer	AdamW [110] / Lion [25] / Sophia [31]	first-order/sign/second-order approximation	A	precision and schedule must be specified
Hyperparameter scaling	muP [132]	transfer of width-scaling rules	A	validated at small-to-medium scale
Sparse architecture	Switch [9] / GLaM [130] / Mixtral [151]	sparse activation saves compute	A	distinguish total/activated parameters

Table 7

Direction	Representative work	Core idea	Evidence	Remarks
Human instructions	FLAN [163] / Super-NI [167]	Task templatization / crowdsourced instructions	A	Broad coverage
Data mixture	Flan Collection [172]	Systematized mixture of data and methods	A	Ablation
Synthetic instructions	Self-Instruct [24]	Model bootstrapping to generate instructions	A	Bias injection
Difficulty evolution	WizardLM/Evol [29]	Evolutionary increase in complexity	B	Diversity risk
Few but well-chosen	LIMA [175]	A thousand high-quality samples suffice for alignment	B	Single setting
Efficient fine-tuning	LoRA [12] / QLoRA [30]	Low-rank updates / quantized base	A	De facto standard

Table 8

Direction	Representative work	Core idea	Evidence	Notes
RLHF	InstructGPT [15] / PPO [196]	RM + PPO + KL	A	alignment base
AI feedback	RLAIF [38] / CAI [23]	AI replaces part of human annotation	B/C	bias risk
Self-rewarding	Self-Rewarding [223]	iterative bootstrapped preference	B	collapse risk
Preference optimization	DPO [32]	RM-free direct optimization	A	length bias [227]
Variants	IPO [214] / KTO [40] / ORPO [41] / SimPO [43]	robust/one-sided labels/merged/reference-free	A	each has trade-offs
Token-level	[229]	fine-grained signal	B	vs. sequence-level

Table 9

Direction	Representative work	Core idea	Evidence	Notes
Domain continued pretraining	[86] / [407]	Mixture ratio and two stages	B	Forgetting risk
Long-context extension	RoPE [11] / PI [143] / YaRN [37]	Positional-encoding adjustment	A/B	Requires long-document calibration
Model merging	Soups [296] / TIES [35]	Weight arithmetic merging	A/B	Requires same base
Structured pruning	LLM-Pruner [304] / Sheared [316]	Removing structural units	B	Requires recovery training
Unstructured pruning	Wanda [309]	Per-weight sparsity	B	Realization depends on hardware
Distillation	DistilBERT [290] / MiniLLM [308]	Sequence-level/on-policy	A/B	Distribution mismatch

Table 10

Paradigm	Representative Works	Core Idea	Evidence	Remarks
Contrastive pretraining	CLIP [10] / SigLIP [46]	Image-text contrast	A	Weak annotation
Bootstrapped pretraining	BLIP [254] / BLIP-2 [265]	Bootstrap image-text	A	Frozen encoder
Encoder–projector–LLM	LLaVA [28] / Qwen-VL [277]	Projector mapping	A	De facto mainstream
Cross-attention	Flamingo [17]	Gated cross-attention	A	Frozen LLM
Interleaved data	OBELISC [273]	Interleaved image-text training	B	Noisy segmentation
Unified discrete token	Chameleon [285] / VQ [3][8]	Any-to-any	B/C	Scalability to be verified

Table 11

Dimension	Pretraining	Post-Training: SFT	Preference Optimization	RL Depth	Continual Optimization	Evaluation
Compute share	High	Low	Medium-low	Medium	Medium	Low
Behavioral leverage	Medium	High	Medium-high	High (when verifiable)	Medium	High (adjudication)
Capability-wall impact	Determines	None (does not change the set)	None	None	Has side effects	None
Primary evidence	B/A	B/C	A/B	C	B/C	B
Key risk	Data contamination	Forgetting	Preference overfitting	Over-optimization	Capability drift	Contamination/bias
Typical cost entry	token compute	Data construction	Preference annotation	Compute + reward	GPU time	Protocol + annotation

Table 12

Conclusion category	Representative work	Evidence strength	Consensus / disagreement	Blank space (research opportunity)
Deduplication and mixture mechanisms	[13][86][87]	A	Consensus	Recipe-level optimal threshold—return curve
Scaling law (pretraining)	[119][16]	A/B	Consensus (basis-dependent)	Optimal-ratio transfer under data quality
MoE and sparsity	[9][151]	A/B	Partial consensus	Industrial-grade routing stability
PEFT / instruction tuning	[12][30][163][172]	A/B	Consensus	Cross-model optimal SFT recipe
Preference optimization (DPO family)	[32][43]	A/B	Consensus	Robustness under multi-objective conflict
RL and verifiable rewards	[15][45][224][400][401]	C	Disagreement (Pass@1 vs Pass@K)	Decoupling experiments of capability expansion vs behavior shaping

Table 13

Category	Representative work	Role	Evidence	Notes
General corpus	The Pile [54] / C4 [49][56]	Pretraining data	A	Licensing issues
Open corpus	Dolma [84]	Auditable data	A	Limited coverage
Image–text data	LAION-5B [64] / OBELISC [273]	Multimodal training	B	Filtering-dependent
Dataset documentation	Datasheets [61]	Reporting standard	A	Foundation of governance
Knowledge and reasoning	MMLU [7] / MMLU-Pro [381]	Capability measurement	A	Contamination present
Math/code	GSM8K [361] / HumanEval [357]	Reasoning/code	A	Protocol-sensitive

Table 14

Figure 4: Practical decision tree—budget $\rightarrow$ scale $\rightarrow$ modality $\rightarrow$ long context $\rightarrow$ post-training recipe $\rightarrow$ checks and rollback

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Model/Work	N (parameters)	D (tokens)	Optimizer/precision	Batch/sequence	Learning rate/schedule	Source
GPT-3	175B	300B	Adam( $\beta$ 1 =0.9, $\beta$ 2 =0.95), wd 0.1, clip 1.0	3.2M / 2048	0.6 $\times$ 10 - 4 , cosine $\rightarrow$ 10%, warmup 3.75 $\times$ 10 8 tokens	[121]
Chinchilla	70B	1.4T	AdamW; bf16 forward/backward + fp32 optimizer weight replica	1.5M $\rightarrow$ 3M (doubled midway) / [to be verified]	1 $\times$ 10 - 4 , cosine	[16]
LLaMA-65B	65.2B	1.4T	AdamW( $\beta$ 1 =0.9, $\beta$ 2 =0.95), wd 0.1, clip 1.0	4M / 2048	1.5 $\times$ 10 - 4 , cosine $\rightarrow$ 10%, warmup 2000 steps	[140]
Mixtral-8x7B	47B sparse / 13B active	[to be verified]	[to be verified]	[to be verified] / 32768	[to be verified]	[151]

Appendix

Table 16

Model/Work	Training FLOPs (approximate)	Hardware/accelerator	Wall-clock time	GPU memory strategy	Source
GPT-3	3.14 $\times$ 10 2 3 (given in the original)	V100 cluster (count undisclosed, provided by Microsoft)	Undisclosed	Intra-matmul + cross-layer model parallelism	[121]
Chinchilla	$\approx$ 5.76 $\times$ 10 2 3 (same compute as Gopher, given in the original)	Undisclosed	Undisclosed	[to be verified]	[16]
LLaMA-65B	$\approx$ 5.5 $\times$ 10 2 3 (estimated via C $\approx$ 6ND)	2048 $\times$ A100-80GB	$\approx$ 21 days (1.022M GPU·hours)	Model + sequence parallelism, activation recomputation	[140]

Appendix

Table 17

Chapter	Topic	Predominantly A/B	Predominantly C/blank
5	Data and tokenization	Cleaning/deduplication/tokenization	Synthetic-data collapse, multilingual scaling law
6	Pretraining	Scaling law, optimization, MoE (partial)	Largest-scale details, industrial-grade stability
7	SFT	Instruction data/format/training	Cross-model optimal recipe for SFT
8	Preference/RL	DPO family, RM mechanism	Depth boundary of RL, cross-model safety alignment
9	Continual optimization	Foundations of compression/editing	Compression–safety interaction, compositional side effects
10	Multimodal	Architecture/data pipeline/weak supervision	Scaling law, root causes of hallucination, unified tokens

Appendix

Table 18

LLM Training

Large language model training studies the data, objectives, optimizers, and systems used to build and adapt language models.

See all 353 papers

Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning
Apr 9, 2026Shiwan Zhao, Zhihu Wang, Xuyang Zhao +10LLM Post-Training RL for Language Models
Nankai University, China · Huawei Technologies Ltd., China
Theoretical Perspectives on Data Quality and Synergistic Effects in Pre- and Post-Training Reasoning Models
Mar 1, 2026Adel Javanmard, Baharan Mirzasoleiman, Vahab MirrokniData Selection for Language Model Fine-Tuning LLM Fine-Tuning
University of Southern California · Google Research · University of California Los Angeles
Post-training is (Massive) Supervised Learning
Apr 20, 2026Michael Hassid, Yossi Adi, Roy SchwartzSupervised Fine-Tuning Language Model Post-Training
1FAIR, Meta AI · 2The Hebrew University of Jerusalem

Apr 9, 2026J/K move · Enter open · S save

Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning

Shiwan Zhao, Zhihu Wang, Xuyang Zhao, Jiaming Zhou+9

Nankai University, China · Huawei Technologies Ltd., China

LLM Post-Training RL for Language Models RL Post-Training

Open paper AskPDF arXiv

Post-training has become central to turning pretrained large language models (LLMs) into aligned, capable, and deployable systems. Recent progress spans supervised fine-tuning (SFT), preference optimization, reinforcement learning (RL), process supervision, verifier-guided methods, distillation, and multi-stage pipelines. Yet these methods are often discussed in fragmented ways, organized by labels or objectives rather than by the behavioral bottlenecks they address. This survey argues that LLM post-training is best understood as structured intervention on model behavior. We organize the field first by trajectory provenance, which defines two primary regimes: off-policy learning on externally supplied trajectories and on-policy learning on learner-generated rollouts. We then interpret methods through two recurring roles -- effective support expansion, which makes useful behaviors more reachable, and policy reshaping, which improves behavior within already reachable regions -- together with a complementary systems-level role, behavioral consolidation, which preserves, transfers, and amortizes useful behavior across stages and model transitions. Under this view, SFT may serve either support expansion or policy reshaping; preference optimization is usually off-policy reshaping, though online variants move closer to learner-generated states. On-policy RL often improves behavior on learner-generated states, but stronger guidance can also make hard-to-reach reasoning paths reachable. Distillation is often better understood as consolidation rather than only compression, and hybrid pipelines emerge as coordinated multi-stage compositions. Overall, the framework helps diagnose post-training bottlenecks and reason about stage composition, suggesting that progress increasingly depends on coordinated systems design rather than any single dominant objective.

Abstract

LLM Training

Explore similar work

Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning

Theoretical Perspectives on Data Quality and Synergistic Effects in Pre- and Post-Training Reasoning Models

Post-training is (Massive) Supervised Learning