Training Foundation Models with a Single Algorithm

Yuda Song; Alvin Zhang; Fahim Tajwar; Daman Arora; Jeff Schneider; Daniel Khashabi; Ruslan Salakhutdinov; J. Andrew Bagnell

cs.LGOct 29, 2026

Training Foundation Models with a Single Algorithm

Authors: Yuda Song, Alvin Zhang, Fahim Tajwar, Daman Arora, Jeff Schneider, Daniel Khashabi, Ruslan Salakhutdinov, J. Andrew Bagnell,

Organizations: Carnegie Mellon University · Johns Hopkins University · Aurora Innovation

Abstract

Foundation models are typically trained in stages: pretraining, midtraining, supervised fine-tuning (SFT), and reinforcement learning (RL), with changes in both data and learning algorithms. We show that this algorithmic separation is not fundamental. By deriving suitable RL objectives as maximum-likelihood fitting to model-generated target distributions, we place all of these stages under a single likelihood-based training algorithm, unified training. This perspective turns the foundation-model training lifecycle into a problem of choosing which data to train on, in what proportions, and when, with conventional staged training as a special case. We instantiate this framework in a unified training system that mixes static data and RL generations within a single trainer. Across controlled and real-world experiments, unified training improves the accuracy-compute tradeoff and task coverage while preserving broader capabilities, and provides a stronger starting point for continual learning. This work lays the groundwork for a unified science of foundation-model training, connecting mathematical principles, training infrastructure, and the development of model capabilities.

Figures & tables

Panel of Figure 1 — Figure 1: Unified training framework. Left: For suitable RL objectives, static data and RL generations share a weighted likelihood loss; the updated model supplies new generations that are added to the data mixture. Right: Unified training reaches higher accuracy and coverage on a reasoning task within the same total compute budget. Each curve shows the best observed accuracy (solid) or coverage (dashed) attainable within a compute budget, across all tested unified (blue) or staged (gray) configurations. Compute includes pretraining, midtraining, and RL. Fig. 5 shows the individual training trajectories.

Source	Prompts $\rho_{s,t}$	Target $\mu_{s,t}$
Pretraining	empty	$\mu_{\mathrm{pre}}$
Midtraining	empty	$\mu_{\mathrm{mid}}$
SFT	$\rho_{\mathrm{sft}}$	$\mu_{\mathrm{sft}}$
RL	$\rho_{\mathrm{rl}}$	$\pi^{+}_{\theta_{t}}$ or $\nu^{+}$

Table 1: Data sources for the shared likelihood objective. For binary rewards,

\nu^{+}

is the success-conditioned distribution of the generation policy

\nu

;

\nu=\pi_{\theta_{t}}

recovers on-policy MaxRL .

Method	Reward	Target $\mu_{s}$	Effective prompts $\rho_{s}$
MaxRL	Binary	$\pi_{\theta_{t}}^{+}$	$\rho(x)$
REINFORCE	Binary	$\pi_{\theta_{t}}^{+}$	$\propto\rho(x)p_{\theta_{t}}(x)$
TailRL	$[0,1]$	$\displaystyle\int_{0}^{1}\pi_{\theta_{t}}(\cdot\mid x,r>\tau)\,d\tau$	$\rho(x)$
REINFORCE	$[0,1]$	$\displaystyle\frac{\pi_{\theta_{t}}(y\mid x)r(x,y)}{\mathbb{E}_{y^{\prime}\sim\pi_{\theta_{t}}(\cdot\mid x)}[r(x,y^{\prime})]}$	$\displaystyle\propto\rho(x)\mathbb{E}_{y\sim\pi_{\theta_{t}}(\cdot\mid x)}[r(x,y)]$

Table 2: RL objectives within the unified likelihood framework. Different objectives correspond to different target and effective prompt distributions.

Panel of Figure 2 — Figure 2: Left: the training system. The generator produces RL data from a recent copy of the weights and appends it to the data buffer, which also holds static data; the trainer samples from the buffer and periodically publishes its weights, which the generator loads. Right: the data format. Each cell is a token and the number is its weight in Eq. 2 ; the stages differ only in the weights.

Panel of Figure 3 — Figure 3: Data composition throughout training. The illustrated run from Section C.8 mixes midtraining tokens into pretraining, then combines RL generations, midtraining data, and pretraining data. Token counts include prompts and are rounded estimates; the axis break omits 10B pretraining tokens, and thin bands are enlarged for legibility.

Figure 4: Final performance and generalization. Mean pass@1 over two seeds after 6000 RL updates in Base and 12000 in Extended. Colors indicate midtraining tokens mixed during pretraining; the horizontal axis measures passes over the same fixed corpus during RL. Dashed lines show the best staged configuration on each setting’s primary test ( Section C.6 ). Extended generalization evaluates the same Extended models on a larger puzzles excluded from RL training ( Appendix C ).

Figure 5: Accuracy and coverage across training compute budgets. The top and bottom rows show pass@1 and pass@16 against cumulative compute, including pretraining, midtraining, and RL. Colored curves show unified configurations: darker colors indicate more midtraining compute before RL, while curves of the same color vary the midtraining mixture during RL. Gray dashed curves show all six staged configurations, spanning three midtraining budgets and two learning rates ( Section C.6 ). Open circles mark the pre-RL evaluations in Section C.5 . The black curve connects the best observed checkpoints for each metric, with colored or gray markers identifying unified or staged training. RL points average performance and estimated compute over two seeds. Each column ends at the largest final compute budget among its staged runs. Extended generalization evaluates the same Extended models on 12-person puzzles. Section A.2 gives the compute estimates.

Figure 6: Pretraining retention before RL. Unified initializations vary the amount of midtraining data mixed during pretraining. Horizontal references show pretraining alone and staged midtraining with one epoch at a peak learning rate of

5\times 10^{-5}

, selected for the best retention on both measures. Loss is measured on held-out pretraining data; mean score averages five general benchmarks. The loss axis is broken, with the lower interval expanded for readability. Section C.10 reports individual benchmarks and all staged configurations.

Figure 7: Effect of the midtraining data proportion. Final results after 12000 updates. Fractions are token proportions estimated before training. Loss changes are relative to the base model. Section D.1 gives the training and evaluation details.

Figure 8: Effect of the data mixing schedule. Four schedules with an average configured midtraining proportion of 0.5%. Full and half annealing reach zero at the end and midpoint, respectively; linear growth starts at zero. Training details are given in Section D.3 .

Figure 9: Coding RL after mathematical training. Curve labels identify the preceding mathematical training, with percentages denoting constant midtraining proportions. All models then use the same coding RL configuration without midtraining data. We report pass@1 and pass@16 on HumanEval+ (HE+) and MBPP+ over 4000 coding RL updates. Section D.7 gives the setup and additional evaluations of mathematics and general knowledge.

Appendix figures & tables42 assets

Supplementary material from the paper’s appendix.

Appendix

Midtraining sequences per update, $k$	Midtraining token fraction $k/(256+k)$
8	3.03%
16	5.88%
32	11.11%
64	20.00%
128	33.33%

Appendix

Table 3: Midtraining exposure during the final 8147 pretraining updates for the Base and Extended reasoning-puzzle settings.

Availability	Example reference
Included in midtraining	Ava’s best friend
	The best friend of Ava’s best friend
	The person directly above Ava
	The person on the second-highest floor
	The best friend of the person on floor 19
Introduced during RL	The best friend of the person on the highest floor

Appendix

Table 4: Reference patterns in the reasoning puzzles. Each row gives a schematic example of how a clue can identify a person or floor. The final three rows combine operations that occur in midtraining but whose composition is omitted from that corpus; the last row represents a family of longer nested references. Fig. 10 illustrates two such compositions in an actual question.

Panel of Figure 10 — Figure 10: Resolving composed references in the eight-person evaluation example. The diagram shows the gold assignment, with occupied floors spaced evenly for readability. The highest floor is occupied by Nora, whose best friend is Mia; Mia’s best friend is Felix. The two highlighted clues therefore reduce to $37-29=8$ and $22+2=24$ . This illustration uses the known answer to explain the task; the assignment is not part of the model input.

	People	Omitted steps	Base	Extended
RL prompts: demonstrated patterns	8–9	0	10000	10000
RL prompts: omitted compositions	8–9	2–4	18570	11666
RL prompts: larger puzzles	10–11	2–4	–	10000
Primary test questions	8–9	2–4	1024	–
Primary test questions	10–11	2–4	–	1024
Size-generalization test questions	12	2–4	–	1024

Appendix

Table 5: Question pools and training horizons for the two reasoning-puzzle settings. “Omitted steps” counts uses of compositions absent from midtraining in each puzzle’s clue chain. These compositions are introduced during RL, while the 12-person puzzle size is used only for evaluation. Test questions are distinct from training questions.

Setting	Value
Parameters / layers	123587328 / 12
Hidden / feedforward size	768 / 2048
Attention heads	12
Embeddings and normalization	Tied embeddings, rotary position embeddings, RMSNorm
Tokenizer / context length	GPT-2 / 2048 tokens
Global batch size	256 packed sequences of 2048 tokens

Appendix

Table 6: Model and original pretraining hyperparameters. Mixed-pretraining branches resume from update 30000 with the saved optimizer state and complete the remaining 8147 updates of the same learning-rate schedule.

Setting	Staged midtraining	RL (both approaches)
Optimizer	AdamW	AdamW
Adam betas	$(0.9,0.95)$	$(0.9,0.999)$
Learning rate	$2\times 10^{-4}$ peak	$10^{-5}$
Schedule	5% warmup, then cosine decay	Constant, no warmup
Weight decay	0.01	0.01
Gradient clip	1.0	1.0

Appendix

Table 7: Staged midtraining and shared RL hyperparameters. The midtraining column gives the selected configuration for both Base and Extended; Section C.6 reports the full sweep. Unified training uses the same RL settings and adds midtraining examples to each batch. To assess the contribution of this mixing during RL, we also run controls that start from the same mixed-pretraining initializations but add no midtraining examples.

Figure 11: Reasoning accuracy and coverage before RL. Each curve shows pass@

k

, the estimated probability of obtaining a correct answer in

k

attempts, for an initialization evaluated before RL. Colors indicate the number of midtraining tokens mixed during pretraining; the dashed gray curve shows the selected staged initialization. Midtraining puzzles contain 6–9 people, while the three panels use the Base, Extended, and Extended generalization test sets defined in Section C.1 . Estimates use 64 responses per question on 1024 questions in each test set. Vertical scales differ across panels; zero-valued curves overlap.

Unified: mixed during pretraining
	Demonstrated patterns		Base		Extended		Extended generalization
Midtraining	pass@1	pass@16	pass@1	pass@16	pass@1	pass@16	pass@1	pass@16
None	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
133M tokens	0.92	12.11	0.12	1.76	0.00	0.00	0.00	0.00
267M tokens	7.23	55.98	0.34	4.67	$<0.01$	0.15	0.00	0.00
534M tokens	16.74	79.67	0.55	7.01	0.05	0.81	$<0.01$	0.02

Appendix

Table 8: Reasoning performance across all initializations before RL. Values are percentages; positive values below 0.01% are shown as

<0.01

, and zeros indicate no observed correct responses. The demonstrated-pattern test contains held-out 8–9-person puzzles using only patterns present in midtraining; Base uses the same puzzle sizes but introduces omitted compositions. The shaded row is the staged configuration selected by final RL accuracy for both Base and Extended. Each row evaluates one initialization before RL, which is subsequently shared by the two RL training seeds.

		Base		Extended		Extended generalization
Epochs	Peak LR	pass@1	pass@16	pass@1	pass@16	pass@1	pass@16
1	$2\times 10^{-4}$	21.79	72.40	16.14	63.51	6.58	41.78
1	$5\times 10^{-5}$	12.24	54.54	9.88	51.30	3.18	26.31
2	$2\times 10^{-4}$	24.54	73.51	16.91	64.80	5.58	35.90
2	$5\times 10^{-5}$	12.74	55.25	7.25	41.13	2.17	18.86
4	$2\times 10^{-4}$	15.15	57.49	11.64	53.20	4.10	28.53

Appendix

Table 9: Staged performance after RL. Each row specifies the number of midtraining passes and peak learning rate used before RL. Pass@1 and pass@16 are percentages averaged over two RL training seeds, after 6000 updates in Base and 12000 in Extended. The shaded row maximizes final pass@1 independently on the Base and Extended primary test sets. Extended generalization evaluates the same Extended checkpoints on 12-person puzzles and is not used to select the reference.

Figure 12: Accuracy and coverage versus compute beyond shared pretraining. The top and bottom rows show pass@1 and pass@16, respectively. Gray dashed curves show all six staged configurations; colored curves show unified configurations, with darker colors indicating more midtraining compute before RL. All checkpoints also appear in Fig. 5 ; only their horizontal coordinates change by subtracting the common pretraining cost. The black curve connects checkpoints that attain the highest observed value within a compute budget across both approaches, with marker colors identifying their source. Open circles mark evaluations of the shared initializations before RL. Subsequent points use evaluations available for both RL seeds at the same update, averaging performance and estimated compute over those two seeds. Curves extend through available evaluations up to 6000 updates in Base and 12000 in Extended, subject to the displayed compute budget. Extended generalization evaluates the Extended checkpoints on 12-person puzzles and uses the same training costs.

Figure 13: Effect of retaining pretraining data during RL. Final accuracy (top) and coverage (bottom) average two training seeds after 6000 updates in Base and 12000 in Extended. Horizontal labels give the midtraining tokens used during pretraining (top line) and midtraining examples per RL update (bottom line).

	Midtraining		Pretraining loss $\downarrow$		Benchmark mean (%)
Setting	Tokens during pretraining	Examples per RL update	Without	With	Without	With
Base	534M	8	2.570	2.542	41.66	42.13
Base	534M	32	2.575	2.535	41.65	41.98
Base	2.14B	8	2.572	2.545	41.93	42.08
Base	2.14B	32	2.571	2.537	42.01	41.85
Extended	534M	8	2.613	2.546	41.60	42.25

Appendix

Table 10: Retention after RL with and without added pretraining data. Pretraining loss is measured in nats and weights held-out ClimbMix and Dolmino by 75/25; the benchmark mean averages the five tasks in Section C.4 . Scores use the same final checkpoints as Fig. 13 and average two training seeds.

Figure 14: Loss aggregation in Base. The upper three rows show pass@1, and the lower three show pass@16, through 6000 RL updates. Within each group, rows vary the number of midtraining tokens used during pretraining; columns vary the number of midtraining examples added to each RL update. Each panel compares unified training with two separately normalized objectives, holding the initialization and example counts fixed. Curves average the scores of two RL training seeds at each update.

Figure 15: Loss aggregation in Extended. Pass@1 (upper three rows) and pass@16 (lower three rows) are measured on the primary Extended test set through 12000 RL updates. Rows and columns follow Fig. 14 , using the same three initializations and five midtraining example counts. The

\alpha=1

curves are included for the 2.14B-token initialization. Each panel holds the initialization and data mixture fixed while varying the loss aggregation. Curves average the scores of two RL training seeds.

Figure 16: Loss aggregation and Extended generalization. The models in Fig. 15 are evaluated on 12-person puzzles, a size absent from RL training. The upper three rows show pass@1 and the lower three show pass@16, with the same arrangement of initializations and midtraining example counts. As in Fig. 15 ,

\alpha=1

is evaluated for the 2.14B-token initialization. Curves average the scores of two RL training seeds through 12000 updates.

Figure 17: Loss aggregation with smaller pretraining mixtures in Base. The upper two rows show pass@1 and the lower two show pass@16 for the 133M- and 267M-token initializations, through 6000 RL updates. Columns vary the number of midtraining examples added to each RL update. Each panel compares the three objectives with the same initialization and example counts, averaging scores over two RL training seeds.

Figure 18: Loss aggregation with a smaller pretraining mixture in Extended. The top row shows pass@1 and the bottom row shows pass@16 for the 267M-token initialization through 12000 RL updates. Columns vary the number of midtraining examples per RL update, with the initialization and example counts held fixed within each comparison. Curves average scores over two RL training seeds.

Figure 19: Extended generalization with a smaller pretraining mixture. The models in Fig. 18 are evaluated on the 12-person test set through 12000 RL updates. The top row shows pass@1 and the bottom row shows pass@16 for the 267M-token initialization. Each curve averages the per-model score over two RL training seeds, without pooling generations from different models.

Unified: mixed during pretraining
	Held-out loss $\downarrow$			Benchmark score (%)
Midtraining	ClimbMix	Dolmino	Mean	MMLU	ARC-E	HellaSwag	TQA	WinoG	Mean
None	2.784	1.794	2.537	28.65	55.09	35.39	39.15	53.59	42.37
133M tokens	2.784	1.795	2.537	28.62	54.67	35.55	39.09	53.20	42.23
267M tokens	2.786	1.796	2.538	28.79	54.97	35.17	39.17	53.99	42.42
534M tokens	2.785	1.797	2.538	28.22	54.67	35.67	39.57	51.14	41.85

Appendix

Table 11: Pretraining retention across all initializations. Every row evaluates one initialization before RL; these results are shared by the Base and Extended experiments. Loss is in nats per token, with a mean weighted by 75% ClimbMix and 25% Dolmino. Benchmark scores are percentages, with an unweighted mean over the five tasks. ARC-E denotes ARC-Easy normalized accuracy, TQA denotes TruthfulQA MC2, and WinoG denotes WinoGrande accuracy; HellaSwag uses normalized accuracy and MMLU uses accuracy. Midtraining token counts include prompts and responses; staged configurations also indicate the number of passes over the corpus. The shaded row is the staged initialization with the best retention on both aggregate measures, used in Fig. 6 .

Setting	Pretraining	Midtraining
Optimizer	AdamW
Adam $(\beta_{1},\beta_{2})$ ; $\varepsilon$	$(0.9,0.95)$ ; $10^{-8}$
Weight decay; gradient clipping norm	0.1; 1
Global batch size (sequences)	2048
Sequence length (tokens)	2048
Peak / minimum learning rate	$4\times 10^{-4}$ / $4\times 10^{-5}$

Appendix

Table 12: Pretraining and midtraining hyperparameters for the 0.76B base model. The midtraining column describes training on the Dolmino corpus.

Setting	Unified	Staged RL
Optimizer	AdamW	AdamW
Adam $(\beta_{1},\beta_{2})$	$(0.9,0.999)$	$(0.9,0.999)$
Adam $\varepsilon$	$10^{-8}$	$10^{-8}$
Weight decay	0	0
Gradient clipping norm	1	1
Learning rate	$10^{-6}$	$10^{-6}$

Appendix

Table 13: RL hyperparameters.

Benchmark	Pass@1	Pass@32
MATH-500	6.54	43.76
GSM8K	17.84	71.63
Math (average)	12.19	57.69
HumanEval+	11.97	32.32
MBPP+	21.05	57.14
Coding (average)	16.51	44.73

Appendix

Table 14: Base-model performance before mathematical RL. Scores are percentages, except held-out loss in nats/token. Pass@1 uses 16 responses per problem; pass@32 uses 256 responses for math and 32 for coding. Math and Coding are the equally weighted benchmark averages defined in Section D.1 .

Figure 20: Training curves for different midtraining data proportions. Each unified curve uses a constant configured proportion of 0, 0.1, 0.2, 0.5, 1, 2, or 5%; the dashed curves show staged training. Math and Coding average the two benchmarks in each domain, as defined in Section D.1 , which also gives the evaluation sample counts.

Mixing	MATH pass@1	GSM pass@1	MATH pass@64	GSM pass@64	MATH pass@256	GSM pass@256
0%	16.83	32.68	66.73	83.88	78.20	91.81
0.1%	16.90	32.14	65.75	85.56	77.00	93.03
0.2%	16.56	32.35	65.03	84.74	76.20	92.49
0.5%	16.83	31.71	65.60	84.06	77.20	91.81
1%	16.11	31.61	65.85	83.71	76.00	91.74
2%	16.68	31.30	67.51	84.90	78.40	93.03

Appendix

Table 15: Mathematical accuracy and coverage at different midtraining proportions. The Base model row reports performance before RL; all other rows report final results after RL. MATH and GSM denote MATH-500 and GSM8K, and all scores are percentages. Pass@1 is estimated from 16 responses per problem; pass@64 and pass@256 use a separate evaluation with 256 responses per problem.

Mixing	$\Delta$ loss $\downarrow$	MMLU	SciQ	ARC-C	MBPP+	HumanEval+
0%	$+0.0737$	37.12	50.20	27.90	25.41	15.17
0.1%	$-0.0154$	39.30	54.60	29.69	24.82	14.44
0.2%	$-0.0516$	40.04	54.30	31.06	23.71	13.76
0.5%	$-0.0871$	39.99	62.00	32.34	21.05	11.97
1%	$-0.1076$	40.54	66.10	33.53	19.31	11.66
2%	$-0.1269$	40.66	67.00	34.22	19.23	11.55

Appendix

Table 16: Held-out loss and performance beyond math. The Base model row gives performance before RL, and the remaining rows give final results after mathematical RL. Scores are percentages;

\Delta

loss is the change in held-out loss from the base model, in nats/token. The coding columns report pass@1 estimated from 16 responses per problem. ARC-C denotes ARC-Challenge.

Figure 21: Accuracy on individual benchmarks. Final pass@1 on MATH-500, GSM8K, HumanEval+, and MBPP+ is estimated from 16 responses per problem. The dashed reference shows staged training.

Figure 22: Coverage and knowledge accuracy across midtraining proportions. Final results are shown for all seven configured midtraining proportions. Mathematical pass@32 is estimated from 256 responses per problem, and coding pass@32 from 32 responses per problem. SciQ and ARC-Challenge use the greedy evaluation protocol in Section D.1 .

Figure 23: Full results for the four mixing schedules. These are the runs in Fig. 8 , with additional panels for Math and Coding accuracy and coverage, held-out loss change, MMLU, SciQ, and ARC-Challenge. The vertical line marks the midpoint of training, when the half-annealing schedule stops adding midtraining data.

Schedule	Math pass@1	Math pass@32	Coding pass@1	Coding pass@32	$\Delta$ loss $\downarrow$	MMLU
Constant 0.5%	24.27	68.32	16.51	46.38	$-0.0871$	39.99
Full anneal 1%	24.70	68.94	17.76	46.16	$-0.0833$	39.96
Ramp to 1%	24.01	68.94	16.33	46.55	$-0.0872$	40.17
Half anneal 2%	24.80	68.49	18.85	46.93	$-0.0634$	40.14

Appendix

Table 17: Results for the four mixing schedules. Each schedule has an average configured midtraining proportion of 0.5%. Scores are percentages, except

\Delta

loss in nats/token. SciQ and ARC-Challenge appear in Fig. 23 .

Figure 24: Results for additional mixing schedules. Full, half, and three-quarter annealing reach zero at the end, midpoint, and three-quarter point of training, respectively. The held-out loss panel reports loss in nats/token, rather than its change from the base model.

Figure 25: Effect of removing or replacing code data. We compare the original corpus with variants that remove its designated code sources or replace them with verified coding examples. Each corpus is evaluated with constant 0.5% mixing and full annealing from 0.5%. Math and Coding are the benchmark averages defined in Section D.1 .

Setting	MATH	GSM	$\Delta$ loss $\downarrow$	SciQ	MBPP+	HumanEval+
constant: original	16.83	31.71	$-0.0871$	62.00	21.05	11.97
constant: code removed	16.05	32.15	$-0.0898$	61.90	23.66	13.64
constant: code replaced	16.54	31.22	$-0.0819$	64.80	20.80	12.12
anneal: original	15.50	32.55	$-0.0552$	59.00	23.69	14.41
anneal: code removed	16.40	32.07	$-0.0633$	56.80	24.01	14.33
anneal: code replaced	17.00	31.81	$-0.0522$	61.70	23.38	13.11

Appendix

Table 18: Results with different code data. Each corpus variant is evaluated with constant 0.5% mixing and with full decay from 0.5% to zero at the end of training. The original corpus is compared with variants that remove or replace its designated code sources, as described in Section D.4 . Mathematical and coding scores are pass@1 (%); SciQ reports accuracy (%). MATH denotes MATH-500, GSM denotes GSM8K, and loss change is measured in nats/token relative to the base model.

Method	Math pass@1	Math pass@32	Coding pass@1	Coding pass@32	$\Delta$ loss $\downarrow$	MMLU	SciQ	ARC-C
MaxRL	23.99	69.61	15.39	45.51	$-0.127$	40.66	67.00	34.22
REINFORCE	24.81	67.70	15.80	45.21	$-0.157$	40.69	71.70	36.43

Appendix

Table 19: MaxRL and REINFORCE with a constant midtraining proportion of 2%. Scores are percentages, except held-out loss change relative to the base model, measured in nats/token. ARC-C denotes ARC-Challenge. Training curves and comparisons across mixing settings appear in Figs. 26 and 27 .

Figure 26: MaxRL and REINFORCE with a constant midtraining proportion of 2%. Training curves for the runs in Table 19 , evaluated using the protocol in Section D.1 . Both algorithms use unified training; blue curves use MaxRL and orange curves use REINFORCE .

Figure 27: Final MaxRL – REINFORCE comparison across mixing settings. Each difference subtracts the MaxRL result from the REINFORCE result at the same mixing proportion and schedule. Positive values therefore indicate a higher value under REINFORCE ; for held-out loss, lower values are preferable. Differences are in percentage points for performance metrics and nats/token for loss.

Mixing setting	Method	Math pass@1	Math pass@32	Coding pass@1	Coding pass@32	$\Delta$ loss $\downarrow$	MMLU	SciQ	ARC-C
Constant 0.5%	MaxRL	24.27	68.32	16.51	46.38	$-0.087$	39.99	62.00	32.34
	REINFORCE	22.98	65.29	14.45	46.00	$-0.120$	41.13	73.90	38.82
Constant 1%	MaxRL	23.86	68.25	15.49	44.89	$-0.108$	40.54	66.10	33.53
	REINFORCE	24.51	66.41	15.60	44.24	$-0.144$	40.82	70.30	35.58
Constant 2%	MaxRL	23.99	69.61	15.39	45.51	$-0.127$	40.66	67.00	34.22
	REINFORCE	24.81	67.70	15.80	45.21	$-0.157$	40.69	71.70	36.43

Appendix

Table 20: MaxRL and REINFORCE across midtraining proportions and schedules. Each pair of rows compares the two algorithms with the same configured proportion and schedule, giving the absolute final scores underlying the differences in Fig. 27 . Half annealing decreases the stated initial proportion linearly to zero halfway through training. Scores are percentages, except held-out loss change relative to the base model, which is measured in nats/token. ARC-C denotes ARC-Challenge.

Setting	Value
Algorithm	On-policy MaxRL with reward-baseline subtraction
Optimizer	Fresh AdamW
Adam $(\beta_{1},\beta_{2})$ ; $\varepsilon$	$(0.9,0.999)$ ; $10^{-8}$
Weight decay; gradient clipping norm	0; 1
Learning rate / schedule	$10^{-6}$ / constant, no warmup
Prompts per update	32

Appendix

Table 21: Hyperparameters for coding and WebInstruct RL. All models use these same RL settings when trained on coding tasks or on WebInstruct after mathematical training.

	HumanEval+		MBPP+
Mathematical training	Pass@1	Pass@16	Pass@1	Pass@16
Staged	22.94	45.12	29.81	52.12
Unified 0.5%	23.67	43.90	30.31	52.91
Unified 1%	23.70	45.12	29.94	54.76
Unified 2%	24.09	44.51	31.76	56.88
Unified 5%	24.96	46.95	32.69	58.47

Appendix

Table 22: Coding performance after subsequent RL. Scores are measured after 4000 coding RL updates and reported in percent. Percentages in the model labels identify the midtraining proportions during mathematical training. Both metrics use 16 responses per problem; bold indicates the highest score in each column.

Figure 28: Mathematical performance and retention during coding RL. The initializations and coding updates are the same as in Fig. 9 . Percentages denote constant midtraining proportions during the preceding mathematical training. Pass@1, pass@16, and benchmark accuracy are in percent; held-out loss is in nats/token. Mathematical accuracy remains close to its starting level, while changes in coverage vary across models. Held-out loss increases for every model.

Figure 29: WebInstruct RL after mathematical training. Percentages identify the midtraining proportions during mathematical training; all models then use the same WebInstruct RL procedure without midtraining data. The top row shows validation accuracy and overall test performance; the bottom row separates test performance by numerical and free-form answers. All performance metrics are in percent.

Figure 30: Midtraining validation loss during KL-regularized RL and unified training. Each curve shows the change from the common base-model value, measured on the trainer’s fixed set of 512 held-out midtraining windows. This loss averages over tokens in the validation windows; it is distinct from the equally weighted average over corpus components in Fig. 31 . Negative values indicate improvement over the base model. Curves show recorded evaluations without smoothing.

Figure 31: Benchmark performance during KL-regularized RL and unified training. Solid blue curves show unified training at five midtraining proportions; dashed orange curves show RL with three KL coefficients. Markers indicate recorded evaluations, joined without smoothing. Math and Coding average the two benchmarks in each domain, as defined in Section D.1 . Math pass@32 uses 256 responses per problem, and Coding pass@32 uses a separate 32-response evaluation. Held-out loss change gives equal weight to each of the 24 midtraining corpus components, with negative values indicating improvement over the base model. It is distinct from the token-averaged validation loss in Fig. 30 .

LLM Training

Large language model training studies the data, objectives, optimizers, and systems used to build and adapt language models.

See all 353 papers

Oct 7, 2026

Sequential Pretraining Favors Large Models

Large neural networks often acquire capabilities that small models fail to learn. Does this stem from large models learning more representative features, or from being more robust to unaccounted-for adverse effects introduced during training? We define and quantify one such adverse effect, primacy bias, as the extent to which exposure to early data distributions impairs later learning. We show that small models can allocate learning capacity inefficiently toward early distributions, whereas sufficiently overparameterized models are robust to this effect. This inefficiency is particularly consequential in pretraining, where foundation models often encounter heterogeneous data distributions sequentially rather than jointly. As a result, small foundation models can struggle to learn distributions encountered late in training, which is particularly harmful when later data emphasizes desirable capabilities such as code, mathematics, and reasoning. Motivated by these findings, we introduce Exposure Therapy (ET), a simple regularization that promotes more efficient allocation of learning capacity during sequential pretraining. We demonstrate that ET improves foundation models' performance on late data distributions as well as overall capability in models up to the billion-parameter scale. Overall, our results suggest that some benefits of large foundation models may arise from greater robustness to adverse training effects, rather than from learning more representative features, and that improved training algorithms can recover some of these advantages in smaller models.

Mohnish Harwani, Yujia Zheng

Purdue University · University of Illinois Urbana–Champaign

Language Model Pretraining Continued Pretraining

Ask PDF

Aug 25, 2026

Unsupervised Post-Training of Foundation Models: A Survey

Foundation-model post-training usually relies on human labels, preference data, stronger teachers, or executable verifiers. We study Unsupervised Post-Training (UPT): update-bearing adaptation on unlabeled inputs whose learning signal is derived from same-lineage model artifacts rather than an external oracle. We catalog 80 strict UPT methods and organize them by the object that supplies the update signal: a prediction statistic, a sample relation, a self-generated target, or an internal evaluator. Beyond inventory, we show how the choice of internal signal and task structure determines whether post-training improves the model or recursively amplifies error. An orthogonal Input Visibility

\times

Update Persistence view maps deployment regimes and defines a unified framework for UPT selection and evaluation.

Yijie Xu, Qianyi Cai, Huizai Yao +9

HKUST(GZ) · Xiaohongshu Inc. · WeChat, Tencent +3

LLM Post-Training Self-Training

Ask PDF

Jul 12, 2026

Learning to Fine-tune Foundation Models under Resource Limitations

We study the problem of optimal continual fine-tuning for a pre-trained Foundation Model deployed at a resource-limited device. At each time slot, a new batch of training data arrives, and the controller is faced with two options: either use the data to fine-tune the model and incur a compute cost, or do not fine-tune the model and discard the data. After the decision, the performance of the current model is measured in terms of an application-specific performance metric such as classification accuracy. Our objective is to learn an optimal policy that determines when to fine-tune the model on a single task (e.g., sentiment analysis), under a finite compute budget. We formulate this online decision-making problem as a constrained Markov Decision Process, where the system state captures three essential aspects: (i) model's performance, (ii) computational budget, and (iii) data distribution relevance to historic data encountered up to that point. The transition to the next state is stochastic and therefore, we propose a reinforcement learning-based method to solve this problem, namely the actor-critic algorithm. We also consider the special case where the performance of fine-tuning for a given model can be predicted or estimated prior to decision; in this case the problem becomes a Dynamic Programming one. Experiments with a large pre-trained model on a widely-used text classification dataset demonstrate that our method consistently outperforms fine-tuning approaches with the same compute budget by more than

4\%

in terms of accuracy and achieves

97\%

of full-parameter fine-tuning accuracy while requiring only

25\%

of the fine-tuning steps.

Thomas Tsouparopoulos, Iordanis Koutsopoulos

Department of Informatics Athens University of Economics and Business Athens, Greece

Fine-Tuning LLM Fine-Tuning Online Continual Learning

Ask PDF

Abstract

LLM Training

Explore similar work

Sequential Pretraining Favors Large Models

Unsupervised Post-Training of Foundation Models: A Survey

Learning to Fine-tune Foundation Models under Resource Limitations