cs.LGOct 29, 2026

Training Foundation Models with a Single Algorithm

Authors: Yuda Song, Alvin Zhang, Fahim Tajwar, Daman Arora, Jeff Schneider, Daniel Khashabi, Ruslan Salakhutdinov, J. Andrew Bagnell, 

Organizations: Carnegie Mellon University · Johns Hopkins University · Aurora Innovation

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Foundation models are typically trained in stages: pretraining, midtraining, supervised fine-tuning (SFT), and reinforcement learning (RL), with changes in both data and learning algorithms. We show that this algorithmic separation is not fundamental. By deriving suitable RL objectives as maximum-likelihood fitting to model-generated target distributions, we place all of these stages under a single likelihood-based training algorithm, unified training. This perspective turns the foundation-model training lifecycle into a problem of choosing which data to train on, in what proportions, and when, with conventional staged training as a special case. We instantiate this framework in a unified training system that mixes static data and RL generations within a single trainer. Across controlled and real-world experiments, unified training improves the accuracy-compute tradeoff and task coverage while preserving broader capabilities, and provides a stronger starting point for continual learning. This work lays the groundwork for a unified science of foundation-model training, connecting mathematical principles, training infrastructure, and the development of model capabilities.

Figures & tables

Figure 1: Unified training framework. Left: For suitable RL objectives, static data and RL generations share a weighted likelihood loss; the updated model supplies new generations that are added to the data mixture. Right: Unified training reaches higher accuracy and coverage on a reasoning task within the same total compute budget. Each curve shows the best observed accuracy (solid) or coverage (dashed) attainable within a compute budget, across all tested unified (blue) or staged (gray) configurations. Compute includes pretraining, midtraining, and RL. Fig. 5 shows the individual training trajectories.
SourcePrompts ρs,t\rho_{s,t}Target μs,t\mu_{s,t}
Pretrainingemptyμpre\mu_{\mathrm{pre}}
Midtrainingemptyμmid\mu_{\mathrm{mid}}
SFTρsft\rho_{\mathrm{sft}}μsft\mu_{\mathrm{sft}}
RLρrl\rho_{\mathrm{rl}}πθt+\pi^{+}_{\theta_{t}} or ν+\nu^{+}
Table 1: Data sources for the shared likelihood objective. For binary rewards, ν+\nu^{+} is the success-conditioned distribution of the generation policy ν\nu ; ν=πθt\nu=\pi_{\theta_{t}} recovers on-policy MaxRL .
MethodRewardTarget μs\mu_{s}Effective prompts ρs\rho_{s}
MaxRLBinaryπθt+\pi_{\theta_{t}}^{+}ρ(x)\rho(x)
REINFORCEBinaryπθt+\pi_{\theta_{t}}^{+}∝ρ(x)pθt(x)\propto\rho(x)p_{\theta_{t}}(x)
TailRL[0,1][0,1]∫01πθt(⋅∣x,r>τ) dτ\displaystyle\int_{0}^{1}\pi_{\theta_{t}}(\cdot\mid x,r>\tau)\,d\tauρ(x)\rho(x)
REINFORCE[0,1][0,1]πθt(y∣x)r(x,y)Ey′∼πθt(⋅∣x)[r(x,y′)]\displaystyle\frac{\pi_{\theta_{t}}(y\mid x)r(x,y)}{\mathbb{E}_{y^{\prime}\sim\pi_{\theta_{t}}(\cdot\mid x)}[r(x,y^{\prime})]}∝ρ(x)Ey∼πθt(⋅∣x)[r(x,y)]\displaystyle\propto\rho(x)\mathbb{E}_{y\sim\pi_{\theta_{t}}(\cdot\mid x)}[r(x,y)]
Table 2: RL objectives within the unified likelihood framework. Different objectives correspond to different target and effective prompt distributions.
Figure 2: Left: the training system. The generator produces RL data from a recent copy of the weights and appends it to the data buffer, which also holds static data; the trainer samples from the buffer and periodically publishes its weights, which the generator loads. Right: the data format. Each cell is a token and the number is its weight in Eq. 2 ; the stages differ only in the weights.
Figure 3: Data composition throughout training. The illustrated run from Section C.8 mixes midtraining tokens into pretraining, then combines RL generations, midtraining data, and pretraining data. Token counts include prompts and are rounded estimates; the axis break omits 10B pretraining tokens, and thin bands are enlarged for legibility.
Figure 4: Final performance and generalization. Mean pass@1 over two seeds after 6000 RL updates in Base and 12000 in Extended. Colors indicate midtraining tokens mixed during pretraining; the horizontal axis measures passes over the same fixed corpus during RL. Dashed lines show the best staged configuration on each setting’s primary test ( Section C.6 ). Extended generalization evaluates the same Extended models on a larger puzzles excluded from RL training ( Appendix C ).
Figure 5: Accuracy and coverage across training compute budgets. The top and bottom rows show pass@1 and pass@16 against cumulative compute, including pretraining, midtraining, and RL. Colored curves show unified configurations: darker colors indicate more midtraining compute before RL, while curves of the same color vary the midtraining mixture during RL. Gray dashed curves show all six staged configurations, spanning three midtraining budgets and two learning rates ( Section C.6 ). Open circles mark the pre-RL evaluations in Section C.5 . The black curve connects the best observed checkpoints for each metric, with colored or gray markers identifying unified or staged training. RL points average performance and estimated compute over two seeds. Each column ends at the largest final compute budget among its staged runs. Extended generalization evaluates the same Extended models on 12-person puzzles. Section A.2 gives the compute estimates.
Figure 6: Pretraining retention before RL. Unified initializations vary the amount of midtraining data mixed during pretraining. Horizontal references show pretraining alone and staged midtraining with one epoch at a peak learning rate of 5×10−55\times 10^{-5} , selected for the best retention on both measures. Loss is measured on held-out pretraining data; mean score averages five general benchmarks. The loss axis is broken, with the lower interval expanded for readability. Section C.10 reports individual benchmarks and all staged configurations.
Figure 7: Effect of the midtraining data proportion. Final results after 12000 updates. Fractions are token proportions estimated before training. Loss changes are relative to the base model. Section D.1 gives the training and evaluation details.
Figure 8: Effect of the data mixing schedule. Four schedules with an average configured midtraining proportion of 0.5%. Full and half annealing reach zero at the end and midpoint, respectively; linear growth starts at zero. Training details are given in Section D.3 .
Figure 9: Coding RL after mathematical training. Curve labels identify the preceding mathematical training, with percentages denoting constant midtraining proportions. All models then use the same coding RL configuration without midtraining data. We report pass@1 and pass@16 on HumanEval+ (HE+) and MBPP+ over 4000 coding RL updates. Section D.7 gives the setup and additional evaluations of mathematics and general knowledge.
Appendix figures & tables42 assets

Supplementary material from the paper’s appendix.

Appendix

Midtraining sequences per update, kkMidtraining token fraction k/(256+k)k/(256+k)
83.03%
165.88%
3211.11%
6420.00%
12833.33%
Appendix
Table 3: Midtraining exposure during the final 8147 pretraining updates for the Base and Extended reasoning-puzzle settings.
AvailabilityExample reference
Included in midtrainingAva’s best friend
 The best friend of Ava’s best friend
 The person directly above Ava
 The person on the second-highest floor
 The best friend of the person on floor 19
Introduced during RLThe best friend of the person on the highest floor
Appendix
Table 4: Reference patterns in the reasoning puzzles. Each row gives a schematic example of how a clue can identify a person or floor. The final three rows combine operations that occur in midtraining but whose composition is omitted from that corpus; the last row represents a family of longer nested references. Fig. 10 illustrates two such compositions in an actual question.
Figure 10: Resolving composed references in the eight-person evaluation example. The diagram shows the gold assignment, with occupied floors spaced evenly for readability. The highest floor is occupied by Nora, whose best friend is Mia; Mia’s best friend is Felix. The two highlighted clues therefore reduce to 37−29=837-29=8 and 22+2=2422+2=24 . This illustration uses the known answer to explain the task; the assignment is not part of the model input.
 PeopleOmitted stepsBaseExtended
RL prompts: demonstrated patterns8–901000010000
RL prompts: omitted compositions8–92–41857011666
RL prompts: larger puzzles10–112–4–10000
Primary test questions8–92–41024–
Primary test questions10–112–4–1024
Size-generalization test questions122–4–1024
Appendix
Table 5: Question pools and training horizons for the two reasoning-puzzle settings. “Omitted steps” counts uses of compositions absent from midtraining in each puzzle’s clue chain. These compositions are introduced during RL, while the 12-person puzzle size is used only for evaluation. Test questions are distinct from training questions.
SettingValue
Parameters / layers123587328 / 12
Hidden / feedforward size768 / 2048
Attention heads12
Embeddings and normalizationTied embeddings, rotary position embeddings, RMSNorm
Tokenizer / context lengthGPT-2 / 2048 tokens
Global batch size256 packed sequences of 2048 tokens
Appendix
Table 6: Model and original pretraining hyperparameters. Mixed-pretraining branches resume from update 30000 with the saved optimizer state and complete the remaining 8147 updates of the same learning-rate schedule.
SettingStaged midtrainingRL (both approaches)
OptimizerAdamWAdamW
Adam betas(0.9,0.95)(0.9,0.95)(0.9,0.999)(0.9,0.999)
Learning rate2×10−42\times 10^{-4} peak10−510^{-5}
Schedule5% warmup, then cosine decayConstant, no warmup
Weight decay0.010.01
Gradient clip1.01.0
Appendix
Table 7: Staged midtraining and shared RL hyperparameters. The midtraining column gives the selected configuration for both Base and Extended; Section C.6 reports the full sweep. Unified training uses the same RL settings and adds midtraining examples to each batch. To assess the contribution of this mixing during RL, we also run controls that start from the same mixed-pretraining initializations but add no midtraining examples.
Figure 11: Reasoning accuracy and coverage before RL. Each curve shows pass@ kk , the estimated probability of obtaining a correct answer in kk attempts, for an initialization evaluated before RL. Colors indicate the number of midtraining tokens mixed during pretraining; the dashed gray curve shows the selected staged initialization. Midtraining puzzles contain 6–9 people, while the three panels use the Base, Extended, and Extended generalization test sets defined in Section C.1 . Estimates use 64 responses per question on 1024 questions in each test set. Vertical scales differ across panels; zero-valued curves overlap.
 Demonstrated patternsBaseExtendedExtended generalization
Midtrainingpass@1pass@16pass@1pass@16pass@1pass@16pass@1pass@16
None0.000.000.000.000.000.000.000.00
Unified: mixed during pretraining
133M tokens0.9212.110.121.760.000.000.000.00
267M tokens7.2355.980.344.67<0.01<0.010.150.000.00
534M tokens16.7479.670.557.010.050.81<0.01<0.010.02
Appendix
Table 8: Reasoning performance across all initializations before RL. Values are percentages; positive values below 0.01% are shown as <0.01<0.01 , and zeros indicate no observed correct responses. The demonstrated-pattern test contains held-out 8–9-person puzzles using only patterns present in midtraining; Base uses the same puzzle sizes but introduces omitted compositions. The shaded row is the staged configuration selected by final RL accuracy for both Base and Extended. Each row evaluates one initialization before RL, which is subsequently shared by the two RL training seeds.
  BaseExtendedExtended generalization
EpochsPeak LRpass@1pass@16pass@1pass@16pass@1pass@16
12×10−42\times 10^{-4}21.7972.4016.1463.516.5841.78
15×10−55\times 10^{-5}12.2454.549.8851.303.1826.31
22×10−42\times 10^{-4}24.5473.5116.9164.805.5835.90
25×10−55\times 10^{-5}12.7455.257.2541.132.1718.86
42×10−42\times 10^{-4}15.1557.4911.6453.204.1028.53
Appendix
Table 9: Staged performance after RL. Each row specifies the number of midtraining passes and peak learning rate used before RL. Pass@1 and pass@16 are percentages averaged over two RL training seeds, after 6000 updates in Base and 12000 in Extended. The shaded row maximizes final pass@1 independently on the Base and Extended primary test sets. Extended generalization evaluates the same Extended checkpoints on 12-person puzzles and is not used to select the reference.
Figure 12: Accuracy and coverage versus compute beyond shared pretraining. The top and bottom rows show pass@1 and pass@16, respectively. Gray dashed curves show all six staged configurations; colored curves show unified configurations, with darker colors indicating more midtraining compute before RL. All checkpoints also appear in Fig. 5 ; only their horizontal coordinates change by subtracting the common pretraining cost. The black curve connects checkpoints that attain the highest observed value within a compute budget across both approaches, with marker colors identifying their source. Open circles mark evaluations of the shared initializations before RL. Subsequent points use evaluations available for both RL seeds at the same update, averaging performance and estimated compute over those two seeds. Curves extend through available evaluations up to 6000 updates in Base and 12000 in Extended, subject to the displayed compute budget. Extended generalization evaluates the Extended checkpoints on 12-person puzzles and uses the same training costs.
Figure 13: Effect of retaining pretraining data during RL. Final accuracy (top) and coverage (bottom) average two training seeds after 6000 updates in Base and 12000 in Extended. Horizontal labels give the midtraining tokens used during pretraining (top line) and midtraining examples per RL update (bottom line).
 MidtrainingPretraining loss ↓\downarrowBenchmark mean (%)
SettingTokens during pretrainingExamples per RL updateWithoutWithWithoutWith
Base534M82.5702.54241.6642.13
Base534M322.5752.53541.6541.98
Base2.14B82.5722.54541.9342.08
Base2.14B322.5712.53742.0141.85
Extended534M82.6132.54641.6042.25
Appendix
Table 10: Retention after RL with and without added pretraining data. Pretraining loss is measured in nats and weights held-out ClimbMix and Dolmino by 75/25; the benchmark mean averages the five tasks in Section C.4 . Scores use the same final checkpoints as Fig. 13 and average two training seeds.
Figure 14: Loss aggregation in Base. The upper three rows show pass@1, and the lower three show pass@16, through 6000 RL updates. Within each group, rows vary the number of midtraining tokens used during pretraining; columns vary the number of midtraining examples added to each RL update. Each panel compares unified training with two separately normalized objectives, holding the initialization and example counts fixed. Curves average the scores of two RL training seeds at each update.
Figure 15: Loss aggregation in Extended. Pass@1 (upper three rows) and pass@16 (lower three rows) are measured on the primary Extended test set through 12000 RL updates. Rows and columns follow Fig. 14 , using the same three initializations and five midtraining example counts. The α=1\alpha=1 curves are included for the 2.14B-token initialization. Each panel holds the initialization and data mixture fixed while varying the loss aggregation. Curves average the scores of two RL training seeds.
Figure 16: Loss aggregation and Extended generalization. The models in Fig. 15 are evaluated on 12-person puzzles, a size absent from RL training. The upper three rows show pass@1 and the lower three show pass@16, with the same arrangement of initializations and midtraining example counts. As in Fig. 15 , α=1\alpha=1 is evaluated for the 2.14B-token initialization. Curves average the scores of two RL training seeds through 12000 updates.
Figure 17: Loss aggregation with smaller pretraining mixtures in Base. The upper two rows show pass@1 and the lower two show pass@16 for the 133M- and 267M-token initializations, through 6000 RL updates. Columns vary the number of midtraining examples added to each RL update. Each panel compares the three objectives with the same initialization and example counts, averaging scores over two RL training seeds.
Figure 18: Loss aggregation with a smaller pretraining mixture in Extended. The top row shows pass@1 and the bottom row shows pass@16 for the 267M-token initialization through 12000 RL updates. Columns vary the number of midtraining examples per RL update, with the initialization and example counts held fixed within each comparison. Curves average scores over two RL training seeds.
Figure 19: Extended generalization with a smaller pretraining mixture. The models in Fig. 18 are evaluated on the 12-person test set through 12000 RL updates. The top row shows pass@1 and the bottom row shows pass@16 for the 267M-token initialization. Each curve averages the per-model score over two RL training seeds, without pooling generations from different models.
 Held-out loss ↓\downarrowBenchmark score (%)
MidtrainingClimbMixDolminoMeanMMLUARC-EHellaSwagTQAWinoGMean
None2.7841.7942.53728.6555.0935.3939.1553.5942.37
Unified: mixed during pretraining
133M tokens2.7841.7952.53728.6254.6735.5539.0953.2042.23
267M tokens2.7861.7962.53828.7954.9735.1739.1753.9942.42
534M tokens2.7851.7972.53828.2254.6735.6739.5751.1441.85
Appendix
Table 11: Pretraining retention across all initializations. Every row evaluates one initialization before RL; these results are shared by the Base and Extended experiments. Loss is in nats per token, with a mean weighted by 75% ClimbMix and 25% Dolmino. Benchmark scores are percentages, with an unweighted mean over the five tasks. ARC-E denotes ARC-Easy normalized accuracy, TQA denotes TruthfulQA MC2, and WinoG denotes WinoGrande accuracy; HellaSwag uses normalized accuracy and MMLU uses accuracy. Midtraining token counts include prompts and responses; staged configurations also indicate the number of passes over the corpus. The shaded row is the staged initialization with the best retention on both aggregate measures, used in Fig. 6 .
SettingPretrainingMidtraining
OptimizerAdamW
Adam (β1,β2)(\beta_{1},\beta_{2}) ; ε\varepsilon(0.9,0.95)(0.9,0.95) ; 10−810^{-8}
Weight decay; gradient clipping norm0.1; 1
Global batch size (sequences)2048
Sequence length (tokens)2048
Peak / minimum learning rate4×10−44\times 10^{-4} / 4×10−54\times 10^{-5}
Appendix
Table 12: Pretraining and midtraining hyperparameters for the 0.76B base model. The midtraining column describes training on the Dolmino corpus.
SettingUnifiedStaged RL
OptimizerAdamWAdamW
Adam (β1,β2)(\beta_{1},\beta_{2})(0.9,0.999)(0.9,0.999)(0.9,0.999)(0.9,0.999)
Adam ε\varepsilon10−810^{-8}10−810^{-8}
Weight decay00
Gradient clipping norm11
Learning rate10−610^{-6}10−610^{-6}
Appendix
Table 13: RL hyperparameters.
BenchmarkPass@1Pass@32
MATH-5006.5443.76
GSM8K17.8471.63
Math (average)12.1957.69
HumanEval+11.9732.32
MBPP+21.0557.14
Coding (average)16.5144.73
Appendix
Table 14: Base-model performance before mathematical RL. Scores are percentages, except held-out loss in nats/token. Pass@1 uses 16 responses per problem; pass@32 uses 256 responses for math and 32 for coding. Math and Coding are the equally weighted benchmark averages defined in Section D.1 .
Figure 20: Training curves for different midtraining data proportions. Each unified curve uses a constant configured proportion of 0, 0.1, 0.2, 0.5, 1, 2, or 5%; the dashed curves show staged training. Math and Coding average the two benchmarks in each domain, as defined in Section D.1 , which also gives the evaluation sample counts.
MixingMATH pass@1GSM pass@1MATH pass@64GSM pass@64MATH pass@256GSM pass@256
0%16.8332.6866.7383.8878.2091.81
0.1%16.9032.1465.7585.5677.0093.03
0.2%16.5632.3565.0384.7476.2092.49
0.5%16.8331.7165.6084.0677.2091.81
1%16.1131.6165.8583.7176.0091.74
2%16.6831.3067.5184.9078.4093.03
Appendix
Table 15: Mathematical accuracy and coverage at different midtraining proportions. The Base model row reports performance before RL; all other rows report final results after RL. MATH and GSM denote MATH-500 and GSM8K, and all scores are percentages. Pass@1 is estimated from 16 responses per problem; pass@64 and pass@256 use a separate evaluation with 256 responses per problem.
MixingΔ\Delta loss ↓\downarrowMMLUSciQARC-CMBPP+HumanEval+
0%+0.0737+0.073737.1250.2027.9025.4115.17
0.1%−0.0154-0.015439.3054.6029.6924.8214.44
0.2%−0.0516-0.051640.0454.3031.0623.7113.76
0.5%−0.0871-0.087139.9962.0032.3421.0511.97
1%−0.1076-0.107640.5466.1033.5319.3111.66
2%−0.1269-0.126940.6667.0034.2219.2311.55
Appendix
Table 16: Held-out loss and performance beyond math. The Base model row gives performance before RL, and the remaining rows give final results after mathematical RL. Scores are percentages; Δ\Delta loss is the change in held-out loss from the base model, in nats/token. The coding columns report pass@1 estimated from 16 responses per problem. ARC-C denotes ARC-Challenge.
Figure 21: Accuracy on individual benchmarks. Final pass@1 on MATH-500, GSM8K, HumanEval+, and MBPP+ is estimated from 16 responses per problem. The dashed reference shows staged training.
Figure 22: Coverage and knowledge accuracy across midtraining proportions. Final results are shown for all seven configured midtraining proportions. Mathematical pass@32 is estimated from 256 responses per problem, and coding pass@32 from 32 responses per problem. SciQ and ARC-Challenge use the greedy evaluation protocol in Section D.1 .
Figure 23: Full results for the four mixing schedules. These are the runs in Fig. 8 , with additional panels for Math and Coding accuracy and coverage, held-out loss change, MMLU, SciQ, and ARC-Challenge. The vertical line marks the midpoint of training, when the half-annealing schedule stops adding midtraining data.
ScheduleMath pass@1Math pass@32Coding pass@1Coding pass@32Δ\Delta loss ↓\downarrowMMLU
Constant 0.5%24.2768.3216.5146.38−0.0871-0.087139.99
Full anneal 1%24.7068.9417.7646.16−0.0833-0.083339.96
Ramp to 1%24.0168.9416.3346.55−0.0872-0.087240.17
Half anneal 2%24.8068.4918.8546.93−0.0634-0.063440.14
Appendix
Table 17: Results for the four mixing schedules. Each schedule has an average configured midtraining proportion of 0.5%. Scores are percentages, except Δ\Delta loss in nats/token. SciQ and ARC-Challenge appear in Fig. 23 .
Figure 24: Results for additional mixing schedules. Full, half, and three-quarter annealing reach zero at the end, midpoint, and three-quarter point of training, respectively. The held-out loss panel reports loss in nats/token, rather than its change from the base model.
Figure 25: Effect of removing or replacing code data. We compare the original corpus with variants that remove its designated code sources or replace them with verified coding examples. Each corpus is evaluated with constant 0.5% mixing and full annealing from 0.5%. Math and Coding are the benchmark averages defined in Section D.1 .
SettingMATHGSMΔ\Delta loss ↓\downarrowSciQMBPP+HumanEval+
constant: original16.8331.71−0.0871-0.087162.0021.0511.97
constant: code removed16.0532.15−0.0898-0.089861.9023.6613.64
constant: code replaced16.5431.22−0.0819-0.081964.8020.8012.12
anneal: original15.5032.55−0.0552-0.055259.0023.6914.41
anneal: code removed16.4032.07−0.0633-0.063356.8024.0114.33
anneal: code replaced17.0031.81−0.0522-0.052261.7023.3813.11
Appendix
Table 18: Results with different code data. Each corpus variant is evaluated with constant 0.5% mixing and with full decay from 0.5% to zero at the end of training. The original corpus is compared with variants that remove or replace its designated code sources, as described in Section D.4 . Mathematical and coding scores are pass@1 (%); SciQ reports accuracy (%). MATH denotes MATH-500, GSM denotes GSM8K, and loss change is measured in nats/token relative to the base model.
MethodMath pass@1Math pass@32Coding pass@1Coding pass@32Δ\Delta loss ↓\downarrowMMLUSciQARC-C
MaxRL23.9969.6115.3945.51−0.127-0.12740.6667.0034.22
REINFORCE24.8167.7015.8045.21−0.157-0.15740.6971.7036.43
Appendix
Table 19: MaxRL and REINFORCE with a constant midtraining proportion of 2%. Scores are percentages, except held-out loss change relative to the base model, measured in nats/token. ARC-C denotes ARC-Challenge. Training curves and comparisons across mixing settings appear in Figs. 26 and 27 .
Figure 26: MaxRL and REINFORCE with a constant midtraining proportion of 2%. Training curves for the runs in Table 19 , evaluated using the protocol in Section D.1 . Both algorithms use unified training; blue curves use MaxRL and orange curves use REINFORCE .
Figure 27: Final MaxRL – REINFORCE comparison across mixing settings. Each difference subtracts the MaxRL result from the REINFORCE result at the same mixing proportion and schedule. Positive values therefore indicate a higher value under REINFORCE ; for held-out loss, lower values are preferable. Differences are in percentage points for performance metrics and nats/token for loss.
Mixing settingMethodMath pass@1Math pass@32Coding pass@1Coding pass@32Δ\Delta loss ↓\downarrowMMLUSciQARC-C
Constant 0.5%MaxRL24.2768.3216.5146.38−0.087-0.08739.9962.0032.34
 REINFORCE22.9865.2914.4546.00−0.120-0.12041.1373.9038.82
Constant 1%MaxRL23.8668.2515.4944.89−0.108-0.10840.5466.1033.53
 REINFORCE24.5166.4115.6044.24−0.144-0.14440.8270.3035.58
Constant 2%MaxRL23.9969.6115.3945.51−0.127-0.12740.6667.0034.22
 REINFORCE24.8167.7015.8045.21−0.157-0.15740.6971.7036.43
Appendix
Table 20: MaxRL and REINFORCE across midtraining proportions and schedules. Each pair of rows compares the two algorithms with the same configured proportion and schedule, giving the absolute final scores underlying the differences in Fig. 27 . Half annealing decreases the stated initial proportion linearly to zero halfway through training. Scores are percentages, except held-out loss change relative to the base model, which is measured in nats/token. ARC-C denotes ARC-Challenge.
SettingValue
AlgorithmOn-policy MaxRL with reward-baseline subtraction
OptimizerFresh AdamW
Adam (β1,β2)(\beta_{1},\beta_{2}) ; ε\varepsilon(0.9,0.999)(0.9,0.999) ; 10−810^{-8}
Weight decay; gradient clipping norm0; 1
Learning rate / schedule10−610^{-6} / constant, no warmup
Prompts per update32
Appendix
Table 21: Hyperparameters for coding and WebInstruct RL. All models use these same RL settings when trained on coding tasks or on WebInstruct after mathematical training.
 HumanEval+MBPP+
Mathematical trainingPass@1Pass@16Pass@1Pass@16
Staged22.9445.1229.8152.12
Unified 0.5%23.6743.9030.3152.91
Unified 1%23.7045.1229.9454.76
Unified 2%24.0944.5131.7656.88
Unified 5%24.9646.9532.6958.47
Appendix
Table 22: Coding performance after subsequent RL. Scores are measured after 4000 coding RL updates and reported in percent. Percentages in the model labels identify the midtraining proportions during mathematical training. Both metrics use 16 responses per problem; bold indicates the highest score in each column.
Figure 28: Mathematical performance and retention during coding RL. The initializations and coding updates are the same as in Fig. 9 . Percentages denote constant midtraining proportions during the preceding mathematical training. Pass@1, pass@16, and benchmark accuracy are in percent; held-out loss is in nats/token. Mathematical accuracy remains close to its starting level, while changes in coverage vary across models. Held-out loss increases for every model.
Figure 29: WebInstruct RL after mathematical training. Percentages identify the midtraining proportions during mathematical training; all models then use the same WebInstruct RL procedure without midtraining data. The top row shows validation accuracy and overall test performance; the bottom row separates test performance by numerical and free-form answers. All performance metrics are in percent.
Figure 30: Midtraining validation loss during KL-regularized RL and unified training. Each curve shows the change from the common base-model value, measured on the trainer’s fixed set of 512 held-out midtraining windows. This loss averages over tokens in the validation windows; it is distinct from the equally weighted average over corpus components in Fig. 31 . Negative values indicate improvement over the base model. Curves show recorded evaluations without smoothing.
Figure 31: Benchmark performance during KL-regularized RL and unified training. Solid blue curves show unified training at five midtraining proportions; dashed orange curves show RL with three KL coefficients. Markers indicate recorded evaluations, joined without smoothing. Math and Coding average the two benchmarks in each domain, as defined in Section D.1 . Math pass@32 uses 256 responses per problem, and Coding pass@32 uses a separate 32-response evaluation. Held-out loss change gives equal weight to each of the 24 midtraining corpus components, with negative values indicating improvement over the base model. It is distinct from the token-averaged validation loss in Fig. 30 .
More in this topic

LLM Training

Large language model training studies the data, objectives, optimizers, and systems used to build and adapt language models.

See all 353 papers

Explore similar work

CardsList

  1. Sequential Pretraining Favors Large Models

    Oct 7, 2026Mohnish Harwani, Yujia ZhengLanguage Model PretrainingContinued Pretraining

    Purdue University · University of Illinois Urbana–Champaign

  2. Unsupervised Post-Training of Foundation Models: A Survey

    Aug 25, 2026Yijie Xu, Qianyi Cai, Huizai Yao +9LLM Post-TrainingSelf-Training

    HKUST(GZ) · Xiaohongshu Inc. · WeChat, Tencent +3

  3. Learning to Fine-tune Foundation Models under Resource Limitations

    Jul 12, 2026Thomas Tsouparopoulos, Iordanis KoutsopoulosFine-TuningLLM Fine-Tuning

    Department of Informatics Athens University of Economics and Business Athens, Greece