We present PRISM, a comprehensive empirical study of mid-training design choices for large language models. Through controlled experiments across seven base models spanning four families (Granite, LLaMA, Mistral, Nemotron-H), two architecture types (dense Transformer and attention-Mamba hybrid), and scales from 3B to 24B parameters, we show that mid-training on approximately 27B high-quality tokens yields consistent gains of +15 to +40 points on math, +5 to +12 points on code, and +6 to +13 points on science benchmarks while preserving general performance. The full PRISM to RL pipeline improves macro-average across six reasoning benchmarks from under 12 to 29-42 (a 3-4x improvement), whereas RL applied directly to most of the base models remains substantially less effective, with AIME scores near zero. Data composition matters most at mid-training, not RL: including science data during mid-training unlocks +17 to +28 point GPQA-Diamond gains during RL, while changing the RL mix produces less than 2 point differences. Mechanistically, mid-training densely restructures over 90% of model weights, while RL makes sparse, front-loaded refinements to approximately 5% of parameters. Representation analysis (CKA) confirms that RL consistently preserves mid-training's representational geometry (over 0.998 CKA) across architectures. Crucially, RL applies identical weight changes regardless of starting point, yet only succeeds on mid-trained models, consistent with mid-training placing the model in a configuration from which RL can effectively improve performance. Our results demonstrate that retention-aware mid-training is highly effective for reliable reasoning enhancement and provide practical guidance for designing robust mid-training pipelines.
Figures & tables
Figure 1: PRISM overview. Mid-training decisions are decomposed into their principal design axes, including retention of general and long-context abilities, domain interaction (math, code, science), benchmark selection, reinforcement learning compatibility, and scaling behavior. PRISM enables holistic evaluation of mid-training choices across model families at scale.
Dataset
Type
Tokens (B)
DCLM-EDU Allal et al. (2025)
General web data
111.46
Open-R1 (MoT) Lozhkov et al. (2025)
Math reasoning
0.60
Nemotron Post-Training v1 Nathawani et al. (2025)
Math
35.93
Megamath-Web-Pro Zhou et al. (2025)
Math web
14.73
Open-R1 (MoT) Penedo et al. (2025)
Code reasoning
1.18
OpenCodeReasoning-2 Ahmad et al. (2025)
Code reasoning
1.12
Table 1: Datasets used in mid-training mixtures. Token counts are reported in billions (Granite 3.3, 8B).
Figure 2: Mid-training data mixture configurations and per-source sampling percentages. The outer ring shows individual data sources; the inner ring groups them by domain category.
Detects generalization regressions hidden by domain-specific gains.
Long-context
RULER
Long-context retrieval
Ensures mid-training does not degrade long-context retrieval capabilities.
Code
LiveCodeBench Jain et al. (2024) , Codeforces Penedo et al. (2025)
Executable program synthesis and reasoning
Captures real-world coding ability.
Math
AIME AIME () , MATH500 Lightman et al. (2023)
Mathematical reasoning
Highly sensitive to data quality and mid-training composition.
Science
GPQA-Diamond Rein et al. (2023)
Expert-level scientific reasoning
Probes scientific reasoning capabilities
Table 2: Benchmark categories recommended for evaluating mid-training design choices.
Leaderbds.
Code
Sci.
Math
Stage
V1
V2
LCB
CF
GPQA
AI24
AI25
M500
Phase 3
63.30
19.44
7.05
8.61
19.53
9.38
16.09
65.88
Phase 4
62.84
20.85
7.89
7.95
17.85
10.00
14.06
61.70
After LC
62.91
20.53
10.39
6.18
25.93
23.59
20.94
77.44
Table 3: Effect of when mid-training is applied on Granite-4 Micro (3B). Phase 3/4 = intermediate/late pretraining; After LC = after long-context extension.
Leaderboard V1
Leaderboard V2
Mixture
ARC
HellaSwag
MMLU
TruthfulQA
Winogrande
GSM8K
OpenLLM V1 Avg
IFEval
BBH
MATH
GPQA
MUSR
MMLU-Pro
OpenLLM V2 Avg
Base
61.95
83.46
62.56
52.24
80.35
56.33
66.15
46.62
24.68
10.20
6.38
8.88
23.82
20.10
Math only
62.54
78.72
64.29
46.04
75.30
71.95
66.47
46.46
25.57
17.75
5.59
9.08
29.86
22.39
Math + Code
61.01
78.09
62.65
47.36
74.74
73.46
66.22
45.56
26.87
18.43
5.93
10.60
28.40
22.63
Math + Code + Science
61.69
78.12
62.98
46.96
74.90
74.22
66.48
46.44
26.32
20.02
7.27
8.60
29.55
23.03
Table 4: Leaderboard V1 and V2 results for Granite-3.3-8B mid-trained with the mixtures in Fig. 2 .
Mixture
Code
Math
GPQA
Base
2.07
8.95
22.56
Math
2.81
36.43
17.34
Math+Code
10.71
44.99
19.02
Math+Code+Sci
10.58
48.75
29.12
Table 5: Domain-specific results for Granite-3.3 (8B). Code/Math are averages; full results in Appendix Table 19 .
Leaderboards
Code
Science
Math
Model
Variant
LB V1
LB V2
LCB
CF
Code Avg
GPQA-D
AIME24
AIME25
MATH500
Math Avg
Granite-3.3 (8B)
Base
66.15
20.10
2.15
1.99
2.07
22.56
0.46
0.31
26.09
8.95
PRISM
66.48
23.03
10.63
10.52
10.58
29.12
37.18
27.96
81.11
48.75
Granite-4 Micro (3B)
Base
66.01
21.82
0.24
2.28
1.26
21.55
16.09
12.34
50.42
26.28
PRISM
62.91
20.53
10.87
6.25
8.56
34.34
27.19
22.29
79.40
42.96
Granite-4-H Micro (3B)
Base
64.49
18.99
0.60
0.88
0.74
20.88
7.08
2.70
30.17
13.32
Table 6: Base versus PRISM (Math+Code+Science) mid-training results across model families. Code Avg is the mean of LiveCodeBench (LCB) and Codeforces (CF). Math Avg is the mean of AIME24, AIME25, and MATH500. All values are reported to two decimal places.
RULER
Code / Science
Math
Model Variant
8k
16k
32k
64k
128k
LCB
CF
Code Avg
GPQA-D
AIME24
AIME25
MATH500
Math Avg
Granite-3.3 Base
85.81
82.40
75.53
64.91
59.09
2.15
1.99
2.07
22.56
0.46
0.31
26.09
8.95
Mid-Train (Math+Code)
89.02
60.44
21.52
11.71
6.46
11.11
10.30
10.71
19.02
32.44
28.33
74.22
44.99
Mid-Train + LC (Attention)
90.04
82.56
71.47
54.63
36.32
23.78
15.53
19.65
17.85
36.56
32.55
67.20
45.44
Mid-Train + LC (Full)
89.29
80.74
70.86
56.02
38.41
29.99
21.04
25.52
14.48
35.21
30.36
62.30
42.62
Merge (15% Base + 85% Mid-Train)
89.12
69.76
32.63
15.44
11.32
10.75
10.96
10.86
22.22
28.39
24.90
72.97
42.09
Table 7: Restoring long-context capability after mid-training for Granite-3.3 (8B). RULER is evaluated from 8k to 128k input lengths. Downstream performance includes Code (LiveCodeBench, Codeforces), Science (GPQA-Diamond), and Math (AIME24, AIME25, MATH500).
Figure 3: Long-context restoration pipeline. After PRISM mid-training degrades RULER@128k from 59.09 to 6.46, a linear merge (15% base + 85% mid-trained) followed by long-context extension recovers performance to 42.16 (full params) or 37.75 (attention-only).
Context
LB-V1
LB-V2
LCB
CF
GPQA
AIME24
AIME25
M500
Base
66.01
21.82
0.24
2.28
21.55
16.09
12.34
50.42
8k
62.91
20.53
10.87
6.25
34.34
27.19
22.29
79.40
16k
64.23
20.37
12.19
8.90
38.89
31.82
25.26
82.47
32k
64.48
21.05
14.93
7.50
39.89
30.98
21.87
82.70
Table 8: Mid-training context length ablation on Granite-4 Micro (3B) with Math+Code+Science mix. V1/V2 = Leaderboard V1/V2.
Tok. (B)
LB-V1
LB-V2
Code
GPQA
Math
Base
66.01
21.82
1.26
21.55
26.28
10.49
63.45
19.50
9.59
19.19
40.21
15.73
63.24
19.79
9.02
23.06
42.07
26.21
63.28
19.63
8.69
19.19
42.22
31.46
63.16
20.05
7.62
21.38
42.42
Table 9: Token budget ablation on Granite-4 Micro (3B), Math+Code mix. Full table in Appendix 20 .
Domain
Sources
Count
Math
DeepScaleR-Preview
294K
INTELLECT-2-RL
Skywork-OR1-RL-Data
Science
Nemotron-PT-v1-stem
100K
Code
DeepCoder-Preview
142K
Skywork-OR1-RL-Data
Table 10: RL datasets and prompt counts.
Figure 4: \textscPRISM→RL : Granite-3.3-8B. RL training curves on the PRISM -mid-trained checkpoint using the unbalanced MCS mix. All benchmarks show consistent, monotonic improvements.
Figure 5: \textscPRISM→RL : Mistral-Small 24B. The largest model tested shows the strongest GPQA-Diamond gains ( +27.95 ) and non-saturating code improvements.
Figure 6: \textscPRISM→RL : Nemotron-H 8B (Hybrid). RL yields stable gains on the hybrid attention-Mamba architecture, confirming that mid-training benefits extend beyond dense Transformers.
Figure 7: RL on Granite-3.3-8B base (no mid-training). AIME24/25 remain near zero throughout training, and overall gains are substantially smaller than the \textscPRISM→RL pipeline (Fig. 4 ).
Figure 8: RL on Nemotron-H 8B base (no mid-training). Even for hybrid architectures, RL on the base model shows limited progress on harder benchmarks compared to \textscPRISM→RL (Fig. 6 ).
Figure 9: \textscPRISM→RL with balanced mix: Granite-3.3-8B. Domain-equalized RL produces comparable math and code gains to the unbalanced mix (Fig. 4 ), with stable training throughout.
Figure 10: Pass rates on initially unsolved (code, score = 0) and hardest (math, score = 1) prompts during RL training of Granite-3.3 (8B) with the balanced mix.
Model
MT
RL
LCB
CF
Code Avg
AIME24
AIME25
MATH500
Math Avg
GPQA
AVG
LLaMA-3.1
–
–
0.00
0.07
0.04
0.05
0.15
6.51
2.24
20.20
7.49
MC
–
6.93
6.03
6.48
20.67
19.58
73.70
37.98
19.53
21.33
MCS
–
6.09
5.45
5.77
16.45
19.32
73.47
36.41
21.04
21.07
MC
MC
12.31
11.85
12.08
25.47
23.23
78.99
42.56
23.06
25.90
MC
MCS
11.83
12.80
12.32
24.43
23.12
78.62
42.06
24.75
26.38
MCS
MC
13.62
11.41
12.51
20.47
21.67
77.10
39.75
34.01
28.76
Table 11: Full Base→Mid-training→RL pipeline results across LLaMA-3.1-8B, Granite-3.3-8B, and Mistral-7B. MC = math + code mix; MCS = math + code + science mix. MT = mid-training mix; RL = RL mix. Highlighted rows show the best configuration per model.
Figure 11: Mid-training densely restructures the network; RL makes sparse, surgical refinements. Top row: layer-wise normalized L2 divergence for Granite-3.3 (8B, left) and Nemotron-H (8B, right). Mid-training (blue) changes weights 370–580 × more than RL (red, orange), broadly across all layers with some layer-wise variation. For Nemotron-H, the repeating pattern reflects its hybrid architecture where Mamba-2, self-attention and FFN are separate sequential layers with independent residual connections ( NVIDIA et al., 2025 ) . Bottom row: update sparsity by component type. Mid-training modifies > 90% of all parameters (attention, MLP, and Mamba alike), while RL leaves > 93% unchanged.
Model
MT
Transition
Attn
MLP
Mamba
Total
Sparsity
Granite-3.3 (8B)
MCS
Base → MT
0.175
0.329
–
0.175
9.3%
MT → RL
0.0003
0.0006
–
0.0003
95.9%
Base → RL (no MT)
0.0004
0.0007
–
0.0004
96.0%
MC
Base → MT
0.177
0.333
–
0.177
9.3%
MT → RL
0.0003
0.0006
–
0.0003
95.8%
Nemotron-H (8B, Hybrid)
MCS
Base → MT
0.230
0.289
0.138
0.112
2.7%
Table 12: Weight divergence summary across models and architectures. Normalized L2 = ∥wnew−wold∥2/∥wold∥2 . Nemotron-H reports all three component types (Attention, MLP, Mamba). Sparsity = fraction of parameters with < 1% relative change. The dense/sparse asymmetry is consistent across all component types and architectures.
Model
Mix
Attn
MLP
Mamba
Total
Granite-3.3 (8B)
MC
0.177
0.333
–
0.177
MCS
0.175
0.329
–
0.175
Nemotron-H (8B)
MC
0.232
0.292
0.140
0.113
MCS
0.230
0.289
0.138
0.112
Table 13: MC vs. MCS weight divergence (Base → MT normalized L2). Both models show nearly identical per-component L2 norms across data compositions, confirming that the training intensity is matched between MC and MCS despite their different downstream capabilities.
Figure 12: Data composition redirects weight updates across all sub-components. Cosine similarity between MC and MCS weight update vectors ( ΔW=WMT−Wbase ) for Granite-3.3 (left) and Nemotron-H (right). Overall cosine similarity of 0.52 and 0.62 respectively confirms that different data compositions steer weights in substantially different directions despite nearly identical magnitudes. The embedding/LM-head layers are most aligned (0.82–0.88), while attention, MLP, and Mamba layers all show low directional similarity (0.48–0.64).
Figure 13: Pass rate landscape on held-out MATH500 problems. (a) Math pass rate at linearly interpolated weight checkpoints along the Base → MT → RL path for Granite-3.3 and LLaMA-3.1, evaluated on 200 held-out MATH500 problems (7680 generation tokens). Pass rate increases monotonically from Base to MT (16.9% → 75.5% for G33, 2.6% → 43.1% for LLaMA) and continues increasing through RL. (b) 2D pass rate landscape for Granite-3.3 centered at MT, with axes toward RL ( α ) and toward Base ( β ). The RL direction consistently improves performance while moving toward Base degrades it.
Figure 14: RL preserves representational geometry; mid-training reshapes it in model-specific ways. Layer-wise linear CKA Kornblith et al. (2019) on Wikipedia (top) and GSM8K math prompts (bottom) for Granite-3.3 (left) and Nemotron-H (right), evaluated on 200 prompts per input type with batch-size-1 encoding. MT vs. RL (green) is ≈ 1.0 at every layer across both models and both input types, confirming RL preserves mid-training’s representational geometry. Base vs. MT and Base vs. RL (blue, pink) are nearly identical, confirming all representational change comes from mid-training. The magnitude and layer pattern of mid-training’s representational shift is model- and input-specific. See Table 14 for the full summary.
Model
Arch.
Wiki
C4
GSM8K
Granite-3.3 (8B)
Dense
0.9999 ± 0.0000
0.9999 ± 0.0000
0.9997 ± 0.0000
LLaMA-3.1 (8B)
Dense
0.9999 ± 0.0000
0.9999 ± 0.0000
0.9996 ± 0.0001
Nemotron-H (8B)
Hybrid
0.9999 ± 0.0000
0.9998 ± 0.0000
0.9993 ± 0.0001
Table 14: MT vs. RL representational similarity (minimum linear CKA ± bootstrap std) across input distributions. Values are the minimum layer-wise CKA across 20 bootstrap resamples of 100 from 200 inputs. RL consistently preserves mid-training’s representational geometry ( > 0.998) across all three models and all three input types, spanning both dense Transformers and hybrid attention-Mamba architectures.
Model
Stage
Pass
Med. Len
Neg-LP
Corr.
Incorr.
Granite-3.3 (8B)
Base
16.9%
120
0.382
–
0.383
MT
75.5%
2,254
0.138
0.128
0.153
RL
79.5%
1,700
0.141
0.135
0.160
LLaMA-3.1 (8B)
Base
2.6%
158
0.758
–
0.780
MT
43.1%
1,052
0.377
0.146
0.469
RL
64.6%
1,188
0.267
0.149
0.320
Table 15: Correctness, response length, and prediction confidence across pipeline stages on 200 held-out MATH500 problems (8 samples/prompt, 7680 max generation tokens, step-by-step reasoning prompt). Pass = mean pass rate across 8 samples per prompt (%). Med. Len = median response length (tokens). Neg-LP = mean negative log-probability. Corr./Incorr. = mean neg-LP for correct/incorrect responses; – indicates too few correct samples. The \textscPRISM→RL pipeline consistently achieves the highest pass rates across all three model families.
Figure 15: Mid-training transforms prediction behavior: models learn to reason longer with calibrated confidence. Evaluated on 200 held-out MATH500 problems. Top row: response length distributions shift from short outputs (Base, gray) to extended reasoning chains (MT, blue), with RL (red) adjusting length. Bottom row: mean negative log-probability at each stage.
Figure 16: RL optimization is front-loaded and starting-point-invariant. Top row: cumulative L2 divergence from the initial checkpoint over RL steps for Granite-3.3 (left) and Nemotron-H (right). Solid lines: MT → RL; dashed lines: Base → RL. Most weight change occurs in the first ∼ 200–400 steps, then plateaus. MT → RL and Base → RL produce nearly identical divergence profiles, confirming that RL’s weight footprint is independent of the starting point. Bottom row: sparsity evolution showing the fraction of parameters within 1% of their initial values. The active parameter set grows progressively from ∼ 1.5% at step 20 to ∼ 5–6% by step 960, with all component types following the same trajectory.
Table 32
Dataset
Type
Tokens (B)
DCLM-EDU
General Web Data
2.51
Nemotron Post-Training v1
Math (QA/Reasoning)
5.08
Megamath-Web-Pro
Math (web)
4.33
StarCoder2
Code (web)
37.52
xenArcAI-codex
Code (QA/Reasoning)
3.860
Table 18: Datasets used in the long-context extension phase. Token counts in billions (Granite 3.3, 8B).
Code
Science
Math
Model / Mixture
LiveCodeBench
Codeforces
Code Avg
GPQA-Diamond
AIME 24
AIME 25
MATH 500
Math Avg
Base
2.15
1.99
2.07
22.56
0.46
0.31
26.09
8.95
Math only
2.15
3.46
2.81
17.34
26.72
22.08
60.50
36.43
Math + Code
11.11
10.30
10.71
19.02
32.44
28.33
74.22
44.33
Math + Code + Science
10.63
10.52
10.58
29.12
37.18
27.96
81.11
48.75
Table 19: Domain-specific evaluation results for Granite-3.3-8B (full breakdown). Code Avg is the mean of LiveCodeBench and Codeforces; Math Avg is the mean of AIME 24, AIME 25, and MATH 500.
Token Budget (B)
LB V1
LB V2
LCB
CF
Code Avg
GPQA-D
AIME24
AIME25
MATH500
Math Avg
Base
66.01
21.82
0.24
2.28
1.26
21.55
16.09
12.34
50.42
26.28
10.49
63.45
19.50
10.51
8.68
9.59
19.19
23.95
19.17
77.52
40.21
15.73
63.24
19.79
10.75
7.28
9.02
23.06
26.14
21.30
78.76
42.07
26.21
63.28
19.63
9.80
7.58
8.69
19.19
28.49
20.10
78.08
42.22
31.46
63.16
20.05
8.24
6.99
7.62
21.38
28.02
22.08
77.15
42.42
Table 20: Effect of increasing mid-training token budget on Granite-4 Micro (3B) using the Math+Code mixture with fixed 8k context length. LCB denotes LiveCodeBench, CF denotes Codeforces, and GPQA-D denotes GPQA-Diamond. Code Avg averages LCB and CF, while Math Avg averages AIME24, AIME25, and MATH500.
Hyperparameter
Value
Sampling Parameters
Responses per Prompt
16
Temperature
1.0
Top- p
1.0
Training Dynamics
Total Batch Size
1024
Table 21: RL training hyperparameters and configuration.
Figure 17: RL training correctness curves for Granite-4 Micro Dense (3B) after PRISM mid-training. All four domains show stable, monotonically improving training dynamics over 2k steps.
Figure 18: \textscPRISM→RL : Mistral-7B. Consistent improvements across code and math benchmarks using the unbalanced MCS mix.
Figure 19: \textscPRISM→RL : LLaMA-3.1-8B. Stable, monotonic gains across all reasoning benchmarks.
Figure 20: \textscPRISM→RL : Granite-4 Micro Dense (3B). Consistent but smaller absolute gains compared to 8B models.
Figure 21: RL on LLaMA-3.1-8B base (no mid-training). AIME24/25 remain near zero; GPQA-Diamond regresses below the base model.
Figure 22: RL on Mistral-7B base (no mid-training). Limited and unstable gains; AIME24/25 remain near zero throughout.
Figure 23: \textscPRISM→RL with balanced mix: Mistral-Small 24B. GPQA-Diamond gain (+25.93) slightly lower than unbalanced mix.
Figure 24: \textscPRISM→RL with balanced mix: Granite-4 Micro Dense (3B). Code and GPQA-Diamond gains are notable (+4.30 LCB, +6.06 GPQA).
Figure 25: \textscPRISM→RL with balanced mix: Granite-4-H Micro (Hybrid, 3B). Largest balanced-mix gains among small models, with +8.09 on Codeforces and +11.95 on GPQA-Diamond.
Threshold (%)
Base → MT unchanged
MT → RL unchanged
0.1
2.1%
82.3%
0.5
2.4%
87.3%
1.0
3.5%
95.0%
2.0
5.3%
97.5%
5.0
11.1%
99.0%
10.0
20.1%
99.5%
Table 22: Per-element sparsity at different relative change thresholds (Granite-3.3, 8B). The dense/sparse asymmetry between mid-training and RL is robust across all tested thresholds. Bold row shows the 1% threshold used in the main text.
Granite-3.3 (8B)
Nemotron-H (8B)
Sub-component
Base → MT
MT → RL
Base → RL
Δ
Base → MT
MT → RL
Base → RL
Δ
Attn: Q
83.5%
4.4%
5.2%
+0.8
97.5%
5.5%
4.9%
− 0.6
Attn: K
83.2%
4.6%
5.2%
+0.6
96.9%
5.5%
4.8%
− 0.7
Attn: V
97.9%
5.7%
7.5%
+1.8
97.1%
8.5%
7.6%
− 0.9
Attn: O
98.0%
5.6%
6.7%
+1.1
97.2%
7.2%
6.4%
− 0.8
MLP: gate
98.3%
5.4%
6.1%
+0.7
–
–
–
–
Table 23: Sub-component weight analysis: % of parameters changed ( > 1% relative change) across pipeline transitions. Δ = difference between Base → RL and MT → RL. The near-zero Δ values confirm that RL targets the same sub-components in the same proportions regardless of starting point. Value and output projections are consistently the most affected by RL across both architectures, while Mamba parameters (A, dt) and norms remain frozen.
Figure 26: RL targets the same sub-components regardless of starting point. Fraction of parameters changed ( > 1% relative change) during RL for Granite-3.3 (left) and Nemotron-H (right), comparing MT → RL (blue) vs. Base → RL (pink). The near-identical distributions confirm that RL’s sub-component targeting is intrinsic to the optimization, not a consequence of mid-training.
Figure 27: CKA across input distributions: Granite-3.3 (8B). MT vs. RL (green) remains ≈ 1.0 on all three inputs. Base vs. MT divergence is input-dependent: strongest on GSM8K math prompts (min 0.55), weakest on C4 general text (min 0.94).
Figure 28: CKA across input distributions: LLaMA-3.1 (8B). Same pattern as Granite-3.3: MT vs. RL ≈ 1.0 everywhere, confirming the finding generalizes across dense Transformer families.
Figure 29: CKA across input distributions: Nemotron-H (8B, Hybrid). MT vs. RL > 0.998 on all three inputs, consistent with all other models. Base vs. MT divergence is most pronounced on GSM8K math prompts (min ≈ 0.41), reflecting mid-training’s targeted restructuring of reasoning-relevant representations.
Figure 30: AIME 2026 accuracy over RL training steps. Both Granite-3.3 (8B) and Mistral-Small (24B) show steady gains on this recently released benchmark, confirming generalization of the \textscPRISM→RL pipeline.
The effectiveness of Reinforcement Learning (RL) in Large Language Models (LLMs) depends on the nature and diversity of the data used before and during RL. In particular, reasoning problems can often be approached in multiple ways that rely on different forms of reasoning, and exposure to only a limited range of such approaches in the training data may limit the effectiveness of RL. Motivated by this, we investigate using diverse self-generated data during mid-training as an intermediate step before RL training. Specifically, we adopt a bootstrapped data-generation framework guided by George Polya's problem-solving approaches for generating multiple variants of correct answers for each question in the training data, and then perform fine-tuning. We first provide a theoretical perspective on how mid-training on such data improves RL and explain how policy-gradient updates can incentivize combining multiple approaches. We then empirically demonstrate that RL-trained models initialized with our mid-training data achieve consistent improvements across various mathematical reasoning benchmarks and other OOD tasks like code generation and narrative reasoning. Overall, our investigative study shows that a language model learning multiple problem-solving approaches, through self-generated data helps subsequent RL.
Aswin RRV, Jacob Dineen, Divij Handa +4
Arizona State University · Google Cloud AI Research · Google DeepMind
Sparse reward reinforcement learning (RL) has become a standard tool for improving LLM reasoning, but its success depends critically on the coverage present in the base model. In practice, models are often primed for RL through mid-training on curated reasoning traces that teach useful primitive skills such as decomposition, verification, or self-correction. Although effective, this strategy requires manually specifying what the model should learn, and it remains unclear whether such primitive coverage is enough for much harder problems, which require combining these skills into broader solution strategies. We study a more automated approach: RL-based mid-training using large corpora of human-written question-answer data. Rather than treating reference solutions as targets to imitate, our method, ExpRL, uses them as reward scaffolds: references are hidden from the policy and used only to construct problem-specific grading rubrics for judging on-policy reasoning traces. The policy samples from the original problem prompt, while an LLM judge compares the sampled reasoning trace against the reference solution and assigns outcome-level or process-level dense rewards. This lets ExpRL reinforce partial progress, useful intermediate reductions, and productive reasoning behaviors that sparse final-answer rewards often fail to upweight. On challenging math reasoning tasks, ExpRL yields stronger RL priming than SFT, sparse-reward GRPO, and self-distillation, and provides a better initialization for subsequent sparse-reward RL. Additional mixed-domain experiments further suggest that ExpRL can extend beyond the original math-only setting.
Violet Xiang, Amrith Setlur, Chase Blagden +2
1Stanford University · 2Carnegie Mellon University · 3OpenAI +1
Reinforcement learning (RL) has become central to improving large language models (LLMs) on complex reasoning tasks, yet RL post-training is largely studied in isolation from the pretraining that precedes it. As a result, two basic questions remain open: (1) how do pretraining choices (model size, data) shape the returns to RL compute, and (2) what does RL actually do to the model? These questions are difficult to study in the standard LLM setting: pretraining corpora are vast and uncontrolled, making it hard to attribute behaviors to pretraining versus RL, and systematic compute sweeps across both stages are prohibitively expensive. To address these challenges, we use chess as a controlled testbed for studying reasoning across the full pretraining-to-post-training pipeline. We follow the standard LLM training pipeline by pretraining language models from 5M to 1B parameters on human chess games, supervised fine-tuning on synthetic reasoning traces, and running RL on chess puzzles with verifiable rewards. Using this framework, we find that the post-RL performance at given RL compute level is well-predicted from the pretraining loss, and slope of the RL reward curves improves approximately linearly with the pretraining tokens. Beyond scaling, we find that RL does not simply sharpen the SFT policy: on easy puzzles it amplifies correct moves the SFT policy already preferred, while on hard puzzles it surfaces correct moves that were nearly absent under SFT. We further test whether our findings transfer beyond chess by training a 1B language model on math-domain text, where the same predictive pattern emerges: longer-pretrained checkpoints reach higher post-RL performance and improve faster under RL. In sum, we provide a quantitative account of the pretraining-to-RL interface and a controlled testbed for studying the science of reasoning across the full pretraining-to-post-training pipeline.
Jingyan Shen, Ang Li, Salman Rahman +4
New York University · Columbia University · University of California, Los Angeles +2