Modern LLM post-training composes supervised fine-tuning (SFT), reinforcement learning with verifiable rewards (RLVR), and on-policy distillation (OPD) into multi-stage pipelines, yet these stages are typically designed and evaluated in isolation. We show that this composition is consequential: a stage that improves the current model can make the next stage less effective. Through controlled experiments with Qwen3 models on math and science reasoning, we first characterize OPD across nine student-teacher pairs spanning 2x to 53x parameter ratios and show that OPD effectiveness depends on student-teacher compatibility rather than teacher scale alone. The surrounding stages of OPD reshape this compatibility in three ways: (1) A brief SFT warm-up improves subsequent OPD, while an RLVR-strengthened student regresses under distillation from the same teacher. (2) Adapting the teacher with RLVR raises downstream OPD accuracy in proportion to the capability it adds. Following these two interventions, we find that combining teacher adaptation and student warm-up alone raise average OPD accuracy from 29.2% to 43.8% (50% relative improvement) after the same number of distillation steps, with additional preparatory training. (3) At comparable accuracy, OPD leaves a stronger initialization for downstream RLVR than SFT, with a gap that widens as RL compute scales. Our results suggest that each post-training stage should be chosen not only for the capability it adds, but for the learning interface it creates for the next stage.
Figures & tables
Figure 1: The outcome of each post-training stage depends on the learning state it inherits. (a) An RLVR-warmed student ( green ) starts stronger but regresses under OPD, while a briefly SFT-warmed student ( gold ) improves steadily from the same teacher. (b) An RLVR-adapted teacher transfers substantially more than a SFT-adapted teacher of the same origin. Inset: teacher accuracy along its RLVR training ( gray ) and final OPD accuracy of a student distilled from the teacher at each step ( green ); both plateau together, so student gains stop when the teacher’s do. (c) At comparable starting accuracy, OPD-initialized models ( blue ) reach higher final accuracy under subsequent RLVR than SFT-initialized models, and the gap widens with RL compute.
Figure 2: A larger teacher does not guarantee better transfer. Red lines show the unrecovered gap, 100−DE (%). (a) Qwen3-4B-Base gains 10.3 points from Qwen3-14B but leaves 55.5% of the teacher’s advantage unrecovered. (b) For the 4B student, DE decreases as teacher size increases from 8B to 32B. (c) With the 14B teacher fixed, smaller students recover a smaller share of the gap.
Figure 3: OPD effectiveness varies across objectives, model states, and training data. (a) Under forward KL, DE decreases with teacher size at every top- k support. (b) Qwen2.5-1.5B base students gain more from OPD than instruct students. (c) Post-trained teachers improve peak accuracy in four of five matched-size comparisons. (d) Qwen3-8B gives the highest accuracy on both training datasets for Qwen3-4B-Base.
Figure 4: Reinforceable signal varies across student–teacher pairings. (a) Initial ρ0+ decreases with teacher size for every student, with point labels showing step-200 accuracy. (b) Selecting maximum ρ0+ identifies the largest step-200 gain for two of three students. (c) Larger teachers leave a greater share of tokens without positive distillation signal.
Figure 5: Student warm-up: the method and the amount both matter. (a) Brief SFT raises the observed OPD performance and reduces the subsequent drop, whereas the RLVR-prepared student regresses under OPD from the same teacher. (b) Both students contract toward the teacher in reverse KL—the RLVR-warmed student from much farther away—so the regression occurs despite successful teacher matching. (c) As SFT continues, student entropy keeps falling while downstream OPD accuracy saturates after a short warm-up. (d) The SFT validation-loss plateau signals when to stop: SFT beyond it no longer improves downstream OPD.
Figure 6: Teacher adaptation transfers capability, not proximity. (a) OPD from an RLVR-adapted teacher surpasses the off-the-shelf teacher’s own accuracy, whereas a SFT-trained teacher improves slightly over vanilla OPD. (b) The student’s gain tracks the teacher’s gain across RLVR checkpoints and flattens once the teacher plateaus (shaded). (c) Some runs regress while reverse KL continues to decrease, so teacher matching alone does not explain accuracy retention.
Figure 7: An OPD checkpoint is a better RLVR initialization than an SFT checkpoint trained on the same trajectories. (a) RLVR from OPD checkpoints (blue) is still rising at step 260 , while RLVR from SFT checkpoints (gold) stalls by step 100 . Shading marks the range across checkpoints. (b) OPD-initialized runs spend entropy as accuracy rises; SFT-initialized runs keep theirs largely unchanged. (c) Among the tested OPD handoffs, the step- 50 checkpoint has both the highest pre-RLVR accuracy and the highest post-RLVR accuracy, reaching 43.1% .
Figure 8: Final OPD accuracy combined with teacher RLVR adaptation and SFT warm-up.
Appendix figures & tables33 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Training data
DAPO-Math-17k
Parameter update
Full-parameter training, LoRA rank 0
Optimizer
AdamW
Learning rate / (β1,β2)
10−6 / (0.9,0.999)
Schedule / warm-up
Constant learning rate / 10 steps
Weight decay / gradient norm clipping
0.01 / 1.0
Appendix
Table 1: Shared OPD configuration for the main experiment grid in § 3 .
Setting
Value
Student / teachers
Qwen3-4B-Base / Qwen3-{8B, 14B, 32B}
Training data
DAPO-Math-17k
Loss mode / top- k
forward_kl_topk / k∈{16,32,64}
Update
Direct distillation loss with policy-gradient mode disabled
Optimizer / learning rate
AdamW / 10−6
Schedule / warm-up
Constant learning rate / 10 steps
Appendix
Table 2: Forward-KL configuration with optimization, sampling, and length settings match Table 1 . The objective uses teacher top- k support.
Table 3: Evaluation protocol where the primary aggregate assigns equal weight to each of the seven benchmarks.
Dataset
Unique prompts
Average prompt tokens
Training
DAPO-Math-17k
17,917
122
OpenThoughts3-Math
30,000
113
Evaluation
AIME 2024
30
150
AIME 2025
30
145
Appendix
Table 4: Training and evaluation datasets, prompt lengths use the Qwen3 tokenizer with the training chat template.
Student
Teacher
A200
A260
Gain 260
DE 260 (%)
0.6B
8B
14.29
14.91
+12.65
33.4
0.6B
14B
15.00
13.24
+10.99
27.6
0.6B
32B
13.66
13.24
+10.99
27.1
1.7B
8B
23.00
22.26
+11.47
39.1
1.7B
14B
20.95
20.13
+9.34
29.8
1.7B
32B
20.34
21.36
+10.58
33.0
Appendix
Table 5: Complete Qwen3 base-student grid where accuracy is the seven-benchmark mean in percent. Gain and DE use the pooled initial score for each student.
Figure 9: Complete OPD learning curves across student and teacher sizes. Each panel compares three teachers for one base student. The 0.6B and 1.7B curves show individual runs. For 4B, bold lines show the three-run mean and faint lines show individual runs. The dotted line marks step 200. All curves extend to the shared final evaluation at step 260.
k
Teacher
A (%)
Gain
DE (%)
Peak (%)
16
8B
33.66
+14.77
69.7
34.43
16
14B
34.48
+15.59
67.2
34.48
16
32B
31.79
+12.91
54.0
33.48
32
8B
34.22
+15.33
72.3
35.11
32
14B
34.39
+15.50
66.8
35.28
32
32B
32.64
+13.75
57.5
33.12
Appendix
Table 6: Forward-KL results. All cells use Qwen3-4B-Base and one run and peak accuracy is restricted to the same budget. Baseline and teacher references match Table 5 .
Student
State
Initial
Qwen2.5-7B/Qwen3-8B teacher
14B teacher
32B teacher
Qwen3-1.7B
Base
10.79
23.00 (+12.22)
20.95 (+10.16)
20.34 (+9.55)
Qwen3-1.7B
Post-trained
29.51
29.82 (+0.31)
31.17 (+1.65)
29.18 (-0.33)
Qwen3-4B
Base
18.89
32.70 (+13.81)
29.23 (+10.34)
29.25 (+10.36)
Qwen3-4B
Post-trained
39.58
38.03 (-1.55)
39.28 (-0.31)
37.69 (-1.89)
Qwen2.5-1.5B
Base
1.30
19.85 (+18.54)
20.01 (+18.71)
20.44 (+19.14)
Qwen2.5-1.5B
Instruct
18.08
20.69 (+2.61)
20.56 (+2.48)
20.73 (+2.66)
Appendix
Table 7: Student-state and model-family comparisons after 200 OPD steps. Entries report accuracy (%) with gains over the pooled initial student score in parentheses. The smallest teacher is 8B for Qwen3 and 7B for Qwen2.5. Results average three runs per teacher for Qwen3-4B-Base and use one run per configuration otherwise.
Student
Teacher size
Base A
Post A
Base peak
Post peak
1.7B
8B
19.66
23.00
21.50
23.00
1.7B
14B
22.30
20.95
23.54
23.14
4B
8B
25.05
32.70
31.52
33.38
4B
14B
30.81
29.23
32.35
33.49
8B
14B
31.19
29.34
33.32
35.10
Appendix
Table 8: Comparison of base and post-trained Qwen3 teachers at matched model sizes, using base students throughout. Endpoint accuracy is measured at step 200, and peaks are computed over evaluations through that step. Results with post-trained teachers average three runs for the 4B student, while all other configurations use one run.
DAPO-Math
OpenThoughts-Math
Teacher
Step 200
Step 200
Step 240
8B
32.70±0.66
31.29
31.10
14B
29.23±0.63
28.80
29.18
32B
29.25±0.34
28.00
30.35
Appendix
Table 9: Qwen3-4B-Base accuracy (%) by distillation dataset. DAPO-Math reports mean ± s.d. over three runs. OpenThoughts-Math is run per teacher.
Benchmark / aggregate
8B teacher
14B teacher
32B teacher
AIME 2024
13.89±1.73
13.61±2.68
13.33±0.83
AIME 2025
12.78±0.48
12.50±0.83
11.11±1.27
AMC 2023
42.97±1.06
39.86±0.92
36.45±1.59
MATH-500
72.23±0.70
61.57±0.70
64.13±1.25
Minerva Math
21.57±0.70
18.29±0.09
17.68±0.41
OlympiadBench
35.89±0.94
30.83±0.97
31.00±0.13
Appendix
Table 10: Qwen3-4B-Base accuracy (%) at step 260, reported as mean ± sample standard deviation over three rollout seeds. Aggregates average the first six, first seven, or all ten benchmarks with equal weight.
Student
0.6B
1.7B
4B
Teacher
8B
14B
32B
8B
14B
32B
8B
14B
32B
Initial rKL
0.740
0.680
0.648
0.447
0.478
0.426
0.370
0.376
0.382
ρ0+ (%)
37.32
33.20
29.39
53.34
49.12
34.37
54.79
47.83
33.11
Appendix
Table 11: Initial reverse KL and ρ0+ for each student–teacher pair.
Step 200
Step 260
Selector
0.6B
1.7B
4B
0.6B
1.7B
4B
Largest teacher
1.35
2.67
3.44
1.67
0.89
2.92
Smallest teacher
0.71
0.00
0.00
0.00
0.00
0.00
Largest ρ0+
0.71
0.00
0.00
0.00
0.00
0.00
Smallest initial rKL
1.35
2.67
0.00
1.67
0.89
0.00
Appendix
Table 12: Teacher-selection regret in accuracy points relative to the best observed teacher at each horizon. Lower is better. 4B comparisons use mean outcomes and diagnostics across three runs.
Figure 10: Task accuracy and teacher matching follow different trajectories. Colored curves show mean benchmark accuracy for Qwen3-4B-Base, with sample-deviation bands across three runs per teacher. Dashed gray curves show mean rKL normalized by each run’s first logged value, using the right axis. Dotted vertical lines mark the peak of each mean accuracy curve through step 200.
Stopping rule
Mean step
Accuracy (%)
Gap (pp)
Updates saved (%)
Fixed step 40
40.0
33.13
0.10
84.6
Fixed step 60
60.0
31.41
1.82
76.9
Fixed step 100
100.0
27.92
5.31
61.5
Fixed step 200
200.0
30.39
2.84
23.1
Fixed step 260
260.0
29.75
3.48
0.0
τ=0.4
66.7
30.13
3.11
74.4
Appendix
Table 13: Stopping comparisons averaged over nine Qwen3-4B-Base runs. Gap is measured from each run’s observed peak through step 260. Update savings use step 260 as the reference.
Figure 11: OPD training dynamics across student and teacher scales.
Setting
Student SFT Warm-Up
Starting student
Qwen3-4B-Base
Solution generator
Qwen3-14B
Prompt source
OpenThoughts-Math
Supervision
Fixed teacher-generated solutions
Update scope
Full-parameter SFT
Training backend / GPUs
FSDP / 8
Appendix
Table 14: Student SFT warm-up hyperparameters.
Setting
Student RLVR Warm-Up
Starting student
Qwen3-4B-Base (Thinking Disabled)
Algorithm
GRPO
Warm-up data
OpenThoughts-Math
Reward
Verifiable outcome (binary)
Update scope
Full-parameter training
Prompt / PPO minibatch size
64 / 64
Appendix
Table 15: Student RLVR warm-up hyperparameters.
Setting
Downstream OPD
Student initialization
Selected SFT checkpoint
Teacher
Qwen3-14B
Training data
DAPO-Math-17K
Objective
Sampled-token reverse KL policy gradient
Task rewards / auxiliary KL loss
Disabled / disabled
Optimizer
AdamW
Appendix
Table 16: Downstream OPD hyperparameters for the SFT warm-up sweep.
Figure 12: Downstream OPD trajectories from different SFT warm-up checkpoints.
SFT Checkpoint K
Initial (%)
Peak (%)
Peak t
Final (%)
Drop
25
30.68
30.87
40
29.15
1.71
75
31.25
35.06
140
34.65
0.41
150
33.42
35.39
160
35.30
0.09
250
33.81
35.08
180
35.08
0.00
348
33.85
35.18
60
34.48
0.70
Appendix
Table 17: Downstream OPD results across SFT warm-up checkpoints.
SFT Checkpoint K
OPD Step t
Student Entropy
Reverse KL
25
1
0.2940
0.1417
75
1
0.2559
0.1227
150
1
0.1950
0.1028
250
1
0.1507
0.1153
348
1
0.1580
0.1117
Appendix
Table 18: Student diagnostics at the beginning of downstream OPD.
Setting
SFT Teacher
Starting teacher
Qwen3-14B (Thinking Disabled)
Adaptation data
OpenThoughts3-Math
SFT format
Full-parameter
Supervision
Fixed, verified self -generated solutions
Unique prompts in batch
64
Learning rate
10−6
Appendix
Table 19: SFT teacher training hyperparameters.
Setting
RLVR-Adapted Teacher
Starting teacher
Qwen3-14B
Algorithm
GRPO
Adaptation data
OpenThoughts-Math
Reward
Verifiable outcome (binary)
Unique prompts in batch
64
Responses per prompt / temperature
8 / 1.0
Appendix
Table 20: RLVR teacher training hyperparameters.
Setting
SFT/RLVR Teacher
Student
Qwen3-4B-Base
Training data
DAPO-Math-17K
Objective
Sampled-token reverse KL policy gradient
Task rewards / auxiliary KL loss
Disabled / disabled
Optimizer
AdamW
Learning rate / warm-up
10−6 / 10 steps
Appendix
Table 21: Downstream OPD training hyperparameters shared across SFT- and RLVR-adapted teachers.
Teacher K
rKL 0
ρ0+ (%)
A200 (%)
50
0.505
47.23
27.41
100
0.476
52.40
28.00
150
0.555
49.08
29.16
200
0.530
52.97
29.77
Appendix
Table 22: Initial diagnostics and step-200 accuracy for Qwen3-1.7B-Base with RLVR-adapted Qwen3-14B teachers. Each configuration uses one run.
Figure 13: Downstream OPD performances across SFT and RLVR adapted teacher checkpoint sweeps.
Setting
Qwen3 family
Qwen2.5 family
Supported base students
0.6B, 1.7B, 4B
1.5B
Teacher
Qwen3-14B
Qwen2.5-14B-Instruct
Training data
OpenThoughts3-Math
Supervision
Fixed, verified teacher-generated solutions
Unique prompts in batch
64
64
Microbatch per GPU
4
4
Appendix
Table 23: SFT initialization settings. The reported comparison on Qwen3-1.7B-Base, Qwen3-4B-Base, and Qwen2.5-1.5B students.
Setting
Qwen3 family
Qwen2.5 family
Supported base students
0.6B, 1.7B, 4B
1.5B, 3B
Teacher
Qwen3-14B
Qwen2.5-14B-Instruct
Training data
OpenThoughts3-Math
Distillation objective
Sampled-token policy gradient reverse-KL
Task rewards
Disabled
Disabled
Prompt / PPO minibatch size
64 / 64
64 / 64
Appendix
Table 24: OPD initialization settings. All students use the same optimization and sampling settings, the teacher and initialization data follow the model family.
Table 25: Common Downstream RLVR settings for SFT and OPD-initialized students. The reported horizons reflect the displayed curves, while the launchers specify an epoch-based training schedule.
Figure 14: The OPD initialization advantage reproduces across matched handoff checkpoints, student scales, and model families. Held-out core-7 mean@4 accuracy for Qwen3-1.7B-Base (left), Qwen3-4B-Base (center), and Qwen2.5-1.5B (right), using their corresponding 14B teachers. Within each student family, OPD and SFT use the same downstream RLVR recipe and matched handoff-checkpoint grid. Blue and orange curves denote OPD and SFT initialization. Shading shows the minimum–maximum range across handoff checkpoints within each arm, not variation across random seeds. The reported RLVR horizons are 260 steps for Qwen3 and 200 for Qwen2.5-1.5B.
Figure 15: Recursive composition under different teacher schedules. Four SFT–OPD rounds on Qwen3-1.7B-Base, each using an RLVR-adapted Qwen3-14B checkpoint. Accuracy averages AIME 2024, AIME 2025, AMC 2023, and Minerva Math.
Schedule
Round
rKL
ρ+
Acc. (%)
Fixed
1
.42
.48
19.5
Fixed
2
.39
.46
19.0
Fixed
3
.38
.45
18.5
Fixed
4
.38
.44
16.6
Reverse
1
.43
.47
19.8
Progressive
4
.41
.44
21.4
Appendix
Table 26: Diagnostics recorded at the start of OPD and four-benchmark accuracy for every round using teacher t200 . Reinforceable fractions are shown on a 0 – 1 scale.
Reinforcement learning with verifiable rewards (RLVR) and on-policy distillation (OPD) have emerged as two dominant methods for post-training reasoning LLMs. Prior work uses OPD's dense token-level supervision to complement the sparse RL reward, fusing the two signals within a single step: either as a weighted-additive combination or a teacher-modulated rescaling of the RL advantage. In this paper, we show that a simple two-stage scheme, OPD-then-RL, consistently outperforms pure OPD, pure RLVR, and all such joint baselines across logic and math reasoning benchmarks. Beyond the empirical results, we further provide a systematic understanding of this through pass@k behavior, learning dynamics, and parameter updates, yielding a consistent explanation: OPD expands the student's coverage of teacher-supported solutions and RL sharpens within that support, while jointly optimizing the two signals causes them to interfere. To provide a practical recipe, we find that the OPD validation score is the key signal for when to switch to RL, and that OPD is a better cold start for RL than SFT. Together, our results establish OPD-then-RL as a simple yet strong way to combine the two methods, turning two entangled signals into complementary stages.
Boyan Li, Bingsen Chen, Chenghao Yang +3
University of Alberta · New York University · NYU Shanghai +3
On-policy distillation (OPD) is an effective post-training paradigm for large language models but requires a live teacher server throughout training, resulting in substantial infrastructure overhead. We investigate whether OPD can be performed offline by precomputing teacher log-probabilities once over SFT rollouts and reusing them during training. We find that naively doing so fails to reliably match standard OPD, and trace the root cause to a previously overlooked condition we term teacher consistency, requiring that the same teacher be used for both supervised fine-tuning and OPD. Violating this condition introduces a gradient bias that degrades performance for both offline and online OPD. Building on this insight, we propose Lightning OPD, an offline on-policy distillation framework that enforces teacher consistency and eliminates the need for a live teacher server entirely. We prove that, under teacher consistency, Lightning OPD shares the same optimum as standard OPD, with bounded gradient discrepancy and an implicit regularization effect that helps prevent policy drift. Experiments on math reasoning and code generation show that Lightning OPD achieves comparable performance to standard OPD while delivering 4.0x higher training efficiency. Starting from an SFT-initialized Qwen3-8B-Base model, Lightning OPD reaches 69.9% on AIME 2024 in just 30 GPU hours. Lightning OPD further scales to MoE architectures, training Qwen3-30B-A3B to 71.0% on AIME 2024 on a single 8xH100 node, substantially lowering the barrier for academic research on LLM post-training. Our code is released at https://github.com/jet-ai-projects/Lightning-OPD.
On-policy distillation (OPD) improves the reasoning capabilities of small language models through token-level teacher supervision on student-generated trajectories. Yet can teachers that excel at solving problems independently also guide student reasoning effectively? Prior work shows that when student prefixes follow reasoning paths that differ from the teacher's own or contain errors, teachers can be less accurate when continuing from these prefixes than when solving problems independently. To this end, we propose Prep-OPD, which uses reinforcement learning (RL) before distillation to train the teacher to adapt to the student's existing reasoning state and correct course when errors arise. Training optimizes teacher continuations from fixed student prefixes using final-answer correctness as the reward. The prepared teacher then trains the student through trajectory guidance and token-level supervision. We evaluate Prep-OPD on eight mathematical reasoning benchmarks, using Qwen3-4B-Instruct-2507 as the teacher and Qwen3-0.6B and Qwen3-1.7B as students. With the 4B teacher and 1.7B student, Prep-OPD improves average accuracy over standard OPD and the strongest baseline, Relay-OPD, by 8.28 and 2.30 percentage points, respectively. Controlled experiments further show that teacher RL conditioned on student-generated prefixes yields higher student accuracy than problem-start teacher RL with and without handoff on Qwen3-1.7B. Reusing the same prepared teacher also improves Qwen3-0.6B.
Xiaoyu Ma, Haoyue Liu, Zhichao Wang +2
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen · Shenzhen Future Network of Intelligence Institute (FNiI-Shenzhen) · Guangdong Provincial Key Laboratory of Future Networks of Intelligence, CUHK(SZ)