Policy-shift distillation reuses the difference between a teacher's log probabilities before and after reinforcement learning (RL) as supervision on student-generated trajectories. However, this endpoint policy shift cannot reveal whether local action preferences evolved consistently during teacher training. The same endpoint shift can result from early and late shifts that reinforce or partially cancel each other. We investigate whether agreement between early and late teacher policy shifts can guide selective transfer. We introduce Waypoint-guided On-Policy Distillation (Waypoint-OPD), which uses an intermediate teacher checkpoint from the same RL run to compare early and late shifts in relative action preferences at student-visited prefixes. We gate the endpoint shift according to the directional agreement between the early and late shifts, suppressing it when they point in opposing directions. Compared with endpoint-only transfer, our method requires only one additional frozen-teacher scoring pass. Using RL-trained Qwen3-4B teachers, we evaluate our method with Qwen3-8B, Qwen3-14B, and Qwen3-30B-A3B as student models in mathematical and code reasoning. In all six domain--student settings, Waypoint-OPD achieves the highest domain-average score among the compared distillation methods. Ablations with the Qwen3-8B student show that removing or shuffling gates reduces mathematical reasoning performance. These results support the use of directional agreement between early and late teacher policy shifts to weight the endpoint shift during student training.
Figures & tables
Figure 1: Mathematical reasoning across three student models. mean@32 accuracy (%), averaged over seven benchmarks. Waypoint-OPD improves over Direct-OPD by 2.18, 1.59, and 1.89 percentage points, respectively. Teacher scores are from the Final Qwen3-4B checkpoint.
Figure 2: Training pipeline of Waypoint-OPD. Initial, Waypoint, and Final score candidates at student-generated prefixes. Their centered segment shifts define an agreement gate on the endpoint shift. Batch scaling and detached candidate advantages drive the student update Waypoint measures consistency only.
Method
AIME24
AIME25
AIME26
HMMT25 Feb
HMMT25 Nov
HMMT26 Feb
AMC23
Avg.
Teacher: Qwen3-4B
Initial
21.67
21.67
21.67
12.50
10.00
15.15
71.25
24.84
Waypoint
58.96
54.79
56.77
31.35
35.73
31.72
93.44
51.82
Final
65.83
60.00
60.83
35.83
40.83
35.13
94.45
56.13
Student: Qwen3-8B
Base
29.17
20.83
18.33
11.67
13.33
18.94
70.00
26.04
Table 1: Mathematical reasoning across three student sizes. mean@32 scores (%) on seven benchmarks and their average. Bold marks the best score per student. Teacher checkpoints are references.
Teacher reference: Qwen3-4B
Initial: 37.64 Waypoint: 52.22 Final: 55.37
Method
Qwen3-8B
Qwen3-14B
Qwen3-30B-A3B
Base
41.03
47.72
50.33
OPD
43.91
51.66
51.91
OPD 2
51.21
55.60
55.96
Direct-OPD
50.59
56.30
56.35
Table 2: Average code reasoning performance. Average mean@8 scores (%) over five evaluations. Qwen3-4B teacher checkpoints are shared references. Bold marks the best score per student. Full results: Appendix D .
Figure 3: Evaluation accuracy during training. Qwen3-8B results on AIME 2025, AIME 2026, and HMMT November 2025 at steps 0, 20, 40, 60, 80, and 100. Scores are mean@32 percentages. Solid red curves denote Waypoint-OPD and dashed blue curves denote Direct-OPD. Gray dashed lines show the Final Qwen3-4B teacher scores.
Method
AIME24
AIME25
AIME26
HMMT25 Feb
HMMT25 Nov
HMMT26 Feb
AMC23
Avg.
No gate
66.25
54.27
56.67
32.60
39.79
34.66
90.23
53.50
Shuffled gate
63.33
52.92
54.58
30.00
37.08
31.06
89.45
51.20
No normalization
68.13
56.46
58.33
34.79
41.56
35.23
92.03
55.22
Direct-OPD
67.29
55.31
57.40
31.77
40.83
33.71
91.41
53.96
Waypoint-OPD
68.85
57.92
60.21
34.38
42.60
36.36
92.66
56.14
Table 3: Gate and normalization ablations on Qwen3-8B mathematical reasoning. Bold marks the best score in each column.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Train batch size
128
Micro batch size
128
Rollout n
8
Maximum prompt length
2,048
Maximum response length
16,384
Temperature
1.0
Appendix
Table 4: GRPO training configuration shared by the math and code teachers.
Setting
Value
Math training dataset
DAPO-Math-17K
Code training dataset
Codeforces-derived
Train batch size
256
Mini batch size
64
Rollout n
1
Maximum prompt length
2,048
Appendix
Table 5: Transfer configuration shared by math and code, except for the transfer datasets.
Item
Value
Initial teacher
Qwen3-4B
Waypoint teacher
RL step 600
Final teacher
RL step 1200
Students
Qwen3-8B, Qwen3-14B, Qwen3-30B-A3B
Math evaluation
AIME 2024, AIME 2025, AIME 2026, HMMT February 2025,HMMT November 2025, HMMT February 2026, AMC 2023
Table 7: Full code reasoning results across three student sizes. Scores are mean@8 percentages. Avg. is the unweighted mean across the five evaluations. Bold marks the best score within each student block. Teacher checkpoints are reference rows.
Waypoint
AIME24
AIME25
AIME26
HMMT25 Feb
HMMT25 Nov
HMMT26 Feb
AMC23
Avg.
Step 500
69.06
58.13
59.79
33.96
42.60
36.74
92.97
56.18
Step 600
68.85
57.92
60.21
34.38
42.60
36.36
92.66
56.14
Step 700
69.27
57.60
59.79
34.58
42.81
36.36
92.50
56.13
Appendix
Table 8: Sensitivity to the Waypoint checkpoint on Qwen3-8B mathematical reasoning. Only the intermediate teacher checkpoint is varied, while the Initial and Final teachers are fixed. Step numbers refer to the teacher’s RL training. Bold marks the best score in each column.
Figure 4: Gate diagnostics on Qwen3-8B mathematical reasoning. Statistics for Waypoint-OPD over 100 training steps. From left to right: mean gate gt , zero-gate fraction, batch-scale normalization multiplier κB , and mean per-prefix reward-vector L2 norm. Raw uses the gated, student-weighted endpoint signal qt(a)gtdt(a) ; Normalized uses the final candidate coefficients At(a) in Eq. 12 from the same run. All statistics aggregate valid response prefixes.
Method
Time (h:m:s)
Seconds/step
GPU-hours
Direct-OPD
8:21:03
300.63
66.81
Waypoint-OPD
8:02:21
289.41
64.31
Appendix
Table 9: Training-time comparison on Qwen3-8B math. Total training time over 100 training steps on eight H20 GPUs, excluding evaluation.
Figure 5: Response length and response truncation rate. The two panels compare Qwen3-8B trained with Waypoint-OPD (solid red) and Direct-OPD (dashed blue) over 100 training steps. (a) Mean number of tokens per rollout response at each training step. (b) Fraction of rollout responses reaching the maximum length of 16,384 tokens; 0.2 corresponds to 20%.
Reinforcement learning can substantially improve a reasoning teacher, but it is unclear which of those improvements survive when the teacher supervises a smaller on-policy student. We study this question in mathematical reasoning by comparing teacher lineages before and after GRPO, multiple student scales, direct GRPO, and several on-policy distillation objectives. The central finding is that transfer is structured rather than scalar: teacher strength alone does not make dense distillation competitive, while an RL-improved teacher creates useful but metric-dependent student gains. This motivates Transfer-Stratified On-Policy Distillation (TS-OPD), which screens training problems by the joint sampled success of the student and teacher, routes acquisition problems to gated forward KL, routes consolidation problems to gated reverse KL, and adds an entropy brake to protect sampled coverage. Across the main comparison, TS-OPD is the strongest student objective for macro average correctness with the GRPO-improved teacher, while pass@K remains more mixed. Ablations show that the gains come from routing and token gating rather than skipping problems. These results support a transfer-aware view of OPD: stronger teachers help when the supervision direction and token budget match the student's observed ability, not merely because the teacher endpoint is stronger.
Xiaoyu Chen, Bo Shao, Tiangang Zhu +5
Institute of Software, Chinese Academy of Sciences · Microsoft · Work done during an internship at Microsoft +1
On-policy distillation is a powerful way to transfer reasoning ability from a strong teacher to a smaller student: the student samples trajectories from its own policy, and the teacher provides dense token-level supervision on the states the student actually visits. However, this supervision is not always reliable: a teacher can assign high likelihood to plausible but incorrect solutions, or low likelihood to correct student solutions that follow different reasoning paths. Unconditionally distilling the teacher can therefore reinforce bad modes or erase useful student behavior. To address these limitations, we introduce RG-OPD: Reward-Gated On-Policy Distillation that uses verifier feedback to decide when teacher logits should be trusted. RG-OPD bridges sparse verifier rewards and dense teacher logits, preserving token-level supervision while filtering misleading teacher signals. Across reasoning and coding benchmarks, RG-OPD produces stronger distilled students, outperforming both vanilla reverse-KL distillation and the recent TSD-KD baseline. At 1K generation length, RG-OPD improves over reverse-KL by 2.9 points and over TSD-KD by 4.9 points; in the long-generation setting, it improves over the untuned student by 8.2 points. Our code is available at https://github.com/UoC-tail/RG-OPD.
Mohammad Sadegh Akhondzadeh, Vijay Lingam, Atula Tejaswi +3
On-policy distillation (OPD) grounds token-level supervision in the student's own trajectory, yet suffers from prefix failure: once the student commits to a wrong reasoning direction, all subsequent generation builds on this deviation, producing misdirected continuations that elicit unreliable supervision and waste compute. We identify a teacher-student continuation asymmetry on failed prefixes, where the teacher tends to redirect while the student continues along the original direction, and convert it into a label-free handoff trigger in Relay On-Policy Distillation (Relay-OPD). During training, Relay-OPD constructs relay trajectories by letting the teacher briefly take over at detected trigger points to produce a teacher leg, after which the student resumes and is optimized on the resulting trajectory. A limited relay budget concentrates intervention on critical early positions while limiting departure from the student policy. With a Qwen3-4B-Instruct-2507 teacher and Qwen3-0.6B/1.7B-Non-Thinking students on eight mathematical reasoning benchmarks, Relay-OPD achieves the best or second-best results on every benchmark, outperforming standard OPD by +5.73% and the strongest baseline FastOPD by +1.49% on average for 1.7B, with consistent gains at 0.6B. Training trajectory length is reduced by over 50%.
Haolei Xu, Xiaowen Xu, Haiwen Hong +5
1Zhejiang University · 2Yuvion Team, Alibaba Group