Policy-shift distillation reuses the difference between a teacher's log probabilities before and after reinforcement learning (RL) as supervision on student-generated trajectories. However, this endpoint policy shift cannot reveal whether local action preferences evolved consistently during teacher training. The same endpoint shift can result from early and late shifts that reinforce or partially cancel each other. We investigate whether agreement between early and late teacher policy shifts can guide selective transfer. We introduce Waypoint-guided On-Policy Distillation (Waypoint-OPD), which uses an intermediate teacher checkpoint from the same RL run to compare early and late shifts in relative action preferences at student-visited prefixes. We gate the endpoint shift according to the directional agreement between the early and late shifts, suppressing it when they point in opposing directions. Compared with endpoint-only transfer, our method requires only one additional frozen-teacher scoring pass. Using RL-trained Qwen3-4B teachers, we evaluate our method with Qwen3-8B, Qwen3-14B, and Qwen3-30B-A3B as student models in mathematical and code reasoning. In all six domain--student settings, Waypoint-OPD achieves the highest domain-average score among the compared distillation methods. Ablations with the Qwen3-8B student show that removing or shuffling gates reduces mathematical reasoning performance. These results support the use of directional agreement between early and late teacher policy shifts to weight the endpoint shift during student training.
Figures & tables
Figure 1: Mathematical reasoning across three student models. mean@32 accuracy (%), averaged over seven benchmarks. Waypoint-OPD improves over Direct-OPD by 2.18, 1.59, and 1.89 percentage points, respectively. Teacher scores are from the Final Qwen3-4B checkpoint.
Figure 2: Training pipeline of Waypoint-OPD. Initial, Waypoint, and Final score candidates at student-generated prefixes. Their centered segment shifts define an agreement gate on the endpoint shift. Batch scaling and detached candidate advantages drive the student update Waypoint measures consistency only.
Method
AIME24
AIME25
AIME26
HMMT25 Feb
HMMT25 Nov
HMMT26 Feb
AMC23
Avg.
Teacher: Qwen3-4B
Initial
21.67
21.67
21.67
12.50
10.00
15.15
71.25
24.84
Waypoint
58.96
54.79
56.77
31.35
35.73
31.72
93.44
51.82
Final
65.83
60.00
60.83
35.83
40.83
35.13
94.45
56.13
Student: Qwen3-8B
Base
29.17
20.83
18.33
11.67
13.33
18.94
70.00
26.04
Table 1: Mathematical reasoning across three student sizes. mean@32 scores (%) on seven benchmarks and their average. Bold marks the best score per student. Teacher checkpoints are references.
Teacher reference: Qwen3-4B
Initial: 37.64 Waypoint: 52.22 Final: 55.37
Method
Qwen3-8B
Qwen3-14B
Qwen3-30B-A3B
Base
41.03
47.72
50.33
OPD
43.91
51.66
51.91
OPD 2
51.21
55.60
55.96
Direct-OPD
50.59
56.30
56.35
Table 2: Average code reasoning performance. Average mean@8 scores (%) over five evaluations. Qwen3-4B teacher checkpoints are shared references. Bold marks the best score per student. Full results: Appendix D .
Figure 3: Evaluation accuracy during training. Qwen3-8B results on AIME 2025, AIME 2026, and HMMT November 2025 at steps 0, 20, 40, 60, 80, and 100. Scores are mean@32 percentages. Solid red curves denote Waypoint-OPD and dashed blue curves denote Direct-OPD. Gray dashed lines show the Final Qwen3-4B teacher scores.
Method
AIME24
AIME25
AIME26
HMMT25 Feb
HMMT25 Nov
HMMT26 Feb
AMC23
Avg.
No gate
66.25
54.27
56.67
32.60
39.79
34.66
90.23
53.50
Shuffled gate
63.33
52.92
54.58
30.00
37.08
31.06
89.45
51.20
No normalization
68.13
56.46
58.33
34.79
41.56
35.23
92.03
55.22
Direct-OPD
67.29
55.31
57.40
31.77
40.83
33.71
91.41
53.96
Waypoint-OPD
68.85
57.92
60.21
34.38
42.60
36.36
92.66
56.14
Table 3: Gate and normalization ablations on Qwen3-8B mathematical reasoning. Bold marks the best score in each column.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Train batch size
128
Micro batch size
128
Rollout n
8
Maximum prompt length
2,048
Maximum response length
16,384
Temperature
1.0
Appendix
Table 4: GRPO training configuration shared by the math and code teachers.
Setting
Value
Math training dataset
DAPO-Math-17K
Code training dataset
Codeforces-derived
Train batch size
256
Mini batch size
64
Rollout n
1
Maximum prompt length
2,048
Appendix
Table 5: Transfer configuration shared by math and code, except for the transfer datasets.
Item
Value
Initial teacher
Qwen3-4B
Waypoint teacher
RL step 600
Final teacher
RL step 1200
Students
Qwen3-8B, Qwen3-14B, Qwen3-30B-A3B
Math evaluation
AIME 2024, AIME 2025, AIME 2026, HMMT February 2025,HMMT November 2025, HMMT February 2026, AMC 2023
Table 7: Full code reasoning results across three student sizes. Scores are mean@8 percentages. Avg. is the unweighted mean across the five evaluations. Bold marks the best score within each student block. Teacher checkpoints are reference rows.
Waypoint
AIME24
AIME25
AIME26
HMMT25 Feb
HMMT25 Nov
HMMT26 Feb
AMC23
Avg.
Step 500
69.06
58.13
59.79
33.96
42.60
36.74
92.97
56.18
Step 600
68.85
57.92
60.21
34.38
42.60
36.36
92.66
56.14
Step 700
69.27
57.60
59.79
34.58
42.81
36.36
92.50
56.13
Appendix
Table 8: Sensitivity to the Waypoint checkpoint on Qwen3-8B mathematical reasoning. Only the intermediate teacher checkpoint is varied, while the Initial and Final teachers are fixed. Step numbers refer to the teacher’s RL training. Bold marks the best score in each column.
Figure 4: Gate diagnostics on Qwen3-8B mathematical reasoning. Statistics for Waypoint-OPD over 100 training steps. From left to right: mean gate gt , zero-gate fraction, batch-scale normalization multiplier κB , and mean per-prefix reward-vector L2 norm. Raw uses the gated, student-weighted endpoint signal qt(a)gtdt(a) ; Normalized uses the final candidate coefficients At(a) in Eq. 12 from the same run. All statistics aggregate valid response prefixes.
Method
Time (h:m:s)
Seconds/step
GPU-hours
Direct-OPD
8:21:03
300.63
66.81
Waypoint-OPD
8:02:21
289.41
64.31
Appendix
Table 9: Training-time comparison on Qwen3-8B math. Total training time over 100 training steps on eight H20 GPUs, excluding evaluation.
Figure 5: Response length and response truncation rate. The two panels compare Qwen3-8B trained with Waypoint-OPD (solid red) and Direct-OPD (dashed blue) over 100 training steps. (a) Mean number of tokens per rollout response at each training step. (b) Fraction of rollout responses reaching the maximum length of 16,384 tokens; 0.2 corresponds to 20%.