Organizations: School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen · Shenzhen Future Network of Intelligence Institute (FNiI-Shenzhen) · Guangdong Provincial Key Laboratory of Future Networks of Intelligence, CUHK(SZ)
On-policy distillation (OPD) improves the reasoning capabilities of small language models through token-level teacher supervision on student-generated trajectories. Yet can teachers that excel at solving problems independently also guide student reasoning effectively? Prior work shows that when student prefixes follow reasoning paths that differ from the teacher's own or contain errors, teachers can be less accurate when continuing from these prefixes than when solving problems independently. To this end, we propose Prep-OPD, which uses reinforcement learning (RL) before distillation to train the teacher to adapt to the student's existing reasoning state and correct course when errors arise. Training optimizes teacher continuations from fixed student prefixes using final-answer correctness as the reward. The prepared teacher then trains the student through trajectory guidance and token-level supervision. We evaluate Prep-OPD on eight mathematical reasoning benchmarks, using Qwen3-4B-Instruct-2507 as the teacher and Qwen3-0.6B and Qwen3-1.7B as students. With the 4B teacher and 1.7B student, Prep-OPD improves average accuracy over standard OPD and the strongest baseline, Relay-OPD, by 8.28 and 2.30 percentage points, respectively. Controlled experiments further show that teacher RL conditioned on student-generated prefixes yields higher student accuracy than problem-start teacher RL with and without handoff on Qwen3-1.7B. Reusing the same prepared teacher also improves Qwen3-0.6B.
Figures & tables
Figure 1: Motivation and main results. (a) A solution from scratch and two continuations from the same erroneous student prefix. (b) Qwen3-1.7B mean benchmark accuracy (%; Table 1 ).
Figure 2: Overview of Prep-OPD. Stage 1 fixes prefixes from S0 and trains Tϕ , initialized from T0 , with outcome rewards and suffix-only GRPO. Stage 2 freezes TP for bounded handoffs and token-level supervision of mixed trajectories. Blue/purple tokens denote student/teacher generation; distillation covers both token types. Inference uses the student alone.
Method
Teacher
AIME
AMC
HMMT
Olympiad
MATH
Avg.
2024
2025
2026
2023
Feb. 26
Nov. 25
Bench
500
Qwen3-0.6B Student
Initial student ( S0 )
–
2.08
1.25
0.00
17.19
0.00
1.25
14.15
38.35
9.28
GRPO
–
9.17
13.23
12.08
46.56
6.25
7.08
35.15
65.60
24.39
OPD
TB
13.33
12.50
14.17
51.25
8.33
9.17
39.93
70.60
27.41
TRD
T0
4.58
7.92
5.94
39.61
6.72
4.69
29.63
61.60
20.09
Table 1: Mathematical reasoning accuracy (%). Avg. is the unweighted mean over eight benchmarks. Distillation runs for both the 1.7 B and 0.6 B students use 16,164 training problems. Bold/underline indicate the best/second-best reported scores within each student block.
Figure 3: Training efficiency on Qwen3-1.7B. (a) Accuracy versus cumulative student training compute for the checkpoint selected on the development set for each distillation method. Filled markers identify Pareto-optimal points among these selected checkpoints; the staircase gives their best accuracy within each compute budget. (b) Mean rollout length over the full 126-step student training run for each method.
Figure 4: Teacher continuation across prefix depths. Accuracy at each depth weights the eight benchmarks equally, and the depth mean weights the four continuation positions equally. The 0% point denotes problem-start solving and is excluded from this mean. Panel titles report the mean recovery gap in percentage points. Shading shows the pointwise gap between problem-start and continuation accuracy, not uncertainty. Ordinary RL denotes problem-start RL.
Teacher
No handoff
With handoff
Δ handoff (pp)
T0
39.96
45.33
+5.36
TB
39.37
45.35
+5.98
TP
41.26
47.65
+6.39
Table 2: Ablations of teacher preparation, handoff, and teacher roles. Results are Qwen3-1.7B student accuracies (%), averaged over the eight benchmarks with equal weights.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
LR
Optimizer
Batch/mini
Schedule
Additional settings
Teacher RL ( TB , TP )
10−6
AdamW
128/128
constant
GRPO group size G=8 ; response cap 16,384 tokens
OPD students
10−6
AdamW
128/128
constant
One PPO optimization epoch; dynamic batching
Handoff students
10−6
AdamW
128/128
constant
Handoff training channel ( train.sh )
FastOPD
10−6
AdamW
128/128
constant
Training rollout cap 4,096 tokens
IW-OPD
10−6
AdamW
128/128
constant
Importance-weight parameter γ=0.5
Appendix
Table 3: Training hyperparameters. Batch/mini denotes the configured training batch and PPO mini-batch sizes. The prepared teacher is denoted by TP throughout the paper.
Arm
Teacher
Channel
Budget
Role
S0
–
–
–
untrained floor
OPD
TB
standard
16K
main distilled baseline
TRD
TB
offline rewrite
16K
re-run of Jiang et al. (2026)
IW-OPD
TB
standard
16K
importance-weighted OPD
FastOPD
TB
truncated rollouts
4K
re-run of Zhang et al. (2026)
Relay-OPD
TB
relay
16K
equal-RL handoff baseline
Appendix
Table 4: Full-corpus Qwen3-1.7B distillation configurations ( 16,164 student-training problems). Budget is the student-training response cap.
Student-prefix depth
Benchmark
Teacher
CT
20%
40%
60%
80%
PT
GT
AIME 2024
T0
60.10
60.94
56.15
53.33
54.17
56.15
3.95
TB
67.92
60.42
54.79
50.10
44.90
52.55
15.37
TP
59.27
61.46
59.48
53.65
56.15
57.68
1.58
AIME 2025
T0
41.04
45.31
46.46
38.23
38.13
42.03
-0.99
TB
53.13
44.58
41.35
35.73
38.02
39.92
13.21
Appendix
Table 5: Detailed teacher continuation results. Problem-start accuracy CT and continuation accuracy at four student-prefix depths are reported in percent. PT is the equal-weight mean over the four depths; GT=CT−PT is the recovery gap in percentage points.
Qwen3-0.6B
Qwen3-1.7B
Benchmark
Relay ( T0 )
Prep ( TP )
Δ
Relay ( T0 )
Prep ( TP )
Δ
AIME 2024
13.33
17.50
+4.17
43.65
48.33
+4.69
AIME 2025
14.17
18.33
+4.17
28.33
31.67
+3.33
AIME 2026
18.33
15.00
−3.33
31.15
32.08
+0.94
AMC 2023
50.62
57.50
+6.88
72.50
75.62
+3.12
HMMT Feb. 2026
8.33
10.61
+2.27
23.77
25.00
+1.23
Appendix
Table 6: Comparison with Relay-OPD using the original teacher. Accuracy is reported in percent; Δ is Prep-OPD minus Relay-OPD in percentage points. Bold marks the higher accuracy within each student size. Avg. weights the eight benchmarks equally. Differences are computed before rounding.
Method
Seed
AIME
AMC
HMMT
Olympiad
MATH
Avg.
2024
2025
2026
2023
Feb. 26
Nov. 25
Bench
500
Prep-OPD ( TP )
42
48.33
31.67
32.08
75.62
25.00
20.83
60.44
87.20
47.65
43
45.83
30.83
34.17
75.00
23.48
20.83
59.93
86.30
47.05
44
43.33
31.67
37.50
73.75
26.52
23.33
58.30
85.00
47.42
Relay-OPD ( TB )
42
38.33
31.25
30.83
74.69
23.48
20.94
58.00
85.25
45.35
43
40.00
29.58
29.58
74.38
23.48
20.83
56.67
84.45
44.87
Appendix
Table 7: Comparison across random seeds on Qwen3-1.7B. Accuracy is reported in percent. The seed-42 rows reproduce Table 1 ; seeds 43 and 44 are additional runs. Avg. is the unweighted mean over the eight benchmarks, computed before rounding the benchmark entries.
On-policy distillation (OPD) trains a student on its own reasoning trajectories using feedback from a stronger teacher. Teacher interventions can improve these trajectories, but also change the distribution on which the student learns. Our controlled studies show that rollout quality alone is an incomplete criterion for allocating teacher guidance. Deeper intervention yields diminishing gains in rollout accuracy while increasing off-policy load. In a training probe with a restricted rollout horizon, peak student accuracy and performance retention favor different intervention strengths. The preferred intervention depth and placement also vary across benchmarks. These findings motivate MAESTRO, which uses local policy disagreement to jointly adapt when the teacher takes over and how long it generates. Its {policy disagreement score} combines teacher-weighted candidate coverage with local distribution similarity and is aggregated within reasoning paragraphs. Across eight mathematical reasoning benchmarks, MAESTRO achieves the highest macro-average accuracy among the compared methods for both 0.6B and 1.7B Qwen3 students, with the 1.7B student leading on every benchmark. MAESTRO also reduces average training response length by 67.3% relative to standard OPD. The code is available at https://github.com/yhao-wang/MAESTRO.
Yuhao Wang, Ruiyang Ren, Yinan Zhang +3
Nanyang Technological University, Singapore · Baidu Inc.
Reinforcement learning can substantially improve a reasoning teacher, but it is unclear which of those improvements survive when the teacher supervises a smaller on-policy student. We study this question in mathematical reasoning by comparing teacher lineages before and after GRPO, multiple student scales, direct GRPO, and several on-policy distillation objectives. The central finding is that transfer is structured rather than scalar: teacher strength alone does not make dense distillation competitive, while an RL-improved teacher creates useful but metric-dependent student gains. This motivates Transfer-Stratified On-Policy Distillation (TS-OPD), which screens training problems by the joint sampled success of the student and teacher, routes acquisition problems to gated forward KL, routes consolidation problems to gated reverse KL, and adds an entropy brake to protect sampled coverage. Across the main comparison, TS-OPD is the strongest student objective for macro average correctness with the GRPO-improved teacher, while pass@K remains more mixed. Ablations show that the gains come from routing and token gating rather than skipping problems. These results support a transfer-aware view of OPD: stronger teachers help when the supervision direction and token budget match the student's observed ability, not merely because the teacher endpoint is stronger.
Xiaoyu Chen, Bo Shao, Tiangang Zhu +5
Institute of Software, Chinese Academy of Sciences · Microsoft · Work done during an internship at Microsoft +1
On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher, but providing such supervision for every rollout requires substantial teacher computation. We introduce Success-Referenced On-Policy Distillation (SR-OPD), which reduces this cost by selecting which prompts and rollouts receive teacher supervision. When the student produces both successful and failed rollouts for the same prompt, a successful rollout can serve as a natural reference for selecting failed rollouts. SR-OPD therefore focuses on such prompts and prioritizes failed rollouts whose hidden-state trajectories show sustained divergence from a successful reference, while accounting for estimated teacher-input cost. Across three teacher-student pairs and six mathematical reasoning benchmarks, SR-OPD uses only 3.46-5.02% of the teacher-input tokens required by Vanilla OPD in the one-pass setting while maintaining comparable reasoning performance. Under a controlled setting matched to 5% of Vanilla OPD's teacher-input budget, further experiments support both key design choices: focusing supervision on prompts with both successful and failed rollouts, and using successful rollouts to guide failure selection. These results indicate that a student's own successful behavior can serve as a useful reference for allocating teacher supervision under a fixed teacher-input budget.
Xiang Chen, Futao Su, Kong Wang +2
Tongji University · Xiaohongshu · Shanghai Jiao Tong University +1