On-policy distillation (OPD) trains a student model on its own rollouts with token-level supervision from a teacher, assuming that the teacher can reach a correct answer from any prefix the student writes. We call this property the recoverability of the teacher and show it fails on incorrect rollouts: the teacher follows student errors, although it recognizes them. We propose CriticOPD, using error recognition to restore recoverability. The teacher critiques an incorrect rollout, allowing the student to rewrite its solution following the error. The resulting output, including the critique as privileged information, is scored by the teacher while the student uses the revision without the critique for training. On two Qwen3 students and six mathematical reasoning benchmarks, CriticOPD improves OPD, with the largest gains on competition problems.
Figures & tables
Figure 1: Where the supervision of each method falls after the student errs, and who writes it.
Figure 2: Overview of CriticOPD. The student rewrites an incorrect rollout from the last error that the teacher critiques. The teacher scores the rewrite with the critique, and the student learns from it without the critique.
Figure 3: Teacher ( ΛT ) and student ( ΛS ) continuation accuracy along (a) correct and (b) incorrect rollouts.
Figure 5: An example of critique-guided rewriting. Left: the original solution, with the critiqued paragraphs marked. Middle: the critique, which quotes each erroneous sentence, gives a fix, and ends with the reference answer. Right: the suffix that the student writes from the last error with the critique.
Figure 6: Recoverability of the teacher and the student on original rollouts (dashed) and critique-guided rewrites (solid), plotted against the position relative to the cut point. On the rewrite, the teacher sees the critique, and the student does not.
Method
AIME24
AIME25
AMC23
MATH500
OlympiadBench
Minerva
Avg
Teacher
62.19
48.12
85.99
94.83
72.26
45.54
68.16
Student: Qwen3-0.6B-Non-Thinking
Student
2.08
1.35
18.98
45.52
16.94
13.69
16.43
SFT @139
4.62 ±0.49
6.84 ±0.12
30.08 ±0.20
58.60 ±0.37
27.24 ±0.20
15.59 ±0.25
23.83 ±0.06
KD @139
4.51 ±0.63
6.15 ±0.73
28.75 ±0.80
57.88 ±0.69
26.62 ±0.52
15.09 ±0.88
23.17 ±0.49
GRPO @80
3.82 ±0.71
3.78 ±1.27
26.46 ±0.92
53.72 ±1.60
23.45 ±1.14
15.12 ±0.71
21.06 ±1.00
Table 1: Results on six mathematical reasoning benchmarks. Entries are means and standard deviations over three seeds, and the subscript after each method gives the selected step. Bold marks the best student at each scale.
Variant
AIME
Avg
Δ
CriticOPD
34.79
54.57
–
(a) Content of the feedback
w/o answer
34.58
53.72
− 0.85
w/o critique
28.22
48.67
− 5.90
critique + answer → generic
33.85
53.11
− 1.46
(b) Writer of the suffix
Table 2: Ablations on Qwen3-1.7B with seed 42 . Δ is the change in Avg from CriticOPD, and from CriticOPD without the answer in (b) and (c). AIME averages AIME 2024 and AIME 2025.
Figure 7: Occupancy mismatch and teacher reliability on the states that each method supervises, and the average accuracy of the Qwen3-1.7B student.
Figure 8: (a) Advantage per token of two continuations of 64 tokens at the cut point, which either continue or correct the error, under the scoring of OPD and of CriticOPD. Numbers give correct minus continue. (b) Success rate of the student of CriticOPD minus that of OPD from the same prefixes on OlympiadBench, with 95% confidence bands.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Expected accuracy of the teacher and the Qwen3-0.6B student from prefixes of the student on (a) correct and (b) incorrect rollouts.
Figure 10: (a) Outcomes of the continuations of the teacher and (b) fraction of the listed errors that are real, for the Qwen3-0.6B student.
Figure 11: Success rates of the teacher and the Qwen3-0.6B student around the cut point.
Figure 12: Token positions of the prefixes that each method supervises (colored) and of the inference prefixes (gray), with the total variation distance in each title. The dotted line marks the training limit of 16,384 tokens.
Figure 13: Hidden states of the student on the prefixes that each method supervises, projected onto the first two principal components of the inference states. The contour encloses 95% of the inference states, and each title gives the classifier bound on the total variation distance.
Setting
Value
Training
Prompts per step
128
Rollouts per prompt
1 (GRPO: 8 )
Optimizer updates per step
1
Learning rate
10−6 , constant, no warmup
Maximum prompt length
2,048
Appendix
Table 3: Hyperparameters of all methods.
Figure 14: Training cost of the Qwen3-1.7B students. (a) A100 GPU-hours until the selected step, including the generation of offline data. (b) Seconds per training step, averaged over five steps.
Table 4: Results with OLMo-3 models on six mathematical reasoning benchmarks with seed 42 . The subscript after each method gives the step, and bold marks the best student.
Method
MMLU-Pro
MMLU
ARC-C
OpenBookQA
HumanEval+
MBPP+
Avg
Teacher
69.75
80.09
94.94
93.95
82.00
74.70
82.57
Student
38.10
55.24
77.69
74.10
62.50
56.50
60.69
SFT
50.80
63.29
84.41
79.70
63.10
55.40
66.12
KD
50.85
63.78
84.28
79.30
64.00
57.10
66.55
GRPO
51.41
61.49
81.68
78.75
63.70
58.50
65.92
OPD
56.61
67.14
87.07
83.80
66.30
61.90
70.47
Appendix
Table 5: Results outside mathematics for Qwen3-1.7B. Code uses EvalPlus pass@1 on the extended tests.
Variant
AIME24
AIME25
AMC23
MATH500
Olympiad
Minerva
Avg
CriticOPD
40.52
29.06
70.18
89.40
60.59
37.64
54.57
(a) Content of the feedback
w/o answer
41.56
27.60
67.43
88.75
59.04
37.91
53.72
w/o critique
34.48
21.96
60.33
85.55
54.83
34.89
48.67
critique + answer → generic
40.00
27.71
67.39
88.05
58.91
36.63
53.11
(b) Writer of the suffix
Appendix
Table 6: Per-benchmark results of the ablations on Qwen3-1.7B with seed 42 . The rows follow Table 2 .
Figure 15: Sensitivity of CriticOPD on the Qwen3-1.7B student with seed 42 . The star marks the default, and the numbers in (b) give the share of truncated critiques.
Figure 16: Training dynamics, averaged over seeds and smoothed over five steps.
Figure 17: Average over six benchmarks when responses longer than the budget count as incorrect, over three seeds of Qwen3-1.7B.
Method
AIME length
Avg@16k
Avg@32k
OPD
8,487
49.99
50.08
FastOPD
11,910
50.27
52.55
CriticOPD
12,457
50.32
54.65
scoring w/o feedback
12,065
50.62
52.26
critique → generic
12,051
50.86
52.87
Appendix
Table 7: Length-controlled comparison on Qwen3-1.7B, averaged over three seeds.
On-policy distillation (OPD) trains a student on its own rollouts under a frozen teacher, mitigating the distribution mismatch of off-policy supervision. While prior work has primarily trained on problems a student can largely solve, we ask whether OPD can improve a student when training is restricted to problems it currently fails. We evaluate five representative OPD mechanisms with Qwen-3 and Phi-4 models under a unified competition-mathematics protocol. We find that OPD does not improve accuracy on this hard subset, with most mechanisms instead degrading performance despite continued decreases in training KL divergence. All eight configurations show a decline, and the two-sided sign test gives p=0.0078. In contrast, the same pipeline improves accuracy by 4.6 percentage points when training is not concentrated on failures, showing that the failure is conditional on the training regime rather than systematic. These findings suggest that OPD is effective when supervision is dominated by correct student prefixes, but can reinforce existing errors when training is concentrated on problems the student currently fails.
Shuo Yang, Zheyu Zhang, Mahito Sugiyama +1
Technical University of Munich · National Institute of Informatics · Munich Center for Machine Learning
On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher, but providing such supervision for every rollout requires substantial teacher computation. We introduce Success-Referenced On-Policy Distillation (SR-OPD), which reduces this cost by selecting which prompts and rollouts receive teacher supervision. When the student produces both successful and failed rollouts for the same prompt, a successful rollout can serve as a natural reference for selecting failed rollouts. SR-OPD therefore focuses on such prompts and prioritizes failed rollouts whose hidden-state trajectories show sustained divergence from a successful reference, while accounting for estimated teacher-input cost. Across three teacher-student pairs and six mathematical reasoning benchmarks, SR-OPD uses only 3.46-5.02% of the teacher-input tokens required by Vanilla OPD in the one-pass setting while maintaining comparable reasoning performance. Under a controlled setting matched to 5% of Vanilla OPD's teacher-input budget, further experiments support both key design choices: focusing supervision on prompts with both successful and failed rollouts, and using successful rollouts to guide failure selection. These results indicate that a student's own successful behavior can serve as a useful reference for allocating teacher supervision under a fixed teacher-input budget.
Xiang Chen, Futao Su, Kong Wang +2
Tongji University · Xiaohongshu · Shanghai Jiao Tong University +1
On-policy distillation (OPD) grounds token-level supervision in the student's own trajectory, yet suffers from prefix failure: once the student commits to a wrong reasoning direction, all subsequent generation builds on this deviation, producing misdirected continuations that elicit unreliable supervision and waste compute. We identify a teacher-student continuation asymmetry on failed prefixes, where the teacher tends to redirect while the student continues along the original direction, and convert it into a label-free handoff trigger in Relay On-Policy Distillation (Relay-OPD). During training, Relay-OPD constructs relay trajectories by letting the teacher briefly take over at detected trigger points to produce a teacher leg, after which the student resumes and is optimized on the resulting trajectory. A limited relay budget concentrates intervention on critical early positions while limiting departure from the student policy. With a Qwen3-4B-Instruct-2507 teacher and Qwen3-0.6B/1.7B-Non-Thinking students on eight mathematical reasoning benchmarks, Relay-OPD achieves the best or second-best results on every benchmark, outperforming standard OPD by +5.73% and the strongest baseline FastOPD by +1.49% on average for 1.7B, with consistent gains at 0.6B. Training trajectory length is reduced by over 50%.
Haolei Xu, Xiaowen Xu, Haiwen Hong +5
1Zhejiang University · 2Yuvion Team, Alibaba Group