On-policy distillation (OPD) trains a student on its own rollouts under a frozen teacher, mitigating the distribution mismatch of off-policy supervision. While prior work has primarily trained on problems a student can largely solve, we ask whether OPD can improve a student when training is restricted to problems it currently fails. We evaluate five representative OPD mechanisms with Qwen-3 and Phi-4 models under a unified competition-mathematics protocol. We find that OPD does not improve accuracy on this hard subset, with most mechanisms instead degrading performance despite continued decreases in training KL divergence. All eight configurations show a decline, and the two-sided sign test gives p=0.0078. In contrast, the same pipeline improves accuracy by 4.6 percentage points when training is not concentrated on failures, showing that the failure is conditional on the training regime rather than systematic. These findings suggest that OPD is effective when supervision is dominated by correct student prefixes, but can reinforce existing errors when training is concentrated on problems the student currently fails.
Figures & tables
Figure 1: Closer to the teacher does not imply more correct. When the student’s prefix y<t already contains an error, on-policy distillation drives pS toward pT(⋅∣x,y<t) , reducing KL divergence yet failing to improve accuracy relative to the frozen student.
Bucket (base acc.)
n
Base
Priv.
Δ (pp)
=0
57
0.0
17.5
+17.5
(0,0.5)
13
25.0
75.0
+50.0
[0.5,1)
46
64.7
96.7
+32.0
=1 (saturated)
684
100.0
100.0
0.0
Table 1: Headroom from the reference solution sits on problems the student fails.
Figure 2: Accuracy change relative to the frozen student on the competent-student hard subset. All evaluated mechanisms yield non-positive changes. Single-run results are shown except for the TRL configuration, for which three training seeds and their mean are reported. The same pipeline improves weaker students, as discussed in Section 3.2 .
Configuration
Metric
Δ
Residual offset / forward KL (8B)
Macro
−2.2
Token attention
Sparse-accuracy RL (–)
Macro
−11.1
Privileged imitation (4B)
AIME24
−11.7
OPD after imitation (4B)
AIME24
+1.7
Table 2: Independent steering results (avg@2). Teacher sizes are in parentheses and “–” denotes no teacher.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Aspect
OPD
OPSD
Teacher input
x only
x and y⋆
Gold anchor
None
Reference path y⋆
Positive gain
Distillation
Elicitation + write-in
Appendix
Table 3: OPD lacks OPSD’s privileged answer anchor, making its training problem harder.
Figure 3: Accuracy with and without the reference solution in context, by the student’s own base accuracy, with thinking enabled and a 16k-token budget.
Figure 4: The five mechanisms, ordered by the extent to which each modifies the frozen model. The first four leave all parameters of θ unchanged, whereas only full fine-tuning updates them.
Configuration
Base
Meth.
Δ
W/L/T
p
Prefix, OPSD-style KL
26.7
13.3
−13.3
2/6/22
0.29
Prefix, OPD (4B)
26.7
20.0
−6.7
0/2/28
0.50
Full FT, forward KL
40.0
30.0
−10.0
2/5/23
0.45
Full FT, reverse KL
46.7
36.7
−10.0
2/5/23
0.45
TRL reverse KL, seed 42
46.7
40.0
−6.7
3/5/22
0.73
TRL reverse KL, seed 0
40.0
20.0
−20.0
0/6/24
0.031
Appendix
Table 4: AIME24 pass@1 accuracy with thirty problems for every configuration: accuracy in percent, delta, items won, lost, and tied by the method, and the exact two-sided McNemar p -value. Each configuration re-samples its own frozen baseline.
Figure 5: Three training seeds of the TRL configuration. The frozen student itself varies between 40.0 and 46.7% across evaluation runs, and its per-problem verdict changes on ten of thirty items.
Figure 6: Trained soft prefixes are nearly collinear across ninety-four problems, drawn from all thirty AIME24 items plus sixty-four further hard-subset items, unlike the pooled problem inputs or an untrained encoder of identical architecture. Static embeddings serve as a proxy input for the checkpoints trained with contextual pooling. A high cosine under this proxy remains informative even so.
Control
Base
Student
Δ
n
p
Qwen2.5-0.5B, GSM8K
43.6
48.2
+4.6
1319
9.5×10−4
Phi-4-mini, MATH-500
67.0
74.0
+7.0
100
0.21
Phi-4-mini, AIME24
10.0
13.3
+3.3
30
1.00
Appendix
Table 5: Positive controls under the identical TRL reverse-KL pipeline. For GSM8K the student wins 196 items and loses 135, with a 95% confidence interval of 2.0 to 7.4 points. For Phi-4-mini on MATH-500 the student wins 15 items and loses 8, with a 95% confidence interval of −2.0 to 16.0 points.
Isolation
Seed repeats
GSM8K
Phi-4
Student
Qwen3-1.7B
Qwen3-1.7B
Qwen2.5-0.5B
Phi-4-mini
Teacher
Qwen3-4B
Qwen3-4B
Qwen2.5-1.5B
Phi-4-reason.
Train set
hard-904
hard-904
GSM8K-7473
MATH-800
Steps run
200
200
934, 1 epoch
200
Learning rate
1.0e-5
1.0e-5
2.0e-5
1.0e-5
Batch × accum.
1 × 4
1 × 4
2 × 4
1 × 4
Appendix
Table 6: Hyperparameters shared by every TRL DistillationTrainer run in the paper, reverse KL at β=1 , weight decay 0.01, gradient clip 1.0, the AdamW optimizer ( Loshchilov and Hutter, 2017 ) with β1=0.9 , β2=0.999 , ϵ=10−8 , and a linear learning-rate decay. Sampled evaluation uses temperature 0.6 with top- p 0.95 and top- k 20.
Fwd KL
Rev KL
Prefix, OPD
Prefix, OPSD
Updates
full params
full params
prefix, 64 tok.
prefix, 96 tok.
Imitation warm-up
none
none
300 steps
500 steps
KL direction
forward
reverse
forward
OPSD-style
KL support
top-64, clip 5
top-64, clip 5
top-64, clip 5
top-64, clip 5
Train rollout cap
16384
16384
4096
4096
Steps run, evaluated
400
800, at 500
400
400
Appendix
Table 7: Hyperparameters for the in-house engine behind the remaining four rows of Table 4 , all with a constant learning rate, no warmup, and the AdamW optimizer ( Loshchilov and Hutter, 2017 ) . Evaluation always uses a 16384-token cap regardless of the training rollout cap above. A shorter training cap is a property of the training setup only.
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
On-policy distillation (OPD) trains a student model on its own rollouts with token-level supervision from a teacher, assuming that the teacher can reach a correct answer from any prefix the student writes. We call this property the recoverability of the teacher and show it fails on incorrect rollouts: the teacher follows student errors, although it recognizes them. We propose CriticOPD, using error recognition to restore recoverability. The teacher critiques an incorrect rollout, allowing the student to rewrite its solution following the error. The resulting output, including the critique as privileged information, is scored by the teacher while the student uses the revision without the critique for training. On two Qwen3 students and six mathematical reasoning benchmarks, CriticOPD improves OPD, with the largest gains on competition problems.
Kaixiang Zhao, Tianrun Yu, Shangzhe Li +4
Brigham Young University · University of North Carolina at Chapel Hill
On-policy distillation (OPD) trains a student on its own trajectories while a teacher supplies dense token-level likelihoods at student-visited prefixes. These likelihoods are often read locally: agreement appears safe to imitate, whereas disagreement appears to identify an error. We show that both readings are confounded by the outcome of the completed trajectory. We introduce an outcome-resolved diagnostic that crosses pointwise teacher-student divergence with final-answer correctness, separating safe imitation, productive divergence, harmful divergence, and agreement-on-failure. In an eight-seed mathematical-reasoning study with a Qwen3-8B student and Qwen3-32B teacher, agreement-on-failure constitutes 67.84% of pooled response-token mass; with a Qwen2.5-7B/32B pair it remains 67.68%. The result persists across threshold, sequence-level, format, and truncation audits. Even on prompts that the Qwen3 teacher solves in all four independent attempts, student accuracy rises to 86.91% but agreement-on-failure remains 14.76%. We then run three matched training probes that use the available signals to imitate, mask, or contrast whole trajectories; none consistently reduces agreement-on-failure. The result points to a localization limitation: local divergence paired with a trajectory-level outcome does not identify where a failed trajectory became unrecoverable. Addressing this limitation requires additional positional information, such as process labels, teacher continuations from student prefixes, or token-level alignment across rollouts. Our contribution is therefore diagnostic rather than a new training method.
On-policy distillation (OPD) is increasingly studied as a cheaper alternative to reinforcement learning, yet it remains a form of knowledge distillation (KD) that relies on teacher supervision. We revisit OPD from the KD perspective by asking where the teacher's advantage lies along student-generated trajectories. Controlled token flips at teacher-student disagreements directly measure the accuracy gained from teacher corrections. Single-token flips yield a modest average gain on selected failed trajectories, whereas repeated flips raise accuracy from 28.4% to 74.5%, close to the teacher's 76.9%. Gains continue to accumulate with more flips, but mass- and KL-based selection outperform random flips by 15-18 percentage points at counts of similar magnitude. Selective intervention remains useful after OPD, but greater argmax agreement does not consistently improve accuracy across model pairs. We advocate training schedules that learn more valuable teacher corrections while maintaining distributional stability. Controlled flips and fork-resolution analysis evaluate OPD beyond accuracy by measuring which teacher preferences the student acquires and what further correction can recover.