On-policy distillation (OPD) trains a student on its own rollouts under a frozen teacher, mitigating the distribution mismatch of off-policy supervision. While prior work has primarily trained on problems a student can largely solve, we ask whether OPD can improve a student when training is restricted to problems it currently fails. We evaluate five representative OPD mechanisms with Qwen-3 and Phi-4 models under a unified competition-mathematics protocol. We find that OPD does not improve accuracy on this hard subset, with most mechanisms instead degrading performance despite continued decreases in training KL divergence. All eight configurations show a decline, and the two-sided sign test gives p=0.0078. In contrast, the same pipeline improves accuracy by 4.6 percentage points when training is not concentrated on failures, showing that the failure is conditional on the training regime rather than systematic. These findings suggest that OPD is effective when supervision is dominated by correct student prefixes, but can reinforce existing errors when training is concentrated on problems the student currently fails.
Figures & tables
Figure 1: Closer to the teacher does not imply more correct. When the student’s prefix y<t already contains an error, on-policy distillation drives pS toward pT(⋅∣x,y<t) , reducing KL divergence yet failing to improve accuracy relative to the frozen student.
Bucket (base acc.)
n
Base
Priv.
Δ (pp)
=0
57
0.0
17.5
+17.5
(0,0.5)
13
25.0
75.0
+50.0
[0.5,1)
46
64.7
96.7
+32.0
=1 (saturated)
684
100.0
100.0
0.0
Table 1: Headroom from the reference solution sits on problems the student fails.
Figure 2: Accuracy change relative to the frozen student on the competent-student hard subset. All evaluated mechanisms yield non-positive changes. Single-run results are shown except for the TRL configuration, for which three training seeds and their mean are reported. The same pipeline improves weaker students, as discussed in Section 3.2 .
Configuration
Metric
Δ
Residual offset / forward KL (8B)
Macro
−2.2
Token attention
Sparse-accuracy RL (–)
Macro
−11.1
Privileged imitation (4B)
AIME24
−11.7
OPD after imitation (4B)
AIME24
+1.7
Table 2: Independent steering results (avg@2). Teacher sizes are in parentheses and “–” denotes no teacher.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Aspect
OPD
OPSD
Teacher input
x only
x and y⋆
Gold anchor
None
Reference path y⋆
Positive gain
Distillation
Elicitation + write-in
Appendix
Table 3: OPD lacks OPSD’s privileged answer anchor, making its training problem harder.
Figure 3: Accuracy with and without the reference solution in context, by the student’s own base accuracy, with thinking enabled and a 16k-token budget.
Figure 4: The five mechanisms, ordered by the extent to which each modifies the frozen model. The first four leave all parameters of θ unchanged, whereas only full fine-tuning updates them.
Configuration
Base
Meth.
Δ
W/L/T
p
Prefix, OPSD-style KL
26.7
13.3
−13.3
2/6/22
0.29
Prefix, OPD (4B)
26.7
20.0
−6.7
0/2/28
0.50
Full FT, forward KL
40.0
30.0
−10.0
2/5/23
0.45
Full FT, reverse KL
46.7
36.7
−10.0
2/5/23
0.45
TRL reverse KL, seed 42
46.7
40.0
−6.7
3/5/22
0.73
TRL reverse KL, seed 0
40.0
20.0
−20.0
0/6/24
0.031
Appendix
Table 4: AIME24 pass@1 accuracy with thirty problems for every configuration: accuracy in percent, delta, items won, lost, and tied by the method, and the exact two-sided McNemar p -value. Each configuration re-samples its own frozen baseline.
Figure 5: Three training seeds of the TRL configuration. The frozen student itself varies between 40.0 and 46.7% across evaluation runs, and its per-problem verdict changes on ten of thirty items.
Figure 6: Trained soft prefixes are nearly collinear across ninety-four problems, drawn from all thirty AIME24 items plus sixty-four further hard-subset items, unlike the pooled problem inputs or an untrained encoder of identical architecture. Static embeddings serve as a proxy input for the checkpoints trained with contextual pooling. A high cosine under this proxy remains informative even so.
Control
Base
Student
Δ
n
p
Qwen2.5-0.5B, GSM8K
43.6
48.2
+4.6
1319
9.5×10−4
Phi-4-mini, MATH-500
67.0
74.0
+7.0
100
0.21
Phi-4-mini, AIME24
10.0
13.3
+3.3
30
1.00
Appendix
Table 5: Positive controls under the identical TRL reverse-KL pipeline. For GSM8K the student wins 196 items and loses 135, with a 95% confidence interval of 2.0 to 7.4 points. For Phi-4-mini on MATH-500 the student wins 15 items and loses 8, with a 95% confidence interval of −2.0 to 16.0 points.
Isolation
Seed repeats
GSM8K
Phi-4
Student
Qwen3-1.7B
Qwen3-1.7B
Qwen2.5-0.5B
Phi-4-mini
Teacher
Qwen3-4B
Qwen3-4B
Qwen2.5-1.5B
Phi-4-reason.
Train set
hard-904
hard-904
GSM8K-7473
MATH-800
Steps run
200
200
934, 1 epoch
200
Learning rate
1.0e-5
1.0e-5
2.0e-5
1.0e-5
Batch × accum.
1 × 4
1 × 4
2 × 4
1 × 4
Appendix
Table 6: Hyperparameters shared by every TRL DistillationTrainer run in the paper, reverse KL at β=1 , weight decay 0.01, gradient clip 1.0, the AdamW optimizer ( Loshchilov and Hutter, 2017 ) with β1=0.9 , β2=0.999 , ϵ=10−8 , and a linear learning-rate decay. Sampled evaluation uses temperature 0.6 with top- p 0.95 and top- k 20.
Fwd KL
Rev KL
Prefix, OPD
Prefix, OPSD
Updates
full params
full params
prefix, 64 tok.
prefix, 96 tok.
Imitation warm-up
none
none
300 steps
500 steps
KL direction
forward
reverse
forward
OPSD-style
KL support
top-64, clip 5
top-64, clip 5
top-64, clip 5
top-64, clip 5
Train rollout cap
16384
16384
4096
4096
Steps run, evaluated
400
800, at 500
400
400
Appendix
Table 7: Hyperparameters for the in-house engine behind the remaining four rows of Table 4 , all with a constant learning rate, no warmup, and the AdamW optimizer ( Loshchilov and Hutter, 2017 ) . Evaluation always uses a 16384-token cap regardless of the training rollout cap above. A shorter training cap is a property of the training setup only.
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.