Online Policy Distillation (OPD) often outperforms offline methods such as Supervised Fine-Tuning (SFT) by training on trajectories sampled directly from the student. However, standard OPD trains the student to follow the teacher at every state of the trajectory, assuming that teacher guidance is always beneficial. We instead distinguish between what the teacher prefers and what actually helps the current student succeed. We introduce Advantage-Based On-Policy Distillation (AOPD), which weights teacher actions by their counterfactual advantage under the student. Teacher actions that improve the student's expected outcome are reinforced, while those that do not are downweighted. Directly computing these advantages requires trying alternative teacher actions at each step and running new rollouts from each. Instead, we show that the AOPD objective can be estimated using only standard student rollouts and their terminal rewards. Our estimator uses teacher importance weights on the student-sampled actions to provide unbiased advantage-weighted supervision, without any additional rollouts or reward evaluations. This allows AOPD to focus teacher supervision on actions that help the current student while preserving the rollout efficiency of standard OPD. Experiments on math, coding, and Medqa show gains over distillation and RL baselines in both external-teacher and self-distillation settings.
Figures & tables
Figure 1: (left) The overall picture of our method. On the student sampled trajectories, not all the suggested actions by the teacher are useful to the student, and many suggestions are destructive. (middle) Performance comparison of our method with OPSD when training Qwen3-8B in non-thinking mode on Open-Thoughts. (right) Comparison with OPSD when training Qwen3-4B-Instruct on TACO for coding.
Figure 2: Training progress under external-teacher distillation with Qwen3-1.7B as the student. (Left/middle) Open-Thoughts evaluation reward and AIME24 performance. (Right) TACO evaluation reward for coding.
Math Training on Open-Thoughts
Code Training on TACO
OpenThoughts
AIME24
AMC
HMMT
HumanEval
MultiPL-E
Models
pass@1
pass@1
pass@1
pass@1
HE
HE+
pass@10
Avg. (7 lang.)
Qwen3-1.7B
56.6
14.7
40.1
5.8
68.3
61.6
82.5
31.4
+ SFT
68.0
23.3
53.5
12.6
69.5
62.3
82.4
31.0
+ GRPO
71.8
22.3
51.4
10.6
71.8
64.0
87.9
31.1
+ KDRL
72.8
25.0
55.8
12.1
71.5
66.7
84.4
31.0
Table 1: External-teacher distillation on math and coding with Qwen3-1.7B as the student. The teachers are Qwen3-4B-Instruct for math and Qwen3-8B for coding.
Qwen3-1.7B
Qwen3-8B
Method
OpenThoughts
AIME24
AMC
HMMT
OpenThoughts
AIME24
AMC
HMMT
pass@1
pass@1
pass@1
pass@1
pass@1
pass@1
pass@1
pass@1
Base
56.6
14.7
40.1
5.8
68.8
27.5
59.6
14.2
+ GRPO
71.8
22.3
51.4
10.6
78.5
30.6
63.8
13.6
+ OPSD
55.2
15.6
45.7
6.4
72.7
41.7
75.0
18.4
+ AOPD
70.2
35.0
61.4
14.2
84.6
52.1
78.6
20.8
Table 2: Math results for self-distillation on Open-Thoughts with Qwen3-1.7B and Qwen3-8B, both in non-thinking mode.
Model
Benchmark
Base
GRPO
OPSD
AOPD
Biology
Qwen2.5-7B-Instruct
SciKnowEval
20.0
42.0
41.0
54.0
MMLU-Pro
71.3
72.5
71.6
73.8
Qwen3-4B-Instruct
SciKnowEval
36.0
38.0
44.0
32.0
MMLU-Pro
84.5
84.1
84.9
84.8
Medical
Qwen2.5-7B-Instruct
MedQA
61.5
70.3
72.1
74.8
MedMCQA
56.5
64.5
66.6
69.8
Table 3: Self-distillation results across biology, medical question answering, and coding. AOPD outperforms GRPO and OPSD across all three domains.
Figure 3: Training progress for GRPO, OPSD, and AOPD in self-distillation. (Left/middle) Open-Thoughts evaluation reward for Qwen3-1.7B and Qwen3-8B. (Right) MedQA test accuracy after training on MedMCQA with Qwen2.5-7B-Instruct.
Teacher
Intervention position
Helpful (%)
Unchanged (%)
Harmful (%)
External
Random
6.9
85.1
8.0
Highest KL disagreement
9.6
80.6
9.8
Self-distilled
Random
8.6
83.4
8.0
Highest KL disagreement
9.0
81.4
9.6
Table 4: Effect of replacing a student token with the teacher’s preferred token under external-teacher and self-distillation settings. Interventions can both improve and harm the student’s final outcome.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Increasing the group size does not significantly improve the performance for AOPD. This is while for methods such GRPO, a larger group size is critical.
Figure 5: Clipping the importance sampling weight improves the stability and the performance of AOPD.
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherently off-policy for it, so the reliability of its supervision, and hence the source of the student's improvement, remains unclear. We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging to comparable performance regardless of whether noisy supervision is retained or removed. Does OPD distill at all? By analyzing what drives its gains, we find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones. This suggests that OPD works largely by suppressing low log-probability tokens, which requires no teacher. These findings motivate On-Policy Self-Adaptation (OPSA), a supervision-free method using entropy-adaptive negative advantages. It assigns stronger learning signals to high-entropy positions, suppressing tail tokens, and evenly redistributing probability mass among head tokens. Compared with the base Qwen3-1.7B, OPSA improves Avg@32 by 35.41 points on AIME24, corresponding to a 263% relative gain, and more than doubles Pass@32 across all three benchmarks. It also outperforms OPD by 16.77 points in Avg@32 on AIME24. Extensive experiments and analyses across model families and tasks further demonstrate its effectiveness and generalizability.
Yi Ding, Ruqi Zhang
Department of Computer Science, Purdue University, USA
On-policy distillation (OPD) is increasingly studied as a cheaper alternative to reinforcement learning, yet it remains a form of knowledge distillation (KD) that relies on teacher supervision. We revisit OPD from the KD perspective by asking where the teacher's advantage lies along student-generated trajectories. Controlled token flips at teacher-student disagreements directly measure the accuracy gained from teacher corrections. Single-token flips yield a modest average gain on selected failed trajectories, whereas repeated flips raise accuracy from 28.4% to 74.5%, close to the teacher's 76.9%. Gains continue to accumulate with more flips, but mass- and KL-based selection outperform random flips by 15-18 percentage points at counts of similar magnitude. Selective intervention remains useful after OPD, but greater argmax agreement does not consistently improve accuracy across model pairs. We advocate training schedules that learn more valuable teacher corrections while maintaining distributional stability. Controlled flips and fork-resolution analysis evaluate OPD beyond accuracy by measuring which teacher preferences the student acquires and what further correction can recover.
On-policy distillation (OPD) has become increasingly popular for transferring teacher capabilities to student models. In this work, we ask a critical research question: Is on-policy sampling always beneficial for distilling arbitrary teacher-student pairs? We show that a simple alternative, Semi-OPD, which distills from offline rollouts generated by the initial student, can often outperform OPD in both accuracy and training efficiency. Across 17 teacher-student pairs ranging from 1.5B to 235B parameters, Semi-OPD outperforms OPD in 14 cases, with up to +13.6% accuracy and 11.4x training speedup. We further find that the choice between OPD and Semi-OPD depends on the alignment between the initial teacher and student, quantified by an output-token overlap ratio: OPD is beneficial only when the two are highly aligned with high overlap ratios. Our deeper investigation suggests that effective distillation requires on-policyness w.r.t. both the student and the teacher. For misaligned pairs, student rollouts can become increasingly off-policy w.r.t. the teacher as context length grows, weakening the distillation signal. In contrast, Semi-OPD is often more stable, as it distills on shorter contexts while covering full trajectories and exposing the student to more teacher-preferred tokens. Beyond proposing Semi-OPD as an efficient alternative, our work motivates the community to rethink when to use OPD and to study stronger OPD variants with meaningful teacher-student pairs.