On-policy distillation (OPD) improves language-model post-training by letting the student learn from its own sampled trajectories under teacher supervision, aligning the training process with inference-time behavior. However, the rollout policy in OPD is usually treated as fixed: each round samples directly from the current student. We show that this choice can create a compounding teacher--student mismatch, because it requires the teacher to provide effective supervision on whichever trajectories the student samples, including ones less reachable by the teacher itself. On such trajectories, teacher supervision can become noisy or even misleading, leading to undesirable student patterns such as repetition. Moreover, since the updated student in turn defines the rollout policy of subsequent rounds, these patterns persist and potentially amplify over training. This failure mode suggests that improving OPD requires treating the rollout policy itself as an optimization target, rather than only modifying the distillation objective. We propose Mixture Proposal On-Policy Distillation (MPOPD), which derives a student-anchored, teacher-guided rollout distribution from a local trust-region objective based on relative teacher support. The resulting target is a geometric mixture of the student and teacher policies, and we introduce a proposal policy that approximates this mixture, enabling rollout generation in a single forward pass. Experiments on 16 math, knowledge, and medical reasoning benchmarks show consistent gains over OPD and recent variants. On math reasoning, our method improves average accuracy by 6.91 over OPD on Qwen3-4B-Base while reducing average generation length by 41%, suggesting that rollout policy design provides an effective way to improve OPD.
Figures & tables
Figure 1: Teacher–student mismatch compounds under OPD . Upper : 10th-percentile similarity between the two completions drops by 0.24 from its peak, while our method remains stable. Lower : fraction of prompts on which the former scores >0.10 (solid) or >0.20 (dashed) below the latter in PRM score; the former reaches ∼24% by the end of training, while our method keeps it near zero.
Figure 2: Overview of MPOPD . Left: Standard on-policy distillation suffers from compounding teacher–student mismatch, where student rollouts continuously drift beyond the teacher-reachable region and receive unreliable supervision. Right: MPOPD derives a tilted target as a geometric mixture of the student and teacher, and trains a proposal policy to approximate it.
Algorithm 1 Mixture Proposal for On-Policy Distillation
Qwen3-1.7B-Base
Benchmark
Base
OPD
ExOPD
EOPD
VETO
MPOPD
Imp.
AMC23
13.05 / 6.96k
39.22 / 16.24k
40.00 / 16.37k
41.02 / 16.16k
39.14 / 10.30k
41.09 / 9.78k
+0.07 / ↓ 39.5%
AIME24
1.67 / 8.00k
11.46 / 16.38k
12.92 / 16.38k
11.56 / 16.35k
10.62 / 14.90k
12.50 / 14.65k
− 0.42 / ↓ 10.6%
AIME25
1.15 / 7.66k
7.19 / 16.36k
7.29 / 16.38k
8.65 / 16.29k
7.40 / 15.06k
8.75 / 14.79k
+0.10 / ↓ 9.2%
AIME26
0.73 / 7.73k
7.71 / 16.33k
7.71 / 16.37k
7.81 / 16.31k
7.92 / 15.00k
9.06 / 14.85k
+1.14 / ↓ 1.0%
H25F
0.21 / 7.82k
3.12 / 16.32k
2.92 / 16.37k
3.12 / 16.32k
2.81 / 15.50k
3.65 / 15.48k
+0.53 / ↓ 5.2%
Table 1: Evaluation results on math benchmarks . Each cell reports accuracy (%) and average generation length . Imp. compares MPOPD with the best baseline in each row, showing the accuracy gain in percentage points and the relative length change. Best and second-best accuracies within each model block are bolded and underlined, respectively.
Knowledge Question Answering
Benchmark
Base
OPD
ExOPD
EOPD
VETO
MPOPD
Imp.
GPQA-D
7.32 / 8.22k
38.51 / 16.37k
40.03 / 16.38k
41.60 / 16.35k
40.47 / 5.78k
42.17 / 6.34k
+0.57 / ↓ 61.3%
ARC-C
24.82 / 3.92k
88.52 / 8.18k
85.38 / 8.19k
87.95 / 8.17k
91.45 / 0.33k
92.37 / 0.32k
+0.92 / ↓ 5.1%
MMLU-P
9.30 / 5.58k
62.45 / 8.19k
63.75 / 8.20k
62.82 / 8.15k
64.10 / 1.42k
65.08 / 1.31k
+0.98 / ↓ 7.2%
Avg.
13.81 / 5.91k
63.16 / 10.91k
63.05 / 10.92k
64.12 / 10.89k
65.34 / 2.51k
66.54 / 2.65k
+1.20 / ↑ 5.7%
Medical Reasoning
Table 2: Evaluation results on knowledge QA and medical reasoning benchmarks on Qwen3-4B-Base. Each cell reports accuracy (%) and average generation length . Imp. compares MPOPD with the best baseline in each row, showing the accuracy gain in percentage points and the relative length change. Best and second-best accuracies within each section are bolded and underlined, respectively.
Figure 6
Figure 5: Training diagnostics. We compare standard OPD and our tilted rollout proposal over training. (a) Training reward. (b) Fraction of responses clipped by the maximum generation length. (c) Average teacher log-probability on generated tokens. (d) Student–teacher entropy gap. (e) Fraction of token-level loss from low-confidence teacher positions.
η
AIME26
H25N
Minerva
Avg.
0.10
23.33
15.10
42.36
26.93
0.15
24.17
15.42
42.65
27.41
0.20
25.00
15.42
42.65
27.69
0.30
24.69
15.31
42.43
27.48
Table 4: Effect of the coefficient η on selected Qwen3-4B-Base math benchmarks. Scores are reported as accuracy (%).
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Math
Knowledge QA
Medical
Batch size
256
256
128
PPO mini-batch
128
128
64
Training data
full split
full split
5K subset
Learning rate
3×10−6
Weight decay
0.1
Train temperature
0.8
Appendix
Table 5: Training configuration across domains. Settings shared across all domains are marked with “–”.
(a) Accuracy at different decoding budgets
Method
Acc@4K
Acc@8K
Acc@16K
Teacher
31.60
46.56
50.16
OPD
21.89
25.32
28.46
VETO
24.52
28.29
31.06
MPOPD
26.89
31.42
35.37
Appendix
Table 6: Decoding budgets and termination behavior. Results are macro-averaged over eight math benchmarks. Accuracy and rates are percentages; K denotes thousands of tokens. In (a), the best and second-best student accuracies are bolded and underlined, respectively.
(a) Approximation to the tilted target
Step
KL(q∗∥q^)
KL(q^∥q∗)
ρ top-16
ρ full vocab.
20
0.012299
0.012704
0.880
0.704
40
0.009787
0.009905
0.862
0.746
60
0.009007
0.009122
0.862
0.820
80
0.009032
0.009111
0.888
0.763
100
0.008369
0.008462
0.882
0.789
Appendix
Table 7: Proposal approximation and effective KL radius. (a) Mean approximation KLs on the teacher top-16 support and rank correlations of displacement from the student. (b) Percentiles of KL(q∗∥π) on the full vocabulary.
Figure 6: Distillation with a larger teacher and student. Bars show macro-average accuracy (%) over eight math benchmarks after two training epochs with Qwen3-30B-A3B-Instruct-2507 as the teacher. “Ours” denotes MPOPD .
k
AIME24
AIME25
AIME26
AMC23
H25F
H25N
Minerva
Olymp.
Avg.
4
27.19
27.60
23.75
66.95
14.69
15.21
42.35
55.36
34.14
16
28.65
28.96
25.00
69.61
15.83
15.42
42.65
56.82
35.37
32
30.00
28.65
24.90
69.84
14.79
14.90
42.68
56.88
35.33
128
29.48
28.96
24.58
70.39
15.62
15.94
42.70
57.60
35.66
Appendix
Table 8: Impact of the teacher support size. Scores are accuracy (%); Avg. is the macro-average over eight math benchmarks. Best and second-best accuracies are bolded and underlined, respectively.
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
In on-policy distillation (OPD), the student determines which states receive teacher supervision. As its policy evolves, earlier response prefixes become less likely even though teacher-student disagreement on them persists. Under matched trajectory and optimization budgets, neither more queries nor more frequent rollout resampling is uniformly beneficial. Current-policy rollouts outperform initial-policy rollouts at shorter response budgets, but this ranking reverses at longer budgets. Replaying initial-policy rollouts after current-policy training improves accuracy, whereas replaying fixed recent rollouts does not reproduce the gain. We propose R-OPD, a gradient-triggered curriculum that adaptively selects when to revisit initial-student trajectories. When changes in mean gradients fall within minibatch-level variation for two consecutive comparisons, training switches from the next iteration onward to initial-policy replay. Across eight mathematics benchmarks and three training-data orders, R-OPD improves average accuracy over continued current-policy sampling by 2.25/4.44 percentage points at 16K/32K for a 0.6B student and 2.81/6.15 points for a 1.7B student. With a 30B-A3B teacher, it improves 8B accuracy by 4.05 points at 32K. At 32K, R-OPD exceeds a fixed schedule of 40 current-policy updates followed by 20 replay updates by 1.39/1.66/1.85 points for 0.6B/1.7B/8B. At the same generation cap, R-OPD also produces longer responses, suggesting that well-timed revisits help students use more of their reasoning capacity.
Large language models are often post-trained with sparse verifier rewards, which indicate whether a sampled trajectory succeeds but provide limited guidance about where reasoning succeeds or fails. On-policy distillation (OPD) offers denser token-level supervision by training on student-generated trajectories, yet existing methods typically distill each rollout independently and ignore the other attempts sampled for the same prompt. We introduce Multi-Rollout On-Policy Distillation (MOPD), a peer-conditioned distillation framework that uses the student's local rollout group to construct more informative teacher signals. MOPD conditions the teacher on both successful and failed peer rollouts: successes provide positive evidence for valid reasoning patterns, while failures provide structured negative evidence about plausible mistakes to avoid. We study two peer-context constructions: positive peer imitation and contrastive success-failure conditioning. Experiments on competitive programming, mathematical reasoning, scientific question answering, and tool-use benchmarks show that MOPD consistently improves over standard on-policy baselines. Further teacher-signal analysis shows that mixed success-failure contexts better align teacher scores with verifier rewards, indicating that the gains arise from more faithful, instance-adaptive supervision. These results indicate that effective on-policy distillation should exploit the student's multi-rollout trial-and-error behavior rather than treating rollouts as isolated samples.
Weichen Yu, Xiaomin Li, Yizhou Zhao +8
1Carnegie Mellon University · 2Microsoft · 3Purdue University
On-policy distillation (OPD) has been widely studied as a post-training method in which a student model obtains token-level supervision from a stronger teacher on its own rollouts. Recent studies have improved OPD through alternative distillation reward formulations and teacher configurations, while the objective of distillation remains centered on mimicking the teacher. However, when a stronger teacher has limited distributional overlap with the student, such positive guidance can provide insufficient learning signals. In this work, we introduce Negative-Policy OPD (NP-OPD), which complements teacher supervision with rollouts from a lower-performing, lower-capability negative policy that serves as a negative reference for the student. Rather than modifying the distillation reward formulation, NP-OPD introduces the negative policy at the rollout stage, continuously supplying tokens preferred by the negative policy over the teacher so that they remain exposed to teacher supervision throughout training. This provides an explicit negative signal through negative-policy rollouts while preserving the positive teacher supervision used in OPD. Through extensive experiments, we show that NP-OPD improves OPD across model scales, generation modes, reasoning domains, and different OPD variants. Furthermore, our analyses show that NP-OPD effectively suppresses tokens preferred by the negative policy over the teacher and moves the student away from the negative policy. These results support our design of introducing negative signals through negative-policy rollouts and provide new insight into the role of the rollout policy in OPD. Code will be available at https://github.com/naver-ai/np-opd.