On-policy distillation (OPD) improves language-model post-training by letting the student learn from its own sampled trajectories under teacher supervision, aligning the training process with inference-time behavior. However, the rollout policy in OPD is usually treated as fixed: each round samples directly from the current student. We show that this choice can create a compounding teacher--student mismatch, because it requires the teacher to provide effective supervision on whichever trajectories the student samples, including ones less reachable by the teacher itself. On such trajectories, teacher supervision can become noisy or even misleading, leading to undesirable student patterns such as repetition. Moreover, since the updated student in turn defines the rollout policy of subsequent rounds, these patterns persist and potentially amplify over training. This failure mode suggests that improving OPD requires treating the rollout policy itself as an optimization target, rather than only modifying the distillation objective. We propose Mixture Proposal On-Policy Distillation (MPOPD), which derives a student-anchored, teacher-guided rollout distribution from a local trust-region objective based on relative teacher support. The resulting target is a geometric mixture of the student and teacher policies, and we introduce a proposal policy that approximates this mixture, enabling rollout generation in a single forward pass. Experiments on 16 math, knowledge, and medical reasoning benchmarks show consistent gains over OPD and recent variants. On math reasoning, our method improves average accuracy by 6.91 over OPD on Qwen3-4B-Base while reducing average generation length by 41%, suggesting that rollout policy design provides an effective way to improve OPD.
Figures & tables
Figure 1: Teacher–student mismatch compounds under OPD . Upper : 10th-percentile similarity between the two completions drops by 0.24 from its peak, while our method remains stable. Lower : fraction of prompts on which the former scores >0.10 (solid) or >0.20 (dashed) below the latter in PRM score; the former reaches ∼24% by the end of training, while our method keeps it near zero.
Figure 2: Overview of MPOPD . Left: Standard on-policy distillation suffers from compounding teacher–student mismatch, where student rollouts continuously drift beyond the teacher-reachable region and receive unreliable supervision. Right: MPOPD derives a tilted target as a geometric mixture of the student and teacher, and trains a proposal policy to approximate it.
Algorithm 1 Mixture Proposal for On-Policy Distillation
Qwen3-1.7B-Base
Benchmark
Base
OPD
ExOPD
EOPD
VETO
MPOPD
Imp.
AMC23
13.05 / 6.96k
39.22 / 16.24k
40.00 / 16.37k
41.02 / 16.16k
39.14 / 10.30k
41.09 / 9.78k
+0.07 / ↓ 39.5%
AIME24
1.67 / 8.00k
11.46 / 16.38k
12.92 / 16.38k
11.56 / 16.35k
10.62 / 14.90k
12.50 / 14.65k
− 0.42 / ↓ 10.6%
AIME25
1.15 / 7.66k
7.19 / 16.36k
7.29 / 16.38k
8.65 / 16.29k
7.40 / 15.06k
8.75 / 14.79k
+0.10 / ↓ 9.2%
AIME26
0.73 / 7.73k
7.71 / 16.33k
7.71 / 16.37k
7.81 / 16.31k
7.92 / 15.00k
9.06 / 14.85k
+1.14 / ↓ 1.0%
H25F
0.21 / 7.82k
3.12 / 16.32k
2.92 / 16.37k
3.12 / 16.32k
2.81 / 15.50k
3.65 / 15.48k
+0.53 / ↓ 5.2%
Table 1: Evaluation results on math benchmarks . Each cell reports accuracy (%) and average generation length . Imp. compares MPOPD with the best baseline in each row, showing the accuracy gain in percentage points and the relative length change. Best and second-best accuracies within each model block are bolded and underlined, respectively.
Knowledge Question Answering
Benchmark
Base
OPD
ExOPD
EOPD
VETO
MPOPD
Imp.
GPQA-D
7.32 / 8.22k
38.51 / 16.37k
40.03 / 16.38k
41.60 / 16.35k
40.47 / 5.78k
42.17 / 6.34k
+0.57 / ↓ 61.3%
ARC-C
24.82 / 3.92k
88.52 / 8.18k
85.38 / 8.19k
87.95 / 8.17k
91.45 / 0.33k
92.37 / 0.32k
+0.92 / ↓ 5.1%
MMLU-P
9.30 / 5.58k
62.45 / 8.19k
63.75 / 8.20k
62.82 / 8.15k
64.10 / 1.42k
65.08 / 1.31k
+0.98 / ↓ 7.2%
Avg.
13.81 / 5.91k
63.16 / 10.91k
63.05 / 10.92k
64.12 / 10.89k
65.34 / 2.51k
66.54 / 2.65k
+1.20 / ↑ 5.7%
Medical Reasoning
Table 2: Evaluation results on knowledge QA and medical reasoning benchmarks on Qwen3-4B-Base. Each cell reports accuracy (%) and average generation length . Imp. compares MPOPD with the best baseline in each row, showing the accuracy gain in percentage points and the relative length change. Best and second-best accuracies within each section are bolded and underlined, respectively.
Figure 6
Figure 5: Training diagnostics. We compare standard OPD and our tilted rollout proposal over training. (a) Training reward. (b) Fraction of responses clipped by the maximum generation length. (c) Average teacher log-probability on generated tokens. (d) Student–teacher entropy gap. (e) Fraction of token-level loss from low-confidence teacher positions.
η
AIME26
H25N
Minerva
Avg.
0.10
23.33
15.10
42.36
26.93
0.15
24.17
15.42
42.65
27.41
0.20
25.00
15.42
42.65
27.69
0.30
24.69
15.31
42.43
27.48
Table 4: Effect of the coefficient η on selected Qwen3-4B-Base math benchmarks. Scores are reported as accuracy (%).
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Math
Knowledge QA
Medical
Batch size
256
256
128
PPO mini-batch
128
128
64
Training data
full split
full split
5K subset
Learning rate
3×10−6
Weight decay
0.1
Train temperature
0.8
Appendix
Table 5: Training configuration across domains. Settings shared across all domains are marked with “–”.
(a) Accuracy at different decoding budgets
Method
Acc@4K
Acc@8K
Acc@16K
Teacher
31.60
46.56
50.16
OPD
21.89
25.32
28.46
VETO
24.52
28.29
31.06
MPOPD
26.89
31.42
35.37
Appendix
Table 6: Decoding budgets and termination behavior. Results are macro-averaged over eight math benchmarks. Accuracy and rates are percentages; K denotes thousands of tokens. In (a), the best and second-best student accuracies are bolded and underlined, respectively.
(a) Approximation to the tilted target
Step
KL(q∗∥q^)
KL(q^∥q∗)
ρ top-16
ρ full vocab.
20
0.012299
0.012704
0.880
0.704
40
0.009787
0.009905
0.862
0.746
60
0.009007
0.009122
0.862
0.820
80
0.009032
0.009111
0.888
0.763
100
0.008369
0.008462
0.882
0.789
Appendix
Table 7: Proposal approximation and effective KL radius. (a) Mean approximation KLs on the teacher top-16 support and rank correlations of displacement from the student. (b) Percentiles of KL(q∗∥π) on the full vocabulary.
Figure 6: Distillation with a larger teacher and student. Bars show macro-average accuracy (%) over eight math benchmarks after two training epochs with Qwen3-30B-A3B-Instruct-2507 as the teacher. “Ours” denotes MPOPD .
k
AIME24
AIME25
AIME26
AMC23
H25F
H25N
Minerva
Olymp.
Avg.
4
27.19
27.60
23.75
66.95
14.69
15.21
42.35
55.36
34.14
16
28.65
28.96
25.00
69.61
15.83
15.42
42.65
56.82
35.37
32
30.00
28.65
24.90
69.84
14.79
14.90
42.68
56.88
35.33
128
29.48
28.96
24.58
70.39
15.62
15.94
42.70
57.60
35.66
Appendix
Table 8: Impact of the teacher support size. Scores are accuracy (%); Avg. is the macro-average over eight math benchmarks. Best and second-best accuracies are bolded and underlined, respectively.
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.