Organizations: Department of Computer Sciences, University of Wisconsin–Madison · Machine Learning Research, Morgan Stanley · Department of Electrical and Computer Engineering, Rutgers University–New Brunswick
On-policy distillation (OPD) trains a student model on its self-generated trajectories with dense token-level teacher feedback. However, naive OPD may suffer from teacher unanchoring, where the student's reasoning trajectory drifts far from the teacher, causing the teacher to be queried on states it would hardly visit and thus provide unreliable supervision. We propose Interactive-Policy Distillation (IPD), which applies adaptive teacher intervention to the student rollout. Under a bidirectional propose-and-verify state machine, the student and teacher alternately exchange their roles as proposer and verifier, and collaboratively generate mixed-source trajectories. Then different supervisions are applied according to the source of each token. This bidirectional propose-and-verify mechanism and the source-split loss make IPD not only a more performant distillation method, but also a unified bridge between on-policy and off-policy paradigms. To make the interleaved dual-model rollouts more efficient, we also design a dedicated fused inference engine that co-hosts both models in one serving instance with separate KV caches and instantiates the state machine model to distribute, collect, and process requests. On math reasoning tasks and across multiple teacher-student model pairs, student models trained with IPD not only outperform those trained with OPD, but also demonstrate higher data efficiency. Specifically, when distilling Qwen3-30B-A3B into Qwen3-1.7B-Base, IPD brings a +3.28 mean@8 and a +3.28 best@8 benchmark-averaged accuracy improvement compared with OPD. Besides, IPD only consumes about 1/4 of the training examples and steps to outperform OPD trained on the whole training dataset for one epoch. We also investigate the impact of different loss variants and takeover / handback configurations, and demonstrate the robustness of IPD on different training data.
Figures & tables
Figure 1: The rollout and training demonstration of IPD. The student and teacher collaboratively generate the rollout under a bidirectional propose-and-verify state machine. Then tokens are grouped by their sources, and the IPD source-split loss is applied to update the student weights.
Teacher/ Student
Method
MATH-500
Minerva
Olympiad
AMC23
AIME24
AIME25
Avg.
mean@8
best@8
mean@8
best@8
mean@8
best@8
mean@8
best@8
mean@8
best@8
mean@8
best@8
mean@8
best@8
Qwen3-4B/ Qwen3-1.7B-Base
OPD
66.68
84.06
26.42
44.04
31.00
48.45
35.69
57.29
7.92
21.77
2.92
8.70
28.44
44.05
SKD
66.70
84.50
27.94
43.69
29.75
49.52
35.09
52.90
8.75
17.41
4.58
10.73
28.80
43.13
IPD (ours)
69.28
84.59
29.60
46.26
31.80
49.63
36.14
57.25
9.17
21.69
6.67
15.06
30.44
45.75
Qwen3-30B-A3B/ Qwen3-1.7B-Base
OPD
63.35
83.54
23.25
41.43
28.14
46.88
32.68
54.39
10.42
18.84
5.83
13.12
27.28
43.03
SKD
67.38
84.34
26.70
44.69
30.39
49.73
33.89
54.67
8.75
22.36
5.83
14.92
28.82
45.12
Table 1: Comparison of different distillation methods across various student and teacher model combinations. We report mean@8 and best@8 accuracy (%) on different math reasoning benchmarks, as well as the benchmark-wise average.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
MATH-500
Minerva
Olympiad
AMC23
AIME24
AIME25
Avg.
Loss Variant
mean@8
best@8
mean@8
best@8
mean@8
best@8
mean@8
best@8
mean@8
best@8
mean@8
best@8
mean@8
best@8
Pure Reverse KL
68.23
84.05
29.18
46.21
32.13
49.70
35.39
56.46
10.42
27.24
4.17
11.56
29.92
45.87
Pure Forward KL
66.80
83.64
28.77
45.15
30.46
49.69
34.19
58.58
6.67
18.41
4.17
11.60
28.51
44.51
Inverse-IPD-Split KL
67.35
84.84
27.16
44.06
31.37
50.31
33.58
56.63
9.17
22.66
6.67
15.09
29.22
45.60
IPD-Split KL (default)
69.28
84.59
29.60
46.26
31.80
49.63
36.14
57.25
9.17
21.69
6.67
15.06
30.44
45.75
Appendix
Table 2: Benchmark-wise mean@8 and best@8 accuracy for the KL-direction ablation.
MATH-500
Minerva
Olympiad
AMC23
AIME24
AIME25
Avg.
Coverage width
mean@8
best@8
mean@8
best@8
mean@8
best@8
mean@8
best@8
mean@8
best@8
mean@8
best@8
mean@8
best@8
K=4
68.30
85.36
28.49
43.59
31.97
49.76
35.24
53.75
10.00
24.21
8.75
18.17
30.46
45.80
K=8
69.20
84.94
28.95
45.00
32.45
50.97
33.43
54.60
8.75
17.68
6.25
14.10
29.84
44.55
K=16 (default)
69.28
84.59
29.60
46.26
31.80
49.63
36.14
57.25
9.17
21.69
6.67
15.06
30.44
45.75
K=32
68.03
83.81
28.77
44.66
32.50
50.50
34.64
53.96
8.75
17.58
5.42
14.30
29.68
44.13
Appendix
Table 3: Benchmark-wise mean@8 and best@8 accuracy for the top- K coverage ablation.
Table 4: Same-origin distillation experiment with DeepSeek-R1-Distill-Qwen-1.5B as the student and JustRL-DeepSeek-1.5B as the teacher on DAPO-Math-17K-Processed. Intervention methods including IPD and SKD do not bring as statistically significant improvement over OPD compared with our main experiments with different-size Qwen3 models.
Off-policy and on-policy distillation have traditionally been formulated as separate paradigms, each favoring a different property of distillation trajectories. Teacher-generated (off-policy) traces are typically high-quality but lie far from the student's distribution, whereas student-generated (on-policy) rollouts are more learnable but often contain erroneous reasoning. We view these paradigms as the endpoints of a policy continuum and posit that a more effective rollout policy may lie in between. We introduce \textbf{Interpolated Policy Distillation (IPD)}, which defines the next-token distribution at every decoding step as an explicit linear interpolation between the student and teacher distributions. The interpolation operates at the distribution level, token by token, and its coefficient provides direct control over the balance between trajectory quality and student learnability. Naively sampling from this policy would require sequentially querying the teacher at every token and is thus expensive. To make IPD practical, we accelerate it with a new speculative-decoding rule while exactly preserving the interpolated next-token distribution.At the trajectory level, the resulting rollouts naturally interleave student- and teacher-generated segments. Unlike recent heuristic segment-interleaving methods, however, this interleaving is induced by an exactly realized token-level interpolated policy rather than by hand-designed switching rules. Across text-only and multimodal reasoning benchmarks, IPD consistently outperforms both endpoint policies (SFT and OPD), their conventional two-stage combination (SFT-then-OPD), and recent heuristic segment-interleaving methods, demonstrating that token-level policy interpolation better balances trajectory quality and student learnability.
Youxu Shi, Yifan Sun, Dacheng Yin +5
University of Science and Technology of China · WeChat Vision, Tencent Inc.
On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher. However, we identify a critical bottleneck, \textbf{Supervision Fidelity Decay (SFD)}: as student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative. Consequently, the teacher-dependent corrective signal in reverse-KL distillation weakens, causing student drift to compound across long reasoning chains. To mitigate SFD, we introduce \textbf{Lookahead Group Reward (\ours{})}. Building on the insight that next-step teacher confidence reflects the discriminative strength of future reverse-KL supervision, \ours{} evaluates the student's top-K candidate tokens by the teacher confidence they induce at the subsequent step and assigns a group-normalized reward. To maintain computational efficiency, we further design an entropy-triggered tree-attention mechanism. Across six math and code benchmarks, \ours{} improves mean@8 by \textbf{2.57} points over OPD for a 7B student, with gains increasing in longer-generation and reaching +\textbf{4.92} points on AIME-26 at 39k tokens.
Yanjiang Liu, Jie Lou, Xinyan Guan +7
University of Chinese Academy of Sciences · 2Chinese Information Processing Laboratory Institute of Software, Chinese Academy of Sciences 2University of Chinese Academy of Sciences, Beijing, China · 3Xiaohongshu
On-policy distillation is a powerful way to transfer reasoning ability from a strong teacher to a smaller student: the student samples trajectories from its own policy, and the teacher provides dense token-level supervision on the states the student actually visits. However, this supervision is not always reliable: a teacher can assign high likelihood to plausible but incorrect solutions, or low likelihood to correct student solutions that follow different reasoning paths. Unconditionally distilling the teacher can therefore reinforce bad modes or erase useful student behavior. To address these limitations, we introduce RG-OPD: Reward-Gated On-Policy Distillation that uses verifier feedback to decide when teacher logits should be trusted. RG-OPD bridges sparse verifier rewards and dense teacher logits, preserving token-level supervision while filtering misleading teacher signals. Across reasoning and coding benchmarks, RG-OPD produces stronger distilled students, outperforming both vanilla reverse-KL distillation and the recent TSD-KD baseline. At 1K generation length, RG-OPD improves over reverse-KL by 2.9 points and over TSD-KD by 4.9 points; in the long-generation setting, it improves over the untuned student by 8.2 points. Our code is available at https://github.com/UoC-tail/RG-OPD.
Mohammad Sadegh Akhondzadeh, Vijay Lingam, Atula Tejaswi +3