On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher, but providing such supervision for every rollout requires substantial teacher computation. We introduce Success-Referenced On-Policy Distillation (SR-OPD), which reduces this cost by selecting which prompts and rollouts receive teacher supervision. When the student produces both successful and failed rollouts for the same prompt, a successful rollout can serve as a natural reference for selecting failed rollouts. SR-OPD therefore focuses on such prompts and prioritizes failed rollouts whose hidden-state trajectories show sustained divergence from a successful reference, while accounting for estimated teacher-input cost. Across three teacher-student pairs and six mathematical reasoning benchmarks, SR-OPD uses only 3.46-5.02% of the teacher-input tokens required by Vanilla OPD in the one-pass setting while maintaining comparable reasoning performance. Under a controlled setting matched to 5% of Vanilla OPD's teacher-input budget, further experiments support both key design choices: focusing supervision on prompts with both successful and failed rollouts, and using successful rollouts to guide failure selection. These results indicate that a student's own successful behavior can serve as a useful reference for allocating teacher supervision under a fixed teacher-input budget.
Figures & tables
Figure 1: Two-stage pre-query routing in SR-OPD. Left: prompt routing retains prompts containing both successful and failed rollouts while skipping all-success and all-failure prompts. Right: a same-prompt success serves only as a student-side reference; persistent-departure area (PDA) and teacher-input cost rank failed rollouts. In this example, only r3 receives frozen-teacher supervision and the standard OPD update.
Pair
Method
AIME24
AIME25
AMC23
HMMT24
HMMT25
MATH-500
Macro Avg
Input (M; ratio)
Qwen
Vanilla OPD
34.31
19.44
61.30
11.60
13.96
84.30
37.48
374.10 (1.00)
TA-OPD
29.72
21.39
61.35
14.31
13.75
83.49
37.34
417.90 (1.12)
TLR-OPD
17.57
15.35
46.98
14.10
12.08
75.57
30.27
19.20 (0.05)
SR-OPD
34.17
19.58
61.56
12.08
13.68
84.21
37.55
13.60 (0.04)
Skywork
Vanilla OPD
36.74
26.67
67.60
15.14
19.24
85.89
41.88
519.80 (1.00)
TA-OPD
35.07
26.81
66.72
14.93
17.22
85.73
41.08
516.20 (0.99)
Table 1: Native operating points. Benchmark columns report accuracy over 16 generation seeds, averaged across three training seeds. Macro Avg is the unweighted mean across six benchmarks for each training seed, then averaged across training seeds. Input is mean serialized teacher-input tokens in millions; parenthesized ratios are decimal values relative to Vanilla OPD in the same pair. Bold marks the highest observed benchmark and Macro Avg mean within each pair and the lowest teacher input.
Figure 2: Controlled routing and mechanism analyses. (a) Absolute Macro Avg for Global+Random and Mixed+Random; each model has its own horizontal score scale. (b) Absolute Macro Avg for the four Mixed-gate selectors, with a separate local scale per model; R, T, S, and P denote Random, TLR, Static SR, and PDA. Paired differences and intervals are in Appendix A.7 . (c) Independent component effects shown as treatment-minus-control paired means and pointwise 95% intervals. (d) Spearman correlations between the Full SR score and three post-query diagnostics; darker cells indicate stronger association. Definitions and numerical results are in Appendix A.5 for (a), Appendices A.6 – A.7 for (b), Appendix A.8 for (c), and Appendix A.9 for (d).
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Pair
Range (%)
Qwen
4.98–5.02
Skywork
4.98–5.01
Granite
4.99–5.02
Appendix
Table A1: Realized teacher-input use across the four mixed-gate selectors, as a percentage of the corresponding Vanilla OPD input.
Pair
Global+Random
Mixed+Random
Difference
95% CI
Qwen
36.14
36.55
+0.410
[+0.213,+0.607]
Skywork
40.02
40.49
+0.467
[+0.265,+0.669]
Granite
14.05
14.39
+0.338
[+0.130,+0.546]
Appendix
Table A2: Controlled prompt routing at the matched 5% teacher-input budget. Effects are paired Mixed+Random minus Global+Random Macro Avg.
Pair
Method
AIME24
AIME25
AMC23
HMMT24
HMMT25
MATH-500
Macro Avg
Qwen
Mixed+Random
30.90
21.39
58.70
12.71
14.72
80.88
36.55
Mixed+TLR
30.97
21.39
59.38
13.13
15.14
80.89
36.81
Mixed+StaticSR
31.11
21.60
59.53
13.26
15.21
81.14
36.98
Mixed+PDA (SR-OPD)
31.11
21.74
59.95
13.61
15.49
81.10
37.17
Skywork
Mixed+Random
35.49
26.04
65.47
13.75
17.15
85.05
40.49
Mixed+TLR
35.83
26.46
65.63
14.17
17.71
85.13
40.82
Appendix
Table A3: Controlled rollout-routing results at the matched 5% teacher-input budget.
Pair
Comparator
Difference
95% CI
Qwen
Mixed+Random
+0.616
[+0.241,+0.991]
Mixed+TLR
+0.352
[−0.305,+1.008]
Mixed+StaticSR
+0.191
[−0.514,+0.895]
Skywork
Mixed+Random
+0.738
[+0.651,+0.825]
Mixed+TLR
+0.408
[+0.255,+0.561]
Mixed+StaticSR
+0.262
[−0.372,+0.895]
Appendix
Table A4: Paired Mixed+PDA minus comparator effects on Macro Avg. Intervals use paired training seeds 41–43.
Comparison
Pair
Difference
95% CI
Same-Prompt − Matched Other-Prompt
Qwen
+0.173
[−0.293,+0.639]
Skywork
+0.076
[−0.159,+0.310]
Granite
+0.154
[−0.184,+0.493]
Same-Prompt − Shuffled Other-Prompt
Qwen
+0.262
[−0.098,+0.621]
Skywork
+0.180
[−0.129,+0.488]
Granite
+0.160
[−0.156,+0.477]
Appendix
Table A5: Independent paired component ablations. Differences are treatment minus control Macro Avg; intervals are pointwise 95%.
Pair
Signal
ULR
Ugrad
Uval
Qwen
TLR
0.071
0.082
0.162
AvgDistance
0.101
0.112
0.126
PDA
0.271
0.301
0.314
Full SR
0.251
0.412
0.196
Skywork
TLR
0.104
0.143
0.151
AvgDistance
0.116
0.071
0.181
Appendix
Table A6: Spearman correlations between pre-query routing signals and post-query diagnostic proxies (200 failed items per pair). AvgDistance is a diagnostic mean-distance score, not the StaticSR training ablation.
Pair
Method
K=8
K=16
Change [95% CI]
Qwen
Mixed+Random
36.49
36.53
+0.040[−0.072,+0.152]
Cost-only
36.85
36.90
+0.050[−0.049,+0.149]
Frontier-TLR
36.79
36.82
+0.030[−0.082,+0.142]
Static SR
36.99
37.05
+0.060[−0.052,+0.172]
SR-OPD
37.16
37.32
+0.160[+0.036,+0.284]
Skywork
Mixed+Random
40.39
40.48
+0.090[−0.022,+0.202]
Appendix
Table A7: Rollout-count sensitivity at the nominal 3% cap. The final column is the paired K=16 minus K=8 change.
Pair
Method
K=8 input (M)
K=8 / Vanilla (%)
K=16 input (M)
K=16 / Vanilla (%)
Qwen
Mixed+Random
11.350
3.034
11.375
3.041
Cost-only
11.130
2.975
11.114
2.971
Frontier-TLR
11.306
3.022
11.333
3.030
Static SR
11.207
2.996
11.242
3.005
SR-OPD
11.207
2.996
11.215
2.998
Skywork
Mixed+Random
15.551
2.992
15.590
2.999
Appendix
Table A8: Realized teacher-input usage for the rollout-count sensitivity experiment. Input is serialized teacher-input tokens in millions; percentages are relative to the corresponding Vanilla OPD teacher-input mean.
Pair
Audit error [95% CI]
0% flip
5% flip
10% flip
Qwen
1.50% [0.69,3.23]
37.17
37.05
36.84
Skywork
2.25% [1.19,4.22]
41.23
41.07
40.83
Granite
1.00% [0.39,2.54]
14.98
14.87
14.72
Appendix
Table A9: Verifier reliability: audit error among 400 items per pair with 95% Wilson intervals, and Macro Avg under random label flips. The 0% column is the unperturbed controlled SR-OPD baseline.
On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher. However, we identify a critical bottleneck, Supervision Fidelity Decay (SFD): as student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative. Consequently, the teacher-dependent corrective signal in reverse-KL distillation weakens, causing student drift to compound across long reasoning chains. To mitigate SFD, we introduce Lookahead Group Reward (\ours{}). Building on the insight that next-step teacher confidence reflects the discriminative strength of future reverse-KL supervision, \ours{} evaluates the student's top-K candidate tokens by the teacher confidence they induce at the subsequent step and assigns a group-normalized reward. To maintain computational efficiency, we further design an entropy-triggered tree-attention mechanism. Across six math and code benchmarks, \ours{} improves mean@8 by 2.57 points over OPD for a 7B student, with gains increasing in longer-generation and reaching +4.92 points on AIME-26 at 39k tokens.
Yanjiang Liu, Jie Lou, Xinyan Guan +7
University of Chinese Academy of Sciences · 2Chinese Information Processing Laboratory Institute of Software, Chinese Academy of Sciences 2University of Chinese Academy of Sciences, Beijing, China · 3Xiaohongshu
On-policy distillation is a powerful way to transfer reasoning ability from a strong teacher to a smaller student: the student samples trajectories from its own policy, and the teacher provides dense token-level supervision on the states the student actually visits. However, this supervision is not always reliable: a teacher can assign high likelihood to plausible but incorrect solutions, or low likelihood to correct student solutions that follow different reasoning paths. Unconditionally distilling the teacher can therefore reinforce bad modes or erase useful student behavior. To address these limitations, we introduce RG-OPD: Reward-Gated On-Policy Distillation that uses verifier feedback to decide when teacher logits should be trusted. RG-OPD bridges sparse verifier rewards and dense teacher logits, preserving token-level supervision while filtering misleading teacher signals. Across reasoning and coding benchmarks, RG-OPD produces stronger distilled students, outperforming both vanilla reverse-KL distillation and the recent TSD-KD baseline. At 1K generation length, RG-OPD improves over reverse-KL by 2.9 points and over TSD-KD by 4.9 points; in the long-generation setting, it improves over the untuned student by 8.2 points. Our code is available at https://github.com/UoC-tail/RG-OPD.
Mohammad Sadegh Akhondzadeh, Vijay Lingam, Atula Tejaswi +3
On-policy distillation (OPD) grounds token-level supervision in the student's own trajectory, yet suffers from prefix failure: once the student commits to a wrong reasoning direction, all subsequent generation builds on this deviation, producing misdirected continuations that elicit unreliable supervision and waste compute. We identify a teacher-student continuation asymmetry on failed prefixes, where the teacher tends to redirect while the student continues along the original direction, and convert it into a label-free handoff trigger in Relay On-Policy Distillation (Relay-OPD). During training, Relay-OPD constructs relay trajectories by letting the teacher briefly take over at detected trigger points to produce a teacher leg, after which the student resumes and is optimized on the resulting trajectory. A limited relay budget concentrates intervention on critical early positions while limiting departure from the student policy. With a Qwen3-4B-Instruct-2507 teacher and Qwen3-0.6B/1.7B-Non-Thinking students on eight mathematical reasoning benchmarks, Relay-OPD achieves the best or second-best results on every benchmark, outperforming standard OPD by +5.73% and the strongest baseline FastOPD by +1.49% on average for 1.7B, with consistent gains at 0.6B. Training trajectory length is reduced by over 50%.
Haolei Xu, Xiaowen Xu, Haiwen Hong +5
1Zhejiang University · 2Yuvion Team, Alibaba Group