Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation

Xiang Chen; Futao Su; Kong Wang; Jiayi Chen; TanLin Li

cs.AIOct 2, 2026

Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation

Authors: Xiang Chen, Futao Su, Kong Wang, Jiayi Chen, TanLin Li

Organizations: Tongji University · Xiaohongshu · Shanghai Jiao Tong University · Hunan University

Abstract

On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher, but providing such supervision for every rollout requires substantial teacher computation. We introduce Success-Referenced On-Policy Distillation (SR-OPD), which reduces this cost by selecting which prompts and rollouts receive teacher supervision. When the student produces both successful and failed rollouts for the same prompt, a successful rollout can serve as a natural reference for selecting failed rollouts. SR-OPD therefore focuses on such prompts and prioritizes failed rollouts whose hidden-state trajectories show sustained divergence from a successful reference, while accounting for estimated teacher-input cost. Across three teacher-student pairs and six mathematical reasoning benchmarks, SR-OPD uses only 3.46-5.02% of the teacher-input tokens required by Vanilla OPD in the one-pass setting while maintaining comparable reasoning performance. Under a controlled setting matched to 5% of Vanilla OPD's teacher-input budget, further experiments support both key design choices: focusing supervision on prompts with both successful and failed rollouts, and using successful rollouts to guide failure selection. These results indicate that a student's own successful behavior can serve as a useful reference for allocating teacher supervision under a fixed teacher-input budget.

Figures & tables

Figure 1: Two-stage pre-query routing in SR-OPD. Left: prompt routing retains prompts containing both successful and failed rollouts while skipping all-success and all-failure prompts. Right: a same-prompt success serves only as a student-side reference; persistent-departure area (PDA) and teacher-input cost rank failed rollouts. In this example, only

r_{3}

receives frozen-teacher supervision and the standard OPD update.

Pair	Method	AIME24	AIME25	AMC23	HMMT24	HMMT25	MATH-500	Macro Avg	Input (M; ratio)
Qwen	Vanilla OPD	34.31	19.44	61.30	11.60	13.96	84.30	37.48	374.10 (1.00)
	TA-OPD	29.72	21.39	61.35	14.31	13.75	83.49	37.34	417.90 (1.12)
	TLR-OPD	17.57	15.35	46.98	14.10	12.08	75.57	30.27	19.20 (0.05)
	SR-OPD	34.17	19.58	61.56	12.08	13.68	84.21	37.55	13.60 (0.04)
Skywork	Vanilla OPD	36.74	26.67	67.60	15.14	19.24	85.89	41.88	519.80 (1.00)
	TA-OPD	35.07	26.81	66.72	14.93	17.22	85.73	41.08	516.20 (0.99)

Table 1: Native operating points. Benchmark columns report accuracy over 16 generation seeds, averaged across three training seeds. Macro Avg is the unweighted mean across six benchmarks for each training seed, then averaged across training seeds. Input is mean serialized teacher-input tokens in millions; parenthesized ratios are decimal values relative to Vanilla OPD in the same pair. Bold marks the highest observed benchmark and Macro Avg mean within each pair and the lowest teacher input.

Figure 2: Controlled routing and mechanism analyses. (a) Absolute Macro Avg for Global+Random and Mixed+Random; each model has its own horizontal score scale. (b) Absolute Macro Avg for the four Mixed-gate selectors, with a separate local scale per model; R, T, S, and P denote Random, TLR, Static SR, and PDA. Paired differences and intervals are in Appendix A.7 . (c) Independent component effects shown as treatment-minus-control paired means and pointwise 95% intervals. (d) Spearman correlations between the Full SR score and three post-query diagnostics; darker cells indicate stronger association. Definitions and numerical results are in Appendix A.5 for (a), Appendices A.6 – A.7 for (b), Appendix A.8 for (c), and Appendix A.9 for (d).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Pair	Range (%)
Qwen	4.98–5.02
Skywork	4.98–5.01
Granite	4.99–5.02

Appendix

Table A1: Realized teacher-input use across the four mixed-gate selectors, as a percentage of the corresponding Vanilla OPD input.

Pair	Global+Random	Mixed+Random	Difference	95% CI
Qwen	36.14	36.55	$+0.410$	$[+0.213,+0.607]$
Skywork	40.02	40.49	$+0.467$	$[+0.265,+0.669]$
Granite	14.05	14.39	$+0.338$	$[+0.130,+0.546]$

Appendix

Table A2: Controlled prompt routing at the matched 5% teacher-input budget. Effects are paired Mixed+Random minus Global+Random Macro Avg.

Pair	Method	AIME24	AIME25	AMC23	HMMT24	HMMT25	MATH-500	Macro Avg
Qwen	Mixed+Random	30.90	21.39	58.70	12.71	14.72	80.88	36.55
	Mixed+TLR	30.97	21.39	59.38	13.13	15.14	80.89	36.81
	Mixed+StaticSR	31.11	21.60	59.53	13.26	15.21	81.14	36.98
	Mixed+PDA (SR-OPD)	31.11	21.74	59.95	13.61	15.49	81.10	37.17
Skywork	Mixed+Random	35.49	26.04	65.47	13.75	17.15	85.05	40.49
	Mixed+TLR	35.83	26.46	65.63	14.17	17.71	85.13	40.82

Appendix

Table A3: Controlled rollout-routing results at the matched 5% teacher-input budget.

Pair	Comparator	Difference	95% CI
Qwen	Mixed+Random	$+0.616$	$[+0.241,+0.991]$
	Mixed+TLR	$+0.352$	$[-0.305,+1.008]$
	Mixed+StaticSR	$+0.191$	$[-0.514,+0.895]$
Skywork	Mixed+Random	$+0.738$	$[+0.651,+0.825]$
	Mixed+TLR	$+0.408$	$[+0.255,+0.561]$
	Mixed+StaticSR	$+0.262$	$[-0.372,+0.895]$

Appendix

Table A4: Paired Mixed+PDA minus comparator effects on Macro Avg. Intervals use paired training seeds 41–43.

Comparison	Pair	Difference	95% CI
Same-Prompt $-$ Matched Other-Prompt	Qwen	$+0.173$	$[-0.293,+0.639]$
	Skywork	$+0.076$	$[-0.159,+0.310]$
	Granite	$+0.154$	$[-0.184,+0.493]$
Same-Prompt $-$ Shuffled Other-Prompt	Qwen	$+0.262$	$[-0.098,+0.621]$
	Skywork	$+0.180$	$[-0.129,+0.488]$
	Granite	$+0.160$	$[-0.156,+0.477]$

Appendix

Table A5: Independent paired component ablations. Differences are treatment minus control Macro Avg; intervals are pointwise 95%.

Pair	Signal	$U_{\mathrm{LR}}$	$U_{\mathrm{grad}}$	$U_{\mathrm{val}}$
Qwen	TLR	0.071	0.082	0.162
	AvgDistance	0.101	0.112	0.126
	PDA	0.271	0.301	0.314
	Full SR	0.251	0.412	0.196
Skywork	TLR	0.104	0.143	0.151
	AvgDistance	0.116	0.071	0.181

Appendix

Table A6: Spearman correlations between pre-query routing signals and post-query diagnostic proxies (200 failed items per pair). AvgDistance is a diagnostic mean-distance score, not the StaticSR training ablation.

Pair	Method	$K=8$	$K=16$	Change [95% CI]
Qwen	Mixed+Random	36.49	36.53	$+0.040\;[-0.072,+0.152]$
	Cost-only	36.85	36.90	$+0.050\;[-0.049,+0.149]$
	Frontier-TLR	36.79	36.82	$+0.030\;[-0.082,+0.142]$
	Static SR	36.99	37.05	$+0.060\;[-0.052,+0.172]$
	SR-OPD	37.16	37.32	$+0.160\;[+0.036,+0.284]$
Skywork	Mixed+Random	40.39	40.48	$+0.090\;[-0.022,+0.202]$

Appendix

Table A7: Rollout-count sensitivity at the nominal 3% cap. The final column is the paired

K=16

minus

K=8

change.

Pair	Method	$K=8$ input (M)	$K=8$ / Vanilla (%)	$K=16$ input (M)	$K=16$ / Vanilla (%)
Qwen	Mixed+Random	11.350	3.034	11.375	3.041
	Cost-only	11.130	2.975	11.114	2.971
	Frontier-TLR	11.306	3.022	11.333	3.030
	Static SR	11.207	2.996	11.242	3.005
	SR-OPD	11.207	2.996	11.215	2.998
Skywork	Mixed+Random	15.551	2.992	15.590	2.999

Appendix

Table A8: Realized teacher-input usage for the rollout-count sensitivity experiment. Input is serialized teacher-input tokens in millions; percentages are relative to the corresponding Vanilla OPD teacher-input mean.

Pair	Audit error [95% CI]	0% flip	5% flip	10% flip
Qwen	1.50% [0.69,3.23]	37.17	37.05	36.84
Skywork	2.25% [1.19,4.22]	41.23	41.07	40.83
Granite	1.00% [0.39,2.54]	14.98	14.87	14.72

Appendix

Table A9: Verifier reliability: audit error among 400 items per pair with 95% Wilson intervals, and Macro Avg under random label flips. The

0\%

column is the unperturbed controlled SR-OPD baseline.

Explore similar work

Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation
May 29, 2026Yanjiang Liu, Jie Lou, Xinyan Guan +7CoT Distillation Language Model Distillation
University of Chinese Academy of Sciences · 2Chinese Information Processing Laboratory Institute of Software, Chinese Academy of Sciences 2University of Chinese Academy of Sciences, Beijing, China · 3Xiaohongshu
Reward-Gated On-Policy Distillation
Jul 4, 2026Mohammad Sadegh Akhondzadeh, Vijay Lingam, Atula Tejaswi +3Language Model Distillation On-Policy Distillation
University of Cologne · AWS AI · UT Austin +1
Pass the Baton: Trajectory-Relayed On-Policy Distillation
Jul 28, 2026Haolei Xu, Xiaowen Xu, Haiwen Hong +5Language Model Distillation On-Policy Distillation
1Zhejiang University · 2Yuvion Team, Alibaba Group

May 29, 2026J/K move · Enter open · S save

Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation

Yanjiang Liu, Jie Lou, Xinyan Guan, Yuqiu Ji+6

University of Chinese Academy of Sciences · 2Chinese Information Processing Laboratory Institute of Software, Chinese Academy of Sciences 2University of Chinese Academy of Sciences, Beijing, China · 3Xiaohongshu

CoT Distillation Language Model Distillation On-Policy Distillation Teacher-Student Learning RL Fine-Tuning

Open paper AskPDF arXiv

On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher. However, we identify a critical bottleneck, Supervision Fidelity Decay (SFD): as student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative. Consequently, the teacher-dependent corrective signal in reverse-KL distillation weakens, causing student drift to compound across long reasoning chains. To mitigate SFD, we introduce Lookahead Group Reward (\ours{}). Building on the insight that next-step teacher confidence reflects the discriminative strength of future reverse-KL supervision, \ours{} evaluates the student's top-K candidate tokens by the teacher confidence they induce at the subsequent step and assigns a group-normalized reward. To maintain computational efficiency, we further design an entropy-triggered tree-attention mechanism. Across six math and code benchmarks, \ours{} improves mean@8 by 2.57 points over OPD for a 7B student, with gains increasing in longer-generation and reaching +4.92 points on AIME-26 at 39k tokens.

Abstract

Explore similar work

Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation

Reward-Gated On-Policy Distillation

Pass the Baton: Trajectory-Relayed On-Policy Distillation