On the Off-Policy Teacher in On-Policy Distillation

Langlin Huang; Hao Liu; Mononito Goswami; Xinyu Li; Prithwith Jana; Nikos Kanakaris; Patrick Blöbaum; Purak Jain

cs.LGSep 29, 2026

On the Off-Policy Teacher in On-Policy Distillation

Authors: Langlin Huang, Hao Liu, Mononito Goswami, Xinyu Li, Prithwith Jana, Nikos Kanakaris, Patrick Blöbaum, Purak Jain

Organizations: Washington University in St. Louis · AWS AI Labs · Carnegie Mellon University · Georgia Institute of Technology

Abstract

On-policy distillation (OPD) has recently emerged as a promising post-training paradigm in which the student learns from trajectories generated by its own policy under dense teacher supervision. However, OPD introduces a fundamental asymmetry: although the sampled trajectories are on-policy for the student, they are off-policy for the teacher. The teacher is typically optimized to continue from prefixes generated by its own policy, but during OPD it must instead supervise prefixes generated by the student. Empirically, we find that its continuation performance degrades as these prefixes grow longer. To address this issue, we propose Student-COnditioned Updates of the Teacher (SCOUT), a co-training framework that adapts the teacher to student-generated prefixes. Alongside standard OPD updates, SCOUT periodically optimizes the teacher's conditional ability using reinforcement learning with verifiable rewards, where the teacher generates continuations from student prefixes and learns from outcome rewards. Controlled experiments show that SCOUT improves the teacher's ability to continue from student-generated prefixes, supporting the intended mechanism of student-conditioned teacher adaptation. Across multiple teacher--student configurations, model scales, and reasoning domains, SCOUT also consistently improves the effectiveness of on-policy distillation.

Figures & tables

Figure 1 : Teacher continuation from student-generated prefixes. As the student prefix grows longer, final-answer accuracy declines and a persistent entropy gap emerges between student and teacher prefixes, indicating increasing teacher uncertainty under student-generated contexts.

Figure 2 : The overview of SCOUT and its comparison with standard OPD. While standard OPD suffers from unreliable teacher supervision at long prefixes, SCOUT periodically trains the teacher $\pi_{\phi}$ given the student prefix $p^{S}$ . As shown in the figure, SCOUT is a combination of standard OPD and teacher-side RL training. $\pi_{\phi}$ is synchronized after its update to provide supervision for the next OPD training steps. Upper right subfigure: $\pi_{\phi}$ becomes gradually better at student prefixes along training. So we linearly increase the student prefix ratio according to training steps to increase the difficulty level.

Method	Mathematical Reasoning							Code Generation
Method	AIME24 Avg@32	AIME25 Avg@32	AMC23 Avg@32	HMMT Avg@32	Olym. Avg@4	MATH500 Avg@8	Mean	LCB v5 Avg@4	HE+ Avg@16	MBPP Avg@8	Mean
Qwen3-1.7B	$13.3{\scriptstyle\pm 4.2}$	$10.3{\scriptstyle\pm 3.2}$	$46.8{\scriptstyle\pm 4.9}$	$5.4{\scriptstyle\pm 2.6}$	$42.3{\scriptstyle\pm 1.5}$	$73.1{\scriptstyle\pm 1.5}$	$31.9$	$27.3{\scriptstyle\pm 0.3}$	$62.4{\scriptstyle\pm 1.5}$	$47.3{\scriptstyle\pm 1.2}$	$45.7$
Qwen3-4B-Ins	$54.9{\scriptstyle\pm 4.6}$	$44.5{\scriptstyle\pm 4.8}$	$92.4{\scriptstyle\pm 3.0}$	$28.0{\scriptstyle\pm 4.5}$	$72.0{\scriptstyle\pm 1.6}$	$93.5{\scriptstyle\pm 0.7}$	$64.2$	$50.3{\scriptstyle\pm 0.3}$	$84.3{\scriptstyle\pm 1.3}$	$79.4{\scriptstyle\pm 1.2}$	$71.3$
GRPO	$30.9{\scriptstyle\pm 4.5}$	$30.4{\scriptstyle\pm 4.1}$	$73.3{\scriptstyle\pm 5.7}$	$18.9{\scriptstyle\pm 3.4}$	$61.1{\scriptstyle\pm 5.8}$	$87.4{\scriptstyle\pm 3.5}$	$50.3{\scriptstyle\pm 4.4}$	$44.3{\scriptstyle\pm 6.2}$	$72.7{\scriptstyle\pm 1.4}$	$69.6{\scriptstyle\pm 5.0}$	$62.2{\scriptstyle\pm 2.8}$
OPD	$33.8{\scriptstyle\pm 3.0}$	$24.3{\scriptstyle\pm 1.8}$	$\underline{73.9{\scriptstyle\pm 1.9}}$	$16.0{\scriptstyle\pm 0.8}$	$\underline{60.2{\scriptstyle\pm 1.6}}$	$\underline{87.0{\scriptstyle\pm 1.1}}$	$49.2{\scriptstyle\pm 1.5}$	$\underline{37.6{\scriptstyle\pm 0.1}}$	$\underline{69.7{\scriptstyle\pm 0.7}}$	$\underline{62.5{\scriptstyle\pm 0.3}}$	$\underline{56.6{\scriptstyle\pm 0.3}}$
ESR	$31.9{\scriptstyle\pm 0.9}$	$25.5{\scriptstyle\pm 1.1}$	$70.6{\scriptstyle\pm 2.6}$	$16.0{\scriptstyle\pm 1.4}$	$59.0{\scriptstyle\pm 0.5}$	$86.3{\scriptstyle\pm 0.6}$	$48.2{\scriptstyle\pm 0.5}$	$36.0{\scriptstyle\pm 2.3}$	$68.1{\scriptstyle\pm 1.0}$	$61.3{\scriptstyle\pm 2.2}$	$55.1{\scriptstyle\pm 1.7}$

Table 1 : Main results for Qwen3-4B-Instruct-2507

\rightarrow

Qwen3-1.7B. SCOUT consistently outperforms competing OPD methods across mathematical reasoning and code generation, demonstrating that its gains generalize across domains.

Method	Mathematical Reasoning
Method	AIME24 Avg@32	AIME25 Avg@32	AMC23 Avg@32	HMMT Avg@32	Olym. Avg@4	MATH500 Avg@8	Mean
Qwen3-1.7B	$13.3{\scriptstyle\pm 4.2}$	$10.3{\scriptstyle\pm 3.2}$	$46.8{\scriptstyle\pm 4.9}$	$5.4{\scriptstyle\pm 2.6}$	$42.3{\scriptstyle\pm 1.5}$	$73.1{\scriptstyle\pm 1.5}$	$31.9$
Qwen3-8B-DAPO	$59.9{\scriptstyle\pm 6.0}$	$43.9{\scriptstyle\pm 5.9}$	$90.7{\scriptstyle\pm 3.9}$	$25.7{\scriptstyle\pm 4.1}$	$74.2{\scriptstyle\pm 0.7}$	$94.6{\scriptstyle\pm 0.5}$	$64.8$
GRPO	$30.9{\scriptstyle\pm 4.5}$	$30.4{\scriptstyle\pm 4.1}$	$73.3{\scriptstyle\pm 5.7}$	$18.9{\scriptstyle\pm 3.4}$	$61.1{\scriptstyle\pm 5.8}$	$87.4{\scriptstyle\pm 3.5}$	$50.3{\scriptstyle\pm 4.4}$
OPD	$33.4{\scriptstyle\pm 0.8}$	$\underline{27.8{\scriptstyle\pm 1.0}}$	$70.1{\scriptstyle\pm 0.7}$	$15.9{\scriptstyle\pm 1.4}$	$60.0{\scriptstyle\pm 1.1}$	$87.0{\scriptstyle\pm 0.5}$	$\underline{49.0{\scriptstyle\pm 0.6}}$
ESR	$\underline{34.8{\scriptstyle\pm 2.5}}$	$26.7{\scriptstyle\pm 1.9}$	$69.5{\scriptstyle\pm 2.0}$	$15.8{\scriptstyle\pm 0.2}$	$\underline{60.6{\scriptstyle\pm 0.4}}$	$86.8{\scriptstyle\pm 0.2}$	$\underline{49.0{\scriptstyle\pm 0.4}}$

Table 2 : Mathematical reasoning with Qwen3-8B-DAPO

\rightarrow

Qwen3-1.7B. SCOUT outperforms all other OPD-based methods across all six benchmarks, demonstrating that its gains generalize to larger model scales.

Method	Mathematical Reasoning
Method	AIME24 Avg@32	AIME25 Avg@32	AMC23 Avg@32	HMMT Avg@32	Olym. Avg@4	MATH500 Avg@8	Mean
DeepSeek-R1-Distill-Qwen-1.5B	$28.1{\scriptstyle\pm 4.7}$	$23.8{\scriptstyle\pm 4.7}$	$70.6{\scriptstyle\pm 4.8}$	$13.9{\scriptstyle\pm 4.8}$	$52.9{\scriptstyle\pm 1.4}$	$83.2{\scriptstyle\pm 0.8}$	$45.4$
Skywork-OR1-Math-7B	$57.7{\scriptstyle\pm 5.4}$	$46.4{\scriptstyle\pm 4.8}$	$90.2{\scriptstyle\pm 2.7}$	$27.3{\scriptstyle\pm 3.9}$	$73.4{\scriptstyle\pm 0.9}$	$94.8{\scriptstyle\pm 0.7}$	$65.0$
GRPO †	$4.1{\scriptstyle\pm 3.0}$	$3.1{\scriptstyle\pm 2.7}$	$35.8{\scriptstyle\pm 5.3}$	$0.4{\scriptstyle\pm 1.1}$	$29.2{\scriptstyle\pm 1.4}$	$63.1{\scriptstyle\pm 1.2}$	$22.6$
OPD	$\underline{37.8{\scriptstyle\pm 0.6}}$	$\mathbf{30.0{\scriptstyle\pm 0.5}}$	$\underline{77.9{\scriptstyle\pm 0.6}}$	$\mathbf{19.1{\scriptstyle\pm 0.1}}$	$\underline{60.6{\scriptstyle\pm 0.2}}$	$\underline{89.1{\scriptstyle\pm 0.1}}$	$\underline{52.4{\scriptstyle\pm 0.2}}$
ESR	$32.6{\scriptstyle\pm 2.0}$	$27.3{\scriptstyle\pm 0.1}$	$71.6{\scriptstyle\pm 1.7}$	$17.3{\scriptstyle\pm 1.1}$	$56.9{\scriptstyle\pm 0.4}$	$87.5{\scriptstyle\pm 0.1}$	$48.8{\scriptstyle\pm 0.4}$

Table 3 : Mathematical reasoning with Skywork-OR1-Math-7B

\rightarrow

DeepSeek-R1-Distill-Qwen-1.5B. GRPO collapses and Prune-OPD degrades substantially in this setting, while SCOUT remains robust and outperforms baselines, demonstrating that its gains generalize across model families.

Figure 3 : SCOUT improves the teacher’s ability to reason from student-generated prefixes. Left: Matched student–teacher checkpoints become more accurate across increasingly long prefixes as training progresses. Right: Holding the student prefixes fixed, the trained teacher outperforms the initial teacher on the same inputs, showing that the gains reflect teacher adaptation rather than stronger student prefixes alone.

Figure 4 : SCOUT reduces teacher uncertainty on student-generated prefixes. After training, teacher entropy on student prefixes decreases over later response positions, narrowing the gap with entropy on the teacher’s own responses. The inset shows the corresponding behavior before training.

Figure 6 : Additional OPD training does not close the gap to SCOUT . In the Qwen3-4B-Instruct-2507

\rightarrow

Qwen3-1.7B math setting, we train SCOUT for one epoch and OPD for three, on the same data, to compare performance over similar training-compute budgets. OPD quickly plateaus across repeated epochs, while SCOUT continues to improve.

Task	Base	w/o SCOUT	+ SCOUT	Gain
Math	OPD	49.2	51.4	+2.2
Math	OPTR	49.8	52.3	+2.5
Code	OPD	56.6	59.7	+3.1
Code	OPTR	59.3	60.5	+1.2

Table 5 : SCOUT provides complementary gains on top of OPTR. In the Qwen3-4B-Instruct-2507

\rightarrow

Qwen3-1.7B setting, adding SCOUT improves both OPD and OPTR across math and code.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Table 11

Teacher Model	Student Model	Task	Student LR	Teacher LR
Qwen3-4B-Instruct	Qwen3-1.7B	Math	$5{\times}10^{-6}$	$5{\times}10^{-6}$
Qwen3-8B-DAPO	Qwen3-1.7B	Math	$3{\times}10^{-6}$	$5{\times}10^{-6}$
Skywork-OR1-MATH-7B	DeepSeek-R1-Distill-Qwen-1.5B	Math	$3{\times}10^{-6}$	$5{\times}10^{-6}$
Qwen3-4B-Instruct	Qwen3-1.7B	Code	$5{\times}10^{-6}$	$5{\times}10^{-6}$

Appendix

Table 8: Learning rates for the student and teacher models across different model pairs and tasks.

Benchmark	Reasoning						Code Generation
Benchmark	AIME24	AIME25	AMC23	HMMT	Olympiad	MATH500	LCB v5	HumanEval+	MBPP
# Questions	30	30	40	30	581	500	880	164	500
# Repeats	32	32	32	32	4	8	4	16	8

Appendix

Table 9 : The statistics of the evaluation benchmarks, including the number of questions and the number of sampled responses per question.

Method	Run	AIME24	AIME25	AMC23	HMMT	Olymp.	MATH500	Avg6
	Avg.@	32	32	32	32	4	8	–
GRPO	Run 1	$29.06{\scriptstyle\,\pm\,0.97}$	$30.10{\scriptstyle\,\pm\,0.61}$	$73.28{\scriptstyle\,\pm\,0.76}$	$16.77{\scriptstyle\,\pm\,0.69}$	$60.80{\scriptstyle\,\pm\,0.48}$	$87.85{\scriptstyle\,\pm\,0.11}$	$49.64$
	Run 2	$27.71{\scriptstyle\,\pm\,0.98}$	$26.46{\scriptstyle\,\pm\,0.70}$	$67.58{\scriptstyle\,\pm\,0.84}$	$16.98{\scriptstyle\,\pm\,0.48}$	$55.46{\scriptstyle\,\pm\,0.22}$	$83.67{\scriptstyle\,\pm\,0.42}$	$46.31$
	Run 3	$36.04{\scriptstyle\,\pm\,1.26}$	$34.58{\scriptstyle\,\pm\,0.87}$	$79.06{\scriptstyle\,\pm\,0.94}$	$22.81{\scriptstyle\,\pm\,0.54}$	$67.00{\scriptstyle\,\pm\,0.33}$	$90.62{\scriptstyle\,\pm\,0.26}$	$55.02$
	Mean	$30.94{\scriptstyle\,\pm\,4.47}$	$30.38{\scriptstyle\,\pm\,4.07}$	$73.31{\scriptstyle\,\pm\,5.74}$	$18.85{\scriptstyle\,\pm\,3.43}$	$61.09{\scriptstyle\,\pm\,5.78}$	$87.38{\scriptstyle\,\pm\,3.50}$	$50.32{\scriptstyle\,\pm\,4.39}$
OPD	Run 1	$30.94{\scriptstyle\,\pm\,1.06}$	$22.40{\scriptstyle\,\pm\,0.81}$	$71.95{\scriptstyle\,\pm\,0.71}$	$15.42{\scriptstyle\,\pm\,0.65}$	$58.30{\scriptstyle\,\pm\,0.45}$	$85.75{\scriptstyle\,\pm\,0.52}$	$47.46$

Appendix

Table 10 : Complete results for the Qwen3-4B-Instruct-2507

\rightarrow

Qwen3-1.7B setting on six math reasoning benchmarks. This table corresponds to the mathematical reasoning task in Table . SCOUT shows significant improvement over OPD on 3 of 6 benchmarks and in the average performance.

Method	Run	LCB v5	HumanEval+	MBPP	Avg.
	Avg.@	4	16	8	–
GRPO	Run 1	$43.12{\scriptstyle\,\pm\,0.65}$	$74.16{\scriptstyle\,\pm\,1.64}$	$64.20{\scriptstyle\,\pm\,0.85}$	$60.49$
	Run 2	$38.81{\scriptstyle\,\pm\,1.50}$	$72.71{\scriptstyle\,\pm\,2.94}$	$70.85{\scriptstyle\,\pm\,1.63}$	$60.79$
	Run 3	$51.02{\scriptstyle\,\pm\,1.53}$	$71.34{\scriptstyle\,\pm\,2.73}$	$73.88{\scriptstyle\,\pm\,1.57}$	$65.41$
	Mean	$44.32{\scriptstyle\,\pm\,6.20}$	$72.74{\scriptstyle\,\pm\,1.41}$	$69.64{\scriptstyle\,\pm\,4.95}$	$62.23{\scriptstyle\,\pm\,2.76}$
OPD	Run 1	$37.67{\scriptstyle\,\pm\,0.25}$	$69.21{\scriptstyle\,\pm\,0.82}$	$62.50{\scriptstyle\,\pm\,0.36}$	$56.46$

Appendix

Table 11 : Per-run results on coding benchmarks for the Qwen3-4B-Instruct-2507

\rightarrow

Qwen3-1.7B setting. This table corresponds to the code generation task in Table . While other OPD-based methods perform worse in the code domain, SCOUT demonstrates its generalizability across domains, with significantly better accuracy on 2 of 3 benchmarks and in the average performance.

Method	Run	AIME24	AIME25	AMC23	HMMT	Olymp.	MATH500	Avg6
	Avg.@	32	32	32	32	4	8	–
GRPO	Run 1	$29.06{\scriptstyle\,\pm\,0.97}$	$30.10{\scriptstyle\,\pm\,0.61}$	$73.28{\scriptstyle\,\pm\,0.76}$	$16.77{\scriptstyle\,\pm\,0.69}$	$60.80{\scriptstyle\,\pm\,0.48}$	$87.85{\scriptstyle\,\pm\,0.11}$	$49.64$
	Run 2	$27.71{\scriptstyle\,\pm\,0.98}$	$26.46{\scriptstyle\,\pm\,0.70}$	$67.58{\scriptstyle\,\pm\,0.84}$	$16.98{\scriptstyle\,\pm\,0.48}$	$55.46{\scriptstyle\,\pm\,0.22}$	$83.67{\scriptstyle\,\pm\,0.42}$	$46.31$
	Run 3	$36.04{\scriptstyle\,\pm\,1.26}$	$34.58{\scriptstyle\,\pm\,0.87}$	$79.06{\scriptstyle\,\pm\,0.94}$	$22.81{\scriptstyle\,\pm\,0.54}$	$67.00{\scriptstyle\,\pm\,0.33}$	$90.62{\scriptstyle\,\pm\,0.26}$	$55.02$
	Mean	$30.94{\scriptstyle\,\pm\,4.47}$	$30.38{\scriptstyle\,\pm\,4.07}$	$73.31{\scriptstyle\,\pm\,5.74}$	$18.85{\scriptstyle\,\pm\,3.43}$	$61.09{\scriptstyle\,\pm\,5.78}$	$87.38{\scriptstyle\,\pm\,3.50}$	$50.32{\scriptstyle\,\pm\,4.39}$
OPD	Run 1	$32.50{\scriptstyle\,\pm\,0.85}$	$27.08{\scriptstyle\,\pm\,1.01}$	$69.38{\scriptstyle\,\pm\,0.78}$	$14.79{\scriptstyle\,\pm\,0.73}$	$59.60{\scriptstyle\,\pm\,0.64}$	$86.52{\scriptstyle\,\pm\,0.36}$	$48.31$

Appendix

Table 12 : Complete per-run results for the Qwen3-8B-DAPO

\rightarrow

Qwen3-1.7B setting on six math reasoning benchmarks. This table corresponds to Table . SCOUT significantly outperforms the OPD baseline in 5 of 6 benchmarks and in average performance. These results suggest that SCOUT ’s gains extend to the larger-teacher setting.

Method	Run	AIME24	AIME25	AMC23	HMMT	Olymp.	MATH500	Avg6
	Avg.@	32	32	32	32	4	8	–
OPD	Run 1	$37.92{\scriptstyle\,\pm\,1.05}$	$30.52{\scriptstyle\,\pm\,0.73}$	$77.27{\scriptstyle\,\pm\,0.77}$	$19.17{\scriptstyle\,\pm\,0.89}$	$60.46{\scriptstyle\,\pm\,0.76}$	$89.00{\scriptstyle\,\pm\,0.26}$	$52.39$
	Run 2	$38.33{\scriptstyle\,\pm\,0.89}$	$29.90{\scriptstyle\,\pm\,0.68}$	$78.44{\scriptstyle\,\pm\,0.66}$	$19.06{\scriptstyle\,\pm\,0.45}$	$60.80{\scriptstyle\,\pm\,0.76}$	$89.15{\scriptstyle\,\pm\,0.25}$	$52.61$
	Run 3	$37.08{\scriptstyle\,\pm\,1.06}$	$29.58{\scriptstyle\,\pm\,0.53}$	$78.05{\scriptstyle\,\pm\,0.94}$	$18.96{\scriptstyle\,\pm\,0.55}$	$60.41{\scriptstyle\,\pm\,0.64}$	$89.10{\scriptstyle\,\pm\,0.23}$	$52.20$
	Mean	$\underline{37.78}{\scriptstyle\,\pm\,0.64}$	$\mathbf{30.00}{\scriptstyle\,\pm\,0.48}$	$\underline{77.92}{\scriptstyle\,\pm\,0.60}$	$\mathbf{19.06}{\scriptstyle\,\pm\,0.11}$	$\underline{60.56}{\scriptstyle\,\pm\,0.21}$	$\underline{89.08}{\scriptstyle\,\pm\,0.08}$	$\underline{52.40}{\scriptstyle\,\pm\,0.21}$
ESR	Run 1	$31.67{\scriptstyle\,\pm\,1.20}$	$27.40{\scriptstyle\,\pm\,0.83}$	$73.44{\scriptstyle\,\pm\,0.82}$	$18.44{\scriptstyle\,\pm\,0.67}$	$56.80{\scriptstyle\,\pm\,0.61}$	$87.38{\scriptstyle\,\pm\,0.36}$	$49.19$

Appendix

Table 13 : Complete per-run results for the Skywork-OR1-Math-7B

\rightarrow

DeepSeek-R1-Distill-Qwen-1.5B setting on six math reasoning benchmarks. This table corresponds to Table . SCOUT significantly outperforms OPD on 4 of 6 benchmarks and in average performance, which extends its generalizability beyond the Qwen model family.

Method	Run	AIME24	AIME25	AMC23	HMMT	Olymp.	MATH500	Avg.
	Avg.@	32	32	32	32	4	8	–
(a) Qwen3-4B-Instruct-2507 $\rightarrow$ Qwen3-1.7B
OPD	Mean	$\underline{33.82}{\scriptstyle\,\pm\,3.03}$	$24.31{\scriptstyle\,\pm\,1.75}$	$\underline{73.93}{\scriptstyle\,\pm\,1.85}$	$\underline{16.04}{\scriptstyle\,\pm\,0.75}$	$\underline{60.15}{\scriptstyle\,\pm\,1.64}$	$87.03{\scriptstyle\,\pm\,1.14}$	$\underline{49.21}{\scriptstyle\,\pm\,1.53}$
OPD + Teacher GRPO	Run 1	$29.17{\scriptstyle\,\pm\,0.82}$	$23.65{\scriptstyle\,\pm\,0.68}$	$73.05{\scriptstyle\,\pm\,0.87}$	$13.96{\scriptstyle\,\pm\,0.55}$	$58.56{\scriptstyle\,\pm\,0.55}$	$86.67{\scriptstyle\,\pm\,0.30}$	47.51
	Run 2	$32.29{\scriptstyle\,\pm\,1.05}$	$26.56{\scriptstyle\,\pm\,0.73}$	$74.53{\scriptstyle\,\pm\,0.96}$	$16.04{\scriptstyle\,\pm\,0.66}$	$60.15{\scriptstyle\,\pm\,0.72}$	$87.83{\scriptstyle\,\pm\,0.17}$	49.57
	Run 3	$32.50{\scriptstyle\,\pm\,1.07}$	$27.40{\scriptstyle\,\pm\,0.83}$	$71.95{\scriptstyle\,\pm\,0.92}$	$17.71{\scriptstyle\,\pm\,0.72}$	$59.42{\scriptstyle\,\pm\,0.46}$	$87.12{\scriptstyle\,\pm\,0.33}$	49.35

Appendix

Table 14 : Per-run mathematical-reasoning results for the OPD + Teacher GRPO control. This control applies the same outcome-reward GRPO update to the teacher as SCOUT, but the teacher rolls out from scratch without conditioning on a student-generated prefix. OPD and SCOUT are shown as three-run aggregates for reference. For individual training runs, the smaller

\pm

term denotes within-evaluation standard error; in Mean rows, it denotes standard deviation across three training runs. Avg. is the macro-average over the six displayed math benchmarks. Best and second-best three-run aggregates are shown in bold and underline , respectively.

Method	Run	LCB v5	HumanEval+	MBPP	Avg.
	Avg.@	4	16	8	–
OPD	Mean	$37.64{\scriptstyle\,\pm\,0.05}$	$69.74{\scriptstyle\,\pm\,0.68}$	$62.48{\scriptstyle\,\pm\,0.29}$	$56.62{\scriptstyle\,\pm\,0.28}$
OPD + Teacher GRPO	Run 1	$38.01{\scriptstyle\,\pm\,0.30}$	$69.21{\scriptstyle\,\pm\,0.50}$	$61.35{\scriptstyle\,\pm\,0.49}$	56.19
	Run 2	$39.40{\scriptstyle\,\pm\,0.29}$	$72.18{\scriptstyle\,\pm\,0.49}$	$65.80{\scriptstyle\,\pm\,0.27}$	59.13
	Run 3	$36.08{\scriptstyle\,\pm\,0.24}$	$69.36{\scriptstyle\,\pm\,0.60}$	$61.72{\scriptstyle\,\pm\,0.63}$	55.72
	Mean	$\underline{37.83}{\scriptstyle\,\pm\,1.67}$	$\underline{70.25}{\scriptstyle\,\pm\,1.67}$	$\underline{62.96}{\scriptstyle\,\pm\,2.47}$	$\underline{57.01}{\scriptstyle\,\pm\,1.85}$

Appendix

Table 15 : Per-run code-generation results for the OPD + Teacher GRPO control in the Qwen3-4B-Instruct-2507

\rightarrow

Qwen3-1.7B setting. This control applies the same outcome-reward GRPO update to the teacher as SCOUT, but the teacher rolls out from scratch without conditioning on a student-generated prefix. OPD and SCOUT are shown as three-run aggregates for reference. For individual training runs, the smaller

\pm

term denotes within-evaluation standard error; in Mean rows, it denotes standard deviation across three training runs. Best and second-best three-run aggregates are shown in bold and underline , respectively.

Model	AIME24	AIME25	AMC23	HMMT	Olympiad	MATH-500	Mean
Qwen3-1.7B	$13.3{\scriptstyle\pm 4.2}$	$10.3{\scriptstyle\pm 3.2}$	$46.8{\scriptstyle\pm 4.9}$	$5.4{\scriptstyle\pm 2.6}$	$42.3{\scriptstyle\pm 1.5}$	$73.1{\scriptstyle\pm 1.5}$	$31.9$
Qwen3-8B-DAPO	$59.9{\scriptstyle\pm 6.0}$	$43.9{\scriptstyle\pm 5.9}$	$90.7{\scriptstyle\pm 3.9}$	$25.7{\scriptstyle\pm 4.1}$	$74.2{\scriptstyle\pm 0.7}$	$94.6{\scriptstyle\pm 0.5}$	$64.8$
OPD	$33.5{\scriptstyle\pm 6.5}$	$28.0{\scriptstyle\pm 5.5}$	$69.2{\scriptstyle\pm 4.9}$	$16.4{\scriptstyle\pm 4.1}$	$59.9{\scriptstyle\pm 0.4}$	$87.2{\scriptstyle\pm 0.9}$	$49.0$
SCOUT ( $f{=}1$ )	$33.0{\scriptstyle\pm 5.8}$	$\underline{29.8}{\scriptstyle\pm 4.6}$	$\mathbf{76.1}{\scriptstyle\pm 4.8}$	$\underline{18.2}{\scriptstyle\pm 3.9}$	$\mathbf{62.9}{\scriptstyle\pm 1.0}$	$87.8{\scriptstyle\pm 0.8}$	$51.3$
SCOUT ( $f{=}5$ )	$\underline{38.4}{\scriptstyle\pm 6.0}$	$\mathbf{30.3}{\scriptstyle\pm 5.4}$	$\underline{75.8}{\scriptstyle\pm 5.2}$	$\mathbf{18.3}{\scriptstyle\pm 4.1}$	$\underline{62.7}{\scriptstyle\pm 1.8}$	$\mathbf{89.6}{\scriptstyle\pm 0.6}$	$\mathbf{52.5}$
SCOUT ( $f{=}10$ )	$\mathbf{39.2}{\scriptstyle\pm 5.0}$	$28.7{\scriptstyle\pm 6.1}$	$74.2{\scriptstyle\pm 5.0}$	$16.5{\scriptstyle\pm 4.1}$	$60.8{\scriptstyle\pm 0.4}$	$\underline{89.4}{\scriptstyle\pm 0.7}$	$\underline{51.5}$

Appendix

Table 16 : Ablation on the teacher update interval

f

with the Qwen3-8B-DAPO

\rightarrow

Qwen3-1.7B student setting. Updating the teacher with

f\leq 10

consistently improves over OPD, whereas the improvement becomes marginal or disappears for

f\geq 15

. Although

f=5

achieves the highest average accuracy,

f=10

retains most of the improvement with fewer teacher updates, providing a favorable performance–computation trade-off.

Method	Run	AIME24	AIME25	AMC23	HMMT	Olymp.	MATH500	Avg.
	Avg.@	32	32	32	32	4	8	–
OPD	Mean	$33.82{\scriptstyle\,\pm\,3.03}$	$24.31{\scriptstyle\,\pm\,1.75}$	$73.93{\scriptstyle\,\pm\,1.85}$	$16.04{\scriptstyle\,\pm\,0.75}$	$60.15{\scriptstyle\,\pm\,1.64}$	$87.03{\scriptstyle\,\pm\,1.14}$	$49.21{\scriptstyle\,\pm\,1.53}$
SCOUT	Mean	$\underline{35.76}{\scriptstyle\,\pm\,0.51}$	$\mathbf{28.82}{\scriptstyle\,\pm\,0.64}$	$\underline{75.76}{\scriptstyle\,\pm\,0.59}$	$\mathbf{17.22}{\scriptstyle\,\pm\,1.90}$	$\underline{62.38}{\scriptstyle\,\pm\,0.84}$	$\underline{88.41}{\scriptstyle\,\pm\,0.25}$	$\underline{51.39}{\scriptstyle\,\pm\,0.57}$
OPTR	Run 1	$33.02{\scriptstyle\,\pm\,1.01}$	$24.06{\scriptstyle\,\pm\,0.86}$	$76.56{\scriptstyle\,\pm\,0.82}$	$17.50{\scriptstyle\,\pm\,0.75}$	$61.14{\scriptstyle\,\pm\,0.56}$	$86.92{\scriptstyle\,\pm\,0.44}$	49.87
	Run 2	$32.81{\scriptstyle\,\pm\,0.95}$	$24.58{\scriptstyle\,\pm\,0.74}$	$73.98{\scriptstyle\,\pm\,0.85}$	$15.52{\scriptstyle\,\pm\,0.66}$	$61.49{\scriptstyle\,\pm\,0.43}$	$86.80{\scriptstyle\,\pm\,0.49}$	49.20
	Run 3	$34.90{\scriptstyle\,\pm\,0.92}$	$26.25{\scriptstyle\,\pm\,0.79}$	$75.08{\scriptstyle\,\pm\,0.66}$	$16.67{\scriptstyle\,\pm\,0.75}$	$61.66{\scriptstyle\,\pm\,0.37}$	$87.35{\scriptstyle\,\pm\,0.28}$	50.32

Appendix

Table 17 : Full results for combining SCOUT with On-policy trust region OPD (OPTR) in the Qwen3-4B-Instruct-2507

\rightarrow

Qwen3-1.7B math setting. OPTR performs better than OPD, and SCOUT further improves OPTR when combined with it. This shows that SCOUT is complementary to other loss-level OPD methods.

Method	Run	LCB v5	HumanEval+	MBPP	Avg.
	Avg.@	4	16	8	–
OPD	Mean	$37.64{\scriptstyle\,\pm\,0.05}$	$69.74{\scriptstyle\,\pm\,0.68}$	$62.48{\scriptstyle\,\pm\,0.29}$	$56.62{\scriptstyle\,\pm\,0.28}$
SCOUT	Mean	$40.57{\scriptstyle\,\pm\,2.51}$	$\underline{73.42}{\scriptstyle\,\pm\,0.88}$	$\underline{65.22}{\scriptstyle\,\pm\,2.62}$	$\underline{59.74}{\scriptstyle\,\pm\,1.88}$
OPTR	Run 1	$41.42{\scriptstyle\,\pm\,0.64}$	$73.67{\scriptstyle\,\pm\,0.52}$	$63.77{\scriptstyle\,\pm\,0.36}$	59.62
	Run 2	$40.40{\scriptstyle\,\pm\,0.26}$	$72.56{\scriptstyle\,\pm\,0.94}$	$64.58{\scriptstyle\,\pm\,0.56}$	59.18
	Run 3	$41.42{\scriptstyle\,\pm\,0.38}$	$72.79{\scriptstyle\,\pm\,0.72}$	$63.45{\scriptstyle\,\pm\,0.71}$	59.22

Appendix

Table 18 : Full results for combining SCOUT with On-policy trust region OPD (OPTR) in the Qwen3-4B-Instruct-2507

\rightarrow

Qwen3-1.7B code setting. When combined with SCOUT , OPTR achieves further improvement, showing the same trend as Table . This proves that the complementary merit of SCOUT generalizes to the code domain.

Explore similar work

May 29, 2026cs.CL

Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation

On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher. However, we identify a critical bottleneck, \textbf{Supervision Fidelity Decay (SFD)}: as student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative. Consequently, the teacher-dependent corrective signal in reverse-KL distillation weakens, causing student drift to compound across long reasoning chains. To mitigate SFD, we introduce \textbf{Lookahead Group Reward (\ours{})}. Building on the insight that next-step teacher confidence reflects the discriminative strength of future reverse-KL supervision, \ours{} evaluates the student's top-K candidate tokens by the teacher confidence they induce at the subsequent step and assigns a group-normalized reward. To maintain computational efficiency, we further design an entropy-triggered tree-attention mechanism. Across six math and code benchmarks, \ours{} improves mean@8 by \textbf{2.57} points over OPD for a 7B student, with gains increasing in longer-generation and reaching +\textbf{4.92} points on AIME-26 at 39k tokens.

Yanjiang Liu, Jie Lou, Xinyan Guan +7

University of Chinese Academy of Sciences · 2Chinese Information Processing Laboratory Institute of Software, Chinese Academy of Sciences 2University of Chinese Academy of Sciences, Beijing, China · 3Xiaohongshu

Teacher Dataset Distillation

Ask PDF

May 29, 2026cs.LG

Trust-Region Behavior Blending for On-Policy Distillation

On-policy distillation (OPD) trains a student on prefixes sampled from its own policy while matching a stronger teacher. This addresses the prefix mismatch of offline distillation, but early student rollouts can still be poor, placing teacher supervision on weak or low-quality prefixes. We propose Trust-Region behavior Blending (TRB), a warmup method that replaces the early rollout policy with the closest-to-teacher behavior policy inside a student-centered KL trust region, while keeping the per-prefix reverse-KL OPD loss unchanged. The KL budget is annealed to zero, so training returns to pure student rollouts after warmup. Across two math-reasoning distillation settings, TRB attains the strongest average among the compared methods.

Daniil Plyusov, Alexey Gorbatovski, Alexey Malakhov +4

T-Tech

Trust Region Distribution Matching Distillation Kullback-Leibler Divergence Agent Behavior Modeling

Ask PDF

Jul 28, 2026cs.CL

Pass the Baton: Trajectory-Relayed On-Policy Distillation

On-policy distillation (OPD) grounds token-level supervision in the student's own trajectory, yet suffers from prefix failure: once the student commits to a wrong reasoning direction, all subsequent generation builds on this deviation, producing misdirected continuations that elicit unreliable supervision and waste compute. We identify a teacher-student continuation asymmetry on failed prefixes, where the teacher tends to redirect while the student continues along the original direction, and convert it into a label-free handoff trigger in Relay On-Policy Distillation (Relay-OPD). During training, Relay-OPD constructs relay trajectories by letting the teacher briefly take over at detected trigger points to produce a teacher leg, after which the student resumes and is optimized on the resulting trajectory. A limited relay budget concentrates intervention on critical early positions while limiting departure from the student policy. With a Qwen3-4B-Instruct-2507 teacher and Qwen3-0.6B/1.7B-Non-Thinking students on eight mathematical reasoning benchmarks, Relay-OPD achieves the best or second-best results on every benchmark, outperforming standard OPD by +5.73% and the strongest baseline FastOPD by +1.49% on average for 1.7B, with consistent gains at 0.6B. Training trajectory length is reduced by over 50%.

Haolei Xu, Xiaowen Xu, Haiwen Hong +5

1Zhejiang University · 2Yuvion Team, Alibaba Group

Teacher Mathematical Reasoning Benchmarks

Ask PDF