Does On-Policy Distillation Need to Complete Every Rollout?

Zhengxin Zhang; Maosen Tang; Ning Wang; Changhao Li; Jingren Zhou; Bo Dai; Sainyam Galhotra; Alex Townsend

cs.LGOct 15, 2026

Does On-Policy Distillation Need to Complete Every Rollout?

Authors: Zhengxin Zhang, Maosen Tang, Ning Wang, Changhao Li, Jingren Zhou, Bo Dai, Sainyam Galhotra, Alex Townsend,

Organizations: Cornell University · Georgia Tech · Alibaba Group

Abstract

On-policy distillation (OPD) transfers teacher knowledge to a student through feedback on student-generated rollouts, but generating complete rollouts and computing per-token teacher feedback are computationally expensive. Existing truncation methods save computation by stopping rollouts early. However, they do not compensate for truncated token gradients, which biases the gradient estimates and changes Vanilla OPD's original training objective. Furthermore, their truncation decisions rely on assumptions about supervision quality or criteria that may not generalize across settings. We introduce Stochastic Truncated On-Policy Distillation (STOPD), which accelerates OPD by randomly truncating student rollouts and correcting the resulting sampling bias with blockwise inverse-probability weighting (IPW). At each training step, we sample a stopping position for each rollout before generation and let the student generate until that position or natural termination. We obtain teacher feedback on every generated token and use blockwise IPW to compensate for the lower probability of retaining later tokens while controlling the weights. We prove that this weighting preserves Vanilla OPD's gradient sum in expectation and derive one-step update-error bounds for SGD, Adam, and Muon. We evaluate STOPD across multiple teacher-student model pairs, task domains, and optimizers. STOPD reduces generated tokens by 25-48% while maintaining accuracy comparable to Vanilla OPD, achieving 1.3-2.2x training speedups.

Figures & tables

Figure 2: Accuracy of the 1.7B student against training step with the base model and the shared warmup in gray.

Qwen3-1.7B-Base $\leftarrow$ Qwen3-4B-Instruct-2507				Time per step (s)
Domain	Method	Tokens	k tok/s	Generation	Update	Other	Total
Math	Vanilla	9,516	3.5	697	189	56	941
	STOPD	6,255	4.5	358	119	36	513
	Speedup	1.5 $\times$	1.3 $\times$	2.0 $\times$	1.6 $\times$	1.5 $\times$	1.8 $\times$
Code	Vanilla	6,587	4.4	383	142	41	566
	STOPD	3,444	5.6	158	78	24	260

Table 1: Cost of a training step in the main setting.

Figure 3: Training dynamics of the 1.7B student with the shared warmup in gray.

Figure 4: Scaling in mathematics: accuracy for 4B and 8B students distilled from Qwen3-30B-A3B-Instruct-2507 (left, middle), and token and training-time speedups across student sizes, including the 1.7B main setting of Section 5.1 (right).

Figure 5: Training dynamics of the 4B and 8B students in mathematics with the shared warmup in gray.

Figure 6: Accuracy against training step in mathematics under AdamW in the main setting of Section 5.1 and under SGD and Muon, with the base model and each optimizer’s warmup in gray.

Figure 7: Training dynamics in mathematics (1.7B

\leftarrow

4B) under AdamW and under SGD and Muon, with each optimizer’s warmup in gray.

Figure 8: Block width in mathematics (1.7B

\leftarrow

4B): accuracy against training step with the base model and the shared warmup in gray (left), the largest weight in each training step as a box plot with its bound

L/K

above and its median beside each box (middle), and tokens and training time relative to Vanilla (right).

Figure 9: Accuracy against training step in mathematics (1.7B

\leftarrow

4B) at

B=128

256

, and

512

(left), and loss jitter (right).

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Upper bounds on Adam’s coefficients

\lambda_{\mathrm{A},j}

: distribution across coordinates and checkpoints (left), the median coordinate at each step (right), and the theoretical ceiling (dashed).

Figure 11: Muon’s coefficients

\Lambda_{\mathrm{M}}

: distribution across blocks and checkpoints (left), the median block (right), and the theoretical ceiling (dashed).

Main results: Qwen3-1.7B-Base $\leftarrow$ Qwen3-4B-Instruct-2507
			Accuracy (%) at step
Setting	Method	Base	25	50	75	100	125	150	175	200	225	250
Math	Vanilla	11.7	23.4 ±0.8	24.1 ±0.0	27.9 ±0.7	26.9 ±0.2	28.4 ±0.6	28.5 ±0.7	29.3 ±0.4	28.1 ±1.8	30.0 ±0.4	30.1 ±0.5
Math	STOPD	11.7	23.4 ±0.8	24.4 ±0.9	27.3 ±0.7	27.5 ±0.5	27.7 ±0.6	29.0 ±0.5	29.1 ±0.1	27.9 ±1.4	28.5 ±0.6	29.4 ±0.6
Code	Vanilla	21.8	27.6 ±0.2	26.4 ±0.2	25.5 ±0.3	26.5 ±0.2	27.2 ±0.3	27.6 ±0.2	27.6 ±0.3	28.1 ±0.3	28.5 ±0.1	28.5 ±0.1
Code	STOPD	21.8	27.6 ±0.2	26.3 ±0.3	25.7 ±0.4	25.8 ±0.3	26.9 ±0.3	27.4 ±0.3	27.2 ±0.1	27.5 ±0.6	27.8 ±0.4	28.1 ±0.4

Appendix

Table 2: Accuracy (%) every 25 steps, mean

\pm

standard deviation, from the base model and the shared step-25 Vanilla warmup to 250 steps.

Mathematics: Qwen3-1.7B-Base $\leftarrow$ Qwen3-4B-Instruct-2507
			Accuracy (%) at step
Optimizer	Method	Base	25	50	75	100	125	150	175	200	225	250
AdamW	Vanilla	11.7	23.4 ±0.8	24.1 ±0.0	27.9 ±0.7	26.9 ±0.2	28.4 ±0.6	28.5 ±0.7	29.3 ±0.4	28.1 ±1.8	30.0 ±0.4	30.1 ±0.5
AdamW	STOPD	11.7	23.4 ±0.8	24.4 ±0.9	27.3 ±0.7	27.5 ±0.5	27.7 ±0.6	29.0 ±0.5	29.1 ±0.1	27.9 ±1.4	28.5 ±0.6	29.4 ±0.6
SGD	Vanilla	11.7	22.6 ±0.5	24.4 ±0.3	24.3 ±0.3	24.5 ±1.6	24.9 ±0.1	24.4 ±0.6	24.9 ±0.2	25.2 ±0.8	25.3 ±0.3	25.2 ±0.5
SGD	STOPD	11.7	22.6 ±0.5	24.2 ±0.5	24.6 ±0.2	24.7 ±0.3	24.2 ±0.2	25.0 ±0.3	24.1 ±0.4	25.2 ±0.2	25.2 ±0.6	25.1 ±0.6

Appendix

Table 3: Accuracy (%) every 25 steps under each optimizer in mathematics, mean

\pm

standard deviation.

Mathematics: Qwen3-1.7B-Base $\leftarrow$ Qwen3-4B-Instruct-2507
			Accuracy (%) at step
$K$	Method	Base	25	50	75	100	125	150	175	200	225	250
	Vanilla	11.7	23.4 ±0.8	24.1 ±0.0	27.9 ±0.7	26.9 ±0.2	28.4 ±0.6	28.5 ±0.7	29.3 ±0.4	28.1 ±1.8	30.0 ±0.4	30.1 ±0.5
32	STOPD			24.1 ±0.2	27.3 ±0.4	27.1 ±0.5	27.6 ±0.1	28.2 ±0.6	29.2 ±0.4	29.2 ±0.5	28.8 ±0.3	29.8 ±0.7
128	STOPD			24.4 ±0.9	27.3 ±0.7	27.5 ±0.5	27.7 ±0.6	29.0 ±0.5	29.1 ±0.1	27.9 ±1.4	28.5 ±0.6	29.4 ±0.6
512	STOPD			24.1 ±0.7	27.7 ±0.5	27.7 ±0.5	27.5 ±0.8	28.8 ±0.2	28.9 ±0.7	28.7 ±0.8	28.8 ±1.1	29.7 ±0.2

Appendix

Table 4: Accuracy (%) every 25 steps for each block width

K

in mathematics, mean

\pm

standard deviation.

Mathematics: Qwen3-1.7B-Base $\leftarrow$ Qwen3-4B-Instruct-2507
			Accuracy (%) at step
$B$	Method	Base	25	50	75	100
128	Vanilla	11.7	23.1 ±0.3	23.5 ±0.6	27.6 ±0.1	27.8 ±0.3
128	STOPD	11.7	23.1 ±0.3	24.3 ±0.4	27.9 ±0.6	28.4 ±0.8
256	Vanilla	11.7	23.4 ±0.8	24.1 ±0.0	27.9 ±0.7	26.9 ±0.2
256	STOPD	11.7	23.4 ±0.8	24.4 ±0.9	27.3 ±0.7	27.5 ±0.5

Appendix

Table 5: Accuracy (%) every 25 steps for each batch size

B

in mathematics, mean

\pm

standard deviation.

Mathematics: Qwen3-1.7B-Base $\leftarrow$ Qwen3-4B-Instruct-2507
		Accuracy (%) at step
Method	Base	25	50	75	100	125	150	175	200	225	250
Vanilla	11.7	23.4 ±0.8	24.1 ±0.0	27.9 ±0.7	26.9 ±0.2	28.4 ±0.6	28.5 ±0.7	29.3 ±0.4	28.1 ±1.8	30.0 ±0.4	30.1 ±0.5
STOPD			24.4 ±0.9	27.3 ±0.7	27.5 ±0.5	27.7 ±0.6	29.0 ±0.5	29.1 ±0.1	27.9 ±1.4	28.5 ±0.6	29.4 ±0.6
ESR			24.5 ±0.1	23.7 ±1.2	23.4 ±0.4	24.0 ±0.4	23.4 ±0.1	23.2 ±0.4	23.0 ±0.4	23.2 ±0.5	23.5 ±0.6

Appendix

Table 6: Accuracy (%) every 25 steps for ESR in mathematics, mean

\pm

standard deviation.

Mathematics: Qwen3-1.7B-Base $\leftarrow$ Qwen3-4B-Instruct-2507
		Accuracy (%) at step
Method	Base	25	50	75	100	125	150	175	200	225	250
Vanilla	11.7	23.4 ±0.8	24.1 ±0.0	27.9 ±0.7	26.9 ±0.2	28.4 ±0.6	28.5 ±0.7	29.3 ±0.4	28.1 ±1.8	30.0 ±0.4	30.1 ±0.5
STOPD			24.4 ±0.9	27.3 ±0.7	27.5 ±0.5	27.7 ±0.6	29.0 ±0.5	29.1 ±0.1	27.9 ±1.4	28.5 ±0.6	29.4 ±0.6
STOPD, $w_{t}=1$			23.8 ±0.2	27.4 ±0.1	26.9 ±0.4	27.7 ±0.4	27.9 ±0.5	29.2 ±0.6	29.0 ±0.5	29.6 ±0.3	29.1 ±0.3
Mathematics: Qwen3-4B-Base $\leftarrow$ Qwen3-30B-A3B-Instruct-2507

Appendix

Table 7: Accuracy (%) every 25 steps for STOPD with

w_{t}=1

in mathematics, mean

\pm

standard deviation.

				Time per step (s)
Student	Method	Tokens	k tok/s	Generation	Update	Other	Total
4B	Vanilla	10,005	3.4	753	360	95	1209
	STOPD	6,511	4.3	392	219	61	672
	Speedup	1.5 $\times$	1.3 $\times$	1.9 $\times$	1.6 $\times$	1.6 $\times$	1.8 $\times$
8B	Vanilla	9,432	5.6	434	303	81	818
	STOPD	6,276	6.5	246	190	54	490

Appendix

Table 8: Cost of a training step for the 4B and 8B students in mathematics.

Qwen3-1.7B-Base $\leftarrow$ Qwen3-4B-Instruct-2507				Time per step (s)
Optimizer	Method	Tokens	k tok/s	Generation	Update	Other	Total
AdamW	Vanilla	9,516	3.5	697	189	56	941
	STOPD	6,255	4.5	358	119	36	513
	Speedup	1.5 $\times$	1.3 $\times$	2.0 $\times$	1.6 $\times$	1.5 $\times$	1.8 $\times$
SGD	Vanilla	9,278	2.9	810	185	50	1045
	STOPD	5,474	4.1	342	109	34	485

Appendix

Table 9: Cost of a training step under each optimizer in mathematics.

Qwen3-1.7B-Base $\leftarrow$ Qwen3-4B-Instruct-2507				Time per step (s)
$K$	Method	Tokens	k tok/s	Generation	Update	Other	Total
	Vanilla	9,516	3.5	697	189	56	941
32	STOPD	6,227	4.4	366	124	41	531
32	Speedup	1.5 $\times$	1.2 $\times$	1.9 $\times$	1.5 $\times$	1.4 $\times$	1.8 $\times$
128	STOPD	6,255	4.5	358	119	36	513
128	Speedup	1.5 $\times$	1.3 $\times$	2.0 $\times$	1.6 $\times$	1.5 $\times$	1.8 $\times$

Appendix

Table 10: Cost of a training step for each block width

K

in mathematics.

Qwen3-1.7B-Base $\leftarrow$ Qwen3-4B-Instruct-2507				Time per step (s)
Batch size	Method	Tokens	k tok/s	Generation	Update	Other	Total
128	Vanilla	9,403	3.0	399	91	26	516
	STOPD	5,781	4.0	185	53	17	256
	Speedup	1.6 $\times$	1.3 $\times$	2.2 $\times$	1.7 $\times$	1.5 $\times$	2.0 $\times$
256	Vanilla	9,650	3.1	793	195	57	1045
	STOPD	5,940	4.3	356	112	33	501

Appendix

Table 11: Cost of a training step for each batch size in mathematics over the first 100 steps.

Figure 12: ESR in mathematics: accuracy (left), response length at evaluation (middle), and truncated share (right).

Figure 13: STOPD with every weight set to

w_{t}=1

, in mathematics with the 1.7B student (top) and the 4B student (bottom): accuracy (left), response length at evaluation (middle), and truncated share (right).

Figure 14: Mathematics, Qwen3-1.7B-Base

\leftarrow

Qwen3-4B-Instruct-2507: accuracy per benchmark against training step.

Figure 15: Code, Qwen3-1.7B-Base

\leftarrow

Qwen3-4B-Instruct-2507: pass rate per benchmark against training step.

Figure 16: Mathematics, Qwen3-4B-Base

\leftarrow

Qwen3-30B-A3B-Instruct-2507: accuracy per benchmark against training step.

Figure 17: Mathematics, Qwen3-8B-Base

\leftarrow

Qwen3-30B-A3B-Instruct-2507: accuracy per benchmark against training step.

Figure 18: Mathematics under SGD, Qwen3-1.7B-Base

\leftarrow

Qwen3-4B-Instruct-2507: accuracy per benchmark against training step.

Figure 19: Mathematics under Muon, Qwen3-1.7B-Base

\leftarrow

Qwen3-4B-Instruct-2507: accuracy per benchmark against training step.

Figure 20: Block width in mathematics, Qwen3-1.7B-Base

\leftarrow

Qwen3-4B-Instruct-2507: accuracy per benchmark against training step.

Figure 21: Mathematics at

B=128

, Qwen3-1.7B-Base

\leftarrow

Qwen3-4B-Instruct-2507: accuracy per benchmark against training step.

Figure 22: Mathematics at

B=256

, Qwen3-1.7B-Base

\leftarrow

Qwen3-4B-Instruct-2507: accuracy per benchmark against training step.

Figure 23: Mathematics at

B=512

, Qwen3-1.7B-Base

\leftarrow

Qwen3-4B-Instruct-2507: accuracy per benchmark against training step.

Figure 24: ESR in mathematics, Qwen3-1.7B-Base

\leftarrow

Qwen3-4B-Instruct-2507: accuracy per benchmark against training step.

Figure 25: STOPD with

w_{t}=1

in mathematics, Qwen3-1.7B-Base

\leftarrow

Qwen3-4B-Instruct-2507: accuracy per benchmark against training step.

Figure 26: STOPD with

w_{t}=1

in mathematics, Qwen3-4B-Base

\leftarrow

Qwen3-30B-A3B-Instruct-2507: accuracy per benchmark against training step.

On-Policy Distillation

On-policy distillation trains a model on its own outputs using teacher guidance, aligning supervision with the states it encounters at deployment.

See all 511 papers

When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better
Oct 8, 2026Siyan Zhao, Yonggan Fu, Jindong Jiang +8Language Model Distillation On-Policy Distillation
NVIDIA
Proposing Better Rollouts for On-Policy Distillation
Oct 27, 2026Sirui Chen, Yunzhe Qi, Haobo Xu +4Language Model Distillation On-Policy Distillation
University of Illinois Urbana-Champaign · Stanford University
Are Full Rollouts Necessary for On-Policy Distillation?
May 29, 2026Yaocheng Zhang, Jiajun Chai, Yuqian Fu +7RL for Language Model Reasoning Language Model Distillation
Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · Meituan +1

Oct 8, 2026J/K move · Enter open · S save

When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better

Siyan Zhao, Yonggan Fu, Jindong Jiang, Shih-Yang Liu+7

NVIDIA

Language Model Distillation On-Policy Distillation Knowledge Distillation Teacher-Student Learning

Open paper AskPDF arXiv

On-policy distillation (OPD) has become increasingly popular for transferring teacher capabilities to student models. In this work, we ask a critical research question: Is on-policy sampling always beneficial for distilling arbitrary teacher-student pairs? We show that a simple alternative, Semi-OPD, which distills from offline rollouts generated by the initial student, can often outperform OPD in both accuracy and training efficiency. Across 17 teacher-student pairs ranging from 1.5B to 235B parameters, Semi-OPD outperforms OPD in 14 cases, with up to +13.6% accuracy and 11.4x training speedup. We further find that the choice between OPD and Semi-OPD depends on the alignment between the initial teacher and student, quantified by an output-token overlap ratio: OPD is beneficial only when the two are highly aligned with high overlap ratios. Our deeper investigation suggests that effective distillation requires on-policyness w.r.t. both the student and the teacher. For misaligned pairs, student rollouts can become increasingly off-policy w.r.t. the teacher as context length grows, weakening the distillation signal. In contrast, Semi-OPD is often more stable, as it distills on shorter contexts while covering full trajectories and exposing the student to more teacher-preferred tokens. Beyond proposing Semi-OPD as an efficient alternative, our work motivates the community to rethink when to use OPD and to study stronger OPD variants with meaningful teacher-student pairs.

Abstract

On-Policy Distillation

Explore similar work

When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better

Proposing Better Rollouts for On-Policy Distillation

Are Full Rollouts Necessary for On-Policy Distillation?