Flash-OPD: Fast On-Policy Distillation

Wei Chen; Junle Chen; Yitong Yang; Zhaoyang Xu; Jiaxin Lin; Yuxuan Liang; Xiaofang Zhou; Kai Wang

cs.LGOct 5, 2026

Flash-OPD: Fast On-Policy Distillation

Authors: Wei Chen, Junle Chen, Yitong Yang, Zhaoyang Xu, Jiaxin Lin, Yuxuan Liang, Xiaofang Zhou, Kai Wang, +1 more

Organizations: Tencent Hy · HKUST(GZ) · HKUST

Abstract

On-policy distillation (OPD) provides dense teacher supervision on student-generated trajectories, but generating and evaluating long rollouts incurs substantial training cost. Existing acceleration methods reduce this cost through open-loop rollout schedules or closed-loop horizon adaptation. However, supervision compatibility can vary substantially across trajectories, making a single rollout horizon difficult to match their heterogeneous reliable lengths: an overly short horizon may truncate useful supervision, while an overly long one wastes computation beyond reliable regions. Our key insight is that the trajectory-specific reliability boundary need not be predicted before generation. By viewing reliability as the first-passage of accumulated low teacher--student compatibility events, the boundary is inherently unknown before sampling, yet whether it has been reached can be determined exactly from the observed prefix. Building on this insight, we propose Flash-OPD, which shifts from rollout-horizon control to adaptive trajectory-level boundary verification. Flash-OPD interleaves cached generation with teacher verification and independently stops each trajectory according to its observed compatibility events. To reduce verification overhead, the recent event rate is used only to schedule the next verification point, while the actual stopping decision always relies on the exact cumulative count. This separation prevents estimation errors from causing premature termination while enabling efficient verification during generation. Extensive experiments across diverse datasets and teacher--student settings show that Flash-OPD achieves $2.2\times$ -- $7.5\times$ speedups over standard OPD while maintaining or improving accuracy.

Figures & tables

Figure 1 : From a shared rollout horizon to adaptive verification. Existing OPD acceleration methods commit to a shared horizon before trajectory sampling, causing premature truncation or redundant generation when usable lengths vary across trajectories; Flash-OPD instead verifies reliability during generation and retires each trajectory at its own boundary, ensuring reliability and efficiency.

	Length decision		Allocation
Method	Driven by	Granularity	Spectrum	All seqs. trained	Stacks on a batch ctrl.
POPD ( Zhang et al., 2026b )	training step	batch	single	✓	✗
TOPD ( Zhang et al., 2026b )	training step	batch	single	✓	✗
Prefix OPD ( Zhang et al., 2026a )	training step	batch	single	✓	✗
ADWIN ( Liang et al., 2026 )	gradient cosine	batch	single	✓	✗
Prune-OPD ( Yang et al., 2026c )	top- $k$ overlap	batch	single	✓	✗

Table 1: Comparison of rollout-control methods for OPD. Driven by : what the length decision consumes. Granularity : the unit the decision is made per. Spectrum : the set of realized lengths within one step. All seqs. trained : whether every sampled sequence contributes to the loss. Stacks on a batch ctrl. : can run under an independent batch-level window controller instead of replacing it.

Figure 2 : Heterogeneous and evolving reliability lengths expose the mismatch of fixed and adaptive batch-shared horizons (left and middle), while our method Flash-OPD substantially reduces excess rollout budget without prematurely truncating reliable prefixes in the illustrated setting (right).

Figure 3 : A schematic diagram of our rollout case, which adaptively adjusts verification positions according to the estimated local event rate and stops once the exact reliability boundary is reached.

Method	AMC23	AIME24	AIME25	HMMT24	HMMT25	Average	Time	Speed Up
Student / Teacher: DeepSeek-R1-Distill-Qwen-1.5B / JustRL-DeepSeek-1.5B
OPD	78.16	46.46	32.92	24.79	18.12	40.09	7.46h	-
TOPD	79.67	45.83	33.33	24.79	19.38	40.60	3.51h	2.12 $\times$
Prune-OPD	78.92	44.79	31.25	22.92	20.62	39.70	3.36h	2.22 $\times$
Flash-OPD	79.37	46.46	32.92	22.29	20.83	40.37	3.35h	2.23 $\times$
Student / Teacher: DeepSeek-R1-Distill-Qwen-1.5B / DeepSeek-R1-Distill-Qwen-7B

Table 2: Overall performance-efficiency comparison under different benchmarks and settings.

Figure 4 : Mean wall-clock seconds per training step.

Figure 5 : Wall-clock training dynamics of vanilla OPD and our Flash-OPD across four teacher-student model pairs and five benchmarks. We perform a benchmark evaluation every 20 steps.

Figure 6 : Transferability of Flash-OPD across evaluation distributions and model generations.

Figure 7 : With a fixed student, Flash-OPD incurs smaller step-time increases than OPD while maintaining stable accuracy and consistently short rollouts as teacher size grows from 4B to 32B.

Figure 8 : Sensitivity of Flash-OPD to the drift threshold $\eta$ , allowance $B$ , and step discount $\rho$ .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Domain	Split used	#Problems	Evaluation budget
Training corpus
DAPO-Math-17K ( Yu et al., 2026 )	Competition mathematics	train (all)	17,917	—
In-domain evaluation
AMC23	Competition mathematics	test	83	$31744$ tok., mean@16
AIME24	Competition mathematics	test	30	$31744$ tok., mean@16
AIME25	Competition mathematics	test	30	$31744$ tok., mean@16

Appendix

Table 3: The training corpus and the five in-domain benchmarks are competition mathematics; the four out-of-domain benchmarks are not, and GPQA is not mathematics at all.

Student	Teacher	Relationship	Max resp.	$k_{0}$	$W$	$k_{\min}$	$L_{\max}$
R1-Distill-Qwen-1.5B	JustRL-DeepSeek-1.5B	Same-size RL-improved copy of the student	12288	1024	1024	512	4400
R1-Distill-Qwen-1.5B	R1-Distill-Qwen-7B	Larger same-family reasoning model	12288	128	128	16	12288
Qwen3-1.7B-Base	Qwen3-4B (Non-thinking)	Stronger same-family teacher, base student	8192	128	128	16	8192
Qwen3-4B-Base	Qwen3-4B (Non-thinking)	Same-size post-trained teacher, base student	12288	128	128	16	12288

Appendix

Table 4: The four teacher–student pairs and the controller each one runs.

Parameter	Value
Optimisation
Train batch size (prompts per step)	$64$
PPO mini-batch size	$64$ (one update per step)
Samples per prompt $n$	$4$
Learning rate	$1\times 10^{-6}$ , constant
Micro-batch per GPU	$1$ , dynamic batching enabled

Appendix

Table 5: Parameter setting of our method Flash-OPD .

Method	Step-time attribution (s/step)					Tokens vs. time
Method	Rollout	Teacher	Log-prob	Update	Step	Length	Token $\downarrow$	Step $\downarrow$
R1-1.5B $\leftarrow$ JustRL-1.5B
OPD	70.3	11.2	10.5	31.5	132.3	6122	–	–
Flash-OPD	26.2	14.2	0.0	14.7	60.0	2708	2.3 $\times$	2.20 $\times$
R1-1.5B $\leftarrow$ R1-7B
OPD	88.8	40.4	13.3	42.4	194.3	7961	–	–

Appendix

Table 6: Where the speedup comes from: mean wall-clock seconds per training step.

Pair	Retained length $\hat{E}$			Stopping rule (of 256)			Verification cost
Pair	min	mean	max	allowance	$L_{\max}$	EOS	rounds	amplif.	overshoot
R1-1.5B $\leftarrow$ JustRL-1.5B	830	2708	3509	130	96	30	2.64	1.54 $\times$	231 (8.5%)
R1-1.5B $\leftarrow$ R1-7B	335	585	1052	256	0	0	6.23	2.41 $\times$	26 (4.5%)
Qwen3-1.7B-Base $\leftarrow$ Qwen3-4B	126	304	672	243	0	13	4.98	2.23 $\times$	14 (4.5%)
Qwen3-4B-Base $\leftarrow$ Qwen3-4B	293	569	2117	250	0	6	6.23	2.19 $\times$	62 (10.9%)

Appendix

Table 7: Mechanism counters, averaged over all training steps.

Figure 9 : Retained lengths over training.

Pair	Method	MATH-500	Minerva	Olympiad	GPQA	Avg.	$\Delta$
R1-1.5B $\leftarrow$ JustRL-1.5B	OPD	90.40	38.40	61.60	36.93	56.83
R1-1.5B $\leftarrow$ JustRL-1.5B	Flash-OPD	90.60	38.10	58.70	34.39	55.45	tie
R1-1.5B $\leftarrow$ R1-7B	OPD	85.20	34.50	51.40	11.80	45.73
R1-1.5B $\leftarrow$ R1-7B	Flash-OPD	85.20	34.30	50.00	15.23	46.18	tie
Qwen3-1.7B-Base $\leftarrow$ Qwen3-4B	OPD	57.60	22.80	26.80	15.36	30.64
Qwen3-1.7B-Base $\leftarrow$ Qwen3-4B	Flash-OPD	59.80	23.30	29.30	13.96	31.59	tie

Appendix

Table 8: Out-of-domain transfer.

Pair	Method	AMC23	AIME24	AIME25	HMMT24	HMMT25	Avg.	Time	Speed Up
R1-1.5B $\leftarrow$ Qwen3-4B	Student	63.35	26.80	20.00	12.40	12.30	26.97	–	–
	OPD	60.02	31.25	22.92	12.92	14.58	28.34	12.02h	1.00 $\times$
	Flash-OPD	64.76	27.92	22.08	13.96	13.54	28.45	1.38h	8.71 $\times$

Appendix

Table 9: Cross-generation transfer.

Explore similar work

Adaptive FastOPD: Progress-Aware Rollout Horizon Expansion for Efficient On-Policy Distillation
Jul 31, 2026Qian Tan, Huaifei Liang, Xuanyu Zhu +2Efficient On-Policy Distillation Autoregressive Rollout
University of Science and Technology of China · 3Shanghai Jiao Tong University · 2Shanghai Artificial Intelligence Laboratory
Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rollouts
Jun 20, 2026Qingfei Zhao, Huan Song, Shuyu Tian +2Unsupervised On-Policy Self-Distillation Rollout
TeleAI · Shanghai Jiao Tong University
Are Full Rollouts Necessary for On-Policy Distillation?
May 29, 2026Yaocheng Zhang, Jiajun Chai, Yuqian Fu +7Efficient On-Policy Distillation Parallel Rollouts
Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · Meituan +1

Jul 31, 2026, cs.LGJ/K move · Enter open · S save

Adaptive FastOPD: Progress-Aware Rollout Horizon Expansion for Efficient On-Policy Distillation

Qian Tan, Huaifei Liang, Xuanyu Zhu, Lei Jiang+1

University of Science and Technology of China · 3Shanghai Jiao Tong University · 2Shanghai Artificial Intelligence Laboratory

Efficient On-Policy Distillation Autoregressive Rollout Decision Thresholds

Open paper AskPDF arXiv

On-policy distillation (OPD) provides dense teacher supervision along student-generated trajectories, but its online rollout process incurs substantial computational cost, particularly when a few long responses delay batch completion. Existing acceleration methods typically control rollout length using fixed budgets or absolute teacher--student agreement thresholds, which may not reflect learning progress across different models and training stages. We propose Adaptive FastOPD, a progress-aware strategy that expands the rollout horizon only when learning near the current boundary region has plateaued and the current horizon is sufficiently utilized. The former is determined from four teacher--student signals measured relative to their values upon entering each horizon, making expansion responsive to stage-specific progress rather than a predefined step interval or an absolute threshold on the raw agreement signals, while the latter prevents a small number of long responses from triggering increases in rollout cost. Across two teacher--student pairs, Adaptive FastOPD achieves the highest average performance while reducing training time by 49.1--71.2% relative to OPD 15K, and remains robust across a range of hyperparameter settings.

Abstract

Explore similar work

Adaptive FastOPD: Progress-Aware Rollout Horizon Expansion for Efficient On-Policy Distillation

Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rollouts

Are Full Rollouts Necessary for On-Policy Distillation?