On-policy distillation (OPD) transfers teacher knowledge to a student through feedback on student-generated rollouts, but generating complete rollouts and computing per-token teacher feedback are computationally expensive. Existing truncation methods save computation by stopping rollouts early. However, they do not compensate for truncated token gradients, which biases the gradient estimates and changes Vanilla OPD's original training objective. Furthermore, their truncation decisions rely on assumptions about supervision quality or criteria that may not generalize across settings. We introduce Stochastic Truncated On-Policy Distillation (STOPD), which accelerates OPD by randomly truncating student rollouts and correcting the resulting sampling bias with blockwise inverse-probability weighting (IPW). At each training step, we sample a stopping position for each rollout before generation and let the student generate until that position or natural termination. We obtain teacher feedback on every generated token and use blockwise IPW to compensate for the lower probability of retaining later tokens while controlling the weights. We prove that this weighting preserves Vanilla OPD's gradient sum in expectation and derive one-step update-error bounds for SGD, Adam, and Muon. We evaluate STOPD across multiple teacher-student model pairs, task domains, and optimizers. STOPD reduces generated tokens by 25-48% while maintaining accuracy comparable to Vanilla OPD, achieving 1.3-2.2x training speedups.
Figures & tables
Figure 2: Accuracy of the 1.7B student against training step with the base model and the shared warmup in gray.
Qwen3-1.7B-Base ← Qwen3-4B-Instruct-2507
Time per step (s)
Domain
Method
Tokens
k tok/s
Generation
Update
Other
Total
Math
Vanilla
9,516
3.5
697
189
56
941
STOPD
6,255
4.5
358
119
36
513
Speedup
1.5 ×
1.3 ×
2.0 ×
1.6 ×
1.5 ×
1.8 ×
Code
Vanilla
6,587
4.4
383
142
41
566
STOPD
3,444
5.6
158
78
24
260
Table 1: Cost of a training step in the main setting.
Figure 3: Training dynamics of the 1.7B student with the shared warmup in gray.
Figure 4: Scaling in mathematics: accuracy for 4B and 8B students distilled from Qwen3-30B-A3B-Instruct-2507 (left, middle), and token and training-time speedups across student sizes, including the 1.7B main setting of Section 5.1 (right).
Figure 5: Training dynamics of the 4B and 8B students in mathematics with the shared warmup in gray.
Figure 6: Accuracy against training step in mathematics under AdamW in the main setting of Section 5.1 and under SGD and Muon, with the base model and each optimizer’s warmup in gray.
Figure 7: Training dynamics in mathematics (1.7B ← 4B) under AdamW and under SGD and Muon, with each optimizer’s warmup in gray.
Figure 8: Block width in mathematics (1.7B ← 4B): accuracy against training step with the base model and the shared warmup in gray (left), the largest weight in each training step as a box plot with its bound L/K above and its median beside each box (middle), and tokens and training time relative to Vanilla (right).
Figure 9: Accuracy against training step in mathematics (1.7B ← 4B) at B=128 , 256 , and 512 (left), and loss jitter (right).
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Upper bounds on Adam’s coefficients λA,j : distribution across coordinates and checkpoints (left), the median coordinate at each step (right), and the theoretical ceiling (dashed).
Figure 11: Muon’s coefficients ΛM : distribution across blocks and checkpoints (left), the median block (right), and the theoretical ceiling (dashed).
Accuracy (%) at step
Setting
Method
Base
25
50
75
100
125
150
175
200
225
250
Main results: Qwen3-1.7B-Base ← Qwen3-4B-Instruct-2507
Math
Vanilla
11.7
23.4 ±0.8
24.1 ±0.0
27.9 ±0.7
26.9 ±0.2
28.4 ±0.6
28.5 ±0.7
29.3 ±0.4
28.1 ±1.8
30.0 ±0.4
30.1 ±0.5
STOPD
24.4 ±0.9
27.3 ±0.7
27.5 ±0.5
27.7 ±0.6
29.0 ±0.5
29.1 ±0.1
27.9 ±1.4
28.5 ±0.6
29.4 ±0.6
Code
Vanilla
21.8
27.6 ±0.2
26.4 ±0.2
25.5 ±0.3
26.5 ±0.2
27.2 ±0.3
27.6 ±0.2
27.6 ±0.3
28.1 ±0.3
28.5 ±0.1
28.5 ±0.1
STOPD
26.3 ±0.3
25.7 ±0.4
25.8 ±0.3
26.9 ±0.3
27.4 ±0.3
27.2 ±0.1
27.5 ±0.6
27.8 ±0.4
28.1 ±0.4
Appendix
Table 2: Accuracy (%) every 25 steps, mean ± standard deviation, from the base model and the shared step-25 Vanilla warmup to 250 steps.
Table 7: Accuracy (%) every 25 steps for STOPD with wt=1 in mathematics, mean ± standard deviation.
Time per step (s)
Student
Method
Tokens
k tok/s
Generation
Update
Other
Total
4B
Vanilla
10,005
3.4
753
360
95
1209
STOPD
6,511
4.3
392
219
61
672
Speedup
1.5 ×
1.3 ×
1.9 ×
1.6 ×
1.6 ×
1.8 ×
8B
Vanilla
9,432
5.6
434
303
81
818
STOPD
6,276
6.5
246
190
54
490
Appendix
Table 8: Cost of a training step for the 4B and 8B students in mathematics.
Qwen3-1.7B-Base ← Qwen3-4B-Instruct-2507
Time per step (s)
Optimizer
Method
Tokens
k tok/s
Generation
Update
Other
Total
AdamW
Vanilla
9,516
3.5
697
189
56
941
STOPD
6,255
4.5
358
119
36
513
Speedup
1.5 ×
1.3 ×
2.0 ×
1.6 ×
1.5 ×
1.8 ×
SGD
Vanilla
9,278
2.9
810
185
50
1045
STOPD
5,474
4.1
342
109
34
485
Appendix
Table 9: Cost of a training step under each optimizer in mathematics.
Qwen3-1.7B-Base ← Qwen3-4B-Instruct-2507
Time per step (s)
K
Method
Tokens
k tok/s
Generation
Update
Other
Total
Vanilla
9,516
3.5
697
189
56
941
32
STOPD
6,227
4.4
366
124
41
531
Speedup
1.5 ×
1.2 ×
1.9 ×
1.5 ×
1.4 ×
1.8 ×
128
STOPD
6,255
4.5
358
119
36
513
Speedup
1.5 ×
1.3 ×
2.0 ×
1.6 ×
1.5 ×
1.8 ×
Appendix
Table 10: Cost of a training step for each block width K in mathematics.
Qwen3-1.7B-Base ← Qwen3-4B-Instruct-2507
Time per step (s)
Batch size
Method
Tokens
k tok/s
Generation
Update
Other
Total
128
Vanilla
9,403
3.0
399
91
26
516
STOPD
5,781
4.0
185
53
17
256
Speedup
1.6 ×
1.3 ×
2.2 ×
1.7 ×
1.5 ×
2.0 ×
256
Vanilla
9,650
3.1
793
195
57
1045
STOPD
5,940
4.3
356
112
33
501
Appendix
Table 11: Cost of a training step for each batch size in mathematics over the first 100 steps.
Figure 12: ESR in mathematics: accuracy (left), response length at evaluation (middle), and truncated share (right).
Figure 13: STOPD with every weight set to wt=1 , in mathematics with the 1.7B student (top) and the 4B student (bottom): accuracy (left), response length at evaluation (middle), and truncated share (right).
Figure 14: Mathematics, Qwen3-1.7B-Base ← Qwen3-4B-Instruct-2507: accuracy per benchmark against training step.
Figure 15: Code, Qwen3-1.7B-Base ← Qwen3-4B-Instruct-2507: pass rate per benchmark against training step.
Figure 16: Mathematics, Qwen3-4B-Base ← Qwen3-30B-A3B-Instruct-2507: accuracy per benchmark against training step.
Figure 17: Mathematics, Qwen3-8B-Base ← Qwen3-30B-A3B-Instruct-2507: accuracy per benchmark against training step.
Figure 18: Mathematics under SGD, Qwen3-1.7B-Base ← Qwen3-4B-Instruct-2507: accuracy per benchmark against training step.
Figure 19: Mathematics under Muon, Qwen3-1.7B-Base ← Qwen3-4B-Instruct-2507: accuracy per benchmark against training step.
Figure 20: Block width in mathematics, Qwen3-1.7B-Base ← Qwen3-4B-Instruct-2507: accuracy per benchmark against training step.
Figure 21: Mathematics at B=128 , Qwen3-1.7B-Base ← Qwen3-4B-Instruct-2507: accuracy per benchmark against training step.
Figure 22: Mathematics at B=256 , Qwen3-1.7B-Base ← Qwen3-4B-Instruct-2507: accuracy per benchmark against training step.
Figure 23: Mathematics at B=512 , Qwen3-1.7B-Base ← Qwen3-4B-Instruct-2507: accuracy per benchmark against training step.
Figure 24: ESR in mathematics, Qwen3-1.7B-Base ← Qwen3-4B-Instruct-2507: accuracy per benchmark against training step.
Figure 25: STOPD with wt=1 in mathematics, Qwen3-1.7B-Base ← Qwen3-4B-Instruct-2507: accuracy per benchmark against training step.
Figure 26: STOPD with wt=1 in mathematics, Qwen3-4B-Base ← Qwen3-30B-A3B-Instruct-2507: accuracy per benchmark against training step.
On-policy distillation (OPD) has become increasingly popular for transferring teacher capabilities to student models. In this work, we ask a critical research question: Is on-policy sampling always beneficial for distilling arbitrary teacher-student pairs? We show that a simple alternative, Semi-OPD, which distills from offline rollouts generated by the initial student, can often outperform OPD in both accuracy and training efficiency. Across 17 teacher-student pairs ranging from 1.5B to 235B parameters, Semi-OPD outperforms OPD in 14 cases, with up to +13.6% accuracy and 11.4x training speedup. We further find that the choice between OPD and Semi-OPD depends on the alignment between the initial teacher and student, quantified by an output-token overlap ratio: OPD is beneficial only when the two are highly aligned with high overlap ratios. Our deeper investigation suggests that effective distillation requires on-policyness w.r.t. both the student and the teacher. For misaligned pairs, student rollouts can become increasingly off-policy w.r.t. the teacher as context length grows, weakening the distillation signal. In contrast, Semi-OPD is often more stable, as it distills on shorter contexts while covering full trajectories and exposing the student to more teacher-preferred tokens. Beyond proposing Semi-OPD as an efficient alternative, our work motivates the community to rethink when to use OPD and to study stronger OPD variants with meaningful teacher-student pairs.
On-policy distillation (OPD) provides dense teacher feedback along student-generated rollouts rather than fixed teacher traces and has emerged as a promising post-training paradigm. However, standard OPD typically generates full rollouts during training, which is computationally expensive and may expose the student to unreliable teacher feedback at late rollout positions, especially during early training. We identify the rollout horizon as a key bottleneck in OPD that substantially impacts training efficiency. Unlike Reinforcement Learning with Verifiable Rewards (RLVR), OPD does not require a final answer reward to provide learning signals. Therefore, full rollouts may not always be necessary for OPD. Motivated by this insight, we propose two simple horizon-control strategies: Progressive OPD (POPD), which gradually expands the rollout horizon during training, and Truncated OPD (TOPD), which permanently performs distillation on reliable truncated rollouts. Experiments on mathematical reasoning show that POPD improves the training efficiency of OPD by up to 3×, while TOPD matches OPD performance using only 10% of the rollout horizon, leading to substantial wall-clock and memory reductions. These results demonstrate that controlling the rollout horizon offers a simple and practical path to more efficient OPD.
Yaocheng Zhang, Jiajun Chai, Yuqian Fu +7
Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · Meituan +1
On-policy distillation (OPD) is increasingly used in LLM post-training because it can leverage a teacher model to provide dense supervision on student rollouts. The standard implementation, however, usually reduces distribution matching to a sampled-token log-ratio, which can make the learning signal fragile on long rollouts whose prefixes drift away from the teacher's typical support. We revisit this formulation from both theoretical and implementation perspectives. Theoretically, token-level OPD is biased relative to sequence-level reverse-KL minimization, but admits a substantially tighter worst-case variance bound; a controlled synthetic study further shows that stronger future-reward coupling increases gradient variance and destabilizes training. Empirically, we identify three failure modes of sampled-token OPD: imbalanced token-level supervision, unreliable teacher guidance on student-generated prefixes, and tokenizer or special-token mismatch. These findings motivate teacher top-K local support matching, a truncated reverse-KL objective that compares teacher and student distributions over a teacher-supported token set at each prefix, together with top-p rollout sampling and special-token masking. Across single-task reasoning and multi-task benchmarks spanning agentic and reasoning settings, this objective improves optimization stability and yields a +19.8% performance gain over standard sampled-token OPD baselines, providing a practical recipe for more stable on-policy distillation.
Yuqian Fu, Haohuan Huang, Kaiwen Jiang +4
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, UCAS · Fudan University +1