Asynchronous On-Policy Self-Distillation

Xidong Wu; Qi Qi; Yuqiong Ji; Shangqian Gao

cs.LGOct 19, 2026

Asynchronous On-Policy Self-Distillation

Authors: Xidong Wu, Qi Qi, Yuqiong Ji, Shangqian Gao

Organizations: University of Pittsburgh · University of Iowa · University of Pennsylvania · Florida State University

Abstract

Unlike on-policy distillation (OPD), which depends on a separate teacher model, on-policy self-distillation (OPSD) leverages a single language model acting simultaneously as the student and the teacher, with the teacher conditioned on privileged information. Despite this architectural simplicity, synchronous rollout generation dominates overall training time, creating a severe systems bottleneck when scaling OPSD. To address this, we introduce Asynchronous On-Policy Self-Distillation (AsyncOPSD), a framework that decouples rollout generation from learner updates to improve training throughput. Since the teacher and student share parameters, AsyncOPSD eliminates the multi-model coordination overhead inherent to OPD architectures in asynchronous training. As the student policy evolves, buffered trajectories generated by older policies become stale, potentially degrading performance. To mitigate this, we introduce a distillation objective based on staleness-aware policy control and token-level entropy weighting. Across three model sizes and mathematical reasoning benchmarks, AsyncOPSD improves the training throughput by $1.96$ -- $2.25\times$ over synchronous OPSD while increasing average benchmark accuracy.

Figures & tables

Figure 1: Overview of AsyncOPSD. Synchronous and asynchronous OPSD under the same budget. (a) Synchronous OPSD executes rollout and learning sequentially on

N

GPUs. (b) Asynchronous OPSD assigns

N_{r}

GPUs to independent rollout actors and

N_{l}

GPUs to a single distributed learner (

N=N_{r}+N_{l}

). Actors add rollout microbatches in a shared buffer, and the learner periodically updates policy to the actors.

Model	Sync Step	Samples/s $\uparrow$	Mean Staleness $\downarrow$
Qwen3-1.7B	$1$	$8.95\pm 0.34$	$4.98\pm 0.08$
	2	$9.22\pm 0.41$	$5.35\pm 0.28$
	$4$	$9.41\pm 0.49$	$6.08\pm 0.37$
	$8$	$8.63\pm 0.21$	$7.93\pm 0.16$
Qwen3-4B	$1$	$5.23\pm 0.22$	$6.09\pm 0.29$
	2	$5.53\pm 0.46$	$6.66\pm 0.37$

Table 1: Effect of sync_step . No max staleness limitation and no entropy weighting. Bold identifies selected interval.

Model	Method	AIME24	AIME25	HMMT25	Average
Qwen3-8B	OPSD Zhao et al. (2026)	77.8	70.8	45.8	64.80
	Vanilla AsyncOPSD	76.9	70.0	45.8	64.23
	+ Staleness Control	77.5	71.1	46.1	64.90
	AsyncOPSD	77.8	71.7	47.5	65.67
Qwen3-4B	OPSD Zhao et al. (2026)	76.4	68.3	46.1	63.60
	Vanilla AsyncOPSD	75.0	67.5	44.4	62.30

Table 2: We evaluate the mathematical reasoning performance of Qwen3 models using Avg@12, following the sampling settings recommended in the Qwen3 blog (temperature 1.0, maximum generation length 38k). Full benchmark details are provided in Table 5 .

Model	Method	Samples/s $\uparrow$	Relative throughput	Mean Staleness
Qwen3-1.7B	Synchronous OPSD	$4.07\pm 0.32$	1.00	-
	Vanilla AsyncOPSD	$9.22\pm 0.41$	2.26	$5.35\pm 0.28$
	+ Staleness Control	$8.20\pm 0.39$	2.01	$4.73\pm 0.22$
	AsyncOPSD	$8.00\pm 0.45$	1.96	$4.61\pm 0.32$
Qwen3-4B	Synchronous OPSD	$2.40\pm 0.33$	1.00	-
	Vanilla AsyncOPSD	$5.53\pm 0.46$	2.31	$6.66\pm 0.37$

Table 3: Accepted-training-sample throughput on eight A100s. Relative throughput is the ratio of the asynchronous and synchronous mean sample rates, computed before rounding. Synchronous rollouts have zero staleness. The comparison measures the complete implementations, including their batching and policy-refresh strategies.

Panel of Figure 2 — Figure 2: GPU-allocation ablation on eight A100s. L/A denotes learner/actor GPUs. (a) shows mean accepted-training-sample throughput; panel (b) averages mean rollout staleness in learner optimizer steps. The sweep uses $\texttt{sync\_step}=8$ vanilla AsyncOPSD.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Training
Optimizer	AdamW
Learning Rate	$5\times 10^{-6}$
Learning Rate Scheduler	Constant
Effective Batch Size	32
Dataset shuffling seed	42
LoRA Rank $r$	64

Appendix

Table 4: Training Configuration.

Parameter	Evaluation
Max New Tokens	38912
Thinking Mode	Enabled
Top-p	0.95
Top-k	-1
Min-p	0.0
Samples per Prompt	12

Appendix

Table 5: Evaluation Configuration.

Model	L/A	Samples/s $\uparrow$	Mean staleness $\downarrow$
Qwen3-1.7B	2/6	$6.06\pm 0.29$	$9.54\pm 0.43$
Qwen3-1.7B	3/5	$6.68\pm 0.25$	$10.56\pm 0.19$
Qwen3-1.7B	4/4	$8.63\pm 0.21$	$7.93\pm 0.16$
Qwen3-1.7B	5/3	$7.21\pm 0.15$	$9.55\pm 0.11$
Qwen3-1.7B	6/2	$5.53\pm 0.13$	$5.86\pm 0.05$
Qwen3-4B	2/6	$3.32\pm 0.24$	$10.65\pm 0.43$

Appendix

Table 6: GPU-allocation ablation on eight A100s at sync interval 8, No max staleness limitation, and uniform token weights. L/A denotes learner/actor GPUs.

$K$	Samples/s $\uparrow$	$R_{K}$	Mean Staleness $\downarrow$	Rejected (%) $\downarrow$
4	$6.96\pm 0.36$	0.755	$3.79\pm 0.21$	$33.91\pm 0.91$
6	$9.22\pm 0.41$	1.000	$5.35\pm 0.28$	$10.70\pm 1.39$
8	$10.27\pm 0.33$	1.114	$6.12\pm 0.38$	$0.00\pm 0.00$
10	$10.22\pm 0.31$	1.108	$6.20\pm 0.39$	$0.00\pm 0.00$

Appendix

Table 7: Staleness-cutoff ablation on Qwen3-1.7B with

\beta_{H}=0

. Relative throughput (

R_{K}=\bar{v}_{K}/\bar{v}_{6}

) is normalized to this sweep’s measured

K=6

mean before rounding. Staleness is measured in learner optimizer steps; rejection is the fraction of claimed queue microbatches discarded by the cutoff.

$\beta_{H}$	AIME 2025
0.03	42.50
0.1	43.90
0.3	42.78
1.0	40.28

Appendix

Table 8: Effect of the entropy-weighting coefficient on Qwen3-1.7B. Bold marks the highest AIME 2025 Avg@12 among the tested values.

On-Policy Self-Distillation

On-policy self-distillation trains a model on its own generated trajectories using a teacher view to provide dense supervision.

See all 162 papers

AsyncOPD: How Stale Can On-Policy Distillation Be?
Jun 23, 2026Wonjun Kang, Kevin Galim, Seunghyuk Oh +9Language Model Distillation On-Policy Distillation
FuriosaAI · Ajou University · UC Berkeley +3
$\boldsymbol{f}$ -OPD: Stabilizing Long-Horizon On-Policy Distillation with Freshness-Aware Control
May 18, 2026Xianwei Chen, Shimin Zhang, Jibin WuLanguage Model Distillation On-Policy Distillation
The Hong Kong Polytechnic University
DualOPSD: Closing the Feedback Loop in On-Policy Self-Distillation
Aug 26, 2026Yutong Chen, Guangfu Guo, Zhichao Xu +1Adaptive Knowledge Distillation On-Policy Self-Distillation
Department of Computer Science, Clemson University, 105 Sikes Hall, Clemson, SC 29634, USA · Department of Computer Science, University of Utah

Jun 23, 2026J/K move · Enter open · S save

AsyncOPD: How Stale Can On-Policy Distillation Be?

Wonjun Kang, Kevin Galim, Seunghyuk Oh, Minjun Kang+8

FuriosaAI · Ajou University · UC Berkeley+3

Language Model Distillation On-Policy Distillation LLM Post-Training Teacher-Student Learning

Open paper AskPDF arXiv

On-policy distillation (OPD) trains a student on its own rollouts guided by teacher feedback and is becoming increasingly important for large language model (LLM) post-training. Like reinforcement learning (RL), however, OPD faces an on-policy systems bottleneck, as rollouts can dominate training time for reasoning workloads. Asynchronous training pipelines can alleviate this bottleneck by decoupling rollout generation from learner updates, but doing so introduces stale-policy data. While prior work has studied stale data in asynchronous RL, its effects in OPD remain underexplored. We present the first systematic study of staleness in asynchronous OPD, focusing on a practical setting where teacher feedback is implemented through local KL losses and full-vocabulary teacher logits are too expensive to store or transfer, necessitating finite teacher-score caches. We first show that KL direction changes the stale-data problem: teacher-weighted forward KL is more robust to stale rollouts, whereas student-weighted reverse KL is vulnerable. Second, for this vulnerable reverse-KL case, we study whether methods designed to stabilize asynchronous RL can mitigate OPD staleness. In our experiments, they do not improve over a simpler OPD-specific surrogate: recomputing the reverse-KL signal under the current student at learner time. Third, we analyze how finite teacher-score caches create a bias-variance tradeoff for sparse and sampled reverse-KL OPD estimators. This motivates multi-sample Monte Carlo (MC), which preserves MC correctability while reducing one-sample variance. Finally, we present and open-source AsyncOPD, a fully asynchronous OPD training pipeline built from these estimator choices. Experiments show that AsyncOPD improves training throughput by

1.6\times

3.8\times

over strict synchronous training while reaching comparable accuracy.

Abstract

On-Policy Self-Distillation

Explore similar work

AsyncOPD: How Stale Can On-Policy Distillation Be?

f\boldsymbol{f}f-OPD: Stabilizing Long-Horizon On-Policy Distillation with Freshness-Aware Control

DualOPSD: Closing the Feedback Loop in On-Policy Self-Distillation

$\boldsymbol{f}$ -OPD: Stabilizing Long-Horizon On-Policy Distillation with Freshness-Aware Control