Unlike on-policy distillation (OPD), which depends on a separate teacher model, on-policy self-distillation (OPSD) leverages a single language model acting simultaneously as the student and the teacher, with the teacher conditioned on privileged information. Despite this architectural simplicity, synchronous rollout generation dominates overall training time, creating a severe systems bottleneck when scaling OPSD. To address this, we introduce Asynchronous On-Policy Self-Distillation (AsyncOPSD), a framework that decouples rollout generation from learner updates to improve training throughput. Since the teacher and student share parameters, AsyncOPSD eliminates the multi-model coordination overhead inherent to OPD architectures in asynchronous training. As the student policy evolves, buffered trajectories generated by older policies become stale, potentially degrading performance. To mitigate this, we introduce a distillation objective based on staleness-aware policy control and token-level entropy weighting. Across three model sizes and mathematical reasoning benchmarks, AsyncOPSD improves the training throughput by 1.96--2.25× over synchronous OPSD while increasing average benchmark accuracy.
Figures & tables
Figure 1: Overview of AsyncOPSD. Synchronous and asynchronous OPSD under the same budget. (a) Synchronous OPSD executes rollout and learning sequentially on N GPUs. (b) Asynchronous OPSD assigns Nr GPUs to independent rollout actors and Nl GPUs to a single distributed learner ( N=Nr+Nl ). Actors add rollout microbatches in a shared buffer, and the learner periodically updates policy to the actors.
Model
Sync Step
Samples/s ↑
Mean Staleness ↓
Qwen3-1.7B
1
8.95±0.34
4.98±0.08
2
9.22±0.41
5.35±0.28
4
9.41±0.49
6.08±0.37
8
8.63±0.21
7.93±0.16
Qwen3-4B
1
5.23±0.22
6.09±0.29
2
5.53±0.46
6.66±0.37
Table 1: Effect of sync_step . No max staleness limitation and no entropy weighting. Bold identifies selected interval.
Model
Method
AIME24
AIME25
HMMT25
Average
Qwen3-8B
OPSD Zhao et al. (2026)
77.8
70.8
45.8
64.80
Vanilla AsyncOPSD
76.9
70.0
45.8
64.23
+ Staleness Control
77.5
71.1
46.1
64.90
AsyncOPSD
77.8
71.7
47.5
65.67
Qwen3-4B
OPSD Zhao et al. (2026)
76.4
68.3
46.1
63.60
Vanilla AsyncOPSD
75.0
67.5
44.4
62.30
Table 2: We evaluate the mathematical reasoning performance of Qwen3 models using Avg@12, following the sampling settings recommended in the Qwen3 blog (temperature 1.0, maximum generation length 38k). Full benchmark details are provided in Table 5 .
Model
Method
Samples/s ↑
Relative throughput
Mean Staleness
Qwen3-1.7B
Synchronous OPSD
4.07±0.32
1.00
-
Vanilla AsyncOPSD
9.22±0.41
2.26
5.35±0.28
+ Staleness Control
8.20±0.39
2.01
4.73±0.22
AsyncOPSD
8.00±0.45
1.96
4.61±0.32
Qwen3-4B
Synchronous OPSD
2.40±0.33
1.00
-
Vanilla AsyncOPSD
5.53±0.46
2.31
6.66±0.37
Table 3: Accepted-training-sample throughput on eight A100s. Relative throughput is the ratio of the asynchronous and synchronous mean sample rates, computed before rounding. Synchronous rollouts have zero staleness. The comparison measures the complete implementations, including their batching and policy-refresh strategies.
Figure 2: GPU-allocation ablation on eight A100s. L/A denotes learner/actor GPUs. (a) shows mean accepted-training-sample throughput; panel (b) averages mean rollout staleness in learner optimizer steps. The sweep uses sync_step=8 vanilla AsyncOPSD.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Training
Optimizer
AdamW
Learning Rate
5×10−6
Learning Rate Scheduler
Constant
Effective Batch Size
32
Dataset shuffling seed
42
LoRA Rank r
64
Appendix
Table 4: Training Configuration.
Parameter
Evaluation
Max New Tokens
38912
Thinking Mode
Enabled
Top-p
0.95
Top-k
-1
Min-p
0.0
Samples per Prompt
12
Appendix
Table 5: Evaluation Configuration.
Model
L/A
Samples/s ↑
Mean staleness ↓
Qwen3-1.7B
2/6
6.06±0.29
9.54±0.43
Qwen3-1.7B
3/5
6.68±0.25
10.56±0.19
Qwen3-1.7B
4/4
8.63±0.21
7.93±0.16
Qwen3-1.7B
5/3
7.21±0.15
9.55±0.11
Qwen3-1.7B
6/2
5.53±0.13
5.86±0.05
Qwen3-4B
2/6
3.32±0.24
10.65±0.43
Appendix
Table 6: GPU-allocation ablation on eight A100s at sync interval 8, No max staleness limitation, and uniform token weights. L/A denotes learner/actor GPUs.
K
Samples/s ↑
RK
Mean Staleness ↓
Rejected (%) ↓
4
6.96±0.36
0.755
3.79±0.21
33.91±0.91
6
9.22±0.41
1.000
5.35±0.28
10.70±1.39
8
10.27±0.33
1.114
6.12±0.38
0.00±0.00
10
10.22±0.31
1.108
6.20±0.39
0.00±0.00
Appendix
Table 7: Staleness-cutoff ablation on Qwen3-1.7B with βH=0 . Relative throughput ( RK=vˉK/vˉ6 ) is normalized to this sweep’s measured K=6 mean before rounding. Staleness is measured in learner optimizer steps; rejection is the fraction of claimed queue microbatches discarded by the cutoff.
βH
AIME 2025
0.03
42.50
0.1
43.90
0.3
42.78
1.0
40.28
Appendix
Table 8: Effect of the entropy-weighting coefficient on Qwen3-1.7B. Bold marks the highest AIME 2025 Avg@12 among the tested values.
On-policy distillation (OPD) trains a student on its own rollouts guided by teacher feedback and is becoming increasingly important for large language model (LLM) post-training. Like reinforcement learning (RL), however, OPD faces an on-policy systems bottleneck, as rollouts can dominate training time for reasoning workloads. Asynchronous training pipelines can alleviate this bottleneck by decoupling rollout generation from learner updates, but doing so introduces stale-policy data. While prior work has studied stale data in asynchronous RL, its effects in OPD remain underexplored. We present the first systematic study of staleness in asynchronous OPD, focusing on a practical setting where teacher feedback is implemented through local KL losses and full-vocabulary teacher logits are too expensive to store or transfer, necessitating finite teacher-score caches. We first show that KL direction changes the stale-data problem: teacher-weighted forward KL is more robust to stale rollouts, whereas student-weighted reverse KL is vulnerable. Second, for this vulnerable reverse-KL case, we study whether methods designed to stabilize asynchronous RL can mitigate OPD staleness. In our experiments, they do not improve over a simpler OPD-specific surrogate: recomputing the reverse-KL signal under the current student at learner time. Third, we analyze how finite teacher-score caches create a bias-variance tradeoff for sparse and sampled reverse-KL OPD estimators. This motivates multi-sample Monte Carlo (MC), which preserves MC correctability while reducing one-sample variance. Finally, we present and open-source AsyncOPD, a fully asynchronous OPD training pipeline built from these estimator choices. Experiments show that AsyncOPD improves training throughput by 1.6× to 3.8× over strict synchronous training while reaching comparable accuracy.
Scaling on-policy distillation (OPD) for large language models (LLMs) confronts a fundamental tension: asynchronous execution is necessary for system efficiency, but structurally deviates from the ideal on-policy objective. To address this challenge, we theoretically decompose the objective discrepancy into rollout drift and supervision drift, capturing staleness in student rollout and teacher context, respectively. Building on this, we introduce a sample-level freshness score that quantifies the reliability of a buffered sample with respect to the on-policy objective. Guided by this signal, we further propose f-OPD, a novel framework that adaptively regulates stale-sample influence and constrains policy drift accumulated under asynchronous training. Across reasoning, tool-use, and coding-agent tasks of increasing interaction horizon, f-OPD consistently achieves task performance comparable to synchronous optimization while largely retaining the throughput advantages of asynchronous execution. Our results establish the first recipe for achieving a performance-efficiency trade-off in OPD, paving the way for long-horizon agentic post-training at scale.
On-policy self-distillation (OPSD) combines student generated trajectories with dense supervision from a privileged teacher of the same model. Its pointwise clipping protects the student from locally incompatible teacher directions, but the fixed teacher receives no feedback about the supervision the student rejects, which limits effective learning from the teacher. We introduce DualOPSD, which alternates updates in a coupled teacher-student system. The student retains OPSD's clipped forward KL objective. The privileged teacher adapts to the updated student after each student step. A residual target based on the frozen initial model preserves the effect of privileged context during teacher adaptation. Across Qwen3-8B/4B/1.7B and three math benchmarks, DualOPSD improves avg@12 accuracy over OPSD by 9.83 percentage points. Further diagnostic analyses show lower bidirectional KL, less persistent mismatch, and retained sensitivity to privileged reference. Together, these results demonstrate that DualOPSD can be a more effective form of privileged self-distillation. Code is available at: https://github.com/on1262/DualOPSD.
Yutong Chen, Guangfu Guo, Zhichao Xu +1
Department of Computer Science, Clemson University, 105 Sikes Hall, Clemson, SC 29634, USA · Department of Computer Science, University of Utah