Asynchronous reinforcement learning has become increasingly central to scaling LLM post-training, delivering major throughput gains by decoupling rollout generation from policy updates. However, widely used policy-gradient objectives such as REINFORCE and GRPO suffer under high asynchrony: stale rollouts produce heavy-tailed importance weights, so a small number of trajectories dominate updates and the policy-gradient estimator becomes markedly higher variance. Through systematic analysis on math, reasoning, and tool-use benchmarks, we find that this increasing variance is reliably predicted by collapsing effective sample size (ESS), which prior stabilization methods largely fail to address. Motivated by this diagnosis, we introduce Variance Controlled Policy Optimization (VCPO), a method that (i) dynamically scales the learning rate with ESS to dampen unreliable updates and (ii) applies a closed-form minimum-variance baseline for off-policy settings, without a critic model and adding minimal overhead. Empirically, across math and general reasoning benchmarks, this enables robustly stable asynchronous training compared to previous stabilization and algorithmic methods, even in highly off-policy regimes (128 steps off-policy). In a long-horizon, tool-use task, VCPO matches synchronous performance while delivering a 2.5× speedup in training time. Code is available at: https://github.com/mit-han-lab/vcpo
Figures & tables
Figure 1: Long-context, tool-integrated multi-turn RL comparing synchronous training ( k=0 ) to a two-step policy lag ( k=2 ) with VCPO versus sequence-level truncated importance sampling (TIS). Left: AIME-2025 validation accuracy vs. cumulative wall-clock time. VCPO matches the best synchronous accuracy 2.5× faster ( ≈ 42h vs. ≈ 105h) and continues to improve thereafter. Right: Gradient norm vs. training steps. The TIS run shows an instability characterized by a brief gradient-norm spike followed by rapid collapse.
Figure 2: Sequence-Level TIS Collapse. Qwen2.5-7B Base trained on MATH task with 10-step off-policy ( PipelineRL- 10). The ESS ratio first degrades and then collapses, leading to a spike in rollout–policy KL divergence and a sharp drop in both training reward and validation accuracy. See Appendix F.3 for hyperparameters and training details.
Figure 3: Compute (left) and memory (right) overhead of baseline-aware updates for Qwen2.5-7B on 4 × H100 GPUs (TP=4) with a sequence length of 8192 tokens.
Figure 4: GSM8K with Qwen2-1.5B under PipelineRL- 12 (high policy lag). Most baselines lead to training collapse (or crash, e.g. Geometric MIS masks all sequences and has no loss), while VCPO remains stable throughout training and matches synchronous performance. Training details and hyperparameters can be found in Appendix F.2 .
Figure 5: Qwen2.5-7B under PipelineRL- 10 (10 Steps Off-Policy) on Countdown and MATH-500. Across both tasks, sequence level truncated importance sampling (TIS) suffers ESS collapse followed by KL/gradient instability and degraded accuracy, whereas VCPO maintains healthy ESS and stable updates, reaching synchronous performance. Training details and hyperparameters are provided in Appendix F.3 .
Method
Countdown ↑
Steps
GPU hours ↓
Base
1.6%
–
–
Sync ( k=0 )
38.4%
400
143.2
VCPO + Async ( k=10 )
41.9%
400
89.6
Table 1: End-to-end training times and validation accuracy for synchronous vs. asynchronous training (lag k ).
Figure 6: Stable training under extreme policy lag : Qwen2.5-7B trained with PipelineRL at policy lag k (steps off-policy). VCPO is stable up to at least k=128 steps off-policy and matches the synchronous baseline ( k=0 ).
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Qwen2-1.5B Base on GSM8K questions. Truncated Importance Sampling (TIS) exhibits both effective sample size (ESS) collapse and KL explosion, leading to training collapse.
Figure 8: Countdown (Qwen2-1.5B Base) under high asynchrony with TIS. Validation accuracy collapses to near zero within the first ∼ 50 training steps, indicating rapid training failure.
Figure 9: Component ablations on MATH-500 under high asynchrony (Qwen2.5-7B, PipelineRL- 10). We compare TIS alone, as well as adding ESS-guided step-size scaling or the off-policy optimal baseline (OPOB). Both components improve stability and accuracy.
Figure 10: Lowering the learning rate is not a substitute for variance-aware step-size control. On MATH-500 under high asynchrony, a very small fixed learning rate (TIS + Low LR) improves stability but converges more slowly and to lower accuracy, despite exhibiting similar ESS levels. VCPO reaches higher accuracy with stable updates without learning-rate sweeps.
Figure 11: Masking and clipping ablations under 12 steps off-policy on GSM8K (Qwen2-1.5B). We compare truncated importance sampling (TIS) and masked importance sampling (MIS) using sequence-level, token-level, and geometric-mean ratio statistics with thresholds c∈{2.0,8.0} . Overall, these methods bounding is highly sensitive to the threshold and fail to prevent training collapse.
Figure 12: M2PO under high asynchrony on GSM8K (Qwen2-1.5B). Top: GSM8K validation accuracy. Bottom-left: rollout–policy KL divergence (log scale). Bottom-right: trusted-token mask fraction (log scale). At k=12 (and similarly at k=10 ), the mask fraction becomes unstable and can collapse toward masking nearly all tokens, coinciding with KL instability and training collapse. Shaded regions indicate run-to-run variability.
Figure 13: Effect of adding a KL penalty in the reward for Seq-TIS under high asynchrony ( k=12 ) on GSM8K with Qwen2-1.5B. KL-in-reward ( β=0.001 ) does not prevent eventual collapse and also lowers the peak validation accuracy compared to Seq-TIS without KL.
Figure 14: Qwen2.5-7B Base trained on MATH-500 under high asynchrony ( k=10 ): the logit-energy proxy baseline (OTB) eventually becomes unstable and collapses, while our off-policy optimal baseline (OPOB) remains stable and sustains higher validation accuracy throughout training.
Figure 15: Qwen2-1.5B Base trajectories on GSM8K questions, with max response length capped to 4096 tokens
Hyperparameter
Value
Optimizer
AdamW
Learning rate
1×10−6
Warmup steps
0
Weight decay
0.1
AdamW β1,β2
0.9, 0.999
AdamW ϵ
10−8
Appendix
Table 2: Optimization hyperparameters
Hyperparameter
Value
Reference
Model
Qwen2-1.5B
Learning Rate
1×10−6
Learning Rate Schedule
Warmup Stable Decay
Learning Rate Warmup Steps
5
Learning Rate Stable Steps
395
Learning Rate Decay Steps
0
Appendix
Table 3: Training hyperparameters for the GSM8K task
Hyperparameter
Value
Reference
Model
Qwen2.5-7B
Learning Rate
1×10−6
Learning Rate Schedule
Warmup Stable Decay
Learning Rate Warmup Steps
0
Learning Rate Stable Steps
400
Learning Rate Decay Steps
0
Appendix
Table 4: Training hyperparameters for the MATH and Countdown task
Hyperparameter
Value
Reference
Model
Qwen2.5-7B
Learning Rate
1×10−6
Learning Rate Schedule
Warmup Stable Decay
Learning Rate Warmup Steps
0
Learning Rate Stable Steps
200
Learning Rate Decay Steps
0
Appendix
Table 5: Training hyperparameters for the multi-turn tool integrated reasoning
Asynchronous execution is essential for scaling reinforcement learning (RL) to modern large model workloads, including large language models and AI agents, but it can fundamentally alter RL optimization behavior. While prior work on asynchronous RL focuses on training throughput and distributional correction, we show that naively applying asynchrony to policy-gradient updates can induce qualitatively different training dynamics and lead to severe training instability. Through systematic empirical and theoretical analysis, we identify a key signature of this instability: asynchronous training exhibits persistently high cosine similarity between consecutive policy gradients, in contrast to the near-orthogonal updates observed under synchronized training. This stale-aligned gradient effect amplifies correlated updates and increases the risk of overshooting and divergence. Motivated by this observation, we propose GRADIENT ALIGNMENT CONTROL(GAC), a simple dynamics-aware stabilization method that regulates asynchronous RL progress along stale-aligned directions via gradient projection. We establish convergence guarantees under bounded staleness and demonstrate empirically that GAC recovers stable, on-policy training dynamics and matches synchronized baselines even at high staleness.
Haofeng Xu, Junwei Su, Yukun Tian +3
School of Computing and Data Science, The University of Hong Kong · Alibaba Group · Southeast University
Asynchronous RL accelerates large language model post-training by decoupling rollout generation from optimization, but trains on stale trajectories. Existing methods primarily correct token-level policy mismatch through importance-ratio control in the actor objective. We show that this policy-side correction alone is insufficient: advantage estimates also inherit mismatch from behavior-policy continuations, which we term advantage staleness. We derive exact bias and variance decompositions for a general two-channel actor update, revealing nonseparable coupling between policy-weight and advantage-estimation errors: their interaction induces multiplicative bias terms, while squared policy weights amplify advantage uncertainty in gradient variance. This motivates the hypothesis that policy- and advantage-side correction should be coordinated. We introduce Coupled Off-Policy Correction (COPC), an actor--critic method combining token-level ratio masking with two-sided clipped-ratio weighting of TD residuals for return and advantage estimation. Joint parameter sweeps across staleness levels support this hypothesis: the effect of one correction parameter depends on, and can reverse with, the other. COPC achieves the highest reported performance on tool-integrated mathematical reasoning and search, outperforming the strongest reported asynchronous baseline in each setting. It also offers a broad high-performing parameter region and improved training stability. In search, COPC remains stable throughout training, while most evaluated asynchronous baselines collapse late in training. These gains persist at 64-step policy staleness. COPC adds minimal step-time overhead over asynchronous PPO and retains a 1.7× step-time speedup over synchronous PPO.
Zicheng Hu, Zhijian Zhou, Xuan Zhang +7
Fudan University · Meituan · East China Normal University
Fully asynchronous reinforcement learning (RL) improves resource utilization in large language model post-training by overlapping rollout generation with policy optimization, but it also introduces policy lag as trajectories are generated and queued while the trainer continues to update. We study how this lag accumulates over a trajectory's lifetime and how it can be controlled without sacrificing the wall-clock benefits of asynchronous execution. We decompose trajectory staleness into Generation Staleness, accumulated before rollout completion, and Waiting Staleness, accumulated after a completed trajectory enters the pool. Motivated by this decomposition, we introduce PACE (Pool-Aware Control of Effective Staleness). PACE converts excess pool occupancy into an adaptive rejection budget and ranks completed trajectories using an effective-staleness score that combines Waiting Staleness with prefix-aware Generation Staleness. This avoids penalizing long or interrupted rollouts solely because they span multiple policy versions. In single-turn mathematical reasoning, PACE improves the six-benchmark average validation accuracy by 18.7% over unfiltered asynchronous RL at the same wall-clock budget and matches synchronous RL performance with 47.1% less GPU time. PACE also improves validation performance in multi-turn tool-integrated reasoning, outperforming both synchronous and unfiltered asynchronous RL. Further experiments with the mixture-of-experts model and an alternative RL algorithm support its applicability across model architectures and training algorithms.
Chenliang Li, Neiwen Ling, Zijun Wei +1
ByteDance · Texas A&M University · University of Virginia