Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs

Luke J. Huang; Zhuoyang Zhang; Qinghao Hu; Shang Yang; Song Han

cs.LGFeb 19, 2026

Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs

Authors: Luke J. Huang, Zhuoyang Zhang, Qinghao Hu, Shang Yang, Song Han

Organizations: MIT · NVIDIA

Abstract

Asynchronous reinforcement learning has become increasingly central to scaling LLM post-training, delivering major throughput gains by decoupling rollout generation from policy updates. However, widely used policy-gradient objectives such as REINFORCE and GRPO suffer under high asynchrony: stale rollouts produce heavy-tailed importance weights, so a small number of trajectories dominate updates and the policy-gradient estimator becomes markedly higher variance. Through systematic analysis on math, reasoning, and tool-use benchmarks, we find that this increasing variance is reliably predicted by collapsing effective sample size (ESS), which prior stabilization methods largely fail to address. Motivated by this diagnosis, we introduce $\textbf{V}$ ariance $\textbf{C}$ ontrolled $\textbf{P}$ olicy $\textbf{O}$ ptimization ( $\textbf{VCPO}$ ), a method that (i) dynamically scales the learning rate with ESS to dampen unreliable updates and (ii) applies a closed-form minimum-variance baseline for off-policy settings, without a critic model and adding minimal overhead. Empirically, across math and general reasoning benchmarks, this enables robustly stable asynchronous training compared to previous stabilization and algorithmic methods, even in highly off-policy regimes (128 steps off-policy). In a long-horizon, tool-use task, VCPO matches synchronous performance while delivering a 2.5 $\times$ speedup in training time. Code is available at: https://github.com/mit-han-lab/vcpo

Figures & tables

Figure 1: Long-context, tool-integrated multi-turn RL comparing synchronous training ( $k{=}0$ ) to a two-step policy lag ( $k{=}2$ ) with VCPO versus sequence-level truncated importance sampling (TIS). Left: AIME-2025 validation accuracy vs. cumulative wall-clock time. VCPO matches the best synchronous accuracy $2.5\times$ faster ( $\approx$ 42h vs. $\approx$ 105h) and continues to improve thereafter. Right: Gradient norm vs. training steps. The TIS run shows an instability characterized by a brief gradient-norm spike followed by rapid collapse.

Figure 2: Sequence-Level TIS Collapse. Qwen2.5-7B Base trained on MATH task with 10-step off-policy ( PipelineRL- 10). The ESS ratio first degrades and then collapses, leading to a spike in rollout–policy KL divergence and a sharp drop in both training reward and validation accuracy. See Appendix F.3 for hyperparameters and training details.

Figure 3: Compute (left) and memory (right) overhead of baseline-aware updates for Qwen2.5-7B on 4 $\times$ H100 GPUs (TP=4) with a sequence length of 8192 tokens.

Figure 4: GSM8K with Qwen2-1.5B under PipelineRL- 12 (high policy lag). Most baselines lead to training collapse (or crash, e.g. Geometric MIS masks all sequences and has no loss), while VCPO remains stable throughout training and matches synchronous performance. Training details and hyperparameters can be found in Appendix F.2 .

Figure 5: Qwen2.5-7B under PipelineRL- 10 (10 Steps Off-Policy) on Countdown and MATH-500. Across both tasks, sequence level truncated importance sampling (TIS) suffers ESS collapse followed by KL/gradient instability and degraded accuracy, whereas VCPO maintains healthy ESS and stable updates, reaching synchronous performance. Training details and hyperparameters are provided in Appendix F.3 .

Method	Countdown $\uparrow$	Steps	GPU hours $\downarrow$
Base	1.6%	–	–
Sync ( $k{=}0$ )	38.4%	400	143.2
VCPO + Async ( $k{=}10$ )	41.9%	400	89.6

Table 1: End-to-end training times and validation accuracy for synchronous vs. asynchronous training (lag

k

Figure 6: Stable training under extreme policy lag : Qwen2.5-7B trained with PipelineRL at policy lag $k$ (steps off-policy). VCPO is stable up to at least $k{=}128$ steps off-policy and matches the synchronous baseline ( $k{=}0$ ).

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Qwen2-1.5B Base on GSM8K questions. Truncated Importance Sampling (TIS) exhibits both effective sample size (ESS) collapse and KL explosion, leading to training collapse.

Figure 8: Countdown (Qwen2-1.5B Base) under high asynchrony with TIS. Validation accuracy collapses to near zero within the first $\sim$ 50 training steps, indicating rapid training failure.

Figure 9: Component ablations on MATH-500 under high asynchrony (Qwen2.5-7B, PipelineRL- 10). We compare TIS alone, as well as adding ESS-guided step-size scaling or the off-policy optimal baseline (OPOB). Both components improve stability and accuracy.

Figure 10: Lowering the learning rate is not a substitute for variance-aware step-size control. On MATH-500 under high asynchrony, a very small fixed learning rate (TIS + Low LR) improves stability but converges more slowly and to lower accuracy, despite exhibiting similar ESS levels. VCPO reaches higher accuracy with stable updates without learning-rate sweeps.

Figure 11: Masking and clipping ablations under 12 steps off-policy on GSM8K (Qwen2-1.5B). We compare truncated importance sampling (TIS) and masked importance sampling (MIS) using sequence-level, token-level, and geometric-mean ratio statistics with thresholds $c\in\{2.0,8.0\}$ . Overall, these methods bounding is highly sensitive to the threshold and fail to prevent training collapse.

Figure 12: M2PO under high asynchrony on GSM8K (Qwen2-1.5B). Top: GSM8K validation accuracy. Bottom-left: rollout–policy KL divergence (log scale). Bottom-right: trusted-token mask fraction (log scale). At $k{=}12$ (and similarly at $k{=}10$ ), the mask fraction becomes unstable and can collapse toward masking nearly all tokens, coinciding with KL instability and training collapse. Shaded regions indicate run-to-run variability.

Figure 13: Effect of adding a KL penalty in the reward for Seq-TIS under high asynchrony ( $k=12$ ) on GSM8K with Qwen2-1.5B. KL-in-reward ( $\beta=0.001$ ) does not prevent eventual collapse and also lowers the peak validation accuracy compared to Seq-TIS without KL.

Figure 14: Qwen2.5-7B Base trained on MATH-500 under high asynchrony ( $k{=}10$ ): the logit-energy proxy baseline (OTB) eventually becomes unstable and collapses, while our off-policy optimal baseline (OPOB) remains stable and sustains higher validation accuracy throughout training.

Figure 15: Qwen2-1.5B Base trajectories on GSM8K questions, with max response length capped to 4096 tokens

Hyperparameter	Value
Optimizer	AdamW
Learning rate	$1\times 10^{-6}$
Warmup steps	0
Weight decay	0.1
AdamW $\beta_{1},\beta_{2}$	0.9, 0.999
AdamW $\epsilon$	$10^{-8}$

Appendix

Table 2: Optimization hyperparameters

Hyperparameter	Value	Reference
Model	Qwen2-1.5B
Learning Rate	$1\times 10^{-6}$
Learning Rate Schedule	Warmup Stable Decay
Learning Rate Warmup Steps	5
Learning Rate Stable Steps	395
Learning Rate Decay Steps	0

Appendix

Table 3: Training hyperparameters for the GSM8K task

Hyperparameter	Value	Reference
Model	Qwen2.5-7B
Learning Rate	$1\times 10^{-6}$
Learning Rate Schedule	Warmup Stable Decay
Learning Rate Warmup Steps	0
Learning Rate Stable Steps	400
Learning Rate Decay Steps	0

Appendix

Table 4: Training hyperparameters for the MATH and Countdown task

Hyperparameter	Value	Reference
Model	Qwen2.5-7B
Learning Rate	$1\times 10^{-6}$
Learning Rate Schedule	Warmup Stable Decay
Learning Rate Warmup Steps	0
Learning Rate Stable Steps	200
Learning Rate Decay Steps	0

Appendix

Table 5: Training hyperparameters for the multi-turn tool integrated reasoning

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 410 papers

GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Control
Mar 2, 2026Haofeng Xu, Junwei Su, Yukun Tian +3RL Stability Asynchronous Advantage Actor-Critic
School of Computing and Data Science, The University of Hong Kong · Alibaba Group · Southeast University
COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning
Oct 7, 2026Zicheng Hu, Zhijian Zhou, Xuan Zhang +7Asynchronous RL RL Post-Training
Fudan University · Meituan · East China Normal University
Where Does Staleness Accumulate? Pool Aware Effective Staleness Control for Asynchronous RL in LLM Post-Training
Sep 29, 2026Chenliang Li, Neiwen Ling, Zijun Wei +1Asynchronous RL RL for Language Model Reasoning
ByteDance · Texas A&M University · University of Virginia

Mar 2, 2026J/K move · Enter open · S save

GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Control

Haofeng Xu, Junwei Su, Yukun Tian, Lansong Diao+2

School of Computing and Data Science, The University of Hong Kong · Alibaba Group · Southeast University

RL Stability Asynchronous Advantage Actor-Critic Asynchronous RL Reinforcement Learning Policy Gradient Methods RL for Language Models

Open paper AskPDF arXiv

Asynchronous execution is essential for scaling reinforcement learning (RL) to modern large model workloads, including large language models and AI agents, but it can fundamentally alter RL optimization behavior. While prior work on asynchronous RL focuses on training throughput and distributional correction, we show that naively applying asynchrony to policy-gradient updates can induce qualitatively different training dynamics and lead to severe training instability. Through systematic empirical and theoretical analysis, we identify a key signature of this instability: asynchronous training exhibits persistently high cosine similarity between consecutive policy gradients, in contrast to the near-orthogonal updates observed under synchronized training. This stale-aligned gradient effect amplifies correlated updates and increases the risk of overshooting and divergence. Motivated by this observation, we propose GRADIENT ALIGNMENT CONTROL(GAC), a simple dynamics-aware stabilization method that regulates asynchronous RL progress along stale-aligned directions via gradient projection. We establish convergence guarantees under bounded staleness and demonstrate empirically that GAC recovers stable, on-policy training dynamics and matches synchronized baselines even at high staleness.

Abstract

RL for Language Models

Explore similar work

GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Control

COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning

Where Does Staleness Accumulate? Pool Aware Effective Staleness Control for Asynchronous RL in LLM Post-Training