GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Control

Haofeng Xu; Junwei Su; Yukun Tian; Lansong Diao; Zhengping Qian; Chuan Wu

cs.LGMar 2, 2026

GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Control

Authors: Haofeng Xu, Junwei Su, Yukun Tian, Lansong Diao, Zhengping Qian, Chuan Wu

Organizations: School of Computing and Data Science, The University of Hong Kong · Alibaba Group · Southeast University

Abstract

Asynchronous execution is essential for scaling reinforcement learning (RL) to modern large model workloads, including large language models and AI agents, but it can fundamentally alter RL optimization behavior. While prior work on asynchronous RL focuses on training throughput and distributional correction, we show that naively applying asynchrony to policy-gradient updates can induce qualitatively different training dynamics and lead to severe training instability. Through systematic empirical and theoretical analysis, we identify a key signature of this instability: asynchronous training exhibits persistently high cosine similarity between consecutive policy gradients, in contrast to the near-orthogonal updates observed under synchronized training. This stale-aligned gradient effect amplifies correlated updates and increases the risk of overshooting and divergence. Motivated by this observation, we propose GRADIENT ALIGNMENT CONTROL(GAC), a simple dynamics-aware stabilization method that regulates asynchronous RL progress along stale-aligned directions via gradient projection. We establish convergence guarantees under bounded staleness and demonstrate empirically that GAC recovers stable, on-policy training dynamics and matches synchronized baselines even at high staleness.

Figures & tables

Panel of Figure 1 — Figure 1: Progressive instability under increasing staleness. Panels (a–b) report training reward and validation accuracy across staleness levels. Panels (c–d) report consecutive-gradient cosine similarity $c_{t}$ , contrasting synchronized training ( $s{=}0$ ) with mild ( $s{=}8$ ) and larger staleness ( $s{=}16,32$ ). Results on MATH with Qwen3-1.7B.

Panel of Figure 2 — Figure 2: Cross-scale learning curves under large staleness. Panels (a–d) report training reward and panels (e–h) report MinervaMath validation accuracy for Qwen3-1.7B/4B/8B and Llama-3.2-3B-Instruct. Curves compare synchronized GRPO, stale-rollout GRPO, M2PO, BAPO, and GAC .

Panel of Figure 3 — Figure 3: Gradient-alignment dynamics under staleness. Consecutive-gradient cosine similarity $c_{t}$ for Qwen3-4B/8B, comparing synchronized GRPO, stale-rollout GRPO, M2PO, BAPO and GAC.

Method	AIME24	AIME25	AMC23	AMC24	Math500	Miner.	Olymp.	Avg.	$\boldsymbol{\Delta_{\textbf{Avg}}}$
Llama-3.2-3B-Instruct
GRPO-Sync	17.2	3.9	35.2	18.9	56.5	22.3	20.5	24.9	–
GRPO	10.8	3.0	30.4	13.8	55.4	19.1	17.6	21.4	–
M2PO	13.3	3.4	32.5	15.6	55.7	19.9	18.9	22.8	–
BAPO	8.6	3.3	33.3	15.2	55.5	19.2	18.6	22.0	–
GAC	17.0	3.7	36.1	19.0	56.2	21.6	20.9	24.9	+2.2

Table 1: Accuracy under Staleness. Accuracy (%) on seven mathematical reasoning benchmarks for Qwen3 models (1.7B/4B/8B) and Llama-3.2-3B-Instruct. We report synchronized on-policy GRPO ( GRPO-Sync ,

s{=}0

) as a reference (shaded rows). All other methods are evaluated under stale-rollout training with

s{=}16

. Avg. denotes the mean accuracy across benchmarks. Bold marks the best average accuracy among stale-rollout methods for each model. The last column reports

\Delta_{\text{Avg}}

, computed as the improvement of GAC over the strongest stale-rollout baseline among {GRPO, M2PO, BAPO} for the same model.

Figure 4: Robustness across staleness levels. Training reward trajectories under

s\in\{8,16,32\}

. GAC remains stable as staleness increases, while stale-rollout GRPO progressively degrades.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Optimization
Parameter	Value
Optimizer	AdamW
Learning rate ( $\eta$ )	$1\times 10^{-6}$
AdamW $\beta$ ’s	$(0.9,\,0.999)$
AdamW $\epsilon_{\mathrm{adam}}$	$1\times 10^{-8}$
Weight decay	$1\times 10^{-2}$

Appendix

Table 2: GRPO hyperparameters. Configuration used across all experiments.

Model	$q_{0.9}(\|c_{t}\|)$	$\max\|c_{t}\|$	$\Pr(\|c_{t}\|\leq 0.05)$
Qwen3-1.7B	0.0471	0.0924	0.91
Llama-3.2-3B-Instruct	0.0687	0.1276	0.83
Qwen3-4B	0.0296	0.0645	0.99
Qwen3-8B	0.0317	0.0625	0.98

Appendix

Table 3: On-policy (

s{=}0

) statistics of consecutive-gradient alignment. We report the 90th percentile

q_{0.9}(|c_{t}|)

, the maximum

\max|c_{t}|

, and the fraction of updates with

|c_{t}|\leq 0.05

(computed over updates after step 50 to exclude early transients). These values motivate anchoring

c_{\text{low}}

to the on-policy range and setting

c_{\text{high}}

well above it.

Panel of Figure 5 — Figure 5: Threshold robustness of GAC . Ablations around the default $(c_{\text{low}},c_{\text{high}}){=}(0.05,0.3)$ show that accuracy varies only mildly across nearby settings, supporting practical robustness without per-model retuning.

Panel of Figure 6 — Figure 6: Consecutive-gradient cosine similarity under synchronized vs. stale-rollout training across scales. Each panel corresponds to a different model and overlays $c_{t}$ for synchronized training ( $s{=}0$ ) and stale-rollout training ( $s{=}16$ ).

Panel of Figure 7 — Figure 7: Per-benchmark accuracy breakdown for Qwen3-4B under staleness ( $s{=}16$ ). Each panel reports final validation accuracy on one benchmark, comparing GAC against synchronized GRPO (on-policy reference) and off-policy baselines under stale rollouts.

RL Stability

See all 29 papers

Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysis
Oct 1, 2026Qijia He, Ruinan Jin, Jun Luo +2Asynchronous RL RL for Language Model Reasoning
The Ohio State University · Arizona State University
Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs
Feb 19, 2026Luke J. Huang, Zhuoyang Zhang, Qinghao Hu +2RL for Language Models Asynchronous RL
MIT · NVIDIA
Where Does Staleness Accumulate? Pool Aware Effective Staleness Control for Asynchronous RL in LLM Post-Training
Sep 29, 2026Chenliang Li, Neiwen Ling, Zijun Wei +1Asynchronous RL RL for Language Model Reasoning
ByteDance · Texas A&M University · University of Virginia

Oct 1, 2026J/K move · Enter open · S save

Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysis

Qijia He, Ruinan Jin, Jun Luo, Shaofeng Zou+1

The Ohio State University · Arizona State University

Asynchronous RL RL for Language Model Reasoning Group Relative Policy Optimization RL Post-Training Off-Policy RL

Open paper AskPDF arXiv

Asynchronous reinforcement learning (RL) improves the efficiency of large language model post-training but introduces stale rollouts generated by earlier policies. Theoretical understanding of how this staleness affects convergence and how to mitigate its impact remains limited. We derive a convergence bound for GRPO-style algorithms that explicitly characterizes the tradeoff between the gradient estimator's second moment and bias. For trajectory-level importance-weighted estimators, our analysis shows that once the second moment is uniformly controlled, delay enters the bound through the bias introduced by clipping or rescaling. Guided by this insight, we propose a novel group mass capping GRPO (GMC-GRPO) method, which minimizes a ratio-based bias bound within a class of weighted estimators sharing a common second-moment guarantee. We establish convergence guarantees for asynchronous GMC-GRPO and show that, compared with TIC-GRPO, it improves the threshold dependence of the fourth-order delay term from

O(ε^{-4})

O(ε^{-2})

ε\to0

, where

1+ε

is the ratio threshold. Under local policy overlap, the delay-dependent term decreases as

G^{-2/5}

after tuning the step size, where

G

is the group size. For fixed behavior and current policies, the bias introduced by group rescaling also vanishes as

G\to\infty

, whereas the bias from trajectory-wise clipping can persist. Experiments across Qwen3 models and reasoning benchmarks demonstrate improved robustness to stale rollouts, with GMC-GRPO achieving the best performance among stable baselines under large rollout delays.

Abstract

RL Stability

Explore similar work

Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysis

Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs

Where Does Staleness Accumulate? Pool Aware Effective Staleness Control for Asynchronous RL in LLM Post-Training