Align and Filter: Improving Performance in Asynchronous On-Policy RL

Homayoun Honari; Roger Creus Castanyer; Michael Przystupa; Michael Noukhovitch; Pablo Samuel Castro; Glen Berseth

cs.LGMar 2, 2026

Align and Filter: Improving Performance in Asynchronous On-Policy RL

Authors: Homayoun Honari, Roger Creus Castanyer, Michael Przystupa, Michael Noukhovitch, Pablo Samuel Castro, Glen Berseth

Organizations: Mila – Quebec AI Institute, Canada · Université de Montréal, Canada · University of Alberta, Canada

Abstract

Distributed training and increasing the gradient update frequency are practical strategies to accelerate learning and improve performance, but both exacerbate a central challenge: policy lag, which is the mismatch between the behavior policy generating data and the learning policy being updated. Policy lag can hinder the scaling of on-policy learning algorithms to larger problems. In this paper, we identify the sources of policy lag caused by distributed learning and high update frequency. We use the findings to propose total Variation-based Advantage aligned Constrained policy Optimization (\methodacronym) as a practical approach to mitigate policy lag. We empirically validate our method and show that it offers better robustness to policy lag in classic RL tasks and a modern RL for LLM math reasoning task.

Figures & tables

Figure 1: (left) Simulated Asynchronous RL setup. After the end of each training phase we store the weights of the new policy in the policy buffer. Subsequently, we sample random policies from the buffer and assign them to the actors. We then generated the trajectories in a synchronous fashion. The setup aims to simulate an asynchronous RL training setup to control the backward policy lag. (right) The typical asynchronous RL training setup. The actors generate the trajectories and train the policy simultaneously. The actor would receive the new policy whenever it is ready. Therefore, the new dataset would contain trajectories from older policies.

Panel of Figure 2 — Figure 2: Algorithmic Choices in VACO . (Top) VACO vs. PPO Clipping : To maintain a fixed Total Variation (TV) divergence, VACO(shown as ’TV Filtering’) selectively removes gradients that would increase the TV divergence. In contrast, PPO naively clips gradients if their policy ratio exceeds a predefined threshold. (Bottom) IMPALA vs. Advantage Realignment : While IMPALA estimates advantage values for the most recent policy using an asynchronously generated dataset, VACO’s ’Advantage Realignment’ first aligns the dataset to the initial policy of the optimization process, then iteratively optimizes based on this aligned dataset. This approach significantly reduces the computational load compared to IMPALA’s on-the-fly realignment.

Figure 3: With more backward policy lag (higher degree of asynchronicity) VACO achieves better performance on the aggregate metrics across various MuJoCo tasks. Higher Median, IQM, and Mean values and lower Optimality Gap imply better performance. The scores are computed over 100M steps across 10 independent random seeds and the shades represent 95% confidence intervals of the metrics.

Figure 4: IQM values of the comparison algorithms in the simulated asynchronous setups during the training process. The scores are computed over 100M steps across 10 independent random seeds and the shades represent the 95% confidence interval. (bottom right) IQM values of the Area under the curve of the normalized return plots. Higher values imply better sample efficiency during the training process.

Panel of Figure 5 — Figure 5: VACO improves over PPO-clipping for training LLMs to reason on GSM8k. (Top) Forward policy lag can improve training efficiency at the cost of eval performance. VACO maintains higher performance as lag increases. (Bottom) PPO-clip is always clipping, proportional to the forward lag. VACO filters more rarely, enabling learning from highly policy-lagged samples, but still maintains stability by filtering a larger part of the batch when activated.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Clip Ratio/TV Threshold	0.2
Learning Rate	3e-4
Learning Rate Annealing	True
Num Envs	500
Num Steps	1000
Discount Factor	0.99

Appendix

Table 1: Important Hyperparameters of PPO, SPO, IMPALA, and VACO

Hyperparameter	Value
PPO-Clip Lower Ratio	0.2
PPO-Clip Higher Ratio	0.272
TV Threshold	0.05
Learning Rate	1e-6
Num Prompts Per Minibatch	32
Num Completions Per Prompt	8

Appendix

Table 2: Hyperparameters of GRPO, PPO-Clip and VACO for GSM8k

Figure 6: Return plot of the comparison algorithms in five environments across five degrees of asynchronicity. The plot represent the average return of the policy over 10 random seeds and the shades showcase the standard deviation of the returns.

Figure 7: IQM values of VACO with various values of $\delta$ for TV divergence threshold in the simulated asynchronous setups during the training process. The scores are computed over 100M steps across 10 independent random seeds and the shades represent the 95% confidence interval. (bottom right) IQM values of the Area under the curve of the normalized return plots. Higher values imply better sample efficiency during the training process.

Figure 8: Ablation analysis on the performance of VACO with various value of the TV divergence threshold $\delta$ . Higher Median, IQM, and Mean values and lower Optimality Gap imply better performance. The scores are computed over 100M steps across 10 independent random seeds and the shades represent 95% confidence intervals of the metrics.

Figure 9: IQM values of VACO with various values of $\rho$ for advantage realignment in the simulated asynchronous setups during the training process. The scores are computed over 100M steps across 10 independent random seeds and the shades represent the 95% confidence interval. (bottom right) IQM values of the Area under the curve of the normalized return plots. Higher values imply better sample efficiency during the training process.

Figure 10: Ablation analysis on the performance of VACO with various value of $\rho$ for advantage realignment. Higher Median, IQM, and Mean values and lower Optimality Gap imply better performance. The scores are computed over 100M steps across 10 independent random seeds and the shades represent 95% confidence intervals of the metrics.

Figure 11: The average Total Variation Divergence between the final policy of the training phase and the behavior policy distribution for VACO and PPO in the simulated asynchronous RL training setup. We can see that VACO with the constraint of $1/2\mathbb{E}[|\pi/\beta-1|]\leq\delta/2$ where $\delta=0.2$ , consistently maintains the same level of TV divergence across all environments and degrees of asynchronicity. The plot represent the average estimate of the TV value over 10 random seeds and the shades showcase the standard deviation.

Figure 12: The performance of VACO with and without the advantage realignment process. We can see that on average, return realignment offers better robustness to off-policy data. The scores are computed over 100M steps across 10 independent random seeds and the shades represent the 95% confidence interval.

Asynchronous RL

Asynchronous reinforcement learning studies learning from staggered, delayed, or decentralized actions and updates without global synchronization.

See all 46 papers

GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Control
Mar 2, 2026Haofeng Xu, Junwei Su, Yukun Tian +3RL Stability Asynchronous Advantage Actor-Critic
School of Computing and Data Science, The University of Hong Kong · Alibaba Group · Southeast University
Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs
Feb 19, 2026Luke J. Huang, Zhuoyang Zhang, Qinghao Hu +2RL for Language Models Asynchronous RL
MIT · NVIDIA
LLMs Can Learn to Reason Via Off-Policy RL
Feb 22, 2026Daniel Ritter, Owen Oertell, Bradley Guo +3RL for Language Model Reasoning RL Post-Training
Cornell University · Databricks · Harvard University

Mar 2, 2026J/K move · Enter open · S save

GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Control

Haofeng Xu, Junwei Su, Yukun Tian, Lansong Diao+2

School of Computing and Data Science, The University of Hong Kong · Alibaba Group · Southeast University

RL Stability Asynchronous Advantage Actor-Critic Asynchronous RL Reinforcement Learning Policy Gradient Methods RL for Language Models

Open paper AskPDF arXiv

Asynchronous execution is essential for scaling reinforcement learning (RL) to modern large model workloads, including large language models and AI agents, but it can fundamentally alter RL optimization behavior. While prior work on asynchronous RL focuses on training throughput and distributional correction, we show that naively applying asynchrony to policy-gradient updates can induce qualitatively different training dynamics and lead to severe training instability. Through systematic empirical and theoretical analysis, we identify a key signature of this instability: asynchronous training exhibits persistently high cosine similarity between consecutive policy gradients, in contrast to the near-orthogonal updates observed under synchronized training. This stale-aligned gradient effect amplifies correlated updates and increases the risk of overshooting and divergence. Motivated by this observation, we propose GRADIENT ALIGNMENT CONTROL(GAC), a simple dynamics-aware stabilization method that regulates asynchronous RL progress along stale-aligned directions via gradient projection. We establish convergence guarantees under bounded staleness and demonstrate empirically that GAC recovers stable, on-policy training dynamics and matches synchronized baselines even at high staleness.

Abstract

Asynchronous RL

Explore similar work

GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Control

Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs

LLMs Can Learn to Reason Via Off-Policy RL