Asynchronous RL

RL: Reinforcement Learning

Asynchronous reinforcement learning studies agents that collect experience, act, or update at different times, as in distributed simulators, multi-agent systems, robots with inference delays, and language-model post-training. Systems methods overlap rollout generation, environment interaction, and optimization, while algorithmic methods control stale data through importance weighting, trust regions, advantage correction, or staleness-aware scheduling. Work is evaluated on settings ranging from tabular Q-learning and Markov games to tool-use, reasoning, web-agent, and robotics tasks. The central tension is gaining throughput by reducing synchronization waits without letting policy lag and delayed feedback destabilize learning or degrade performance; theory and practical safeguards address this tradeoff with varying guarantees.

Latest papers

46 papers14 papers in the last four weeks, up 250% on the four weeks before. 0.1% of all new papers.

AsyncWebRL: Efficient Multi-Step RL for Visual Web Agents
Jun 4, 2026Hao Bai, Rui Yang, Chenlu Ye +3Asynchronous RL Reinforcement Learning
1UIUC · 2Microsoft · 3CMU
Failure Modes of Deep Multi-Agent RL in Asynchronous Pricing: Reproducible Triggers, Trace Diagnostics, and a Partial Fix
Jun 3, 2026Shree Murthy, Rohan PandeyAsynchronous RL Algorithmic Collusion
DigitalOcean, USA
ASymPO: Asymmetric-Scale Policy Optimization for Asynchronous LLM Post-Training Without Behavior Information
Jun 2, 2026Zehua Liu, Yuxuan Yao, Xiaojin Fu +2Asynchronous RL Reinforcement Learning
Huawei Technologies
Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction
May 12, 2026Zhong Guan, Yongjian Guo, Haoran Sun +5Asynchronous RL Reinforcement Learning
Tianjin University · Tsinghua University · JDT AI Infra +1
MARLaaS: Multi-Tenant Asynchronous Reinforcement Learning as a Service
May 8, 2026Timothy Tin Long Yu, Gursimran Singh, Ge Shi +3Asynchronous RL Reinforcement Learning
Huawei Technologies Canada
DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training
Apr 29, 2026Tianhao Hu, Xiangcheng Liu, Youshao Xiao +15RL for Language Models Asynchronous RL
Meituan, China
Relax: An Asynchronous Reinforcement Learning Engine for Omni-Modal Post-Training at Scale
Apr 13, 2026Liujie Zhang, Benzhe Ning, Rui Yang +8Reinforcement Learning RL Post-Training
AI Platform, Xiaohongshu Inc · The University of Hong Kong · University of Science and Technology of China
Gaussian Approximation for Asynchronous Q-learning
Apr 8, 2026Artemy Rubtsov, Sergey Samsonov, Vladimir Ulyanov +1Polyak-Ruppert Averaging Q-Learning
HSE University, Moscow, Russia · HSE University & Lomonosov MSU, Moscow, Russia · HSE University & Steklov Mathematical Institute of Russian Academy of Sciences, Moscow, Russia
Dual-Gated Epistemic Time-Dilation: Autonomous Compute Modulation in Asynchronous MARL
Mar 24, 2026Igor JankowskiSemi-Markov Decision Process Reinforcement Learning
AcceRL: A Distributed Asynchronous Reinforcement Learning and World Model Framework for Vision-Language-Action Models
Mar 19, 2026Chengxuan Lu, Shukuan Wang, Yanjie Li +10World Models for VLA Models Asynchronous RL
IROOTECH TECHNOLOGY Wolf 1069 b Lab, Sany Group
GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Control
Mar 2, 2026Haofeng Xu, Junwei Su, Yukun Tian +3RL Stability Asynchronous Advantage Actor-Critic
School of Computing and Data Science, The University of Hong Kong · Alibaba Group · Southeast University
Align and Filter: Improving Performance in Asynchronous On-Policy RL
Mar 2, 2026Homayoun Honari, Roger Creus Castanyer, Michael Przystupa +3Asynchronous RL Policy Gradient Methods
Mila – Quebec AI Institute, Canada · Université de Montréal, Canada · University of Alberta, Canada
Mean-Field Reinforcement Learning without Synchrony
Feb 20, 2026Shan YangMean-Field MARL Reinforcement Learning
Department of Industrial Systems Engineering and Management, National University of Singapore, Singapore.
Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs
Feb 19, 2026Luke J. Huang, Zhuoyang Zhang, Qinghao Hu +2RL for Language Models Asynchronous RL
MIT · NVIDIA
AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training
Jul 2, 2025Zhenyu Han, Ansheng You, Haibo Wang +16RL for Language Models Asynchronous RL
Huawei · Individual Researcher
Graph Neural Network-Based Reinforcement Learning for Controlling Biological Networks - the GATTACA Framework
May 5, 2025Andrzej Mizera, Jakub ZarzyckiRL Control Asynchronous RL
Faculty of Mathematics, Informatics and Mechanics, University of Warsaw, Banacha 2, 02-097 Warsaw, Poland · IDEAS Research Institute, Królewska 27, 00-060 Warsaw, Poland · IDEAS NCBR Sp. z o.o., Chmielna 69, 00-801 Warsaw, Poland

Jun 4, 2026J/K move · Enter open · S save

AsyncWebRL: Efficient Multi-Step RL for Visual Web Agents

Hao Bai, Rui Yang, Chenlu Ye, Spencer Whitehead+2

1UIUC · 2Microsoft · 3CMU

Asynchronous RL Reinforcement Learning Group Relative Policy Optimization RL for VLMs Web Agents

Open paper AskPDF arXiv

Training vision-language web agents with multi-step RL is compute-intensive, with two dominant forms of inefficiency: idle GPUs in synchronous RL, and trajectories that use more steps and tokens than necessary. We present AsyncWebRL, which addresses both. On the system side, an asynchronous design overlaps rollout, gradient update, and policy refresh across iterations, paired with two web-agent-specific adaptations, namely an everlasting rollout pool and lightweight screenshot handling, that together deliver up to a

2.9\times

end-to-end training-throughput speedup over the previously fastest open synchronous pipeline (WebGym). On the algorithmic side, we identify the per-trajectory normalizer

1/|τ_i|

in multi-step GRPO as the root cause of trajectory-level and token-level inefficiency: because failures are systematically longer than successes, it down-weights the negative gradient on failed tokens, so the policy keeps producing verbose memory schemas. Replacing

1/|τ_i|

with a constant

1/k

breaks this coupling, contracting trajectories while preserving aggregate success. Together, these contributions set a new open-source state of the art on the WebGym out-of-distribution test split (+5.8% relative over the 42.9% prior best), with the largest gains on the harder slices (+42% relative on Medium, +48% relative on Hard).