Asynchronous RL

RL: Reinforcement Learning

Asynchronous reinforcement learning studies agents that collect experience, act, or update at different times, as in distributed simulators, multi-agent systems, robots with inference delays, and language-model post-training. Systems methods overlap rollout generation, environment interaction, and optimization, while algorithmic methods control stale data through importance weighting, trust regions, advantage correction, or staleness-aware scheduling. Work is evaluated on settings ranging from tabular Q-learning and Markov games to tool-use, reasoning, web-agent, and robotics tasks. The central tension is gaining throughput by reducing synchronization waits without letting policy lag and delayed feedback destabilize learning or degrade performance; theory and practical safeguards address this tradeoff with varying guarantees.

Latest papers

46 papers14 papers in the last four weeks, up 250% on the four weeks before. 0.1% of all new papers.

Arctic-SWE: Revisiting the Long Horizon Agent Training by Modeling-System Co-Design
Oct 17, 2026Boyi Liu, Yite Wang, Xinyu Lian +6Long-Horizon RL RL for LLM Agents
Snowflake · University of Illinois at Urbana-Champaign · University of Houston +1
ReFlow: Replay-Driven Decoupling for Efficient RL Post-Training
Oct 15, 2026Yukun Tian, Junwei Su, Haofeng Xu +6RL Post-Training RL for Language Model Reasoning
The University of Hong Kong · Alibaba Group · University of Science and Technology of China +1
Long-Horizon RL Needs Muon
Oct 14, 2026Kai Ruan, Jinghao Lin, Tao Chen +3Long-Horizon RL Long-Horizon LLM Agents
Gaoling School of Artificial Intelligence, Renmin University of China · Duke University · Institute of Automation, Chinese Academy of Sciences +1
LoongSage: A Production-Grade, High-Efficiency Agentic RL Framework for Frontier LLMs with Validated Recipes
Oct 12, 2026LoongSage Team, Bowei Song, Caiwei Zhao +45Agentic RL LLM Agent Training
KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization
Oct 10, 2026Sihwan Park, Sung-Yub Kim, Doohyuk Jang +1KV-Cache Quantization RL Post-Training
KAIST · NAVER Cloud · AITRICS
Hitting-Time Bounds for Asynchronous Q-Learning
Oct 9, 2026Yang PengReinforcement Learning Q-Learning
Yau Mathematical Sciences Center, Tsinghua University
A Closed-Loop Non-Asymptotic Convergence Analysis of PPO with Learned Critics and Clipping
Oct 7, 2026Junwei Su, Mengfan Liu, Yanyong Zhang +1Proximal Policy Optimization Actor-Critic Methods
School of Artificial Intelligence and Data Science, University of Science and Technology of China · Department of Computer Science, University of Hong Kong
COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning
Oct 7, 2026Zicheng Hu, Zhijian Zhou, Xuan Zhang +7Asynchronous RL RL Post-Training
Fudan University · Meituan · East China Normal University
On KL-Regularized Policy Optimization
Oct 6, 2026Yifan ZhangRL for LLM Agents KL-Regularized RL
Princeton University
Asynchrony Is Nearly Free for Evolution Strategies on Long-Horizon LLM Agentic Tasks
Oct 3, 2026William Hoy, Jingxuan Fan, Nurcin Celik +1Evolution Strategies RL Post-Training
University of Miami · Harvard University
Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysis
Oct 1, 2026Qijia He, Ruinan Jin, Jun Luo +2Asynchronous RL RL for Language Model Reasoning
The Ohio State University · Arizona State University
Near-Minimax-Optimal Rates for Asynchronous TD Learning with Markovian Data
Sep 30, 2026Yang PengAsynchronous RL Stochastic Approximation
Yau Mathematical Sciences Center, Tsinghua University
Where Does Staleness Accumulate? Pool Aware Effective Staleness Control for Asynchronous RL in LLM Post-Training
Sep 29, 2026Chenliang Li, Neiwen Ling, Zijun Wei +1Asynchronous RL RL for Language Model Reasoning
ByteDance · Texas A&M University · University of Virginia
PEARL: Adaptive Prefill-Decode Execution with Elasticity for Agentic Reinforcement Learning
Sep 28, 2026Jiaan Zhu, Wei Gao, Youhui Bai +6Agentic RL Asynchronous RL
USTC China
EBRL: Asynchronous Embodied RL by Multi-Grained Resource Management
Sep 23, 2026Liang Mi, Weijun Wang, Bowen Gao +12Asynchronous RL Reinforcement Learning
Nanjing University · Institute for AI Industry Research (AIR), Tsinghua University · Nanjing University of Posts and Telecommunications +1
Stable and Efficient Real-World Online VLA Post-Training via Asynchronous Replay-Anchored Policy Improvement
Sep 19, 2026Jiarui Yang, Jiajin Zhang, Bin Zhu +2Robot Policy Learning RL Post-Training
Institute of Trustworthy Embodied AI, Fudan University · Shanghai Key Laboratory of Multimodal Embodied AI · Singapore Management University
High-Probability Nash Regret for Decentralized Learning in Markov $α$ -Potential Games: Episodic and Fully Online Asynchronous Algorithms with Applications to Markov Congestion Games
Sep 14, 2026S. Rasoul EtesamiPolicy Gradient Asynchronous RL
Department of Industrial and Systems Engineering Coordinated Science Laboratory University of Illinois Urbana-Champaign, Urbana, IL, USA
BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RL
Sep 9, 2026Guanqun Zhao, Zijun Xie, Binbin Zheng +3Asynchronous RL Off-Policy RL
Beijing University of Posts and Telecommunications · Baidu Inc. · Peking University +1
SmoothRL: Online Reinforcement Learning During Asynchronous Execution
Aug 30, 2026Guang Gao, Yuxuan Nong, Baifu Huang +1Robot Policy Learning Asynchronous RL
Astribot Team
SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RL
Aug 25, 2026Kai Ruan, Jinghao Lin, Qianshan Wei +2RL for LLM Agents Agentic RL
Gaoling School of Artificial Intelligence, Renmin University of China · Independent Researcher · Institute of Automation, Chinese Academy of Sciences +2
Learning to Act While Waiting: RL Finetuning of Generalist Robot Policies Under Inference Latency
Aug 24, 2026Brian Zhu, Momen Khalil, E Harrison +17RL for Robotics Robot Policy Learning
Siemens · UC Berkeley · Microsoft +1
TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling
Aug 11, 2026Yanyu Ren, Xizheng Wang, Xiao Liu +8Agentic RL Asynchronous RL
Tsinghua University · Zhongguancun Laboratory
Learning under Opponent Unawareness in Linear-Quadratic Stochastic Games
Aug 8, 2026Dantong Chu, Xuefeng Gao, Yufei ZhangNash Equilibrium Asynchronous RL
Department of Systems Engineering and Engineering Management, The Chinese University of Hong Kong, Hong Kong, China. · Department of Mathematics, Imperial College London, UK.
Deconstructing Off-Policy Ratios: Entropy-Normalized Trust Regions for Asynchronous Reinforcement Learning
Jul 24, 2026Guanqun Zhao, Zijun Xie, Binbin Zheng +5Asynchronous RL RL for Language Model Reasoning
Beijing University of Posts and Telecommunications · Baidu Inc. · Peking University +2
Robust Asynchronous Q-Learning under Reward and State Corruption via Batching
Jul 23, 2026Sreejeet Maity, Aritra MitraAsynchronous RL Q-Learning
Department of Electrical and Computer Engineering, North Carolina State University.
CHMAS: A Coupled Hierarchical Framework for Multi-Agent Reinforcement Learning
Jul 21, 2026Dongming Wang, Jie Xu, Yanyu Zhang +1Multi-Agent Coordination Asynchronous RL
Department of Electrical and Computer Engineering, University of California, Riverside, CA 92521, USA
Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning
Jul 21, 2026Junyao Yang, Yucheng Shi, Zongxia Li +6Asynchronous RL Proximal Policy Optimization
1Tencent Hy LLM Frontier · 2National University of Singapore · University of Maryland, College Park +3
Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning
Jul 8, 2026Zhenyu Hou, Yujiang Li, Jie Tang +1Agentic RL Asynchronous RL
Tsinghua University · Work done while ZH and YL interned at Z.AI.
Scaling Laws for Collapse in Asynchronous GRPO
Jul 1, 2026Jingwei Song, Haofeng Xu, Jie Xiao +7Asynchronous RL Reinforcement Learning
The University of Hong Kong · Shanghai Jiao Tong University · Gradient +2
Retroactive Advantage Correction: Closed-Form V-Trace Bias Correction for Delay-Aware RLHF
Jun 25, 2026Arnav RajAsynchronous RL Reinforcement Learning
Department of Computer Science and Engineering, Indian Institute of Technology Delhi, New Delhi, India.

Oct 17, 2026J/K move · Enter open · S save

Arctic-SWE: Revisiting the Long Horizon Agent Training by Modeling-System Co-Design

Boyi Liu, Yite Wang, Xinyu Lian, Fan Shu+5

Snowflake · University of Illinois at Urbana-Champaign · University of Houston+1

Long-Horizon RL RL for LLM Agents Reinforcement Learning Asynchronous RL Software Engineering Benchmarks

Open paper AskPDF arXiv

Long-horizon agent training, including software engineering, is advancing rapidly, but the recipe and infrastructure needed for a strong reinforcement-learning baseline remain difficult to reproduce. This makes gains from new algorithms or data hard to distinguish from improvements to an under-tuned baseline. We present Arctic-SWE, a fully asynchronous RL framework that establishes such a baseline through model-system co-design, using overfitting on a small task pool as the testbed. This testbed exposes failures that early reward gains conceal and guides the integration of established components: gated reward, importance-weight control, aligned training and inference engines, and consistent state across policy updates. No single component is new, yet together they form the missing foundation. On raw R2E-Gym data, without new algorithms or data work, Arctic-SWE trains Qwen3.5-4B to 62.0% pass@1 on SWE-bench Verified. This is comparable to concurrent FrogNano (61.5%), which relies on online task synthesis, after about one-sixth as many training rollouts. Building on this foundation, we further filter and refine R2E-Gym, SWE-rebench V2, and Scale-SWE. With this curated subset, the 4B model improves from 47.4%/25.0%/37.9% to 65.0%/38.3%/59.7% on SWE-bench Verified/Pro-v1/Pro-v2, and the recipe also improves Qwen3.5-9B from 56.2%/31.9%/48.8% to 65.7%/39.6%/70.2% on Verified/Pro-v1/Pro-v2. Controlled comparisons show that both curation and test strengthening improve held-out performance, although stronger tests reduce apparent training-task mastery. Arctic-SWE provides a documented starting point for studying better algorithms, data and environments for asynchronous agent RL.