Asynchronous RL

RL: Reinforcement Learning

Overview

Asynchronous reinforcement learning studies agents that collect experience, act, or update at different times, as in distributed simulators, multi-agent systems, robots with inference delays, and language-model post-training. Systems methods overlap rollout generation, environment interaction, and optimization, while algorithmic methods control stale data through importance weighting, trust regions, advantage correction, or staleness-aware scheduling. Work is evaluated on settings ranging from tabular Q-learning and Markov games to tool-use, reasoning, web-agent, and robotics tasks. The central tension is gaining throughput by reducing synchronization waits without letting policy lag and delayed feedback destabilize learning or degrade performance; theory and practical safeguards address this tradeoff with varying guarantees.

Latest papers

CardsList

52 papers4 papers in the last four weeks, down 73% on the four weeks before. 0.0% of all new papers.

  1. Regularized Score Centering

    Nov 16, 2026Hao Bai, Chenlu Ye, Xuanchang Zhang +2Agentic RLAsynchronous RL

  2. Sharp Gaussian Approximation for Q-Learning

    Nov 15, 2026Xingtu LiuQ-LearningReinforcement Learning

  3. Adaptive Scheduling Pipeline for Multi-Instance Asynchronous Reinforcement Learning

    Nov 6, 2026Salah Chikhi, Luis Zuin, Entong Li +1LLM Inference SchedulingReinforcement Learning

  4. Arctic-SWE: Revisiting the Long Horizon Agent Training by Modeling-System Co-Design

    Oct 17, 2026Boyi Liu, Yite Wang, Xinyu Lian +6Long-Horizon RLRL for LLM Agents

  5. ReFlow: Replay-Driven Decoupling for Efficient RL Post-Training

    Oct 15, 2026Yukun Tian, Junwei Su, Haofeng Xu +6RL Post-TrainingRL for Language Model Reasoning

  6. Long-Horizon RL Needs Muon

    Oct 14, 2026Kai Ruan, Jinghao Lin, Tao Chen +3Long-Horizon RLLong-Horizon LLM Agents

  7. KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization

    Oct 10, 2026Sihwan Park, Sung-Yub Kim, Doohyuk Jang +1KV-Cache QuantizationRL Post-Training

  8. Hitting-Time Bounds for Asynchronous Q-Learning

    Oct 9, 2026Yang PengReinforcement LearningQ-Learning

  9. A Closed-Loop Non-Asymptotic Convergence Analysis of PPO with Learned Critics and Clipping

    Oct 7, 2026Junwei Su, Mengfan Liu, Yanyong Zhang +1Proximal Policy OptimizationActor-Critic Methods

  10. COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning

    Oct 7, 2026Zicheng Hu, Zhijian Zhou, Xuan Zhang +7Asynchronous RLRL Post-Training

  11. Asynchrony Is Nearly Free for Evolution Strategies on Long-Horizon LLM Agentic Tasks

    Oct 3, 2026William Hoy, Jingxuan Fan, Nurcin Celik +1Evolution StrategiesRL Post-Training

  12. Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysis

    Oct 1, 2026Qijia He, Ruinan Jin, Jun Luo +2Asynchronous RLRL for Language Model Reasoning

  13. Near-Minimax-Optimal Rates for Asynchronous TD Learning with Markovian Data

    Sep 30, 2026Yang PengAsynchronous RLStochastic Approximation

  14. Where Does Staleness Accumulate? Pool Aware Effective Staleness Control for Asynchronous RL in LLM Post-Training

    Sep 29, 2026Chenliang Li, Neiwen Ling, Zijun Wei +1Asynchronous RLRL for Language Model Reasoning

  15. EBRL: Asynchronous Embodied RL by Multi-Grained Resource Management

    Sep 23, 2026Liang Mi, Weijun Wang, Bowen Gao +12Asynchronous RLReinforcement Learning

  16. Stable and Efficient Real-World Online VLA Post-Training via Asynchronous Replay-Anchored Policy Improvement

    Sep 19, 2026Jiarui Yang, Jiajin Zhang, Bin Zhu +2Robot Policy LearningRL Post-Training

  17. High-Probability Nash Regret for Decentralized Learning in Markov αα-Potential Games: Episodic and Fully Online Asynchronous Algorithms with Applications to Markov Congestion Games

    Sep 14, 2026S. Rasoul EtesamiPolicy GradientAsynchronous RL

  18. BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RL

    Sep 9, 2026Guanqun Zhao, Zijun Xie, Binbin Zheng +3Asynchronous RLOff-Policy RL

  19. SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RL

    Aug 25, 2026Kai Ruan, Jinghao Lin, Qianshan Wei +2RL for LLM AgentsAgentic RL

  20. Learning to Act While Waiting: RL Finetuning of Generalist Robot Policies Under Inference Latency

    Aug 24, 2026Brian Zhu, Momen Khalil, E Harrison +17RL for RoboticsRobot Policy Learning

  21. TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling

    Aug 11, 2026Yanyu Ren, Xizheng Wang, Xiao Liu +8Agentic RLAsynchronous RL

  22. Learning under Opponent Unawareness in Linear-Quadratic Stochastic Games

    Aug 8, 2026Dantong Chu, Xuefeng Gao, Yufei ZhangNash EquilibriumAsynchronous RL

  23. Deconstructing Off-Policy Ratios: Entropy-Normalized Trust Regions for Asynchronous Reinforcement Learning

    Jul 24, 2026Guanqun Zhao, Zijun Xie, Binbin Zheng +5Asynchronous RLRL for Language Model Reasoning