Asynchronous RL

RL: Reinforcement Learning

Follow this topic Ask about this topic

Momentum

7 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 27

All topics

A Closed-Loop Non-Asymptotic Convergence Analysis of PPO with Learned Critics and Clipping
Oct 7, 2026Junwei Su, Mengfan Liu, Yanyong Zhang +1Proximal Policy Optimization Actor-Critic Methods
School of Artificial Intelligence and Data Science, University of Science and Technology of China · Department of Computer Science, University of Hong Kong
COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning
Oct 7, 2026Zicheng Hu, Zhijian Zhou, Xuan Zhang +7Asynchronous RL RL Post-Training
Fudan University · Meituan · East China Normal University
On KL-Regularized Policy Optimization
Oct 6, 2026Yifan ZhangRL for LLM Agents KL-Regularized RL
Princeton University
Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysis
Oct 1, 2026Qijia He, Ruinan Jin, Jun Luo +2Asynchronous RL RL for Language Model Reasoning
The Ohio State University · Arizona State University
Sharp Statistical Rates for Asynchronous TD Learning with Markovian Data
Sep 30, 2026Yang PengAsynchronous RL Stochastic Approximation
Yau Mathematical Sciences Center, Tsinghua University
Where Does Staleness Accumulate? Pool Aware Effective Staleness Control for Asynchronous RL in LLM Post-Training
Sep 29, 2026Chenliang Li, Neiwen Ling, Zijun Wei +1Asynchronous RL RL for Language Model Reasoning
ByteDance · Texas A&M University · University of Virginia
PEARL: Adaptive Prefill-Decode Execution with Elasticity for Agentic Reinforcement Learning
Sep 28, 2026Jiaan Zhu, Wei Gao, Youhui Bai +6Agentic RL Asynchronous RL
USTC China
EBRL: Asynchronous Embodied RL by Multi-Grained Resource Management
Sep 23, 2026Liang Mi, Weijun Wang, Bowen Gao +12Asynchronous RL Reinforcement Learning
Nanjing University · Institute for AI Industry Research (AIR), Tsinghua University · Nanjing University of Posts and Telecommunications +1
High-Probability Nash Regret for Decentralized Learning in Markov $α$ -Potential Games: Episodic and Fully Online Asynchronous Algorithms with Applications to Markov Congestion Games
Sep 14, 2026S. Rasoul EtesamiPolicy Gradient Asynchronous RL
Department of Industrial and Systems Engineering Coordinated Science Laboratory University of Illinois Urbana-Champaign, Urbana, IL, USA
BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RL
Sep 9, 2026Guanqun Zhao, Zijun Xie, Binbin Zheng +3Asynchronous RL Off-Policy RL
Beijing University of Posts and Telecommunications · Baidu Inc. · Peking University +1
SmoothRL: Online Reinforcement Learning During Asynchronous Execution
Aug 30, 2026Guang Gao, Yuxuan Nong, Baifu Huang +1Robot Policy Learning Asynchronous RL
Astribot Team
TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling
Aug 11, 2026Yanyu Ren, Xizheng Wang, Xiao Liu +8Agentic RL Asynchronous RL
Tsinghua University · Zhongguancun Laboratory
Learning under Opponent Unawareness in Linear-Quadratic Stochastic Games
Aug 8, 2026Dantong Chu, Xuefeng Gao, Yufei ZhangNash Equilibrium Asynchronous RL
Department of Systems Engineering and Engineering Management, The Chinese University of Hong Kong, Hong Kong, China. · Department of Mathematics, Imperial College London, UK.
Deconstructing Off-Policy Ratios: Entropy-Normalized Trust Regions for Asynchronous Reinforcement Learning
Jul 24, 2026Guanqun Zhao, Zijun Xie, Binbin Zheng +5Asynchronous RL RL for Language Model Reasoning
Beijing University of Posts and Telecommunications · Baidu Inc. · Peking University +2
Robust Asynchronous Q-Learning under Reward and State Corruption via Batching
Jul 23, 2026Sreejeet Maity, Aritra MitraAsynchronous RL Q-Learning
Department of Electrical and Computer Engineering, North Carolina State University.
CHMAS: A Coupled Hierarchical Framework for Multi-Agent Reinforcement Learning
Jul 21, 2026Dongming Wang, Jie Xu, Yanyu Zhang +1Multi-Agent Coordination Asynchronous RL
Department of Electrical and Computer Engineering, University of California, Riverside, CA 92521, USA
Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning
Jul 21, 2026Junyao Yang, Yucheng Shi, Zongxia Li +6Asynchronous RL Proximal Policy Optimization
1Tencent Hy LLM Frontier · 2National University of Singapore · University of Maryland, College Park +3
Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning
Jul 8, 2026Zhenyu Hou, Yujiang Li, Jie Tang +1Agentic RL Asynchronous RL
Tsinghua University · Work done while ZH and YL interned at Z.AI.
Scaling Laws for Collapse in Asynchronous GRPO
Jul 1, 2026Jingwei Song, Haofeng Xu, Jie Xiao +7Asynchronous RL Reinforcement Learning
The University of Hong Kong · Shanghai Jiao Tong University · Gradient +2
Retroactive Advantage Correction: Closed-Form V-Trace Bias Correction for Delay-Aware RLHF
Jun 25, 2026Arnav RajAsynchronous RL Reinforcement Learning
Department of Computer Science and Engineering, Indian Institute of Technology Delhi, New Delhi, India.
AsyncWebRL: Efficient Multi-Step RL for Visual Web Agents
Jun 4, 2026Hao Bai, Rui Yang, Chenlu Ye +3Asynchronous RL Reinforcement Learning
1UIUC · 2Microsoft · 3CMU
Failure Modes of Deep Multi-Agent RL in Asynchronous Pricing: Reproducible Triggers, Trace Diagnostics, and a Partial Fix
Jun 3, 2026Shree Murthy, Rohan PandeyAsynchronous RL Algorithmic Collusion
DigitalOcean, USA
ASymPO: Asymmetric-Scale Policy Optimization for Asynchronous LLM Post-Training Without Behavior Information
Jun 2, 2026Zehua Liu, Yuxuan Yao, Xiaojin Fu +2Asynchronous RL Reinforcement Learning
Huawei Technologies
Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction
May 12, 2026Zhong Guan, Yongjian Guo, Haoran Sun +5Asynchronous RL Reinforcement Learning
Tianjin University · Tsinghua University · JDT AI Infra +1
MARLaaS: Multi-Tenant Asynchronous Reinforcement Learning as a Service
May 8, 2026Timothy Tin Long Yu, Gursimran Singh, Ge Shi +3Asynchronous RL Reinforcement Learning
Huawei Technologies Canada
DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training
Apr 29, 2026Tianhao Hu, Xiangcheng Liu, Youshao Xiao +15RL for Language Models Asynchronous RL
Meituan, China
AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training
Date pendingZhenyu Han, Ansheng You, Haibo Wang +16RL for Language Models Asynchronous RL
Huawei · Individual Researcher

Oct 7, 2026, cs.LGJ/K move · Enter open · S save

A Closed-Loop Non-Asymptotic Convergence Analysis of PPO with Learned Critics and Clipping

Junwei Su, Mengfan Liu, Yanyong Zhang, Chuan Wu

School of Artificial Intelligence and Data Science, University of Science and Technology of China · Department of Computer Science, University of Hong Kong

Proximal Policy Optimization Actor-Critic Methods Policy Gradient Methods Sample-Efficient RL Asynchronous RL

Open paper AskPDF arXiv

Despite its widespread use, Proximal Policy Optimization with clipping (PPO-Clip) remains difficult to tune, and the interactions among critic learning, clipping, and rollout reuse remain incompletely understood. We develop a \emph{non-asymptotic} analysis of PPO-Clip as a \emph{closed-loop actor--critic} system. It captures actor--critic coupling, nonsmooth probability-ratio clipping, finite-batch reuse, and predictable early stopping under explicit coverage and critic regularity assumptions, using raw GAE and Monte Carlo critic targets. Our synchronous and asynchronous guarantees jointly characterize policy stationarity and the tracking accuracy of the learned critic, with explicit dependence on algorithmic parameters. A sufficient coupling condition gives optimization, critic tracking, clipping, and finite-batch errors a common amplification bound. The asynchronous result also requires a delay-dependent critic stepsize restriction; violating these conditions does not establish divergence. For finite layered MDPs with tabular critics, a uniform bound on the actual clipped-gradient class replaces complete-trajectory counting. A verified growing-horizon family has polynomial sample complexity, and a two-time-scale schedule gives

O(T^{-2/5})

stationarity and critic-tracking bounds with explicit fresh-rollout accounting. These results together advance our understanding about PPO and provide theoretical guidance in tuning.

Momentum

Related topics

Latest papers 27