Actor-Critic Methods

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 62

All topics

A Closed-Loop Non-Asymptotic Convergence Analysis of PPO with Learned Critics and Clipping
Oct 7, 2026Junwei Su, Mengfan Liu, Yanyong Zhang +1Proximal Policy Optimization Actor-Critic Methods
School of Artificial Intelligence and Data Science, University of Science and Technology of China · Department of Computer Science, University of Hong Kong
Foundation Model-Aided Multi-Agent Reinforcement Learning for Wireless Random Access Network Optimization
Oct 6, 2026Myeung Suk Oh, Zhiyao Zhang, Alvaro Velasquez +2Wireless Resource Allocation Decentralized MARL
The Ohio State University Columbus, Ohio, USA · University of Colorado Boulder Boulder, Colorado, USA · Pathfinder Applied Research, LLC Ashburn, Virginia, USA
Taylor Representations for Model-Free RL in Networked MDPs
Oct 4, 2026Salah Chikhi, Abdelhaq Chaoui, Asuman Ozdaglar +1Markov Decision Processes Representation Learning for RL
Massachusetts Institute of Technology
After Cooperation Is Learned: Gradient Routing and Optimizer-Dependent Maintenance in Multi-Agent Reinforcement Learning
Oct 1, 2026Chaoyuan Hao, Wentao Yue, Tianyou Lai +4Actor-Critic Methods Cooperative MARL
Trust the Critic More
Sep 30, 2026Kaiyue Wen, Luke Bailey, Arvind Mahankali +1RL for Language Model Reasoning Credit Assignment in RL
Stanford University
DAMPER: Return-Prioritized Gradient Control for Smooth Policies
Sep 30, 2026Seokmin Ko, Taewon Goo, Kihyuk HongRL Control Reinforcement Learning
KAIST
Privy to the Foil: Recasting Value Estimation with a Self-Privileged Critic for RLVR
Sep 29, 2026Kun Liang, Chenming Tang, Clive Bai +5Value Function Estimation RL for Language Model Reasoning
School of Computer Science, Peking University · National Key Laboratory for Multimedia Information Processing, Peking University · Foundation Model Department, Tencent
A Concentration Bound for Two-Timescale Actor-Critic Algorithm
Sep 24, 2026Prashansa Panda, Shalabh BhatnagarAverage-Reward RL Two-Timescale Stochastic Approximation
Department of Computer Science and Automation IISc Bangalore
PACT: From Credit Assignment to Critic Alignment
Sep 22, 2026Jiayan Fu, Hang Xu, Yong Zhang +4Reinforcement Learning Token-Level Credit Assignment
AllSpark Team
Fully Byzantine-Resilient Multi-Agent Reinforcement Learning
Sep 22, 2026Haejoon Lee, Dimitra PanagouDecentralized MARL Actor-Critic Methods
Robotics Department, University of Michigan, Ann Arbor, MI, USA
CARE-VI: Conservative Adaptive Reliability Estimation for Value Improvement in Off-Policy Actor-Critic Learning
Sep 17, 2026Xiang Zou, Shengzhu Shi, Junqi Gao +1Temporal-Difference Learning Actor-Critic Methods
School of Mathematics, Harbin Institute of Technology, Harbin 150001, China
Locally-Guided Actor-Critic: Training a Goal-conditioned Actor with a Subgoal-aware Critic
Aug 31, 2026Olivier Serris, Stéphane Doncieux, Olivier SigaudReinforcement Learning Sparse-Reward RL
ISIR · Sorbonne Université, CNRS
The Time Value of Evolution
Aug 13, 2026Matthew Siper, Ahmed Khalifa, Julian TogeliusEvolutionary Optimization Policy Gradient Methods
Nof1, New York University · Nof1, University of Malta
Learning Suffers More Than the Policy Class Under Partial Observability: A Closed-Form Analysis
Aug 7, 2026Idil GözelRL Control Reinforcement Learning
University College London · Department of Physics and Astronomy University College London, London, UK
Toward Integrating Adaptive Experience Replay and Online Uncertainty Estimation in Safe Actor-Critic Optimal Control
Aug 5, 2026Mahshad Rastegarmoghaddam, Davoud Nikkhouy, Shima SamadzadehRL Control Reinforcement Learning
Department of Mechanical Engineering, Politecnico di Milano, Milan, Italy
Foundations of Reinforcement Learning and Control:Connections and New Perspectives
Aug 3, 2026Claire Vernade, Onno Eberhard, Martha White +4RL Control Data-Driven Control
University of Technology Nuremberg · Max Planck Institute for Intelligent Systems, Tübingen, Germany · University of Alberta and Alberta Machine Intelligence Institute (Amii) +2
ReBRAC-v2: The Return of the King
Aug 2, 2026Denis Tarasov, Robert K. KatzschmannReinforcement Learning Offline RL
Soft Robotics Lab, D-MAVT ETH Zurich, Switzerland
Hierarchical Multilevel Monte Carlo for Order-Optimal Neural Actor-Critic in Average-Reward CMDPs
Jul 30, 2026Ankur Naskar, Vaneet AggarwalAverage-Reward RL Constrained RL
Indian Institute of Science, India · Purdue University, USA
Collaborative Weighting with Pessimistic Critic for Mitigating Overestimation in Off-Policy Reinforcement Learning
Jul 29, 2026Gong Gao, Xiao Lai, Ziqi Xie +3Reinforcement Learning Actor-Critic Methods
School of Computer Science and Technology, Tongji University, 200092, Shanghai, China · Shanghai University of Engineering Science, 201620, Shanghai, China
Hybrid Advantage Estimation with Unified Critic for VLM Agentic Reinforcement Learning
Jul 26, 2026Wenxuan Zhang, Yuhui Wang, Donggang Jia +5Agentic RL Actor-Critic Methods
KAUST
Comparative Study of Multi-Agent Actor-Critic Algorithms in Parameterized Action Reinforcement Learning
Jul 21, 2026Ubayd Ali Bapoo, Clement N NyirendaReinforcement Learning Soft Actor-Critic
Department of Computer Science University of the Western Cape Cape Town, South Africa
Deconstructing Actor-Critic: A Large-scale Empirical Study of Design Components for Practitioners
Jul 14, 2026Haseeb Shah, Lingwei Zhu, Adam White +1RL Control Reinforcement Learning
Department of Computing Science, University of Alberta, Canada · Alberta Machine Intelligence Institute (amii), Canada · Great Bay University, China +1
Actor-Critic Learning for Extended Mean Field Control with Deterministic Policies
Jul 13, 2026Ziheng Cheng, Xin Guo, Huyên Pham +1Policy Gradient Continuous-Time RL
University of California, Berkeley · Ecole Polytechnique, CMAP · Imperial College London
Gauging, Measuring, and Controlling Critic Complexity in Actor-Critic Reinforcement Learning
Jul 1, 2026Konstantin GarbersReinforcement Learning Actor-Critic Methods
Peking University
Revisiting Action Factorization for Complex Action Spaces
Jun 25, 2026Timothy Flavin, Sandip SenReinforcement Learning Actor-Critic Methods
The University of Tulsa, Tulsa, Oklahoma, USA
Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Data
Jun 25, 2026Erhan Bayraktar, Martin Hernandez, Qinxin Yan +1Reinforcement Learning Stochastic Optimal Control
Department of Mathematics, University of Michigan, Ann Arbor, MI, USA. · Department of Statistics and Data Science, University of California, Los Angeles, CA, USA. · Program in Applied and Computational Mathematics, Princeton University, Princeton, NJ, USA.
EVOM: Agentic Meta-Evolution of Actor-Critic Architectures for Reinforcement Learning
Jun 24, 2026Boyun Zhang, Chao Wang, Kai WuLarge Language Model-Guided NAS Neural Architecture Search
Xidian University
Bias-Controlled Primal-Dual Natural Actor-Critic: Optimal Rates for Constrained Multi-Objective Average-Reward RL
Jun 23, 2026Ankur Naskar, Swetha Ganesh, Vaneet AggarwalMulti-Objective Reinforcement Learning Reinforcement Learning
Indian Institute of Science, Bengaluru, India · Purdue University, USA
Critic Architecture Matters: Dual vs. Unified Critics for Humanoid Loco-Manipulation
Jun 10, 2026Mehmet Turan YardımcıMulti-Objective Reinforcement Learning Humanoid Loco-Manipulation
Department of Computer Engineering C¸ukurova University Adana, Türkiye
Phi-Actor-Critic: Steering General-Sum Games to Pareto-Efficient Correlated Equilibria
Jun 9, 2026Wongyu Lee, Francesco Lelli, Omran Ayoub +1Multi-Agent Coordination Regret Minimization
Politecnico di Milano · Tilburg University · University of Applied Sciences and Arts of Southern Switzerland

Oct 7, 2026, cs.LGJ/K move · Enter open · S save

A Closed-Loop Non-Asymptotic Convergence Analysis of PPO with Learned Critics and Clipping

Junwei Su, Mengfan Liu, Yanyong Zhang, Chuan Wu

School of Artificial Intelligence and Data Science, University of Science and Technology of China · Department of Computer Science, University of Hong Kong

Proximal Policy Optimization Actor-Critic Methods Policy Gradient Methods Sample-Efficient RL Asynchronous RL

Open paper AskPDF arXiv

Despite its widespread use, Proximal Policy Optimization with clipping (PPO-Clip) remains difficult to tune, and the interactions among critic learning, clipping, and rollout reuse remain incompletely understood. We develop a \emph{non-asymptotic} analysis of PPO-Clip as a \emph{closed-loop actor--critic} system. It captures actor--critic coupling, nonsmooth probability-ratio clipping, finite-batch reuse, and predictable early stopping under explicit coverage and critic regularity assumptions, using raw GAE and Monte Carlo critic targets. Our synchronous and asynchronous guarantees jointly characterize policy stationarity and the tracking accuracy of the learned critic, with explicit dependence on algorithmic parameters. A sufficient coupling condition gives optimization, critic tracking, clipping, and finite-batch errors a common amplification bound. The asynchronous result also requires a delay-dependent critic stepsize restriction; violating these conditions does not establish divergence. For finite layered MDPs with tabular critics, a uniform bound on the actual clipped-gradient class replaces complete-trajectory counting. A verified growing-horizon family has polynomial sample complexity, and a two-time-scale schedule gives

O(T^{-2/5})

stationarity and critic-tracking bounds with explicit fresh-rollout accounting. These results together advance our understanding about PPO and provide theoretical guidance in tuning.

Momentum

Related topics

Latest papers 62