Actor-Critic Methods

Follow this topic Ask about this topic

Actor-critic methods pair an actor, which chooses actions, with a critic, which estimates their value and helps improve the policy. On-policy methods such as A2C and PPO and off-policy methods such as SAC, TD3, and DDPG are widely used for continuous control, robotics, multi-agent learning, and language-model agents. Researchers also adapt critics for offline learning, risk and safety constraints, partial observability, and long-horizon credit assignment, while developing convergence and sample-complexity guarantees. A central challenge is that inaccurate or unstable critic estimates can misdirect policy updates, so better value estimation and training stability must be balanced against data efficiency and computational cost.

Latest papers

108 papers20 papers in the last four weeks, up 82% on the four weeks before. 0.1% of all new papers.

Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
Mar 30, 2026Max Qiushi Lin, Reza Asad, Kevin Tan +3Actor-Critic Methods Policy Learning
Simon Fraser University · University of Pennsylvania · Mila, McGill University +1
Can an Actor-Critic Optimization Framework Improve Analog Design?
Mar 25, 2026Sounak Dutta, Fin Amin, Sushil Panda +3Analog Circuit Sizing Electronic Design Automation
North Carolina State University, Department of Electrical and Computer Engineering, Raleigh, North Carolina, USA
Visualizing Critic Match Loss Landscapes for Interpretation of Online Reinforcement Learning Control Algorithms
Mar 15, 2026Jingyi Liu, Jian Guo, Eberhard GillRL Control Actor-Critic Methods
Faculty of Aerospace Engineering, Delft University of Technology, Kluyverweg 1, Delft, 2629 HS, The Netherlands
When Learning Rates Go Wrong: Early Structural Signals in PPO Actor-Critic
Mar 10, 2026Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +3Reinforcement Learning Proximal Policy Optimization
Universitat Polit`ecnica de Val`encia, Valencia, Spain · Universitat Jaume I, Castell´o de la Plana, Spain · Openchip & Software Technologies, Barcelona, Spain
A Recipe for Stable Offline Multi-agent Reinforcement Learning
Mar 9, 2026Dongsu Lee, Daehee Lee, Amy ZhangOffline MARL Actor-Critic Methods
University of Texas at Austin · Sungkyunkwan University
Global Convergence of Average Reward Constrained MDPs with Neural Critic and General Policy Parameterization
Mar 8, 2026Anirudh Satheesh, Pankaj Kumar Barman, Washim Uddin Mondal +1Natural Policy Gradient Constrained RL
Department of Computer Science, University of Maryland, College Park, MD 20740, USA · Indian Institute of Technology, Kanpur, Uttar Pradesh, 208016, India · Department of Electrical Engineering, Indian Institute of Technology, Kanpur, Uttar Pradesh, 208016, India +1
Wasserstein Proximal Policy Gradient
Mar 3, 2026Zhaoyu Zhu, Shuhan Zhang, Rui Gao +1Policy Optimization Policy Gradient Methods
Zhiyuan College, Shanghai Jiao Tong University, Shanghai 200240, China · School of Data Science, The Chinese University of Hong Kong, Shenzhen, Guangdong, China · McCombs School of Business, The University of Texas at Austin, Austin, TX, USA
Actor-Critic Pretraining for Proximal Policy Optimization
Feb 27, 2026Andreas Kernbach, Amr Elsheikh, Nicolas Grupp +2Robotic RL Sample-Efficient RL
Fraunhofer Institute for Manufacturing Engineering and Automation IPA, Stuttgart, Germany · Institute of Industrial Manufacturing and Management IFF, University of Stuttgart, Germany · Institute for Control Engineering of Machine Tools and Manufacturing Units ISW, University of Stuttgart, Germany +1
CACTO-BIC: Scalable Actor-Critic Learning via Biased Sampling and GPU-Accelerated Trajectory Optimization
Feb 23, 2026Elisa Alboni, Pietro Noah Crestaz, Elias Fontanari +1Actor-Critic Methods Trajectory Optimization
Dept. of Industrial Engineering, University of Trento, Italy · LAAS-CNRS, Université de Toulouse, CNRS, Toulouse, France
Flow Actor-Critic for Offline Reinforcement Learning
Feb 20, 2026Jongseong Chae, Jongeui Park, Yongjae Shin +3Flow-Based Policy Learning Conservative RL
KAIST · UNIST
SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfer
Feb 19, 2026Nathan Samuel de Lara, Florian ShkurtiOffline RL Offline-To-Online RL
Department of Computer Science, University of Toronto, Toronto, Canada · Vector Institute
Decoupled Continuous-Time Reinforcement Learning via Hamiltonian Flow
Feb 16, 2026Minh NguyenReinforcement Learning Continuous-Time RL
Google · University of Texas at Austin
Pretraining in Actor-Critic Reinforcement Learning for Locomotion
Oct 14, 2025Jiale Fan, Andrei Cramariuc, Tifanny Portela +1RL for Robotics Reinforcement Learning
Section of Microengineering, EPFL, 1015 Lausanne, Switzerland · Robotic Systems Lab, ETH Zurich, 8092 Zürich, Switzerland · ETH AI Center, ETH Zurich, 8092 Zürich, Switzerland
Context Value Informed In-Context Reinforcement Learning
Sep 27, 2025Wenhao Zhang, Shao Zhang, Xihuai Wang +2In-Context RL Test-Time RL
Shanghai Jiao Tong University
Efficient Q-Learning and Actor-Critic Methods for Robust Average-Reward Reinforcement Learning
Jun 8, 2025Yang Xu, Swetha Ganesh, Vaneet AggarwalRobust Markov Decision Processes Reinforcement Learning
Purdue University, West Lafayette, Indiana, USA 47907
Streaming Deep Reinforcement Learning Finally Works
Oct 18, 2024Mohamed Elsayed, Elena Sorina Lupu, Gautham Vasan +1Reinforcement Learning Actor-Critic Methods
Department of Computing Science, University of Alberta, Edmonton, Canada · Alberta Machine Intelligence Institute (Amii) · Canada CIFAR AI Chair
Topology-Guided Modular Actor-Critic Learning for Continuous Systems under Temporal Objectives
Apr 20, 2023Lening Li, Zhentian Qian, Jianan Xia +7RL Control Reinforcement Learning
Ningbo University, Ningbo, China · Robotics Engineering Program, Worcester Polytechnic Institute, Worcester, MA, USA · Ningbo Zhenhai Silver-Ball Bearing Co., Ltd., Ningbo, China
Diversity-Enriched Option-Critic
Nov 4, 2020Anand Kamat, Doina PrecupHierarchical RL Intrinsic Motivation
McGill University · Mila · 3Google deepmind

Mar 30, 2026J/K move · Enter open · S save

Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes

Max Qiushi Lin, Reza Asad, Kevin Tan, Haque Ishfaq+2

Simon Fraser University · University of Pennsylvania · Mila, McGill University+1

Actor-Critic Methods Policy Learning Linear Markov Decision Processes Sample-Efficient RL

Open paper AskPDF arXiv

Although actor-critic methods have been successful in practice, their theoretical analyses have several limitations. Specifically, existing theoretical work either sidesteps the exploration problem by making strong assumptions or analyzes impractical methods with complicated algorithmic modifications. Moreover, the actor-critic methods analyzed for linear MDPs often employ natural policy gradient and construct "implicit" policies without explicit parameterization. Such policies are computationally expensive to sample from, making the environment interactions inefficient. To that end, we focus on the finite-horizon linear MDPs and propose an optimistic actor-critic framework that uses parametric log-linear policies. In particular, we introduce a tractable

\textit{logit-matching}

regression objective for the actor. For the critic, we use approximate Thompson sampling via Langevin Monte Carlo to obtain optimistic value estimates. We prove that the resulting algorithm achieves

\widetilde{\mathcal{O}}(ε^{-4})

and

\widetilde{\mathcal{O}}(ε^{-2})

sample complexity in the on-policy and off-policy setting, respectively. Our results match prior theoretical work in achieving the state-of-the-art sample complexity, while our algorithm is more aligned with practice.