Actor-Critic Methods

Follow this topic Ask about this topic

Actor-critic methods pair an actor, which chooses actions, with a critic, which estimates their value and helps improve the policy. On-policy methods such as A2C and PPO and off-policy methods such as SAC, TD3, and DDPG are widely used for continuous control, robotics, multi-agent learning, and language-model agents. Researchers also adapt critics for offline learning, risk and safety constraints, partial observability, and long-horizon credit assignment, while developing convergence and sample-complexity guarantees. A central challenge is that inaccurate or unstable critic estimates can misdirect policy updates, so better value estimation and training stability must be balanced against data efficiency and computational cost.

Latest papers

108 papers20 papers in the last four weeks, up 82% on the four weeks before. 0.1% of all new papers.

A Unified Analysis of Importance-Corrected Momentum Actor-Critic over a Markovian Trajectory
Oct 24, 2026Yanjie Dong, Zhou Su, Wei Wang +1Reinforcement Learning Actor-Critic Methods
School of Cyber-Science and Engineering, Xi’an Jiaotong University · School of Information and Communications Engineering, Xi’an Jiaotong University · State Key Laboratory of Autonomous Intelligent Unmanned Systems, Beijing Institute of Technology
Financial Context-Rich Strategy for Actor-Critic based Agents: A Deep Reinforcement Learning Approach for Stock Trading
Oct 23, 2026Hrishikesh Singh, Aarti Sharma, Millie PantAlgorithmic Trading Actor-Critic Methods
Department of Applied Mathematics & Scientific Computing, Indian Institute of Technology Roorkee, Roorkee, India
Continuous-Time Policy Gradient from Discrete Data: Convergence Analysis and Model-Free Algorithms
Oct 20, 2026Martin Hernandez, Xinyan Qing, Mo Zhou +1Continuous-Time RL Model-Free RL
Department of Statistics and Data Science, University of California, Los Angeles · Department of Statistics, University of Michigan · School of Mathematical Sciences, Peking University
From Outcomes to Steps: Reliable Success-Aware Critic Learning for Long-Horizon Language Agents
Oct 20, 2026Jiaxi Liu, Shixiong Kai, Zhentao Tang +4Reinforcement Learning RL for LLM Agents
Shanghai Jiao Tong University · Huawei Noah’s Ark Lab
Sample-Efficient Reinforcement Learning with General Utilities via Neural Dual Critics
Oct 17, 2026Ankur Naskar, Vaneet AggarwalReinforcement Learning Sample-Efficient RL
Indian Institute of Science, India · Purdue University, USA
ACRO: Actor-Critic Rollout Orchestration
Oct 14, 2026Junhoo Lee, Seungyeon Kim, Baekseung Kim +4Robotic Manipulation Robot Policy Learning
Korea Advanced Institute of Science and Technology (KAIST) · Seoul National University
FOCUS: From Privileged States to RGB-D with Controlled Modality Switching and Representation Alignment
Oct 8, 2026Filip Grigorov, Kourosh Darvish, Nandita VijaykumarRobotic RL Visual RL
University of Toronto Canada
A Closed-Loop Non-Asymptotic Convergence Analysis of PPO with Learned Critics and Clipping
Oct 7, 2026Junwei Su, Mengfan Liu, Yanyong Zhang +1Proximal Policy Optimization Actor-Critic Methods
School of Artificial Intelligence and Data Science, University of Science and Technology of China · Department of Computer Science, University of Hong Kong
Foundation Model-Aided Multi-Agent Reinforcement Learning for Wireless Random Access Network Optimization
Oct 6, 2026Myeung Suk Oh, Zhiyao Zhang, Alvaro Velasquez +2Wireless Resource Allocation Decentralized MARL
The Ohio State University Columbus, Ohio, USA · University of Colorado Boulder Boulder, Colorado, USA · Pathfinder Applied Research, LLC Ashburn, Virginia, USA
Taylor Representations for Model-Free RL in Networked MDPs
Oct 4, 2026Salah Chikhi, Abdelhaq Chaoui, Asuman Ozdaglar +1Markov Decision Processes Representation Learning for RL
Massachusetts Institute of Technology
Policy as Data: Replay-Based Policy Dual Averaging via Advantage Regression
Oct 3, 2026Nianli Peng, Geoffrey J. Gordon, Kianté BrantleyActor-Critic Methods Experience Replay
Harvard University · Carnegie Mellon University
Amortized Score-Hamiltonian Policy Iteration: A Grid-Free Scheme for Relaxed Stochastic Control Problems
Oct 3, 2026Qi Feng, Gu WangStochastic Optimal Control Policy Iteration
Department of Mathematics, Florida State University, Tallahassee, 32306 · Department of Mathematical Sciences, Worcester Polytechnic Institute, Worcester, MA 01609
Memory-State Critic for Asymmetric Actor-Critic with Application to Vision-Based Pursuit-Evasion
Oct 2, 2026Arthur Louette, Alejandro Sánchez Roncero, Gaspard Lambrechts +4Asynchronous Advantage Actor-Critic Actor-Critic Methods
University of Liège, Belgium · KTH Royal Institute of Technology, Sweden · McGill University & Mila Québec AI Institute, Canada +1
Learning What to Investigate Next: Meta-Reasoning for Long-Horizon Research Agents
Oct 1, 2026Ankur Samanta, Yonathan Efroni, Paul Sajda +2Autonomous Research Agents Long-Horizon Agent Tasks
Meta AI · Columbia University · Work done at Meta +1
After Cooperation Is Learned: Gradient Routing and Optimizer-Dependent Maintenance in Multi-Agent Reinforcement Learning
Oct 1, 2026Chaoyuan Hao, Wentao Yue, Tianyou Lai +4Actor-Critic Methods Cooperative MARL
Trust the Critic More
Sep 30, 2026Kaiyue Wen, Luke Bailey, Arvind Mahankali +1RL for Language Model Reasoning Credit Assignment in RL
Stanford University
DAMPER: Return-Prioritized Gradient Control for Smooth Policies
Sep 30, 2026Seokmin Ko, Taewon Goo, Kihyuk HongRL Control Reinforcement Learning
KAIST
Privy to the Foil: Recasting Value Estimation with a Self-Privileged Critic for RLVR
Sep 29, 2026Kun Liang, Chenming Tang, Clive Bai +5Value Function Estimation RL for Language Model Reasoning
School of Computer Science, Peking University · National Key Laboratory for Multimedia Information Processing, Peking University · Foundation Model Department, Tencent
EasyPPO: Stabilizing the Critic Is Key
Sep 29, 2026Xuanyi Zhou, Qiuyang Mang, Huanzhi Mao +7RL for Language Models Proximal Policy Optimization
University of California, Berkeley · Princeton University
ABC: Advantage-Based Control Variates for Reinforcement Learning with Verifiable Rewards
Sep 28, 2026Hsiao-Ru Pan, Florent Draye, Bernhard SchölkopfReinforcement Learning Actor-Critic Methods
MPI-IS · ELLIS Institute Tübingen
Schrödinger--Föllmer Actor--Critic: Diffusion Policy Improvement with Finite-Sample Analysis
Sep 27, 2026Yuling Jiao, Lican Kang, Jerry Zhijian Yang +1Diffusion Policy RL for Diffusion Models
School of Artificial Intelligence, and Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, 430072, China. · Institute for Math and AI, Hubei Key Laboratory of Computational Science, and School of Artificial Intelligence, Wuhan University, Wuhan, 430072, China. · School of Mathematics and Statistics, and Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, 430072, China. +1
$T^5$ : Twin-Critic Training for Token-Level Thoughts in Reinforcement Mid-Training
Sep 26, 2026Nan Qiao, Yebin Yang, Weinong Wang +10RL for Language Models Reinforcement Learning
Tsinghua University · Tencent · Shanghai Jiao Tong University +4
Deep Reinforcement Learning for Equity Trading: Benchmarking Actor-Critic Methods with Forward Retraining
Sep 25, 2026Bicheng Wang, Xinyi ZhangAlgorithmic Trading Reinforcement Learning
Stanford University
A Concentration Bound for Two-Timescale Actor-Critic Algorithm
Sep 24, 2026Prashansa Panda, Shalabh BhatnagarAverage-Reward RL Two-Timescale Stochastic Approximation
Department of Computer Science and Automation IISc Bangalore
PACT: From Credit Assignment to Critic Alignment
Sep 22, 2026Jiayan Fu, Hang Xu, Yong Zhang +4Reinforcement Learning Token-Level Credit Assignment
AllSpark Team
Fully Byzantine-Resilient Multi-Agent Reinforcement Learning
Sep 22, 2026Haejoon Lee, Dimitra PanagouDecentralized MARL Actor-Critic Methods
Robotics Department, University of Michigan, Ann Arbor, MI, USA
Design of Adaptive PID Controller Based On Asynchronous Advantage Actor Critic Learning Method for QuadCopter Control
Sep 17, 2026Ali Jokar, Aria AlastyQuadrotor Control Adaptive Control
Sharif AgRoLab School of Mechanical Engineering Sharif University of Technology Tehran, Iran
CARE-VI: Conservative Adaptive Reliability Estimation for Value Improvement in Off-Policy Actor-Critic Learning
Sep 17, 2026Xiang Zou, Shengzhu Shi, Junqi Gao +1Temporal-Difference Learning Actor-Critic Methods
School of Mathematics, Harbin Institute of Technology, Harbin 150001, China
Nonparametric Variance-Penalized Actor-Critic: Statistical Inference for Risk-Sensitive Reinforcement Learning
Sep 13, 2026Saunak Kumar Panda, Tong Li, Yisha Xiang +1Risk-Sensitive RL Actor-Critic Methods
Human-Centered AI Institute, C.T. Bauer College of Business, University of Houston, Houston, TX 77204 USA · Department of Industrial and Systems Engineering, University of Houston, Houston, TX 77204 USA · Department of Mathematics and Statistics, Texas Tech University, Lubbock, TX 79409 USA
Behavioral Cloning Outperforms Entropy-Regularized RL: Critic-Driven Failure of Actor-Critic Methods on Adaptive Tumor Treatment
Sep 6, 2026Aleksandar Dimitrov, Giacomo SpiglerActor-Critic Methods Soft Actor-Critic
Department of Cognitive Science & Artificial Intelligence, Tilburg University, Tilburg, The Netherlands

Oct 24, 2026J/K move · Enter open · S save

A Unified Analysis of Importance-Corrected Momentum Actor-Critic over a Markovian Trajectory

Yanjie Dong, Zhou Su, Wei Wang, Gang Wang

School of Cyber-Science and Engineering, Xi’an Jiaotong University · School of Information and Communications Engineering, Xi’an Jiaotong University · State Key Laboratory of Autonomous Intelligent Unmanned Systems, Beijing Institute of Technology

Reinforcement Learning Actor-Critic Methods Policy Gradient Methods Temporal-Difference Learning Importance Sampling for RL

Open paper AskPDF arXiv

We present a generic actor-critic method that combines a ridge-regularized pathwise least-squares temporal-difference (LSTD) critic with importance-corrected momentum (ICM) actors. The critic solves a ridge-regularized system per frame by Newton-Schulz inversion and needs no stepsize, no knowledge of the mixing time, and no rank condition. Averaging stochastic policy gradients over frames of

T

consecutive transitions, the ICM actors incur a frame-dependent gradient bias of order

\frac{1}{T}

. To our knowledge, we establish the first finite-time bounds for ICM actor-critic methods with normalized and unnormalized steps along a single Markovian trajectory. Up to a stationary-to-discounted distribution mismatch, the averaged gradient norm over

K

updates is bounded by

{\cal O}(K^{-1/3}+\frac{1}{T})

for normalized steps and by

{\cal O}(K^{-1/4}+\frac{1}{T})

for unnormalized steps. We specialize these bounds to Muon, normalized stochastic gradient descent (SGD) with momentum, SGD with momentum, and AdaNorm, and compare them in the nuclear norm. With a dimension-free stepsize, Muon confines the factor

\min\{m,n\}^{1/2}

associated with an

m\times n

actor parameter matrix to the noise and correction terms. The comparison also characterizes when orthogonalization guarantees a larger noiseless ascent than normalization: exactly when the gradient's nuclear rank exceeds the spectral-to-Frobenius curvature ratio.