Actor Critic Algorithm

Actor-critic algorithms are a class of reinforcement learning methods that learn optimal policies by iteratively improving a policy (the actor) and estimating its value (the critic). Current research focuses on improving the stability and efficiency of these algorithms, particularly through advancements in model architectures like neural networks (including those using ReLU networks and moment neural networks), and addressing challenges such as bias in gradient estimation, sample efficiency, and robustness to uncertainty in the environment. These improvements are driving progress in various applications, including robotics, resource management, and even LLM alignment, where actor-critic methods are increasingly used for policy optimization.

Papers

May 5, 2024

Finite-Time Convergence and Sample Complexity of Actor-Critic Multi-Objective Reinforcement Learning
Tianchen Zhou, FNU Hairi, Haibo Yang, Jia Liu, Tian Tong, Fan Yang, Michinari Momma, Yan Gao
Reinforcement Learning Policy Gradient Sample Complexity Multi Objective Reinforcement Learning Actor Critic Algorithm Finite Time Convergence

May 3, 2024

Natural Policy Gradient and Actor Critic Methods for Constrained Multi-Task Reinforcement Learning
Sihan Zeng, Thinh T. Doan, Justin Romberg
Multi Task Optimal Policy Actor Critic Actor Critic Algorithm Multi Task Reinforcement Learning Natural Policy Gradient Primal Dual Algorithm

April 16, 2024

Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, Yi Wu
Reinforcement Learning Proximal Policy Optimization Direct Preference Optimization Comprehensive Study Policy Optimization Actor Critic Algorithm LLM Alignment Reward Free

March 25, 2024

Weak Convergence Analysis of Online Neural Actor-Critic Algorithms
Samuel Chun-Hei Lam, Justin Sirignano, Ziheng Wang
Neural Network Policy Gradient Actor Critic Algorithm Critic Model Weak Convergence Online Deep Learning

February 3, 2024

SQT -- std $Q$-target
Nitsan Soffair, Dotan Di-Castro, Orly Avner, Shie Mannor
Q Learning Actor Critic Algorithm Q Function

February 2, 2024

Two-Timescale Critic-Actor for Average Reward MDPs with Function Approximation
Prashansa Panda, Shalabh Bhatnagar
Actor Critic Algorithm Function Approximation Average Reward MDPs Timescale Actor Critic

November 30, 2023

Anomaly Detection via Learning-Based Sequential Controlled Sensing
Geethu Joseph, Chen Zhong, M. Cenk Gursoy, Senem Velipasalar, Pramod K. Varshney
Anomaly Detection Markov Decision Process Policy Gradient Actor Critic Algorithm Sequential Selection Sequential Hypothesis Testing

November 25, 2023

Projected Off-Policy Q-Learning (POP-QL) for Stabilizing Offline Reinforcement Learning
Melrose Roderick, Gaurav Manek, Felix Berkenkamp, J. Zico Kolter
Offline Reinforcement Learning Policy Reinforcement Learning Actor Critic Algorithm

November 8, 2023

Real-Time Recurrent Reinforcement Learning
Julian Lemmel, Radu Grosu
Actor Critic Algorithm Real Time Recurrent Learning

November 5, 2023

On the Second-Order Convergence of Biased Policy Gradient Algorithms
Siqiao Mu, Diego Klabjan
Policy Gradient Actor Critic Algorithm Vanilla Reinforcement Learning

October 18, 2023

Learning to Optimise Climate Sensor Placement using a Transformer
Chen Wang, Victoria Huang, Gang Chen, Hui Ma, Bryce Chen, Jochen Schmidt
LeArning Abstract Deep Reinforcement Learning Transformer Based Actor Critic Algorithm Sensor Placement

September 29, 2023

Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning
Zihan Ding, Chi Jin
Reinforcement Learning Generative Model Score Based Generative Actor Critic Algorithm Consistency Model Generating Rich Expressive Policy

September 23, 2023

Limits of Actor-Critic Algorithms for Decision Tree Policies Learning in IBMDPs
Hector Kohler, Riad Akrour, Philippe Preux
Decision Tree Continuum Limit Actor Critic Algorithm Observable Markov Decision Process Constrained Markov Decision Process Decision Problem Decision Tree Policy

September 8, 2023

Actor critic learning algorithms for mean-field control with moment neural networks
Huyên Pham, Xavier Warin
Reinforcement Learning Neural Network Practical Algorithm Mean Field Actor Critic Algorithm Temporal Moment

August 19, 2023

PACE: Improving Prompt with Actor-Critic Editing for Large Language Model
Yihong Dong, Kangcheng Luo, Xue Jiang, Zhi Jin, Ge Li
Large Language Model Complex Prompt Style PROMPT Actor Critic Algorithm Diverse Prompt Prompt Based Editing

August 18, 2023

DoCRL: Double Critic Deep Reinforcement Learning for Mapless Navigation of a Hybrid Aerial Underwater Vehicle with Medium Transition
Ricardo B. Grando, Junior C. de Jesus, Victor A. Kich, Alisson H. Kolling, Rodrigo S. Guerra, Paulo L. J. Drews-Jr
Deep Reinforcement Learning Actor Critic Algorithm Deep RL Mapless Navigation Late Time Transition Aerial Underwater Vehicle

August 4, 2023

Synthesizing Programmatic Policies with Actor-Critic Algorithms and ReLU Networks
Spyros Orfanos, Levi H. S. Lelis
Reinforcement Learning ReLU Network Actor Critic Algorithm Online Policy Customization

July 6, 2023

A Neuromorphic Architecture for Reinforcement Learning from Real-Valued Observations
Sergio F. Chevtchenko, Yeshwanth Bethi, Teresa B. Ludermir, Saeed Afshar
Reinforcement Learning Spiking Neural Network Proximal Policy Optimization Actor Critic Algorithm Efficient Reinforcement Learning Real Data Neuromorphic Architecture

June 28, 2023

SARC: Soft Actor Retrospective Critic
Sukriti Verma, Ayush Chopra, Jayakumar Subramanian, Mausoom Sarkar, Nikaash Puri, Piyush Gupta, Balaji Krishnamurthy
Policy Gradient Actor Critic Algorithm Gradient Estimation Soft Actor Critic Critic Model

June 24, 2023

Towards Optimal Pricing of Demand Response -- A Nonparametric Constrained Policy Optimization Approach
Jun Song, Chaoyue Zhao
Reinforcement Learning Policy OpTimization Policy Optimization Actor Critic Algorithm Demand Response Constrained Policy Optimization Optimal Pricing