Primal Dual Reinforcement Learning

Primal-dual reinforcement learning (RL) tackles constrained Markov decision processes (CMDPs) by framing the optimization problem as a saddle-point problem, balancing reward maximization with constraint satisfaction. Recent research focuses on developing provably efficient algorithms, such as primal-dual actor-critic methods and variations employing policy gradient approaches, that achieve optimal or near-optimal policies under various assumptions (e.g., linear function approximation, offline settings). This approach offers improved theoretical guarantees, including sample complexity and convergence rates, and addresses challenges in both online and offline RL settings, particularly for constrained problems and non-stationary environments, leading to more robust and reliable RL agents.

Papers

January 31, 2024

A Policy Gradient Primal-Dual Algorithm for Constrained MDPs with Uniform PAC Guarantees
Toshinori Kitamura, Tadashi Kozuno, Masahiro Kato, Yuki Ichihara, Soichiro Nishimori, Akiyoshi Sannai, Sho Sonoda, Wataru Kumagai, Yutaka Matsuo
Policy Gradient Sublinear Regret Constrained Markov Decision Process Primal Dual Algorithm Primal Dual Reinforcement Learning

June 13, 2023

A Primal-Dual-Critic Algorithm for Offline Constrained Reinforcement Learning
Kihyuk Hong, Yuhang Li, Ambuj Tewari
Reinforcement Learning Regret Algorithm Primal Dual Algorithm Primal Dual Reinforcement Learning

May 22, 2023

Offline Primal-Dual Reinforcement Learning for Linear MDPs
Germano Gabbianelli, Gergely Neu, Nneka Okolo, Matteo Papini
Offline Reinforcement Learning Optimal Policy Primal Dual Near Optimal Policy Linear MDPs Average Reward MDPs Primal Dual Reinforcement Learning

February 28, 2022

Provably Efficient Convergence of Primal-Dual Actor-Critic with Nonlinear Function Approximation
Jing Dong, Li Shen, Yinggan Xu, Baoxiang Wang
Reinforcement Learning Primal Dual Actor Critic Algorithm Nonlinear Function Approximation Stochastic Gradient Descent Ascent Primal Dual Reinforcement Learning

January 28, 2022

Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraints
Yuhao Ding, Javad Lavaei
Reinforcement Learning Time Varying Participation Constraint Constrained Markov Decision Process Episodic Markov Decision Process Safe Reinforcement Learning Algorithm Time Varying Objective Primal Dual Reinforcement Learning

December 28, 2021

Efficient Performance Bounds for Primal-Dual Reinforcement Learning from Demonstrations
Angeliki Kamoutsi, Goran Banjac, John Lygeros
Inverse Reinforcement Learning Noisy Demonstration Policy Optimization Stochastic Convex Optimization Low Regret Performance Bound Primal Dual Reinforcement Learning

Primal Dual Reinforcement Learning

Papers

A Policy Gradient Primal-Dual Algorithm for Constrained MDPs with Uniform PAC Guarantees

A Primal-Dual-Critic Algorithm for Offline Constrained Reinforcement Learning

Offline Primal-Dual Reinforcement Learning for Linear MDPs

Provably Efficient Convergence of Primal-Dual Actor-Critic with Nonlinear Function Approximation

Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraints

Efficient Performance Bounds for Primal-Dual Reinforcement Learning from Demonstrations