Constrained Markov Decision Process

Constrained Markov Decision Processes (CMDPs) extend standard reinforcement learning by incorporating constraints on actions or cumulative costs, enabling safe and efficient decision-making in real-world scenarios with limitations. Current research heavily focuses on developing efficient algorithms, particularly primal-dual methods and policy optimization approaches, that achieve optimal regret (difference between achieved and optimal reward) and constraint violation bounds under various feedback (full or bandit) and constraint types (stochastic or adversarial). These advancements are crucial for deploying reinforcement learning in safety-critical applications like autonomous driving and robotics, where adhering to constraints is paramount.

Papers

January 31, 2024

A Policy Gradient Primal-Dual Algorithm for Constrained MDPs with Uniform PAC Guarantees
Toshinori Kitamura, Tadashi Kozuno, Masahiro Kato, Yuki Ichihara, Soichiro Nishimori, Akiyoshi Sannai, Sho Sonoda, Wataru Kumagai, Yutaka Matsuo
Policy Gradient Sublinear Regret Constrained Markov Decision Process Primal Dual Algorithm Primal Dual Reinforcement Learning

December 1, 2023

A safe exploration approach to constrained Markov decision processes
Tingting Ni, Maryam Kamgarpour
Markov Decision Process Optimal Policy Constrained Markov Decision Process Safe Exploration Policy Parameterization Interior Point

November 9, 2023

Anytime-Constrained Reinforcement Learning
Jeremy McMahan, Xiaojin Zhu
Markov Decision Process Optimal Policy Constrained Reinforcement Learning Constrained Markov Decision Process Deterministic Policy Markovian Policy

November 2, 2023

Anytime-Competitive Reinforcement Learning with Policy Prior
Jianyi Yang, Pengfei Li, Tongxin Li, Adam Wierman, Shaolei Ren
Constrained Markov Decision Process Prior Policy Cost Constraint

November 1, 2023

SCPO: Safe Reinforcement Learning with Safety Critic Policy Optimization
Jaafar Mhamed, Shangding Gu
Reinforcement Learning Safe Reinforcement Learning Safety Constraint Constrained Markov Decision Process Safe Policy Safe Reinforcement Learning Algorithm

October 25, 2023

Finite-Time Analysis of Three-Timescale Constrained Actor-Critic and Constrained Natural Actor-Critic Algorithms
Prashansa Panda, Shalabh Bhatnagar
Actor Critic Constrained Markov Decision Process Finite Time Reinforcement Learning Task

September 27, 2023

September 23, 2023

Limits of Actor-Critic Algorithms for Decision Tree Policies Learning in IBMDPs
Hector Kohler, Riad Akrour, Philippe Preux
Decision Tree Continuum Limit Actor Critic Algorithm Observable Markov Decision Process Constrained Markov Decision Process Decision Problem Decision Tree Policy

June 20, 2023

Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPs
Dongsheng Ding, Chen-Yu Wei, Kaiqing Zhang, Alejandro Ribeiro
Primal Dual Constrained Markov Decision Process

June 12, 2023

Cancellation-Free Regret Bounds for Lagrangian Approaches in Constrained Markov Decision Processes
Adrian Müller, Pragnya Alatur, Giorgia Ramponi, Niao He
Regret Bound Primal Dual Constrained Markov Decision Process Tabular Markov Decision Process Lagrangian Method

June 1, 2023

Identifiability and Generalizability in Constrained Inverse Reinforcement Learning
Andreas Schlaginhaufen, Maryam Kamgarpour
Reinforcement Learning Reward Function Inverse Reinforcement Learning Stronger Generalizability Constrained Markov Decision Process Novel Identifiability Test Inverse Reward Design Inverse Constrained Reinforcement Learning

May 23, 2023

Constrained Reinforcement Learning for Dynamic Material Handling
Chengpeng Hu, Ziming Wang, Jialin Liu, Junyi Wen, Bifei Mao, Xin Yao
Constrained Reinforcement Learning Constrained Markov Decision Process Constrained Reinforcement Learning Algorithm

May 14, 2023

Inverse Reinforcement Learning With Constraint Recovery
Nirjhar Das, Arpan Chattopadhyay
Inverse Reinforcement Learning Constrained Markov Decision Process Non Convex Optimization Problem Backward Learning Constraint Acquisition

April 29, 2023

Semi-Infinitely Constrained Markov Decision Processes and Efficient Reinforcement Learning
Liangyu Zhang, Yang Peng, Wenhao Yang, Zhihua Zhang
Model Based Reinforcement Learning Efficient Reinforcement Learning Constrained Markov Decision Process

April 27, 2023

A Best-of-Both-Worlds Algorithm for Constrained MDPs with Long-Term Constraints
Jacopo Germano, Francesco Emanuele Stradi, Gianmarco Genalti, Matteo Castiglioni, Alberto Marchesi, Nicola Gatti
Online Learning Constrained Markov Decision Process Episodic Markov Decision Process Best of Both World Algorithm Long Term Constraint

March 2, 2023

Model-based Constrained MDP for Budget Allocation in Sequential Incentive Marketing
Shuai Xiao, Le Guo, Zaifan Jiang, Lei Lv, Yuanbo Chen, Jun Zhu, Shuang Yang
Constrained Markov Decision Process Joint Optimization Budget Allocation Dual Optimization

February 25, 2023

On Bellman's principle of optimality and Reinforcement learning for safety-constrained Markov decision process
Rahul Misra, Rafał Wisniewski, Carsten Skovmose Kallesøe
Reinforcement Learning Safe Reinforcement Learning General Principle Near Optimality Constrained Markov Decision Process Reinforcement Learning Problem Bellman Operator

February 6, 2023

State-wise Safe Reinforcement Learning: A Survey
Weiye Zhao, Tairan He, Rui Chen, Tianhao Wei, Changliu Liu
Reinforcement Learning Timely Survey Constrained Markov Decision Process Constraint Satisfaction Hard Constraint State Constraint

February 2, 2023

ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPs
Ted Moskovitz, Brendan O'Donoghue, Vivek Veeriah, Sebastian Flennerhag, Satinder Singh, Tom Zahavy
Reinforcement Learning Markov Decision Process Constrained Reinforcement Learning Constrained Markov Decision Process Last Iterate Convergence Gradient Descent Ascent Optimistic Gradient

Constrained Markov Decision Process

Papers

A Policy Gradient Primal-Dual Algorithm for Constrained MDPs with Uniform PAC Guarantees

A safe exploration approach to constrained Markov decision processes

Anytime-Constrained Reinforcement Learning

Anytime-Competitive Reinforcement Learning with Policy Prior

SCPO: Safe Reinforcement Learning with Safety Critic Policy Optimization

Finite-Time Analysis of Three-Timescale Constrained Actor-Critic and Constrained Natural Actor-Critic Algorithms

Provably Efficient Exploration in Constrained Reinforcement Learning:Posterior Sampling Is All You Need

Model-Free, Regret-Optimal Best Policy Identification in Online CMDPs

Limits of Actor-Critic Algorithms for Decision Tree Policies Learning in IBMDPs

Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPs

Cancellation-Free Regret Bounds for Lagrangian Approaches in Constrained Markov Decision Processes

Identifiability and Generalizability in Constrained Inverse Reinforcement Learning

Constrained Reinforcement Learning for Dynamic Material Handling

Inverse Reinforcement Learning With Constraint Recovery

Semi-Infinitely Constrained Markov Decision Processes and Efficient Reinforcement Learning

A Best-of-Both-Worlds Algorithm for Constrained MDPs with Long-Term Constraints

Model-based Constrained MDP for Budget Allocation in Sequential Incentive Marketing

On Bellman's principle of optimality and Reinforcement learning for safety-constrained Markov decision process

State-wise Safe Reinforcement Learning: A Survey

ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPs