Constrained Markov Decision Process

Constrained Markov Decision Processes (CMDPs) extend standard reinforcement learning by incorporating constraints on actions or cumulative costs, enabling safe and efficient decision-making in real-world scenarios with limitations. Current research heavily focuses on developing efficient algorithms, particularly primal-dual methods and policy optimization approaches, that achieve optimal regret (difference between achieved and optimal reward) and constraint violation bounds under various feedback (full or bandit) and constraint types (stochastic or adversarial). These advancements are crucial for deploying reinforcement learning in safety-critical applications like autonomous driving and robotics, where adhering to constraints is paramount.

Papers

November 5, 2024

Embedding Safety into RL: A New Take on Trust Region Methods
Nikola Milosevic, Johannes Müller, Nico Scherf
Reinforcement Learning Policy OpTimization Human SAFETY New Perspective Constrained Markov Decision Process Natural Policy Gradient Trust Region Trust Region Policy Optimization

October 3, 2024

August 21, 2024

Last-Iterate Convergence of General Parameterized Policies in Constrained MDPs
Washim Uddin Mondal, Vaneet Aggarwal
Primal Dual Constrained Markov Decision Process Policy Parameterization Last Iterate Convergence Policy Parametrization

July 8, 2024

Enhanced Safety in Autonomous Driving: Integrating Latent State Diffusion Model for End-to-End Navigation
Detian Chu, Linyuan Bai, Jianuo Huang, Zhenlong Fang, Peng Zhang, Wei Kang, Haifeng Lin
Autonomous Driving Motion Planning Policy OpTimization Constrained Markov Decision Process Safe Exploration Safety Enhancement

June 26, 2024

Confident Natural Policy Gradient for Local Planning in $q_π$-realizable Constrained MDPs
Tian Tian, Lin F. Yang, Csaba Szepesvári
Policy Gradient Linear Function Approximation Reinforcement Learning Approach Constrained Markov Decision Process Local Planning

June 23, 2024

Monte Carlo Planning for Stochastic Control on Constrained Markov Decision Processes
Larkin Liu, Shiqi Liu, Matej Jusup
Markov Decision Process Constrained Markov Decision Process Stochastic Control Monte Carlo Planning Stochastic Decision

June 17, 2024

Constrained Reinforcement Learning with Average Reward Objective: Model-Based and Model-Free Algorithms
Vaneet Aggarwal, Washim Uddin Mondal, Qinbo Bai
Reinforcement Learning Primal Dual Constrained Reinforcement Learning Constrained Markov Decision Process Average Reward Markov Decision Process Model Free Algorithm Average Reward

May 29, 2024

Efficient Exploration in Average-Reward Constrained Reinforcement Learning: Achieving Near-Optimal Regret With Posterior Sampling
Danil Provodin, Maurits Kaptein, Mykola Pechenizkiy
Optimal Regret Efficient Exploration Posterior Sampling Constrained Reinforcement Learning Constrained Markov Decision Process Bayesian Regret Average Reward Reinforcement Learning

May 23, 2024

Learning Constrained Markov Decision Processes With Non-stationary Rewards and Constraints
Francesco Emanuele Stradi, Anna Lunghi, Matteo Castiglioni, Alberto Marchesi, Nicola Gatti
Participation Constraint Bandit Feedback Constrained Markov Decision Process Stochastic Reward Adversarial Markov Decision Process Constraint Violation Adversarial Reward

May 17, 2024

Sample-Efficient Constrained Reinforcement Learning with General Parameterization
Washim Uddin Mondal, Vaneet Aggarwal
Primal Dual Constrained Markov Decision Process Global Optimality Momentum Acceleration

May 9, 2024

Approximate Dec-POMDP Solving Using Multi-Agent A*
Wietze Koops, Sebastian Junges, Nils Jansen
Practical Algorithm Near Optimality Specific Heuristic Constrained Markov Decision Process

April 22, 2024

Generalizing Multi-Step Inverse Models for Representation Learning to Finite-Memory POMDPs
Lili Wu, Ben Evans, Riashat Islam, Raihan Seraj, Yonathan Efroni, Alex Lamb
Representation Learning High Dimensional Constrained Markov Decision Process Multi Step Agent Centric

April 10, 2024

Structured Reinforcement Learning for Media Streaming at the Wireless Edge
Archana Bura, Sarat Chandra Bobbili, Shreyas Rameshkumar, Desik Rengarajan, Dileep Kalathil, Srinivas Shakkottai
Constrained Reinforcement Learning Constrained Markov Decision Process Based Policy Natural Policy Gradient Online Streaming Wireless Edge

March 23, 2024

Safe Reinforcement Learning for Constrained Markov Decision Processes with Stochastic Stopping Time
Abhijit Mazumdar, Rafal Wisniewski, Manuela L. Bujorianu
Optimal Policy Safe Reinforcement Learning Efficient Exploration Safety Constraint Online Reinforcement Learning Constrained Markov Decision Process Stopping Time

March 20, 2024

POLICEd RL: Learning Closed-Loop Robot Control Policies with Provable Satisfaction of Hard Constraints
Jean-Baptiste Bouvier, Kartik Nagpal, Negar Mehr
Reinforcement Learning Algorithm Constrained Markov Decision Process Human Like RL Hard Constraint State Constraint

March 6, 2024

Learning Adversarial MDPs with Stochastic Hard Constraints
Francesco Emanuele Stradi, Matteo Castiglioni, Alberto Marchesi, Nicola Gatti
LeArning Abstract Sublinear Regret Constrained Markov Decision Process Adversarial Markov Decision Process Loss Design Online Learning Problem Stochastic Constraint

February 26, 2024

Achieving $\tilde{O}(1/\epsilon)$ Sample Complexity for Constrained Markov Decision Process
Jiashuo Jiang, Yinyu Ye
Sample Complexity Sequential Learning Constrained Markov Decision Process Reinforcement Learning Problem Logarithmic Regret Optimal Guarantee

February 24, 2024

Truly No-Regret Learning in Constrained MDPs
Adrian Müller, Pragnya Alatur, Volkan Cevher, Giorgia Ramponi, Niao He
Regret Bound Primal Dual Sublinear Regret Constrained Markov Decision Process Primal Dual Algorithm

February 3, 2024

Learning General Parameterized Policies for Infinite Horizon Average Reward Constrained MDPs via Primal-Dual Policy Gradient Algorithm
Qinbo Bai, Washim Uddin Mondal, Vaneet Aggarwal
Optimal Policy Primal Dual Constrained Markov Decision Process Regret Guarantee Simple Regret Policy Gradient Algorithm Average Reward Markov Decision Process Policy Parametrization

Constrained Markov Decision Process

Papers

Embedding Safety into RL: A New Take on Trust Region Methods

Optimal Strong Regret and Violation in Constrained MDPs via Policy Optimization

Best-of-Both-Worlds Policy Optimization for CMDPs with Bandit Feedback

Last-Iterate Convergence of General Parameterized Policies in Constrained MDPs

Enhanced Safety in Autonomous Driving: Integrating Latent State Diffusion Model for End-to-End Navigation

Confident Natural Policy Gradient for Local Planning in $q_π$-realizable Constrained MDPs

Monte Carlo Planning for Stochastic Control on Constrained Markov Decision Processes

Constrained Reinforcement Learning with Average Reward Objective: Model-Based and Model-Free Algorithms

Efficient Exploration in Average-Reward Constrained Reinforcement Learning: Achieving Near-Optimal Regret With Posterior Sampling

Learning Constrained Markov Decision Processes With Non-stationary Rewards and Constraints

Sample-Efficient Constrained Reinforcement Learning with General Parameterization

Approximate Dec-POMDP Solving Using Multi-Agent A*

Generalizing Multi-Step Inverse Models for Representation Learning to Finite-Memory POMDPs

Structured Reinforcement Learning for Media Streaming at the Wireless Edge

Safe Reinforcement Learning for Constrained Markov Decision Processes with Stochastic Stopping Time

POLICEd RL: Learning Closed-Loop Robot Control Policies with Provable Satisfaction of Hard Constraints

Learning Adversarial MDPs with Stochastic Hard Constraints

Achieving $\tilde{O}(1/\epsilon)$ Sample Complexity for Constrained Markov Decision Process

Truly No-Regret Learning in Constrained MDPs

Learning General Parameterized Policies for Infinite Horizon Average Reward Constrained MDPs via Primal-Dual Policy Gradient Algorithm