Constrained Reinforcement Learning

Constrained Reinforcement Learning (CRL) addresses the challenge of training agents to maximize rewards while simultaneously satisfying safety or resource constraints, crucial for deploying RL in real-world applications. Current research focuses on developing efficient algorithms, such as primal-dual methods, penalty function methods, and those incorporating techniques like log barrier functions or posterior sampling, often within model-based or model-free frameworks. These advancements improve the safety and reliability of RL agents across diverse domains, including robotics, resource allocation, and safe navigation, by ensuring learned policies adhere to critical operational limitations. The resulting improvements in robustness and safety are significant for transitioning RL from simulated to real-world deployments.

Papers

May 23, 2023

Constrained Reinforcement Learning for Dynamic Material Handling
Chengpeng Hu, Ziming Wang, Jialin Liu, Junyi Wen, Bifei Mao, Xin Yao
Constrained Reinforcement Learning Constrained Markov Decision Process Constrained Reinforcement Learning Algorithm

April 30, 2023

Joint Learning of Policy with Unknown Temporal Constraints for Safe Reinforcement Learning
Lunet Yifru, Ali Baheri
Reinforcement Learning Temporal Logic Safe Reinforcement Learning Joint Learning Safety Constraint Constrained Reinforcement Learning Health Policy Recommendation Unknown Constraint Optimal Reinforcement Learning

April 18, 2023

Benchmarking Actor-Critic Deep Reinforcement Learning Algorithms for Robotics Control with Action Constraints
Kazumi Kasaura, Shuwa Miura, Tadashi Kozuno, Ryo Yonetani, Kenta Hoshino, Yohei Hosoe
New Benchmark Actor Critic Robot Control Constrained Reinforcement Learning Robot Environment Task Constraint

March 21, 2023

Style Miner: Find Significant and Stable Explanatory Factors in Time Series with Constrained Reinforcement Learning
Dapeng Li, Feiyang Pan, Jia He, Zhiwei Xu, Dandan Tu, Guoliang Fan
Time Series Style Representation Constrained Reinforcement Learning High Dimensional Time Series Learning Based Method Stable Explanation Real World Financial Data

March 6, 2023

Constrained Reinforcement Learning and Formal Verification for Safe Colonoscopy Navigation
Davide Corsi, Luca Marzari, Ameya Pore, Alessandro Farinelli, Alicia Casals, Paolo Fiorini, Diego Dall'Alba
Deep Reinforcement Learning Autonomous Navigation Constrained Reinforcement Learning Formal Verification Flexible Endoscope

February 2, 2023

ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPs
Ted Moskovitz, Brendan O'Donoghue, Vivek Veeriah, Sebastian Flennerhag, Satinder Singh, Tom Zahavy
Reinforcement Learning Markov Decision Process Constrained Reinforcement Learning Constrained Markov Decision Process Last Iterate Convergence Gradient Descent Ascent Optimistic Gradient

January 27, 2023

Solving Richly Constrained Reinforcement Learning through State Augmentation and Reward Penalties
Hao Jiang, Tien Mai, Pradeep Varakantham, Minh Huy Hoang
Constrained Reinforcement Learning State Augmentation Cost Constraint Appropriate Penalty Term

January 24, 2023

December 9, 2022

Networked Restless Bandits with Positive Externalities
Christine Herlihy, John P. Dickerson
Multi Armed Bandit Constrained Reinforcement Learning Restless Bandit Positive Externality

December 3, 2022

Constrained Reinforcement Learning via Dissipative Saddle Flow Dynamics
Tianqi Zheng, Pengcheng You, Enrique Mallada
Primal Dual Constrained Reinforcement Learning Gradient Descent Ascent Stochastic Gradient Descent Ascent Saddle to Saddle Dynamic

November 28, 2022

Quantile Constrained Reinforcement Learning: A Reinforcement Learning Framework Constraining Outage Probability
Whiyoung Jung, Myungsik Cho, Jongeui Park, Youngchul Sung
Distributional Reinforcement Learning Constrained Reinforcement Learning Outage Probability

October 25, 2022

UNIFY: a Unified Policy Designing Framework for Solving Constrained Optimization Problems with Machine Learning
Mattia Silvestri, Allegra De Filippo, Michele Lombardi, Michela Milano
Machine Learning Constrained Optimization Constrained Reinforcement Learning Anti Unification Decision Problem Decision Focused Learning Dexterous Policy

October 15, 2022

When to Update Your Model: Constrained Model-based Reinforcement Learning
Tianying Ji, Yu Luo, Fuchun Sun, Mingxuan Jing, Fengxiang He, Wenbing Huang
Full Model Policy OpTimization Constrained Reinforcement Learning Model Learning

October 14, 2022

Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm
Ashish Kumar Jayant, Shalabh Bhatnagar
Reinforcement Learning Deep Reinforcement Learning Proximal Policy Optimization Safe Reinforcement Learning Constrained Reinforcement Learning Model Based Safe Reinforcement Learning

October 10, 2022

A policy gradient approach for Finite Horizon Constrained Markov Decision Processes
Soumyajit Guin, Shalabh Bhatnagar
Reinforcement Learning Policy Gradient Constrained Reinforcement Learning Finite Horizon Infinite Horizon

October 2, 2022

Policy Gradients for Probabilistic Constrained Reinforcement Learning
Weiqin Chen, Dharmashankar Subramanian, Santiago Paternain
Reinforcement Learning Policy Gradient Policy Optimization Constrained Reinforcement Learning Probabilistic Safety Probabilistic Constraint Policy Based Algorithm

September 21, 2022

LCRL: Certified Policy Synthesis via Logically-Constrained Reinforcement Learning
Hosein Hasanbeig, Daniel Kroening, Alessandro Abate
Markov Decision Process Reinforcement Learning Algorithm Model Free Reinforcement Learning Constrained Reinforcement Learning Policy Synthesis B\"uchi Automaton

September 14, 2022

Robust Constrained Reinforcement Learning
Yue Wang, Fei Miao, Shaofeng Zou
Primal Dual Model Uncertainty Robust Policy Constrained Reinforcement Learning Dynamic Feasibility

September 8, 2022

An Empirical Evaluation of Posterior Sampling for Constrained Reinforcement Learning
Danil Provodin, Pratik Gajane, Mykola Pechenizkiy, Maurits Kaptein
Efficient Exploration Posterior Sampling Constrained Reinforcement Learning Empirical Evaluation Optimistic Algorithm

Constrained Reinforcement Learning

Papers

Constrained Reinforcement Learning for Dynamic Material Handling

Joint Learning of Policy with Unknown Temporal Constraints for Safe Reinforcement Learning

Benchmarking Actor-Critic Deep Reinforcement Learning Algorithms for Robotics Control with Action Constraints

Style Miner: Find Significant and Stable Explanatory Factors in Time Series with Constrained Reinforcement Learning

Constrained Reinforcement Learning and Formal Verification for Safe Colonoscopy Navigation

ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPs

Solving Richly Constrained Reinforcement Learning through State Augmentation and Reward Penalties

AutoCost: Evolving Intrinsic Cost for Zero-violation Reinforcement Learning

Constrained Reinforcement Learning for Dexterous Manipulation

Networked Restless Bandits with Positive Externalities

Constrained Reinforcement Learning via Dissipative Saddle Flow Dynamics

Quantile Constrained Reinforcement Learning: A Reinforcement Learning Framework Constraining Outage Probability

UNIFY: a Unified Policy Designing Framework for Solving Constrained Optimization Problems with Machine Learning

When to Update Your Model: Constrained Model-based Reinforcement Learning

Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm

A policy gradient approach for Finite Horizon Constrained Markov Decision Processes

Policy Gradients for Probabilistic Constrained Reinforcement Learning

LCRL: Certified Policy Synthesis via Logically-Constrained Reinforcement Learning

Robust Constrained Reinforcement Learning

An Empirical Evaluation of Posterior Sampling for Constrained Reinforcement Learning