Constrained Reinforcement Learning

Constrained Reinforcement Learning (CRL) addresses the challenge of training agents to maximize rewards while simultaneously satisfying safety or resource constraints, crucial for deploying RL in real-world applications. Current research focuses on developing efficient algorithms, such as primal-dual methods, penalty function methods, and those incorporating techniques like log barrier functions or posterior sampling, often within model-based or model-free frameworks. These advancements improve the safety and reliability of RL agents across diverse domains, including robotics, resource allocation, and safe navigation, by ensuring learned policies adhere to critical operational limitations. The resulting improvements in robustness and safety are significant for transitioning RL from simulated to real-world deployments.

Papers

October 28, 2024

Adversarial Constrained Policy Optimization: Improving Constrained Reinforcement Learning by Adapting Budgets
Jianmina Ma, Jingtian Ji, Yue Gao
Reinforcement Learning Participation Constraint Constrained Reinforcement Learning Constraint Satisfaction Constrained Policy Optimization Adversarial Constraint

October 10, 2024

September 29, 2024

Constrained Reinforcement Learning for Safe Heat Pump Control
Baohe Zhang, Lilli Frison, Thomas Brox, Joschka Bödecker
Constrained Optimization Constrained Reinforcement Learning Soft Actor Critic Thermal Management Building Energy

September 27, 2024

Autoregressive Policy Optimization for Constrained Allocation Tasks
David Winkel, Niklas Strauß, Maximilian Bernhard, Zongyue Li, Thomas Seidl, Matthias Schubert
Policy OpTimization Constrained Reinforcement Learning Synthetic Benchmark

September 20, 2024

SoloParkour: Constrained Reinforcement Learning for Visual Locomotion from Privileged Experience
Elliot Chane-Sane, Joseph Amigo, Thomas Flayols, Ludovic Righetti, Nicolas Mansard
Legged Robot Constrained Reinforcement Learning Parkour Skill Privileged Feature Visual Locomotion

July 24, 2024

SoNIC: Safe Social Navigation with Adaptive Conformal Inference and Constrained Reinforcement Learning
Jianpeng Yao, Xiaopan Zhang, Yu Xia, Zejin Wang, Amit K. Roy-Chowdhury, Jiachen Li
Reinforcement Learning Social Robot Constrained Reinforcement Learning Social Navigation Conformal Regression

July 22, 2024

Exterior Penalty Policy Optimization with Penalty Metric Network under Constraints
Shiqing Gao, Jiaxin Ding, Luoyi Fu, Xinbing Wang, Chenghu Zhou
Participation Constraint Constrained Reinforcement Learning Penalty Method

July 15, 2024

Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning
Alessandro Montenegro, Marco Mussi, Matteo Papini, Alberto Maria Metelli
Policy Gradient Primal Dual Constrained Reinforcement Learning Last Iterate Convergence Stochastic Policy Agnostic Exploration

June 17, 2024

Constrained Reinforcement Learning with Average Reward Objective: Model-Based and Model-Free Algorithms
Vaneet Aggarwal, Washim Uddin Mondal, Qinbo Bai
Reinforcement Learning Primal Dual Constrained Reinforcement Learning Constrained Markov Decision Process Average Reward Markov Decision Process Model Free Algorithm Average Reward

June 13, 2024

e-COP : Episodic Constrained Optimization of Policies
Akhil Agnihotri, Rahul Jain, Deepak Ramachandran, Sahil Singla
Reinforcement Learning Constrained Reinforcement Learning Prior Policy Policy Network

June 3, 2024

Multi-agent assignment via state augmented reinforcement learning
Leopoldo Agorio, Sean Van Alen, Miguel Calvo-Fullana, Santiago Paternain, Juan Andres Bazerque
Multi Agent Multiple Agent Constrained Reinforcement Learning State Augmentation

May 29, 2024

Efficient Exploration in Average-Reward Constrained Reinforcement Learning: Achieving Near-Optimal Regret With Posterior Sampling
Danil Provodin, Maurits Kaptein, Mykola Pechenizkiy
Optimal Regret Efficient Exploration Posterior Sampling Constrained Reinforcement Learning Constrained Markov Decision Process Bayesian Regret Average Reward Reinforcement Learning

May 23, 2024

Deterministic Policies for Constrained Reinforcement Learning in Polynomial Time
Jeremy McMahan
Constrained Reinforcement Learning Near Optimal Policy Deterministic Policy Approximate Dynamic Programming Polynomial Time Approximation

May 14, 2024

I-CTRL: Imitation to Control Humanoid Robots Through Constrained Reinforcement Learning
Yashuai Yan, Esteve Valls Mascaro, Tobias Egle, Dongheui Lee
Legged Robot Humanoid Robot Feature Imitation Constrained Reinforcement Learning Motion Imitation GTR CTRL

May 2, 2024

Constrained Reinforcement Learning Under Model Mismatch
Zhongchang Sun, Sihong He, Fei Miao, Shaofeng Zou
Model Uncertainty Robust Policy Constrained Reinforcement Learning Reward Maximization Model Mismatch

April 16, 2024

Simplex Decomposition for Portfolio Allocation Constraints in Reinforcement Learning
David Winkel, Niklas Strauß, Matthias Schubert, Thomas Seidl
Reinforcement Learning Action Space Constrained Reinforcement Learning Portfolio Optimization Probability Simplex

April 10, 2024

Structured Reinforcement Learning for Media Streaming at the Wireless Edge
Archana Bura, Sarat Chandra Bobbili, Shreyas Rameshkumar, Desik Rengarajan, Dileep Kalathil, Srinivas Shakkottai
Constrained Reinforcement Learning Constrained Markov Decision Process Based Policy Natural Policy Gradient Online Streaming Wireless Edge

April 2, 2024

Asymptotics of Language Model Alignment
Joy Qiping Yang, Salman Salamatian, Ziteng Sun, Ananda Theertha Suresh, Ahmad Beirami
Constrained Reinforcement Learning Language Model Alignment Asymptotic Behavior

March 25, 2024

Arm-Constrained Curriculum Learning for Loco-Manipulation of the Wheel-Legged Robot
Zifan Wang, Yufei Jia, Lu Shi, Haoyu Wang, Haizhou Zhao, Xueyang Li, Jinni Zhou, Jun Ma, Guyue Zhou
Legged Robot Curriculum Learning Constrained Reinforcement Learning Loco Manipulation

Constrained Reinforcement Learning

Papers

Adversarial Constrained Policy Optimization: Improving Constrained Reinforcement Learning by Adapting Budgets

Probabilistic Satisfaction of Temporal Logic Constraints in Reinforcement Learning via Adaptive Policy-Switching

Offline Inverse Constrained Reinforcement Learning for Safe-Critical Decision Making in Healthcare

Constrained Reinforcement Learning for Safe Heat Pump Control

Autoregressive Policy Optimization for Constrained Allocation Tasks

SoloParkour: Constrained Reinforcement Learning for Visual Locomotion from Privileged Experience

SoNIC: Safe Social Navigation with Adaptive Conformal Inference and Constrained Reinforcement Learning

Exterior Penalty Policy Optimization with Penalty Metric Network under Constraints

Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning

Constrained Reinforcement Learning with Average Reward Objective: Model-Based and Model-Free Algorithms

e-COP : Episodic Constrained Optimization of Policies

Multi-agent assignment via state augmented reinforcement learning

Efficient Exploration in Average-Reward Constrained Reinforcement Learning: Achieving Near-Optimal Regret With Posterior Sampling

Deterministic Policies for Constrained Reinforcement Learning in Polynomial Time

I-CTRL: Imitation to Control Humanoid Robots Through Constrained Reinforcement Learning

Constrained Reinforcement Learning Under Model Mismatch

Simplex Decomposition for Portfolio Allocation Constraints in Reinforcement Learning

Structured Reinforcement Learning for Media Streaming at the Wireless Edge

Asymptotics of Language Model Alignment

Arm-Constrained Curriculum Learning for Loco-Manipulation of the Wheel-Legged Robot