Constrained Policy Optimization

Constrained policy optimization (CPO) in reinforcement learning focuses on training agents to maximize rewards while adhering to safety or other operational constraints, crucial for real-world deployment. Current research emphasizes developing algorithms that reliably satisfy constraints, even with stochastic systems and limited data, often employing techniques like Lagrangian methods, trust regions, or distributional representations within policy optimization frameworks. This area is vital for advancing safe and reliable AI in applications such as robotics, autonomous driving, and healthcare, where unintended consequences are unacceptable. The development of robust, efficient, and theoretically grounded CPO methods is a significant focus of ongoing research.

Papers

November 7, 2024

Hypercube Policy Regularization Framework for Offline Reinforcement Learning
Yi Shen, Hanyan Huang
LeArning Abstract Offline Reinforcement Learning Constrained Policy Optimization Policy Regularization

October 28, 2024

Adversarial Constrained Policy Optimization: Improving Constrained Reinforcement Learning by Adapting Budgets
Jianmina Ma, Jingtian Ji, Yue Gao
Reinforcement Learning Participation Constraint Constrained Reinforcement Learning Constraint Satisfaction Constrained Policy Optimization Adversarial Constraint

October 2, 2024

Absolute State-wise Constrained Policy Optimization: High-Probability State-wise Constraints Satisfaction
Weiye Zhao, Feihan Li, Yifan Sun, Yujie Wang, Rui Chen, Tianhao Wei, Changliu Liu
Constrained Policy Optimization State Constraint

September 26, 2024

CRoP: Context-wise Robust Static Human-Sensing Personalization
Sawinder Kaur, Avery Gump, Jingyu Xin, Yi Xiao, Harshit Sharma, Nina R Benway, Jonathan L Preston, Asif Salekin
Stronger Generalizability Human Sensing Constrained Policy Optimization

August 4, 2024

SelfBC: Self Behavior Cloning for Offline Reinforcement Learning
Shirong Liu, Chenjia Bai, Zixian Guo, Hao Zhang, Gaurav Sharma, Yang Liu
Offline Reinforcement Learning Behavior Cloning Constrained Policy Optimization Policy Constraint Self Consistent Constraint

June 9, 2024

Information Theoretic Guarantees For Policy Alignment In Large Language Models
Youssef Mroueh
Large Language Model Constrained Policy Optimization Optimal Alignment Policy Alignment R\'enyi Divergence

April 17, 2024

Stepwise Alignment for Constrained Language Model Policy Optimization
Akifumi Wachi, Thien Q. Tran, Rei Sato, Takumi Tanabe, Youhei Akimoto
Language Model Direct Preference Optimization Constrained Policy Optimization Bidirectional Alignment Reward Maximizing Policy

February 24, 2024

Concurrent Learning of Policy and Unknown Safety Constraints in Reinforcement Learning
Lunet Yifru, Ali Baheri
Reinforcement Learning Safe Reinforcement Learning Safety Constraint Health Policy Recommendation Constrained Policy Optimization

January 20, 2024

Constraint-Generation Policy Optimization (CGPO): Nonlinear Programming for Policy Optimization in Mixed Discrete-Continuous MDPs
Michael Gimelfarb, Ayal Taitler, Scott Sanner
Optimal Policy Policy OpTimization Nonlinear Programming Constrained Policy Optimization Continuous MDPs

December 15, 2023

ACPO: AI-Enabled Compiler-Driven Program Optimization
Amir H. Ashouri, Muhammad Asif Manzoor, Duc Minh Vu, Raymond Zhang, Ziwen Wang, Angel Zhang, Bryan Chan, Tomasz S. Czajkowski, Yaoqing Gao
Modern Compiler Performance Optimization Constrained Policy Optimization Compiler Optimization Level Virtual Machine

October 11, 2023

Robust Safe Reinforcement Learning under Adversarial Disturbances
Zeyang Li, Chuxiong Hu, Shengbo Eben Li, Jia Cheng, Yunan Wang
Reinforcement Learning Constrained Reinforcement Learning Constrained Policy Optimization Adversarial Disturbance

October 5, 2023

Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learning
Yihang Yao, Zuxin Liu, Zhepeng Cen, Jiacheng Zhu, Wenhao Yu, Tingnan Zhang, Ding Zhao
Reinforcement Learning Policy OpTimization Safe Reinforcement Learning Safety Constraint Constrained Policy Optimization

September 27, 2023

Evaluation of Constrained Reinforcement Learning Algorithms for Legged Locomotion
Joonho Lee, Lukas Schroth, Victor Klemm, Marko Bjelonic, Alexander Reske, Marco Hutter
Deep Reinforcement Learning Global Evaluation Constrained Reinforcement Learning Legged Locomotion Constrained Policy Optimization Standard Reinforcement Learning Constrained Reinforcement Learning Algorithm

June 24, 2023

Towards Optimal Pricing of Demand Response -- A Nonparametric Constrained Policy Optimization Approach
Jun Song, Chaoyue Zhao
Reinforcement Learning Policy OpTimization Policy Optimization Actor Critic Algorithm Demand Response Constrained Policy Optimization Optimal Pricing

February 28, 2023

Efficient Exploration Using Extra Safety Budget in Constrained Policy Optimization
Haotian Xu, Shengjie Wang, Zhaolei Wang, Yunzhe Zhang, Qing Zhuo, Yang Gao, Tao Zhang
Reinforcement Learning Learning Based Efficient Exploration Constrained Policy Optimization Safety Enhancement Exploration Efficiency Inefficient Exploration

February 4, 2023

Online Reinforcement Learning in Non-Stationary Context-Driven Environments
Pouya Hamadanian, Arash Nasr-Esfahany, Malte Schwarzkopf, Siddartha Sen, Mohammad Alizadeh
Policy OpTimization Context Aware Non Stationary Online Reinforcement Learning Constrained Policy Optimization

February 3, 2023

Distributional constrained reinforcement learning for supply chain optimization
Jaime Sabal Bermúdez, Antonio del Rio Chanona, Calvin Tsay
Policy Gradient Distributional Reinforcement Learning Reinforcement Learning Research Constrained Policy Optimization Supply Chain Optimization

February 2, 2023

ACPO: A Policy Optimization Algorithm for Average MDPs with Constraints
Akhil Agnihotri, Rahul Jain, Haipeng Luo
Reinforcement Learning Policy OpTimization Participation Constraint Policy Optimization Constrained Policy Optimization Trust Region Policy Optimization

January 28, 2023

Constrained Policy Optimization with Explicit Behavior Density for Offline Reinforcement Learning
Jing Zhang, Chi Zhang, Wenjia Wang, Bing-Yi Jing
Offline Reinforcement Learning Constrained Policy Optimization Advantage Learning

January 24, 2023

Constrained Reinforcement Learning for Dexterous Manipulation
Abhineet Jain, Jack Kolb, Harish Ravichandar
Dexterous Manipulation Constrained Reinforcement Learning Constrained Policy Optimization Knowledge Constraint Real World Dexterous Manipulation Constrained Problem