Policy Constraint

Policy constraint in reinforcement learning focuses on ensuring learned agent policies adhere to predefined safety or behavioral limitations, preventing undesirable actions while optimizing for a primary objective. Current research emphasizes developing dynamic and adaptive constraint methods, often integrated into offline reinforcement learning algorithms like TD3-BC and CQL, or employing novel architectures such as decision transformers and conditional sequence models (e.g., SaFormer). These advancements aim to address limitations of static constraints, improve sample efficiency, and enable robust policy learning from diverse or imperfect datasets, with applications ranging from robotics to autonomous systems.

Papers

October 10, 2024

Probabilistic Satisfaction of Temporal Logic Constraints in Reinforcement Learning via Adaptive Policy-Switching
Xiaoshan Lin, Sadık Bera Yüksel, Yasin Yazıcıoğlu, Derya Aksaray
Reinforcement Learning Adaptive Importance Constrained Reinforcement Learning Reward Maximization Traditional Reinforcement Learning Policy Constraint

October 8, 2024

Stochastic Bandits for Egalitarian Assignment
Eugene Lim, Vincent Y. F. Tan, Harold Soh
Cumulative Regret Stochastic Bandit Diverse Policy Policy Constraint Stochastic Multi Armed Bandit Egalitarian Fairness

August 4, 2024

SelfBC: Self Behavior Cloning for Offline Reinforcement Learning
Shirong Liu, Chenjia Bai, Zixian Guo, Hao Zhang, Gaurav Sharma, Yang Liu
Offline Reinforcement Learning Behavior Cloning Constrained Policy Optimization Policy Constraint Self Consistent Constraint

May 23, 2024

Exclusively Penalized Q-learning for Offline Reinforcement Learning
Junghyuk Yeom, Yonghyeon Jo, Jungmo Kim, Sanghyeon Lee, Seungyul Han
Offline Reinforcement Learning Q Learning Estimation Bias Policy Constraint Regularized Reinforcement Learning

April 25, 2024

A Dual Perspective of Reinforcement Learning for Imposing Policy Constraints
Bram De Cooman, Johan Suykens
Reinforcement Learning Primal Dual Model Free Reinforcement Learning Multiple View Actor Critic Reinforcement Learning Policy Constraint

March 12, 2024

A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspective
Yunpeng Qing, Shunyu liu, Jingyuan Cong, Kaixuan Chen, Yihe Zhou, Mingli Song
Policy Constraint Conditional Variational Auto Encoder Disentanglement Constraint Aware Policy

November 18, 2023

Constraint-aware Policy for Compliant Manipulation
Daichi Saito, Kazuhiro Sasabuchi, Naoki Wake, Atsushi Kanehira, Jun Takamatsu, Hideki Koike, Katsushi Ikeuchi
Robot Manipulation Home Robot Policy Constraint

October 27, 2023

Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learning
Shenzhi Wang, Qisen Yang, Jiawei Gao, Matthieu Gaetan Lin, Hao Chen, Liwei Wu, Ning Jia, Shiji Song, Gao Huang
High Speed Train System Single Parent Family Online Fine Tuning Policy Constraint Offline to Online Reinforcement Learning State Adaptive

September 12, 2023

Human-Centered Autonomy for UAS Target Search
Hunter M. Ray, Zakariya Laouar, Zachary Sunberg, Nisar Ahmed
Mission Specification Policy Constraint UAV Based Search Human Supervision

September 4, 2023

Hundreds Guide Millions: Adaptive Offline Reinforcement Learning with Expert Guidance
Qisen Yang, Shenzhi Wang, Qihang Zhang, Gao Huang, Shiji Song
Offline Reinforcement Learning Offline Reinforcement Learning Algorithm Policy Constraint

June 26, 2023

A General Framework for Sequential Decision-Making under Adaptivity Constraints
Nuoya Xiong, Zhaoran Wang, Zhuoran Yang
Markov Decision Process Sequential Decision General Framework Policy Constraint Adaptivity Constraint

June 21, 2023

Inverse Constraint Learning and Generalization by Transferable Reward Decomposition
Jaehwi Jang, Minjae Song, Daehyung Park
Strong Generalization Task Transferability Constraint Learning Policy Constraint rEward Decomposition Adaptivity Constraint

June 8, 2023

Decoupled Prioritized Resampling for Offline RL
Yang Yue, Bingyi Kang, Xiao Ma, Qisen Yang, Gao Huang, Shiji Song, Shuicheng Yan
Offline Reinforcement Learning Experience Replay Offline Reinforcement Learning Algorithm Policy Constraint

March 26, 2023

Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learning
Alex Beeson, Giovanni Montana
Offline Reinforcement Learning Optimal Policy Robust Policy Online Fine Tuning Policy Constraint Robust Offline Reinforcement Learning

February 14, 2023

Constrained Decision Transformer for Offline Safe Reinforcement Learning
Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, Ding Zhao
Safe Reinforcement Learning Decision Transformer Policy Constraint Offline Safe Reinforcement Learning

January 28, 2023

SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learning
Qin Zhang, Linrui Zhang, Haoran Xu, Li Shen, Bowen Wang, Yongzhe Chang, Xueqian Wang, Bo Yuan, Dacheng Tao
Constructive Approach Sequence Model Policy Constraint Offline Safe Reinforcement Learning

December 7, 2022

Accelerating Self-Imitation Learning from Demonstrations via Policy Constraints and Q-Ensemble
Chao Li
Reinforcement Learning Deep Reinforcement Learning Noisy Demonstration Online Reinforcement Learning Expert Demonstration Policy Constraint

November 21, 2022

Improving TD3-BC: Relaxed Policy Constraint for Offline Learning and Stable Online Fine-Tuning
Alex Beeson, Giovanni Montana
Reinforcement Learning Behavior Cloning Optimal Behavior Policy Constraint Sub Optimal Policy

October 19, 2022

Robust Offline Reinforcement Learning with Gradient Penalty and Constraint Relaxation
Chengqian Gao, Ke Xu, Liu Liu, Deheng Ye, Peilin Zhao, Zhiqiang Xu
Offline Reinforcement Learning Policy Constraint Robust Offline Reinforcement Learning Constraint Relaxation Gradient Penalty

June 28, 2022

Linear programming-based solution methods for constrained partially observable Markov decision processes
Robert K. Helmeczi, Can Kavaklioglu, Mucahit Cevik
Markov Decision Process Integer Programming Linear Ordered Data Policy Constraint Programming Solver Continuous POMDP

Policy Constraint

Papers

Probabilistic Satisfaction of Temporal Logic Constraints in Reinforcement Learning via Adaptive Policy-Switching

Stochastic Bandits for Egalitarian Assignment

SelfBC: Self Behavior Cloning for Offline Reinforcement Learning

Exclusively Penalized Q-learning for Offline Reinforcement Learning

A Dual Perspective of Reinforcement Learning for Imposing Policy Constraints

A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspective

Constraint-aware Policy for Compliant Manipulation

Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learning

Human-Centered Autonomy for UAS Target Search

Hundreds Guide Millions: Adaptive Offline Reinforcement Learning with Expert Guidance

A General Framework for Sequential Decision-Making under Adaptivity Constraints

Inverse Constraint Learning and Generalization by Transferable Reward Decomposition

Decoupled Prioritized Resampling for Offline RL

Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learning

Constrained Decision Transformer for Offline Safe Reinforcement Learning

SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learning

Accelerating Self-Imitation Learning from Demonstrations via Policy Constraints and Q-Ensemble

Improving TD3-BC: Relaxed Policy Constraint for Offline Learning and Stable Online Fine-Tuning

Robust Offline Reinforcement Learning with Gradient Penalty and Constraint Relaxation

Linear programming-based solution methods for constrained partially observable Markov decision processes