Constrained RL

RL: Reinforcement Learning

Constrained reinforcement learning studies how agents can maximize rewards while obeying limits on costs such as collisions, energy use, latency, or harm. Many methods model the problem as a constrained Markov decision process and use Lagrangian or primal-dual updates; others enforce feasibility with action masks, projections, safety shields, control barrier functions, or reachability analysis. These approaches are used in settings from robot control and power systems to language-model alignment, with work spanning online and offline learning, multi-agent tasks, and formal guarantees for regret and constraint satisfaction. A central challenge is balancing useful performance with reliable safety: average-cost limits can miss rare severe failures, while strict safeguards may make learning conservative or difficult to scale.

Latest papers

226 papers38 papers in the last four weeks, up 46% on the four weeks before. 0.2% of all new papers.

Safe Reinforcement Learning with Preference-based Constraint Inference
Mar 24, 2026Chenglin Li, Grant Ruan, Hua GengConstrained RL Safe RL
Department of Automation, Tsinghua University, Beijing, China · Laboratory for Information & Decision Systems, Massachusetts Institute of Technology, Cambridge, MA, USA
Optimizing Resource-Constrained Non-Pharmaceutical Interventions for Multi-Cluster Outbreak Control Using Hierarchical Reinforcement Learning
Mar 19, 2026Xueqiao Peng, Andrew PerraultHealthcare Constrained RL
The Ohio State University
Enhancing Reinforcement Learning Fine-Tuning with an Online Refiner
Mar 18, 2026Hao Ma, Zhiqiang Pu, Yang Liu +1RL Fine-Tuning RL for Language Models
School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences
Shielded Reinforcement Learning Under Dynamic Temporal Logic Constraints
Mar 17, 2026Sadık Bera Yüksel, Ali Tevfik Buyukkocak, Derya AksarayConstrained RL Reinforcement Learning
Department of Electrical and Computer Engineering at Northeastern University.
Task-Specified Compliance Bounds for Humanoids via Lipschitz-Constrained Policies
Mar 17, 2026Zewen He, Yoshihiko NakamuraCompliant Robot Control Robotic Control
Department of Robotics, Mohamed bin Zayed University of Artificial Intelligence, Masdar City, Abu Dhabi, United Arab Emirates
HIPO: Instruction Hierarchy via Constrained Reinforcement Learning
Mar 17, 2026Keru Chen, Jun Luo, Sen Lin +4LLM Alignment RL for Language Models
School of ECEE Arizona State University · Department of ECE The Ohio State University · Computer Science Department University of Houston +2
Iterative Learning Control-Informed Reinforcement Learning for Batch Process Control
Mar 16, 2026Runze Lin, Ziqi Zhuo, Junghui Chen +2Iterative Learning Control RL Control
State Key Laboratory of Industrial Control Technology, Institute of Cyber-Systems and Control, Zhejiang University, Hangzhou 310027, China · Department of Chemical Engineering, Chung-Yuan Christian University, Taoyuan 32023, Taiwan, R.O.C.
SmoothVLA: Aligning Vision-Language-Action Models with Physical Constraints via Intrinsic Smoothness Optimization
Mar 14, 2026Jiashun Li, Xiaoyu Shi, Hong Xie +2VLMs for Robotics RL for VLMs
Chongqing University of Posts and Telecommunications · Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences · The First Affiliated Hospital, University of Science and Technology of China
Reinforcement Learning for Elliptical Cylinder Motion Control Tasks
Mar 13, 2026Pawel Marczewski, Paulina Superczynska, Jakub Bernat +1RL Control Feedback Control
Institute of Computing Science Poznan University of Technology Poznan, 61-138 · Institute of Automatic Control and Robotics Poznan University of Technology Poznan, 61-138
Safe RLHF Beyond Expectation: Stochastic Dominance for Universal Spectral Risk Control
Mar 11, 2026Yaswanth Chittepu, Ativ Joshi, Rajarshi Bhattacharjee +1Constrained RL LLM Safety Alignment
University of Massachusetts Amherst
Improving Search Agent with One Line of Code
Mar 10, 2026Jian Li, Dongsheng Chen, Zhenhua Xu +5RL for Tool Use Group Relative Policy Optimization
Nanjing University · Tencent YoutuLab
Global Convergence of Average Reward Constrained MDPs with Neural Critic and General Policy Parameterization
Mar 8, 2026Anirudh Satheesh, Pankaj Kumar Barman, Washim Uddin Mondal +1Natural Policy Gradient Constrained RL
Department of Computer Science, University of Maryland, College Park, MD 20740, USA · Indian Institute of Technology, Kanpur, Uttar Pradesh, 208016, India · Department of Electrical Engineering, Indian Institute of Technology, Kanpur, Uttar Pradesh, 208016, India +1
Beyond Hard Constraints: Budget-Conditioned Reachability For Safe Offline Reinforcement Learning
Mar 8, 2026Janaka Chathuranga Brahmanage, Akshat KumarSafe RL Constrained RL
School of Computing and Information Systems, Singapore Management University.
Expert Knowledge-driven Reinforcement Learning for Autonomous Racing via Trajectory Guidance and Dynamics Constraints
Mar 6, 2026Bo Leng, Weiqi Zhang, Zhuoren Li +4Constrained RL Autonomous Racing
College of Automotive and Energy Engineering, Tongji University, Shanghai, 201804, China · School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore, 639798, Singapore
A Constrained RL Approach for Cost-Efficient Delivery of Latency-Sensitive Applications
Mar 4, 2026Ozan Aygün, Vincenzo Norman Vitale, Antonia M. Tulino +3Online Resource Allocation Real-Time Scheduling
New York University, NY, USA · Università degli Studi Napoli Federico II, Naples, Italy · Intel Labs +1
Swimming Under Constraints: A Safe Reinforcement Learning Framework for Quadrupedal Bio-Inspired Propulsion
Mar 4, 2026Xinyu Cui, Fei Han, Hang Xu +9Constrained RL Underwater Robotics
FSI lab, Westlake University, Hangzhou, 310030, China. · Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China. · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China +3
Conformal Policy Control
Mar 2, 2026Drew Prinster, Clara Fannjiang, Ji Won Park +4Constrained RL RL Exploration
Prescient Design, Genentech, U.S.A. · Johns Hopkins University, Baltimore, U.S.A. · New York University, NYC, U.S.A. +1
Selection as Power: Constrained Reinforcement for Bounded Decision Authority
Mar 2, 2026Jose Manuel de la Chica Rodriguez, Juan Manuel Vera DíazAI Agent Governance Constrained RL
AI Lab, Grupo Santander Madrid, Spain
Learning Thermal-Aware Locomotion Policies for an Electrically-Actuated Quadruped Robot
Mar 2, 2026Letian Qian, Yuhang Wan, Shuhan Wang +1Robot Locomotion Quadruped Locomotion
State Key Laboratory of Intelligent Manufacturing Equipment and Technology, Huazhong University of Science and Technology, Wuhan, China
LLM-assisted Semantic Option Discovery for Facilitating Adaptive Deep Reinforcement Learning
Mar 2, 2026Chang Yao, Jinghui Qin, Kebing Jin +1Reinforcement Learning LLM-Guided RL
Integrating LTL Constraints into PPO for Safe Reinforcement Learning
Mar 1, 2026Maifang Zhang, Hang Yu, Qian Zuo +3Safe RL Constrained RL
School of Informatics, University of Edinburgh, Edinburgh EH8 9AB, Scotland. · School of Computer Science, Faculty of Engineering, University of Sydney, Darlington NSW 2008, Australia. · School of Engineering and Physical Sciences, Heriot-Watt University, Edinburgh EH14 4AS, Scotland.
Multi-Objective Reinforcement Learning for Large-Scale Tote Allocation in Human-Robot Collaborative Fulfillment Centers
Feb 27, 2026Sikata Sengupta, Guangyi Liu, Omer Gottesman +4Multi-Objective Reinforcement Learning Supply Chain Management
Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA, USA · Amazon.
Learning to maintain safety through expert demonstrations in settings with unknown constraints: A Q-learning perspective
Feb 27, 2026George Papadopoulos, George A. VourosSafe RL Constrained RL
University of Piraeus, Piraeus, Greece
RLShield: Practical Multi-Agent RL for Financial Cyber Defense with Attack-Surface MDPs and Real-Time Response Orchestration
Feb 26, 2026Srikumar NayakAutonomous Cyber Defense Cybersecurity
Incedo Inc, USA
Reinforcing Real-world Service Agents: Balancing Utility and Cost in Task-oriented Dialogue
Feb 26, 2026Ning Gao, Wei Zhang, Yuqin Dai +6Task-Oriented Dialogue Systems Conversational Agents
Meituan, Beijing, China.
Provable Last-Iterate Convergence for Multi-Objective Safe LLM Alignment via Optimistic Primal-Dual
Feb 25, 2026Yining Li, Peizhong Ju, Ness ShroffLLM Safety Alignment RL from Human Feedback
The Ohio State University · University of Kentucky
Fuz-RL: A Fuzzy-Guided Robust Framework for Safe Reinforcement Learning under Uncertainty
Feb 24, 2026Xu Wan, Chao Yang, Cheng Yang +2Safe RL Reinforcement Learning
Zhejiang University · Alibaba DAMO Academy · Peking University
Safe Continuous-time Multi-Agent Reinforcement Learning via Epigraph Form
Feb 19, 2026Xuefeng Wang, Lei Zhang, Henglin Pu +2Multi-Agent Reinforcement Learning Continuous-Time RL
Purdue University
Demonstration-Guided Observation Attacks on Black-Box Safe Reinforcement Learning Controllers for Robotic Systems
Feb 18, 2026Jialiang Fan, Shixiong Jiang, Mengyu Liu +1Adversarial Attacks Black-Box Adversarial Attacks
University of Notre Dame, Notre Dame, IN, USA. · Washington State University Tri-Cities, Richland, WA, USA.
Capacity-constrained demand response in smart grids using deep reinforcement learning
Feb 18, 2026Shafagh Abband Pashaki, Sepehr Maleki, Amir BadieeDemand Response Constrained RL
Lincoln AI Lab, University of Lincoln, Lincoln, United Kingdom

Mar 24, 2026J/K move · Enter open · S save

Safe Reinforcement Learning with Preference-based Constraint Inference

Chenglin Li, Grant Ruan, Hua Geng

Department of Automation, Tsinghua University, Beijing, China · Laboratory for Information & Decision Systems, Massachusetts Institute of Technology, Cambridge, MA, USA

Constrained RL Safe RL Preference-Based RL

Open paper AskPDF arXiv

Safe reinforcement learning (RL) is a standard paradigm for safety-critical decision making. However, real-world safety constraints can be complex, subjective, and even hard to explicitly specify. Existing works on constraint inference rely on restrictive assumptions or extensive expert demonstrations, which are not realistic in many real-world applications. How to cheaply and reliably learn these constraints is the major challenge we focus on in this study. While inferring constraints from human preferences offers a data-efficient alternative, we identify popular Bradley-Terry (BT) models fail to capture the asymmetric, heavy-tailed nature of safety costs, resulting in risk underestimation. It is still rare in the literature to understand the impacts of BT models on the downstream policy learning. To address the above knowledge gaps, we propose a novel approach namely Preference-based Constrained Reinforcement Learning (PbCRL). We introduce a novel dead zone mechanism into preference modeling and theoretically prove that it encourages heavy-tailed cost distributions, thereby achieving better constraint alignment. Additionally, we incorporate a Signal-to-Noise Ratio (SNR) loss to encourage exploration by cost variances, which is found to benefit policy learning. Further, two-stage training strategy is deployed to lower online labeling burdens while adaptively enhancing constraint satisfaction. Empirical results demonstrate that PbCRL achieves superior alignment with true safety requirements and outperforms state-of-the-art baselines in terms of safety and reward. Our work explores a promising and effective way for constraint inference in Safe RL, with great potential in various safety-critical applications.