Constrained RL

RL: Reinforcement Learning

Constrained reinforcement learning studies how agents can maximize rewards while obeying limits on costs such as collisions, energy use, latency, or harm. Many methods model the problem as a constrained Markov decision process and use Lagrangian or primal-dual updates; others enforce feasibility with action masks, projections, safety shields, control barrier functions, or reachability analysis. These approaches are used in settings from robot control and power systems to language-model alignment, with work spanning online and offline learning, multi-agent tasks, and formal guarantees for regret and constraint satisfaction. A central challenge is balancing useful performance with reliable safety: average-cost limits can miss rare severe failures, while strict safeguards may make learning conservative or difficult to scale.

Latest papers

226 papers38 papers in the last four weeks, up 46% on the four weeks before. 0.2% of all new papers.

Last-Iterate Convergence for Constrained MDPs: From Dual Descent to Primal-Dual
Oct 29, 2026Michael Lu, Max Qiushi Lin, Mo Chen +1Last-Iterate Convergence Constrained RL
Simon Fraser University · Canada CIFAR AI Chair, Amii, Simon Fraser University
Force-Driven Hook Disassembly via Constraint-Aware Object-Centric Reinforcement Learning
Oct 28, 2026Nadeen Elhadad, Takuya Kiyokawa, Kensuke HaradaRobotic Manipulation Reinforcement Learning
N. Elhadad, T. Kiyokawa, and K. Harada are with the Department of Systems Innovation, Graduate School of Engineering Science, The University of Osaka, 1-3 Machikaneyama, Toyonaka, Osaka, Japan. · K. Harada is with the Industrial Cyber-physical Systems Research Center, The National Institute of Advanced Industrial Science and Technology (AIST), 2-3-26 Aomi, Koto-ku, Tokyo, Japan.
Variational Consequence-Driven Offline Reinforcement Learning
Oct 28, 2026Wen Jiang, Ke Jiang, Xiaoyang TanOffline RL Constrained RL
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence · Graduate School of Information Science and Technology, The University of Osaka
Reinforcement learning for constrained stochastic linear--quadratic control in continuous time
Oct 26, 2026Weiping Wu, Qitong Pan, Xun Li +1Stochastic Optimal Control Linear Quadratic Regulator
School of Economics and Management, Fuzhou University, Fuzhou, 350108, Fujian, China · Department of Applied Mathematics, The Hong Kong Polytechnic University, Hong Kong, China
CORMA: Constraint-Routed Decision Mamba for Generalizable Multi-Constraint Offline Safe Reinforcement Learning
Oct 25, 2026Changxu Ni, Feiran You, Yongan Guo +5Safe RL Reinforcement Learning
Southeast University · The University of Hong Kong · Nanjing University of Posts and Telecommunications
Physics-Informed Graph-Based Safe RL with Constraint-Specific Critics for Community Battery Coordination in Distribution Network
Oct 24, 2026Chenghao Huang, Yue Yang, Hao WangPower Systems Safe RL
Department of Data Science and AI, Faculty of IT and Monash Energy Institute, Monash University, Melbourne, VIC 3800, Australia
RBGA: Residual-Budget Guide Arbitration for Offline-to-Online Safe Reinforcement Learning in Dexterous Manipulation
Oct 24, 2026Zongyou Yu, Ao Zhou, Xiongbo Shi +7Reinforcement Learning Safe RL
Tongji University · Shanghai Innovation Institute · Xiaomi EV +2
MDPO: Multi-constraint Dexterous Policy Optimization with Reinforcement Learning
Oct 24, 2026Ao Zhou, Zongyou Yu, Ke Wu +5Safe Robot Manipulation Robotic Manipulation
Tongji University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · Fudan University, Shanghai, China +2
Primal-Dual Representation Learning for Low-Rank Constrained MDPs
Oct 22, 2026Chenhao Zhou, Chao Zhang, Wensong Bai +2Markov Decision Processes Constrained RL
College of Computer Science and Technology, Zhejiang University
Confessions of a Masked MDP
Oct 20, 2026Sebastian Reboul, Lorenzo Croissant, Hélène HalconruyReinforcement Learning Constrained RL
SAMOVAR, Télécom SudParis, Institut Polytechnique de Paris · Modal’X, Université Paris-Nanterre · Wiremind Cargo, AI & ML Group +1
Next Best Action: Causal Constrained Offline RL for Business Action Targeting
Oct 19, 2026Surya Ramachandiran, Yiming Li, Juan Mancilla-Caceres +2Offline RL Constrained RL
Lyft, Inc., San Francisco, CA, USA
LOCU: Lowdin-Orthogonalized Constraint Updates for Multi-Constraint Policy Optimization
Oct 19, 2026Joonyoung Lim, Younghwan YooConstrained RL Trust Region Policy Optimization
Department of Information Convergence Engineering, Pusan National University · School of Computer Science and Engineering, Pusan National University
Specification Gaming and a Constrained Reinforcement Learning Mitigation in a Cleaning Gridworld
Oct 15, 2026Hilda Adwubi OseiReward Hacking Reward Design for RL
Department of Industrial Engineering, Kwame Nkrumah University of Science and Technology, Kumasi, Ghana
Finite-Time Guarantees for Constrained Reinforcement Learning under Partial Observability
Oct 15, 2026Valentina Zaccaria, Philip Jordan, Tingting Ni +2Reinforcement Learning Constrained RL
University of Padova · SYCAMORE, EPFL
Constrained Multi-Agent Reinforcement Learning in Imagination
Oct 15, 2026Raz Shmueli, Shahaf S. ShperbergConstrained MARL Multi-Agent Reinforcement Learning
Ben-Gurion University of the Negev, Beer-Sheva, Israel
Constrained Differentiable Policy Learning for Safe Quadrotor Navigation
Oct 12, 2026Yan Rui Tan, Praveen Krishnapur, Zeyu Feng +3Robot Policy Learning Constrained RL
National University of Singapore, Singapore. · Agency for Science, Technology and Research (A*STAR), Singapore.
Phase-Gated Constrained Reinforcement Learning for End-to-End Capture and Detumbling of Non-Cooperative Targets
Oct 11, 2026Álvaro Belmonte-Baeza, Celia Redondo-Verdú, José L. Ramón +3Space Robotics Constrained RL
Department of Computer Science and Artificial Intelligence, University of Alicante, Spain. · Department of Physics, Systems Engineering and Signal Theory, University of Alicante, Spain. · Centre for Space Systems, Cranfield University, Cranfield, UK.
Learning Safety-Constrained Whole-Body Loco-Manipulation for Lunar Surface Operations
Oct 11, 2026Alvaro Belmonte-Baeza, Miguel Cazorla, Gabriel J. García +2Robotics Whole-Body Robotic Manipulation
Goal-Oriented Bundle Management and Flow Control for Deep-Space Communications via Reinforcement Learning
Oct 9, 2026Sajjad Baghaee, Aimin Li, Shanmugarajan Balasubramanian +1Wireless Communications Age of Information
Communication Networks Research Group (CNG), Dept. of Electrical and Electronics Engineering, Middle East Technical University (METU), Ankara, Türkiye
FAITH: Feasibility-Aware Safety-Filtered RL for High-Dimensional Systems
Oct 8, 2026Songyuan Zhang, Baljeet Singh, Sarthak Ranjeet Kaingade +2Safe RL Constrained RL
Massachusetts Institute of Technology · Amazon
A Unified Bellman Operator for Safety-Critical Reinforcement Learning
Oct 8, 2026Nishanth Arun Rao, Royina Karegoudra Jayanth, Benjamin Eysenbach +1Safe RL Constrained RL
Department of Electrical and Computer Engineering Princeton University · Department of Computer Science Princeton University
Toward Optimal Regret in Adversarial MDPs with Stochastic Hard Constraints
Oct 8, 2026Qian Zuo, Francesco Emanuele StradiMarkov Decision Processes Constrained RL
University of Edinburgh · Politecnico di Milano
Constrained Command-Conditioned Reinforcement Learning with Bandit Strategy Selection in Real-Time Strategy Games
Oct 8, 2026Nick Leenders, Roy Lindelauf, Joost van Oijen +1Reinforcement Learning Game-Playing Agents
NLR Royal Netherlands Aerospace Centre Amsterdam, The Netherlands · Data Science Center of Excellence Faculty of Military Sciences, Breda, NL · Departement Intelligent Systems Tilburg University, Tilburg, NL
Safe on Average, Unsafe in the Tail: When Is the Episodic-Cost Tail Controllable?
Oct 7, 2026Samuel Tetteh, Cody FlemingSafe RL Constrained RL
Iowa State University Ames, Iowa, USA
Learning Unknown Constraints without Unsafe Data via Optimality and Counterfactual Regularization
Oct 7, 2026Zhouyu Zhang, Chih-Yuan Chiu, Glen ChouConstrained RL Learning from Demonstration
School of Electrical and Computer Engineering Georgia Institute of Technology Atlanta, GA 30309 · School of Cybersecurity and Privacy School of Aerospace Engineering Georgia Institute of Technology
Who Bears the Burden? Learning Responsibility for Shared Constraints in Multi-Agent Reinforcement Learning
Oct 5, 2026Xiaoyang Cao, Jingqi Li, Zhe Fu +1Game Theory Constrained MARL
Massachusetts Institute of Technology · The University of Texas at Austin · Stanford University +1
Constrained Goal-directed Planar Graph Generation with Grammar-based Reinforcement Learning
Oct 5, 2026Nicolas Hochuli, Lorenzo Miele, Kristina Shea +1Constrained Generative Modeling Graph Generation
Engineering Design and Computing Laboratory ETH Zürich
Reachability-Aware Diffusion Policy Optimization
Oct 5, 2026Hikmet Simsir, Kutay Demiray, Ozgur S. OguzConstrained RL Diffusion Policy
Department of Computer Engineering, Bilkent University
FSPO: Policy-Consistent Risk and Pareto-Feasible Control for Budgeted LLM RL Post-Training
Oct 2, 2026Miaobo Hu, Shuhao Hu, Xiaobo Guo +3Reinforcement Learning RL Post-Training
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China
Instance-Dependent Regret for CMDPs with Step-Wise Constraints
Oct 1, 2026Qian Zuo, Francesco Emanuele Stradi, Leyang Xue +1Constrained RL Safe RL
University of Edinburgh · Politecnico di Milano · University College London

Oct 29, 2026J/K move · Enter open · S save

Last-Iterate Convergence for Constrained MDPs: From Dual Descent to Primal-Dual

Michael Lu, Max Qiushi Lin, Mo Chen, Sharan Vaswani

Simon Fraser University · Canada CIFAR AI Chair, Amii, Simon Fraser University

Last-Iterate Convergence Constrained RL Primal-Dual Optimization

Open paper AskPDF arXiv

We study policy optimization for infinite-horizon, discounted constrained Markov decision processes. While primal-dual (

\texttt{PD}

) methods are widely used in practice, their theoretical guarantees typically hold for an impractical mixture policy. In contrast, dual descent (

\texttt{DD}

) methods output a single policy ("last iterate") with guarantees on its performance. However, such methods require iteratively solving entropy-regularized MDPs in an inner-loop, limiting their practical effectiveness. Moreover, existing analyses rely on strong coverage assumptions that are difficult to satisfy in practice. Consequently, we first provide a new analysis of

\texttt{DD}

methods that does not require coverage assumptions. We then employ entropy-regularized natural policy gradient and dual regularization to eliminate the inner loop, resulting in an efficient

\texttt{PD}

algorithm. We develop a novel tracking analysis which shows that, compared to existing methods, the proposed algorithm achieves improved last-iterate convergence guarantees. Furthermore, our theoretical framework allows us to analyze a truncated bilevel method that interpolates between

\texttt{DD}

and

\texttt{PD}

methods. Finally, we generalize our analysis to handle linear function approximation. Compared to prior last-iterate methods, we prove stronger guarantees under less restrictive assumptions.