Safe RL

RL: Reinforcement Learning

Ask about this topic

Overview

Safe reinforcement learning aims to maximize task performance while limiting hazards such as collisions, equipment damage, or unsafe medical decisions, including during exploration and deployment. Common approaches formulate constraints as costs in constrained Markov decision processes and use primal-dual or Lagrangian optimization; others add control-barrier-function, Lyapunov, or reachability shields that filter actions, sometimes using learned models to account for uncertainty. Work also addresses offline learning, human preference feedback, temporal-logic specifications, and applications from robotics and driving to healthcare and energy control. A central challenge is balancing strong, reliable safety guarantees with reward, data efficiency, and adaptation to unfamiliar conditions; benchmarks also show that methods vary across tasks and that average costs can hide rare severe failures.

Latest papers

CardsList

217 papers24 papers in the last four weeks, up 14% on the four weeks before. 0.1% of all new papers.

Mar 17, 2026

SLowRL: Safe Low-Rank Adaptation Reinforcement Learning for Locomotion

Sim-to-real transfer of locomotion policies often leads to performance degradation due to the inevitable sim-to-real gap. Naively fine-tuning these policies directly on hardware is problematic, as it poses risks of mechanical failure and suffers from high sample inefficiency. In this paper, we address the challenge of safely and efficiently fine-tuning reinforcement learning (RL) policies for dynamic locomotion tasks. Specifically, we focus on fine-tuning policies learned in simulation directly on hardware, while explicitly enforcing safety constraints. In doing so, we introduce SLowRL, a framework that combines Low-Rank Adaptation (LoRA) with training-time safety enforcement via a recovery policy. We evaluate our method both in simulation and on a real Unitree Go2 quadruped robot for jump and trot tasks. Experimental results show that our method achieves a 46.5%46.5\% reduction in fine-tuning time and near-zero safety violations compared to standard proximal policy optimization (PPO) baselines. Notably, we find that a rank-1 adaptation alone is sufficient to recover pre-trained performance in the real world, while maintaining stable and safe real-world fine-tuning. These results demonstrate the practicality of safe, efficient fine-tuning for dynamic real-world robotic applications.

Elham Daneshmand, Shafeef Omar, Glen Berseth +2

McGill University · Mila - Quebec AI Institute · Technical University of Munich +1

RL Fine-TuningRL for Legged LocomotionSim-to-Real TransferSafe RL
AskPDF
Mar 16, 2026

Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies

Offline safe reinforcement learning (RL) seeks reward-maximizing policies from static datasets under strict safety constraints. Existing methods often rely on soft expected-cost objectives or iterative generative inference, which can be insufficient for safety-critical real-time control. We propose Safe Flow Q-Learning (SafeFQL), which extends FQL to safe offline RL by combining a Hamilton--Jacobi reachability-inspired safety value function with an efficient one-step flow policy. SafeFQL learns the safety value via a self-consistency Bellman recursion, trains a flow policy by behavioral cloning, and distills it into a one-step actor for reward-maximizing safe action selection without rejection sampling at deployment. Empirically, SafeFQL trades modestly higher offline training cost for substantially lower inference latency than diffusion-style safe generative baselines, which is advantageous for real-time safety-critical deployment. Across boat navigation, and Safety Gymnasium MuJoCo tasks, SafeFQL matches or exceeds prior offline safe RL performance while substantially reducing constraint violations.

Mumuksh Tayal, Manan Tayal, Ravi Prakash

Indian Institute of Science, India

Flow MatchingOffline RLSafe RLFlow-Based Policy Learning
AskPDF
Mar 11, 2026

Ensuring Safety in Automated Mechanical Ventilation through Offline Reinforcement Learning and Digital Twin Verification

Mechanical ventilation (MV) is a life-saving intervention for patients with acute respiratory failure (ARF) in the ICU. However, inappropriate ventilator settings could cause ventilator-induced lung injury (VILI). Also, clinicians workload is shown to be directly linked to patient outcomes. Hence, MV should be personalized and automated to improve patient outcomes. Previous attempts to incorporate personalization and automation in MV include traditional supervised learning and offline reinforcement learning (RL) approaches, which often neglect temporal dependencies and rely excessively on mortality-based rewards. As a result, early stage physiological deterioration and the risk of VILI are not adequately captured. To address these limitations, we propose Transformer-based Conservative Q-Learning (T-CQL), a novel offline RL framework that integrates a Transformer encoder for effective temporal modeling of patient dynamics, conservative adaptive regularization based on uncertainty quantification to ensure safety, and consistency regularization for robust decision-making. We build a clinically informed reward function that incorporates indicators of VILI and a score for severity of patients illness. Also, previous work predominantly uses Fitted Q-Evaluation (FQE) for RL policy evaluation on static offline data, which is less responsive to dynamic environmental changes and susceptible to distribution shifts. To overcome these evaluation limitations, interactive digital twins of ARF patients were used for online "at the bedside" evaluation. Our results demonstrate that T-CQL consistently outperforms existing state-of-the-art offline RL methodologies, providing safer and more effective ventilatory adjustments. Our framework demonstrates the potential of Transformer-based models combined with conservative RL strategies as a decision support tool in critical care.

Hang Yu, Huidong Liu, Qingchen Zhang +5

School of Engineering, University of Warwick, Coventry, UK · School of Computer Science, Chongqing University, China · School of Computer Science and Technology, Hainan University, China +1

Offline RLSafe RLTransformer-Based RLRobust RL
AskPDF
Mar 10, 2026

SEA-Nav: Efficient Policy Learning for Safe and Agile Quadruped Navigation in Cluttered Environments

Efficiently learning safe and agile quadruped navigation in densely cluttered environments remains difficult: existing methods often lack safety and agility, or become conservative in complex scenes and require long training schedules. We propose SEA-Nav (Safe, Efficient, and Agile Navigation), a safe reinforcement learning framework for quadruped navigation in cluttered environments. A differentiable control barrier function (CBF) shield constrains the policy to produce safe velocity commands. An adaptive collision-state initialization mechanism increases the probability of learning from safety-critical near-collision experience. An action regularization term further suppresses infeasible commands for physical deployment. The policy converges after about one hour of training on a single RTX 4090 and transfers zero-shot to real-world cluttered scenes.

Shiyi Chen, Mingye Yang, Haiyi Liu +6

Tsinghua University, Beijing, China. · Imperial College London, London, UK.

Quadruped Robot NavigationRobot NavigationSafe Robot NavigationSafe RLData-Efficient Robot Learning
AskPDF
Mar 10, 2026

SPAARS: Safer RL Policy Alignment through Abstract Exploration and Refined Exploitation of Action Space

Offline-to-online reinforcement learning (RL) offers a promising paradigm for robotics by pre-training policies on safe, offline demonstrations and fine-tuning them via online interaction. However, a fundamental challenge remains: how to safely explore online without deviating from the behavioral support of the offline data? While recent methods leverage conditional variational autoencoders (CVAEs) to bound exploration within a latent space, they inherently suffer from an exploitation gap -- a performance ceiling imposed by the decoder's reconstruction loss. We introduce SPAARS, a curriculum learning framework that initially constrains exploration to the low-dimensional latent manifold for sample-efficient, safe behavioral improvement, then seamlessly transfers control to the raw action space, bypassing the decoder bottleneck. SPAARS has two instantiations: the CVAE-based variant requires only unordered (s,a) pairs and no trajectory segmentation; SPAARS-SUPE pairs SPAARS with OPAL temporal skill pretraining for stronger exploration structure at the cost of requiring trajectory chunks. We prove an upper bound on the exploitation gap using the Performance Difference Lemma, establish that latent-space policy gradients achieve provable variance reduction over raw-space exploration, and show that concurrent behavioral cloning during the latent phase directly controls curriculum transition stability. Empirically, SPAARS-SUPE achieves 0.825 normalized return on kitchen-mixed-v0 versus 0.75 for SUPE, with 5x better sample efficiency; standalone SPAARS achieves 92.7 and 102.9 normalized return on hopper-medium-v2 and walker2d-medium-v2 respectively, surpassing IQL baselines of 66.3 and 78.3 respectively, confirming the utility of the unordered-pair CVAE instantiation.

Swaminathan S K, Aritra Hazra

Department of Computer Science and Engineering, IIT Kharagpur

Reinforcement LearningOffline-To-Online RLRL ExplorationSafe RLSample-Efficient RLCurriculum RL
AskPDF
Mar 8, 2026

Beyond Hard Constraints: Budget-Conditioned Reachability For Safe Offline Reinforcement Learning

Sequential decision making using Markov Decision Process underpins many realworld applications. Both model-based and model free methods have achieved strong results in these settings. However, real-world tasks must balance reward maximization with safety constraints, often conflicting objectives, that can lead to unstable min/max, adversarial optimization. A promising alternative is safety reachability analysis, which precomputes a forward-invariant safe state, action set, ensuring that an agent starting inside this set remains safe indefinitely. Yet, most reachability based methods address only hard safety constraints, and little work extends reachability to cumulative cost constraints. To address this, first, we define a safetyconditioned reachability set that decouples reward maximization from cumulative safety cost constraints. Second, we show how this set enforces safety constraints without unstable min/max or Lagrangian optimization, yielding a novel offline safe RL algorithm that learns a safe policy from a fixed dataset without environment interaction. Finally, experiments on standard offline safe RL benchmarks, and a real world maritime navigation task demonstrate that our method matches or outperforms state of the art baselines while maintaining safety.

Janaka Chathuranga Brahmanage, Akshat Kumar

School of Computing and Information Systems, Singapore Management University.

Safe RLConstrained RLOffline RL
AskPDF
Mar 8, 2026

COOL-MC: Verifying and Explaining RL Policies for Multi-bridge Network Maintenance

Aging bridge networks require proactive, verifiable, and interpretable maintenance strategies, yet reinforcement learning (RL) policies trained solely on reward signals provide no formal safety guarantees and remain opaque to infrastructure managers. We demonstrate COOL-MC as a tool for verifying and explaining RL policies for multi-bridge network maintenance, building on a single-bridge Markov decision process (MDP) from the literature and extending it to a parallel network of three heterogeneous bridges with a shared periodic budget constraint, encoded in the PRISM modeling language. We train an RL agent on this MDP and apply probabilistic model checking and explainability methods to the induced discrete-time Markov chain (DTMC) that arises from the interaction between the learned policy and the underlying MDP. Probabilistic model checking reveals that the trained policy has a safety-violation probability of 3.5% over the planning horizon, being slightly above the theoretical minimum of 0% and indicating the suboptimality of the learned policy, noting that these results are based on artificially constructed transition probabilities and deterioration rates rather than real-world data, so absolute performance figures should be interpreted with caution. The explainability analysis further reveals, for instance, a systematic bias in the trained policy toward the state of bridge 1 over the remaining bridges in the network. These results demonstrate COOL-MC's ability to provide formal, interpretable, and practical analysis of RL maintenance policies.

Dennis Gross

Formal Methods for RLInterpretability in RLSafe RLFormal Verification
AskPDF
Mar 6, 2026

Beyond Scalar Rewards: Distributional Reinforcement Learning with Preordered Objectives for Safe and Reliable Autonomous Driving

Autonomous driving involves multiple, often conflicting objectives such as safety, efficiency, and comfort. In reinforcement learning (RL), these objectives are typically combined through weighted summation, which collapses their relative priorities and often yields policies that violate safety-critical constraints. To overcome this limitation, we introduce the Preordered Multi-Objective MDP (Pr-MOMDP), which augments standard MOMDPs with a preorder over reward components. This structure enables reasoning about actions with respect to a hierarchy of objectives rather than a scalar signal. To make this structure actionable, we extend distributional RL with a novel pairwise comparison metric, Quantile Dominance (QD), that evaluates action return distributions without reducing them into a single statistic. Building on QD, we propose an algorithm for extracting optimal subsets, the subset of actions that remain non-dominated under each objective, which allows precedence information to shape both decision-making and training targets. Our framework is instantiated with Implicit Quantile Networks (IQN), establishing a concrete implementation while preserving compatibility with a broad class of distributional RL methods. Experiments in Carla show improved success rates, fewer collisions and off-road events, and deliver statistically more robust policies than IQN and ensemble-IQN baselines. By ensuring policies respect rewards preorder, our work advances safer, more reliable autonomous driving systems.

Ahmed Abouelazm, Jonas Michel, Daniel Bogdoll +2

FZI Research Center for Information Technology, Germany · Karlsruhe Institute of Technology, Germany

Multi-Objective Reinforcement LearningRL for Autonomous DrivingSafe RLDistributional RL
AskPDF
Mar 2, 2026

Conformal Policy Control

An agent must try new behaviors to explore and improve. In high-stakes environments, an agent that violates safety constraints may cause harm and must be taken offline, curtailing any future interaction. Imitating old behavior is safe, but excessive conservatism discourages exploration. How much behavior change is too much? We show how to use any safe reference policy as a probabilistic regulator for any optimized but untested policy. Conformal calibration on data from the safe policy determines how aggressively the new policy can act, while provably enforcing the user's declared risk tolerance. Unlike conservative optimization methods, we do not assume the user has identified the correct model class nor tuned any hyperparameters. Unlike previous conformal methods, our theory provides finite-sample guarantees even for non-monotonic bounded loss functions, and it introduces a new policy control setting. Our experiments on applications ranging from natural language question answering to biomolecular engineering show that safe exploration is not only possible from the first moment of deployment, but can also improve performance.

Drew Prinster, Clara Fannjiang, Ji Won Park +4

Prescient Design, Genentech, U.S.A. · Johns Hopkins University, Baltimore, U.S.A. · New York University, NYC, U.S.A. +1

Constrained RLRL ExplorationConformal Risk ControlSafe RL
AskPDF
Mar 1, 2026

Integrating LTL Constraints into PPO for Safe Reinforcement Learning

This paper proposes Proximal Policy Optimization with Linear Temporal Logic Constraints (PPO-LTL), a framework that integrates safety constraints written in LTL into PPO for safe reinforcement learning. LTL constraints offer rigorous representations of complex safety requirements, such as regulations that broadly exist in robotics, enabling systematic monitoring of safety requirements. Violations against LTL constraints are monitored by limit-deterministic Büchi automata, and then translated by a logic-to-cost mechanism into penalty signals. The signals are further employed for guiding the policy optimization via the Lagrangian scheme. Extensive experiments on the Zones and CARLA environments show that our PPO-LTL can consistently reduce safety violations, while maintaining competitive performance, against the state-of-the-art methods. The code is at https://github.com/EVIEHub/PPO-LTL.

Maifang Zhang, Hang Yu, Qian Zuo +3

School of Informatics, University of Edinburgh, Edinburgh EH8 9AB, Scotland. · School of Computer Science, Faculty of Engineering, University of Sydney, Darlington NSW 2008, Australia. · School of Engineering and Physical Sciences, Heriot-Watt University, Edinburgh EH14 4AS, Scotland.

Safe RLConstrained RLProximal Policy Optimization
AskPDF
Feb 27, 2026

Learning to maintain safety through expert demonstrations in settings with unknown constraints: A Q-learning perspective

Given a set of trajectories demonstrating the execution of a task safely in a constrained MDP with observable rewards but with unknown constraints and non-observable costs, we aim to find a policy that maximizes the likelihood of demonstrated trajectories trading the balance between being conservative and increasing significantly the likelihood of high-rewarding trajectories but with potentially unsafe steps. Having these objectives, we aim towards learning a policy that maximizes the probability of the most promisingpromising trajectories with respect to the demonstrations. In so doing, we formulate the ``promise" of individual state-action pairs in terms of QQ values, which depend on task-specific rewards as well as on the assessment of states' safety, mixing expectations in terms of rewards and safety. This entails a safe Q-learning perspective of the inverse learning problem under constraints: The devised Safe QQ Inverse Constrained Reinforcement Learning (SafeQIL) algorithm is compared to state-of-the art inverse constraint reinforcement learning algorithms to a set of challenging benchmark tasks, showing its merits.

George Papadopoulos, George A. Vouros

University of Piraeus, Piraeus, Greece

Safe RLConstrained RLLearning from DemonstrationQ-Learning
AskPDF
Feb 24, 2026

Fuz-RL: A Fuzzy-Guided Robust Framework for Safe Reinforcement Learning under Uncertainty

Safe Reinforcement Learning (RL) is crucial for achieving high performance while ensuring safety in real-world applications. However, the complex interplay of multiple uncertainty sources in real environments poses significant challenges for interpretable risk assessment and robust decision-making. To address these challenges, we propose Fuz-RL, a fuzzy measure-guided robust framework for safe RL. Specifically, our framework develops a novel fuzzy Bellman operator for estimating robust value functions using Choquet integrals. Theoretically, we prove that solving the Fuz-RL problem (in Constrained Markov Decision Process (CMDP) form) is equivalent to solving distributionally robust safe RL problems (in robust CMDP form), effectively avoiding min-max optimization. Empirical analyses on safe-control-gym and safety-gymnasium scenarios demonstrate that Fuz-RL effectively integrates with existing safe RL baselines in a model-free manner, significantly improving both safety and control performance under various types of uncertainties in observation, action, and dynamics.

Xu Wan, Chao Yang, Cheng Yang +2

Zhejiang University · Alibaba DAMO Academy · Peking University

Safe RLReinforcement LearningConstrained RLDistributionally Robust RLModel-Free RL
AskPDF
Feb 23, 2026

Bellman Value Decomposition for Task Logic in Safe Optimal Control

Real-world tasks involve nuanced combinations of goal and safety specifications. In high dimensions, the challenge is exacerbated: formal automata become cumbersome, and the combination of sparse rewards tends to require laborious tuning. In this work, we consider the innate structure of the Bellman Value as a means to naturally organize the problem for improved automatic performance. Namely, we prove the Bellman Value for a complex task defined in temporal logic can be decomposed into a graph of Bellman Values, connected by a set of well-known Bellman equations (BEs): the Reach-Avoid BE, the Avoid BE, and a novel type, the Reach-Avoid-Loop BE. To solve the Value and optimal policy, we propose VDPPO, which embeds the decomposed Value graph into a two-layer neural net, bootstrapping the implicit dependencies. We conduct a variety of simulated and hardware experiments to test our method on complex, high-dimensional tasks involving heterogeneous teams and nonlinear dynamics. Ultimately, we find this approach greatly improves performance over existing baselines, balancing safety and liveness automatically.

William Sharpless, Oswin So, Dylan Hirsch +2

UCSD · MIT

Safe RLReinforcement Learning
AskPDF
Feb 20, 2026

Interacting safely with cyclists using Hamilton-Jacobi reachability and reinforcement learning

In this paper, we present a framework for enabling autonomous vehicles to interact with cyclists in a manner that balances safety and optimality. The approach integrates Hamilton-Jacobi reachability analysis with deep Q-learning to jointly address safety guarantees and time-efficient navigation. A value function is computed as the solution to a time-dependent Hamilton-Jacobi-Bellman inequality, providing a quantitative measure of safety for each system state. This safety metric is incorporated as a structured reward signal within a reinforcement learning framework. The method further models the cyclist's latent response to the vehicle, allowing disturbance inputs to reflect human comfort and behavioral adaptation. The proposed framework is evaluated through simulation and comparison with human driving behavior and an existing state-of-the-art method.

Aarati Andrea Noronha, Jean Oh

Autonomous Driving PlanningSafe RLRL for Autonomous Driving
AskPDF
Feb 19, 2026

LexiSafe: Offline Safe Reinforcement Learning with Lexicographic Safety-Reward Hierarchy

Offline safe reinforcement learning (RL) is increasingly important for cyber-physical systems (CPS), where safety violations during training are unacceptable and only pre-collected data are available. Existing offline safe RL methods typically balance reward-safety tradeoffs through constraint relaxation or joint optimization, but they often lack structural mechanisms to prevent safety drift. We propose LexiSafe, a lexicographic offline RL framework designed to preserve safety-aligned behavior. We first develop LexiSafe-SC, a single-cost formulation for standard offline safe RL, and derive safety-violation and performance-suboptimality bounds that together yield sample-complexity guarantees. We then extend the framework to hierarchical safety requirements with LexiSafe-MC, which supports multiple safety costs and admits its own sample-complexity analysis. Empirically, LexiSafe demonstrates reduced safety violations and improved task performance compared to constrained offline baselines. By unifying lexicographic prioritization with structural bias, LexiSafe offers a practical and theoretically grounded approach for safety-critical CPS decision-making.

Hsin-Jung Yang, Zhanhong Jiang, Prajwal Koirala +3

Dept. of Mechanical Engineering Iowa State University Ames, USA · Translational AI Center Iowa State University Ames, USA · Sibley School of Mechanical and Aerospace Engineering Cornell University Ithaca, USA

Reinforcement LearningSafe RLOffline RLCyber-Physical Systems
AskPDF
Feb 19, 2026

Safe Continuous-time Multi-Agent Reinforcement Learning via Epigraph Form

Multi-agent reinforcement learning (MARL) has made significant progress in recent years, but most algorithms still rely on a discrete-time Markov Decision Process (MDP) with fixed decision intervals. This formulation is often ill-suited for complex multi-agent dynamics, particularly in high-frequency or irregular time-interval settings, leading to degraded performance and motivating the development of continuous-time MARL (CT-MARL). Existing CT-MARL methods are mainly built on Hamilton-Jacobi-Bellman (HJB) equations. However, they rarely account for safety constraints such as collision penalties, since these introduce discontinuities that make HJB-based learning difficult. To address this challenge, we propose a continuous-time constrained MDP (CT-CMDP) formulation and a novel MARL framework that transforms discrete MDPs into CT-CMDPs via an epigraph-based reformulation. We then solve this by proposing a novel physics-informed neural network (PINN)-based actor-critic method that enables stable and efficient optimization in continuous time. We evaluate our approach on continuous-time safe multi-particle environments (MPE) and safe multi-agent MuJoCo benchmarks. Results demonstrate smoother value approximations, more stable training, and improved performance over safe MARL baselines, validating the effectiveness and robustness of our method.

Xuefeng Wang, Lei Zhang, Henglin Pu +2

Purdue University

Multi-Agent Reinforcement LearningContinuous-Time RLSafe RLConstrained RL
AskPDF
Feb 18, 2026

Demonstration-Guided Observation Attacks on Black-Box Safe Reinforcement Learning Controllers for Robotic Systems

Safe reinforcement learning (Safe RL) learns robotic controllers that optimize task rewards under safety constraints, yet observation perturbations can induce safety violations. Existing safety-directed attacks often require access to victim networks, gradients, critics, or explicit specifications -- assumptions rarely met once a controller is deployed as a black box. We propose a demonstration-guided observation attack for analyzing unknown Safe RL controllers. The framework recovers a state constraint and a surrogate policy through inverse constrained reinforcement learning, and learns dynamics from demonstration transitions. Their composed gradient generates bounded observation perturbations without victim parameters, gradients, or queries; demonstrations are the only victim-specific information. Across four Bullet tasks and one MetaDrive map with three budgets per victim, the attack exceeds every same-access baseline in 12 of 15 environment-budget conditions, and in 7 of those 12 it also exceeds the privileged reference attacks with access to the victim's reward and cost critics. Demonstrations released to support safe learning thus provide an attack surface for deployed black-box controllers. A defense study shows that state-adversarial regularization reduces attack cost, whereas the tested adversarial-training and demonstration-contamination schemes provide inconsistent protection.

Jialiang Fan, Shixiong Jiang, Mengyu Liu +1

University of Notre Dame, Notre Dame, IN, USA. · Washington State University Tri-Cities, Richland, WA, USA.

Adversarial AttacksBlack-Box Adversarial AttacksRobotic RLSafe RLConstrained RL
AskPDF
Feb 18, 2026

Dual-Quadruped Collaborative Transportation in Narrow Environments via Safe Reinforcement Learning

Collaborative transportation, where multiple robots collaboratively transport a payload, has garnered significant attention in recent years. While ensuring safe and high-performance inter-robot collaboration is critical for effective task execution, it is difficult to pursue in narrow environments where the feasible region is extremely limited. To address this challenge, we propose a novel approach for dual-quadruped collaborative transportation via safe reinforcement learning (RL). Specifically, we model the task as a fully cooperative constrained Markov game, where collision avoidance is formulated as constraints. We introduce a cost-advantage decomposition method that enforces the sum of team constraints to remain below an upper bound, thereby guaranteeing task safety within an RL framework. Furthermore, we propose a constraint allocation method that assigns shared constraints to individual robots to maximize the overall task reward, encouraging autonomous task-assignment among robots, thereby improving collaborative task performance. Simulation and real-time experimental results demonstrate that the proposed approach achieves superior performance and a higher success rate in dual-quadruped collaborative transportation compared to existing methods.

Zhezhi Lei, Zhihai Bi, Wenxin Wang +1

Robotics and Autonomous Systems Thrust, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China · Advanced Remanufacturing & Technology Centre (ARTC), Agency for Science, Technology and Research (A*STAR), Singapore 637143 · Cheng Kar-Shun Robotics Institute, The Hong Kong University of Science and Technology, Hong Kong SAR, China

Multi-Robot Cargo TransportationMulti-Robot SystemsCooperative MARLSafe RLRobot Collision Avoidance
AskPDF
Feb 17, 2026

Solving Parameter-Robust Avoid Problems with Unknown Feasibility using Reinforcement Learning

Recent advances in deep reinforcement learning (RL) have achieved strong results on high-dimensional control tasks, but applying RL to reachability problems raises a fundamental mismatch: reachability seeks to maximize the set of states from which a system remains safe indefinitely, while RL optimizes expected returns over a user-specified distribution. This mismatch can result in policies that perform poorly on low-probability states that are still within the safe set. A natural alternative is to frame the problem as a robust optimization over a set of initial conditions that specify the initial state, dynamics and safe set, but whether this problem has a solution depends on the feasibility of the specified set, which is unknown a priori. We propose Feasibility-Guided Exploration (FGE), a method that simultaneously identifies a subset of feasible initial conditions under which a safe policy exists, and learns a policy to solve the reachability problem over this set of initial conditions. Empirical results demonstrate that FGE learns policies with over 50% more coverage than the best existing method for challenging initial conditions across tasks in the MuJoCo simulator and the Kinetix simulator with pixel observations.

Oswin So, Eric Yang Yu, Songyuan Zhang +3

Department of Aeronautics and Astronautics, MIT · MIT Lincoln Laboratory

Safe RLReachability AnalysisRobust RLRL Exploration
AskPDF
Feb 16, 2026

Formally Verifying and Explaining Sepsis Treatment Policies with COOL-MC

Safe and interpretable sequential decision-making is critical in healthcare, yet reinforcement learning (RL) policies for sepsis treatment optimization remain opaque and difficult to verify. Standard probabilistic model checkers operate on the full state space, which becomes infeasible for larger MDPs, and cannot explain why a learned policy makes particular decisions. COOL-MC wraps the model checker Storm but adds three key capabilities: it constructs only the reachable state space induced by a trained policy, yielding a smaller discrete-time Markov chain amenable to verification even when full-MDP analysis is intractable; it automatically labels states with clinically meaningful atomic propositions; and it integrates explainability methods with probabilistic computation tree logic (PCTL) queries to reveal which features drive decisions across treatment trajectories. We demonstrate COOL-MC's capabilities on the ICU-Sepsis MDP, a benchmark derived from approximately 17,000 sepsis patient records, which serves as a case study for applying COOL-MC to the formal analysis of sepsis treatment policies. Our analysis establishes hard bounds via full MDP verification, trains a safe RL policy that achieves optimal survival probability, and analyzes its behavior via PCTL verification and explainability on the induced DTMC. This reveals, for instance, that our trained policy relies predominantly on prior dosing history rather than the patient's evolving condition, a weakness that is invisible to standard evaluation but is exposed by COOL-MC's integration of formal verification and explainability. Our results illustrate how COOL-MC could serve as a tool for clinicians to investigate and debug sepsis treatment policies before deployment.

Dennis Gross

Formal Methods for RLFormal VerificationHealthcareSafe RLInterpretability in RL
AskPDF
Feb 16, 2026

Near-Optimal Sample Complexity for Online Constrained MDPs

Safety is a fundamental challenge in reinforcement learning (RL), particularly in real-world applications such as autonomous driving, robotics, and healthcare. To address this, Constrained Markov Decision Processes (CMDPs) are commonly used to enforce safety constraints while optimizing performance. However, existing methods often suffer from significant safety violations or require a high sample complexity to generate near-optimal policies. We address two settings: relaxed feasibility, where small violations are allowed, and strict feasibility, where no violation is allowed. We propose a model-based primal-dual algorithm that balances regret and bounded constraint violations, drawing on techniques from online RL and constrained optimization. For relaxed feasibility, we prove that our algorithm returns an ε\varepsilon-optimal policy with ε\varepsilon-bounded violation with arbitrarily high probability, requiring O~(SAH3ε2)\tilde{O}\left(\frac{SAH^3}{\varepsilon^2}\right) learning episodes, matching the lower bound for unconstrained MDPs. For strict feasibility, we prove that our algorithm returns an ε\varepsilon-optimal policy with zero violation with arbitrarily high probability, requiring O~(SAH5ε2ζ2)\tilde{O}\left(\frac{SAH^5}{\varepsilon^2ζ^2}\right) learning episodes, where ζζ is the problem-dependent Slater constant characterizing the size of the feasible region. This result matches the lower bound for learning CMDPs with access to a generative model. Our results demonstrate that learning CMDPs in an online setting is as easy as learning with a generative model and is no more challenging than learning unconstrained MDPs when small violations are allowed.

Chang Liu, Yunfan Li, Lin F. Yang

University of California, Los Angeles

Constrained RLSafe RLSample-Efficient RLModel-Based RL
AskPDF
Feb 9, 2026

High-Speed Vision-Based Flight in Clutter with Safety-Shielded Reinforcement Learning

Quadrotor unmanned aerial vehicles (UAVs) are increasingly deployed in complex missions that demand reliable autonomous navigation and robust obstacle avoidance. However, traditional modular pipelines often incur cumulative latency, whereas purely reinforcement learning (RL) approaches typically provide limited formal safety guarantees. To bridge this gap, we propose an end-to-end RL framework augmented with model-based safety mechanisms. We incorporate physical priors in both training and deployment. During training, we design a physics-informed reward structure that provides global navigational guidance. During deployment, we integrate a real-time safety filter that projects the policy outputs onto a provably safe set to enforce strict collision-avoidance constraints. This hybrid architecture reconciles high-speed flight with robust safety assurances. Benchmark evaluations demonstrate that our method outperforms both traditional planners and recent end-to-end obstacle avoidance approaches based on differentiable physics. Extensive experiments demonstrate strong generalization, enabling reliable high-speed navigation in dense clutter and challenging outdoor forest environments at velocities up to 7.5 m/s}.

Jiarui Zhang, Chengyong Lei, Chengjiang Dai +4

Institute of Cyber-Systems and Control, College of Control Science and Engineering, Zhejiang University, Hangzhou 310027, China · Differential Robotics, Hangzhou 311121, China.

RL for RoboticsUAV NavigationSafe RLVision-Based Robot ControlRobot Collision Avoidance
AskPDF
Feb 6, 2026

The impact of temporal resampling on the safety of clinical reinforcement learning: A simulated trial

Reinforcement learning (RL) is a type of artificial intelligence for making optimal choices. In healthcare, researchers generally use offline RL (ORL), where models are trained and evaluated from retrospective observational data. To accommodate naturally irregular clinical records, researchers often resample the data into uniform time intervals before training (known as binning). However, binning may create counterfactual trajectories that do not represent clinical practice. We first quantified these distortions in patients receiving continuous insulin infusions in the MIMIC-IV intensive care database. Binning reversed the apparent causal order for over 70% of paired glucose measurements and insulin adjustments within the same interval, and frequently masked significant dysglycaemic events and merged distinct treatment decisions together. We then conducted an in silico clinical trial using 30 virtual patients with type 1 diabetes from the FDA-approved UVA/Padova simulator. The simulator was modified to include stochastic intervals between decisions and used to generate a training dataset for offline RL. We trained three ORL algorithms on both the unprocessed dataset and equivalent datasets resampled at 10-minute, 2-hour, and 4-hour intervals. When deployed back into the simulated environment, temporal resampling was found to reduce model performance by up to 60% relative to unprocessed data, with 4-hour binning causing all agents to perform worse than the dataset's baseline. Retrospective evaluation on resampled data actively obscured this effect, predicting 1.5-3x better returns than agents achieved in practice. We recommend that future research in this area prioritises datasets with natural clinical timings between decisions, which may be a necessary step before these models can be safely deployed into patient care.

Thomas Frost, Hrisheekesh Vaidya, Steve Harris

University College London, London, United Kingdom · University College London Hospitals NHS Foundation Trust, London, United Kingdom

Reinforcement LearningOffline RLSafe RL
AskPDF
Feb 5, 2026

Constrained Group Relative Policy Optimization

Group Relative Policy Optimization (GRPO) remains the dominant critic-free approach for fine-tuning LLMs and VLMs, but its compatibility with constrained policy optimization (e.g. for safety-critical domains) has not been carefully examined. In this work, we introduce Constrained GRPO, a Lagrangian-based extension of GRPO for constrained policy optimization. We show that the standard practice of scalarizing rewards before normalization introduces a critical Lagrangian-specific failure mode: GRPO's within-group normalization makes constrained optimization highly sensitive to how multi-component learning signals are aggregated. We show that scalarizing rewards before normalization introduces shared-denominator coupling, so that changing one multiplier alters not only the emphasis on its corresponding constraint, but also the relative weighting of the reward and other constraints. We address this with a simple but crucial modification: scalarizing standardized advantages rather than rewards. This yields a better-conditioned update by addressing the coupling induced by reward scalarization, resulting in better-behaved multiplier dynamics and more stable constraint enforcement in practice. Empirically, across a controlled gridworld, a real-world autonomous driving benchmark, and a mathematical reasoning task, Constrained GRPO consistently achieves better adherence to specified constraints while maintaining or improving task performance.

Roger Girgis, Rodrigue de Schaetzen, Luke Rowe +3

Mila – Quebec AI Institute Université de Montréal

Group Relative Policy OptimizationConstrained RLCritic-Free RLSafe RLRL Fine-TuningPolicy Learning
AskPDF
Jan 27, 2026

Reversible Simplex Supervision with Post-Action Debt Accounting for Goal-Reaching RL

Deploying reinforcement learning (RL) on multi-tonne robots calls for supervisory mechanisms that address both operational safety and progress toward task completion. However, repeated switching need not preserve task progress when a learned action increases storage before recovery takes control. We introduce reversible Simplex supervision with post-action debt accounting for a frozen finite-state policy and robust-adaptive recovery. Under exact sampled-state information and stated model and certificate conditions, we prove that recovery repayment exceeding a uniform triggering-edge debt bound guarantees finite switching and finite-sample goal entry. We formulate reachability-based certificate constructions for establishing these sufficient conditions. In 20 matched simulations, goal-entry counts are 20 with debt gating and 18 without it; the two remaining runs terminate under the supervisor's admissibility stopping rule. On an experimental 6000 kg robot, 24 asphalt and soft-terrain trials evaluate 50 ms supervision above a 1 kHz actuator stack; all eight triggered recoveries complete debt-gated re-entry. The experiments demonstrate the supervisory mechanism in the tested trials.

Mehdi Heydari Shahna, Joongheon Kim, Jouni Mattila

Faculty of Engineering and Natural Sciences, Tampere University, Tampere, Finland · Department of Electrical and Computer Engineering, Korea University, Seoul, Republic of Korea

RL for RoboticsRobot SafetyRuntime AssuranceSafe RL
AskPDF
Dec 29, 2025

SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraints

In safety-critical domains, reinforcement learning (RL) systems must satisfy strict, zero-cost safety constraints while achieving meaningful task performance. Existing model-free methods can struggle to achieve high safety without substantially compromising task performance. We introduce Safety-Biased Trust Region Policy Optimisation (SB-TRPO), a principled approach to RL with zero-cost constraints, which requires only a fixed fraction of the maximal cost reduction achievable within the trust region, thus retaining flexibility for reward optimisation. We show that the idealised update nevertheless converges to zero cost and maximal reward amongst zero-cost policies in finite MDPs. A practical gradient-based approximation provides local improvements in both safety and reward under suitable gradient alignment. Experiments on Safety Gymnasium demonstrate high safety alongside strong task performance.

Dominik Wagner, Ankit Kanwar, Luke Ong

NTU Singapore · Sony Corporation

Safety-Critical ControlPolicy OptimizationConstrained RLTrust Region Policy OptimizationSafe RLPolicy Learning
AskPDF
Dec 10, 2025

Safe Learning for Contact-Rich Robot Tasks: A Survey from Classical Learning-Based Methods to Safe Foundation Models

Contact-rich tasks pose significant challenges for robotic systems due to inherent uncertainty, complex dynamics, and the high risk of damage during interaction. Recent advances in learning-based control have shown great potential in enabling robots to acquire and generalize complex manipulation skills in such environments, but ensuring safety, both during exploration and execution, remains a critical bottleneck for reliable real-world deployment. This survey provides a comprehensive overview of safe learning-based methods for robot contact-rich tasks. We categorize existing approaches into two main domains: safe exploration and safe execution. We review key techniques, including constrained reinforcement learning, risk-sensitive optimization, uncertainty-aware modeling, control barrier functions, and model predictive safety shields, and highlight how these methods incorporate prior knowledge, task structure, and online adaptation to balance safety and efficiency. A particular emphasis of this survey is on how these safe learning principles extend to and interact with emerging robotic foundation models, especially vision-language models (VLMs) and vision-language-action models (VLAs), which unify perception, language, and control for contact-rich manipulation. We discuss both the new safety opportunities enabled by VLM/VLA-based methods, such as language-level specification of constraints and multimodal grounding of safety signals, and the amplified risks and evaluation challenges they introduce. Finally, we outline current limitations and promising future directions toward deploying reliable, safety-aligned, and foundation-model-enabled robots in complex contact-rich environments. More details and materials are available at our \href{ https://github.com/jack-sherman01/Awesome-Learning4Safe-Contact-rich-tasks}{Project GitHub Repository}.

Heng Zhang, Rui Dai, Gokhan Solak +4

Human-Robot Interfaces and Interaction Lab, Istituto Italiano di Tecnologia, Genova, Italy. · Ph.D. program of national interest in Robotics and Intelligent Machines (DRIM), Universit`a di Genova, Genoa, Italy. · Edwardson School of Industrial Engineering, Purdue University, West Lafayette, IN, USA. +3

Safe Robot ManipulationContact-Rich Robotic ManipulationSafe RLRobot Foundation ModelsVLMs for Robotics
AskPDF
Nov 6, 2025

Exchange Policy Optimization Algorithm for Semi-Infinite Safe Reinforcement Learning

Safe reinforcement learning (RL) aims to optimize long-term performance while adhering to safety requirements. However, many practical applications involve an infinite number of constraints, forming semi-infinite safe RL (SI-safe RL). Such scenarios typically appear when safety conditions must be enforced across an entire continuous parameter space, such as ensuring adequate resource distribution at every spatial location. Existing approaches typically tackle these continuous constraints through naive spatial discretization or stochastic sampling. Such methods inherently suffer from residual violations or provide only probabilistic safety guarantees. Therefore, no current framework can handle infinitely many constraints to provide reliable safety certificates. In this paper, we propose exchange policy optimization (EPO), an algorithmic framework that achieves optimal policy performance with provably bounded safety guarantees. EPO operates by iteratively solving safe RL subproblems restricted to a finite constraint set, adaptively adjusting the active set through constraint expansion and deletion. Specifically, at each iteration, constraints violating a predefined tolerance are added to refine the policy, while those with zero Lagrange multipliers are removed after the policy update. This exchange rule limits the subproblem complexity to ensure computational tractability while driving policy convergence. Theoretically, we establish that, under mild assumptions, EPO achieves finite convergence to a policy that both ensures the optimal reward performance and keeps the global constraint violation within the prescribed tolerance. Furthermore, we derive an upper bound on the required number of iterations and quantify the gap between the obtained policy and the true optimum.

Jiaming Zhang, Yujie Yang, Haoning Wang +2

Department of Mathematical Sciences Tsinghua University Beijing, 100084, China · School of Vehicle and Mobility Tsinghua University Beijing, 100084, China · Department of Mathematical Sciences, Tsinghua University Beijing, 100084, China +1

Safe RLConstrained RL
AskPDF
Oct 16, 2025

CBF-RL: Safety Filtering Reinforcement Learning in Training with Control Barrier Functions

Reinforcement learning (RL), while powerful and expressive, can often prioritize performance at the expense of safety. Yet safety violations can lead to catastrophic outcomes in real-world deployments. Control Barrier Functions (CBFs) offer a principled method to enforce dynamic safety -- traditionally deployed online via safety filters. While the result is safe behavior, the fact that the RL policy does not have knowledge of the CBF can lead to conservative behaviors. This paper proposes CBF-RL, a framework for generating safe behaviors with RL by enforcing CBFs in training. CBF-RL has two key attributes: (1) minimally modifying a nominal RL policy to encode safety constraints via a CBF term, (2) and safety filtering of the policy rollouts in training. Theoretically, we prove that continuous-time safety filters can be deployed via closed-form expressions on discrete-time roll-outs. Practically, we demonstrate that CBF-RL internalizes the safety constraints in the learned policy -- both enforcing safer actions and biasing towards safer rewards -- enabling safe deployment without the need for an online safety filter. We validate our framework through ablation studies on navigation tasks and on the Unitree G1 humanoid robot, where CBF-RL enables safer exploration, faster convergence, and robust performance under uncertainty, enabling the humanoid robot to avoid obstacles and climb stairs safely in real-world settings without a runtime safety filter.

Lizhi Yang, Blake Werner, Massimiliano de Sa +1

Caltech MCE

RL for RoboticsControl Barrier FunctionsConstrained RLSafety FilteringSafe RL
AskPDF
Jun 2, 2025

SafeOR-Gym: A Benchmark Suite for Safe Reinforcement Learning Algorithms on Practical Operations Research Problems

Most existing safe reinforcement learning (RL) benchmarks focus on robotics and control tasks, offering limited relevance to high-stakes domains that involve structured constraints, mixed-integer decisions, and industrial complexity. This gap hinders the advancement and deployment of safe RL in critical areas such as energy systems, manufacturing, and supply chains. To address this limitation, we present SafeOR-Gym, a benchmark suite of nine operations research (OR) environments tailored for safe RL under complex constraints. Each environment captures a realistic planning, scheduling, or control problems characterized by cost-based constraint violations, planning horizons, and hybrid discrete-continuous action spaces. The suite integrates seamlessly with the Constrained Markov Decision Process (CMDP) interface provided by OmniSafe. We evaluate several state-of-the-art safe RL algorithms across these environments, revealing a wide range of performance: while some tasks are tractable, others expose fundamental limitations in current approaches. SafeORGym provides a challenging and practical testbed that aims to catalyze future research in safe RL for real-world decision-making problems.

Asha Ramanujam, Adam Elyoumi, Hao Chen +5

Davidson School of Chemical Engineering, Purdue University, West Lafayette, IN · Energy Sciences, ExxonMobil Technology and Engineering Company, Annandale, NJ · Energy Sciences, ExxonMobil Technology and Engineering Company, Annadale, NJ

RL BenchmarksConstrained RLOperations ResearchSafe RL
AskPDF