Dexterous manipulation requires robots to leverage existing datasets for rapid skill acquisition while maintaining safety during real-world deployment to prevent damage to the robot or its surroundings. Offline-to-online safe reinforcement learning (O2O safe RL) provides a promising solution for this problem by leveraging prior experience for online improvement under safety constraints, thereby mitigating the sample inefficiency and exploration risk of learning safe policies from scratch. However, existing O2O safe RL approaches do not explicitly account for the safety budget already consumed when making decisions within an episode. Excessive cost incurred early can therefore leave little room for subsequent exploration and policy improvement. To address this limitation, we propose Residual-Budget Guide Arbitration (RBGA), an algorithm that explicitly tracks the remaining episode safety budget and uses it to adaptively select between offline guidance and the evolving online policy. At each step, RBGA compares the cost-to-go estimates of the guide and online actions against the remaining budget, enabling step-wise action arbitration throughout the episode. We provide theoretical analysis characterizing the safety implications of residual-budget-based selection and its effect on guide-assisted exploration. Extensive experiments demonstrate that RBGA enables effective online policy improvement while maintaining strong safety performance. Our code is available at this link.
Figures & tables
Figure 1: The figure depicts the reward, cost, and success rate curves of RBGA and the baseline algorithms in the comparison experiment. The shaded areas represent variance over three random seeds. The dashed line indicates the episode cost limit d=30 .
Figure 2: The figure depicts the reward, cost, and success rate curves under different guide horizon scheduling strategies. The shaded areas represent variance over three random seeds. The labels 40k, 60k, and 80k indicate the iterations at which guidance ends, while Random and Fixed retain guidance throughout training. The dashed line indicates the episode cost limit d=30 .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Network architecture
Hidden layers of each actor and critic
[1024,1024,512]
Hidden activation
ELU
Online policy
Tanh-squashed Gaussian
Reward critics / SAC-Lag cost critics
2 / 2
Auxiliary cost critics
2
Appendix
Table 1: RBGA hyperparameters used in the main experiments. The online SAC-Lag cost critic and the auxiliary cost estimator are separate networks.
Figure 3: The figure depicts the reward, cost, and success rate curves of RBGA and RBGA without residual-budget arbitration on four tasks. The shaded areas on the curves represent the variance obtained from online testing conducted with three random seeds. The dashed line indicates the episode cost limit d=30 .
Figure 4: The figure depicts the reward, cost, and success rate curves of RBGA and the baseline algorithms using DDPG-Lag as the online learner. The shaded areas on the curves represent the variance obtained from online testing conducted with three random seeds. The dashed line indicates the episode cost limit d=30 .
Figure 5: The figure depicts the reward, cost, and success rate curves of RBGA and the baseline algorithms on the Close Drawer and Pick and Place tasks. The shaded areas on the curves represent the variance obtained from online testing conducted with three random seeds. The dashed line indicates the episode cost limit d=30 .
Figure 6: Simulation and real-world deployment of RBGA on the Cube Grasping task. Top: simulation. Bottom: the physical robot. The platform consists of a RealMan RM65 robotic arm equipped with a Linker Hand L20 dexterous hand.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Safe exploration remains a fundamental challenge in reinforcement learning (RL), limiting the deployment of RL agents in the real world. We propose Sampling-Based Safe Reinforcement Learning (SBSRL), a model-based RL algorithm that maintains safety throughout the learning process by enforcing constraints jointly across a finite set of dynamics samples. This formulation approximates an intractable worst-case optimization over uncertain dynamics and enables practical safety guarantees in continuous domains. We further introduce an exploration strategy based on constraining epistemic uncertainty, eliminating the need for explicit exploration bonuses. Under regularity conditions, we derive high-probability guarantees of safety throughout learning and a finite-time sample complexity bound for recovering a near-optimal policy. Empirically, SBSRL achieves safe and efficient exploration both in simulation and in real robotic hardware, and readily extends to practical deep-ensemble implementations that scale to high-dimensional continuous control problems.
Offline safe reinforcement learning often requires policies to adapt at deployment time to safety budgets that vary across episodes or change within a single episode. While diffusion-based planners enable flexible trajectory generation, existing guidance schemes often treat reward improvement and constraint satisfaction as competing gradient objectives, which can lead to unreliable safety compliance under cost limits. We reinterpret adaptive safe trajectory generation as sampling from a constrained trajectory distribution, where the budget restricts the trajectory region, and reward shapes preferences within that region. This perspective motivates Safe Decoupled Guidance Diffusion (SDGD), which conditions classifier-free guidance on the cost limit to bias sampling toward trajectories satisfying the specified limit, while using reward-gradient guidance to refine trajectories for higher return. Because direct reward guidance can increase return while also steering samples toward trajectories with higher cumulative cost, we introduce Feasible Trajectory Relabeling (FTR) to reshape reward targets and discourage such directions. We further provide a first-order sampling-time analysis showing that FTR suppresses reward-induced cost drift under a prefix-restorative alignment condition. Extensive evaluations on the DSRL benchmark show that SDGD achieves the strongest safety compliance among baselines, satisfying the constraint on 94.7% of tasks (36/38), while obtaining the highest reward among safe methods on 21 tasks.
Rufeng Chen, Zhaofan Zhang, Zhejiang Yang +2
1The Hong Kong University of Science and Technology (Guangzhou) · 2Jilin University
Real-world reinforcement learning has enabled contact-rich manipulation tasks to be trained directly on physical robots, but unsafe interactions during online learning remain a key barrier to practical deployment. We propose Safe Evidential Real-World Reinforcement Learning (SEVI), a framework for reliability-aware safety intervention during real-world policy learning. SEVI introduces an Evidential Safety Critic that estimates both future constraint-violation risk and epistemic uncertainty in a single forward pass. During online learning, SEVI evaluates each task-policy action before execution; if the action is predicted to be unsafe or the prediction is insufficiently supported by evidence, SEVI overrides it with an offline-trained recovery policy that guides the robot toward a lower-risk region. We evaluate SEVI on two simulated manipulation tasks and a real-world USB insertion task under in-distribution and distribution-shifted conditions. Across task-learning and safety-critic baselines, SEVI achieves the strongest trajectory-level safety prediction, the highest final task success, and the lowest unsafe-event counts. SEVI also reduces unsafe events by 13.4 to 57.1 percent relative to recovery-based safe RL while preserving low-latency execution-time filtering. The project page is available at: https://sevi-safe.github.io/
Shaekh Mohammad Shithil, Lantao Liu
Luddy School of Informatics, Computing, and Engineering, Indiana University, Bloomington, IN 47408 USA