SEVI: Safe Evidential Real-World Reinforcement Learning

Shaekh Mohammad Shithil; Lantao Liu

cs.ROOct 22, 2026

SEVI: Safe Evidential Real-World Reinforcement Learning

Authors: Shaekh Mohammad Shithil, Lantao Liu

Organizations: Luddy School of Informatics, Computing, and Engineering, Indiana University, Bloomington, IN 47408 USA

Abstract

Real-world reinforcement learning has enabled contact-rich manipulation tasks to be trained directly on physical robots, but unsafe interactions during online learning remain a key barrier to practical deployment. We propose Safe Evidential Real-World Reinforcement Learning (SEVI), a framework for reliability-aware safety intervention during real-world policy learning. SEVI introduces an Evidential Safety Critic that estimates both future constraint-violation risk and epistemic uncertainty in a single forward pass. During online learning, SEVI evaluates each task-policy action before execution; if the action is predicted to be unsafe or the prediction is insufficiently supported by evidence, SEVI overrides it with an offline-trained recovery policy that guides the robot toward a lower-risk region. We evaluate SEVI on two simulated manipulation tasks and a real-world USB insertion task under in-distribution and distribution-shifted conditions. Across task-learning and safety-critic baselines, SEVI achieves the strongest trajectory-level safety prediction, the highest final task success, and the lowest unsafe-event counts. SEVI also reduces unsafe events by 13.4 to 57.1 percent relative to recovery-based safe RL while preserving low-latency execution-time filtering. The project page is available at: https://sevi-safe.github.io/

Figures & tables

Fig. 1: Illustration of SEVI on the real-world USB insertion task. When the task policy attempts an unsafe insertion, SEVI detects elevated safety risk or epistemic uncertainty and triggers the recovery policy. The recovery policy steers the robot toward a safer region before returning control to the task policy for continued online RL training.

Fig. 2: Overview of the SEVI framework . In offline training, failure and recovery demonstrations are used to train the Evidential Safety Critic and recovery policy. During the runtime pipeline, the ESC evaluates task-policy actions using predicted risk and epistemic uncertainty and switches to the recovery policy when needed. In online RL, RLPD updates the task policy using executed transitions from the RL buffer together with prior demonstrations from the demo buffer.

Fig. 3: Block Plucking example of SEVI task–recovery switching. An unsafe $\pi^{\mathrm{task}}$ action that could destabilize the stack (red) triggers $\pi^{\mathrm{rec}}$ , which returns the robot to a safer configuration before task execution resumes successfully (green).

Panel of Figure 4 — Fig. 4: Characterization of the Evidential Safety Critic. (a) Epistemic uncertainty under increasing training coverage in a fixed USB failure region; shading denotes $\pm 1$ standard deviation across three ESC trainings. (b) Detection efficiency of safety critics in simulation and hardware.

Task	Metric	SQRL	Dropout	Ensemble	SEVI(w/o Unc.)	SEVI
Block ID	BA $\uparrow$	$.85{\pm}.01$	$.82{\pm}.03$	$.90{\pm}.03$	$.83{\pm}.04$	$\mathbf{.93{\pm}.03}$
	TPR $\uparrow$	$.77{\pm}.03$	$.74{\pm}.04$	$.86{\pm}.04$	$.80{\pm}.05$	$\mathbf{.92{\pm}.03}$
	TNR $\uparrow$	$.93{\pm}.03$	$.90{\pm}.03$	$\mathbf{.95{\pm}.02}$	$.86{\pm}.04$	$.94{\pm}.04$
Block OOD	BA $\uparrow$	$.66{\pm}.03$	$.69{\pm}.01$	$.74{\pm}.05$	$.66{\pm}.03$	$\mathbf{.80{\pm}.02}$
	TPR $\uparrow$	$.61{\pm}.03$	$.54{\pm}.03$	$.63{\pm}.07$	$.61{\pm}.04$	$\mathbf{.73{\pm}.03}$
	TNR $\uparrow$	$.71{\pm}.03$	$.84{\pm}.03$	$.85{\pm}.04$	$.72{\pm}.03$	$\mathbf{.88{\pm}.03}$

TABLE I: Trajectory-level safety prediction under ID and OOD conditions. SEVI achieves the highest balanced accuracy across tasks with consistent TPR–TNR balance.

	Simulation (Block Plucking)		Hardware (USB Insert)
Method	DT $\downarrow$	Lat. (ms) $\downarrow$	DT $\downarrow$	Lat. (ms) $\downarrow$
SQRL	$0.37{\pm}0.03$	$1.00{\pm}0.08$	$0.28{\pm}0.02$	$2.40{\pm}0.15$
Dropout	$0.43{\pm}0.04$	$3.00{\pm}0.20$	$0.29{\pm}0.03$	$4.21{\pm}0.25$
Ensemble	$0.53{\pm}0.05$	$7.00{\pm}0.40$	$0.37{\pm}0.04$	$8.23{\pm}0.50$
SEVI	$\mathbf{0.29{\pm}0.02}$	$\mathbf{1.00{\pm}0.05}$	$\mathbf{0.26{\pm}0.02}$	$\mathbf{2.60{\pm}0.10}$

TABLE II: Detection time (DT) and inference latency of the evaluated safety critics corresponding to Fig. 4 (b).

Set.	Metric	Dropout	Ensemble	SEVI
ID	AUROC $\uparrow$	$0.68$	$0.88$	$\mathbf{0.92}$
	APR $\uparrow$	$0.74$	$0.87$	$\mathbf{0.96}$
	ECE $\downarrow$	$0.29$	$0.38$	$\mathbf{0.07}$
	Brier $\downarrow$	$0.29$	$0.28$	$\mathbf{0.04}$
OOD	AUROC $\uparrow$	$0.62$	$0.89$	$\mathbf{0.94}$
	APR $\uparrow$	$0.25$	$0.21$	$\mathbf{0.39}$

TABLE III: Uncertainty estimation quality on the hardware task under ID and OOD conditions. SEVI provides stronger discrimination and calibration than baselines.

Fig. 5: Effect of recovery-gating criteria on closed-loop execution. Combining risk and uncertainty gives the best safety–performance trade-off across tasks.

Fig. 6: Learning curves from a single representative hardware training run on the USB insertion task, showing success rate, intervention rate, and cycle time for SQRL and SEVI during online training. SEVI reaches higher success while progressively reducing intervention rate and cycle time.

Recovery Behavior	Gating	Success $\uparrow$	Unsafe $\downarrow$
Hit-and-Recover	Risk only	$0.64\pm 0.03$	$32\pm 4$
Hit-and-Recover	Risk+Unc.	$\mathbf{0.82\pm 0.03}$	$\mathbf{16\pm 4}$
Stuck-and-Recover	Risk only	$0.64\pm 0.01$	$33\pm 4$
Stuck-and-Recover	Risk+Unc.	$\mathbf{0.83\pm 0.03}$	$\mathbf{15\pm 5}$

TABLE IV: Effect of recovery behavior on USB Insertion task.

Method	Block Plucking		Transport Nut		USB Insertion
Method	Succ. $\uparrow$	Unsafe $\downarrow$	Succ. $\uparrow$	Unsafe $\downarrow$	Succ. $\uparrow$	Unsafe $\downarrow$
SERL [ 2 ]	$0.67{\pm}.02$	$270{\pm}7$	$0.52{\pm}.01$	$450{\pm}11$	$0.71{\pm}.03$	$56{\pm}7$
SAC [ 15 ]	$0.63{\pm}.01$	$298{\pm}5$	$0.49{\pm}.01$	$467{\pm}9$	$0.61{\pm}.03$	$49{\pm}4$
SAC-C [ 10 ]	$0.71{\pm}.03$	$281{\pm}3$	$0.55{\pm}.02$	$410{\pm}7$	$0.70{\pm}.04$	$37{\pm}6$
SQRL [ 4 ]	$0.75{\pm}.02$	$220{\pm}5$	$0.61{\pm}.01$	$358{\pm}5$	$0.68{\pm}.04$	$35{\pm}3$
SEVI(w/o Unc.)	$0.77{\pm}.03$	$234{\pm}5$	$0.63{\pm}.03$	$360{\pm}7$	$0.66{\pm}.04$	$32{\pm}5$

TABLE V: Closed-loop task execution performance. Success is evaluated using the final policy, while unsafe events are counted throughout training.

Task	$\boldsymbol{\tau_{\mathrm{risk}}}$	$\boldsymbol{\tau_{\mathrm{epi}}}$	$\boldsymbol{\Delta}$ Success
Block Plucking	$0.44$	$0.56$	$<8\%$
Transport Nut	$0.61$	$0.64$	$<9\%$
USB Insertion	$0.65$	$0.45$	$<5\%$

TABLE VI: Task-specific recovery-gating thresholds and sensitivity to

\pm 20\%

threshold perturbation.

Hyperparameter	Value
Critic architecture	Twin critic
Visual encoder	Frozen ResNet-v1-10
Camera views	Front + wrist RGB
Hidden layers	$[256,256]$
Activation	Swish
Normalization	LayerNorm

TABLE VII: Evidential Safety Critic hyperparameters.

Hyperparameter	Value
Algorithm	SERL/RLPD
Actor hidden layers	$[256,256]$
Critic hidden layers	$[256,256]$
Actor learning rate	$3\times 10^{-4}$
Critic learning rate	$3\times 10^{-4}$
Replay buffer size	$200$ k transitions

TABLE VIII: Task policy hyperparameters.

Hyperparameter	Value
Algorithm	TD3+BC
Actor hidden layers	$[256,256]$
Critic hidden layers	$[256,256]$
Actor learning rate	$1\times 10^{-4}$
Critic learning rate	$1\times 10^{-4}$
Discount factor $\gamma$	$0.95$

TABLE IX: Recovery policy hyperparameters.

Method	Success $\uparrow$	Unsafe $\downarrow$
SERL	$0.71\pm 0.03$	$56\pm 7$
SAC	$0.61\pm 0.03$	$49\pm 4$
SAC-C	$0.70\pm 0.04$	$37\pm 6$
SQRL	$0.68\pm 0.04$	$35\pm 3$
SEVI (w/o Unc.)	$0.66\pm 0.04$	$32\pm 5$
SEVI	$\mathbf{0.83\pm 0.03}$	$\mathbf{15\pm 4}$

TABLE X: Repeated hardware evaluation on the USB Insertion task. Success is evaluated using the final policy, while unsafe events are counted throughout online training. Results are mean

\pm

standard deviation over three independent 3-hour hardware training runs.

Recovery Behavior	Gating	Success $\uparrow$	Unsafe $\downarrow$
Hit-and-Recover	Risk only	$0.64\pm 0.03$	$32\pm 4$
	Risk + Unc.	$\mathbf{0.82\pm 0.03}$	$\mathbf{16\pm 4}$
Stuck-and-Recover	Risk only	$0.64\pm 0.01$	$33\pm 4$
	Risk + Unc.	$\mathbf{0.83\pm 0.03}$	$\mathbf{15\pm 5}$

TABLE XI: Closed-loop performance under two qualitatively distinct recovery behaviors on the USB Insertion task. Results are mean

\pm

standard deviation over three independent evaluation runs.

Safe RL

Safe reinforcement learning studies how agents can learn and act while respecting safety constraints and limiting harmful outcomes.

See all 217 papers

Provably Safe Sim-to-Real Transfer
Sep 1, 2026Tingting Ni, Maryam KamgarpourReinforcement Learning Sim-to-Real Transfer
SYCAMORE, EPFL
Sampling-Based Safe Reinforcement Learning
May 19, 2026Luca Vignola, Bruce D. Lee, Manish Prajapat +4Robot Safety Safe RL
ETH Zurich
SafeVLA-Bench: A Benchmark for the Success-Safety Gap in Vision-Language-Action Models
May 30, 2026Jialiang Fan, Weizhe Xu, Zijun Wang +3Robot Safety Robotic Manipulation
University of Notre Dame · University of Pennsylvania

Sep 1, 2026J/K move · Enter open · S save

Provably Safe Sim-to-Real Transfer

Tingting Ni, Maryam Kamgarpour

SYCAMORE, EPFL

Reinforcement Learning Sim-to-Real Transfer Reward-Free RL Safe RL Sample-Efficient RL

Open paper AskPDF arXiv

We address safe sim-to-real transfer, in which an agent leverages an imperfect simulator and limited real-world interaction while ensuring safety throughout data collection in the real system. This problem arises in applications such as robotics and healthcare: simulators provide cheap data, but sim-to-real mismatch makes direct transfer unreliable, and collecting real-world data to correct this mismatch must itself be safe. Moreover, deployment objectives may vary across tasks, making it costly to collect new data for each reward function. We therefore formulate safe sim-to-real transfer as a reward-free safe reinforcement learning (RL) problem, in which data are collected once and reused to plan for arbitrary reward functions. We develop a computationally efficient algorithm that identifies where the simulator and real dynamics differ, uses certified simulator transitions where they are reliable, and estimates mismatched transitions from safely collected data. With high probability, every policy deployed during learning is feasible, and the collected data support the computation of a feasible and near-optimal policy for any reward function. When the simulator is uninformative, our algorithm recovers online reward-free safe RL while improving the best-known sample complexity by a factor of

\widetildeΘ(H/ξ^2)

, where

ξ

is the safety margin of a baseline policy. When the simulator is accurate on most transitions, this improvement grows to

\widetildeΘ(H^2|\mc S||\mc A|/(ξ^2|\mc B|))

, where

|\mc B|

denotes the size of the sim-to-real mismatch region.

Abstract

Safe RL

Explore similar work

Provably Safe Sim-to-Real Transfer

Sampling-Based Safe Reinforcement Learning

SafeVLA-Bench: A Benchmark for the Success-Safety Gap in Vision-Language-Action Models