RECREG: Recovery-based Shielding for Regular Safety Properties in Reinforcement Learning

Alexander W. Goodall; Francesco Belardinelli

cs.LGOct 16, 2026

RECREG: Recovery-based Shielding for Regular Safety Properties in Reinforcement Learning

Authors: Alexander W. Goodall, Francesco Belardinelli

Organizations: Department of Computer Science Imperial College London

Abstract

In many real-world scenarios we need to consider formal requirements, such as safety and constraint compliance, to deploy Reinforcement Learning (RL) agents. In this paper we introduce RecReg, a safe RL framework capable of enforcing non-Markovian regular safety properties via recovery-based shielding. RecReg is a unified approach that supports different levels of a priori knowledge and safety guarantees. The paper primarily focuses on building the theoretical foundations for RecReg in the tabular setting, providing rigorous safety and convergence guarantees that are not present in related work. We provide an extensive set of empirical results using the MASA-Safe-RL library, and we outline an extension of RecReg to deep RL domains using SOTA model-based RL. Our empirical results demonstrate that RecReg achieves efficient learning and matches the theory in this paper.

Figures & tables

Figure 1: Experimental results for the MASA-Safe-RL [ 19 ] benchmark. Normalized returns and safety probabilities are averaged over 10 random seeds for the final 100 episodes in training. The dashed horizontal line in the safety probability plots denotes the theoretical safety probability given by Theorem 1 . Scores aggregated across all environments are provided in the bottom right corner.

Algorithm	Metric	MediaStreaming	BridgeCrossing	ColourBomb…V2 (Prop. 3)	MiniPacman
Algorithm	Metric	$1-\epsilon=0.96$	$1-\epsilon=0.95$	$1-\epsilon=0.99$	$1-\epsilon=0.99$
RecReg-Exact	Ep. Return	$-21.355\pm 0.240$	$\mathbf{1.000\pm 0.000}$	$\mathbf{26.294\pm 0.359}$	$\mathbf{0.995\pm 0.00}1$
RecReg-Exact	Safety Prob.	$0.951\pm 0.009$	$1.000\pm 0.000$	$0.997\pm 0.002$	$0.997\pm 0.002$
ProbShield	Ep. Return	$\mathbf{-25.964\pm 1.040}$	$0.933\pm 0.067$	$19.046\pm 1.490$	$0.966\pm 0.006$
ProbShield	Safety Prob.	$0.990\pm 0.005$	$0.978\pm 0.011$	$0.985\pm 0.006$	$0.993\pm 0.003$

Table 1: Experimental results for ProbShield and RecReg-Exact . Bold text denotes the best return; grey text denotes algorithms that fail the safety threshold.

Figure 2: Learning curves for Atari Seaquest. Episodic reward and violation rate are averaged over 5 random seeds.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Estimating the safety probability with RSSM.

Panel of Figure 5 — Figure 5: Safety Gym vehicles.

Panel of Figure 6 — Figure 6: Safety Gym tasks and constraints.

		DreamerV3	Aug. Lag.	AMBS	Deep-RecReg	PPO-Lag	CPO
seaquest ( $\varphi_{1}$ )	Return	$2440\pm 400$	$706\pm 162$	$2350\pm 340$	$\bm{3140\pm 326}$	$506\pm 7.57$	$543\pm 47.4$
	Viol. rate	$5.30\pm 0.657$	$9.83\pm 1.39$	$\bm{4.20\pm 0.215}$	$4.43\pm 0.187$	$25.6\pm 0.468$	$23.9\pm 1.97$
seaquest ( $\varphi_{2}$ )	Return	$2440\pm 400$	$580\pm 54.0$	$2480\pm 483$	$\bm{2700\pm 364}$	$79.6\pm 10.5$	$342\pm 17.1$
	Viol. rate	$17.9\pm 5.77$	$\bm{1.86\pm 0.891}$	$2.00\pm 0.767$	$2.52\pm 0.673$	$16.8\pm 2.17$	$22.0\pm 4.45$

Appendix

Table 2: Results for the deep RL experiments on Atari Seaquest. The violation rate is scaled by

10^{-4}

Figure 7: Learning curves for Atari Seaquest. Episodic reward and violation rate are averaged over 5 random seeds.

		DreamerV3	Aug. Lag.	AMBS	Deep-RecReg
PointGoal1	Return	$\bm{22.2\pm 0.629}$	$19.1\pm 0.661$	$16.7\pm 2.58$	$17.3\pm 0.628$
	Viol. rate	$0.0562\pm 0.00239$	$0.0111\pm 0.00867$	$\bm{0.00383\pm 0.00346}$	$0.00500\pm 0.00237$
PointGoal2	Return	$\bm{20.6\pm 1.22}$	$12.7\pm 1.58$	$6.42\pm 1.04$	$7.88\pm 2.08$
	Viol. rate	$0.188\pm 0.0177$	$0.0337\pm 0.0112$	$\bm{0.0126\pm 0.00909}$	$0.0213\pm 0.0179$
CarGoal1	Return	$\bm{17.4\pm 2.13}$	$11.2\pm 2.96$	$7.26\pm 3.52$	$7.56\pm 3.23$
	Viol. rate	$0.0836\pm 0.0101$	$0.0444\pm 0.0308$	$0.00552\pm 0.00393$	$\bm{0.00367\pm 0.00415}$

Appendix

Table 3: Results for the deep RL experiments on SafetyGym .

Figure 8: Learning curves for Safety Gym environments. Episodic reward and violation rate are averaged over 5 random seeds.

Name	Symbol	Value
Replay capacity	$D$	$10^{6}$
Batch size	$B$	16
Batch length	-	64
Number of envs	-	8
Train ratio	-	64
Number of MLP layers	-	5

Appendix

Table 4: Hyperparameter details for the DreamerV3-based algorithms: Aug. Lag., AMBS and Deep-RecReg .

Algorithm	Metric	MediaStreaming	BridgeCrossing	BridgeCrossingV2	Colour…World
Algorithm	Metric	$1-\epsilon=0.96$	$1-\epsilon=0.95$	$1-\epsilon=0.99$	$1-\epsilon=0.95$
QL	Ep. Return	$-0.017\pm 0.009$	$0.303\pm 0.287$	$0.986\pm 0.004$	$1.000\pm 0.000$
QL	Safety Prob.	$0.000\pm 0.000$	$0.368\pm 0.260$	$0.986\pm 0.004$	$0.000\pm 0.000$
QL( $\lambda$ )	Ep. Return	$-22.291\pm 0.377$	$\mathbf{0.998\pm 0.001}$	$0.414\pm 0.258$	$\mathbf{1.000\pm 0.000}$
QL( $\lambda$ )	Safety Prob.	$0.975\pm 0.011$	$0.998\pm 0.001$	$0.985\pm 0.010$	$0.998\pm 0.001$
RecReg	Ep. Return	$\mathbf{-21.726\pm 0.234}$	$\mathbf{0.998\pm 0.002}$	$\mathbf{0.665\pm 0.272}$	$0.911\pm 0.174$

Appendix

Table 5: Experimental results with raw returns . Bold text denotes the best value among algorithms satisfying the safety threshold; grey text denotes algorithms that fail the safety threshold.

Algorithm	Metric	ColourBomb…(Prop. 2)	ColourBomb…V2	ColourBomb…V2 (Prop. 2)	Colour…V2 (Prop. 3)	MiniPacman
Algorithm	Metric	$1-\epsilon=0.95$	$1-\epsilon=0.99$	$1-\epsilon=0.99$	$1-\epsilon=0.99$	$1-\epsilon=0.99$
QL	$\ldots$	$1.000\pm 0.000$	$32.701\pm 0.599$	$32.701\pm 0.599$	$32.701\pm 0.599$	$0.989\pm 0.003$
QL	$\ldots$	$0.076\pm 0.007$	$0.000\pm 0.000$	$0.000\pm 0.000$	$0.000\pm 0.000$	$0.396\pm 0.018$
QL( $\lambda$ )	$\ldots$	$0.996\pm 0.002$	$20.484\pm 2.408$	$22.663\pm 1.379$	$22.394\pm 0.364$	$0.925\pm 0.009$
QL( $\lambda$ )	$\ldots$	$0.996\pm 0.002$	$0.877\pm 0.048$	$0.949\pm 0.039$	$0.947\pm 0.013$	$0.883\pm 0.007$
RecReg	$\ldots$	$0.935\pm 0.126$	$21.358\pm 0.687$	$21.952\pm 1.277$	$26.466\pm 0.095$	$0.912\pm 0.012$

Appendix

Table 6: Experimental results with normalized returns . Bold text denotes the best value among algorithms satisfying the safety threshold; grey text denotes algorithms that fail the safety threshold.

Figure 9: Learning curves for Media Streaming.

Figure 10: Learning curves for Bridge Crossing.

Figure 11: Learning curves for Bridge Crossing V2.

Figure 12: Learning curves for Colour Bomb Gridworld (Property 1).

Figure 13: Learning curves for Colour Bomb Gridworld (Property 2).

Figure 14: Learning curves for Colour Bomb Gridworld V2 (Property 1).

Figure 15: Learning curves for Colour Bomb Gridworld V2 (Property 2).

Figure 16: Learning curves for Colour Bomb Gridworld V2 (Property 3).

Figure 17: Learning curves for Mini Pacman.

Figure 18: Learning curves for Media Streaming.

Figure 19: Learning curves for Bridge Crossing.

Figure 20: Learning curves for Colour Bomb Gridworld V2 (Property 3).

Figure 21: Learning curves for Mini Pacman.

Figure 22: Reward Shaping for Colour Bomb Gridworld V2 with RecReg-ModelFree .

Figure 23: Reward Shaping for Colour Bomb Gridworld V2 with QL- $\lambda$ .

	Exact	ModelBased	ModelBased
	& ModelFree	(exact)	(stat. $m$ =10,000)
media_streaming	$\sim 2600$ fps	$\sim 1300$ fps	$\sim 90$ fps
bridge_crossing	$\sim 2200$ fps	$\sim 1000$ fps	$\sim 80$ fps
$15\times 15$ _gridworld (3)	$\sim 2000$ fps	$\sim 525$ fps	$\sim 60$ fps
mini_pacman	$\sim 2600$ fps	$\sim 250$ fps	$\sim 27$ fps

Appendix

Table 7: Frames per second (fps) for different RecReg modes.

Figure 24: The Bridge Crossing environment.

Figure 25: The Bridge Crossing V2 environment.

Figure 26: The Colour Bomb Gridworld environment.

Figure 27: The Colour Bomb Gridworld V2 environment

Q-Learning (shared)
Name	Symbol	Value
Learning rate	$\alpha$	$\{0.05,0.1\}$
Discount factor	$\gamma$	$\{0.95,0.99\}$
Exploration type	-	$\{\texttt{Boltzmann},\texttt{Epsilon-greedy}\}$
Boltzmann Temp.	$\tau$	$0.05$
Epsilon start	$\epsilon_{\text{init}}$	1.0

Appendix

Table 8: Hyperparameter details for Q-Learning based algorithms: RecReg Rec. RL, QL-

\lambda

, SEM, LCRL.

PPO-Lag, TRPO-Lag, CPO shared (Tabular)
Name	Symbol	Value
Actor learning rate (PPO-Lag)	$\eta$	$3\times 10^{-4}$
Critic learning rate (CPO/TRPO-Lag)	$\eta_{v}$	$1\times 10^{-3}$
Discount factor	$\gamma$	$\{0.95,0.99\}$
Cost coefficient	$c$	$1.0$
Cost threshold	-	Variable (c.f., Section D )

Appendix

Table 9: Hyperparameter details for PPO-Lag, TRPO-Lag and CPO.

Name	Symbol	Value
ProbShieldWrapperDisc
Initial safety bound	$\epsilon_{\text{init}}$	0.01
Max VI Steps	-	10,000
VI Cut off	-	$1\times 10^{-10}$
Disc. Granularity	-	20
PPO (underlying algorithm)

Appendix

Table 10: Hyperparameter details for ProbShield .

Safe RL

Safe reinforcement learning studies how agents can learn and act while respecting safety constraints and limiting harmful outcomes.

See all 209 papers

May 29, 2026

Robust Shielding for Safe Reinforcement Learning

Shielding is an effective approach to formally guarantee the safety of reinforcement learning agents in Markov decision processes (MDPs). However, existing shielding techniques typically assume knowledge of the safety-relevant transition dynamics - a requirement that is seldom met in practice. To address this limitation, we introduce a novel shielding framework for robust MDPs (RMDPs), i.e., MDPs with sets of transition probabilities. We define safety as the satisfaction of a linear temporal logic (LTL) formula with a certain threshold probability under the worst-case transition probabilities of the RMDP. We prove that our shielding framework is both sound and optimal for the RMDP: every policy admissible by the shield is safe, and conversely, every safe RMDP policy is admissible by the shield. We combine our approach with existing sampling methods for learning transition probabilities of MDPs with probably approximately correct (PAC) guarantees. This combination enables the construction of shields for MDPs that, with high confidence, guarantee safety while remaining minimally restrictive. Our experiments show that our shields for learned RMDPs guarantee safety in unknown MDPs while recovering strong expected return as the number of samples increases.

Edwin Hamel-De le Court, Thom Badings, Alessandro Abate +2

Department of Computer Science, University of Manchester, United Kingdom · Faculty of Computer Science & DSME, RWTH Aachen University, Germany · Department of Computer Science, University of Oxford, United Kingdom +1

Robust Markov Decision Processes Formal Verification Safe RL Robust RL

Ask PDF

May 19, 2026

Sampling-Based Safe Reinforcement Learning

Safe exploration remains a fundamental challenge in reinforcement learning (RL), limiting the deployment of RL agents in the real world. We propose Sampling-Based Safe Reinforcement Learning (SBSRL), a model-based RL algorithm that maintains safety throughout the learning process by enforcing constraints jointly across a finite set of dynamics samples. This formulation approximates an intractable worst-case optimization over uncertain dynamics and enables practical safety guarantees in continuous domains. We further introduce an exploration strategy based on constraining epistemic uncertainty, eliminating the need for explicit exploration bonuses. Under regularity conditions, we derive high-probability guarantees of safety throughout learning and a finite-time sample complexity bound for recovering a near-optimal policy. Empirically, SBSRL achieves safe and efficient exploration both in simulation and in real robotic hardware, and readily extends to practical deep-ensemble implementations that scale to high-dimensional continuous control problems.

Luca Vignola, Bruce D. Lee, Manish Prajapat +4

ETH Zurich

Robot Safety Safe RL Sample-Efficient RL Model-Based RL

Ask PDF

May 11, 2026

Robust Probabilistic Shielding for Safe Offline Reinforcement Learning

In offline reinforcement learning (RL), we learn policies from fixed datasets without environment interaction. The major challenges are to provide guarantees on the (1) performance and (2) safety of the resulting policy. A technique called safe policy improvement (SPI) provides a performance guarantee: with high probability, the new policy outperforms a given baseline policy, which is assumed to be safe. Orthogonally, in the context of safe RL, a shield provides a safety guarantee by restricting the action space to those actions that are provably safe with respect to a given safety-relevant model. We integrate these paradigms by extending shielding to offline RL, relying solely on the available dataset and knowledge of safe and unsafe states. Then, we shield the policy improvement steps, guaranteeing, with high probability, a safe policy. Experimental results demonstrate that shielded SPI outperforms its unshielded counterpart, improving both average and worst-case performance, particularly in low-data regimes.

Maris F. L. Galesloot, Thomas Rhemrev, Nils Jansen

Radboud University · Nijmegen, The Netherlands · Ruhr University & Radboud University +1

Offline RL Safe RL Robust RL

Ask PDF

Abstract

Safe RL

Explore similar work

Robust Shielding for Safe Reinforcement Learning

Sampling-Based Safe Reinforcement Learning

Robust Probabilistic Shielding for Safe Offline Reinforcement Learning