RECREG: Recovery-based Shielding for Regular Safety Properties in Reinforcement Learning

Alexander W. Goodall; Francesco Belardinelli

cs.LGOct 16, 2026

RECREG: Recovery-based Shielding for Regular Safety Properties in Reinforcement Learning

Authors: Alexander W. Goodall, Francesco Belardinelli

Organizations: Department of Computer Science Imperial College London

Abstract

In many real-world scenarios we need to consider formal requirements, such as safety and constraint compliance, to deploy Reinforcement Learning (RL) agents. In this paper we introduce RecReg, a safe RL framework capable of enforcing non-Markovian regular safety properties via recovery-based shielding. RecReg is a unified approach that supports different levels of a priori knowledge and safety guarantees. The paper primarily focuses on building the theoretical foundations for RecReg in the tabular setting, providing rigorous safety and convergence guarantees that are not present in related work. We provide an extensive set of empirical results using the MASA-Safe-RL library, and we outline an extension of RecReg to deep RL domains using SOTA model-based RL. Our empirical results demonstrate that RecReg achieves efficient learning and matches the theory in this paper.

Figures & tables

Figure 1: Experimental results for the MASA-Safe-RL [ 19 ] benchmark. Normalized returns and safety probabilities are averaged over 10 random seeds for the final 100 episodes in training. The dashed horizontal line in the safety probability plots denotes the theoretical safety probability given by Theorem 1 . Scores aggregated across all environments are provided in the bottom right corner.

Algorithm	Metric	MediaStreaming	BridgeCrossing	ColourBomb…V2 (Prop. 3)	MiniPacman
Algorithm	Metric	$1-\epsilon=0.96$	$1-\epsilon=0.95$	$1-\epsilon=0.99$	$1-\epsilon=0.99$
RecReg-Exact	Ep. Return	$-21.355\pm 0.240$	$\mathbf{1.000\pm 0.000}$	$\mathbf{26.294\pm 0.359}$	$\mathbf{0.995\pm 0.00}1$
RecReg-Exact	Safety Prob.	$0.951\pm 0.009$	$1.000\pm 0.000$	$0.997\pm 0.002$	$0.997\pm 0.002$
ProbShield	Ep. Return	$\mathbf{-25.964\pm 1.040}$	$0.933\pm 0.067$	$19.046\pm 1.490$	$0.966\pm 0.006$
ProbShield	Safety Prob.	$0.990\pm 0.005$	$0.978\pm 0.011$	$0.985\pm 0.006$	$0.993\pm 0.003$

Table 1: Experimental results for ProbShield and RecReg-Exact . Bold text denotes the best return; grey text denotes algorithms that fail the safety threshold.

Figure 2: Learning curves for Atari Seaquest. Episodic reward and violation rate are averaged over 5 random seeds.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Estimating the safety probability with RSSM.

Panel of Figure 5 — Figure 5: Safety Gym vehicles.

Panel of Figure 6 — Figure 6: Safety Gym tasks and constraints.

		DreamerV3	Aug. Lag.	AMBS	Deep-RecReg	PPO-Lag	CPO
seaquest ( $\varphi_{1}$ )	Return	$2440\pm 400$	$706\pm 162$	$2350\pm 340$	$\bm{3140\pm 326}$	$506\pm 7.57$	$543\pm 47.4$
	Viol. rate	$5.30\pm 0.657$	$9.83\pm 1.39$	$\bm{4.20\pm 0.215}$	$4.43\pm 0.187$	$25.6\pm 0.468$	$23.9\pm 1.97$
seaquest ( $\varphi_{2}$ )	Return	$2440\pm 400$	$580\pm 54.0$	$2480\pm 483$	$\bm{2700\pm 364}$	$79.6\pm 10.5$	$342\pm 17.1$
	Viol. rate	$17.9\pm 5.77$	$\bm{1.86\pm 0.891}$	$2.00\pm 0.767$	$2.52\pm 0.673$	$16.8\pm 2.17$	$22.0\pm 4.45$

Appendix

Table 2: Results for the deep RL experiments on Atari Seaquest. The violation rate is scaled by

10^{-4}

Figure 7: Learning curves for Atari Seaquest. Episodic reward and violation rate are averaged over 5 random seeds.

		DreamerV3	Aug. Lag.	AMBS	Deep-RecReg
PointGoal1	Return	$\bm{22.2\pm 0.629}$	$19.1\pm 0.661$	$16.7\pm 2.58$	$17.3\pm 0.628$
	Viol. rate	$0.0562\pm 0.00239$	$0.0111\pm 0.00867$	$\bm{0.00383\pm 0.00346}$	$0.00500\pm 0.00237$
PointGoal2	Return	$\bm{20.6\pm 1.22}$	$12.7\pm 1.58$	$6.42\pm 1.04$	$7.88\pm 2.08$
	Viol. rate	$0.188\pm 0.0177$	$0.0337\pm 0.0112$	$\bm{0.0126\pm 0.00909}$	$0.0213\pm 0.0179$
CarGoal1	Return	$\bm{17.4\pm 2.13}$	$11.2\pm 2.96$	$7.26\pm 3.52$	$7.56\pm 3.23$
	Viol. rate	$0.0836\pm 0.0101$	$0.0444\pm 0.0308$	$0.00552\pm 0.00393$	$\bm{0.00367\pm 0.00415}$

Appendix

Table 3: Results for the deep RL experiments on SafetyGym .

Figure 8: Learning curves for Safety Gym environments. Episodic reward and violation rate are averaged over 5 random seeds.

Name	Symbol	Value
Replay capacity	$D$	$10^{6}$
Batch size	$B$	16
Batch length	-	64
Number of envs	-	8
Train ratio	-	64
Number of MLP layers	-	5

Appendix

Table 4: Hyperparameter details for the DreamerV3-based algorithms: Aug. Lag., AMBS and Deep-RecReg .

Algorithm	Metric	MediaStreaming	BridgeCrossing	BridgeCrossingV2	Colour…World
Algorithm	Metric	$1-\epsilon=0.96$	$1-\epsilon=0.95$	$1-\epsilon=0.99$	$1-\epsilon=0.95$
QL	Ep. Return	$-0.017\pm 0.009$	$0.303\pm 0.287$	$0.986\pm 0.004$	$1.000\pm 0.000$
QL	Safety Prob.	$0.000\pm 0.000$	$0.368\pm 0.260$	$0.986\pm 0.004$	$0.000\pm 0.000$
QL( $\lambda$ )	Ep. Return	$-22.291\pm 0.377$	$\mathbf{0.998\pm 0.001}$	$0.414\pm 0.258$	$\mathbf{1.000\pm 0.000}$
QL( $\lambda$ )	Safety Prob.	$0.975\pm 0.011$	$0.998\pm 0.001$	$0.985\pm 0.010$	$0.998\pm 0.001$
RecReg	Ep. Return	$\mathbf{-21.726\pm 0.234}$	$\mathbf{0.998\pm 0.002}$	$\mathbf{0.665\pm 0.272}$	$0.911\pm 0.174$

Appendix

Table 5: Experimental results with raw returns . Bold text denotes the best value among algorithms satisfying the safety threshold; grey text denotes algorithms that fail the safety threshold.

Algorithm	Metric	ColourBomb…(Prop. 2)	ColourBomb…V2	ColourBomb…V2 (Prop. 2)	Colour…V2 (Prop. 3)	MiniPacman
Algorithm	Metric	$1-\epsilon=0.95$	$1-\epsilon=0.99$	$1-\epsilon=0.99$	$1-\epsilon=0.99$	$1-\epsilon=0.99$
QL	$\ldots$	$1.000\pm 0.000$	$32.701\pm 0.599$	$32.701\pm 0.599$	$32.701\pm 0.599$	$0.989\pm 0.003$
QL	$\ldots$	$0.076\pm 0.007$	$0.000\pm 0.000$	$0.000\pm 0.000$	$0.000\pm 0.000$	$0.396\pm 0.018$
QL( $\lambda$ )	$\ldots$	$0.996\pm 0.002$	$20.484\pm 2.408$	$22.663\pm 1.379$	$22.394\pm 0.364$	$0.925\pm 0.009$
QL( $\lambda$ )	$\ldots$	$0.996\pm 0.002$	$0.877\pm 0.048$	$0.949\pm 0.039$	$0.947\pm 0.013$	$0.883\pm 0.007$
RecReg	$\ldots$	$0.935\pm 0.126$	$21.358\pm 0.687$	$21.952\pm 1.277$	$26.466\pm 0.095$	$0.912\pm 0.012$

Appendix

Table 6: Experimental results with normalized returns . Bold text denotes the best value among algorithms satisfying the safety threshold; grey text denotes algorithms that fail the safety threshold.

Figure 9: Learning curves for Media Streaming.

Figure 10: Learning curves for Bridge Crossing.

Figure 11: Learning curves for Bridge Crossing V2.

Figure 12: Learning curves for Colour Bomb Gridworld (Property 1).

Figure 13: Learning curves for Colour Bomb Gridworld (Property 2).

Figure 14: Learning curves for Colour Bomb Gridworld V2 (Property 1).

Figure 15: Learning curves for Colour Bomb Gridworld V2 (Property 2).

Figure 16: Learning curves for Colour Bomb Gridworld V2 (Property 3).

Figure 17: Learning curves for Mini Pacman.

Figure 18: Learning curves for Media Streaming.

Figure 19: Learning curves for Bridge Crossing.

Figure 20: Learning curves for Colour Bomb Gridworld V2 (Property 3).

Figure 21: Learning curves for Mini Pacman.

Figure 22: Reward Shaping for Colour Bomb Gridworld V2 with RecReg-ModelFree .

Figure 23: Reward Shaping for Colour Bomb Gridworld V2 with QL- $\lambda$ .

	Exact	ModelBased	ModelBased
	& ModelFree	(exact)	(stat. $m$ =10,000)
media_streaming	$\sim 2600$ fps	$\sim 1300$ fps	$\sim 90$ fps
bridge_crossing	$\sim 2200$ fps	$\sim 1000$ fps	$\sim 80$ fps
$15\times 15$ _gridworld (3)	$\sim 2000$ fps	$\sim 525$ fps	$\sim 60$ fps
mini_pacman	$\sim 2600$ fps	$\sim 250$ fps	$\sim 27$ fps

Appendix

Table 7: Frames per second (fps) for different RecReg modes.

Figure 24: The Bridge Crossing environment.

Figure 25: The Bridge Crossing V2 environment.

Figure 26: The Colour Bomb Gridworld environment.

Figure 27: The Colour Bomb Gridworld V2 environment

Q-Learning (shared)
Name	Symbol	Value
Learning rate	$\alpha$	$\{0.05,0.1\}$
Discount factor	$\gamma$	$\{0.95,0.99\}$
Exploration type	-	$\{\texttt{Boltzmann},\texttt{Epsilon-greedy}\}$
Boltzmann Temp.	$\tau$	$0.05$
Epsilon start	$\epsilon_{\text{init}}$	1.0

Appendix

Table 8: Hyperparameter details for Q-Learning based algorithms: RecReg Rec. RL, QL-

\lambda

, SEM, LCRL.

PPO-Lag, TRPO-Lag, CPO shared (Tabular)
Name	Symbol	Value
Actor learning rate (PPO-Lag)	$\eta$	$3\times 10^{-4}$
Critic learning rate (CPO/TRPO-Lag)	$\eta_{v}$	$1\times 10^{-3}$
Discount factor	$\gamma$	$\{0.95,0.99\}$
Cost coefficient	$c$	$1.0$
Cost threshold	-	Variable (c.f., Section D )

Appendix

Table 9: Hyperparameter details for PPO-Lag, TRPO-Lag and CPO.

Name	Symbol	Value
ProbShieldWrapperDisc
Initial safety bound	$\epsilon_{\text{init}}$	0.01
Max VI Steps	-	10,000
VI Cut off	-	$1\times 10^{-10}$
Disc. Granularity	-	20
PPO (underlying algorithm)

Appendix

Table 10: Hyperparameter details for ProbShield .

Safe RL

Safe reinforcement learning studies how agents can learn and act while respecting safety constraints and limiting harmful outcomes.

See all 209 papers

Robust Shielding for Safe Reinforcement Learning
May 29, 2026Edwin Hamel-De le Court, Thom Badings, Alessandro Abate +2Robust Markov Decision Processes Formal Verification
Department of Computer Science, University of Manchester, United Kingdom · Faculty of Computer Science & DSME, RWTH Aachen University, Germany · Department of Computer Science, University of Oxford, United Kingdom +1
Sampling-Based Safe Reinforcement Learning
May 19, 2026Luca Vignola, Bruce D. Lee, Manish Prajapat +4Robot Safety Safe RL
ETH Zurich
Robust Probabilistic Shielding for Safe Offline Reinforcement Learning
May 11, 2026Maris F. L. Galesloot, Thomas Rhemrev, Nils JansenOffline RL Safe RL
Radboud University · Nijmegen, The Netherlands · Ruhr University & Radboud University +1

May 29, 2026J/K move · Enter open · S save

Robust Shielding for Safe Reinforcement Learning

Edwin Hamel-De le Court, Thom Badings, Alessandro Abate, Francesco Belardinelli+1

Department of Computer Science, University of Manchester, United Kingdom · Faculty of Computer Science & DSME, RWTH Aachen University, Germany · Department of Computer Science, University of Oxford, United Kingdom+1

Robust Markov Decision Processes Formal Verification Safe RL Robust RL

Open paper AskPDF arXiv

Shielding is an effective approach to formally guarantee the safety of reinforcement learning agents in Markov decision processes (MDPs). However, existing shielding techniques typically assume knowledge of the safety-relevant transition dynamics - a requirement that is seldom met in practice. To address this limitation, we introduce a novel shielding framework for robust MDPs (RMDPs), i.e., MDPs with sets of transition probabilities. We define safety as the satisfaction of a linear temporal logic (LTL) formula with a certain threshold probability under the worst-case transition probabilities of the RMDP. We prove that our shielding framework is both sound and optimal for the RMDP: every policy admissible by the shield is safe, and conversely, every safe RMDP policy is admissible by the shield. We combine our approach with existing sampling methods for learning transition probabilities of MDPs with probably approximately correct (PAC) guarantees. This combination enables the construction of shields for MDPs that, with high confidence, guarantee safety while remaining minimally restrictive. Our experiments show that our shields for learned RMDPs guarantee safety in unknown MDPs while recovering strong expected return as the number of samples increases.

Abstract

Safe RL

Explore similar work

Robust Shielding for Safe Reinforcement Learning

Sampling-Based Safe Reinforcement Learning

Robust Probabilistic Shielding for Safe Offline Reinforcement Learning