Safe Reinforcement Learning via Recovery-based Shielding with Gaussian Process Dynamics Models

Alexander W. Goodall; Francesco Belardinelli

cs.LGFeb 12, 2026

Safe Reinforcement Learning via Recovery-based Shielding with Gaussian Process Dynamics Models

Authors: Alexander W. Goodall, Francesco Belardinelli

Organizations: Imperial College London, Department of Computing London, United Kingdom

Abstract

Reinforcement learning (RL) is a powerful framework for optimal decision-making and control but often lacks provable guarantees for safety-critical applications. In this paper, we introduce a novel recovery-based shielding framework that enables safe RL with a provable safety lower bound for unknown and non-linear continuous dynamical systems. The proposed approach integrates a backup policy (shield) with the RL agent, leveraging Gaussian process (GP) based uncertainty quantification to predict potential violations of safety constraints, dynamically recovering to safe trajectories only when necessary. Experience gathered by the 'shielded' agent is used to construct the GP models, with policy optimization via internal model-based sampling - enabling unrestricted exploration and sample efficient learning, without compromising safety. Empirically our approach demonstrates strong performance and strict safety-compliance on a suite of continuous control environments.

Figures & tables

Figure 1.

99.99\%

-CI uncertainty sets for recoverable (blue) and irrecoverable (red) trajectories under the backup policy for cartpole (i)/(ii) environment.

Env.	Metric	A2C-GP-Shield	MPS	DMPS	CPO	PPO-Lag	Thm. 3.3
cartpole	Return	$\bm{200.0\pm 0.00}$	$\bm{200.0\pm 0.00}$	$\bm{200.0\pm 0.00}$	$\bm{200.0\pm 0.00}$	$188.0\pm 5.53$	$1-\epsilon=0.98$
	Safety Prob.	$\bm{1.000\pm 0.00}$	$\bm{1.000\pm 0.00}$	$\bm{1.000\pm 0.00}$	$\bm{1.000\pm 0.00}$	$0.910\pm 0.04$	$1-\epsilon=0.98$
cartpole2	Return	$30.7\pm 10.4$	$43.2\pm 11.1$	$65.1\pm 23.1$ *	$152.0\pm 31.2$	$\bm{171.0\pm 7.53}$	$1-\epsilon=0.98$
	Safety prob.	$\bm{1.000\pm 0.00}$	$\bm{1.000\pm 0.00}$	$\bm{1.000\pm 0.00}$	$0.980\pm 0.01$	$0.811\pm 0.12$	$1-\epsilon=0.98$
mountain_car	Return	$\bm{91.3\pm 1.08}$ *	$85.1\pm 8.17$	$81.2\pm 0.28$	$-30.4\pm 6.74$	$73.2\pm 18.7$	$1-\epsilon=0.9$
	Safety prob.	$\bm{1.000\pm 0.00}$	$\bm{1.000\pm 0.00}$	$\bm{1.000\pm 0.00}$	$0.995\pm 0.00$	$0.884\pm 0.05$	$1-\epsilon=0.9$

Table 1. Empirical mean return and safety probability (at the end of training), bold text denotes the best score, asterisk (*) denoted the best reward that also satisfies the safety constraint with probability 1, standard error (SE) bars (averaged over 5 independent runs) are reported. We also include the theoretical lower bound established by Thm. 3.3 (based on

\epsilon_{t}=10^{-5}

Panel of Figure 2 — Figure 2. Learning curves (log scale on the x-axis).

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3. Learning curves (reward and safety probability) for A2C-Shield and A2C-Eval.

Panel of Figure 4 — Figure 4. Overrides (per episode) for A2C-Shield

Panel of Figure 5 — Figure 5. Learning curves (reward and safety probability) for CPO and PPO-Lag.

Recovery Horizon	gaussian_approx	posterior_sampling_approx
Recovery Horizon	( $\epsilon_{t}=0.9999$ )	$K=5$	$K=10$	$K=50$	$K=100$	$K=500$	$K=1000$	$K=5000$
$N=5$	0.749	0.537	0.533	0.648	0.619	0.683	0.910	6.890
$N=10$	1.090	0.985	1.015	1.151	1.149	1.241	1.659	12.614
$N=20$	1.652	1.857	1.911	2.221	2.159	2.358	3.155	35.563
$N=40$	2.788	3.634	3.804	4.247	4.278	4.649	6.162	47.005
$N=100$	6.495	8.857	8.764	10.566	10.482	11.294	15.187	115.771

Appendix

Table 2. Runtime comparison (seconds) for gaussian_approx and posterior_sampling_approx on cartpole (

n=4

), with GP posterior with

D=1000

inputs.

Recovery Horizon	gaussian_approx	posterior_sampling_approx
Recovery Horizon	( $\epsilon_{t}=0.9999$ )	$K=5$	$K=10$	$K=50$	$K=100$	$K=500$	$K=1000$	$K=5000$
$N=5$	0.703	15.752	15.794	16.135	16.284	17.591	19.841	47.535
$N=10$	1.230	28.854	28.943	29.562	29.845	32.243	36.369	87.142
$N=20$	2.384	55.087	55.253	56.441	56.992	61.547	69.431	166.362
$N=40$	4.353	107.614	107.881	110.180	111.251	120.161	135.552	324.794
$N=100$	10.4313	264.880	265.655	271.443	274.051	296.042	333.899	800.059

Appendix

Table 3. Runtime comparison (seconds) for gaussian_approx and posterior_sampling_approx on cartpole (

n=4

), with SGPR posterior ( Titsias, 2009 ) with

D=5000

inputs and

Z=500

inducing points.

	overall pass	$\text{skew}(t)$	skew $p$ -value	skew pass	$\text{kurt}(t)$	kurt $p$ -value	kurt pass
$t=0$	Pass	0.0228	0.5245	Pass	24.0808	0.6803	Pass
$t=1$	Pass	0.0244	0.4374	Pass	24.0433	0.8252	Pass
$t=2$	Pass	0.0313	0.1629	Pass	24.0392	0.8416	Pass
$t=3$	Pass	0.0339	0.1038	Pass	24.1253	0.5225	Pass
$t=4$	Pass	0.0280	0.2745	Pass	24.0372	0.8495	Pass
$t=5$	Pass	0.0287	0.2456	Pass	24.2263	0.2482	Pass

Appendix

Table 4. Skewness and kurtosis analysis on cartpole (

n=4

) with GP posterior with

D=1000

inputs.

	overall pass	$\text{skew}(t)$	skew $p$ -value	skew pass	$\text{kurt}(t)$	kurt $p$ -value	kurt pass
$t=0$	Pass	0.0049	0.3999	Pass	8.1132	0.3169	Pass
$t=1$	Fail	0.0132	0.0269	Fail	8.0025	0.9824	Pass
$t=2$	Fail	0.0141	0.0191	Fail	7.8566	0.2050	Pass
$t=3$	Pass	0.0095	0.0935	Pass	7.9025	0.3889	Pass
$t=4$	Pass	0.0109	0.0595	Pass	7.9244	0.5043	Pass
$t=5$	Pass	0.0099	0.0834	Pass	8.0009	0.9938	Pass

Appendix

Table 5. Skewness and kurtosis analysis on mountain_car (

n=2

) with GP posterior with

D=1000

inputs.

	overall pass	$\text{skew}(t)$	skew $p$ -value	skew pass	$\text{kurt}(t)$	kurt $p$ -value	kurt pass
$t=0$	Pass	0.0222	0.5542	Pass	24.2452	0.2108	Pass
$t=1$	Pass	0.0297	0.2123	Pass	23.8672	0.4980	Pass
$t=2$	Fail	0.0487	0.0042	Fail	23.7801	0.2618	Pass
$t=3$	Pass	0.0368	0.0595	Pass	23.8696	0.5057	Pass
$t=4$	Pass	0.0361	0.0684	Pass	23.7830	0.2681	Pass
$t=5$	Pass	0.0262	0.3484	Pass	23.7300	0.1682	Pass

Appendix

Table 6. Skewness and kurtosis analysis on obstacle (

n=4

) with GP posterior with

D=1000

inputs.

	overall pass	$\text{skew}(t)$	skew $p$ -value	skew pass	$\text{kurt}(t)$	kurt $p$ -value	kurt pass
$t=1$	Pass	0.0139	0.9291	Pass	24.0364	0.8525	Pass
$t=2$	Pass	0.0313	0.1639	Pass	24.0531	0.7862	Pass
$t=3$	Pass	0.0352	0.0810	Pass	24.0604	0.7580	Pass
$t=4$	Pass	0.0303	0.1912	Pass	24.0422	0.8295	Pass
$t=5$	Fail	0.0463	0.0075	Fail	24.4101	0.0364	Fail
$t=6$	Fail	0.0676	0.0000	Fail	24.2153	0.2719	Pass

Appendix

Table 7. Skewness and kurtosis analysis on obstacle2/3/4 (

n=4

) with GP posterior with

D=1000

inputs.

	overall pass	$\text{skew}(t)$	skew $p$ -value	skew pass	$\text{kurt}(t)$	kurt $p$ -value	kurt pass
$t=0$	Pass	0.0047	0.4179	Pass	8.1394	0.2179	Pass
$t=1$	Pass	0.0050	0.3840	Pass	8.0203	0.8577	Pass
$t=2$	Pass	0.0075	0.1803	Pass	7.8849	0.3090	Pass
$t=3$	Pass	0.0067	0.2320	Pass	7.8799	0.2885	Pass
$t=4$	Pass	0.0105	0.0673	Pass	8.0387	0.7323	Pass
$t=5$	Fail	0.0181	0.0045	Fail	8.0728	0.5198	Pass

Appendix

Table 8. Skewness and kurtosis analysis on road (

n=2

) with GP posterior with

D=1000

inputs.

	overall pass	$\text{skew}(t)$	skew $p$ -value	skew pass	$\text{kurt}(t)$	kurt $p$ -value	kurt pass
$t=0$	Pass	0.0213	0.6065	Pass	24.1454	0.4582	Pass
$t=1$	Pass	0.0232	0.4977	Pass	23.6600	0.0827	Pass
$t=2$	Pass	0.0202	0.6635	Pass	23.8459	0.4317	Pass
$t=3$	Fail	0.0233	0.4936	Pass	24.4087	0.0370	Fail
$t=4$	Pass	0.0217	0.5827	Pass	23.9153	0.6657	Pass
$t=5$	Pass	0.0111	0.9799	Pass	24.0857	0.6618	Pass

Appendix

Table 9. Skewness and kurtosis analysis on road_2d (

n=4

) with GP posterior with

D=1000

inputs.

Figure 6.

K=50

empirical samples and 99.99%-CI uncertainty sets

\mathcal{E}(0),\ldots\mathcal{E}(N)

for cartpole (i)/(ii) , with GP posterior with

D=400

inputs.

Shield
Name	Symbol	Value
Recovery horizon	$N$	20
Sigma coef.	$z$ ( $\epsilon_{t}$ )	3.82 (0.9999)
Gaussian process
Learning rate	-	0.01
Optim. iters.	-	100

Appendix

Table 10. Default hyperparameters for A2C-Shield

Shared
Name	Symbol	Value
Actor learning rate	$\eta$	$3\times 10^{-4}$
Discount factor	$\gamma$	$\{0.99,0.997\}$
Cost coefficient	$c$	$1.0$
Cost threshold	-	$0.01$
Cost gamma	$\gamma_{c}$	$\{0.99,0.997\}$

Appendix

Table 11. Hyperparameter details for CPO and PPO-Lag.

Figure 7. Visualization of the cartpole dynamical system

Figure 8. Visualization of the mountain car problem

RL Shielding

See all 14 papers

May 29, 2026

Robust Shielding for Safe Reinforcement Learning

Shielding is an effective approach to formally guarantee the safety of reinforcement learning agents in Markov decision processes (MDPs). However, existing shielding techniques typically assume knowledge of the safety-relevant transition dynamics - a requirement that is seldom met in practice. To address this limitation, we introduce a novel shielding framework for robust MDPs (RMDPs), i.e., MDPs with sets of transition probabilities. We define safety as the satisfaction of a linear temporal logic (LTL) formula with a certain threshold probability under the worst-case transition probabilities of the RMDP. We prove that our shielding framework is both sound and optimal for the RMDP: every policy admissible by the shield is safe, and conversely, every safe RMDP policy is admissible by the shield. We combine our approach with existing sampling methods for learning transition probabilities of MDPs with probably approximately correct (PAC) guarantees. This combination enables the construction of shields for MDPs that, with high confidence, guarantee safety while remaining minimally restrictive. Our experiments show that our shields for learned RMDPs guarantee safety in unknown MDPs while recovering strong expected return as the number of samples increases.

Edwin Hamel-De le Court, Thom Badings, Alessandro Abate +2

Department of Computer Science, University of Manchester, United Kingdom · Faculty of Computer Science & DSME, RWTH Aachen University, Germany · Department of Computer Science, University of Oxford, United Kingdom +1

Robust Markov Decision Processes Formal Verification Safe RL Robust RL RL Shielding

Ask PDF

Oct 16, 2026

RECREG: Recovery-based Shielding for Regular Safety Properties in Reinforcement Learning

In many real-world scenarios we need to consider formal requirements, such as safety and constraint compliance, to deploy Reinforcement Learning (RL) agents. In this paper we introduce RecReg, a safe RL framework capable of enforcing non-Markovian regular safety properties via recovery-based shielding. RecReg is a unified approach that supports different levels of a priori knowledge and safety guarantees. The paper primarily focuses on building the theoretical foundations for RecReg in the tabular setting, providing rigorous safety and convergence guarantees that are not present in related work. We provide an extensive set of empirical results using the MASA-Safe-RL library, and we outline an extension of RecReg to deep RL domains using SOTA model-based RL. Our empirical results demonstrate that RecReg achieves efficient learning and matches the theory in this paper.

Alexander W. Goodall, Francesco Belardinelli

Department of Computer Science Imperial College London

RL Shielding Safe RL Formal Methods for RL Non-Markovian RL

Ask PDF

May 19, 2026

Sampling-Based Safe Reinforcement Learning

Safe exploration remains a fundamental challenge in reinforcement learning (RL), limiting the deployment of RL agents in the real world. We propose Sampling-Based Safe Reinforcement Learning (SBSRL), a model-based RL algorithm that maintains safety throughout the learning process by enforcing constraints jointly across a finite set of dynamics samples. This formulation approximates an intractable worst-case optimization over uncertain dynamics and enables practical safety guarantees in continuous domains. We further introduce an exploration strategy based on constraining epistemic uncertainty, eliminating the need for explicit exploration bonuses. Under regularity conditions, we derive high-probability guarantees of safety throughout learning and a finite-time sample complexity bound for recovering a near-optimal policy. Empirically, SBSRL achieves safe and efficient exploration both in simulation and in real robotic hardware, and readily extends to practical deep-ensemble implementations that scale to high-dimensional continuous control problems.

Luca Vignola, Bruce D. Lee, Manish Prajapat +4

ETH Zurich

Robot Safety Safe RL Sample-Efficient RL Model-Based RL

Ask PDF

Abstract

RL Shielding

Explore similar work

Robust Shielding for Safe Reinforcement Learning

RECREG: Recovery-based Shielding for Regular Safety Properties in Reinforcement Learning

Sampling-Based Safe Reinforcement Learning