Safe Reinforcement Learning via Recovery-based Shielding with Gaussian Process Dynamics Models

Alexander W. Goodall; Francesco Belardinelli

cs.LGFeb 12, 2026

Safe Reinforcement Learning via Recovery-based Shielding with Gaussian Process Dynamics Models

Authors: Alexander W. Goodall, Francesco Belardinelli

Organizations: Imperial College London, Department of Computing London, United Kingdom

Abstract

Reinforcement learning (RL) is a powerful framework for optimal decision-making and control but often lacks provable guarantees for safety-critical applications. In this paper, we introduce a novel recovery-based shielding framework that enables safe RL with a provable safety lower bound for unknown and non-linear continuous dynamical systems. The proposed approach integrates a backup policy (shield) with the RL agent, leveraging Gaussian process (GP) based uncertainty quantification to predict potential violations of safety constraints, dynamically recovering to safe trajectories only when necessary. Experience gathered by the 'shielded' agent is used to construct the GP models, with policy optimization via internal model-based sampling - enabling unrestricted exploration and sample efficient learning, without compromising safety. Empirically our approach demonstrates strong performance and strict safety-compliance on a suite of continuous control environments.

Figures & tables

Figure 1.

99.99\%

-CI uncertainty sets for recoverable (blue) and irrecoverable (red) trajectories under the backup policy for cartpole (i)/(ii) environment.

Env.	Metric	A2C-GP-Shield	MPS	DMPS	CPO	PPO-Lag	Thm. 3.3
cartpole	Return	$\bm{200.0\pm 0.00}$	$\bm{200.0\pm 0.00}$	$\bm{200.0\pm 0.00}$	$\bm{200.0\pm 0.00}$	$188.0\pm 5.53$	$1-\epsilon=0.98$
	Safety Prob.	$\bm{1.000\pm 0.00}$	$\bm{1.000\pm 0.00}$	$\bm{1.000\pm 0.00}$	$\bm{1.000\pm 0.00}$	$0.910\pm 0.04$	$1-\epsilon=0.98$
cartpole2	Return	$30.7\pm 10.4$	$43.2\pm 11.1$	$65.1\pm 23.1$ *	$152.0\pm 31.2$	$\bm{171.0\pm 7.53}$	$1-\epsilon=0.98$
	Safety prob.	$\bm{1.000\pm 0.00}$	$\bm{1.000\pm 0.00}$	$\bm{1.000\pm 0.00}$	$0.980\pm 0.01$	$0.811\pm 0.12$	$1-\epsilon=0.98$
mountain_car	Return	$\bm{91.3\pm 1.08}$ *	$85.1\pm 8.17$	$81.2\pm 0.28$	$-30.4\pm 6.74$	$73.2\pm 18.7$	$1-\epsilon=0.9$
	Safety prob.	$\bm{1.000\pm 0.00}$	$\bm{1.000\pm 0.00}$	$\bm{1.000\pm 0.00}$	$0.995\pm 0.00$	$0.884\pm 0.05$	$1-\epsilon=0.9$

Table 1. Empirical mean return and safety probability (at the end of training), bold text denotes the best score, asterisk (*) denoted the best reward that also satisfies the safety constraint with probability 1, standard error (SE) bars (averaged over 5 independent runs) are reported. We also include the theoretical lower bound established by Thm. 3.3 (based on

\epsilon_{t}=10^{-5}

Panel of Figure 2 — Figure 2. Learning curves (log scale on the x-axis).

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3. Learning curves (reward and safety probability) for A2C-Shield and A2C-Eval.

Panel of Figure 4 — Figure 4. Overrides (per episode) for A2C-Shield

Panel of Figure 5 — Figure 5. Learning curves (reward and safety probability) for CPO and PPO-Lag.

Recovery Horizon	gaussian_approx	posterior_sampling_approx
Recovery Horizon	( $\epsilon_{t}=0.9999$ )	$K=5$	$K=10$	$K=50$	$K=100$	$K=500$	$K=1000$	$K=5000$
$N=5$	0.749	0.537	0.533	0.648	0.619	0.683	0.910	6.890
$N=10$	1.090	0.985	1.015	1.151	1.149	1.241	1.659	12.614
$N=20$	1.652	1.857	1.911	2.221	2.159	2.358	3.155	35.563
$N=40$	2.788	3.634	3.804	4.247	4.278	4.649	6.162	47.005
$N=100$	6.495	8.857	8.764	10.566	10.482	11.294	15.187	115.771

Appendix

Table 2. Runtime comparison (seconds) for gaussian_approx and posterior_sampling_approx on cartpole (

n=4

), with GP posterior with

D=1000

inputs.

Recovery Horizon	gaussian_approx	posterior_sampling_approx
Recovery Horizon	( $\epsilon_{t}=0.9999$ )	$K=5$	$K=10$	$K=50$	$K=100$	$K=500$	$K=1000$	$K=5000$
$N=5$	0.703	15.752	15.794	16.135	16.284	17.591	19.841	47.535
$N=10$	1.230	28.854	28.943	29.562	29.845	32.243	36.369	87.142
$N=20$	2.384	55.087	55.253	56.441	56.992	61.547	69.431	166.362
$N=40$	4.353	107.614	107.881	110.180	111.251	120.161	135.552	324.794
$N=100$	10.4313	264.880	265.655	271.443	274.051	296.042	333.899	800.059

Appendix

Table 3. Runtime comparison (seconds) for gaussian_approx and posterior_sampling_approx on cartpole (

n=4

), with SGPR posterior ( Titsias, 2009 ) with

D=5000

inputs and

Z=500

inducing points.

	overall pass	$\text{skew}(t)$	skew $p$ -value	skew pass	$\text{kurt}(t)$	kurt $p$ -value	kurt pass
$t=0$	Pass	0.0228	0.5245	Pass	24.0808	0.6803	Pass
$t=1$	Pass	0.0244	0.4374	Pass	24.0433	0.8252	Pass
$t=2$	Pass	0.0313	0.1629	Pass	24.0392	0.8416	Pass
$t=3$	Pass	0.0339	0.1038	Pass	24.1253	0.5225	Pass
$t=4$	Pass	0.0280	0.2745	Pass	24.0372	0.8495	Pass
$t=5$	Pass	0.0287	0.2456	Pass	24.2263	0.2482	Pass

Appendix

Table 4. Skewness and kurtosis analysis on cartpole (

n=4

) with GP posterior with

D=1000

inputs.

	overall pass	$\text{skew}(t)$	skew $p$ -value	skew pass	$\text{kurt}(t)$	kurt $p$ -value	kurt pass
$t=0$	Pass	0.0049	0.3999	Pass	8.1132	0.3169	Pass
$t=1$	Fail	0.0132	0.0269	Fail	8.0025	0.9824	Pass
$t=2$	Fail	0.0141	0.0191	Fail	7.8566	0.2050	Pass
$t=3$	Pass	0.0095	0.0935	Pass	7.9025	0.3889	Pass
$t=4$	Pass	0.0109	0.0595	Pass	7.9244	0.5043	Pass
$t=5$	Pass	0.0099	0.0834	Pass	8.0009	0.9938	Pass

Appendix

Table 5. Skewness and kurtosis analysis on mountain_car (

n=2

) with GP posterior with

D=1000

inputs.

	overall pass	$\text{skew}(t)$	skew $p$ -value	skew pass	$\text{kurt}(t)$	kurt $p$ -value	kurt pass
$t=0$	Pass	0.0222	0.5542	Pass	24.2452	0.2108	Pass
$t=1$	Pass	0.0297	0.2123	Pass	23.8672	0.4980	Pass
$t=2$	Fail	0.0487	0.0042	Fail	23.7801	0.2618	Pass
$t=3$	Pass	0.0368	0.0595	Pass	23.8696	0.5057	Pass
$t=4$	Pass	0.0361	0.0684	Pass	23.7830	0.2681	Pass
$t=5$	Pass	0.0262	0.3484	Pass	23.7300	0.1682	Pass

Appendix

Table 6. Skewness and kurtosis analysis on obstacle (

n=4

) with GP posterior with

D=1000

inputs.

	overall pass	$\text{skew}(t)$	skew $p$ -value	skew pass	$\text{kurt}(t)$	kurt $p$ -value	kurt pass
$t=1$	Pass	0.0139	0.9291	Pass	24.0364	0.8525	Pass
$t=2$	Pass	0.0313	0.1639	Pass	24.0531	0.7862	Pass
$t=3$	Pass	0.0352	0.0810	Pass	24.0604	0.7580	Pass
$t=4$	Pass	0.0303	0.1912	Pass	24.0422	0.8295	Pass
$t=5$	Fail	0.0463	0.0075	Fail	24.4101	0.0364	Fail
$t=6$	Fail	0.0676	0.0000	Fail	24.2153	0.2719	Pass

Appendix

Table 7. Skewness and kurtosis analysis on obstacle2/3/4 (

n=4

) with GP posterior with

D=1000

inputs.

	overall pass	$\text{skew}(t)$	skew $p$ -value	skew pass	$\text{kurt}(t)$	kurt $p$ -value	kurt pass
$t=0$	Pass	0.0047	0.4179	Pass	8.1394	0.2179	Pass
$t=1$	Pass	0.0050	0.3840	Pass	8.0203	0.8577	Pass
$t=2$	Pass	0.0075	0.1803	Pass	7.8849	0.3090	Pass
$t=3$	Pass	0.0067	0.2320	Pass	7.8799	0.2885	Pass
$t=4$	Pass	0.0105	0.0673	Pass	8.0387	0.7323	Pass
$t=5$	Fail	0.0181	0.0045	Fail	8.0728	0.5198	Pass

Appendix

Table 8. Skewness and kurtosis analysis on road (

n=2

) with GP posterior with

D=1000

inputs.

	overall pass	$\text{skew}(t)$	skew $p$ -value	skew pass	$\text{kurt}(t)$	kurt $p$ -value	kurt pass
$t=0$	Pass	0.0213	0.6065	Pass	24.1454	0.4582	Pass
$t=1$	Pass	0.0232	0.4977	Pass	23.6600	0.0827	Pass
$t=2$	Pass	0.0202	0.6635	Pass	23.8459	0.4317	Pass
$t=3$	Fail	0.0233	0.4936	Pass	24.4087	0.0370	Fail
$t=4$	Pass	0.0217	0.5827	Pass	23.9153	0.6657	Pass
$t=5$	Pass	0.0111	0.9799	Pass	24.0857	0.6618	Pass

Appendix

Table 9. Skewness and kurtosis analysis on road_2d (

n=4

) with GP posterior with

D=1000

inputs.

Figure 6.

K=50

empirical samples and 99.99%-CI uncertainty sets

\mathcal{E}(0),\ldots\mathcal{E}(N)

for cartpole (i)/(ii) , with GP posterior with

D=400

inputs.

Shield
Name	Symbol	Value
Recovery horizon	$N$	20
Sigma coef.	$z$ ( $\epsilon_{t}$ )	3.82 (0.9999)
Gaussian process
Learning rate	-	0.01
Optim. iters.	-	100

Appendix

Table 10. Default hyperparameters for A2C-Shield

Shared
Name	Symbol	Value
Actor learning rate	$\eta$	$3\times 10^{-4}$
Discount factor	$\gamma$	$\{0.99,0.997\}$
Cost coefficient	$c$	$1.0$
Cost threshold	-	$0.01$
Cost gamma	$\gamma_{c}$	$\{0.99,0.997\}$

Appendix

Table 11. Hyperparameter details for CPO and PPO-Lag.

Figure 7. Visualization of the cartpole dynamical system

Figure 8. Visualization of the mountain car problem

RL Shielding

See all 14 papers

Robust Shielding for Safe Reinforcement Learning
May 29, 2026Edwin Hamel-De le Court, Thom Badings, Alessandro Abate +2Robust Markov Decision Processes Formal Verification
Department of Computer Science, University of Manchester, United Kingdom · Faculty of Computer Science & DSME, RWTH Aachen University, Germany · Department of Computer Science, University of Oxford, United Kingdom +1
RECREG: Recovery-based Shielding for Regular Safety Properties in Reinforcement Learning
Oct 16, 2026Alexander W. Goodall, Francesco BelardinelliRL Shielding Safe RL
Department of Computer Science Imperial College London
Sampling-Based Safe Reinforcement Learning
May 19, 2026Luca Vignola, Bruce D. Lee, Manish Prajapat +4Robot Safety Safe RL
ETH Zurich

May 29, 2026J/K move · Enter open · S save

Robust Shielding for Safe Reinforcement Learning

Edwin Hamel-De le Court, Thom Badings, Alessandro Abate, Francesco Belardinelli+1

Department of Computer Science, University of Manchester, United Kingdom · Faculty of Computer Science & DSME, RWTH Aachen University, Germany · Department of Computer Science, University of Oxford, United Kingdom+1

Robust Markov Decision Processes Formal Verification Safe RL Robust RL RL Shielding

Open paper AskPDF arXiv

Shielding is an effective approach to formally guarantee the safety of reinforcement learning agents in Markov decision processes (MDPs). However, existing shielding techniques typically assume knowledge of the safety-relevant transition dynamics - a requirement that is seldom met in practice. To address this limitation, we introduce a novel shielding framework for robust MDPs (RMDPs), i.e., MDPs with sets of transition probabilities. We define safety as the satisfaction of a linear temporal logic (LTL) formula with a certain threshold probability under the worst-case transition probabilities of the RMDP. We prove that our shielding framework is both sound and optimal for the RMDP: every policy admissible by the shield is safe, and conversely, every safe RMDP policy is admissible by the shield. We combine our approach with existing sampling methods for learning transition probabilities of MDPs with probably approximately correct (PAC) guarantees. This combination enables the construction of shields for MDPs that, with high confidence, guarantee safety while remaining minimally restrictive. Our experiments show that our shields for learned RMDPs guarantee safety in unknown MDPs while recovering strong expected return as the number of samples increases.

Abstract

RL Shielding

Explore similar work

Robust Shielding for Safe Reinforcement Learning

RECREG: Recovery-based Shielding for Regular Safety Properties in Reinforcement Learning

Sampling-Based Safe Reinforcement Learning