Adaptive Probabilistic Shielding by Learning MDPs for Safe Reinforcement Learning

Astrid Horn Brorholt; Maris F. L. Galesloot; Nils Jansen; Kim Guldstrand Larsen; Christian Schilling

cs.LGAug 20, 2026

Adaptive Probabilistic Shielding by Learning MDPs for Safe Reinforcement Learning

Authors: Astrid Horn Brorholt, Maris F. L. Galesloot, Nils Jansen, Kim Guldstrand Larsen, Christian Schilling

Organizations: Aalborg University, Aalborg, Denmark · Radboud University, Nijmegen, Netherlands · Ruhr University Bochum, Bochum, Germany

Abstract

Probabilistic shielding is a technique for safe reinforcement learning (RL). Typically, a static observer -- called the shield -- constrains the learning agent's actions to those for which acting safely remains feasible. Traditionally, the shield is computed from the transition probabilities of the underlying Markov decision process (MDP). Thus, this technique is not applicable when the MDP model is not given a priori, which, unfortunately, is the case in typical RL applications. In this paper, we study the problem of computing a shield in the setting where the transition graph of the MDP is known, but the transition probabilities are unknown. Our approach integrates probabilistic shielding with online model learning: as the RL agent explores the environment, we estimate the transition probabilities. From this estimate, we compute a shield. While the shield may be conservative initially, it adapts as the model estimate becomes more precise. Thus, the shield improves in tandem with the RL agent. This paradigm of adaptive probabilistic shielding raises a number of challenges, such as when to recompute the shield and how to balance between exploration and safety during learning. We empirically evaluate multiple variants of this paradigm across several environments.

Figures & tables

Figure 1: A high-level overview of our adaptive probabilistic shielding approach.

Panel of Figure 2 — Algorithm 1 Safe RL via Adaptive Probabilistic Shielding

Panel of Figure 3 — Figure 3: Heat map showing how frequently a state is visited in the antlion environment. The initial state, predator, and goal are at the bottom, center, and top, respectively.

Estimator	Aircraft	Antlion	Sinkholes	Crossroads	Gravity
Robust LUI	14.89 8.3%	5.27 4.5%	57.05 3.6%	5.12 0.0%	8.27 3.9%
Robust PAC	15.54 8.2%	6.33 2.8%	46.59 3.8%	5.12 0.0%	$-$ 2.56 0.0%
MAP	13.24 4.0%	$-$ 8.99 8.0%	49.16 3.0%	5.12 0.0%	$-$ 2.35 0.0%
Optimistic LUI	14.17 5.2%	6.66 6.0%	67.78 4.1%	5.11 0.0%	21.40 19.6%
Optimistic PAC	14.35 6.8%	6.36 10.0%	65.35 3.8%	5.12 0.0%	23.51 47.8%
Unshielded	14.36 7.3%	6.62 9.0%	65.62 3.7%	9.57 40.1%	30.35 99.2%

Table 1: Comparison for different model estimators. The numbers in each cell respectively denote the reward in the final evaluation (left) and the probability that the final policy produces an unsafe episode (right). The numbers are the mean outcomes of

100

repetitions and bold entries mark the best result for each column.

Exploration	Aircraft	Antlion	Sinkholes	Crossroads	Gravity
$\mathrm{Unif}_{A}$	14.89 8.3%	5.27 4.5%	57.05 3.6%	5.12 0.0%	8.27 3.9%
$\mathrm{Unif}_{\text{{\color[rgb]{1,1,1}$ \nabla $} \hskip-11.95836pt \hbox to3.5pt{\vbox to4.55pt{\pgfpicture\makeatletter\hbox{\hskip 0.175pt\lower-0.875pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} { {{\lx@inpgf@ignorespaces}{}}{{}}{} {\lx@inpgf@ignorespaces}{} {\lx@inpgf@ignorespaces}{} {\lx@inpgf@ignorespaces}{} {{\lx@inpgf@ignorespaces}{}}{} {\lx@inpgf@ignorespaces}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.35pt} \lx@inpgf@ignorespaces{}\lxSVG@stroke\lxSVG@drawpath@unclipped{M 0 4.84 L 4.36 4.84 L 4.36 1.29 L 2.18 -0.97 L 0 1.29 Z}{fill:none} \lx@inpgf@ignorespaces \lxSVG@closescope } \lxSVG@closescope {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}} \hskip-5.1819pt }(s)}$	15.04 9.8%	1.93 5.1%	57.29 3.7%	5.00 0.0%	1.58 0.6%

Table 2: Comparison for the two exploration variants. Setup as in Table 1 .

$u$	Aircraft	Antlion	Sinkholes	Crossroads	Gravity
0 250	14.94 12.9%	$-$ 10.74 9.7%	78.71 5.4%	5.12 0.0%	$-$ 2.32 0.0%
0 500	14.99 10.5%	$-$ 10.46 11.0%	84.78 4.1%	5.12 0.0%	2.95 1.9%
1 000	14.89 8.3%	5.27 4.5%	57.05 3.6%	5.12 0.0%	8.27 3.9%
1 500	14.81 7.8%	6.17 3.7%	54.40 3.6%	5.12 0.0%	11.02 3.5%
2 000	14.72 7.0%	6.06 4.0%	57.93 3.6%	5.12 0.0%	12.92 8.2%

Table 3: Comparison for different update delays

u

. Setup as in Table 1 .

$h$	Aircraft	Antlion	Sinkholes	Crossroads	Gravity
00 6	14.50 8.2%	6.42 3.1%	48.50 3.3%	9.57 40.1%	24.61 47.1%
0 12	14.78 8.2%	5.90 3.8%	54.81 3.4%	9.57 40.1%	23.16 33.3%
0 25	14.89 8.3%	5.71 4.2%	56.31 3.7%	9.57 40.1%	22.16 21.5%
0 50	14.89 8.3%	4.67 4.9%	57.12 3.4%	9.57 40.1%	19.36 13.2%
0 75	14.89 8.3%	4.39 4.4%	56.35 3.7%	9.57 40.1%	13.55 6.9%
100	14.89 8.3%	5.27 4.4%	57.05 3.7%	5.12 0.0%	8.27 4.0%

Table 4: Comparison for varying shield horizon (

h

\varphi|h

). Setup as in Table 1 .

Safe RL

Safe reinforcement learning studies how agents can learn and act while respecting safety constraints and limiting harmful outcomes.

See all 215 papers

Robust Shielding for Safe Reinforcement Learning
May 29, 2026Edwin Hamel-De le Court, Thom Badings, Alessandro Abate +2Robust Markov Decision Processes Formal Verification
Department of Computer Science, University of Manchester, United Kingdom · Faculty of Computer Science & DSME, RWTH Aachen University, Germany · Department of Computer Science, University of Oxford, United Kingdom +1
Shields to Guarantee Probabilistic Safety in MDPs
May 11, 2026Linus Heck, Filip Macák, Roman Andriushchenko +2Markov Decision Processes Safety-Critical Control
Radboud University, Nijmegen, the Netherlands · Brno University of Technology, Czechia
Robust Probabilistic Shielding for Safe Offline Reinforcement Learning
May 11, 2026Maris F. L. Galesloot, Thomas Rhemrev, Nils JansenOffline RL Safe RL
Radboud University · Nijmegen, The Netherlands · Ruhr University & Radboud University +1

May 29, 2026J/K move · Enter open · S save

Robust Shielding for Safe Reinforcement Learning

Edwin Hamel-De le Court, Thom Badings, Alessandro Abate, Francesco Belardinelli+1

Department of Computer Science, University of Manchester, United Kingdom · Faculty of Computer Science & DSME, RWTH Aachen University, Germany · Department of Computer Science, University of Oxford, United Kingdom+1

Robust Markov Decision Processes Formal Verification Safe RL Robust RL

Open paper AskPDF arXiv

Shielding is an effective approach to formally guarantee the safety of reinforcement learning agents in Markov decision processes (MDPs). However, existing shielding techniques typically assume knowledge of the safety-relevant transition dynamics - a requirement that is seldom met in practice. To address this limitation, we introduce a novel shielding framework for robust MDPs (RMDPs), i.e., MDPs with sets of transition probabilities. We define safety as the satisfaction of a linear temporal logic (LTL) formula with a certain threshold probability under the worst-case transition probabilities of the RMDP. We prove that our shielding framework is both sound and optimal for the RMDP: every policy admissible by the shield is safe, and conversely, every safe RMDP policy is admissible by the shield. We combine our approach with existing sampling methods for learning transition probabilities of MDPs with probably approximately correct (PAC) guarantees. This combination enables the construction of shields for MDPs that, with high confidence, guarantee safety while remaining minimally restrictive. Our experiments show that our shields for learned RMDPs guarantee safety in unknown MDPs while recovering strong expected return as the number of samples increases.

Abstract

Safe RL

Explore similar work

Robust Shielding for Safe Reinforcement Learning

Shields to Guarantee Probabilistic Safety in MDPs

Robust Probabilistic Shielding for Safe Offline Reinforcement Learning