Probabilistic shielding is a technique for safe reinforcement learning (RL). Typically, a static observer -- called the shield -- constrains the learning agent's actions to those for which acting safely remains feasible. Traditionally, the shield is computed from the transition probabilities of the underlying Markov decision process (MDP). Thus, this technique is not applicable when the MDP model is not given a priori, which, unfortunately, is the case in typical RL applications. In this paper, we study the problem of computing a shield in the setting where the transition graph of the MDP is known, but the transition probabilities are unknown. Our approach integrates probabilistic shielding with online model learning: as the RL agent explores the environment, we estimate the transition probabilities. From this estimate, we compute a shield. While the shield may be conservative initially, it adapts as the model estimate becomes more precise. Thus, the shield improves in tandem with the RL agent. This paradigm of adaptive probabilistic shielding raises a number of challenges, such as when to recompute the shield and how to balance between exploration and safety during learning. We empirically evaluate multiple variants of this paradigm across several environments.
Figures & tables
Figure 1: A high-level overview of our adaptive probabilistic shielding approach.
Algorithm 1 Safe RL via Adaptive Probabilistic Shielding
Figure 2: Mean outcome of 100 repetitions for different configurations. We plot the standard deviation as a ribbon around the lines. Vertical grid lines mark updates of the adaptive shield. Column I: Cumulative safety violations during training. Column II: Reward during training. Column III: Per-episode rate of using the fallback \text{{\color[rgb]{1,1,1} \nabla} \hskip-15.00002pt \hbox to5pt{\vbox to6.5pt{\pgfpicture\makeatletter\hbox{\hskip 0.25pt\lower-1.25pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} { {{\lx@inpgf@ignorespaces}{}}{{}}{} {\lx@inpgf@ignorespaces}{} {\lx@inpgf@ignorespaces}{} {\lx@inpgf@ignorespaces}{} {{\lx@inpgf@ignorespaces}{}}{} {\lx@inpgf@ignorespaces}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.5pt} \lx@inpgf@ignorespaces{}\lxSVG@stroke\lxSVG@drawpath@unclipped{M 0 6.92 L 6.23 6.92 L 6.23 1.84 L 3.11 -1.38 L 0 1.84 Z}{fill:none} \lx@inpgf@ignorespaces \lxSVG@closescope } \lxSVG@closescope {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}} \hskip-6.49994pt }_{\kappa} . Column IV: Average total variation between true model and model estimate during training.
Figure 3: Heat map showing how frequently a state is visited in the antlion environment. The initial state, predator, and goal are at the bottom, center, and top, respectively.
Estimator
Aircraft
Antlion
Sinkholes
Crossroads
Gravity
Robust LUI
14.89 8.3%
5.27 4.5%
57.05 3.6%
5.12 0.0%
8.27 3.9%
Robust PAC
15.54 8.2%
6.33 2.8%
46.59 3.8%
5.12 0.0%
− 2.56 0.0%
MAP
13.24 4.0%
− 8.99 8.0%
49.16 3.0%
5.12 0.0%
− 2.35 0.0%
Optimistic LUI
14.17 5.2%
6.66 6.0%
67.78 4.1%
5.11 0.0%
21.40 19.6%
Optimistic PAC
14.35 6.8%
6.36 10.0%
65.35 3.8%
5.12 0.0%
23.51 47.8%
Unshielded
14.36 7.3%
6.62 9.0%
65.62 3.7%
9.57 40.1%
30.35 99.2%
Table 1: Comparison for different model estimators. The numbers in each cell respectively denote the reward in the final evaluation (left) and the probability that the final policy produces an unsafe episode (right). The numbers are the mean outcomes of 100 repetitions and bold entries mark the best result for each column.
Shielding is an effective approach to formally guarantee the safety of reinforcement learning agents in Markov decision processes (MDPs). However, existing shielding techniques typically assume knowledge of the safety-relevant transition dynamics - a requirement that is seldom met in practice. To address this limitation, we introduce a novel shielding framework for robust MDPs (RMDPs), i.e., MDPs with sets of transition probabilities. We define safety as the satisfaction of a linear temporal logic (LTL) formula with a certain threshold probability under the worst-case transition probabilities of the RMDP. We prove that our shielding framework is both sound and optimal for the RMDP: every policy admissible by the shield is safe, and conversely, every safe RMDP policy is admissible by the shield. We combine our approach with existing sampling methods for learning transition probabilities of MDPs with probably approximately correct (PAC) guarantees. This combination enables the construction of shields for MDPs that, with high confidence, guarantee safety while remaining minimally restrictive. Our experiments show that our shields for learned RMDPs guarantee safety in unknown MDPs while recovering strong expected return as the number of samples increases.
Edwin Hamel-De le Court, Thom Badings, Alessandro Abate +2
Department of Computer Science, University of Manchester, United Kingdom · Faculty of Computer Science & DSME, RWTH Aachen University, Germany · Department of Computer Science, University of Oxford, United Kingdom +1
Shielding is a prominent model-based technique to ensure safety of autonomous agents. Classical shielding aims to ensure that nothing bad ever happens and comes with strong guarantees about safety and maximal permissiveness. However, shielding systems for probabilistic safety, where something bad is allowed to happen with an acceptable probability, has proven to be more intricate. This paper presents a formal framework that conservatively extends classical shields to probabilistic safety. In this framework, we (i) demonstrate the impossibility of preserving the strong guarantees on safety and permissiveness, (ii) provide natural shields with weaker guarantees, and (iii) introduce offline and online shield constructions ensuring strong safety guarantees. The empirical evaluation highlights the practical advantages of the new shields, as well as their computational feasibility.
Linus Heck, Filip Macák, Roman Andriushchenko +2
Radboud University, Nijmegen, the Netherlands · Brno University of Technology, Czechia
In offline reinforcement learning (RL), we learn policies from fixed datasets without environment interaction. The major challenges are to provide guarantees on the (1) performance and (2) safety of the resulting policy. A technique called safe policy improvement (SPI) provides a performance guarantee: with high probability, the new policy outperforms a given baseline policy, which is assumed to be safe. Orthogonally, in the context of safe RL, a shield provides a safety guarantee by restricting the action space to those actions that are provably safe with respect to a given safety-relevant model. We integrate these paradigms by extending shielding to offline RL, relying solely on the available dataset and knowledge of safe and unsafe states. Then, we shield the policy improvement steps, guaranteeing, with high probability, a safe policy. Experimental results demonstrate that shielded SPI outperforms its unshielded counterpart, improving both average and worst-case performance, particularly in low-data regimes.
Maris F. L. Galesloot, Thomas Rhemrev, Nils Jansen
Radboud University · Nijmegen, The Netherlands · Ruhr University & Radboud University +1