Safety guarantees are a prerequisite to the deployment of reinforcement learning (RL) agents in safety-critical tasks. Often, deployment environments exhibit non-stationary dynamics or are subject to changing performance goals, requiring updates to the learned policy. This leads to a fundamental challenge: how to update an RL policy while preserving its safety properties on previously encountered tasks? The majority of current approaches either do not provide formal guarantees or verify policy safety only a posteriori. We propose a novel a priori approach to safe policy updates in continual RL by introducing the Rashomon set: a region in policy parameter space certified to meet safety constraints within the demonstration data distribution. We then show that one can provide formal, provable guarantees for arbitrary RL algorithms used to update a policy by projecting their updates onto the Rashomon set. Empirically, we validate this approach across grid-world navigation environments (Frozen Lake and Poisoned Apple) where we guarantee an a priori provably deterministic safety on the source task during downstream adaptation. In contrast, we observe that regularisation-based baselines experience catastrophic forgetting of safety constraints while our approach enables strong adaptation with provable guarantees that safety is preserved.
Figures & tables
Method
Safety structure
Formal?
Space
CL
EWC ( Kirkpatrick et al., 2017 )
None beyond source-task data
No
Parameters
Yes
Shielding ( Alshiekh et al., 2018 )
Abstract environment model
Yes
Action
No
SPoRt ( Cloete et al., 2025 )
Base policy + scenario-based safety data
Yes
Policy ratio
No
Zhang et al. (2024)
Explicit transition dynamics model
Yes
Action/state
No
SaGui ( Yang et al., 2023 )
Safety guidance / exploration constraints
No
Action
No
Held et al. (2017)
Learned damage model
Yes
Action
No
Table 1. Comparison of safety-aware adaptation methods. Safety structure : the type of prior safety/environment structure assumed by the method. Formal? : whether the method can provide a formal safety guarantee. Space : where the safety constraint is enforced. CL : whether the method addresses continual learning.
(5)
θdown=argmaxθ∈ΘsourcesafeJdown(θ),
Algorithm 1 SafeAdapt : Safe Continual RL via Rashomon Set Computation
Figure 1. Overview of the proposed method. The unsafety labelling function Usource yields safe demonstrations Dsourcesafe (Phase 1) . A certified safe parameter region Θsourcesafe is computed around πsource (Phase 2) , and the downstream policy πdown is adapted within this region (Phase 3) .
Figure 2. Safe demonstration dataset construction in Frozen Lake environment. We label state-action pairs that lead to ice holes as unsafe. Then, we derive a demonstration of safe state-action pairs, which exclude states without unsafe actions.
Figure 3. Examples of experiment environments: Frozen Lake standard_4x4 (first row) and Poisoned Apple simple_5x5 (second row).
Setting
Frozen Lake
Poisoned Apple
Task ID in state representation
Yes
No
Termination when unsafe
Yes
No
Single destination
Yes
No
Table 2. Frozen Lake vs Poisoned Apple experiment structure.
Environment
Policy
Provably Safe?
Φscsafe(π)
Φtraj.safe(π)
Total Reward
Frozen Lake ( standard_4x4 )
Source
✓
1.00±0.00
1.00±0.00
1.00±0.00
UnsafeAdapt
×
0.88±0.00
0.10±0.30
0.00±0.00
EWC
×
0.94±0.06
0.60±0.49
0.30±0.46
SafeAdapt (ours)
✓
1.00±0.00
1.00±0.00
0.90±0.30
Poisoned Apple ( simple_5x5 )
Source
✓
1.00±0.00
1.00±0.00
1.96±0.00
UnsafeAdapt
×
0.93±0.11
1.00±0.00
0.91±0.00
Table 3. Source Task safety and performance metrics in Frozen Lake and Poisoned Apple (mean ± standard deviation over 10 seeds). We report (i) critical state safety rate Φscsafe(π) , measuring the fraction of safety-critical states in which the policy selects only safe actions, (ii) trajectory-level safety Φtraj.safe(π) , measuring the probability of executing a fully safe trajectory, and (iii) total reward. Standard continual learning baselines (UnsafeAdapt and EWC) may violate safety constraints, as reflected by degraded Φscsafe in both Frozen Lake and Poisoned Apple and degraded Φtraj.safe in Frozen Lake. In contrast, SafeAdapt (ours) consistently retains perfect safety ( Φscsafe=Φtraj.safe=1 ) across all environments while maintaining competitive total reward in the source task.
Environment
Policy
Total Reward
Success Rate
Frozen Lake ( standard_4x4 )
Source
0.00±0.00
0.00±0.00
UnsafeAdapt
1.00±0.00
1.00±0.00
EWC
1.00±0.00
1.00±0.00
SafeAdapt (ours)
1.00±0.00
1.00±0.00
Poisoned Apple ( simple_5x5 )
Source
−0.04±0.00
1.00±0.00
UnsafeAdapt
0.96±0.00
1.00±0.00
Table 4. Downstream Task performance metrics in Frozen Lake and Poisoned Apple (mean ± standard deviation over 10 seeds).
Figure 4. Scalability analysis across diagonal Frozen Lake layouts.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5. Relationship between the safety surrogate and the hard safety specification.
Setting
Frozen Lake
Poisoned Apple
Environment
FrozenLake-v1 with custom wrappers
PoisonedAppleEnv (custom Gymnasium environment)
State representation
One-hot-encoded position over grid cells and appended task ID
Flat grid vector with the following entries: 0=empty, 1=agent, 2=safe apple, 3=poisoned apple
Action space
Discrete(4) : Left, Down, Right, Up
Discrete(4) : Left, Down, Right, Up
Unsafe event
Entering a hole tile ( H )
Stepping onto a poisoned apple ( info["safe"]=False , cost=1 )
Termination on unsafe event?
Yes
No
What changes from Task 1 to Task 2?
Positions of ice holes
Safe/poisoned apple positions
Appendix
Table 5. Frozen Lake vs. Poisoned Apple experiment setup.
Figure 6. Environment frames at initial states.
Setting
Frozen Lake
Poisoned Apple
Source PPO max timesteps
500,000
20,000
PPO eval episodes
1
1
Actor/Critic architecture
MLP 64-64
MLP 256-256
Rollout steps
256
256
Update epochs
8
6
Mini-batch size
64
64
Appendix
Table 6. Source-task policy training and safety finetuning settings.
Setting
Frozen Lake
Poisoned Apple
Rashomon dataset
Task-1 safety-critical states
Task-1 safety dataset
Label type / aggregation
Multi-label safe-action masks; aggregation= min
Multi-label safe-action masks; aggregation= min
Rashomon iterations ( n_iters )
5,000
20,000
Surrogate threshold
maxs∣Asafe(s)∣/(1+maxs∣Asafe(s)∣)
Same formula
Inverse-temperature search
Smallest T∈[10,1000] satisfying surrogate mass constraint
Smallest T∈[10,1000] satisfying surrogate mass constraint
s min_acc_limit
Surrogate threshold
Surrogate threshold
Appendix
Table 7. Rashomon set computation settings.
Setting
Frozen Lake
Poisoned Apple
Downstream max timesteps
50,000
20,000
Entropy coef
0.1
0.01
Learning rate
3×10−4 (PPO default)
3×10−4 (PPO default)
Rollout / epochs / minibatch
2048 / 10 / 64
2048 / 10 / 64
γ / GAE λ / clip / vf / grad norm
0.99 / 0.95 / 0.2 / 0.5 / 0.5
0.99 / 0.95 / 0.2 / 0.5 / 0.5
Eval episodes
1
1
Appendix
Table 8. Downstream adaptation PPO settings for UnsafeAdapt and SafeAdapt .
Setting
Frozen Lake
Poisoned Apple
EWC λ
5000
5000
Fisher data source
Source training states
Source training states
Fisher sample-size cap
min(1000,Nsource states)
min(1000,Nsource states)
Compute critic Fisher?
No
No
Apply EWC to critic during training?
No (default)
No (explicit)
EWC adaptation timesteps
50,000
20,000
Appendix
Table 9. EWC-PPO settings.
Environment
Policy
Provably Safe?
Φsc(π)
Φtraj.(π)
Total Reward
diagonal_4x4
Source
✓
1.00±0.00
1.00±0.00
1.00±0.00
UnsafeAdapt
×
0.53±0.07
0.00±0.00
0.00±0.00
EWC
×
0.67±0.20
0.30±0.46
0.30±0.46
SafeAdapt (ours)
✓
1.00±0.00
1.00±0.00
1.00±0.00
diagonal_6x6
Source
✓
1.00±0.00
1.00±0.00
1.00±0.00
UnsafeAdapt
×
0.52±0.11
0.00±0.00
0.00±0.00
Appendix
Table 10. Scalability analysis: Frozen Lake Task 1 results across configurations (mean ± std over 10 seeds).
Environment
Policy
Total Reward
Success Rate
diagonal_4x4
Source
0.00±0.00
0.00±0.00
UnsafeAdapt
1.00±0.00
1.00±0.00
EWC
1.00±0.00
1.00±0.00
SafeAdapt (ours)
1.00±0.00
1.00±0.00
diagonal_6x6
Source
0.00±0.00
0.00±0.00
UnsafeAdapt
1.00±0.00
1.00±0.00
Appendix
Table 11. Scalability analysis: Frozen Lake Task 2 results across configurations (mean ± std over 10 seeds).
Figure 7. Logit bounds illustrate the following guarantee of the Rashomon set: in any state, there is a safe action whose lower bound logit is greater than upper bound of any unsafe action’s logit. States are represented using their x and y coordinates in the grid world.
Figure 8. Neural policy probabilities for the worst-case logit vector in the source task of Frozen Lake ( standard 4x4 ).
Safe reinforcement learning (RL) aims to learn policies that optimize rewards while satisfying constraints. Predominant approaches rely on soft-constrained policy optimization, which has achieved empirical success but does not provide formal safety guarantees for the learned policy. In contrast, methods with strict guarantees typically rely on explicit certificate functions, whose construction requires the direct synthesis and verification of control-invariant sets, a process that scales poorly with state dimension and often yields overly conservative behavior. In this paper, we present the Provably Safe, yet Scalable RL (PS2-RL) framework, a novel two-phase architecture for learning provably safe policies in a scalable manner, designed to overcome the key bottlenecks of prior methods. Rather than explicitly computing invariant sets, PS2-RL leverages a learned backup policy to forward-integrate the system dynamics, generating an implicit control-invariant set online. In the first phase, the backup policy is trained with our proposed safe-arrival value function, which characterizes the optimal backup policy for invariant-set construction. In the second phase, an RL policy is trained end-to-end through a differentiable projection layer that strictly enforces the safety guarantees induced by the learned backup policy. By maximizing the volume of the implicit control-invariant set in the first phase, the resulting PS2 policy from the second phase is performant and scalable, while maintaining provable safety. Crucially, PS2-RL imposes no restrictions on the underlying RL algorithm and can be plugged into any existing training pipeline. We establish theoretical guarantees for the proposed framework and evaluate it on robotic control tasks with state dimensions up to 10, a regime in which prior provably safe RL methods struggle or become impractical.
Reinforcement learning (RL) offers a compelling data-driven paradigm for synthesizing controllers for complex systems when accurate physical models are unavailable; however, most existing control-oriented RL methods assume stationarity and, therefore, struggle in real-world non-stationary deployments where system dynamics and operating conditions can change unexpectedly. Moreover, RL controllers acting in physical environments must satisfy safety constraints throughout their learning and execution phases, rendering transient violations during adaptation unacceptable. Although continual RL and safe RL have each addressed non-stationarity and safety, respectively, their intersection remains comparatively unexplored, motivating the study of safe continual RL algorithms that can adapt over the system's lifetime while preserving safety. In this work, we systematically investigate safe continual reinforcement learning by introducing three benchmark environments that capture safety-critical continual adaptation and by evaluating representative approaches from safe RL, continual RL, and their combinations. Our empirical results reveal a fundamental tension between maintaining safety constraints and preventing catastrophic forgetting under non-stationary dynamics, with existing methods generally failing to achieve both objectives simultaneously. To address this shortcoming, we examine regularization-based strategies that partially mitigate this trade-off and characterize their benefits and limitations. Finally, we outline key open challenges and research directions toward developing safe, resilient learning-based controllers capable of sustained autonomous operation in changing environments.
Austin Coursey, Abel Diaz-Gonzalez, Marcos Quinones-Grueiro +1
Institute for Software Integrated Systems, Vanderbilt University
Safe exploration remains a fundamental challenge in reinforcement learning (RL), limiting the deployment of RL agents in the real world. We propose Sampling-Based Safe Reinforcement Learning (SBSRL), a model-based RL algorithm that maintains safety throughout the learning process by enforcing constraints jointly across a finite set of dynamics samples. This formulation approximates an intractable worst-case optimization over uncertain dynamics and enables practical safety guarantees in continuous domains. We further introduce an exploration strategy based on constraining epistemic uncertainty, eliminating the need for explicit exploration bonuses. Under regularity conditions, we derive high-probability guarantees of safety throughout learning and a finite-time sample complexity bound for recovering a near-optimal policy. Empirically, SBSRL achieves safe and efficient exploration both in simulation and in real robotic hardware, and readily extends to practical deep-ensemble implementations that scale to high-dimensional continuous control problems.