SafeAdapt: Provably Safe Policy Updates in Deep Reinforcement Learning

Maksim Anisimov; Francesco Belardinelli; Matthew Wicker

cs.LGApr 10, 2026

SafeAdapt: Provably Safe Policy Updates in Deep Reinforcement Learning

Authors: Maksim Anisimov, Francesco Belardinelli, Matthew Wicker

Organizations: Imperial College London · Imperial College London, London, United Kingdom

Abstract

Safety guarantees are a prerequisite to the deployment of reinforcement learning (RL) agents in safety-critical tasks. Often, deployment environments exhibit non-stationary dynamics or are subject to changing performance goals, requiring updates to the learned policy. This leads to a fundamental challenge: how to update an RL policy while preserving its safety properties on previously encountered tasks? The majority of current approaches either do not provide formal guarantees or verify policy safety only a posteriori. We propose a novel a priori approach to safe policy updates in continual RL by introducing the Rashomon set: a region in policy parameter space certified to meet safety constraints within the demonstration data distribution. We then show that one can provide formal, provable guarantees for arbitrary RL algorithms used to update a policy by projecting their updates onto the Rashomon set. Empirically, we validate this approach across grid-world navigation environments (Frozen Lake and Poisoned Apple) where we guarantee an a priori provably deterministic safety on the source task during downstream adaptation. In contrast, we observe that regularisation-based baselines experience catastrophic forgetting of safety constraints while our approach enables strong adaptation with provable guarantees that safety is preserved.

Figures & tables

Method	Safety structure	Formal?	Space	CL
EWC ( Kirkpatrick et al., 2017 )	None beyond source-task data	No	Parameters	Yes
Shielding ( Alshiekh et al., 2018 )	Abstract environment model	Yes	Action	No
SPoRt ( Cloete et al., 2025 )	Base policy + scenario-based safety data	Yes	Policy ratio	No
Zhang et al. (2024)	Explicit transition dynamics model	Yes	Action/state	No
SaGui ( Yang et al., 2023 )	Safety guidance / exploration constraints	No	Action	No
Held et al. (2017)	Learned damage model	Yes	Action	No

Table 1. Comparison of safety-aware adaptation methods. Safety structure : the type of prior safety/environment structure assumed by the method. Formal? : whether the method can provide a formal safety guarantee. Space : where the safety constraint is enforced. CL : whether the method addresses continual learning.

(5)

\theta_{\mathrm{down}}=\argmax_{\theta\,\in\,\Theta_{\mathrm{source}}^{\mathrm{safe}}}\;J_{\mathrm{down}}(\theta),

Algorithm 1 SafeAdapt : Safe Continual RL via Rashomon Set Computation

Panel of Figure 1 — Figure 1. Overview of the proposed method. The unsafety labelling function $U_{\mathrm{source}}$ yields safe demonstrations $D_{\mathrm{source}}^{\mathrm{safe}}$ (Phase 1) . A certified safe parameter region $\Theta_{\mathrm{source}}^{\mathrm{safe}}$ is computed around $\pi_{\mathrm{source}}$ (Phase 2) , and the downstream policy $\pi_{\mathrm{down}}$ is adapted within this region (Phase 3) .

Figure 2. Safe demonstration dataset construction in Frozen Lake environment. We label state-action pairs that lead to ice holes as unsafe. Then, we derive a demonstration of safe state-action pairs, which exclude states without unsafe actions.

Figure 3. Examples of experiment environments: Frozen Lake standard_4x4 (first row) and Poisoned Apple simple_5x5 (second row).

Setting	Frozen Lake	Poisoned Apple
Task ID in state representation	Yes	No
Termination when unsafe	Yes	No
Single destination	Yes	No

Table 2. Frozen Lake vs Poisoned Apple experiment structure.

Environment	Policy	Provably Safe?	$\Phi^{\mathrm{safe}}_{\mathrm{sc}}(\pi)$	$\Phi^{\mathrm{safe}}_{\mathrm{traj.}}(\pi)$	Total Reward
Frozen Lake ( standard_4x4 )	Source	✓	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$
	UnsafeAdapt	$\times$	$0.88\pm 0.00$	$0.10\pm 0.30$	$0.00\pm 0.00$
	EWC	$\times$	$0.94\pm 0.06$	$0.60\pm 0.49$	$0.30\pm 0.46$
	SafeAdapt (ours)	✓	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$	$0.90\pm 0.30$
Poisoned Apple ( simple_5x5 )	Source	✓	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.96\pm 0.00}$
	UnsafeAdapt	$\times$	$0.93\pm 0.11$	$\mathbf{1.00\pm 0.00}$	$0.91\pm 0.00$

Table 3. Source Task safety and performance metrics in Frozen Lake and Poisoned Apple (mean

\pm

standard deviation over 10 seeds). We report (i) critical state safety rate

\Phi^{\mathrm{safe}}_{\mathrm{sc}}(\pi)

, measuring the fraction of safety-critical states in which the policy selects only safe actions, (ii) trajectory-level safety

\Phi^{\mathrm{safe}}_{\mathrm{traj.}}(\pi)

, measuring the probability of executing a fully safe trajectory, and (iii) total reward. Standard continual learning baselines (UnsafeAdapt and EWC) may violate safety constraints, as reflected by degraded

\Phi^{\mathrm{safe}}_{\mathrm{sc}}

in both Frozen Lake and Poisoned Apple and degraded

\Phi^{\mathrm{safe}}_{\mathrm{traj.}}

in Frozen Lake. In contrast, SafeAdapt (ours) consistently retains perfect safety (

\Phi^{\mathrm{safe}}_{\mathrm{sc}}=\Phi^{\mathrm{safe}}_{\mathrm{traj.}}=1

) across all environments while maintaining competitive total reward in the source task.

Environment	Policy	Total Reward	Success Rate
Frozen Lake ( standard_4x4 )	Source	$0.00\pm 0.00$	$0.00\pm 0.00$
	UnsafeAdapt	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$
	EWC	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$
	SafeAdapt (ours)	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$
Poisoned Apple ( simple_5x5 )	Source	$-0.04\pm 0.00$	$\mathbf{1.00\pm 0.00}$
	UnsafeAdapt	$\mathbf{0.96\pm 0.00}$	$\mathbf{1.00\pm 0.00}$

Table 4. Downstream Task performance metrics in Frozen Lake and Poisoned Apple (mean

\pm

standard deviation over 10 seeds).

Figure 4. Scalability analysis across diagonal Frozen Lake layouts.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5. Relationship between the safety surrogate and the hard safety specification.

Setting	Frozen Lake	Poisoned Apple
Environment	FrozenLake-v1 with custom wrappers	PoisonedAppleEnv (custom Gymnasium environment)
State representation	One-hot-encoded position over grid cells and appended task ID	Flat grid vector with the following entries: 0=empty, 1=agent, 2=safe apple, 3=poisoned apple
Action space	Discrete(4) : Left, Down, Right, Up	Discrete(4) : Left, Down, Right, Up
Unsafe event	Entering a hole tile ( H )	Stepping onto a poisoned apple ( info["safe"]=False , cost=1 )
Termination on unsafe event?	Yes	No
What changes from Task 1 to Task 2?	Positions of ice holes	Safe/poisoned apple positions

Appendix

Table 5. Frozen Lake vs. Poisoned Apple experiment setup.

Panel of Figure 6 — Figure 6. Environment frames at initial states.

Setting	Frozen Lake	Poisoned Apple
Source PPO max timesteps	500,000	20,000
PPO eval episodes	1	1
Actor/Critic architecture	MLP 64-64	MLP 256-256
Rollout steps	256	256
Update epochs	8	6
Mini-batch size	64	64

Appendix

Table 6. Source-task policy training and safety finetuning settings.

Setting	Frozen Lake	Poisoned Apple
Rashomon dataset	Task-1 safety-critical states	Task-1 safety dataset
Label type / aggregation	Multi-label safe-action masks; aggregation= min	Multi-label safe-action masks; aggregation= min
Rashomon iterations ( n_iters )	5,000	20,000
Surrogate threshold	$\max_{s}\|\mathcal{A}^{\mathrm{safe}}(s)\|/(1+\max_{s}\|\mathcal{A}^{\mathrm{safe}}(s)\|)$	Same formula
Inverse-temperature search	Smallest $T\in[10,1000]$ satisfying surrogate mass constraint	Smallest $T\in[10,1000]$ satisfying surrogate mass constraint
s min_acc_limit	Surrogate threshold	Surrogate threshold

Appendix

Table 7. Rashomon set computation settings.

Setting	Frozen Lake	Poisoned Apple
Downstream max timesteps	50,000	20,000
Entropy coef	0.1	0.01
Learning rate	$3\times 10^{-4}$ (PPO default)	$3\times 10^{-4}$ (PPO default)
Rollout / epochs / minibatch	2048 / 10 / 64	2048 / 10 / 64
$\gamma$ / GAE $\lambda$ / clip / vf / grad norm	0.99 / 0.95 / 0.2 / 0.5 / 0.5	0.99 / 0.95 / 0.2 / 0.5 / 0.5
Eval episodes	1	1

Appendix

Table 8. Downstream adaptation PPO settings for UnsafeAdapt and SafeAdapt .

Setting	Frozen Lake	Poisoned Apple
EWC $\lambda$	5000	5000
Fisher data source	Source training states	Source training states
Fisher sample-size cap	$\min(1000,N_{\text{source states}})$	$\min(1000,N_{\text{source states}})$
Compute critic Fisher?	No	No
Apply EWC to critic during training?	No (default)	No (explicit)
EWC adaptation timesteps	50,000	20,000

Appendix

Table 9. EWC-PPO settings.

Environment	Policy	Provably Safe?	$\Phi_{\mathrm{sc}}(\pi)$	$\Phi_{\mathrm{traj.}}(\pi)$	Total Reward
diagonal_4x4	Source	✓	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$
	UnsafeAdapt	$\times$	$0.53\pm 0.07$	$0.00\pm 0.00$	$0.00\pm 0.00$
	EWC	$\times$	$0.67\pm 0.20$	$0.30\pm 0.46$	$0.30\pm 0.46$
	SafeAdapt (ours)	✓	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$
diagonal_6x6	Source	✓	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$
	UnsafeAdapt	$\times$	$0.52\pm 0.11$	$0.00\pm 0.00$	$0.00\pm 0.00$

Appendix

Table 10. Scalability analysis: Frozen Lake Task 1 results across configurations (mean

\pm

std over 10 seeds).

Environment	Policy	Total Reward	Success Rate
diagonal_4x4	Source	$0.00\pm 0.00$	$0.00\pm 0.00$
	UnsafeAdapt	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$
	EWC	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$
	SafeAdapt (ours)	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$
diagonal_6x6	Source	$0.00\pm 0.00$	$0.00\pm 0.00$
	UnsafeAdapt	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$

Appendix

Table 11. Scalability analysis: Frozen Lake Task 2 results across configurations (mean

\pm

std over 10 seeds).

Figure 7. Logit bounds illustrate the following guarantee of the Rashomon set: in any state, there is a safe action whose lower bound logit is greater than upper bound of any unsafe action’s logit. States are represented using their x and y coordinates in the grid world.

Figure 8. Neural policy probabilities for the worst-case logit vector in the source task of Frozen Lake ( standard 4x4 ).

Safe RL

Safe reinforcement learning studies how agents can learn and act while respecting safety constraints and limiting harmful outcomes.

See all 209 papers

Provably Safe, Yet Scalable Reinforcement Learning
Jun 12, 2026Kai S. Yun, Zeyang Li, Navid AzizanRL Control Safety-Critical Control
MIT
Safe Continual Reinforcement Learning in Non-stationary Environments
Apr 21, 2026Austin Coursey, Abel Diaz-Gonzalez, Marcos Quinones-Grueiro +1RL Control Non-Stationary RL
Institute for Software Integrated Systems, Vanderbilt University
Sampling-Based Safe Reinforcement Learning
May 19, 2026Luca Vignola, Bruce D. Lee, Manish Prajapat +4Robot Safety Safe RL
ETH Zurich

Jun 12, 2026J/K move · Enter open · S save

Provably Safe, Yet Scalable Reinforcement Learning

Kai S. Yun, Zeyang Li, Navid Azizan

MIT

RL Control Safety-Critical Control Robotic RL Safety Filtering Safe RL

Open paper AskPDF arXiv

Safe reinforcement learning (RL) aims to learn policies that optimize rewards while satisfying constraints. Predominant approaches rely on soft-constrained policy optimization, which has achieved empirical success but does not provide formal safety guarantees for the learned policy. In contrast, methods with strict guarantees typically rely on explicit certificate functions, whose construction requires the direct synthesis and verification of control-invariant sets, a process that scales poorly with state dimension and often yields overly conservative behavior. In this paper, we present the Provably Safe, yet Scalable RL (PS2-RL) framework, a novel two-phase architecture for learning provably safe policies in a scalable manner, designed to overcome the key bottlenecks of prior methods. Rather than explicitly computing invariant sets, PS2-RL leverages a learned backup policy to forward-integrate the system dynamics, generating an implicit control-invariant set online. In the first phase, the backup policy is trained with our proposed safe-arrival value function, which characterizes the optimal backup policy for invariant-set construction. In the second phase, an RL policy is trained end-to-end through a differentiable projection layer that strictly enforces the safety guarantees induced by the learned backup policy. By maximizing the volume of the implicit control-invariant set in the first phase, the resulting PS2 policy from the second phase is performant and scalable, while maintaining provable safety. Crucially, PS2-RL imposes no restrictions on the underlying RL algorithm and can be plugged into any existing training pipeline. We establish theoretical guarantees for the proposed framework and evaluate it on robotic control tasks with state dimensions up to 10, a regime in which prior provably safe RL methods struggle or become impractical.

Abstract

Safe RL

Explore similar work

Provably Safe, Yet Scalable Reinforcement Learning

Safe Continual Reinforcement Learning in Non-stationary Environments

Sampling-Based Safe Reinforcement Learning