Safe reinforcement learning (Safe RL) learns robotic controllers that optimize task rewards under safety constraints, yet observation perturbations can induce safety violations. Existing safety-directed attacks often require access to victim networks, gradients, critics, or explicit specifications -- assumptions rarely met once a controller is deployed as a black box. We propose a demonstration-guided observation attack for analyzing unknown Safe RL controllers. The framework recovers a state constraint and a surrogate policy through inverse constrained reinforcement learning, and learns dynamics from demonstration transitions. Their composed gradient generates bounded observation perturbations without victim parameters, gradients, or queries; demonstrations are the only victim-specific information. Across four Bullet tasks and one MetaDrive map with three budgets per victim, the attack exceeds every same-access baseline in 12 of 15 environment-budget conditions, and in 7 of those 12 it also exceeds the privileged reference attacks with access to the victim's reward and cost critics. Demonstrations released to support safe learning thus provide an attack surface for deployed black-box controllers. A defense study shows that state-adversarial regularization reduces attack cost, whereas the tested adversarial-training and demonstration-contamination schemes provide inconsistent protection.
Figures & tables
Fig. 1 : Demonstration-guided observation attack framework. (a) From safe expert demonstrations, recover a state constraint c^ϕ(o′) and a surrogate policy πICRL with ICRL using the attacker’s own learner rollouts as contrast, and fit one-step dynamics fψ . (b) Optimize a bounded observation perturbation through the composed objective c^ϕ(fψ(ot,πICRL(ot+δ))) ; the dynamics model keeps the clean ot as its state input. (c) Deliver the perturbed observation to the victim policy and evaluate the safety consequences of its executed actions. True safety cost is used only for evaluation.
Attack
Victim gradients
Victim critics
Victim queries
Safety spec.
Demos
Task env. interaction
Uniform / boundary-random-sign perturbations
No
No
No
No
No
No
Gradient-based MAD variants [ 11 , 5 ]
Yes
No
Yes
No
No
No
Liu MaxCost/MaxReward/AMAD [ 5 ]
Yes
Yes
Yes
No
No
No
STL-guided [ 6 ]
Yes
No
Yes
Yes
No
No
Substitute-model transfer [ 13 , 14 ]
No
No
Yes
No
No
No
BIA [ 12 ]
No
No
Yes
No
Yes
Yes
TABLE I : Information requirements of representative observation attacks. Victim gradients, critics, and queries all refer to access to the attacked policy. The final column indicates whether attacker training requires interaction with a task environment.
Demonstration fitting
Train dynamics by Eq. ( 5 ).
Constraint recovery
In each round, first run πICRL to collect learner trajectories, then update c^ϕ from expert and learner states using Eq. ( 4 ) and improve πICRL under the relabeled cost.
Freeze models
Save c^ϕ , fψ , and πICRL .
Each control step
Read ot , initialize u0=ot , and apply K updates using Eqs. ( 7 )–( 8 ).
Attack execution
Deliver uK to the victim and execute its action in the real environment.
Table 3
Fig. 2 : Simulation environments. Panels (a)–(d) show the four Bullet-Safety-Gym tasks and panel (e) the MetaDrive map.
Setting
Value
Expert demonstrations
20 episodes per victim; reset seeds 1200000–1200019; 6,000 transitions
Constraint network
128–128 ReLU; scalar linear output
Constraint update
512 expert + 512 learner states per step; Adam, learning rate 10−3
Constraint recovery
two alternating rounds; 6,000 learner interaction steps (20 episodes) per round; 100 constraint steps per round
64–64 ReLU; linear output; 16 training / 4 validation episodes; 200 epochs; Adam, learning rate 10−3 ; batch 256
TABLE II : Learning and attack settings for the main experiments.
Fig. 3 : Reward (top) and safety cost (bottom) under observation attacks on MetaDrive, CarCircle, CarRun, DroneRun, and DroneCircle versus the ℓ∞ budget in raw observation coordinates, spaced evenly within each column; cost axes differ per column. Asterisks mark victim-critic access; MAD-MSE-PGD and Liu MAD use victim-actor gradients; our attack uses only the attacker’s own models.
CarCircle
CarRun
DroneCircle
DroneRun
No attack
0.7
0.1
0.0
0.0
Uniform random
39.1
34.0
11.7
128.8
Scrambled constraint (mean of 3)
52.8
49.8
14.9
83.9
Recovered constraint
100.7
70.2
68.8
267.1
TABLE III : Constraint ablation at ϵ=0.10 : mean episode cost over 20 episodes per cell, differentiating through the victim actor.
Victim
CarCircle
DroneCircle
DroneRun
Vanilla
96.8
43.2
65.6
ADV-PPOL, max-reward
66.6
29.1
210.3
ADV-PPOL, max-cost
65.1
43.2
108.2
Uniform-noise training
71.9
46.7
259.2
SA-PPOL, kl
56.1
15.3
28.6
SA-PPOL, klmc
0.8
9.9
0.0
TABLE IV : Mean episode safety cost under our attack against defended victims at ϵ=0.10 (50 episodes per cell). Vanilla is the victim of Figure 3 ; —: excluded, clean task return near zero.
Fig. 4 : Physical deployment on the ROSMASTER R2. (a) Closed-loop driving on the indoor track. (b) Cost per segment: speed or yaw-rate violations plus steps not driven after an off-track exit (Section V-F ); bars are means, whiskers ±1 s.e.m., dots individual segments, and numbers below bars count off-track exits over segments. Methods left of the dashed line issue no victim queries beyond closed-loop execution (hollow bars: MAD objectives through the attacker’s surrogate); to the right, methods use victim-actor gradients (hatched: our objective) or the victim’s cost critic (Max-cost*).
Safe exploration remains a fundamental challenge in reinforcement learning (RL), limiting the deployment of RL agents in the real world. We propose Sampling-Based Safe Reinforcement Learning (SBSRL), a model-based RL algorithm that maintains safety throughout the learning process by enforcing constraints jointly across a finite set of dynamics samples. This formulation approximates an intractable worst-case optimization over uncertain dynamics and enables practical safety guarantees in continuous domains. We further introduce an exploration strategy based on constraining epistemic uncertainty, eliminating the need for explicit exploration bonuses. Under regularity conditions, we derive high-probability guarantees of safety throughout learning and a finite-time sample complexity bound for recovering a near-optimal policy. Empirically, SBSRL achieves safe and efficient exploration both in simulation and in real robotic hardware, and readily extends to practical deep-ensemble implementations that scale to high-dimensional continuous control problems.
Offline safe reinforcement learning (Safe RL) enables policy learning without online interactions, making it suitable for safety-critical systems such as robotics systems. However, its reliance on static datasets exposes offline Safe RL to data poisoning attacks, where adversaries inject malicious samples that compromise safety and induce unsafe policy behavior. In this work, we propose a new learning paradigm, named safe reinforcement unlearning (Safe-RULE), used as a defense framework to remove the influence of poisoned data without retraining from scratch or requiring access to the original training environment. We further extend reinforcement unlearning to offline Safe RL by explicitly accounting for both task performance and safety constraints during the unlearning process. Experiments across benchmark Safe RL tasks demonstrate that our approach effectively enhances safety performance against data poisoning attacks.
Shixiong Jiang, Taozheng Zhu, Fanxin Kong
Department of Computer Science and Engineering, University of Notre Dame
Inverse Reinforcement Learning (IRL) algorithms are powerful tools for learning from and generalizing expert demonstrations, but they often rely on unconstrained exploration, rendering them unsafe for real-world deployment. Meanwhile, Control Barrier Functions (CBFs) can guarantee the safety of control systems, but the analytical design of CBFs can be time-consuming and esoteric. In this work, we address these limitations jointly by constraining reward function candidacy during IRL to the space of CBFs, yielding a formulation that exhibits safe online control with continuous experiential improvement. Crucially, this framework enables the data-driven recovery of barrier functions directly from unlabeled expert observations. We demonstrate that the recovered barrier function is robust to unsafe states entirely absent from the expert data. Furthermore, we benchmark our method against standard IRL baselines in a simulated navigation environment, demonstrating improved safety performance. Finally, we investigate the trade-offs of planning-based versus policy-based IRL methods across both simulation and a real world obstacle avoidance task.