Safe reinforcement learning (Safe RL) learns robotic controllers that optimize task rewards under safety constraints, yet observation perturbations can induce safety violations. Existing safety-directed attacks often require access to victim networks, gradients, critics, or explicit specifications -- assumptions rarely met once a controller is deployed as a black box. We propose a demonstration-guided observation attack for analyzing unknown Safe RL controllers. The framework recovers a state constraint and a surrogate policy through inverse constrained reinforcement learning, and learns dynamics from demonstration transitions. Their composed gradient generates bounded observation perturbations without victim parameters, gradients, or queries; demonstrations are the only victim-specific information. Across four Bullet tasks and one MetaDrive map with three budgets per victim, the attack exceeds every same-access baseline in 12 of 15 environment-budget conditions, and in 7 of those 12 it also exceeds the privileged reference attacks with access to the victim's reward and cost critics. Demonstrations released to support safe learning thus provide an attack surface for deployed black-box controllers. A defense study shows that state-adversarial regularization reduces attack cost, whereas the tested adversarial-training and demonstration-contamination schemes provide inconsistent protection.
Figures & tables
Fig. 1 : Demonstration-guided observation attack framework. (a) From safe expert demonstrations, recover a state constraint c^ϕ(o′) and a surrogate policy πICRL with ICRL using the attacker’s own learner rollouts as contrast, and fit one-step dynamics fψ . (b) Optimize a bounded observation perturbation through the composed objective c^ϕ(fψ(ot,πICRL(ot+δ))) ; the dynamics model keeps the clean ot as its state input. (c) Deliver the perturbed observation to the victim policy and evaluate the safety consequences of its executed actions. True safety cost is used only for evaluation.
Attack
Victim gradients
Victim critics
Victim queries
Safety spec.
Demos
Task env. interaction
Uniform / boundary-random-sign perturbations
No
No
No
No
No
No
Gradient-based MAD variants [ 11 , 5 ]
Yes
No
Yes
No
No
No
Liu MaxCost/MaxReward/AMAD [ 5 ]
Yes
Yes
Yes
No
No
No
STL-guided [ 6 ]
Yes
No
Yes
Yes
No
No
Substitute-model transfer [ 13 , 14 ]
No
No
Yes
No
No
No
BIA [ 12 ]
No
No
Yes
No
Yes
Yes
TABLE I : Information requirements of representative observation attacks. Victim gradients, critics, and queries all refer to access to the attacked policy. The final column indicates whether attacker training requires interaction with a task environment.
Demonstration fitting
Train dynamics by Eq. ( 5 ).
Constraint recovery
In each round, first run πICRL to collect learner trajectories, then update c^ϕ from expert and learner states using Eq. ( 4 ) and improve πICRL under the relabeled cost.
Freeze models
Save c^ϕ , fψ , and πICRL .
Each control step
Read ot , initialize u0=ot , and apply K updates using Eqs. ( 7 )–( 8 ).
Attack execution
Deliver uK to the victim and execute its action in the real environment.
Table 3
Fig. 2 : Simulation environments. Panels (a)–(d) show the four Bullet-Safety-Gym tasks and panel (e) the MetaDrive map.
Setting
Value
Expert demonstrations
20 episodes per victim; reset seeds 1200000–1200019; 6,000 transitions
Constraint network
128–128 ReLU; scalar linear output
Constraint update
512 expert + 512 learner states per step; Adam, learning rate 10−3
Constraint recovery
two alternating rounds; 6,000 learner interaction steps (20 episodes) per round; 100 constraint steps per round
64–64 ReLU; linear output; 16 training / 4 validation episodes; 200 epochs; Adam, learning rate 10−3 ; batch 256
TABLE II : Learning and attack settings for the main experiments.
Fig. 3 : Reward (top) and safety cost (bottom) under observation attacks on MetaDrive, CarCircle, CarRun, DroneRun, and DroneCircle versus the ℓ∞ budget in raw observation coordinates, spaced evenly within each column; cost axes differ per column. Asterisks mark victim-critic access; MAD-MSE-PGD and Liu MAD use victim-actor gradients; our attack uses only the attacker’s own models.
CarCircle
CarRun
DroneCircle
DroneRun
No attack
0.7
0.1
0.0
0.0
Uniform random
39.1
34.0
11.7
128.8
Scrambled constraint (mean of 3)
52.8
49.8
14.9
83.9
Recovered constraint
100.7
70.2
68.8
267.1
TABLE III : Constraint ablation at ϵ=0.10 : mean episode cost over 20 episodes per cell, differentiating through the victim actor.
Victim
CarCircle
DroneCircle
DroneRun
Vanilla
96.8
43.2
65.6
ADV-PPOL, max-reward
66.6
29.1
210.3
ADV-PPOL, max-cost
65.1
43.2
108.2
Uniform-noise training
71.9
46.7
259.2
SA-PPOL, kl
56.1
15.3
28.6
SA-PPOL, klmc
0.8
9.9
0.0
TABLE IV : Mean episode safety cost under our attack against defended victims at ϵ=0.10 (50 episodes per cell). Vanilla is the victim of Figure 3 ; —: excluded, clean task return near zero.
Fig. 4 : Physical deployment on the ROSMASTER R2. (a) Closed-loop driving on the indoor track. (b) Cost per segment: speed or yaw-rate violations plus steps not driven after an off-track exit (Section V-F ); bars are means, whiskers ±1 s.e.m., dots individual segments, and numbers below bars count off-track exits over segments. Methods left of the dashed line issue no victim queries beyond closed-loop execution (hollow bars: MAD objectives through the attacker’s surrogate); to the right, methods use victim-actor gradients (hatched: our objective) or the victim’s cost critic (Max-cost*).