Deep reinforcement learning (DRL) agents are vulnerable to perturbations of their observed state inputs. Prior works often constrain state adversaries with Lp-norm uncertainty sets and evaluate policy performance under the resulting perturbations. However, in continuous control tasks like robotics control, different state dimensions have distinct physical meanings and influence policy behavior to different degrees, which Lp-norm sets fail to capture or explain. Moreover, it is important to identify when and along which state dimensions a policy is vulnerable, independently of the algorithm used to train it. In this paper, we focus on evaluation-time attacks applied to pretrained policies. We propose Performance-Degradation Non-Uniform Uncertainty Set (\method{}), an interpretable formulation of uncertainty sets for state perturbations. \method{} is state-dependent, dimension-wise non-uniform, and can be synthesized online during policy evaluation. We first formulate an ideal constrained optimization problem for \method{} computation, then introduce a surrogate constraint to make the problem tractable. Experiments across multiple continuous-control environments, policy algorithms, and state attacks show that \method{} reveals variation in policy sensitivity across states and dimensions, while its performance degradation is moderately controlled.
Figures & tables
Noise ( δ )
State Dimensions
Hopper Height
Torso Angle
Joint Angles
Torso Linear Velocities
Torso Angular Velocity
Joint Velocities
Thigh
Leg
Foot
X-Axis
Z-Axis
Thigh
Leg
Foot
0
3547.9
0.1
3282.5
3580.5
3486.3
3497.9
3499.9
3490.9
3435.4
3475.6
3579.6
3561.0
3580.4
0.2
2169.2
3575.7
3440.7
3540.5
3294.4
3245.6
3035.1
3395.5
3484.2
3472.8
2966.1
0.3
1501.0
3468.9
2918.6
3153.6
2632.2
3217.5
1967.7
1627.0
3373.2
3342.9
2523.7
Table 1: A toy experiment in which each state dimension is perturbed for a PPO ( schulman2017proximal ) policy trained on Hopper environment. To perturb the i -th state dimension, we apply constant noise, i.e. si←si+δ . We report the mean total reward over 20 episodes. For each value of noise δ , the most and least robust dimensions are shown in bold and underlined , respectively.
PPO
SA-PPO-SGLD
ATLA-PPO
WocaR-PPO
Hopper
0.15
0.25
0.0075
0.2
HalfCheetah
0.06
0.2
0.02
0.075
Walker2d
0.025
0.075
0.05
0.15
Table 2: The coefficient α of the KL divergence regularizer in Eq. ( 11 ) for each environment and algorithm during policy evaluation with PD-NUUS.
Set
Parameter
Attacker
Algorithm
PPO
SA-PPO-SGLD
ATLA-PPO
WocaR-PPO
No Attack
–
–
421.5 ± 57.5
349.4 ± 40.3
345.7 ± 24.9
331.4 ± 10.4
L∞ -Norm
ϵ=0.05
Random
415.0 ± 46.7 (6.5)
350.5 ± 40.7 (-1.1)
342.9 ± 21.0 (2.8)
331.3 ± 10.1 (0.1)
Critic
404.7 ± 72.6 (16.8)
350.4 ± 39.3 (-1.0)
344.3 ± 18.8 (1.4)
331.6 ± 10.7 (-0.2)
MAD
336.7 ± 47.7 (84.8)
349.9 ± 37.5 (-0.5)
339.1 ± 21.4 (6.6)
328.4 ± 9.3 (3.0)
PA-AD
338.3 ± 67.0 (83.2)
348.8 ± 38.6 (0.6)
313.1 ± 38.0 (32.6)
324.8 ± 7.1 (6.6)
Table 3: Comparison of L∞ -norm uncertainty sets and PD-NUUS on HalfCheetah. Each entry reports the discounted return, with degradation in parentheses. The largest degradation across attackers is in bold . The episode rewards of clean policies are 5275.1 ± 1586.0 (PPO), 4320.3 ± 611.3 (SA-PPO-SGLD), 4267.1 ± 854.5 (ATLA-PPO) and 3952.2 ± 566.3 (WocaR-PPO).
Environment
Algorithm
L∞ -Norm
PD-NUUS
ϵ=0.05
ϵ=0.10
ϵ=0.15
ϵ=0.20
β=5
β=10
β=20
β=40
Hopper
PPO
52.0
162.3
168.9
198.5
1.7
10.0
2.7
21.0
SA-PPO-SGLD
0.6
0.6
0.6
28.4
3.9
5.8
14.6
30.8
ATLA-PPO
17.3
78.7
103.0
136.8
1.0
2.5
24.5
127.7
WocaR-PPO
1.2
10.6
25.7
151.7
2.7
2.4
9.3
21.7
HalfCheetah
PPO
84.8
253.5
321.2
399.2
12.3
3.9
17.2
34.2
Table 4: Maximum empirical degradation in discounted return, observed among all state attackers for each environment, algorithm and uncertainty set parameter.
τ
0
0.1
0.2
0.25
C(τ)
77.08%
77.08%
81.25%
85.42%
Table 5: C(τ) values calculated across all PD-NUUS tests.
Figure 1: We report the mean and confidence range of the calculated budget function ri(s) for each state dimension across four algorithms and β∈{20,40} on Hopper under PA-AD attack. The statistics are computed over 200 sampled trajectories.
Environment
Algorithm
β=20
β=40
w/
w/o
w/
w/o
Hopper
PPO
2.7
228.7
21.0
241.4
SA-PPO-SGLD
14.6
55.0
30.8
72.4
ATLA-PPO
24.5
128.1
127.7
176.7
WocaR-PPO
9.3
134.5
21.7
183.2
HalfCheetah
PPO
17.2
285.3
34.2
341.7
Table 6: Empirical degradation J(π)−J(πν) in discounted total returns using PD-NUUS with (w/) and without (w/o) the KL divergence regularizer. With regularization, we report the maximum empirical degradation across four attackers; without regularization ( α=0 ), we report degradation under PA-AD. Even under this conservative comparison, degradation without regularization substantially exceeds β .
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Set
Parameter
Attacker
Algorithm
PPO
SA-PPO-SGLD
ATLA-PPO
WocaR-PPO
No Attack
–
–
269.2 ± 10.0
256.4 ± 6.5
257.5 ± 6.2
267.9 ± 4.5
L∞ -Norm
ϵ=0.05
Random
268.2 ± 9.1 (1.0)
256.1 ± 6.3 (0.3)
257.2 ± 5.8 (0.3)
268.0 ± 4.3 (-0.1)
Critic
269.0 ± 11.0 (0.2)
255.9 ± 6.2 (0.5)
258.3 ± 5.7 (-0.8)
268.4 ± 5.7 (-0.5)
MAD
265.9 ± 9.1 (3.3)
256.1 ± 5.5 (0.3)
254.8 ± 6.5 (2.7)
267.4 ± 4.3 (0.5)
PA-AD
217.2 ± 29.3 (52.0)
255.8 ± 4.8 (0.6)
240.2 ± 17.7 (17.3)
266.7 ± 2.1 (1.2)
Appendix
Table 7: Discounted returns ( γ=0.99 ) under state attackers on Hopper environment. When using PD-NUUS, the values of α for PPO, SA-PPO-SGLD, ATLA-PPO and WocaR-PPO are 0.15, 0.25, 0.0075 and 0.2, respectively. The values in parentheses represent degradation compared to no attack.
Set
Parameter
Attacker
Algorithm
PPO
SA-PPO-SGLD
ATLA-PPO
WocaR-PPO
No Attack
–
–
421.5 ± 57.5
349.4 ± 40.3
345.7 ± 24.9
331.4 ± 10.4
L∞ -Norm
ϵ=0.05
Random
415.0 ± 46.7 (6.5)
350.5 ± 40.7 (-1.1)
342.9 ± 21.0 (2.8)
331.3 ± 10.1 (0.1)
Critic
404.7 ± 72.6 (16.8)
350.4 ± 39.3 (-1.0)
344.3 ± 18.8 (1.4)
331.6 ± 10.7 (-0.2)
MAD
336.7 ± 47.7 (84.8)
349.9 ± 37.5 (-0.5)
339.1 ± 21.4 (6.6)
328.4 ± 9.3 (3.0)
PA-AD
338.3 ± 67.0 (83.2)
348.8 ± 38.6 (0.6)
313.1 ± 38.0 (32.6)
324.8 ± 7.1 (6.6)
Appendix
Table 8: Discounted returns ( γ=0.99 ) under state attackers on HalfCheetah environment. When using PD-NUUS, the value of α for PPO, SA-PPO-SGLD, ATLA-PPO and WocaR-PPO are 0.06, 0.2, 0.02 and 0.075, respectively. The values in parentheses represent degradation compared to no attack.
Set
Parameter
Attacker
Algorithm
PPO
SA-PPO-SGLD
ATLA-PPO
WocaR-PPO
No Attack
–
–
278.5 ± 8.5
298.3 ± 12.6
276.7 ± 14.0
287.9 ± 17.6
L∞ -Norm
ϵ=0.05
Random
278.2 ± 9.6 (0.3)
299.0 ± 12.7 (-0.7)
278.6 ± 11.5 (-1.9)
287.6 ± 17.8 (0.3)
Critic
275.9 ± 9.4 (2.6)
302.6 ± 10.8 (-4.3)
275.6 ± 16.3 (1.1)
289.5 ± 19.0 (-1.6)
MAD
265.1 ± 14.5 (13.4)
299.6 ± 10.4 (-1.3)
260.3 ± 10.4 (16.4)
286.9 ± 17.6 (1.0)
PA-AD
228.2 ± 44.6 (50.3)
297.3 ± 14.4 (1.0)
227.7 ± 35.1 (49.0)
285.0 ± 19.0 (2.9)
Appendix
Table 9: Discounted returns ( γ=0.99 ) under state attackers on Walker2d environment. When using PD-NUUS, the value of α for PPO, SA-PPO-SGLD, ATLA-PPO and WocaR-PPO are 0.025, 0.075, 0.05 and 0.15, respectively. The values in parentheses represent degradation compared to no attack.
Environment
Policy
α
Budget r(s) at rmax ( % )
Degradation in Discounted Return
Hopper
PPO
0
36.219
241.4
0.015
4.855
234.7
0.15 ∗
0.137
21.0
SA-PPO-SGLD
0
18.343
72.3
0.25 ∗
7.515
29.6
0.3
8.758
22.4
Appendix
Table 10: Percentage of perturbation budgets r(s) that reach the upper bound rmax under PA-AD attacks with PD-NUUS at β=40 . We compare the selected value (with ∗ ) of α with other α values. An inappropriate choice of α yields degradation in discounted total return substantially greater than β and cause a larger fraction of the learned budgets to saturate at rmax .
Figure 2: The mean and confidence range of calculated budget ri(s) of policies under PA-AD attack sampled from PD-NUUS with β∈{5,10,20,40} on Hopper. SA-PPO-SGLD, ATLA-PPO and WocaR-PPO admit higher perturbation tolerance than PPO in nearly all state dimensions.
Environment
β
PPO
SA-PPO-SGLD
ATLA-PPO
WocaR-PPO
Hopper
5
0.458±0.077
0.026±0.001
0.136±0.024
0.063±0.003
10
0.643±0.109
0.034±0.002
0.182±0.036
0.086±0.004
20
0.906±0.154
0.045±0.002
0.249±0.052
0.118±0.005
40
1.279±0.218
0.062±0.003
0.347±0.075
0.166±0.007
HalfCheetah
5
0.155±0.029
0.021±0.006
0.130±0.029
0.015±0.003
10
0.216±0.041
0.028±0.007
0.180±0.041
0.019±0.004
Appendix
Table 11: The mean and standard deviation of κ values estimated from sampled trajectories, when synthesizing PD-NUUS.
Environment
Attacker
β
PPO
SA-PPO-SGLD
ATLA-PPO
WocaR-PPO
Hopper
–
–
3471.7±391.7
3203.0±378.5
3227.5±634.9
3358.3±452.8
PD-NUUS
20
3112.9±757.8
1802.7±1311.8
1573.8±1111.0
2501.1±1173.0
PD-NUUS
40
2489.8±1287.3
1198.2±1085.3
924.3±1323.3
1663.1±1081.5
HalfCheetah
–
–
5275.1±1586.0
4320.3±611.3
4267.1±854.5
3952.2±566.3
PD-NUUS
20
4793.0±1810.3
4241.0±561.7
3930.9±954.0
2339.6±1389.2
PD-NUUS
40
4356.1±1986.7
3952.1±751.8
3730.3±971.5
2165.6±1193.5
Appendix
Table 12: Episode rewards E[∑t=0TR(st,at)] without attack and under PA-AD attack with PD-NUUS ( β∈{20,40} ).
Reinforcement learning policies parametrized by deep neural networks have achieved strong performance for continuous control, yet even small input perturbations may lead to unpredictable behavior. This sensitivity limits their use in safety-critical domains, where robustness guarantees are required. Our work addresses this gap between state-of-the-art adversarial training methods and formal verification to train verifiably robust agents. Previous works train networks with individual adversarial perturbations, making them only robust against the specific adversarial attacks used. In contrast, our approach propagates entire perturbed input sets, enclosing all possible adversarial attacks within a single network pass. We leverage this to explicitly penalize the size of the output set (minimizing closed-loop uncertainty) and thereby make the actor robust against all possible attacks. This is realized by the use of set-based policy gradients, where each output within the set has a different gradient, thereby balancing the accuracy and robustness of the network. Doing so, we achieve formal verifiability across different verification frameworks for up to 9 times larger input perturbations compared to standard reinforcement learning and improve certified worst-case performance.
Significant progress has been made in safeguarding Deep Reinforcement Learning (DRL) policies against input perturbations. Developing robust DRL involves three main stages: algorithm design, implementation, and evaluation. In this work, we identify and address a key limitation at each stage. First, we introduce Adversarial Importance Sampling (Advis), a method that estimates and optimizes worst-case returns directly from trajectories already collected during standard training. Advis satisfies three desirable criteria that prior work does not jointly achieve: it requires no additional environment interactions, no auxiliary networks, and captures long-term robustness. Second, we introduce advrl, a modular PyTorch library that provides clean, single-file implementations of existing robustness methods and adversarial attacks, enabling rapid prototyping and reproducible evaluations. Third, we revisit evaluation under learned adversaries and show that optimal adversarial hyperparameters do not transfer across agents, which can lead to an overestimation of robustness when using a limited set of attacker configurations. Accordingly, we evaluate policies against a large and diverse set of attackers, using 6 to 14 times more configurations than prior work. Finally, experiments on continuous control environments show that Advis achieves stronger worst-case performance and greater training efficiency than existing baselines.
Amine Andam, Jamal Bentahar, Mustapha Hedabou
Mohammed VI Polytechnic University · Khalifa University · Concordia University
Regularization-based methods have become a standard approach for training Deep Reinforcement Learning policies against adversarial input perturbations. In this paper, we unify these methods by deriving new upper bounds on the performance gap between the nominal and worst-case policies. Each upper bound is expressed as an existing regularization objective plus a KL-divergence penalty between the nominal and worst-case policies, which further explains why adding a KL penalty improves robustness in practice. Building on these bounds, we formulate robust training as a constrained optimization problem, showing that existing methods correspond to the special case of a fixed Lagrange multiplier. We instead update the multiplier jointly with the policy to automatically tune the regularization weight. Finally, we conduct extensive adversarial evaluations across several continuous control tasks to validate our theoretical analysis.
Amine Andam, Jamal Bentahar, Mustapha Hedabou
Mohammed VI Polytechnic University · Khalifa University · Concordia University