Deep reinforcement learning (DRL) agents are vulnerable to perturbations of their observed state inputs. Prior works often constrain state adversaries with Lp-norm uncertainty sets and evaluate policy performance under the resulting perturbations. However, in continuous control tasks like robotics control, different state dimensions have distinct physical meanings and influence policy behavior to different degrees, which Lp-norm sets fail to capture or explain. Moreover, it is important to identify when and along which state dimensions a policy is vulnerable, independently of the algorithm used to train it. In this paper, we focus on evaluation-time attacks applied to pretrained policies. We propose Performance-Degradation Non-Uniform Uncertainty Set (\method{}), an interpretable formulation of uncertainty sets for state perturbations. \method{} is state-dependent, dimension-wise non-uniform, and can be synthesized online during policy evaluation. We first formulate an ideal constrained optimization problem for \method{} computation, then introduce a surrogate constraint to make the problem tractable. Experiments across multiple continuous-control environments, policy algorithms, and state attacks show that \method{} reveals variation in policy sensitivity across states and dimensions, while its performance degradation is moderately controlled.
Figures & tables
Noise ( δ )
State Dimensions
Hopper Height
Torso Angle
Joint Angles
Torso Linear Velocities
Torso Angular Velocity
Joint Velocities
Thigh
Leg
Foot
X-Axis
Z-Axis
Thigh
Leg
Foot
0
3547.9
0.1
3282.5
3580.5
3486.3
3497.9
3499.9
3490.9
3435.4
3475.6
3579.6
3561.0
3580.4
0.2
2169.2
3575.7
3440.7
3540.5
3294.4
3245.6
3035.1
3395.5
3484.2
3472.8
2966.1
0.3
1501.0
3468.9
2918.6
3153.6
2632.2
3217.5
1967.7
1627.0
3373.2
3342.9
2523.7
Table 1: A toy experiment in which each state dimension is perturbed for a PPO ( schulman2017proximal ) policy trained on Hopper environment. To perturb the i -th state dimension, we apply constant noise, i.e. si←si+δ . We report the mean total reward over 20 episodes. For each value of noise δ , the most and least robust dimensions are shown in bold and underlined , respectively.
PPO
SA-PPO-SGLD
ATLA-PPO
WocaR-PPO
Hopper
0.15
0.25
0.0075
0.2
HalfCheetah
0.06
0.2
0.02
0.075
Walker2d
0.025
0.075
0.05
0.15
Table 2: The coefficient α of the KL divergence regularizer in Eq. ( 11 ) for each environment and algorithm during policy evaluation with PD-NUUS.
Set
Parameter
Attacker
Algorithm
PPO
SA-PPO-SGLD
ATLA-PPO
WocaR-PPO
No Attack
–
–
421.5 ± 57.5
349.4 ± 40.3
345.7 ± 24.9
331.4 ± 10.4
L∞ -Norm
ϵ=0.05
Random
415.0 ± 46.7 (6.5)
350.5 ± 40.7 (-1.1)
342.9 ± 21.0 (2.8)
331.3 ± 10.1 (0.1)
Critic
404.7 ± 72.6 (16.8)
350.4 ± 39.3 (-1.0)
344.3 ± 18.8 (1.4)
331.6 ± 10.7 (-0.2)
MAD
336.7 ± 47.7 (84.8)
349.9 ± 37.5 (-0.5)
339.1 ± 21.4 (6.6)
328.4 ± 9.3 (3.0)
PA-AD
338.3 ± 67.0 (83.2)
348.8 ± 38.6 (0.6)
313.1 ± 38.0 (32.6)
324.8 ± 7.1 (6.6)
Table 3: Comparison of L∞ -norm uncertainty sets and PD-NUUS on HalfCheetah. Each entry reports the discounted return, with degradation in parentheses. The largest degradation across attackers is in bold . The episode rewards of clean policies are 5275.1 ± 1586.0 (PPO), 4320.3 ± 611.3 (SA-PPO-SGLD), 4267.1 ± 854.5 (ATLA-PPO) and 3952.2 ± 566.3 (WocaR-PPO).
Environment
Algorithm
L∞ -Norm
PD-NUUS
ϵ=0.05
ϵ=0.10
ϵ=0.15
ϵ=0.20
β=5
β=10
β=20
β=40
Hopper
PPO
52.0
162.3
168.9
198.5
1.7
10.0
2.7
21.0
SA-PPO-SGLD
0.6
0.6
0.6
28.4
3.9
5.8
14.6
30.8
ATLA-PPO
17.3
78.7
103.0
136.8
1.0
2.5
24.5
127.7
WocaR-PPO
1.2
10.6
25.7
151.7
2.7
2.4
9.3
21.7
HalfCheetah
PPO
84.8
253.5
321.2
399.2
12.3
3.9
17.2
34.2
Table 4: Maximum empirical degradation in discounted return, observed among all state attackers for each environment, algorithm and uncertainty set parameter.
τ
0
0.1
0.2
0.25
C(τ)
77.08%
77.08%
81.25%
85.42%
Table 5: C(τ) values calculated across all PD-NUUS tests.
Figure 1: We report the mean and confidence range of the calculated budget function ri(s) for each state dimension across four algorithms and β∈{20,40} on Hopper under PA-AD attack. The statistics are computed over 200 sampled trajectories.
Environment
Algorithm
β=20
β=40
w/
w/o
w/
w/o
Hopper
PPO
2.7
228.7
21.0
241.4
SA-PPO-SGLD
14.6
55.0
30.8
72.4
ATLA-PPO
24.5
128.1
127.7
176.7
WocaR-PPO
9.3
134.5
21.7
183.2
HalfCheetah
PPO
17.2
285.3
34.2
341.7
Table 6: Empirical degradation J(π)−J(πν) in discounted total returns using PD-NUUS with (w/) and without (w/o) the KL divergence regularizer. With regularization, we report the maximum empirical degradation across four attackers; without regularization ( α=0 ), we report degradation under PA-AD. Even under this conservative comparison, degradation without regularization substantially exceeds β .
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Set
Parameter
Attacker
Algorithm
PPO
SA-PPO-SGLD
ATLA-PPO
WocaR-PPO
No Attack
–
–
269.2 ± 10.0
256.4 ± 6.5
257.5 ± 6.2
267.9 ± 4.5
L∞ -Norm
ϵ=0.05
Random
268.2 ± 9.1 (1.0)
256.1 ± 6.3 (0.3)
257.2 ± 5.8 (0.3)
268.0 ± 4.3 (-0.1)
Critic
269.0 ± 11.0 (0.2)
255.9 ± 6.2 (0.5)
258.3 ± 5.7 (-0.8)
268.4 ± 5.7 (-0.5)
MAD
265.9 ± 9.1 (3.3)
256.1 ± 5.5 (0.3)
254.8 ± 6.5 (2.7)
267.4 ± 4.3 (0.5)
PA-AD
217.2 ± 29.3 (52.0)
255.8 ± 4.8 (0.6)
240.2 ± 17.7 (17.3)
266.7 ± 2.1 (1.2)
Appendix
Table 7: Discounted returns ( γ=0.99 ) under state attackers on Hopper environment. When using PD-NUUS, the values of α for PPO, SA-PPO-SGLD, ATLA-PPO and WocaR-PPO are 0.15, 0.25, 0.0075 and 0.2, respectively. The values in parentheses represent degradation compared to no attack.
Set
Parameter
Attacker
Algorithm
PPO
SA-PPO-SGLD
ATLA-PPO
WocaR-PPO
No Attack
–
–
421.5 ± 57.5
349.4 ± 40.3
345.7 ± 24.9
331.4 ± 10.4
L∞ -Norm
ϵ=0.05
Random
415.0 ± 46.7 (6.5)
350.5 ± 40.7 (-1.1)
342.9 ± 21.0 (2.8)
331.3 ± 10.1 (0.1)
Critic
404.7 ± 72.6 (16.8)
350.4 ± 39.3 (-1.0)
344.3 ± 18.8 (1.4)
331.6 ± 10.7 (-0.2)
MAD
336.7 ± 47.7 (84.8)
349.9 ± 37.5 (-0.5)
339.1 ± 21.4 (6.6)
328.4 ± 9.3 (3.0)
PA-AD
338.3 ± 67.0 (83.2)
348.8 ± 38.6 (0.6)
313.1 ± 38.0 (32.6)
324.8 ± 7.1 (6.6)
Appendix
Table 8: Discounted returns ( γ=0.99 ) under state attackers on HalfCheetah environment. When using PD-NUUS, the value of α for PPO, SA-PPO-SGLD, ATLA-PPO and WocaR-PPO are 0.06, 0.2, 0.02 and 0.075, respectively. The values in parentheses represent degradation compared to no attack.
Set
Parameter
Attacker
Algorithm
PPO
SA-PPO-SGLD
ATLA-PPO
WocaR-PPO
No Attack
–
–
278.5 ± 8.5
298.3 ± 12.6
276.7 ± 14.0
287.9 ± 17.6
L∞ -Norm
ϵ=0.05
Random
278.2 ± 9.6 (0.3)
299.0 ± 12.7 (-0.7)
278.6 ± 11.5 (-1.9)
287.6 ± 17.8 (0.3)
Critic
275.9 ± 9.4 (2.6)
302.6 ± 10.8 (-4.3)
275.6 ± 16.3 (1.1)
289.5 ± 19.0 (-1.6)
MAD
265.1 ± 14.5 (13.4)
299.6 ± 10.4 (-1.3)
260.3 ± 10.4 (16.4)
286.9 ± 17.6 (1.0)
PA-AD
228.2 ± 44.6 (50.3)
297.3 ± 14.4 (1.0)
227.7 ± 35.1 (49.0)
285.0 ± 19.0 (2.9)
Appendix
Table 9: Discounted returns ( γ=0.99 ) under state attackers on Walker2d environment. When using PD-NUUS, the value of α for PPO, SA-PPO-SGLD, ATLA-PPO and WocaR-PPO are 0.025, 0.075, 0.05 and 0.15, respectively. The values in parentheses represent degradation compared to no attack.
Environment
Policy
α
Budget r(s) at rmax ( % )
Degradation in Discounted Return
Hopper
PPO
0
36.219
241.4
0.015
4.855
234.7
0.15 ∗
0.137
21.0
SA-PPO-SGLD
0
18.343
72.3
0.25 ∗
7.515
29.6
0.3
8.758
22.4
Appendix
Table 10: Percentage of perturbation budgets r(s) that reach the upper bound rmax under PA-AD attacks with PD-NUUS at β=40 . We compare the selected value (with ∗ ) of α with other α values. An inappropriate choice of α yields degradation in discounted total return substantially greater than β and cause a larger fraction of the learned budgets to saturate at rmax .
Figure 2: The mean and confidence range of calculated budget ri(s) of policies under PA-AD attack sampled from PD-NUUS with β∈{5,10,20,40} on Hopper. SA-PPO-SGLD, ATLA-PPO and WocaR-PPO admit higher perturbation tolerance than PPO in nearly all state dimensions.
Environment
β
PPO
SA-PPO-SGLD
ATLA-PPO
WocaR-PPO
Hopper
5
0.458±0.077
0.026±0.001
0.136±0.024
0.063±0.003
10
0.643±0.109
0.034±0.002
0.182±0.036
0.086±0.004
20
0.906±0.154
0.045±0.002
0.249±0.052
0.118±0.005
40
1.279±0.218
0.062±0.003
0.347±0.075
0.166±0.007
HalfCheetah
5
0.155±0.029
0.021±0.006
0.130±0.029
0.015±0.003
10
0.216±0.041
0.028±0.007
0.180±0.041
0.019±0.004
Appendix
Table 11: The mean and standard deviation of κ values estimated from sampled trajectories, when synthesizing PD-NUUS.
Environment
Attacker
β
PPO
SA-PPO-SGLD
ATLA-PPO
WocaR-PPO
Hopper
–
–
3471.7±391.7
3203.0±378.5
3227.5±634.9
3358.3±452.8
PD-NUUS
20
3112.9±757.8
1802.7±1311.8
1573.8±1111.0
2501.1±1173.0
PD-NUUS
40
2489.8±1287.3
1198.2±1085.3
924.3±1323.3
1663.1±1081.5
HalfCheetah
–
–
5275.1±1586.0
4320.3±611.3
4267.1±854.5
3952.2±566.3
PD-NUUS
20
4793.0±1810.3
4241.0±561.7
3930.9±954.0
2339.6±1389.2
PD-NUUS
40
4356.1±1986.7
3952.1±751.8
3730.3±971.5
2165.6±1193.5
Appendix
Table 12: Episode rewards E[∑t=0TR(st,at)] without attack and under PA-AD attack with PD-NUUS ( β∈{20,40} ).