Advantage-based Temporal Attack in Reinforcement Learning

Shenghong He

cs.LGFeb 23, 2026

Advantage-based Temporal Attack in Reinforcement Learning

Authors: Shenghong He

Organizations: the School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou 510006, China

Abstract

Extensive research demonstrates that Deep Reinforcement Learning (DRL) models are susceptible to adversarially constructed inputs (i.e., adversarial examples), which can mislead the agent to take suboptimal or unsafe actions. Recent methods improve attack effectiveness by leveraging future rewards to guide adversarial perturbation generation over sequential time steps (i.e., reward-based attacks). However, these methods are unable to capture dependencies between different time steps in the perturbation generation process, resulting in a weak temporal correlation between the current perturbation and previous perturbations.In this paper, we propose a novel method called Advantage-based Adversarial Transformer (AAT), which can generate adversarial examples with stronger temporal correlations (i.e., time-correlated adversarial examples) to improve the attack performance. AAT employs a multi-scale causal self-attention (MSCSA) mechanism to dynamically capture dependencies between historical information from different time periods and the current state, thus enhancing the correlation between the current perturbation and the previous perturbation. Moreover, AAT introduces a weighted advantage mechanism, which quantifies the effectiveness of a perturbation in a given state and guides the generation process toward high-performance adversarial examples by sampling high-advantage regions. Extensive experiments demonstrate that the performance of AAT matches or surpasses mainstream adversarial attack baselines on Atari, DeepMind Control Suite and Google football tasks.

Figures & tables

notation	Explanation
$\mathbf{M}$	Markov Decision Process
$S,\mathbb{A},\mathcal{R},\mathcal{P},\gamma$	State space, action space, reward function, transition probaility, discount factor
$\pi$ , $\tau$ , $P_{\pi}(\tau)$	Policy, trajectory, trajectory distribution under policy $\pi$
$J(\pi)$	Expected cumulative reward
$\delta_{t}$ ,	Adversarial perturbation at time $t$
$s^{\prime}_{t}$	Adversarial state, $s^{\prime}_{t}=s_{t}+\delta_{t}$

TABLE I: Symbol explanation.

Fig. 1: The AAT training structure.

\oplus

denotes the concatenation of vectors,

\otimes

denotes the element-wise multiplication of vectors and MLP is a multilayer perception machine. During the training phase, AAT generates adversarial perturbations using the weighted advantage calculated from the

Q_{\theta}

and

V_{\psi}

value functions.

Breakout
	White-box				Black-box
Method	DQN	A3C	TRPO	PPO	DQN	A3C	TRPO	PPO
* FGSM Skip S-T EDGE PA-AD AdvRL-GAN TSGE PIA AAT	355.83 $\pm$ 0.89 34.30 $\pm$ 3.78 31.45 $\pm$ 4.15 57.68 $\pm$ 5.61 43.26 $\pm$ 3.51 46.35 $\pm$ 4.31 32.44 $\pm$ 5.42 6.42 $\pm$ 4.21 35.17 $\pm$ 4.44 6.56 $\pm$ 2.54	384.52 $\pm$ 1.45 35.45 $\pm$ 4.05 32.46 $\pm$ 4.33 55.44 $\pm$ 2.78 46.32 $\pm$ 3.49 53.87 $\pm$ 3.27 9.11 $\pm$ 2.56 15.29 $\pm$ 3.42 27.84 $\pm$ 4.23 8.21 $\pm$ 3.51	415.64 $\pm$ 2.73 34.58 $\pm$ 4.54 38.79 $\pm$ 3.98 58.47 $\pm$ 3.12 44.89 $\pm$ 4.27 47.96 $\pm$ 4.32 33.98 $\pm$ 3.75 25.46 $\pm$ 4.21 40.51 $\pm$ 3.41 5.98 $\pm$ 3.81	426.58 $\pm$ 2.14 40.56 $\pm$ 3.82 38.42 $\pm$ 4.08 44.36 $\pm$ 3.84 50.33 $\pm$ 4.35 65.73 $\pm$ 3.91 38.21 $\pm$ 2.74 34.32 $\pm$ 3.27 53.47 $\pm$ 2.83 6.88 $\pm$ 2.56	355.83 $\pm$ 0.89 118.32 $\pm$ 8.32 88.46 $\pm$ 7.74 68.32 $\pm$ 5.24 56.88 $\pm$ 6.13 98.48 $\pm$ 7.88 40.60 $\pm$ 9.56 43.73 $\pm$ 6.02 53.45 $\pm$ 7.23 28.32 $\pm$ 6.21	384.52 $\pm$ 1.45 120.42 $\pm$ 9.84 90.34 $\pm$ 9.47 68.21 $\pm$ 8.54 60.36 $\pm$ 7.43 123.63 $\pm$ 6.83 45.28 $\pm$ 7.47 53.46 $\pm$ 8.23 48.72 $\pm$ 9.01 20.36 $\pm$ 7.14	412.64 $\pm$ 2.73 128.24 $\pm$ 4.32 92.34 $\pm$ 10.56 70.24 $\pm$ 11.05 59.86 $\pm$ 9.35 113.74 $\pm$ 8.46 46.68 $\pm$ 9.37 52.61 $\pm$ 7.83 63.47 $\pm$ 8.42 22.22 $\pm$ 8.73	426.58 $\pm$ 2.14 130.26 $\pm$ 7.68 95.42 $\pm$ 8.75 69.21 $\pm$ 9.42 60.84 $\pm$ 8.35 99.87 $\pm$ 9.84 48.23 $\pm$ 7.81 89.36 $\pm$ 8.43 100.47 $\pm$ 9.41 20.21 $\pm$ 7.21

TABLE II: Comparisons between the ATT attack and other methods in the Breakout game.

*

denotes the cumulative reward of the target policy under no attack. The bold indicates the best attack performance. Values represent the average cumulative reward (in game points) of the target policy under attack, calculated over 10 independent evaluation runs. Lower values indicate better attack performance.

Panel of Figure 4 — Fig. 2: All experimental results are the average of 10 experiments. In the black-box experimental results, we generate adversarial examples using A3C and D4PG as substitution strategies, respectively.

Pong
	Adversarial training				SA-MDP				POTBSG
Method	DQN	A3C	TRPO	PPO	DQN	A3C	TRPO	PPO	DQN	A3C	TRPO	PPO
* FGSM Skip S-T EDGE PA-AD AdvRL-GAN AAT	17.88 $\pm$ 0.33 16.08 $\pm$ 0.32 16.11 $\pm$ 1.45 15.12 $\pm$ 0.86 10.12 $\pm$ 1.33 -4.67 $\pm$ 1.37 -3.56 $\pm$ 1.54 - 10.21 $\pm$ 0.67	20.10 $\pm$ 0.40 19.19 $\pm$ 0.58 18.11 $\pm$ 0.64 19.98 $\pm$ 1.47 5.34 $\pm$ 1.57 -6.78 $\pm$ 1.37 -2.41 $\pm$ 1.43 - 8.15 $\pm$ 1.52	19.92 $\pm$ 0.35 18.78 $\pm$ 0.64 18.13 $\pm$ 0.37 18.68 $\pm$ 0.62 3.73 $\pm$ 1.35 -7.36 $\pm$ 1.05 -5.76 $\pm$ 0.94 - 12.06 $\pm$ 1.34	20.96 $\pm$ 0.05 18.86 $\pm$ 0.74 19.38 $\pm$ 1.01 20.11 $\pm$ 0.32 -2.98 $\pm$ 1.84 -4.62 $\pm$ 2.01 -5.46 $\pm$ 1.51 - 7.98 $\pm$ 1.18	18.88 $\pm$ 1.03 17.38 $\pm$ 0.24 16.11 $\pm$ 1.36 17.32 $\pm$ 0.51 13.42 $\pm$ 1.41 -1.41 $\pm$ 1.74 -2.27 $\pm$ 1.35 - 8.21 $\pm$ 1.34	19.12 $\pm$ 0.35 19.07 $\pm$ 0.91 18.74 $\pm$ 1.07 17.18 $\pm$ 0.67 9.74 $\pm$ 1.35 - 3.41 $\pm$ 2.17 -1.21 $\pm$ 2.36 -1.48 $\pm$ 2.41	20.02 $\pm$ 0.41 18.43 $\pm$ 0.5 19.13 $\pm$ 1.32 17.76 $\pm$ 1.54 10.54 $\pm$ 2.07 -8.13 $\pm$ 1.87 -6.26 $\pm$ 1.97 - 9.47 $\pm$ 1.84	20.16 $\pm$ 0.51 19.99 $\pm$ 0.84 18.31 $\pm$ 1.54 18.21 $\pm$ 2.07 13.78 $\pm$ 1.74 1.32 $\pm$ 1.98 - 3.46 $\pm$ 2.13 -3.24 $\pm$ 2.07	17.96 $\pm$ 1.52 16.63 $\pm$ 1.69 16.41 $\pm$ 2.04 14.12 $\pm$ 1.87 12.42 $\pm$ 1.64 -6.21 $\pm$ 2.07 -9.56 $\pm$ 1.73 - 9.74 $\pm$ 2.13	18.74 $\pm$ 0.25 16.04 $\pm$ 0.74 17.41 $\pm$ 1.86 16.13 $\pm$ 1.62 8.44 $\pm$ 1.73 - 7.37 $\pm$ 1.61 -6.19 $\pm$ 1.87 -4.15 $\pm$ 1.92	19.12 $\pm$ 0.53 17.01 $\pm$ 1.02 17.19 $\pm$ 0.98 18.18 $\pm$ 1.41 18.41 $\pm$ 1.23 -5.46 $\pm$ 1.17 -7.21 $\pm$ 1.32 - 9.67 $\pm$ 1.12	19.46 $\pm$ 0.34 17.43 $\pm$ 2.4 18.74 $\pm$ 1.52 18.22 $\pm$ 2.14 -4.33 $\pm$ 1.52 -6.74 $\pm$ 1.74 - 7.88 $\pm$ 1.32 -7.32 $\pm$ 2.07

TABLE III: Comparisons between the ATT attack and other methods in the Pong game.

*

indicates the cumulative reward when the target strategy adopts a defensive strategy but is not attacked.

Panel of Figure 6 — Fig. 3: The performance of different attack methods in Gfootball. The expected cumulative rewards achieved by DQN and PPO target policies are 7.12 and 7.88 respectively.

Panel of Figure 7 — Fig. 4: Visualization of immediate rewards for target policy. The x-axis represents the moments from the start to the end of the target policy’s execution in the Gfootball environment, and the y-axis is the reward value obtained at each time step.

Method	Pong	Breakout	Seaquest	Humanoid	Hopper	Walker
FGSM Skip S-T EDGE AdvRL-GAN AAT	5.3 8.5 13.4 12.8 14.2 1.56	8.4 13.7 9.8 16.6 18.8 2.2	4.3 7.2 5.5 9.4 9.83 2.4	6.3 8.4 8.3 4.1 10.58 1.5	9.6 13.2 14.1 13.5 15.4 2.1	7.8 12.9 15.6 16.4 16.8 2.6

TABLE IV: The average time to generate an adversarial examples. Note that the time unit is milliseconds.

Environments	Data type	DQN	A3C	TRPO	PPO
Seaquest	Expert data Medium data Random data	359.82 367.41 2325.65	353.64 373.56 2354.29	360.23 359.89 2332.79	359.67 393.45 2467.83
Qbert	Expert data Medium data Random data	236.89 379.46 5020.57	183.92 245.67 5596.35	203.89 299.67 4478.36	211.67 374.56 6340.27

TABLE V: The impact of different data quality on AAT performance. The table displays the mean values obtained from 10 attack experiments.

Fig. 5: The impact of data scale on AAT. The x-axis represents the scale of the data. For example, 5 represents 5,000 historical trajectories.

Panel of Figure 11 — Fig. 6: The impact of weighted advantages on AAT performance.

Fig. 7: The trade-off relationship between model performance and computational overhead. Cost means that each output token of the A100 GPU costs approximately.

Shenghong He is currently a PhD student under the supervision of Prof. Chao Yu in the School of Computer Science and Engineering, Sun Yat-sen University, China. He has published a few works at IEEE TDSC and TKDE, and is now working on the problem of offline learning and security with deep reinforcement learning.

Table 13

Chao Yu received the Ph.D. degree in computer science from the University of Wollongong, Australia, in 2014. He is currently a professor at the School of Computer Science and Engineering, Sun Yat-sen University, China. He has published more than 100 papers in prestigious conferences and journals, such as ICML, AAAI, NeurIPS, IEEE TCYB, IEEE TVT, and IEEE TKDE. His research interests include multi-agent systems, reinforcement learning, and their wide applications in autonomous driving, smart grid, robotic control, and intelligent healthcare.

Table 14

Danying Mo is currently a PhD student under the supervision of Prof. Chao Yu in the School of Computer Science and Engineering, Sun Yat-sen University, China. She is now working on the problem of Agent.

Table 15

Yucong Zhang is currently a PhD student under the supervision of Prof. Chao Yu in the School of Computer Science and Engineering, Sun Yat-sen University, China. He has published a few works at AAMAS and is now working on the problem of deep reinforcement learning.

Table 16

Yinqi Wei received the B.Sc. degree in engineering from Sun Yat-Sen University, Shenzhen, Guangdong, China, in 2024, and the M.Sc. degree in electronic from Nanyang Technological University, Singapore, in 2025, respectively.

Table 17

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter
GPT		Perturbation network
Name	value	Name	value
Number of layers	6	Linear layer	128, 3136
Attention heads	8	Aactivite function	Rlue
Embedding dimension	128	ConvTranspose2d	64,32,4
Batch size	Atari:128 DeepMind:256	kernel_size	3,4,8

Appendix

TABLE A.1: Hyperparameter

Pong
	White-box				Black-box
Method	DQN	A3C	TRPO	PPO	DQN	A3C	TRPO	PPO
* FGSM Skip S-T EDGE PA-AD AdvRL-GAN TSGE PIA AAT	19.83 $\pm$ 1.31 -20.31 $\pm$ 0.48 -18.35 $\pm$ 0.74 -17.86 $\pm$ 1.23 -19.36 $\pm$ 0.98 - 21.00 $\pm$ 0.00 -12.44 $\pm$ 0.22 -15.34 $\pm$ 2.37 -17.23 $\pm$ 1.95 -20.96 $\pm$ 0.04	21.00 $\pm$ 0.00 -20.35 $\pm$ 0.25 -19.36 $\pm$ 1.03 -10.84 $\pm$ 3.41 -14.33 $\pm$ 2.54 -19.8 $\pm$ 0.21 -11.18 $\pm$ 1.78 -12.47 $\pm$ 2.44 -18.34 $\pm$ 0.58 - 20.61 $\pm$ 0.33	20.64 $\pm$ 0.32 - 20.48 $\pm$ 0.15 -18.79 $\pm$ 0.27 -12.77 $\pm$ 2.14 -14.89 $\pm$ 1.95 -9.98 $\pm$ 2.13 -18.4 $\pm$ 1.21 -19.76 $\pm$ 0.34 -17.41 $\pm$ 1.43 -19.97 $\pm$ 1.24	21.00 $\pm$ 0.00 -19.56 $\pm$ 0.24 -15.42 $\pm$ 2.84 -14.66 $\pm$ 2.31 -15.13 $\pm$ 3.41 -10.08 $\pm$ 3.42 - 20.12 $\pm$ 0.24 -16.38 $\pm$ 2.18 -17.73 $\pm$ 0.31 -19.87 $\pm$ 0.27	19.83 $\pm$ 1.31 18.38 $\pm$ 0.54 18.10 $\pm$ 0.21 18.12 $\pm$ 0.06 -9.22 $\pm$ 3.45 -10.31 $\pm$ 2.78 -13.56 $\pm$ 1.44 -11.53 $\pm$ 0.97 -9.87 $\pm$ 0.12 - 18.34 $\pm$ 0.43	21.00 $\pm$ 0.00 20.25 $\pm$ 0.14 20.21 $\pm$ 0.34 19.65 $\pm$ 0.46 -6.54 $\pm$ 2.37 -9.31 $\pm$ 3.56 -11.36 $\pm$ 2.77 -8.24 $\pm$ 3.07 -9.21 $\pm$ 2.89 - 18.45 $\pm$ 1.94	20.64 $\pm$ 0.32 19.88 $\pm$ 0.23 18.02 $\pm$ 0.71 19.88 $\pm$ 1.02 -8.98 $\pm$ 3.07 -13.86 $\pm$ 2.94 - 16.36 $\pm$ 0.97 -13.36 $\pm$ 1.41 -11.49 $\pm$ 2.51 -16.06 $\pm$ 1.35	21.00 $\pm$ 0.00 20.26 $\pm$ 0.34 19.18 $\pm$ 0.81 20.01 $\pm$ 0.43 -5.98 $\pm$ 2.34 -15.67 $\pm$ 1.95 -10.46 $\pm$ 2.47 -10.73 $\pm$ 2.85 -13.02 $\pm$ 3.14 - 18.98 $\pm$ 1.07

Appendix

TABLE B.2: Comparisons between the ATT attack and other methods in the Pong game.

*

denotes the cumulative reward of the target policy under no attack. The bold indicates the best attack performance.

Chopper Command
	White-box				Black-box
Method	DQN	A3C	TRPO	PPO	DQN	A3C	TRPO	PPO
* FGSM Skip S-T EDGE PA-AD AdvRL-GAN TSGE PIA AAT	3555.83 $\pm$ 23.54 401.32 $\pm$ 14.53 770.21 $\pm$ 15.39 1180.65 $\pm$ 17.41 879.36 $\pm$ 21.06 789.35 $\pm$ 24.68 1461.37 $\pm$ 19.46 876.34 $\pm$ 23.44 854.37 $\pm$ 19.81 408.68 $\pm$ 19.74	3864.12 $\pm$ 19.87 468.36 $\pm$ 17.84 778.33 $\pm$ 21.33 1098.46 $\pm$ 17.49 906.24 $\pm$ 20.19 1335.46 $\pm$ 23.54 1389.56 $\pm$ 23.78 1031.27 $\pm$ 21.94 986.54 $\pm$ 20.84 410.21 $\pm$ 21.07	3615.44 $\pm$ 26.41 470.32 $\pm$ 19.97 789.34 $\pm$ 21.42 1206.26 $\pm$ 25.46 896.63 $\pm$ 21.53 784.36 $\pm$ 25.19 1280.62 $\pm$ 22.97 987.31 $\pm$ 22.43 1052.48 $\pm$ 21.68 420.98 $\pm$ 20.97	3842.52 $\pm$ 28.33 469.36 $\pm$ 20.89 780.56 $\pm$ 21.41 1240.36 $\pm$ 23.01 900.44 $\pm$ 20.89 578.69 $\pm$ 24.81 1350.43 $\pm$ 23.07 1003.42 $\pm$ 22.74 1203.47 $\pm$ 20.08 418.36 $\pm$ 19.35	3555.83 $\pm$ 23.54 2015.46 $\pm$ 29.35 2898.36 $\pm$ 31.04 2754.38 $\pm$ 28.74 2012.46 $\pm$ 31.14 893.72 $\pm$ 30.82 1864.32 $\pm$ 31.79 1562.49 $\pm$ 34.03 1671.39 $\pm$ 29.04 568.56 $\pm$ 27.89	3864.12 $\pm$ 19.87 2212.42 $\pm$ 31.81 2934.98 $\pm$ 29.87 2856.42 $\pm$ 30.47 2984.56 $\pm$ 33.04 1893.67 $\pm$ 31.78 1906.48 $\pm$ 32.46 1935.47 $\pm$ 32.75 2085.81 $\pm$ 30.84 628.54 $\pm$ 30.07	3615.44 $\pm$ 26.41 2189.38 $\pm$ 31.25 2856.94 $\pm$ 32.18 2789.35 $\pm$ 31.45 2365.38 $\pm$ 32.85 1352.71 $\pm$ 30.94 1832.91 $\pm$ 29.67 1489.26 $\pm$ 30.85 1793.91 $\pm$ 32.42 609.88 $\pm$ 30.33	3842.52 $\pm$ 28.33 2306.85 $\pm$ 30.84 2890.52 $\pm$ 29.47 2745.35 $\pm$ 33.42 2689.36 $\pm$ 32.81 987.62 $\pm$ 31.74 1896.54 $\pm$ 32.83 1328.46 $\pm$ 29.84 1573.23 $\pm$ 33.45 635.86 $\pm$ 29.04

Appendix

TABLE B.3: Comparisons between the ATT attack and other methods in the Chopper Command game.

*

denotes the cumulative reward of the target policy under no attack. The bold indicates the best attack performance.

Seaquest
	White-box				Black-box
Method	DQN	A3C	TRPO	PPO	DQN	A3C	TRPO	PPO
* FGSM Skip S-T EDGE PA-AD AdvRL-GAN TSGE PIA AAT	2385.32 $\pm$ 15.43 17.35 $\pm$ 3.56 89.42 $\pm$ 3.02 150.35 $\pm$ 2.97 75.46 $\pm$ 4.03 18.97 $\pm$ 0.98 108.23 $\pm$ 4.56 19.32 $\pm$ 2.78 23.71 $\pm$ 3.54 18.34 $\pm$ 1.07	2856.96 $\pm$ 16.84 65.42 $\pm$ 3.07 158.34 $\pm$ 4.13 221.84 $\pm$ 3.27 98.55 $\pm$ 3.08 17.63 $\pm$ 2.41 129.33 $\pm$ 2.83 35.43 $\pm$ 1.76 48.29 $\pm$ 2.81 20.22 $\pm$ 1.23	1856.89 $\pm$ 14.87 77.54 $\pm$ 3.21 198.33 $\pm$ 3.44 210.04 $\pm$ 2.97 98.45 $\pm$ 3.12 23.54 $\pm$ 2.87 119.83 $\pm$ 3.04 54.82 $\pm$ 2.14 49.41 $\pm$ 2.52 19.84 $\pm$ 2.03	2108.38 $\pm$ 16.53 74.32 $\pm$ 2.17 178.26 $\pm$ 3.82 256.83 $\pm$ 3.04 100.82 $\pm$ 3.51 24.81 $\pm$ 3.71 132.42 $\pm$ 2.87 54.39 $\pm$ 3.17 53.28 $\pm$ 2.89 21.42 $\pm$ 3.07	2385.32 $\pm$ 15.43 290.32 $\pm$ 5.74 300.48 $\pm$ 6.52 328.18 $\pm$ 4.31 298.43 $\pm$ 7.98 178.46 $\pm$ 5.04 245.44 $\pm$ 6.15 167.89 $\pm$ 6.71 216.74 $\pm$ 7.16 80.45 $\pm$ 5.34	2856.96 $\pm$ 16.84 65.42 $\pm$ 7.32 158.34 $\pm$ 6.89 221.84 $\pm$ 7.18 98.55 $\pm$ 7.24 17.63 $\pm$ 5.37 129.33 $\pm$ 7.05 35.43 $\pm$ 6.05 48.29 $\pm$ 6.19 20.22 $\pm$ 5.23	1856.89 $\pm$ 14.87 77.54 $\pm$ 6.83 198.33 $\pm$ 8.47 210.04 $\pm$ 5.24 98.45 $\pm$ 6.53 23.54 $\pm$ 4.32 119.83 $\pm$ 3.56 54.82 $\pm$ 5.41 49.41 $\pm$ 4.32 19.84 $\pm$ 3.17	2108.38 $\pm$ 16.53 74.32 $\pm$ 6.44 178.26 $\pm$ 7.59 256.83 $\pm$ 6.37 100.82 $\pm$ 6.04 24.81 $\pm$ 3.21 132.42 $\pm$ 4.68 54.39 $\pm$ 5.24 53.28 $\pm$ 4.52 21.42 $\pm$ 4.37

Appendix

TABLE B.4: Comparisons between the ATT attack and other methods in the Seaquest game. The expected cumulative rewards achieved by DQN, A3C, TRPO and PPO target policies are 2385.32, 2856.89, 1856.89 and 2108.38 respectively. The bold indicates the best attack performance.

Qbert
	White-box				Black-box
Method	DQN	A3C	TRPO	PPO	DQN	A3C	TRPO	PPO
* FGSM Skip S-T EDGE PA-AD AdvRL-GAN TSGE PIA AAT	17890.93 $\pm$ 2.35 165.43 $\pm$ 2.49 678.35 $\pm$ 3.13 543.44 $\pm$ 2.87 235.68 $\pm$ 3.04 152.34 $\pm$ 2.89 124.82 $\pm$ 1.87 48.39 $\pm$ 4.46 98.93 $\pm$ 3.51 50.56 $\pm$ 1.74	21840.35 $\pm$ 3.41 189.34 $\pm$ 2.93 543.27 $\pm$ 3.97 569.38 $\pm$ 4.08 253.96 $\pm$ 2.96 148.92 $\pm$ 3.18 139.84 $\pm$ 2.47 83.98 $\pm$ 3.21 129.46 $\pm$ 4.32 60.22 $\pm$ 2.19	22840.56 $\pm$ 1.52 205.33 $\pm$ 3.01 667.32 $\pm$ 2.95 510.35 $\pm$ 3.17 199.84 $\pm$ 3.46 175.62 $\pm$ 3.58 129.83 $\pm$ 4.02 58.01 $\pm$ 3.96 109.49 $\pm$ 2.34 58.42 $\pm$ 2.78	2458.68 $\pm$ 2.87 210.32 $\pm$ 3.73 680.34 $\pm$ 3.15 568.23 $\pm$ 4.01 1246.21 $\pm$ 2.77 101.23 $\pm$ 3.28 130.58 $\pm$ 3.92 94.35 $\pm$ 2.81 117.43 $\pm$ 3.07 54.32 $\pm$ 2.81	17890.93 $\pm$ 2.35 165.43 $\pm$ 5.46 678.35 $\pm$ 8.32 543.44 $\pm$ 3.45 235.68 $\pm$ 7.01 152.34 $\pm$ 5.32 124.82 $\pm$ 4.98 48.39 $\pm$ 5.46 98.93 $\pm$ 4.92 50.56 $\pm$ 3.41	21840.35 $\pm$ 3.41 189.34 $\pm$ 6.42 543.27 $\pm$ 6.51 569.38 $\pm$ 6.73 253.96 $\pm$ 5.82 148.92 $\pm$ 7.03 139.84 $\pm$ 6.54 83.98 $\pm$ 4.98 129.46 $\pm$ 5.94 60.22 $\pm$ 5.33	22840.56 $\pm$ 1.52 205.33 $\pm$ 5.93 667.32 $\pm$ 6.48 510.35 $\pm$ 6.82 199.84 $\pm$ 7.03 175.62 $\pm$ 6.35 129.83 $\pm$ 5.98 58.01 $\pm$ 7.13 109.49 $\pm$ 6.32 58.42 $\pm$ 5.18	2458.68 $\pm$ 2.87 210.32 $\pm$ 6.25 680.34 $\pm$ 5.48 568.23 $\pm$ 6.95 1246.21 $\pm$ 5.39 101.23 $\pm$ 6.33 130.58 $\pm$ 6.43 94.35 $\pm$ 7.36 117.43 $\pm$ 5.83 54.32 $\pm$ 4.73

Appendix

TABLE B.5: Comparisons between the ATT attack and other methods in the Qbert game.

*

denotes the cumulative reward of the target policy under no attack. The bold indicates the best attack performance.

Space Invaders
	White-box				Black-box
Method	DQN	A3C	TRPO	PPO	DQN	A3C	TRPO	PPO
* FGSM Skip S-T EDGE PA-AD AdvRL-GAN TSGE PIA AAT	1005.38 $\pm$ 2.41 123.50 $\pm$ 3.42 169.80 $\pm$ 4.57 210.32 $\pm$ 2.78 156.33 $\pm$ 4.02 151.27 $\pm$ 2.54 160.32 $\pm$ 3.87 103.42 $\pm$ 2.74 123.57 $\pm$ 1.98 59.59 $\pm$ 2.03	1185.42 $\pm$ 1.98 136.34 $\pm$ 4.07 145.83 $\pm$ 4.26 189.56 $\pm$ 3.52 189.32 $\pm$ 3.49 163.52 $\pm$ 2.97 176.32 $\pm$ 3.44 86.43 $\pm$ 3.14 98.47 $\pm$ 2.18 51.26 $\pm$ 2.16	1456.26 $\pm$ 2.74 138.23 $\pm$ 3.78 141.56 $\pm$ 4.01 208.93 $\pm$ 3.42 172.34 $\pm$ 2.97 132.84 $\pm$ 3.15 166.26 $\pm$ 3.28 79.98 $\pm$ 3.61 104.72 $\pm$ 2.73 60.35 $\pm$ 2.32	1653.48 $\pm$ 2.43 149.23 $\pm$ 3.52 185.39 $\pm$ 3.93 219.27 $\pm$ 4.01 188.93 $\pm$ 3.58 89.73 $\pm$ 3.82 170.32 $\pm$ 3.75 59.87 $\pm$ 3.93 93.42 $\pm$ 3.07 61.38 $\pm$ 2.54	1005.38 $\pm$ 2.41 123.50 $\pm$ 5.43 169.80 $\pm$ 7.02 210.32 $\pm$ 6.54 156.33 $\pm$ 5.93 151.27 $\pm$ 6.28 160.32 $\pm$ 4.35 103.42 $\pm$ 5.28 123.57 $\pm$ 6.53 68.59 $\pm$ 4.02	1185.42 $\pm$ 1.98 136.34 $\pm$ 6.03 145.83 $\pm$ 6.43 189.56 $\pm$ 5.86 189.32 $\pm$ 6.87 163.52 $\pm$ 7.13 176.32 $\pm$ 4.56 86.43 $\pm$ 5.23 98.47 $\pm$ 6.33 51.26 $\pm$ 5.27	1456.26 $\pm$ 2.74 138.23 $\pm$ 5.91 141.56 $\pm$ 6.23 208.93 $\pm$ 5.97 172.34 $\pm$ 6.37 132.84 $\pm$ 6.84 166.26 $\pm$ 5.36 79.98 $\pm$ 6.38 104.72 $\pm$ 5.86 60.35 $\pm$ 4.89	1653.48 $\pm$ 2.43 149.23 $\pm$ 6.32 185.39 $\pm$ 6.74 219.27 $\pm$ 5.86 188.93 $\pm$ 7.03 89.73 $\pm$ 6.74 170.32 $\pm$ 5.84 59.87 $\pm$ 5.98 93.42 $\pm$ 6.07 61.38 $\pm$ 5.07

Appendix

TABLE B.6: Comparisons between the ATT attack and other methods in the Space Invaders game.

*

denotes the cumulative reward of the target policy under no attack. The bold indicates the best attack performance.

Panel of Figure 24 — Fig. B.1: All experimental results are the average of 10 experiments. In the black-box experimental results, we generate adversarial examples using A3C and D4PG as substitution strategies, respectively.

Pong
	MF				MimicDiffusion
Method	DQN	A3C	TRPO	PPO	DQN	A3C	TRPO	PPO
	18.18	19.13	18.72	20.02	19.48	20.22	19.84	20.46
FGSM	17.65	18.32	18.04	18.73	18.74	18.97	19.12	19.34
Skip	17.94	18.78	18.13	17.94	18.97	19.34	18.74	19.64
S-T	17.44	17.99	18.14	18.24	18.41	18.95	19.13	18.41

Appendix

TABLE B.7: Defense results of smoothing and input reconstruction against adversarial examples.

*

denotes the cumulative reward obtained by the policy in the absence of an attack.

Panel of Figure 26 — Fig. B.2: Adversarial example visualization. In each subfigure, the first row is the original state, the second row is the adversarial example, the third row is the adversarial perturbation that is visualizable, the fourth row is the MSE between the original state and the adversarial example and five row is the SSIM metric. The six columns correspond to FGSM, Skip, S-T, EDGE, advRL-GAN, and AAT respectively.

Detection method	Environment	FGSM	Skip	S-T	EDGE	PA-AD	AdvRL-GAN	TSGE	PIA	AAT
	Pong	0.83	0.74	0.69	0.63	0.52	0.43	0.53	0.44	0.45
EPS-AD	Squest	0.87	0.65	0.74	0.64	0.66	0.65	0.57	0.49	0.43
	Qbert	0.79	0.67	0.73	0.61	0.59	0.56	0.48	0.44	0.47
	Pong	0.77	0.54	0.68	0.71	0.64	0.34	0.39	0.41	0.35
LR	Squest	0.81	0.61	0.79	0.68	0.45	0.45	0.54	0.42	0.39
	Qbert	0.80	0.69	0.82	0.72	0.47	0.35	0.48	0.39	0.35

Appendix

TABLE B.8: The results of adversarial detection.

Panel of Figure 28 — Fig. B.3: The impact of single and multiple substitution policies in black-box scenarios.

Panel of Figure 29 — Fig. B.4: The impact of data diversity on AAT performance. The y-axis represents the cumulative rewards of agent, while the x-axis is the training epochs of AAT.

Panel of Figure 30 — Fig. B.5: The performance of the target policy after being attacked by different attack methods. The y-axis represents the cumulative rewards of agent, while the x-axis is the training epochs of AAT.

Panel of Figure 31 — Fig. B.6: The impact of random perturbation on AAT attack performance.

Environments	Data type	DQN	A3C	TRPO	PPO
Pong	$D_{FGSM}$ $D_{SKip}$ $D_{EDGE}$ $D_{AGAN}$	-15.18 -9.36 -12.33 -9.67	-13.27 -11.59 -10.64 -12.91	-12.63 -10.84 -13.01 -10.79	-14.14 -12.56 -10.01 -9.95
Chopper Command	$D_{FGSM}$ $D_{SKip}$ $D_{EDGE}$ $D_{AGAN}$	1457.34 1678.43 1832.56 1467.43	1356.98 1834.14 1754.36 1548.21	1543.55 1672.81 1854.66 1456.32	1437.63 1783.29 1659.26 1499.57

Appendix

TABLE B.9: The impact of different data collection methods on AAT performance.

Method	Pong	Breakout	Seaquest	Humanoid	Hopper	Walker
FGSM Skip S-T EDGE AdvRL-GAN AAT	5.3 8.5 13.4 12.8 14.2 1.56	8.4 13.7 9.8 16.6 18.8 2.2	4.3 7.2 5.5 9.4 9.83 2.4	6.3 8.4 8.3 4.1 10.58 1.5	9.6 13.2 14.1 13.5 15.4 2.1	7.8 12.9 15.6 16.4 16.8 2.6

Appendix

TABLE B.10: The average time to generate an adversarial examples. Note that the time unit is milliseconds.

Panel of Figure 34 — Fig. B.7: AAT employs varied learning approaches for attacking different target policies across different environments. The y-axis represents the cumulative rewards of agent, while the x-axis is the training epochs of AAT.

		White-box		Black-box
Environment	Method	D4PG	A3C	D4PG	A3C
HalfCheetah	SA MSCSA	51.25 25.56	42.34 25.17	52.23 46.59	56.37 40.27
Hoper	SA MSCSA	190.31 118.61	74.42 68.72	154.43 138.31	101.45 108.52
Humanoid	SA MSCSA	26.43 5.65	13.47 4.65	37.82 28.31	41.37 30.64
Pendulum	SA MSCSA	40.62 36.51	45.34 30.26	80.61 78.34	82.42 68.35
Walker	SA MSCSA	85.34 60.61	35.36 20.42	175.64 128.43	105.52 108.36

Appendix

TABLE B.11: The performance of the MSCSA

Strategy	$\boldsymbol{Len}$	Window Seq.	Reward	Cost (ms)	Notes
Exponential	5	[5, 10, 20]	118.3	125	Best trade-off of local and global modeling
Exponential	8	[8, 16, 32]	108.7	356	High cost with little gain
Linear	5	[5, 10, 15]	235.6	119	Less effective in modeling long-term context
Linear	8	[8, 12, 16]	209.3	298	Overhead increases without performance gain
Fixed	5	[5, 5, 5]	583.1	89	Poor modeling capacity, lacks multi-scale
Exponential	2	[2, 4, 8]	629.8	90	Efficient but may underutilize long-term info

Appendix

TABLE B.12: Ablation on different window growth strategies and initial window sizes

Len

. The experimental results are the average of 10 runs using D4PG as the target algorithm in the Hopper environment.

Panel of Figure 37 — Fig. B.8: The impact of different window lengths on AAT performance. We adopt PPO as the target policy in the Atari and employ D4PG as the target policy in MuJoCo.

Panel of Figure 38 — Fig. B.9: AAT uses $L_{1}$ , $L_{2}$ , and $L_{\infty}$ losses to affect different target policies in different environments during the learning process.

		White-box				Black-box
Environment	Method	DQN	A3C	TRPO	PPO	DQN	A3C	TRPO	PPO
Pong	AAT-GA AAT-WA	-19.83 - 20.96	-18.18 - 20.61	- 19.98 -19.97	-19.25 - 19.87	-17.26 - 18.34	-17.31 - 18.45	-15.92 - 16.06	-17.61 - 18.98
Chopper Command	AAT-GA AAT-WA	400.31 408.68	567.33 410.21	435.65 420.98	420.68 418.36	693.27 568.56	784.56 628.54	610.32 609.88	642.55 635.36
Seaquest	AAT-GA AAT-WA	19.32 18.34	20.11 20.22	21.52 19.84	23.01 21.42	90.23 80.45	84.51 70.89	79.46 60.42	79.87 73.58
Qbert	AAT-GA AAT-WA	69.35 50.56	60.03 60.22	65.46 58.42	71.21 54.32	635.67 598.55	535.34 478.35	635.78 483.21	638.42 556.34
Space Invaders	AAT-GA AAT-WA	70.23 59.59	63.46 51.26	68.35 60.35	73.21 61.38	110.42 103.66	134.58 126.31	124.68 109.84	136.27 118.28

Appendix

TABLE B.13: Advantageous ablation results.

Panel of Figure 40 — Fig. B.10: The attack outcome of AAT in a multi-agent setting. The experimental results are the average of 10 random seeds.

Attention-Based Adversarial Attacks

See all 12 papers

Boosting Transferable Adversarial Attacks against Deep Reinforcement Learning
Oct 5, 2026Zexin Li, Ruili Yao, Yiming Zeng +1Adversarial Attacks Transfer-Based Adversarial Attacks
Nanyang Technological University · University of California, Riverside · University of Connecticut +1
GENADA: efficient generative time series adversarial attack framework
Aug 12, 2026Michael Baronov, Denis Vorobev, Margarita Rusanova +2Adversarial Attacks Adversarial Perturbations
Moscow Independent Research Institute of Artificial Intelligence · HSE University
Robust Policy Optimization via Adversarial Importance Sampling
Sep 11, 2026Amine Andam, Jamal Bentahar, Mustapha HedabouRobust RL Policy Learning
Mohammed VI Polytechnic University · Khalifa University · Concordia University

Oct 5, 2026J/K move · Enter open · S save

Boosting Transferable Adversarial Attacks against Deep Reinforcement Learning

Zexin Li, Ruili Yao, Yiming Zeng, Xiaoxue Gao

Nanyang Technological University · University of California, Riverside · University of Connecticut+1

Adversarial Attacks Transfer-Based Adversarial Attacks Black-Box Adversarial Attacks Adversarial Transferability

Open paper AskPDF arXiv

Most adversarial attacks on deep reinforcement learning (DRL) assume white-box access to the victim policy, which rarely holds in practice. This paper studies transfer-based black-box attacks on DRL: the attacker crafts observation perturbations on a white-box surrogate agent and feeds them to an unknown victim. We formulate the attack as return minimization under a per-step perturbation budget. We first show that transplanting transferable image-classification attacks (FGSM, MI-FGSM, and NI-FGSM) with a per-step objective yields perturbations that transfer but are no stronger than random noise of the same budget. We then propose a trajectory-level attack that optimizes a sequence of perturbations over a receding horizon through a differentiable model of the environment and a temperature-smoothed surrogate policy, with the same optimizers. On CartPole-v1 with ten DQN and DDQN agents and 100 surrogate--victim pairs, the trajectory-level attack outperforms per-step attacks and random noise in the white-box, cross-model, and cross-algorithm settings.

Abstract

Attention-Based Adversarial Attacks

Explore similar work

Boosting Transferable Adversarial Attacks against Deep Reinforcement Learning

GENADA: efficient generative time series adversarial attack framework

Robust Policy Optimization via Adversarial Importance Sampling