Taming the Adversary: A Cost-to-Disturbance Ratio Approach to Adversarial Reinforcement Learning

Taeho Lee; Donghwan Lee

cs.LGMar 12, 2026

Taming the Adversary: A Cost-to-Disturbance Ratio Approach to Adversarial Reinforcement Learning

Authors: Taeho Lee, Donghwan Lee

Organizations: School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea

Abstract

Reinforcement learning (RL) policies trained in simulation often degrade once deployed on real systems, where the controller must reject external disturbances that were never encountered in simulation. Robust RL addresses this by exposing the controller to perturbations while it learns, through domain randomization, adversarial minimax formulations, or probabilistic mixtures of protagonist and adversarial behavior. However, an unregulated disturbance mechanism destabilizes training and often collapses nominal performance relative to standard, non-robust methods. We propose cost-to-disturbance ratio adversarial training (CoDRA), a framework that expresses the controller--adversary trade-off as a ratio of accumulated cost to accumulated squared disturbance norm, and optimizes it through a self-normalized actor--critic update. In this algorithm, each value term is scaled by a stop-gradient normalization constant computed from the current batch. This moderates the adversary's incentive without altering the controller's own update, and requires neither an explicit disturbance penalty nor an auxiliary trade-off parameter. We evaluate CoDRA on two MuJoCo pendulum environments under force and mass sweeps. On InvertedDoublePendulum, CoDRA attains the lowest cost at every force level, including a force outside the range seen during training, and in all but one cell of the mass grid, whereas its advantage is less pronounced on the milder InvertedPendulum.

Figures & tables

Fig. 1: Overview of cost-to-disturbance ratio adversarial training (CoDRA). The controller and the adversary interact with the environment, generating the action $a_{k}$ and the disturbance $w_{k}$ from the state $s_{k}$ . The critics $Q_{\psi_{1}}(s,a,w)$ and $Q_{\psi_{2}}(s,w)$ are trained using $c_{k+1}$ and $\|w_{k}\|_{2}^{2}$ , respectively. The policy of controller $\pi_{\theta}$ is updated to minimize the self-normalized objective derived from $J^{\pi_{\theta},\mu_{\phi}}$ while the policy of adversary $\mu_{\phi}$ is updated to maximize it.

	$\displaystyle{\psi_{1}}$	$\displaystyle\leftarrow{\psi_{1}}-{\alpha_{{\rm{critic}}}}{\nabla_{\psi_{1}}}L({\psi_{1}};B)$
	$\displaystyle{\psi_{2}}$	$\displaystyle\leftarrow{\psi_{2}}-{\alpha_{{\rm{critic}}}}{\nabla_{\psi_{2}}}L({\psi_{2}};B)$

Algorithm 1 Cost-to-disturbance ratio adversarial training (CoDRA) with DDPG

	InvertedPendulum	InvertedDoublePendulum
State dimension	$4$	$6$
Action range	$[-3,3]$	$[-1,1]$
Motor gear	$100$	$500$
Integrator, timestep	RK4, $0.02$ s	RK4, $0.01$ s
Frame skip	$2$	$2$
Control period	$0.04$ s	$0.02$ s

TABLE I: Simulation parameters.

Hidden layers	$2\times 256$ , ReLU
Output layer	$\tanh$ , scaled to the action/force bound
Optimizer	Adam
Actor / adversary learning rate	$10^{-4}$
Critic learning rate	$10^{-3}$
Discount factor $\gamma$	$0.99$
Mini-batch size $\|B\|$	$128$

TABLE II: Training hyperparameters, shared by all algorithms unless noted.

Algorithm	Nominal	Force $=6$	Worst mass cell
DDPG [ 3 ]	0.61	28.76	19.40
DR-DDPG	0.78	53.59	11.78
RARL [ 6 ]	0.59	28.48	21.85
PR-DDPG [ 13 ]	4.24	43.33	20.28
NR-DDPG [ 13 ]	8.17	34.66	28.73
CoDRA (ours)	1.91	34.14	16.67

TABLE III: InvertedPendulum: mean discounted cost at the nominal plant, under the largest force, and in the worst mass cell. Nominal : force-

0

entry of the force sweep (unit mass scale). Force

=6

: fixed lateral pole force, nominal mass. Worst mass cell : max over the pole

\{0.25,0.5,1,2,4\}\times

cart

\{0.5,1,2\}

grid. Lower is better; bold is best per column.

Fig. 2: InvertedDoublePendulum: mean discounted cost under a constant lateral force on pole2 , swept from $0$ to $6$ at the nominal mass. The adversary was restricted to $\pm 5$ during training, so the largest force lies outside that range. CoDRA (bold border) has the lowest cost at every force level; the gap to the baselines is largest up to $F=4$ and narrows at $F=6$ .

Fig. 3: InvertedDoublePendulum: mean discounted cost over the pole $\times$ pole2 mass $\{0.5,1,2\}\times\{0.5,1,2\}$ grid, nominal at $1\times/1\times$ , with no disturbance applied. Darker is higher (worse) cost. CoDRA (bold border) is lowest in every cell except (pole $0.5\times$ , pole2 $2\times$ ), where PR-DDPG is lowest.

Fig. 4: InvertedDoublePendulum force sweep with all seven methods. The adversarial methods were trained with a $\pm 5$ bound, so only $F=6$ lies outside the training range.

Fig. 5: InvertedDoublePendulum mass grid with all seven methods, pole $\times$ pole2 mass $\{0.5,1,2\}\times\{0.5,1,2\}$ , no disturbance applied.

Fig. 6: InvertedPendulum force sweep with all seven methods. The adversarial methods were trained with a $\pm 2.5$ bound, so $F=4$ and $F=6$ lie outside the training range.

Fig. 7: InvertedPendulum mass grid with all seven methods, pole $\{0.25,0.5,1,2,4\}\times$ cart $\{0.5,1,2\}$ , no disturbance applied.

Algorithm	Nominal	Force $=6$	Worst mass cell
DDPG [ 3 ]	$0.61$	$28.76$	$19.40$
DR-DDPG	$0.78$	$53.59$	$\mathbf{11.78}$
RARL [ 6 ]	$\mathbf{0.59}$	$\mathbf{28.48}$	$21.85$
PR-DDPG [ 13 ]	$4.24$	$43.33$	$20.28$
NR-DDPG [ 13 ]	$8.17$	$34.66$	$28.73$
DDPG-RF	$1.45$	$41.01$	$39.23$

TABLE IV: InvertedPendulum with all seven methods: mean discounted cost. Nominal is the force-

0

entry of the force sweep; the other columns are defined as in Table III . Bold is best per column.

Environment	Quantity	CoDRA	Lowest-mean baseline	$t$	$\min\|t\|$
InvertedDoublePendulum	$F=0$	$0.08\pm 0.09$	DDPG-RF, $0.91\pm 0.90$	$-4.1$	$3.7$
	$F=2$	$0.62\pm 0.11$	DDPG-RF, $2.54\pm 1.45$	$-5.9$	$5.1$
	$F=4$	$2.63\pm 0.74$	RARL, $5.13\pm 1.47$	$-6.8$	$6.8$
	$F=6$	$7.29\pm 1.20$	RARL, $8.01\pm 0.91$	$-2.1$	$2.1$
	Mass-grid mean	$0.89\pm 0.71$	DDPG-RF, $1.28\pm 1.16$	$-1.3$	$1.3$
	Cell (pole $0.5\times$ , pole2 $2\times$ )	$6.17\pm 4.07$	PR-DDPG, $1.65\pm 1.74$	$+4.6$	$0.4$

TABLE V: Mean

\pm

standard deviation of the discounted cost over the

20

seeds.

t

is Welch’s

t

-statistic between CoDRA and the baseline with the lowest mean (among all six baselines, including DDPG-RF); negative values favor CoDRA. The last column is the smallest

|t|

over the six baselines. The InvertedPendulum mass cell is CoDRA’s worst cell (pole

0.25\times

, cart

2\times

); grid means are computed per seed before averaging.

Adversarial MARL

See all 17 papers

Robust Adversarial Reinforcement Learning with Risk Sensitivity and Critic Consistency Regularization
Sep 23, 2026Jiaxi Wu, Tiantian Zhang, Yuxing Wang +2Reinforcement Learning Risk-Sensitive RL
Shenzhen International Graduate School, Tsinghua University Shenzhen, China
Adversarial Reinforcement Learning for Robust Planetary Landing
Oct 14, 2026Miguel Leiva-Vélez, Nicolas Gaston-Rozado, Andrea Scorsoglio +2Adversarial MARL Reinforcement Learning
ETSI Aeronáutica y del Espacio, Universidad Politécnica de Madrid, Madrid, Spain · Indra Sistemas, Madrid, Spain · Department of Systems and Industrial Engineering, University of Arizona, Tucson, USA +2
A Unified and Constrained View of Regularization-Based Robust Reinforcement Learning
Sep 11, 2026Amine Andam, Jamal Bentahar, Mustapha HedabouKL-Regularized RL Adversarial Robustness
Mohammed VI Polytechnic University · Khalifa University · Concordia University

Sep 23, 2026J/K move · Enter open · S save

Robust Adversarial Reinforcement Learning with Risk Sensitivity and Critic Consistency Regularization

Jiaxi Wu, Tiantian Zhang, Yuxing Wang, Yongzhe Chang+1

Shenzhen International Graduate School, Tsinghua University Shenzhen, China

Reinforcement Learning Risk-Sensitive RL Adversarial MARL Robust RL Consistency Regularization

Open paper AskPDF arXiv

Reinforcement learning (RL) achieves strong performance in sequential decision-making but remains brittle under dynamic uncertainty and distributional shifts. Robust Adversarial Reinforcement Learning (RARL) improves robustness via worst-case perturbations, but existing approaches frequently suffer from unstable optimization and degraded value estimation. In particular, overly aggressive adversaries can drive the agent toward uninformative failure states, while adversarial perturbations amplify disagreement between double critics and introduce biased value targets. We propose a unified framework, RACER (Risk-sensitive robust Adversarial critic ConsistEncy-regularized Reinforcement learning), that revisits adversarial RL from a risk-sensitive perspective. First, we introduce a state-dependent adversarial objective that adaptively regulates perturbation strength, suppressing harmful disturbances while preserving informative exploration. Second, we propose critic consistency regularization to reduce disagreement between Q-value estimators and stabilize learning. Comprehensive experiments on challenging continuous control benchmarks demonstrate that RACER consistently improves performance, robustness, and training stability over strong robust RL baselines.

Abstract

Adversarial MARL

Explore similar work

Robust Adversarial Reinforcement Learning with Risk Sensitivity and Critic Consistency Regularization

Adversarial Reinforcement Learning for Robust Planetary Landing

A Unified and Constrained View of Regularization-Based Robust Reinforcement Learning