Taming the Adversary: A Cost-to-Disturbance Ratio Approach to Adversarial Reinforcement Learning

Taeho Lee; Donghwan Lee

cs.LGMar 12, 2026

Taming the Adversary: A Cost-to-Disturbance Ratio Approach to Adversarial Reinforcement Learning

Authors: Taeho Lee, Donghwan Lee

Organizations: School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea

Abstract

Reinforcement learning (RL) policies trained in simulation often degrade once deployed on real systems, where the controller must reject external disturbances that were never encountered in simulation. Robust RL addresses this by exposing the controller to perturbations while it learns, through domain randomization, adversarial minimax formulations, or probabilistic mixtures of protagonist and adversarial behavior. However, an unregulated disturbance mechanism destabilizes training and often collapses nominal performance relative to standard, non-robust methods. We propose cost-to-disturbance ratio adversarial training (CoDRA), a framework that expresses the controller--adversary trade-off as a ratio of accumulated cost to accumulated squared disturbance norm, and optimizes it through a self-normalized actor--critic update. In this algorithm, each value term is scaled by a stop-gradient normalization constant computed from the current batch. This moderates the adversary's incentive without altering the controller's own update, and requires neither an explicit disturbance penalty nor an auxiliary trade-off parameter. We evaluate CoDRA on two MuJoCo pendulum environments under force and mass sweeps. On InvertedDoublePendulum, CoDRA attains the lowest cost at every force level, including a force outside the range seen during training, and in all but one cell of the mass grid, whereas its advantage is less pronounced on the milder InvertedPendulum.

Figures & tables

Fig. 1: Overview of cost-to-disturbance ratio adversarial training (CoDRA). The controller and the adversary interact with the environment, generating the action $a_{k}$ and the disturbance $w_{k}$ from the state $s_{k}$ . The critics $Q_{\psi_{1}}(s,a,w)$ and $Q_{\psi_{2}}(s,w)$ are trained using $c_{k+1}$ and $\|w_{k}\|_{2}^{2}$ , respectively. The policy of controller $\pi_{\theta}$ is updated to minimize the self-normalized objective derived from $J^{\pi_{\theta},\mu_{\phi}}$ while the policy of adversary $\mu_{\phi}$ is updated to maximize it.

	$\displaystyle{\psi_{1}}$	$\displaystyle\leftarrow{\psi_{1}}-{\alpha_{{\rm{critic}}}}{\nabla_{\psi_{1}}}L({\psi_{1}};B)$
	$\displaystyle{\psi_{2}}$	$\displaystyle\leftarrow{\psi_{2}}-{\alpha_{{\rm{critic}}}}{\nabla_{\psi_{2}}}L({\psi_{2}};B)$

Algorithm 1 Cost-to-disturbance ratio adversarial training (CoDRA) with DDPG

	InvertedPendulum	InvertedDoublePendulum
State dimension	$4$	$6$
Action range	$[-3,3]$	$[-1,1]$
Motor gear	$100$	$500$
Integrator, timestep	RK4, $0.02$ s	RK4, $0.01$ s
Frame skip	$2$	$2$
Control period	$0.04$ s	$0.02$ s

TABLE I: Simulation parameters.

Hidden layers	$2\times 256$ , ReLU
Output layer	$\tanh$ , scaled to the action/force bound
Optimizer	Adam
Actor / adversary learning rate	$10^{-4}$
Critic learning rate	$10^{-3}$
Discount factor $\gamma$	$0.99$
Mini-batch size $\|B\|$	$128$

TABLE II: Training hyperparameters, shared by all algorithms unless noted.

Algorithm	Nominal	Force $=6$	Worst mass cell
DDPG [ 3 ]	0.61	28.76	19.40
DR-DDPG	0.78	53.59	11.78
RARL [ 6 ]	0.59	28.48	21.85
PR-DDPG [ 13 ]	4.24	43.33	20.28
NR-DDPG [ 13 ]	8.17	34.66	28.73
CoDRA (ours)	1.91	34.14	16.67

TABLE III: InvertedPendulum: mean discounted cost at the nominal plant, under the largest force, and in the worst mass cell. Nominal : force-

0

entry of the force sweep (unit mass scale). Force

=6

: fixed lateral pole force, nominal mass. Worst mass cell : max over the pole

\{0.25,0.5,1,2,4\}\times

cart

\{0.5,1,2\}

grid. Lower is better; bold is best per column.

Fig. 2: InvertedDoublePendulum: mean discounted cost under a constant lateral force on pole2 , swept from $0$ to $6$ at the nominal mass. The adversary was restricted to $\pm 5$ during training, so the largest force lies outside that range. CoDRA (bold border) has the lowest cost at every force level; the gap to the baselines is largest up to $F=4$ and narrows at $F=6$ .

Fig. 3: InvertedDoublePendulum: mean discounted cost over the pole $\times$ pole2 mass $\{0.5,1,2\}\times\{0.5,1,2\}$ grid, nominal at $1\times/1\times$ , with no disturbance applied. Darker is higher (worse) cost. CoDRA (bold border) is lowest in every cell except (pole $0.5\times$ , pole2 $2\times$ ), where PR-DDPG is lowest.

Fig. 4: InvertedDoublePendulum force sweep with all seven methods. The adversarial methods were trained with a $\pm 5$ bound, so only $F=6$ lies outside the training range.

Fig. 5: InvertedDoublePendulum mass grid with all seven methods, pole $\times$ pole2 mass $\{0.5,1,2\}\times\{0.5,1,2\}$ , no disturbance applied.

Fig. 6: InvertedPendulum force sweep with all seven methods. The adversarial methods were trained with a $\pm 2.5$ bound, so $F=4$ and $F=6$ lie outside the training range.

Fig. 7: InvertedPendulum mass grid with all seven methods, pole $\{0.25,0.5,1,2,4\}\times$ cart $\{0.5,1,2\}$ , no disturbance applied.

Algorithm	Nominal	Force $=6$	Worst mass cell
DDPG [ 3 ]	$0.61$	$28.76$	$19.40$
DR-DDPG	$0.78$	$53.59$	$\mathbf{11.78}$
RARL [ 6 ]	$\mathbf{0.59}$	$\mathbf{28.48}$	$21.85$
PR-DDPG [ 13 ]	$4.24$	$43.33$	$20.28$
NR-DDPG [ 13 ]	$8.17$	$34.66$	$28.73$
DDPG-RF	$1.45$	$41.01$	$39.23$

TABLE IV: InvertedPendulum with all seven methods: mean discounted cost. Nominal is the force-

0

entry of the force sweep; the other columns are defined as in Table III . Bold is best per column.

Environment	Quantity	CoDRA	Lowest-mean baseline	$t$	$\min\|t\|$
InvertedDoublePendulum	$F=0$	$0.08\pm 0.09$	DDPG-RF, $0.91\pm 0.90$	$-4.1$	$3.7$
	$F=2$	$0.62\pm 0.11$	DDPG-RF, $2.54\pm 1.45$	$-5.9$	$5.1$
	$F=4$	$2.63\pm 0.74$	RARL, $5.13\pm 1.47$	$-6.8$	$6.8$
	$F=6$	$7.29\pm 1.20$	RARL, $8.01\pm 0.91$	$-2.1$	$2.1$
	Mass-grid mean	$0.89\pm 0.71$	DDPG-RF, $1.28\pm 1.16$	$-1.3$	$1.3$
	Cell (pole $0.5\times$ , pole2 $2\times$ )	$6.17\pm 4.07$	PR-DDPG, $1.65\pm 1.74$	$+4.6$	$0.4$

TABLE V: Mean

\pm

standard deviation of the discounted cost over the

20

seeds.

t

is Welch’s

t

-statistic between CoDRA and the baseline with the lowest mean (among all six baselines, including DDPG-RF); negative values favor CoDRA. The last column is the smallest

|t|

over the six baselines. The InvertedPendulum mass cell is CoDRA’s worst cell (pole

0.25\times

, cart

2\times

); grid means are computed per seed before averaging.

Adversarial MARL

See all 17 papers

Sep 23, 2026

Robust Adversarial Reinforcement Learning with Risk Sensitivity and Critic Consistency Regularization

Reinforcement learning (RL) achieves strong performance in sequential decision-making but remains brittle under dynamic uncertainty and distributional shifts. Robust Adversarial Reinforcement Learning (RARL) improves robustness via worst-case perturbations, but existing approaches frequently suffer from unstable optimization and degraded value estimation. In particular, overly aggressive adversaries can drive the agent toward uninformative failure states, while adversarial perturbations amplify disagreement between double critics and introduce biased value targets. We propose a unified framework, RACER (Risk-sensitive robust Adversarial critic ConsistEncy-regularized Reinforcement learning), that revisits adversarial RL from a risk-sensitive perspective. First, we introduce a state-dependent adversarial objective that adaptively regulates perturbation strength, suppressing harmful disturbances while preserving informative exploration. Second, we propose critic consistency regularization to reduce disagreement between Q-value estimators and stabilize learning. Comprehensive experiments on challenging continuous control benchmarks demonstrate that RACER consistently improves performance, robustness, and training stability over strong robust RL baselines.

Jiaxi Wu, Tiantian Zhang, Yuxing Wang +2

Shenzhen International Graduate School, Tsinghua University Shenzhen, China

Reinforcement Learning Risk-Sensitive RL Adversarial MARL Robust RL Consistency Regularization

Ask PDF

Oct 14, 2026

Adversarial Reinforcement Learning for Robust Planetary Landing

Deep reinforcement learning (DRL) policies for planetary landing and other aerospace guidance, navigation, and control tasks are commonly trained under randomized disturbances to improve robustness. However, typical stochastic perturbation sampling assumes independent variations, whereas real mission disturbances often exhibit spatial and temporal correlation and cannot be reproduced through independent random draws. This work investigates adversarial reinforcement learning to discover challenging off-nominal conditions. A secondary agent generates physically constrained perturbations that degrade the performance of a landing controller, forming an asymmetric sequential zero-sum game. Training proceeds in three stages: baseline controller learning, robustness enhancement with stochastic perturbations, and an iterative adversarial phase in which the controller and disturbance generator are alternately optimized. Preliminary experiments in a three-degree-of-freedom landing environment show that adversarial disturbances reveal vulnerabilities missed by conventional randomization and provide early indications that, with a more extensive training campaign, the approach can yield improved robustness and stronger generalization under previously unseen perturbation regimes.

Miguel Leiva-Vélez, Nicolas Gaston-Rozado, Andrea Scorsoglio +2

ETSI Aeronáutica y del Espacio, Universidad Politécnica de Madrid, Madrid, Spain · Indra Sistemas, Madrid, Spain · Department of Systems and Industrial Engineering, University of Arizona, Tucson, USA +2

Adversarial MARL Reinforcement Learning Robust RL Adversarial Perturbations

Ask PDF

Sep 11, 2026

A Unified and Constrained View of Regularization-Based Robust Reinforcement Learning

Regularization-based methods have become a standard approach for training Deep Reinforcement Learning policies against adversarial input perturbations. In this paper, we unify these methods by deriving new upper bounds on the performance gap between the nominal and worst-case policies. Each upper bound is expressed as an existing regularization objective plus a KL-divergence penalty between the nominal and worst-case policies, which further explains why adding a KL penalty improves robustness in practice. Building on these bounds, we formulate robust training as a constrained optimization problem, showing that existing methods correspond to the special case of a fixed Lagrange multiplier. We instead update the multiplier jointly with the policy to automatically tune the regularization weight. Finally, we conduct extensive adversarial evaluations across several continuous control tasks to validate our theoretical analysis.

Amine Andam, Jamal Bentahar, Mustapha Hedabou

Mohammed VI Polytechnic University · Khalifa University · Concordia University

KL-Regularized RL Adversarial Robustness Constrained RL Robust RL Policy Learning

Ask PDF

Abstract

Adversarial MARL

Explore similar work

Robust Adversarial Reinforcement Learning with Risk Sensitivity and Critic Consistency Regularization

Adversarial Reinforcement Learning for Robust Planetary Landing

A Unified and Constrained View of Regularization-Based Robust Reinforcement Learning