RL-FAT: Reinforcement Learning for Fair Adversarial Training

Tejaswini Medi; Levan Mikeladze; Margret Keuper

cs.LGAug 29, 2026

RL-FAT: Reinforcement Learning for Fair Adversarial Training

Authors: Tejaswini Medi, Levan Mikeladze, Margret Keuper

Organizations: Chair for Machine Learning University of Mannheim Germany · MPI for Informatics Saarland Informatics Campus Germany

Abstract

Deep neural networks remain highly vulnerable to adversarial perturbations, and adversarial training (AT) has become a widely used approach for improving robustness. However, improvements in average robust accuracy often mask substantial class-wise disparities: while some classes become more robust, others may remain disproportionately vulnerable under attack. This imbalance raises an important adversarial fairness concern, particularly in vision tasks where reliable robustness is expected across all categories. To address this challenge, we propose RL-FAT, a reinforcement-learning-inspired fair adversarial training framework that uses policy-gradient based feedback from adversarial predictions. RL-FAT interprets the prediction distribution as a policy and combines correctness-based prediction rewards with class-wise value estimates to compute class-specific advantages for policy-gradient optimization. This enables the model to adaptively focus on class-wise misclassification. Furthermore, we introduce a fairness-emphasis adversarial loss that assigns stronger training pressure to classes with high adversarial loss, thereby mitigating class-wise robustness disparity. By combining reinforcement-driven adaptation with fairness-emphasis regularization, RL-FAT improves adversarial robustness while promoting a more balanced robustness distribution across classes. Extensive experiments demonstrate that our method achieves competitive robust accuracy and substantially reduces class-wise robustness imbalance compared with standard adversarial training baselines.

Figures & tables

Panel of Figure 1 — Figure 1: Per-class clean and AutoAttack robust accuracy at $\epsilon=8/255$ on CIFAR-10. The shaded region indicates the clean-to-robust performance gap, while dotted horizontal lines denote the corresponding mean average clean and robust accuracies.

Figure 2: Overview of the proposed RL-FAT framework. The model is first initialized using warm-up adversarial training to learn stable robust representations from clean and adversarial examples. After warm-up, RL-FAT performs reinforcement-inspired fair adversarial training. Given an adversarial input, the model prediction distribution is interpreted as a state, a class prediction is sampled as an action, and a reward is assigned according to prediction correctness. A class-wise running value estimate is used to compute the advantage, which guides the reinforcement loss $\mathrm{L}_{\mathrm{RL}}$ . In parallel, the fairness-emphasis loss $\mathrm{L}_{\mathrm{fair}}$ assigns stronger training pressure to classes with above-average adversarial loss. The final post-warm-up objective combines both components as $\mathrm{L}_{\mathrm{RL}}+\lambda\mathrm{L}_{\mathrm{fair}}$ .

Method	Avg. Clean	Worst Clean	Avg. Robust	Worst Robust	$\rho_{\mathrm{rob}}$
TRADES	81.75 $\pm$ 0.61	66.52 $\pm$ 1.53	$49.22\pm 0.74$	23.80 $\pm$ 1.35	0.00
FRL	82.02 $\pm$ 0.20	70.27 $\pm$ 0.76	44.91 $\pm$ 0.30	27.60 $\pm$ 0.74	24.72
WAT	79.02 $\pm$ 0.21	70.00 $\pm$ 1.31	45.92 $\pm$ 0.19	29.44 $\pm$ 1.39	30.40
CFA	79.02 $\pm$ 0.13	64.01 $\pm$ 0.04	49.30 $\pm$ 0.21	27.21 $\pm$ 0.67	14.17
DAFA	81.49 $\pm$ 0.63	66.66 $\pm$ 0.66	48.03 $\pm$ 0.80	31.02 $\pm$ 0.66	32.75
RL-FAT	82.99 $\pm$ 0.09	67.52 $\pm$ 0.44	47.88 $\pm$ 0.20	32.38 $\pm$ 0.44	38.77

Table 1: Robust fairness comparison on CIFAR-10. We report average and worst-class accuracy under clean and adversarial evaluation for the best checkpoint evaluations using AutoAttack Lorenz et al. (2022) with epsilon

\epsilon=8/255

on ResNet-18 over five seeds. The fairness metric

\rho_{\mathrm{rob}}

measures the relative improvement in worst-class robust accuracy over the TRADES baseline, adjusted by the relative change in average robust accuracy. Higher

\rho_{\mathrm{rob}}

indicates better robust fairness improvement. The best result in each column is set in bold.

Method	Avg. Clean	Worst Clean	Avg. Robust	Worst Robust	$\rho_{\mathrm{rob}}$
TRADES	$58.12\pm 0.82$	$16.80\pm 2.17$	$25.13\pm 0.36$	$1.30\pm 0.84$	$0.00$
FRL	$\mathbf{59.03\pm 1.02}$	$\mathbf{26.00\pm 3.32}$	$21.97\pm 0.49$	$1.40\pm 0.89$	$20.27$
WAT	$45.34\pm 0.71$	$19.60\pm 1.52$	$16.01\pm 0.47$	$1.60\pm 0.84$	$59.37$
CFA	$58.25\pm 0.82$	$18.20\pm 2.17$	$\mathbf{29.44\pm 0.32}$	$1.80\pm 0.84$	$21.31$
DAFA	$57.94\pm 0.53$	$19.80\pm 2.17$	$24.42\pm 0.58$	$2.10\pm 1.64$	$64.36$
RL-FAT	$56.64\pm 0.24$	$18.80\pm 1.30$	$24.81\pm 0.15$	$\mathbf{2.20\pm 0.45}$	$\mathbf{70.50}$

Table 2: Robust fairness comparison on CIFAR-100. We report mean and standard deviation over five seeds for the best checkpoint evaluations using AutoAttack on ResNet-18 with

\epsilon=8/255

. The fairness metric

\rho_{\mathrm{rob}}

measures the relative improvement in worst-class robust accuracy over the TRADES baseline, adjusted by the relative change in average robust accuracy. Higher

\rho_{\mathrm{rob}}

indicates better robust fairness improvement.

Method	Avg. Clean	Worst Clean	Avg. Robust	Worst Robust	$\rho_{\mathrm{rob}}\uparrow$
TRADES	$76.00\pm 0.83$	$58.74\pm 0.88$	$48.66\pm 0.38$	$22.90\pm 1.18$	$0.00$
FRL	$\mathbf{76.76\pm 1.38}$	$\mathbf{64.34\pm 1.42}$	$45.75\pm 1.48$	$27.35\pm 3.04$	$25.41$
WAT	$50.24\pm 1.20$	$39.77\pm 2.27$	$24.13\pm 0.34$	$11.26\pm 1.90$	$-0.42$
CFA	$75.46\pm 0.25$	$51.23\pm 2.52$	$\mathbf{50.68\pm 0.86}$	$23.50\pm 1.79$	$-1.53$
DAFA	$75.81\pm 0.29$	$63.89\pm 1.56$	$48.17\pm 0.63$	$28.08\pm 1.53$	$23.63$
RL-FAT	$75.92\pm 0.31$	$61.96\pm 0.87$	$47.91\pm 0.52$	$\mathbf{28.79\pm 1.77}$	$\mathbf{27.26}$

Table 3: Robust fairness comparison on ImageNette. We report average and worst-class accuracy under clean and adversarial evaluation, with values shown as mean and standard deviation over five seeds. Adversarial robustness is evaluated using AutoAttack with ResNet-18. we evaluate the robust fairness metric

\rho_{\mathrm{rob}}

, computed relative to the TRADES baseline. Higher values indicate stronger robust fairness improvement. Best results are set in bold.

Steered Class	Avg. Clean	Worst Clean	Worst Clean Class	Target Clean	Avg. Robust	Worst Robust	Worst Robust Class	Target Robust
Airplane	$71.20\pm 0.50$	$32.26\pm 0.86$	Dog (4/5)	$96.12\pm 0.13$	$43.90\pm 0.31$	$13.90\pm 0.46$	Dog (3/5)	$79.18\pm 0.65$
Automobile	$70.77\pm 0.55$	$33.20\pm 0.87$	Dog (5/5)	$97.52\pm 0.49$	$44.80\pm 0.24$	$14.68\pm 0.54$	Dog (4/5)	$80.88\pm 1.47$
Bird	$60.70\pm 0.76$	$14.46\pm 2.15$	Deer (5/5)	$99.56\pm 0.11$	$40.01\pm 0.65$	$4.64\pm 0.45$	Deer (5/5)	$91.84\pm 1.58$
Cat	$59.01\pm 0.28$	$17.88\pm 1.29$	Dog (4/5)	$\mathbf{99.72\pm 0.11}$	$38.97\pm 0.26$	$5.14\pm 0.93$	Deer (5/5)	$\mathbf{94.42\pm 0.61}$
Deer	$64.41\pm 0.68$	$28.14\pm 1.07$	Bird (5/5)	$99.56\pm 0.18$	$42.78\pm 0.38$	$11.86\pm 0.86$	Frog (5/5)	$92.98\pm 0.85$
Dog	$61.03\pm 0.63$	$1.30\pm 1.05$	Cat (5/5)	$99.60\pm 0.10$	$40.36\pm 0.31$	$0.26\pm 0.30$	Cat (5/5)	$93.24\pm 0.42$

Table 4: Class-specific performance steering on CIFAR-10 using a ResNet-18 model. Each row reports results when RL-FAT is steered toward the target class listed in the first column. We report the average accuracy across all classes, the worst-class accuracy, and the accuracy of the steered target class under both clean evaluation and adversarial evaluation with AutoAttack. For the worst-class columns, we also report the class that most frequently attains the lowest accuracy across five random seeds; the value in parentheses indicates how many of the five seeds identified that class as the worst-performing class.

Panel of Figure 3 — Figure 3: SVHN out-of-distribution detection performance with CIFAR-10 as the in-distribution dataset. The left plot shows overall MaxLogit metrics, while the right plot shows worst-class MaxLogit metrics. Green circles highlight RL-FAT results. Higher values indicate better performance.

Variant	Avg. Clean $\uparrow$	Worst Clean $\uparrow$	Avg. Robust $\uparrow$	Worst Robust $\uparrow$	$\rho_{\mathrm{rob}}$ $\uparrow$
TRADES	$81.75\pm 0.61$	$66.52\pm 1.53$	$\textbf{49.22}\pm\textbf{0.74}$	$23.80\pm 1.35$	$0.00$
RL-FAT w/o fairness loss	$\textbf{83.06}\pm\textbf{0.27}$	$67.06\pm 0.87$	$48.88\pm 0.14$	$27.00\pm 0.67$	$14.14$
RL-FAT worst-case loss	$83.01\pm 0.35$	$\textbf{67.77}\pm\textbf{2.01}$	$47.76\pm 0.43$	$31.60\pm 0.53$	$35.74$
RL-FAT	$82.99\pm 0.09$	$67.52\pm 0.44$	$47.88\pm 0.20$	$\textbf{32.38}\pm\textbf{0.44}$	38.77

Table 5: Ablation study of RL-FAT components on CIFAR-10. All variants are evaluated using AutoAttack at

\epsilon=8/255

using ResNet-18. We report average and worst-class accuracy under clean and adversarial evaluation along with fairness metric

\rho_{\mathrm{rob}}

. Higher clean accuracy, robust accuracy, worst-class accuracy, and

\rho_{\mathrm{rob}}

are better. The best result in each column is set in bold.

Method	Avg. Clean	Worst Clean	Avg. Robust	Worst Robust	$\rho_{\mathrm{rob}}\uparrow$
PGD	$84.06\pm 0.65$	$64.58\pm 1.25$	$48.80\pm 0.38$	$19.24\pm 1.06$	$0.00$
FRL	$83.53\pm 0.31$	$64.88\pm 2.01$	$45.54\pm 0.37$	$23.16\pm 1.30$	$27.05$
CFA	$81.36\pm 0.20$	$63.80\pm 0.31$	$\textbf{49.86}\pm\textbf{0.30}$	$22.83\pm 0.50$	$16.49$
DAFA	$\textbf{84.18}\pm\textbf{0.92}$	$\textbf{67.24}\pm\textbf{2.36}$	$48.70\pm 0.32$	$26.36\pm 1.36$	$37.21$
RL-FAT	$83.01\pm 0.43$	$65.06\pm 1.52$	$48.18\pm 0.48$	$\textbf{31.98}\pm\textbf{0.78}$	67.49

Table 6: Robust fairness comparison on CIFAR-10 using PGD adversarial training on ResNet-18. We report average and worst-class accuracy under clean and adversarial evaluation for the best checkpoint evaluations using AutoAttack Lorenz et al. (2022) with epsilon

\epsilon=8/255

. The best result in each accuracy column is highlighted in bold, while the highest fairness metric is highlighted in bold.

Method	Avg. Clean	Worst Clean	Avg. Robust	Worst Robust	$\rho_{\mathrm{rob}}\uparrow$
PGD	$\mathbf{90.06\pm 0.30}$	$\mathbf{79.20\pm 1.47}$	$\mathbf{56.11\pm 0.60}$	$30.30\pm 1.72$	$0.00$
FRL	$89.26\pm 0.32$	$78.70\pm 0.91$	$48.75\pm 0.40$	$27.50\pm 1.03$	$3.88$
RL-FAT	$89.34\pm 0.17$	$77.10\pm 1.32$	$54.42\pm 0.30$	$\mathbf{37.80\pm 0.69}$	$\mathbf{27.76}$

Table 7: Robust fairness comparison on CIFAR-10 using PGD adversarial training with XCiT-S12. We report average and worst-class accuracy under clean and adversarial evaluation for the best checkpoint using AutoAttack Lorenz et al. (2022) with

\epsilon=8/255

over three seeds. The fairness metric

\rho_{\mathrm{rob}}

measures the relative improvement in worst-class robust accuracy over the PGD baseline, adjusted by the relative change in average robust accuracy. Higher

\rho_{\mathrm{rob}}

indicates better robust fairness improvement. The best result in each column is set in bold.

Method	Overall AUROC $\uparrow$	Overall AUPR-IN $\uparrow$	Overall AUPR-OUT $\uparrow$	Worst AUROC $\uparrow$	Worst AUPR-IN $\uparrow$	Worst AUPR-OUT $\uparrow$
TRADES Zhang et al. (2019)	79.05	69.27	86.56	65.84	7.59	97.25
FRL Xu et al. (2021a)	85.90	79.01	91.16	76.58	12.83	98.36
WAT Li and Liu (2023)	64.68	50.14	76.25	46.98	4.46	94.44
DAFA Lee et al. (2024)	78.82	71.51	85.04	67.35	8.78	97.30
RL-FAT	83.78	78.57	88.65	77.79	24.86	98.40

Table 8: SVHN out-of-distribution detection with CIFAR-10 as the in-distribution dataset. We report overall and worst-class MaxLogit results. Higher AUROC, AUPR-IN, and AUPR-OUT indicate better OOD detection.

Corruption	S1	S2	S3	S4	S5
Gaussian noise	+0.5	+0.5	+0.3	+0.5	+0.4
Shot noise	+0.6	+0.4	+0.5	+0.4	+0.1
Impulse noise	+0.9	+1.0	+0.4	+0.5	-0.9
Snow	+0.4	+0.7	+0.5	+1.6	+2.3

Table 9: Selected CIFAR-10-C corruption robustness gains of RL-FAT over TRADES. We include corruptions where RL-FAT consistently improves worst-class accuracy across all five severity levels. Each value is computed as RL-FAT minus TRADES and is reported in %. Left: average accuracy gain. Right: worst-class accuracy gain, which reflects class-wise robustness under corruption shift.

Method	Time/epoch	Mem.
TRADES	28.25s	1.84G
WAT	69.18s	1.80G
FRL	34.60s	1.89G
DAFA	28.27s	1.85G
CFA	39.40s	1.98G
RL-FAT	40.00s	1.85G

Table 10: Training time and memory usage on a single NVIDIA H100 (94 GB VRAM) with batch size 128.

Adversarial Training

Adversarial training improves model reliability by training on examples or conditions designed to expose weaknesses.

See all 177 papers

Apr 29, 2026

Robust Alignment: Harmonizing Clean Accuracy and Adversarial Robustness in Adversarial Training

Adversarial Training (AT) is one of the most effective methods for developing robust deep neural networks (DNNs). However, AT faces a trade-off problem between clean accuracy and adversarial robustness. In this work, we reveal a surprising phenomenon for the first time: Varying input perturbation intensities for training samples near decision boundaries in AT have minimal impact on model robustness. This finding directly exposes the inconsistency between accuracy and robustness score fluctuations, leading us to identify the misalignment between input and latent spaces as a critical driver of the robustness-accuracy trade-off. To mitigate this misalignment for harmonizing accuracy and robustness, we define Robust Alignment as a new AT target, encouraging the model perception to change with input perturbations provided the final label prediction remains unchanged, which can be achieved via two novel ideas. First, we suggest a reduced and fixed perturbation intensity for those boundary samples, which facilitates the model to utilize the perturbations as learnable patterns, instead of noises that complicate decision boundaries meaninglessly. Second, we propose a Domain Interpolation Consistency Adversarial Regularization (DICAR), based on rigorous theoretical derivations, which explicitly introduces semantic alignment between input and latent spaces into AT. Based on these two ideas, we end up with a new Robust Alignment Adversarial Training (RAAT) method, effectively harmonizing accuracy and robustness. Extensive experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet with ResNet-18, PreActResNet-18, and WideResNet-28-10 demonstrate the effectiveness of RAAT in improving the trade-off beyond four common baselines and a total of 14 related state-of-the-art (SOTA) works.

Yanyun Wang, Qingqing Ye, Li Liu +2

1HK PolyU · 2HKUST (GZ) · 2HKUST (GZ

Adversarial Training Neural Network Robustness Adversarial Robustness Representation Alignment

Ask PDF

May 13, 2026

Taming the Long Tail: Rebalancing Adversarial Training via Adaptive Perturbation

Deep neural networks are highly vulnerable to adversarial examples, i.e.,small perturbations that can significantly degrade model performance. While adversarial training has become the primary defense strategy, most studies focus on balanced datasets, overlooking the challenges posed by real-world long-tail data. Motivated by the fact that perturbations in adversarial examples inherently alter the training distribution, we theoretically investigate their impact. We first revisit adversarial training for long-tail data and identify two key limitations: (i) a skewed training objective caused by class imbalance, and (ii) unstable evolution of adversarial distributions. Furthermore, we show that perturbations can simultaneously address both adversarial vulnerability and class imbalance. Based on these insights, we propose RobustLT, a plug-and-play framework that adaptively adjusts perturbations during adversarial training. Extensive experiments demonstrate that RobustLT consistently enhances adversarial robustness and class-balance on long-tailed datasets. The code is available at https://github.com/zhang-lilin/RobustLT.

Lilin Zhang, Yimo Guo, Yue Li +2

1Sichuan University · 2Dongfang Electric (Chengdu) Innovation Research Co., Ltd. · 3Southwest China Research Institute of Electronic Equipment

Adversarial Training Class-Imbalanced Learning Long-Tail Learning Adversarial Robustness Adversarial Perturbations

Ask PDF

Sep 30, 2026

Probabilistic Adversarial Training

Building on a probabilistic perspective in which adversarial examples arise from the overlap between a distance-based distribution

p_{\mathrm{dis}}

and a victim-classifier-induced distribution

p_{\mathrm{vic}}

, we start from a simple intuition: adversarial examples become harder to generate when these two distributions are pushed apart, as their overlap becomes smaller, thereby increasing robustness. This intuition naturally motivates a KL-based robustness objective. We then prove that

\mathrm{KL}(p_{\mathrm{dis}}\|p_{\mathrm{vic}})-\log Z_{\mathrm{vic}}

is a lower bound on probabilistic robustness (PR), where

Z_{\mathrm{vic}}

denotes the normalizing constant of

p_{\mathrm{vic}}

. Since PR is generally intractable to compute directly, maximizing this KL-based lower bound provides a tractable surrogate objective for improving PR. We further show that this objective recovers a scaled form of adversarial training, offering a probabilistic interpretation of adversarial training and a principled route to robustness improvement. We call the resulting method probabilistic adversarial training. Experiments show that it consistently improves PR, and ablation studies demonstrate that the induced scaling factor can even enhance the PR of non-probabilistic adversarial training methods.

Andi Zhang, Xingyu Zhao, Siddartha Khastgir

WMG, University of Warwick · Wuhan University

Adversarial Training Adversarial Robustness

Ask PDF

Abstract

Adversarial Training

Explore similar work

Robust Alignment: Harmonizing Clean Accuracy and Adversarial Robustness in Adversarial Training

Taming the Long Tail: Rebalancing Adversarial Training via Adaptive Perturbation

Probabilistic Adversarial Training