Adversarial robustness is essential for deploying neural networks in safety-critical applications, yet standard evaluation methods either require expensive adversarial attacks or report only a single aggregate score that obscures how robustness is distributed across classes. We introduce the GF-Score (GREAT-Fairness Score), a framework that decomposes the certified GREAT Score into per-class robustness profiles and quantifies their disparity through four metrics grounded in welfare economics: the Robustness Disparity Index (RDI), the Normalized Robustness Gini Coefficient (NRGC), Worst-Case Class Robustness (WCR), and a Fairness-Penalized GREAT Score (FP-GREAT). The framework further eliminates the original method's dependence on adversarial attacks through a self-calibration procedure that tunes the temperature parameter using only clean accuracy correlations. Evaluating 22 models from RobustBench across CIFAR-10 and ImageNet, we find that the decomposition is exact, that per-class scores reveal consistent vulnerability patterns (e.g., ``cat'' is the weakest class in 76% of CIFAR-10 models), and that more robust models tend to exhibit greater class-level disparity. These results establish a practical, attack-free auditing pipeline for diagnosing where certified robustness guarantees fail to protect all classes equally. We release our code on GitHub.
Figures & tables
Figure 1: Per-class GREAT Scores for 17 CIFAR-10 models reveal substantial class-level robustness disparity hidden by aggregate scores. Each row is a model (sorted by aggregate GREAT Score); each column is a class. The class “cat” is consistently the most vulnerable (darkest column), while “automobile” is most often the most robust (10 of 17 models). Models with higher aggregate robustness (top rows) tend to exhibit greater disparity between their strongest and weakest classes.
Method
Certified
Per-class
Attack-free
Post-hoc
Fairness metrics
AutoAttack ( Croce and Hein, 2020 )
✗
✗
✗
✓
✗
RobustBench ( Croce et al., 2021 )
✗
✗
✗
✓
✗
Rand. Smoothing ( Cohen et al., 2019 )
✓
✓ *
✓
✓
✗
GREAT Score ( Li et al., 2024 )
✓
✗
✓
✓
✗
FRL ( Xu et al., 2021 )
✗
✓
✗
✗
✗
CFA ( Wei et al., 2023 )
✗
✓
✗
✗
✗
Table 1: Comparison of the GF-Score with prior methods across five key dimensions. Certified : provides provable robustness guarantees. Per-class : offers class-level granularity. Attack-free : does not require adversarial attacks. Post-hoc : applicable to already-trained models. Fairness metrics : includes formal disparity quantification grounded in established theory.
CIFAR-10 ( ℓ2 , 17 models)
ImageNet ( ℓ∞ , 5 models)
Uncalibrated
Calibrated
Uncalibrated
Calibrated
Original ( Li et al., 2024 )
0.662
0.897 †
0.800
— ‡
GF-Score (Ours)
0.662
0.871
0.900
1.000
† Uses C&W attack for calibration; ours is fully attack-free.
‡ Calibration not performed for ImageNet in the original paper.
Table 2: Spearman rank correlation ( ρ ) with RobustBench accuracy rankings. Self-calibration matches or exceeds the original attack-based calibration while being fully attack-free.
Figure 2: Aggregate GREAT Score vs. RDI. Higher robustness correlates with greater class-level disparity on both datasets, revealing a quantifiable robustness-fairness tension.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Model
RB Acc
GS
Cal. GS
RDI
NRGC
WCR
WCR Class
FP-GR
Rebuffi_extra
82.32
0.465
0.330
0.333
0.135
0.283
cat
0.298
Gowal_extra
80.53
0.480
0.344
0.348
0.138
0.288
cat
0.306
Rebuffi_70_ddpm
80.42
0.381
0.277
0.360
0.178
0.166
cat
0.201
Augustin_WRN_ext
78.79
0.526
0.330
0.319
0.105
0.335
cat
0.366
Rebuffi_28_ddpm
78.80
0.352
0.255
0.359
0.191
0.144
cat
0.173
Sehwag_Proxy
77.24
0.232
0.290
0.302
0.250
0.060
cat
0.081
Appendix
Table 3: Full GF-Score results for 17 CIFAR-10 ℓ2 models, sorted by RobustBench accuracy. RB Acc : RobustBench robust accuracy (%). GS : uncalibrated GREAT Score. Cal. GS : calibrated GREAT Score ( T∗=2.70 ). FP-GR : Fairness-Penalized GREAT ( λ=0.5 ).
Model
RB Acc
GS
Cal. GS
RDI
NRGC
WCR
WCR Class
FP-GR
Salman_WRN50-2
38.14
0.545
0.800
1.231
0.299
0.009
n01756291
− 0.070
Salman_R50
34.96
0.444
0.730
1.198
0.350
0.003
n04525038
− 0.155
Engstrom2019
29.22
0.446
0.717
1.196
0.361
0.003
n03710637
− 0.152
Wong2020
26.24
0.360
0.591
1.148
0.388
0.000
n04525038
− 0.214
Salman_R18
25.32
0.280
0.575
1.126
0.454
0.000
n04525038
− 0.283
Appendix
Table 4: Full GF-Score results for 5 ImageNet ℓ∞ models, sorted by RobustBench accuracy. Calibrated at T∗=0.10 . Two models achieve WCR=0.000 , indicating at least one class with zero certified robustness.
Model
airplane
auto
bird
cat
deer
dog
frog
horse
ship
truck
Agg.
Aug._WRN_ext
.579
.654
.463
.335
.430
.444
.518
.598
.613
.621
.526
Aug._WRN
.486
.628
.435
.242
.404
.346
.548
.577
.584
.585
.483
Aug.2020
.526
.652
.440
.218
.419
.336
.537
.582
.584
.588
.488
Ding_MMA
.084
.128
.074
.039
.064
.047
.090
.166
.090
.080
.086
Engstrom
.115
.232
.092
.038
.077
.024
.102
.168
.158
.258
.126
Gowal2020
.112
.122
.082
.061
.098
.046
.118
.167
.153
.146
.111
Appendix
Table 5: Per-class GREAT Scores for all 17 CIFAR-10 ℓ2 models (uncalibrated, T=1.0 ). The class “cat” (column 4) is consistently the lowest-scoring class, while “automobile” (column 2) is consistently the highest. The final column shows the aggregate score, which equals the weighted mean of per-class scores with zero numerical error.
Figure 3: Disparity metric bar charts. Models are sorted by aggregate GREAT Score. Higher RDI and NRGC indicate greater disparity; lower WCR indicates worse fairness. All ImageNet models exhibit high RDI ( >1.1 ) due to the extreme class diversity (1,000 classes).
Figure 4: Class vulnerability analysis. On CIFAR-10, “cat” appears as the worst-case class in 13 of 17 models (76%), while “automobile” is the best class in 10 of 17 (59%). On ImageNet, “n04525038” (viaduct) is worst in 3 of 5 models, while “n12057211” (cattail) is best in all 5.
Figure 5: FP-GREAT re-ranking. On CIFAR-10, models with low disparity (e.g., Wu2020) rise, while those with high disparity (e.g., Augustin2020) fall. On ImageNet, the high RDI values ( >1.1 ) cause all FP-GREAT scores to be negative at λ=0.5 .
Figure 6: Self-calibration curves. Both curves are smooth, confirming that the two-phase grid search reliably finds the optimum. On ImageNet, perfect agreement with RobustBench rankings is achieved using only clean accuracies.
Figure 7: RDI concentration bounds. The bound tightens as per-class sample size increases. Our CIFAR-10 evaluation with 1,000 samples per class provides tight estimates; the ImageNet union bound over 1,000 classes requires more samples, but 50 per class still achieves a bound of approximately 0.35.
Figure 8: Radar chart of per-class GREAT Scores for selected CIFAR-10 models. Each axis represents one class. Models with higher aggregate scores (outer polygons) show more pronounced asymmetry between classes, visually confirming the robustness-fairness tension.
Figure 9: Per-class GREAT Scores for 5 ImageNet ℓ∞ models. The extreme contrast between the brightest and darkest columns reflects the high RDI values ( >1.1 ) reported in Table 4 . Several classes receive near-zero scores across all models.
Deep neural networks are vulnerable to adversarial perturbations that can simultaneously degrade prediction robustness and individual fairness across diverse application settings. However, existing evaluation protocols typically assess these dimensions in isolation, thereby obscuring critical failure modes. To bridge this gap, we formalize Robust Individual Fairness (RIF): under semantic-preserving (truth-condition-preserving) perturbations, predictions should remain both correct with respect to the ground truth and invariant across semantically equivalent individuals. To surface RIF violations in practice, we introduce RIFair, a black-box adversarial framework that leverages a decoupled perturbation strategy to construct semantically preserved yet unrobust and/or unfair instance pairs. Experiments across multiple model architectures and real-world textual datasets show that robustness-only or fairness-only metrics often miss Robust Biased and Unrobust Fair behaviors. RIFair}reliably exposes these hidden vulnerabilities, supporting RIF as a necessary criterion for trustworthy model assessment. The experimental code is publicly available at https://github.com/Xuran-LI/RIFair.
Xuran Li, Hao Xue, Peng Wu +4
University of New South Wales Sydney, Australia · The Hong Kong University of Science and Technology Guangzhou, China · Key Laboratory of System Software, Institute of Software, Chinese Academy of Sciences Beijing, China +3
The robustness of deep neural networks is crucial for safety-critical deployments, yet existing evaluation methods are often attack-dependent and lack interpretability. We propose a principled, attack-agnostic robustness metric based on the spectral norm of the Fisher Information Matrix (FIM), which quantifies the worst-case sensitivity of the model's output distribution to input perturbations. Theoretically, we establish that the FIM equals the variance of the input Jacobian and derive closed-form spectral bounds for common architectures, including VGG, ResNet, DenseNet, and Transformer, providing the first theoretical robustness ranking. To enable scalable evaluation, we develop efficient algorithms, including power iteration and Hutchinson-based estimation, that support both white-box and black-box settings. Extensive experiments across multiple datasets, including CIFAR, ImageNet, and medical images, and across multiple architectures show a strong correlation between our metric and adversarial vulnerability. Our framework serves as an interpretable diagnostic tool that complements attack-based evaluations, offering insights into architectural sensitivity and guiding the design of more robust models. Code is available at: https://github.com/franz-chang/SRP/.
Chong Zhang, Xiang Li, Jia Wang +2
Xi’an Jiaotong-Liverpool University · The Chinese University of Hong Kong
Deep neural networks remain highly vulnerable to adversarial perturbations, and adversarial training (AT) has become a widely used approach for improving robustness. However, improvements in average robust accuracy often mask substantial class-wise disparities: while some classes become more robust, others may remain disproportionately vulnerable under attack. This imbalance raises an important adversarial fairness concern, particularly in vision tasks where reliable robustness is expected across all categories. To address this challenge, we propose RL-FAT, a reinforcement-learning-inspired fair adversarial training framework that uses policy-gradient based feedback from adversarial predictions. RL-FAT interprets the prediction distribution as a policy and combines correctness-based prediction rewards with class-wise value estimates to compute class-specific advantages for policy-gradient optimization. This enables the model to adaptively focus on class-wise misclassification. Furthermore, we introduce a fairness-emphasis adversarial loss that assigns stronger training pressure to classes with high adversarial loss, thereby mitigating class-wise robustness disparity. By combining reinforcement-driven adaptation with fairness-emphasis regularization, RL-FAT improves adversarial robustness while promoting a more balanced robustness distribution across classes. Extensive experiments demonstrate that our method achieves competitive robust accuracy and substantially reduces class-wise robustness imbalance compared with standard adversarial training baselines.
Tejaswini Medi, Levan Mikeladze, Margret Keuper
Chair for Machine Learning University of Mannheim Germany · MPI for Informatics Saarland Informatics Campus Germany