GF-Score: Certified Class-Conditional Robustness Evaluation with Fairness Guarantees

Arya Shah; Kaveri Visavadiya; Manisha Padala

cs.LGApr 14, 2026

GF-Score: Certified Class-Conditional Robustness Evaluation with Fairness Guarantees

Authors: Arya Shah, Kaveri Visavadiya, Manisha Padala

Organizations: IIT Gandhinagar

Abstract

Adversarial robustness is essential for deploying neural networks in safety-critical applications, yet standard evaluation methods either require expensive adversarial attacks or report only a single aggregate score that obscures how robustness is distributed across classes. We introduce the GF-Score (GREAT-Fairness Score), a framework that decomposes the certified GREAT Score into per-class robustness profiles and quantifies their disparity through four metrics grounded in welfare economics: the Robustness Disparity Index (RDI), the Normalized Robustness Gini Coefficient (NRGC), Worst-Case Class Robustness (WCR), and a Fairness-Penalized GREAT Score (FP-GREAT). The framework further eliminates the original method's dependence on adversarial attacks through a self-calibration procedure that tunes the temperature parameter using only clean accuracy correlations. Evaluating 22 models from RobustBench across CIFAR-10 and ImageNet, we find that the decomposition is exact, that per-class scores reveal consistent vulnerability patterns (e.g., ``cat'' is the weakest class in 76% of CIFAR-10 models), and that more robust models tend to exhibit greater class-level disparity. These results establish a practical, attack-free auditing pipeline for diagnosing where certified robustness guarantees fail to protect all classes equally. We release our code on GitHub.

Figures & tables

Figure 1: Per-class GREAT Scores for 17 CIFAR-10 models reveal substantial class-level robustness disparity hidden by aggregate scores. Each row is a model (sorted by aggregate GREAT Score); each column is a class. The class “cat” is consistently the most vulnerable (darkest column), while “automobile” is most often the most robust (10 of 17 models). Models with higher aggregate robustness (top rows) tend to exhibit greater disparity between their strongest and weakest classes.

Method	Certified	Per-class	Attack-free	Post-hoc	Fairness metrics
AutoAttack ( Croce and Hein, 2020 )	✗	✗	✗	✓	✗
RobustBench ( Croce et al., 2021 )	✗	✗	✗	✓	✗
Rand. Smoothing ( Cohen et al., 2019 )	✓	✓ *	✓	✓	✗
GREAT Score ( Li et al., 2024 )	✓	✗	✓	✓	✗
FRL ( Xu et al., 2021 )	✗	✓	✗	✗	✗
CFA ( Wei et al., 2023 )	✗	✓	✗	✗	✗

Table 1: Comparison of the GF-Score with prior methods across five key dimensions. Certified : provides provable robustness guarantees. Per-class : offers class-level granularity. Attack-free : does not require adversarial attacks. Post-hoc : applicable to already-trained models. Fairness metrics : includes formal disparity quantification grounded in established theory.

$\dagger$ Uses C&W attack for calibration; ours is fully attack-free.
	CIFAR-10 ( $\ell_{2}$ , 17 models)		ImageNet ( $\ell_{\infty}$ , 5 models)
	Uncalibrated	Calibrated	Uncalibrated	Calibrated
Original ( Li et al., 2024 )	0.662	0.897 $\dagger$	0.800	— $\ddagger$
GF-Score (Ours)	0.662	0.871	0.900	1.000
$\ddagger$ Calibration not performed for ImageNet in the original paper.

Table 2: Spearman rank correlation (

\rho

) with RobustBench accuracy rankings. Self-calibration matches or exceeds the original attack-based calibration while being fully attack-free.

Panel of Figure 2 — Figure 2: Aggregate GREAT Score vs. RDI. Higher robustness correlates with greater class-level disparity on both datasets, revealing a quantifiable robustness-fairness tension.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Model	RB Acc	GS	Cal. GS	RDI	NRGC	WCR	WCR Class	FP-GR
Rebuffi_extra	82.32	0.465	0.330	0.333	0.135	0.283	cat	0.298
Gowal_extra	80.53	0.480	0.344	0.348	0.138	0.288	cat	0.306
Rebuffi_70_ddpm	80.42	0.381	0.277	0.360	0.178	0.166	cat	0.201
Augustin_WRN_ext	78.79	0.526	0.330	0.319	0.105	0.335	cat	0.366
Rebuffi_28_ddpm	78.80	0.352	0.255	0.359	0.191	0.144	cat	0.173
Sehwag_Proxy	77.24	0.232	0.290	0.302	0.250	0.060	cat	0.081

Appendix

Table 3: Full GF-Score results for 17 CIFAR-10

\ell_{2}

models, sorted by RobustBench accuracy. RB Acc : RobustBench robust accuracy (%). GS : uncalibrated GREAT Score. Cal. GS : calibrated GREAT Score (

T^{*}=2.70

). FP-GR : Fairness-Penalized GREAT (

\lambda=0.5

Model	RB Acc	GS	Cal. GS	RDI	NRGC	WCR	WCR Class	FP-GR
Salman_WRN50-2	38.14	0.545	0.800	1.231	0.299	0.009	n01756291	$-$ 0.070
Salman_R50	34.96	0.444	0.730	1.198	0.350	0.003	n04525038	$-$ 0.155
Engstrom2019	29.22	0.446	0.717	1.196	0.361	0.003	n03710637	$-$ 0.152
Wong2020	26.24	0.360	0.591	1.148	0.388	0.000	n04525038	$-$ 0.214
Salman_R18	25.32	0.280	0.575	1.126	0.454	0.000	n04525038	$-$ 0.283

Appendix

Table 4: Full GF-Score results for 5 ImageNet

\ell_{\infty}

models, sorted by RobustBench accuracy. Calibrated at

T^{*}=0.10

. Two models achieve

\mathrm{WCR}=0.000

, indicating at least one class with zero certified robustness.

Model	airplane	auto	bird	cat	deer	dog	frog	horse	ship	truck	Agg.
Aug._WRN_ext	.579	.654	.463	.335	.430	.444	.518	.598	.613	.621	.526
Aug._WRN	.486	.628	.435	.242	.404	.346	.548	.577	.584	.585	.483
Aug.2020	.526	.652	.440	.218	.419	.336	.537	.582	.584	.588	.488
Ding_MMA	.084	.128	.074	.039	.064	.047	.090	.166	.090	.080	.086
Engstrom	.115	.232	.092	.038	.077	.024	.102	.168	.158	.258	.126
Gowal2020	.112	.122	.082	.061	.098	.046	.118	.167	.153	.146	.111

Appendix

Table 5: Per-class GREAT Scores for all 17 CIFAR-10

\ell_{2}

models (uncalibrated,

T=1.0

). The class “cat” (column 4) is consistently the lowest-scoring class, while “automobile” (column 2) is consistently the highest. The final column shows the aggregate score, which equals the weighted mean of per-class scores with zero numerical error.

Panel of Figure 3 — Figure 3: Disparity metric bar charts. Models are sorted by aggregate GREAT Score. Higher RDI and NRGC indicate greater disparity; lower WCR indicates worse fairness. All ImageNet models exhibit high RDI ( $>1.1$ ) due to the extreme class diversity (1,000 classes).

Panel of Figure 4 — Figure 4: Class vulnerability analysis. On CIFAR-10, “cat” appears as the worst-case class in 13 of 17 models (76%), while “automobile” is the best class in 10 of 17 (59%). On ImageNet, “n04525038” (viaduct) is worst in 3 of 5 models, while “n12057211” (cattail) is best in all 5.

Panel of Figure 5 — Figure 5: FP-GREAT re-ranking. On CIFAR-10, models with low disparity (e.g., Wu2020) rise, while those with high disparity (e.g., Augustin2020) fall. On ImageNet, the high RDI values ( $>1.1$ ) cause all FP-GREAT scores to be negative at $\lambda=0.5$ .

Panel of Figure 6 — Figure 6: Self-calibration curves. Both curves are smooth, confirming that the two-phase grid search reliably finds the optimum. On ImageNet, perfect agreement with RobustBench rankings is achieved using only clean accuracies.

Panel of Figure 7 — Figure 7: RDI concentration bounds. The bound tightens as per-class sample size increases. Our CIFAR-10 evaluation with 1,000 samples per class provides tight estimates; the ImageNet union bound over 1,000 classes requires more samples, but 50 per class still achieves a bound of approximately 0.35.

Figure 8: Radar chart of per-class GREAT Scores for selected CIFAR-10 models. Each axis represents one class. Models with higher aggregate scores (outer polygons) show more pronounced asymmetry between classes, visually confirming the robustness-fairness tension.

Figure 9: Per-class GREAT Scores for 5 ImageNet $\ell_{\infty}$ models. The extreme contrast between the brightest and darkest columns reflects the high RDI values ( $>1.1$ ) reported in Table 4 . Several classes receive near-zero scores across all models.

Adversarial Robustness

Adversarial robustness studies how well AI systems withstand deliberately crafted inputs, training-data corruption, and other attacks.

See all 463 papers

Perturbation Effects on Robustness and Individual Fairness
Apr 1, 2024Xuran Li, Hao Xue, Peng Wu +4Neural Network Robustness Algorithmic Fairness
University of New South Wales Sydney, Australia · The Hong Kong University of Science and Technology Guangzhou, China · Key Laboratory of System Software, Institute of Software, Chinese Academy of Sciences Beijing, China +3
Measuring Model Robustness via Fisher Information: Spectral Bounds, Theoretical Guarantees, and Practical Algorithms
Jun 3, 2026Chong Zhang, Xiang Li, Jia Wang +2Neural Network Robustness Adversarial Robustness
Xi’an Jiaotong-Liverpool University · The Chinese University of Hong Kong
RL-FAT: Reinforcement Learning for Fair Adversarial Training
Aug 29, 2026Tejaswini Medi, Levan Mikeladze, Margret KeuperAdversarial Training Adversarial Robustness
Chair for Machine Learning University of Mannheim Germany · MPI for Informatics Saarland Informatics Campus Germany

Apr 1, 2024J/K move · Enter open · S save

Perturbation Effects on Robustness and Individual Fairness

Xuran Li, Hao Xue, Peng Wu, Xingjun Ma+3

University of New South Wales Sydney, Australia · The Hong Kong University of Science and Technology Guangzhou, China · Key Laboratory of System Software, Institute of Software, Chinese Academy of Sciences Beijing, China+3

Neural Network Robustness Algorithmic Fairness Adversarial Robustness Black-Box Adversarial Attacks Adversarial Text Attacks

Open paper AskPDF arXiv

Deep neural networks are vulnerable to adversarial perturbations that can simultaneously degrade prediction robustness and individual fairness across diverse application settings. However, existing evaluation protocols typically assess these dimensions in isolation, thereby obscuring critical failure modes. To bridge this gap, we formalize Robust Individual Fairness (RIF): under semantic-preserving (truth-condition-preserving) perturbations, predictions should remain both correct with respect to the ground truth and invariant across semantically equivalent individuals. To surface RIF violations in practice, we introduce RIFair, a black-box adversarial framework that leverages a decoupled perturbation strategy to construct semantically preserved yet unrobust and/or unfair instance pairs. Experiments across multiple model architectures and real-world textual datasets show that robustness-only or fairness-only metrics often miss Robust Biased and Unrobust Fair behaviors. RIFair}reliably exposes these hidden vulnerabilities, supporting RIF as a necessary criterion for trustworthy model assessment. The experimental code is publicly available at https://github.com/Xuran-LI/RIFair.

Abstract

Adversarial Robustness

Explore similar work

Perturbation Effects on Robustness and Individual Fairness

Measuring Model Robustness via Fisher Information: Spectral Bounds, Theoretical Guarantees, and Practical Algorithms

RL-FAT: Reinforcement Learning for Fair Adversarial Training