A Rank Graduation metric for Algorithmic fairness

Dalia Atif; Paolo Giudici

cs.LGSep 30, 2026

A Rank Graduation metric for Algorithmic fairness

Authors: Dalia Atif, Paolo Giudici

Abstract

Fairness assessment in algorithmic decisions that affect individuals, such as credit scoring, often relies on parity measures calculated at the aggregate group level. Such measures may not reveal which individuals experience unfairness or which explanatory factors contribute to it. In this paper, we propose a rank-based framework that evaluates fairness through the distribution of model prediction errors, thereby linking fairness assessment with predictive accuracy and explainability. The framework combines Rank Graduation Fairness (RGF), its integrated measure AURGF, a centered Cramer--von Mises permutation test, and a feature removal procedure for fairness explainability. We evaluate the methodology using logistic regression, random forest, gradient boosting, and a multilayer perceptron. The simulation study shows that protected-group imbalance can reverse descriptive fairness comparisons, whereas the proposed inferential procedure correctly distinguishes fair from unfair mechanisms. Its application to HMDA mortgage data produces model rankings that differ from those obtained with classical fairness criteria. Tree-based models, rather than logistic regression, provide the strongest combination of predictive accuracy and rank-based fairness, while the fairness null hypothesis is rejected for all four models. The persistence of disparity across statistical, bagging, boosting, and neural network specifications, together with the feature removal results, indicates that the observed unfairness is not specific to a single algorithm or predictor, but is associated with group differences embedded in the characteristics of the lending data. These findings support a broader approach to trustworthy artificial intelligence that combines predictive accuracy, fairness measurement, statistical inference, and explainability.

Figures & tables

Figure 1 : Illustration of the RGF construction: pooled error ranking and numerical construction (A–B), raw and uniformly completed group curves (C–D), pairwise disparity and normalized fairness measures (E–G), and Cramér–von Mises permutation inference (H).

Figure 2 : Conceptual representation of the Rank Graduation Fairness curve. The area under the curve summarizes fairness persistence across the ranked distribution of error burdens.

Model	Scenario	Group-level prediction error		Rank-based fairness			Statistical inference
Model	Scenario	Majority MAE	Minority MAE	RGD	RGF [95% CI]	AURGF	CvM- $T$	$p$ -value
Logistic regression	Fair	$0.057\,022$	$0.051\,698$	$0.4138$	0.5862 [0.5321, 0.6514]	$0.5870$	$0.1560$	$0.6952$
Logistic regression	Unfair	$0.057\,022$	$0.110\,005$	$0.2825$	0.7175 [0.6493, 0.7976]	$0.7271$	$14.1567$	0.0009
Random forest	Fair	$0.058\,536$	$0.051\,959$	$0.4151$	0.5849 [0.5389, 0.6409]	$0.5810$	$0.2690$	$0.6177$
Random forest	Unfair	$0.058\,536$	$0.112\,069$	$0.3023$	0.6977 [0.6373, 0.7684]	$0.7155$	$12.3189$	0.0005
Gradient boosting	Fair	$0.053\,645$	$0.044\,183$	$0.4212$	0.5788 [0.5280, 0.6427]	$0.5790$	$0.5095$	$0.5287$

Table 1 : Fairness diagnostics under protected-group imbalance

Panel of Figure 3 — Figure 3 : AURGF curves and associated permutation p-values for the four classifiers under the simulated unfair scenario. The horizontal axis represents the retained fraction $q$ of observations with the largest error burdens. The shaded area corresponds to the Area Under the Rank Group Fairness curve (AURGF), while the annotations report permutation $p$ -values at selected burden fractions. Smaller values of $q$ correspond to the most severe prediction errors, with $q=1$ indicating the full sample.

	Predictive performance		Rank-based fairness				Statistical inference
Model	ROC–AUC	PR–AUC	RGD	RGF	RGF 95% CI	AURGF	CvM– $T$	Permutation $p$ -value	$q_{\min}$
Logistic regression	0.6794	0.3198	0.3370	0.6630	[0.6580, 0.6684]	0.5223	78.5705	0.0005	0.0200
Random forest	0.7348	0.4213	0.3295	0.6705	[0.6637, 0.6787]	0.7216	19.4222	0.0005	0.0100
Gradient boosting	0.7364	0.4232	0.3427	0.6573	[0.6523, 0.6628]	0.5454	72.7159	0.0005	0.0100
MLP	0.7303	0.4108	0.3441	0.6559	[0.6510, 0.6615]	0.5488	70.2581	0.0005	0.0100

Table 2 : Predictive performance and rank-based fairness results on the original dataset with imbalanced protected groups

Panel of Figure 4 — Figure 4 : AURGF curves for the four classifiers. The horizontal axis represents the retained fraction $q$ of the observations with the largest error burdens. The shaded area corresponds to the Area Under the Rank Group Fairness curve (AURGF). Labels indicate permutation $p$ -values computed at selected retained fractions. Smaller $q$ values focus on the highest-error observations, whereas $q=1$ corresponds to the full test sample.

Model	Removed feature	$RGF_{\mathrm{full}}$	$RGF_{-X_{r}}$	$\Delta RGF_{r}$	$FC_{r}$	CvM– $T_{-X_{r}}$	$p_{-X_{r}}$
Logistic regression	loan_purpose	0.6630	0.6898	0.0268	0.0796	137.3899	0.0005
	minority_population	0.6630	0.6601	-0.0029	-0.0044	64.4720	0.0005
	co_applicant_sex	0.6630	0.6642	0.0012	0.0036	79.9383	0.0005
	loan_type	0.6630	0.6618	-0.0012	-0.0018	75.1155	0.0005
	number_of_owner_occupied_units	0.6630	0.6635	0.0005	0.0015	80.2181	0.0005
Random forest	loan_purpose	0.6705	0.7538	0.0834	0.2530	40.8801	0.0005

Table 3 : Feature-removal fairness explainability results under imbalanced protected-group representation

Panel of Figure 5 — Figure 5 : Fairness explainability across features under imbalanced protected-group representation. The figures report the normalized fairness contribution ( $FC_{r}$ ) obtained by removing each predictor and re-estimating the corresponding classifier. Positive contributions are shown in green and indicate that removing the corresponding feature increases RGF and therefore improves fairness. Negative contributions are shown in red and indicate that removing the feature decreases RGF and therefore worsens fairness.

Model	Removed feature	$\Delta RGF_{r}$	$FC_{r}$	$\Delta$ ROC–AUC	$\Delta$ PR–AUC
Logistic regression	loan_purpose	$0.0268\,\uparrow$	0.0796	$-0.0757\,\downarrow$	$-0.0560\,\downarrow$
	minority_population	$-0.0029\,\downarrow$	-0.0044	$-0.0028\,\downarrow$	$-0.0082\,\downarrow$
	co_applicant_sex	$0.0012\,\uparrow$	0.0036	$-0.0033\,\downarrow$	$-0.0058\,\downarrow$
	loan_type	$-0.0012\,\downarrow$	-0.0018	$-0.0021\,\downarrow$	$0.0010\,\uparrow$
	number_of_owner_occupied_units	$0.0005\,\uparrow$	0.0015	$-0.0003\,\downarrow$	$0.0004\,\uparrow$
Random forest	loan_purpose	$0.0834\,\uparrow$	0.2530	$-0.0538\,\downarrow$	$-0.0458\,\downarrow$

Table 4 : Main fairness contributions of individual features and associated changes in predictive performance under imbalanced protected-group representation

	Predictive performance		Rank-based fairness				Statistical inference
Model	ROC–AUC	PR–AUC	RGD	RGF	RGF 95% CI	AURGF	CvM– $T$	Permutation $p$ -value	$q_{\min}$
Logistic regression	0.7028	0.4252	0.1906	0.8094	[0.7930, 0.8234]	0.6023	70.6250	0.0005	0.0101
Random forest	0.7296	0.4817	0.2123	0.7877	[0.7607, 0.8152]	0.5492	28.0435	0.0005	0.0101
Gradient boosting	0.7327	0.4903	0.1744	0.8256	[0.8077, 0.8409]	0.6790	65.3432	0.0005	0.0101
MLP	0.7122	0.4541	0.1892	0.8108	[0.7918, 0.8259]	0.6417	67.8869	0.0005	0.0101

Table 5 : Predictive performance and rank-based fairness results after protected-group undersampling

Model	SPD	DI	EOD	FPR difference	PPD
Logistic regression	0.0150	11.4081	0.0203	0.0118	0.0093
Random forest	0.2071	1.6852	0.1566	0.1627	0.1284
Gradient boosting	0.0706	2.7861	0.0766	0.0419	0.0149
MLP	0.0388	2.4498	0.0391	0.0195	0.0298

Table 6 : Classical fairness metrics on the original dataset with imbalanced protected groups

Explore similar work

May 12, 2026cs.LG

Do Fair Models Reason Fairly? Counterfactual Explanation Consistency for Procedural Fairness in Credit Decisions

Machine learning algorithms in socially sensitive domains (e.g., credit decisions) often focus on equalizing predictive outcomes. However, satisfying these metrics does not guarantee that models use the same reasoning for different groups. We show that existing outcome-fair models can still apply fundamentally different reasoning to individuals, a

hidden procedural bias'' missed by standard fairness metrics and algorithms. We propose Counterfactual Explanation Consistency (CEC), a framework that detects and mitigates this bias by aligning feature attributions between individuals and their counterfactual counterparts. Key contributions include a nearest-neighbor counterfactual generation method, a modified baseline for integrated gradient comparisons, an individual-level procedural fairness metric, and a corresponding training loss. We introduce a taxonomy identifying

Regime B'' (same outcome, different reasoning) as a critical blind spot. Experiments on synthetic data, German Credit, Adult Income, and HMDA mortgage data demonstrate that outcome-fair baselines exhibit substantial hidden bias, while CEC substantially reduces it with modest utility cost.

Gideon Popoola, John Sheppard

Gianforte School of Computing Montana State University Bozeman, MT, USA, 59715

Algorithmic Fairness Counterfactual Explanation

Ask PDF

Jul 9, 2026cs.LG

FairSelect: A Systematic Evaluation of Multi-Level and Intersectional Algorithmic Fairness

Algorithmic fairness methods are increasingly used to identify and mitigate bias in machine learning models, yet most approaches are evaluated in isolation and along single demographic axes. This limits practical guidance for selecting fairness strategies, where disparities may arise across intersectional subgroups and across multiple stages of the modeling lifecycle. This work presents FairSelect, a toolkit for systematically evaluating fairness mitigation strategies applied individually and in combination across preprocessing, inprocessing, and postprocessing stages. FairSelect supports multiple model architectures, intersectional subgroup evaluation, and comparison of fairness utility tradeoffs across baseline, single method, and multi level configurations. The framework was validated using synthetic clinical datasets designed to represent specific bias mechanisms and a real-world replication of two-year stroke risk prediction among patients with atrial fibrillation. Synthetic experiments showed that targeted fairness methods generally reduced intended subgroup disparities, while combined strategies produced larger average fairness improvements with modest utility tradeoffs. In the clinical prediction task, mitigation effects were highly variable, with some combinations improving both fairness and predictive performance while others were ineffective or counterproductive. These findings demonstrate that fairness interventions interact in nonadditive and context dependent ways. FairSelect provides a practical framework for systematically identifying fairness strategies that improve subgroup equity while preserving model performance in clinical machine learning.

Nick Souligne, Isabella Mixton-Garcia, Vignesh Subbian

College of Engineering, University of Arizona, Tucson, AZ

Algorithmic Fairness Subgroups Selection Bias

Ask PDF

Apr 16, 2026cs.LG

When Fairness Metrics Disagree: Evaluating the Reliability of Demographic Fairness Assessment in Machine Learning

The evaluation of fairness in machine learning systems has become a central concern in high-stakes applications, including biometric recognition, healthcare decision-making, and automated risk assessment. Existing approaches typically rely on a small number of fairness metrics to assess model behaviour across group partitions, implicitly assuming that these metrics provide consistent and reliable conclusions. However, different fairness metrics capture distinct statistical properties of model performance and may therefore produce conflicting assessments when applied to the same system. In this work, we investigate the consistency of fairness evaluation by conducting a systematic multi-metric analysis of demographic bias in machine learning models. Using face recognition as a controlled experimental setting, we evaluate model performance across multiple group partitions under a range of commonly used fairness metrics, including error-rate disparities and performance-based measures. Our results demonstrate that fairness assessments can vary significantly depending on the choice of metrics, leading to contradictory conclusions regarding model bias. To quantify this phenomenon, we introduce the Fairness Disagreement Index (FDI), a measure designed to capture the degree of inconsistency across fairness metrics. We further show that disagreement remains high across thresholds and model configurations. These findings highlight a critical limitation in current fairness evaluation practices and suggest that single-metric reporting is insufficient for reliable bias assessment.

Khalid Adnan Alsayed

School of Computing, Engineering & Digital Technologies, Teesside University, UK

Algorithmic Fairness Evaluation Metrics Equal Error Rate

Ask PDF