cs.LGApr 6, 2026

FairLogue: A Toolkit for Intersectional Fairness Analysis in Clinical Machine Learning Models

Authors: Nick Souligne, Vignesh Subbian

Organizations: Department of Biomedical Engineering, University of Arizona, 1127 E. James E. Rogers Way, Tucson, AZ, 85721-0020, United States

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Objective: Algorithmic fairness is essential for equitable and trustworthy machine learning in healthcare. Most fairness tools emphasize single-axis demographic comparisons and may miss compounded disparities affecting intersectional populations. This study introduces Fairlogue, a toolkit designed to operationalize intersectional fairness assessment in observational and counterfactual contexts within clinical settings. Methods: Fairlogue is a Python-based toolkit composed of three components: 1) an observational framework extending demographic parity, equalized odds, and equal opportunity difference to intersectional populations; 2) a counterfactual framework evaluating fairness under treatment-based contexts; and 3) a generalized counterfactual framework assessing fairness under interventions on intersectional group membership. The toolkit was evaluated using electronic health record data from the All of Us Controlled Tier V8 dataset in a glaucoma surgery prediction task using logistic regression with race and gender as protected attributes. Results: Observational analysis identified substantial intersectional disparities despite moderate model performance (AUROC = 0.709; accuracy = 0.651). Intersectional evaluation revealed larger fairness gaps than single-axis analyses, including demographic parity differences of 0.20 and equalized odds true positive and false positive rate gaps of 0.33 and 0.15, respectively. Counterfactual analysis using permutation-based null distributions produced unfairness ("u-value") estimates near zero, suggesting observed disparities were consistent with chance after conditioning on covariates. Conclusion: Fairlogue provides a modular toolkit integrating observational and counterfactual methods for quantifying and evaluating intersectional bias in clinical machine learning workflows.

Figures & tables

Problem or IssueTraditional algorithmic fairness evaluations in clinical machine learning primarily rely on single-axis demographic comparisons, which may overlook compounded disparities affecting intersectional subgroups and limit equitable deployment of predictive models.
What is Already KnownExisting fairness toolkits and auditing methods can detect bias across individual protected attributes, and counterfactual frameworks have been proposed; however, practical implementations that integrate intersectional analysis with real-world clinical workflows remain limited.
What This Paper AddsThis paper introduces FairLogue , a modular toolkit that combines observational and counterfactual intersectional fairness assessments within a unified framework. Applied to All of Us clinical data, the toolkit demonstrates that intersectional analyses can reveal disparities not captured by single-axis evaluations and provides practical workflows for diagnosing whether disparities reflect model unfairness or underlying data structure.
Who would benefit from the new knowledge in this paperClinical informaticians, machine learning researchers, healthcare data scientists, and developers deploying predictive models who require practical methods to evaluate and improve fairness across diverse patient populations.
Table 1
Figure 1: Component 1 workflow for observational intersectional fairness assessment . Input data is used to train a predictive model and compute single-axis and intersectional fairness metrics (demographic parity, equalized odds, equal opportunity difference), producing group-level disparities summaries without causal assumptions.
Figure 2: Components 2 and 3 workflows for counterfactual intersectional fairness analysis. Observed dataset evaluated using permutation-based null distributions to compute counterfactual error metrics and unfairness (u-values), enabling assessment of whether disparities persist beyond chance.
Group (n)TPRFPR
Black (300)0.690.40
White (485)0.610.30
Female (433)0.720.33
Male (402)0.560.34
Black | Female (170)0.660.39
Black | Male (130)0.720.43
Table 1: True positive rates and false positive rates for the non-intersectional and intersectional Race/Gender demographics
MetricRace OnlyGender OnlyIntersectional
Demographic parity gap0.10380.06920.200
Equalized odds FPR gap0.10310.01330.1462
Equal opportunity gap0.07960.16610.3316
Table 2: Model Performance and Fairness Metric Gaps for non-intersectional and intersectional Race/Gender demographics
Figure 3: Plots from Component 3 showing the aggregate disparity metrics with the u-values overlaid in red. Threshold for acceptable unfairness set to 0.1.
More in this topic

Algorithmic Fairness

Algorithmic fairness studies how automated decisions distribute errors, opportunities, and harms across groups, and how to measure and reduce disparities.

See all 419 papers

Explore similar work

CardsList
Apr 7, 2026

Evaluating Intersectional Fairness across Clinical Machine Learning Use Cases using Fairlogue and the All of Us Research Program

Intersectional biases in healthcare data can produce compound disparities in clinical machine learning models, yet most fairness evaluations assess demographic attributes independently. FairLogue, a toolkit for intersectional fairness auditing, was applied across multiple clinical prediction tasks to evaluate disparities across combined demographic groups. Using the All of Us dataset, two published models were selected for replication and evaluation: (A) prediction of selective serotonin reuptake inhibitor associated bleeding events and (B) two-year stroke risk in patients with atrial fibrillation. Observational fairness metrics were computed across race, gender, and intersectional subgroups, followed by counterfactual analysis to evaluate whether disparities were attributable to group membership. Intersectional evaluation revealed larger disparities than single-axis analyses; however, counterfactual diagnostics indicated that most observed disparities were comparable to those expected under randomized group membership. These results highlight the importance of intersectional fairness auditing and demonstrate how FairLogue provides deeper insight into bias in clinical machine learning systems.

Nick Souligne, Vignesh Subbian

College of Engineering, The University of Arizona, Tucson, AZ, United States

Algorithmic FairnessIntersectional Fairness
AskPDF
Jul 9, 2026

FairSelect: A Systematic Evaluation of Multi-Level and Intersectional Algorithmic Fairness

Algorithmic fairness methods are increasingly used to identify and mitigate bias in machine learning models, yet most approaches are evaluated in isolation and along single demographic axes. This limits practical guidance for selecting fairness strategies, where disparities may arise across intersectional subgroups and across multiple stages of the modeling lifecycle. This work presents FairSelect, a toolkit for systematically evaluating fairness mitigation strategies applied individually and in combination across preprocessing, inprocessing, and postprocessing stages. FairSelect supports multiple model architectures, intersectional subgroup evaluation, and comparison of fairness utility tradeoffs across baseline, single method, and multi level configurations. The framework was validated using synthetic clinical datasets designed to represent specific bias mechanisms and a real-world replication of two-year stroke risk prediction among patients with atrial fibrillation. Synthetic experiments showed that targeted fairness methods generally reduced intended subgroup disparities, while combined strategies produced larger average fairness improvements with modest utility tradeoffs. In the clinical prediction task, mitigation effects were highly variable, with some combinations improving both fairness and predictive performance while others were ineffective or counterproductive. These findings demonstrate that fairness interventions interact in nonadditive and context dependent ways. FairSelect provides a practical framework for systematically identifying fairness strategies that improve subgroup equity while preserving model performance in clinical machine learning.

Nick Souligne, Isabella Mixton-Garcia, Vignesh Subbian

College of Engineering, University of Arizona, Tucson, AZ

Algorithmic FairnessGroup FairnessIntersectional Fairness
AskPDF
Sep 30, 2026

A Rank Graduation metric for Algorithmic fairness

Fairness assessment in algorithmic decisions that affect individuals, such as credit scoring, often relies on parity measures calculated at the aggregate group level. Such measures may not reveal which individuals experience unfairness or which explanatory factors contribute to it. In this paper, we propose a rank-based framework that evaluates fairness through the distribution of model prediction errors, thereby linking fairness assessment with predictive accuracy and explainability. The framework combines Rank Graduation Fairness (RGF), its integrated measure AURGF, a centered Cramer--von Mises permutation test, and a feature removal procedure for fairness explainability. We evaluate the methodology using logistic regression, random forest, gradient boosting, and a multilayer perceptron. The simulation study shows that protected-group imbalance can reverse descriptive fairness comparisons, whereas the proposed inferential procedure correctly distinguishes fair from unfair mechanisms. Its application to HMDA mortgage data produces model rankings that differ from those obtained with classical fairness criteria. Tree-based models, rather than logistic regression, provide the strongest combination of predictive accuracy and rank-based fairness, while the fairness null hypothesis is rejected for all four models. The persistence of disparity across statistical, bagging, boosting, and neural network specifications, together with the feature removal results, indicates that the observed unfairness is not specific to a single algorithm or predictor, but is associated with group differences embedded in the characteristics of the lending data. These findings support a broader approach to trustworthy artificial intelligence that combines predictive accuracy, fairness measurement, statistical inference, and explainability.

Dalia Atif, Paolo Giudici

Algorithmic AuditingAlgorithmic Fairness
AskPDF