Fairness Metrics based on the AUC in Clinical Predictive AI

João Matos; Ben Van Calster; Richard D. Riley; Paula Dhiman; Gary S. Collins

cs.LGNov 3, 2026

Fairness Metrics based on the AUC in Clinical Predictive AI

Authors: João Matos, Ben Van Calster, Richard D. Riley, Paula Dhiman, Gary S. Collins

Organizations: Centre for Statistics in Medicine, University of Oxford, UK · Department of Development and Regeneration, KU Leuven, Belgium · Department of Applied Health Sciences, University of Birmingham, UK · National Institute for Health and Care Research (NIHR) Biomedical Research Centre: Birmingham, University Hospitals Birmingham NHS Foundation Trust and University of Birmingham, UK

Abstract

Fairness evaluation of clinical prediction models for binary outcomes often involves fairness metrics that compare the area under the receiver operating characteristic curve (AUC, or c-statistic) across subgroups. However, the AUC has been shown to be a non-collapsible metric, meaning that its subgroup specific and overall values are not directly comparable as often implicitly assumed. This raises important questions about how existing AUC-based fairness metrics should be used and interpreted. We examine eight AUC-based fairness metrics proposed in the literature, including: within-group AUC parity; cross-group AUC (xAUC) parity; gap-to-overall AUC; equity-scaled AUC; minimum-based metrics; $\mathrm{xAUC}^1$ ; $\mathrm{xAUC}^0$ ; and $\mathrm{xAUC}^1$ or $\mathrm{xAUC}^0$ parity. We characterise their estimands, relate them to the decomposition of overall AUC (into within- and cross-group components), and critically appraise them. We then illustrate their behaviour using simulations based on the EuroSCORE II, a clinical prediction model for in-hospital mortality. We recommend reporting an AUC matrix, containing the overall AUC and all within- and cross-group AUC components, alongside population characteristics. When fairness metrics are needed, we recommend reporting the within-group AUC parity, with cross-group AUC parity providing supplementary information but requiring caution due to its potential conflict with calibration. Metrics involving $\mathrm{xAUC}^1$ and $\mathrm{xAUC}^0$ have merit, summarising within- and cross-group AUCs into single values, but their interpretation is non-trivial, and they should be used with caution. Gap-to-overall AUC, equity-scaled AUC, and minimum-based metrics compare quantities that should not be compared and it is unclear how to operationalise them, making them not fit for purpose and generally not recommended.

Figures & tables

Panel of Figure 1 — Figure 1. Relationship between different AUC metrics (for binary outcome $y\in\{0,1\}$ and assuming two subgroups $A$ and $B$ ).

Metric	Formula	Meaning	Critical appraisal and recommendation
Within-group AUC parity ( Cheng et al., 2021 ; Byrd et al., 2023 )	$\displaystyle\left\|\mathrm{AUC}_{A}-\mathrm{AUC}_{B}\right\|$	Absolute difference in within-group discrimination	Recommended. Appropriate as the primary fairness metric for discrimination performance. Should be reported alongside overall AUC. Can be used on its own, with cross-group AUC parity providing important additional information for a more comprehensive evaluation of fairness in model discrimination.

Cross-group AUC parity ( Kallus and Zhou, 2019 )	$\displaystyle\left\|\mathrm{xAUC}_{A,B}-\mathrm{xAUC}_{B,A}\right\|$	Absolute difference in cross-group discrimination	Use with caution. Provides important additional information on xAUC disparities, but it may be influenced by differences in outcome prevalence and calibration across subgroups. Should be reported alongside within-group AUC parity and overall AUC. Interpretation in isolation may be misleading.

Gap to overall AUC ( Albuquerque et al., 2024 )	$\displaystyle\left\|\mathrm{AUC}-\mathrm{AUC}_{g}\right\|$	Absolute difference between overall and within-group discrimination	Not recommended. Compares quantities estimating fundamentally different aspects of discrimination. Interpretation is generally convoluted.

Table 1. Summary of AUC-based fairness metrics, their meanings, critical appraisal, and recommendations.

Scenario	$p^{M}$	$\pi^{M}$	$\pi^{F}$	$\mu_{0}^{M}$	$\mu_{1}^{M}$	$\mu_{0}^{F}$	$\mu_{1}^{F}$	$\sigma$
(i)	0.8	0.02	0.03	0.015	0.031	0.025	0.034	0.01
(ii)	0.8	0.02	0.03	0.015	0.031	0.033	0.045	0.01
(iii)	0.8	0.02	0.03	0.016	0.031	0.030	0.045	0.01

Table 2. Simulation parameters.

Scenario	$\mathrm{AUC}_{\mathrm{global}}$	$\mathrm{AUC}_{M}$	$\mathrm{AUC}_{F}$	$\mathrm{xAUC}_{M,F}$	$\mathrm{xAUC}_{F,M}$	$\mathrm{xAUC}^{1}_{M}$	$\mathrm{xAUC}^{1}_{F}$	$\mathrm{AUC}_{\mathrm{parity}}$	$\mathrm{xAUC}_{\mathrm{parity}}$	$\mathrm{xAUC}^{1}_{\mathrm{parity}}$
(i)	0.85	0.88	0.73	0.67	0.92	0.84	0.88	0.15	0.25	0.04
(ii)	0.84	0.88	0.80	0.44	0.99	0.79	0.95	0.08	0.55	0.16
(iii)	0.84	0.86	0.86	0.53	0.98	0.80	0.96	0.00	0.45	0.16

Table 3. AUC metrics and AUC-based fairness metrics, across subgroups.

Panel of Figure 2 — Figure 2. Illustrative cases comparing subgroup and overall AUC relationships. Within each subgroup $g\in{M,F}$ (male, female, respectively) the estimated probabilities for non-events and events are drawn from truncated normal distributions on $[0,1]$ , using 100,000 samples in total. For all three scenarios, $p^{M}=0.8$ , $\pi^{M}=0.02$ , $\pi^{F}=0.03$ , and the estimated probabilities follow $\sigma=0.01$ for all strata. $\mu_{0}^{M}$ , $\mu_{1}^{M}$ , $\mu_{0}^{F}$ , and $\mu_{1}^{F}$ are specific to each scenario. Labels of 0 denote patients without an event, labels of 1 denote patients with an event.

Algorithmic Fairness

Algorithmic fairness studies how automated decisions distribute errors, opportunities, and harms across groups, and how to measure and reduce disparities.

See all 434 papers

Collapsibility of Performance Metrics in Clinical Predictive AI
Aug 31, 2026João Matos, Ben Van Calster, Richard D. Riley +2Algorithmic Fairness Clinical Prediction
Centre for Statistics in Medicine, University of Oxford · KU Leuven · University of Birmingham +1
Beyond Demographic Balance: Multi-Metric and Intersectional Evaluation of Fairness in MIMIC-IV Mortality Prediction
Oct 1, 2026Abdullah Al Noman, Fahmid Al Rifat, Tahrima Hashem +3Algorithmic Fairness Intersectional Fairness
Virginia Tech Blacksburg, Virginia, USA · University of Texas at Arlington Arlington, Texas, USA · University of Melbourne Melbourne, Victoria, Australia +3
Local Discrimination of Binary Classifiers via Localized AUC Functional
Oct 22, 2026Wenye Ren, Anna Győrffy-Kerekes, Tancredi Dorsi +1Binary Classification Clinical Risk Prediction
Department of Computer Science, ETH Zurich, Switzerland · Department of Computer Science, Tsinghua University, China · Center for Learning Systems, Max Planck Institute, Tübingen, Germany +2

Aug 31, 2026J/K move · Enter open · S save

Collapsibility of Performance Metrics in Clinical Predictive AI

João Matos, Ben Van Calster, Richard D. Riley, Paula Dhiman+1

Centre for Statistics in Medicine, University of Oxford · KU Leuven · University of Birmingham+1

Algorithmic Fairness Clinical Prediction

Open paper AskPDF arXiv

Background: Population level assessments of predictive artificial intelligence (AI) can conceal performance disparities across subgroups. Fairness evaluations commonly rely on performance analyses across subgroups. However, some performance metrics are non-collapsible, meaning that the overall population performance value does not equal the weighted average of subgroup specific values. Objective: To examine the collapsibility properties of commonly reported performance metrics in predictive AI, with a focus on the area under the receiver operating characteristic curve (AUC, also known as c-statistic). Methods: We investigate the collapsibility of 15 performance metrics, either by expressing each metric as a linear combination of its stratum specific values or, where non-collapsible, by providing a counterexample inspired by Simpson's paradox as a formal disproof. Results: Five performance metrics (AUC, calibration intercept, calibration slope, expected calibration error, and Nagelkerke R^2) are shown to be non-collapsible, and ten (O:E ratio, logloss, Brier score, accuracy, F1-score, true positive rate, true negative rate, positive predictive value, negative predictive value, and net benefit) are shown to be collapsible. The AUC is shown to be non-collapsible because it decomposes into within- and cross-group AUC terms when subpopulations coexist, such that its overall value may fall outside the range of subgroup specific AUCs. Conclusions: Non-collapsibility of performance metrics has important consequences for reporting, model appraisal, and fairness evaluation. It can generate spurious differences between subgroup and overall performance, which may mislead fairness evaluations. Explicitly acknowledging and reporting the collapsibility properties of performance metrics improves both the interpretability and transparency of fairness assessments.

Abstract

Algorithmic Fairness

Explore similar work

Collapsibility of Performance Metrics in Clinical Predictive AI

Beyond Demographic Balance: Multi-Metric and Intersectional Evaluation of Fairness in MIMIC-IV Mortality Prediction

Local Discrimination of Binary Classifiers via Localized AUC Functional