Beyond Demographic Balance: Multi-Metric and Intersectional Evaluation of Fairness in MIMIC-IV Mortality Prediction

Abdullah Al Noman; Fahmid Al Rifat; Tahrima Hashem; Syed Muhammad Ibne Zulfiker; Rishov Paul; Tanzima HAshem

cs.LGOct 1, 2026

Beyond Demographic Balance: Multi-Metric and Intersectional Evaluation of Fairness in MIMIC-IV Mortality Prediction

Authors: Abdullah Al Noman, Fahmid Al Rifat, Tahrima Hashem, Syed Muhammad Ibne Zulfiker, Rishov Paul, Tanzima HAshem

Organizations: Virginia Tech Blacksburg, Virginia, USA · University of Texas at Arlington Arlington, Texas, USA · University of Melbourne Melbourne, Victoria, Australia · University of Toronto Toronto, Ontario, Canada · University of Virginia Charlottesville, Virginia, USA · Bangladesh University of Engineering and Technology Dhaka, Bangladesh

Abstract

Fairness conclusions in clinical prediction can depend strongly on both the metrics reported and the demographic resolution at which performance is evaluated. We revisit these evaluation choices for ICU mortality prediction on MIMIC-IV, comparing predictive-utility and subgroup-error metrics across several fairness interventions. As a complementary case study, we introduce a lightweight adaptation strategy that jointly balances ethnicity--gender--insurance representation without conditioning on mortality outcomes, allowing demographic representation balancing to be examined separately from outcome-conditioned or direct error-rate interventions. We evaluate its behavior at both marginal and corresponding three-way intersectional subgroup levels, while accounting for the statistical support of finer-grained estimates. The results show that interventions can receive substantially different assessments across accuracy/AUROC, sensitivity, and false-positive rate, and that marginal demographic summaries can conceal heterogeneous error profiles within their constituent intersections, including among larger subgroups. These findings highlight the importance of evaluating fairness interventions at both complementary metric and subgroup resolutions, while accounting for the intervention target and the reliability of subgroup estimates.

Figures & tables

Work	Data/task	Intervention target	Main distinction
Mao et al. ( Mao et al., 2023 )	Vision benchmarks	Sensitive group $\times$ label; optional fairness regularization	Last-layer adaptation outside clinical EHRs
Lett et al. ( Lett et al., 2025 )	MIMIC-IV ED admission	Ethnoracial identity $\times$ gender performance objectives	Direct intersectional performance optimization; different endpoint
FairPlay ( Theodorou et al., 2025 )	MIMIC-IV/eICU mortality	Demographic–outcome-conditioned augmentation	Synthetic records with explicit outcome conditioning
Yoon and Kwak ( Yoon and Kwak, 2026 )	MIMIC-IV mortality	Race or gender targeted separately	Single-axis interventions with cross-axis/intersectional evaluation
This study	MIMIC-IV ICU mortality	Ethnicity $\times$ gender $\times$ insurance representation	Joint three-way balancing of observed records without outcome conditioning

Table 1: Closest methodological contrasts in intervention target and demographic resolution.

Panel of Figure 1 — Figure 1 : Study design. IBHFT performs equal-count adaptation over observed ethnicity–gender–insurance intersections and updates only the replacement classifier head. The resulting model is evaluated along two dimensions: predictive-utility versus subgroup-error metrics, and marginal versus intersectional demographic resolution.

Panel of Figure 2 — Figure 2 : Group-wise true-positive rate (top) and false-positive rate (bottom) across repeated evaluations. The series labeled “Ours” corresponds to IBHFT. Reweighting attains very low FPR but substantially lower TPR, whereas IBHFT preserves a sensitivity profile closer to the baseline.

Panel of Figure 3 — Figure 3 : Predictive utility and sensitivity support different assessments of reweighting. High accuracy/AUROC (top) coexists with low sensitivity (bottom).

Figure 4 : Ethnicity-marginal sensitivity (dashed line) and sensitivity estimates for larger nested ethnicity–gender–insurance intersections. Main-text points include intersections with $N\geq 100$ and at least five mortality-positive cases; horizontal bars show 95% Wilson intervals. Dashed lines denote marginal sensitivity from the IBHFT predictions used for the intersectional analysis. Appendix B reports intersections with $N_{+}\geq 5$ .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Ethnicity	Gender	Insurance	$N$	$N_{+}$	Prev.	TPR	FPR	Spec.	Acc.	AUROC
Asian	M	Other	48	6	0.125	0.833	0.381	0.619	0.646	0.877
Black	F	Medicare	113	18	0.159	0.944	0.116	0.884	0.894	0.946
Black	F	Other	106	5	0.047	0.400	0.188	0.812	0.792	0.774
Black	M	Medicare	81	11	0.136	0.727	0.200	0.800	0.790	0.800
Hispanic/Latino	F	Other	39	5	0.128	1.000	0.147	0.853	0.872	0.965
Hispanic/Latino	M	Medicare	27	7	0.259	0.571	0.350	0.650	0.630	0.750

Appendix

Table 2 : Three-way intersectional audit for subgroups with at least five mortality-positive cases (

N_{+}\geq 5

). Point estimates should be interpreted together with subgroup support; Figure 6 reports Wilson intervals for sensitivity.

Figure 5 : TPR–FPR profiles for ethnicity–gender–insurance intersections with at least five mortality-positive cases ( $N_{+}\geq 5$ ). Marker size scales with subgroup size.

Figure 6 : Sensitivity estimates with 95% Wilson intervals for ethnicity–gender–insurance intersections with at least five mortality-positive cases ( $N_{+}\geq 5$ ).

Explore similar work

Improving Fairness of Large Language Model-Based ICU Mortality Prediction via Case-Based Prompting
Dec 17, 2025Gangxiong Zhang, Yongchao Long, Yuxi Zhou +2Intensive Care Unit Mortality Prediction Algorithmic Fairness
School of Computer Science and Engineering, Tianjin University of Technology, Tianjin, China · Institute for Artificial Intelligence, Peking University, Beijing, China · DCST, BNRist, RIIT, Institute of Internet Industry, Tsinghua University, Beijing, China +1
An Integrated Framework for Explainable, Fair, and Observable Hospital Readmission Prediction: Development and Validation on MIMIC-IV
Apr 24, 2026Isaac Tosin AdisaReadmission Prediction Roc-Auc
Department of Statistics, Florida State University, Tallahassee, FL 32306, USA
FairSelect: A Systematic Evaluation of Multi-Level and Intersectional Algorithmic Fairness
Jul 9, 2026Nick Souligne, Isabella Mixton-Garcia, Vignesh SubbianAlgorithmic Fairness Subgroups
College of Engineering, University of Arizona, Tucson, AZ

Dec 17, 2025, cs.LGJ/K move · Enter open · S save

Improving Fairness of Large Language Model-Based ICU Mortality Prediction via Case-Based Prompting

Gangxiong Zhang, Yongchao Long, Yuxi Zhou, Yong Zhang+1

School of Computer Science and Engineering, Tianjin University of Technology, Tianjin, China · Institute for Artificial Intelligence, Peking University, Beijing, China · DCST, BNRist, RIIT, Institute of Internet Industry, Tsinghua University, Beijing, China+1

Intensive Care Unit Mortality Prediction Algorithmic Fairness

Open paper AskPDF arXiv

Accurately predicting mortality risk in intensive care unit (ICU) patients is critical for clinical decision-making. Large language models (LLMs) are increasingly explored for clinical prediction using structured medical data, but their outputs may exhibit demographic disparities. Mitigating such disparities without degrading predictive performance remains challenging. We systematically investigate demographic bias in LLM-based ICU mortality prediction and propose Case-Based Prompting (CAP), a training-free framework designed to improve the empirical balance between predictive performance and subgroup fairness. CAP retrieves clinically similar historical misprediction and demographic-sensitive cases with known outcomes and incorporates them as case-level contextual evidence. We further evaluate model discrimination, subgroup fairness, and feature-dependence consistency. On MIMIC-IV, CAP improved AUROC from 0.806 to 0.873 and AUPRC from 0.497 to 0.694 compared with the Baseline Prompt. CAP also reduced several subgroup disparities, particularly for sex and White-Black comparisons, although age-related disparities remained non-negligible. Controlled demographic-information ablations showed that removing demographic information did not uniformly improve both predictive performance and subgroup fairness, indicating distinct performance-fairness patterns rather than a uniform benefit from demographic suppression. Feature-dependence analysis showed generally consistent expressed clinical factor patterns across demographic subgroups. These findings suggest that CAP is a promising inference-time strategy for LLM-based clinical prediction under the evaluated retrospective setting. Further validation across additional LLMs, external datasets, and prospective clinical scenarios is required before deployment.

Abstract

Explore similar work

Improving Fairness of Large Language Model-Based ICU Mortality Prediction via Case-Based Prompting

An Integrated Framework for Explainable, Fair, and Observable Hospital Readmission Prediction: Development and Validation on MIMIC-IV

FairSelect: A Systematic Evaluation of Multi-Level and Intersectional Algorithmic Fairness