Detection and Validation of Hidden Subgroup Disparities in Clinical AI
Authors: Alceu Bissoto, Trung-Dung Hoang, Tim Flühmann, Nina Weng, Susu Sun, Christian F. Baumgartner, Lisa M. Koch
Organizations: Department of Digital Medicine, University of Bern, Switzerland · Department of Diabetes, Endocrinology, Nutritional Medicine and Metabolism, Bern University Hospital, Switzerland · Diabetes Center Berne, Switzerland · Technical University of Denmark, Denmark · Cluster of Excellence: Machine Learning - New Perspectives for Science, University of Tübingen, Germany · Faculty of Health Sciences and Medicine, University of Lucerne, Switzerland
Machine learning (ML) models used in clinical workflows often perform systematically differently across patient groups, raising concerns about equity and patient safety. Traditional subgroup analyses can expose such disparities only when rich metadata (e.g., patient demographics) is available and captures the sources of performance variation, conditions rarely met in practice. Algorithmic subgroup discovery methods offer a solution by identifying coherent groups from learned feature representations, but because true underlying stratifications are unknown, these methods are hard to evaluate. As a result, subgroup discovery has not been systematically applied or assessed for performance monitoring. Our main objective is to evaluate algorithmic subgroup discovery for performance monitoring, assessing the discovered subgroups' performance gaps and semantic coherence. Investigating cardiomegaly classification in a dataset of 223,228 chest X-rays, we showed that the discovered subgroups exhibited larger performance gaps (accuracies of 0.61-0.90) than traditional metadata-based analysis (0.80-0.94) and were frequently interpretable using patient, acquisition, and model metadata. As a proof of concept, we also showed that annotations from learned subgroups can be used to mitigate the biases they expose. We conclude that subgroup discovery is a practical and effective tool to enhance the performance validation and monitoring of ML systems in healthcare.
Figures & tables
Figure 1: From traditional subgroup analysis to algorithmic subgroup discovery for performance monitoring. (a) Traditional subgroup analysis detects disparate patient outcomes, but it is limited to annotated metadata. (b) Subgroup discovery reveals hidden stratifications but lacks performance validation. (c) We bridge this gap by applying subgroup discovery for performance analysis in both (d) controlled synthetic settings and (e) real-world scenarios with unknown subgroups.
Metadata Subgroups
Sample Size
Accuracy
Sensitivity
Specificity
AUROC
Full dataset
22,281
0.872±0.005
0.471±0.018
0.936±0.004
0.856±0.007
Sex: Male
13,134
0.875±0.006
0.479±0.023
0.938±0.004
0.859±0.009
Sex: Female
9,127
0.866±0.007
0.459±0.028
0.932±0.006
0.850±0.011
Age: 0–60
10,032
0.905±0.006
0.510±0.030
0.952±0.005
0.880±0.011
Age: 60+
12,229
0.844±0.007
0.449±0.022
0.921±0.005
0.832±0.010
Race: White
12,635
0.870±0.006
0.430±0.024
0.936±0.005
0.845±0.010
Table 1: Traditional subgroup analysis. Subgroup performance metrics reported as value ± half 95% CI. PA = posterior-to-anterior, AP = anterior-to-posterior.
Figure 2: Performances of discovered subgroups (blue) and traditional metadata subgroups (red). Filled points indicate statistically significant performance disparities from its complement. For each group, we show their size (A), cardiomegaly class ratio (B), and their performance in prevalence-dependent (C) and independent (D-F) metrics.
Figure 3: Characteristics of discovered subgroups. (A) Discovered subgroup’s metadata profile. Each subgroup (column) is characterized by a set of metadata (row), and its effect (cells) is measured by Cohen’s h for categorical values and Cohen’s d for continuous ones. (B) This characterization enables us to understand the subgroups, identify key characteristics important to the classification models, and create performance gaps. For a few selected subgroups, we show four randomly selected images from this group, and their most distinctive characteristic.
SG
Category
Description
Accuracy
Sensibility
Specificity
AUROC
A5
Positive preds
0.612±0.030
0.984±0.010
0.017±0.014
0.668±0.034
A0
High BMI, males
0.886±0.028
0.103±0.097
0.952±0.019
0.766±0.063
A12
Males, unknown
0.909±0.017
0.371±0.094
0.963±0.012
0.849±0.038
A9
Females, positive preds, support devices
0.771±0.028
0.586±0.066
0.829±0.029
0.809±0.031
A8
Support devices, interpreter needed
0.890±0.018
0.369±0.074
0.964±0.011
0.844±0.029
A6
–
Unknown
0.876±0.010
0.224±0.037
0.962±0.006
0.823±0.018
Table 2: Performance by discovered subgroups. Subgroups are categorized into patient ( ), acquisition ( ), and model ( ) factors. Values are mean ± 95% confidence-interval half-width. PA = posterior-to-anterior, AP = anterior-to-posterior.
Figure 4: Sanity checks of subgroup discovery. (A) Effects of known simulated artifacts. Subgroup discovery successfully dedicates subgroups for each possible combination of the presence of the two artifacts. (B) Random samples from the found subgroups. (C) Detailed subgroup accuracies, reproduced from [ 4 ] . Darker dots represent found subgroups with higher effects. These capture the true performance gap characterized by hidden subgroups, which are overlooked by traditional subgroup analysis based on a single artifact (known subgroups), and by overall performance.
SG
Category
Description
Accuracy
Sensibility
Specificity
AUROC
B4
Lateral
0.804±0.033
0.432±0.091
0.896±0.029
0.810±0.044
B0
Young, healthy, lateral
0.919±0.014
0.317±0.075
0.977±0.008
0.839±0.035
B1
Lateral, no support devices
0.839±0.020
0.528±0.064
0.904±0.018
0.814±0.030
B6
PA, healthy, negative preds
0.944±0.013
0.258±0.104
0.979±0.008
0.855±0.043
B2
Positive preds
0.608±0.026
0.961±0.014
0.104±0.026
0.668±0.030
B9
Pacemaker, PA, positive preds
0.730±0.042
0.775±0.060
0.698±0.056
0.815±0.038
Table 3: Discovered subgroups using BioMedCLIP at threshold 0.5. Subgroups are categorized into patient ( ), acquisition ( ), and model ( ) factors. Metrics are reported as mean ± half 95% CI. PA = posterior-to-anterior.
In biomedical Subgroup Discovery, practitioners are interested in discovering interpretable and homogeneous subgroups within a group of patients. In this paper, assuming that healthy subjects (i.e., controls) share common but irrelevant factors of variation with the patients, we motivate and develop a Contrastive Subgroup Discovery method, entitled Deep UCSL. By contrasting patients with controls, Deep UCSL identifies subgroups driven solely by pathological factors, ignoring common variability shared with healthy subjects. Our framework employs a deep feature extractor to learn a discriminative representation space. Mathematically, we derive a novel loss based on the conditional joint likelihood of latent clusters and patient/control labels, optimized via an Expectation-Maximization strategy alternating between subgroup inference and feature encoder updates. A regularization term further encourages representations to capture disease-specific variability while ignoring variability shared with controls. Compared to previous related works, our approach quantitatively improves the quality of the estimated subgroups, as demonstrated on a MNIST example and four distinct real medical imaging datasets. Code and datasets are available at: https://github.com/rlouiset/deep_ucsl.
Robin Louiset, Edouard Duchesnay, Benoit Dufumier +2
1NeuroSpin, Universit´e Paris-Saclay, CEA, Gif-sur-Yvette, 91191, France. · 2LTCI, Institut Polytechnique de Paris, T´el´ecom Paris, Palaiseau,2026 91120, France.
Medical imaging models are often deployed without the demographic, acquisition, and quality metadata needed for subgroup auditing. Once those metadata disappear, clinically critical failure modes can be masked by strong aggregate performance, and many robust-learning methods lose the group structure they rely on. We present CAPRA, a calibrated proxy-axis framework for hidden subgroup analysis under missing metadata. CAPRA predicts image-derived semantic axes, calibrates axis posteriors on a small metadata-labeled split via patient-level cross-fitting, and organizes those posteriors into a calibrated subgroup interface that supports both deployment-time failure analysis and downstream robust learning without requiring subgroup labels at deployment. Across fundus, dermoscopy, and chest radiography, CAPRA reveals disparity patterns missed by metadata-only slicing, remains informative under dataset shift, and produces subgroup partitions that align more closely with explicit failure axes than image-only or latent-slice baselines. The same interface can also be reused by downstream robust learners, although those gains are domain-dependent. Overall, CAPRA turns hidden subgroup analysis under missing metadata into a calibrated, interpretable, and reusable subgroup interface for deployment-time analysis and robust transfer.
Yawen Li, Yan Li, Zhe Xue +3
Beijing University of Posts and Telecommunications Beijing, China
In high-stakes settings where machine learning models are used to automate decision-making about individuals, the presence of algorithmic bias can exacerbate systemic harm to certain subgroups of people. These biases often stem from the underlying training data. In practice, interventions to "fix the data" depend on the actual additional data sources available -- where many are less than ideal. In these cases, the effects of data scaling on subgroup performance become volatile, as the improvements from increased sample size are counteracted by the introduction of distribution shifts in the training set. In this paper, we investigate the limitations of combining data sources to improve subgroup performance within the context of healthcare. Clinical models are commonly trained on datasets comprised of patient electronic health record (EHR) data from different hospitals or admission departments. Across two such datasets, the eICU Collaborative Research Database and the MIMIC-IV dataset, we find that data addition can both help and hurt model fairness and performance, and many intuitive strategies for data selection are unreliable. We compare model-based post-hoc calibration and data-centric addition strategies to find that the combination of both is important to improve subgroup performance. Our work questions the traditional dogma of "better data" for overcoming fairness challenges by comparing and combining data- and model-based approaches.
Erin Tan, Judy Hanwen Shen, Irene Y. Chen
University of California, Berkeley · Stanford University