Leakage- and Label-Aware Benchmarking of Class-Imbalance Strategies for Multimodal Chest X-Ray and EHR Classification
Organizations: Faculty of Computer Science and Engineering, Ho Chi Minh City University of Technology (HCMUT), VNU-HCM, Vietnam · Applied AI & Data Science Division, AK Technologies Company Limited, Ho Chi Minh City, Vietnam · Vietnam National University Ho Chi Minh City, Vietnam
Abstract
Benchmarks of class-imbalance strategies for multimodal chest X-ray (CXR) and electronic health record (EHR) prediction are often evaluated on record-level splits of cohorts in which one patient contributes several radiographs. We audit such a benchmark on 29,083 MIMIC-CXR radiographs of 10,479 patients with eight diagnosis targets. First, the record-split benchmark was misleading: a lookup table that copies the label of a development-set radiograph with the same EHR vector nearly matched the best model, and on unseen patients every model fell to the level of predicting every case positive. Second, tracing the labels back to MIMIC-IV explained why: each patient's labels are keyword matches on the diagnosis codes of a single admission, copied, with one EHR vector per patient, to all of the patient's radiographs, so about a third of the radiographs carry labels that differ from the codes of their own admission. Third, under patient-separated evaluation with admission-aligned and rule-refined labels, the initial conclusions do not hold: in the evaluated configurations no imbalance strategy raised the observed mean macro F2 over no augmentation by more than 0.01, tuned tree ensembles showed no consistent advantage over tuned logistic regression, and the released EHR variables stayed at the level of the all-positive baseline, whereas the diagnostic history available at imaging time and image embeddings exceeded it, including, with rule-refined labels, a CheXpert-trained encoder that never saw these radiographs; RAD-DINO had seen 96.6% of them during self-supervised pretraining. Adding either encoder to the diagnostic history gave no statistically significant F2 gain; exploratory AUROC gains were significant with RAD-DINO but not with DenseNet. We provide reusable diagnostics, a leakage-aware protocol and a checklist.
Figures & tables
| Released labels (R) | Positives (%) | Cons. | A: prev. % | C: prev. % | ||||
|---|---|---|---|---|---|---|---|---|
| Target | Radiogr. | Patients | Prev. % | own-neg. | later | (%) | (patients) | (patients) |
| Pneumonia | 7,980 | 4,318 | 27.44 | 33.4 | 18.4 | 89.5 | 30.84 (7,014) | 28.03 (6,434) |
| Lung cancer | 7,787 | 2,822 | 26.78 | 13.1 | 8.4 | 14.4 | 26.83 (3,372) | 3.54 (518) |
| Fracture | 2,078 | 918 | 7.15 | 40.2 | 24.0 | 75.6 | 7.27 (1,563) | 5.12 (1,222) |
| Bronchitis | 1,485 | 678 | 5.11 | 40.5 | 15.4 | 100.0 | 5.74 (1,231) | 5.74 (1,231) |
| Arthritis | 1,191 | 428 | 4.10 | 33.2 | 26.4 | 99.5 | 4.11 (700) | 4.07 (691) |
| Protocol | Folds (rows) | Labels | Pat. | Label | EHR |
|---|---|---|---|---|---|
| P1 | record | R | – | – | – |
| P2 | patient | R | ✓ | – | – |
| P3 | patient (1 row/patient) | R | ✓ | – | – |
| P1-A | record | A | – | ✓ | – |
| P4, P4-C | patient | A, C | ✓ | ✓ | – |
| P5 | first-image year | R, A | ✓ | A only | – |
| Representation | F2 | F2 95% CI | [95% CI] | PR-AUC | AUROC | Brier |
|---|---|---|---|---|---|---|
| P1 record split | ||||||
| EHR only | 0.791 0.012 | [0.766, 0.815] | +0.509 ∗ [+0.484, +0.533] | 0.842 0.010 | 0.960 0.005 | 0.025 0.001 |
| Image: 11 scores | 0.283 0.003 | [0.278, 0.289] | +0.001 [ 0.004, +0.007] | 0.118 0.004 | 0.569 0.007 | 0.163 0.031 |
| Fusion: 11 scores | 0.560 0.002 | [0.556, 0.564] | +0.278 ∗ [+0.274, +0.282] | 0.540 0.016 | 0.834 0.007 | 0.062 0.002 |
| All-positive | 0.282 | 0.500 | ||||
| EHR lookup (hit 78.4%) | 0.788 0.008 | +0.506 | ||||
| Representation | F2 | F2 95% CI | [95% CI] | PR-AUC | AUROC |
|---|---|---|---|---|---|
| P1-A record split, admission-aligned labels | |||||
| EHR only | 0.421 0.013 | [0.395, 0.448] | +0.131 ∗ [+0.112, +0.149] | 0.287 0.015 | 0.719 0.012 |
| Image: 11 scores | 0.290 0.003 | [0.285, 0.296] | 0.000 [ 0.010, +0.009] | 0.132 0.003 | 0.567 0.010 |
| All-positive / lookup (hit 78.4%) | 0.291 / 0.355 | ||||
| P4 patient-grouped, admission-aligned labels | |||||
| EHR only | 0.294 0.007 | [0.281, 0.308] | +0.004 [ 0.008, +0.015] | 0.140 0.005 | 0.579 0.008 |
| Contrast | Labels | F2 [95% CI] | ||
|---|---|---|---|---|
| DenseNet image all-positive | A | +0.007 [ 0.000, +0.013] | 0.052 | 0.193 |
| C | +0.022 [+0.012, +0.032] | 0.004 | 0.025 | |
| DenseNet + EHR all-positive | A | +0.013 [+0.000, +0.027] | 0.048 | 0.193 |
| C | +0.026 [+0.020, +0.031] | 0.001 | 0.001 | |
| RAD-DINO image DenseNet image | A | +0.019 [+0.002, +0.035] | 0.034 | 0.180 |
| C | +0.032 [+0.005, +0.058] | 0.030 | 0.180 |
| Check | Setting, result and conclusion | Fits |
|---|---|---|
| Tuned models | Depth 3 or 5, minimum leaf 20 or 50, 100–1,000 trees or rounds; LR ; no augmentation. P1: depth 3 removes the tree advantage (EHR F2: RF 0.788 0.366, ET 0.793 0.315; LR 0.313), yet validation picks unlimited depth in 75 of 80 choices. Tuned trees vs. tuned LR: to ; 1,000 boosting rounds overfit (down to ). No consistent tree advantage. | 10,560 |
| Joint tuning | P4-C; for each strategy, LR ( ) or LightGBM (four settings) chosen on inner validation. vs. no augmentation: EHR to , fusion to (all ; CIs up to ); tuned LightGBM vs. tuned LR to . No significant or consistent strategy gain within this grid. | 2,240 |
| Aligned subset | The 10,479 radiographs whose own admission supplied the labels (and presumably the EHR values); laboratory and vital-sign values may postdate the radiograph, so this evaluates admission alignment, not prediction-time validity. EHR over all-positive; RAD-DINO fusion vs. EHR ( ); C labels: RAD-DINO vs. EHR ( ). EHR stays weak. | 4,080 |
| Frontal only | AP/PA radiographs only (20,294). F2 vs. all views to ; EHR at the all-positive level; RAD-DINO vs. EHR (R, ), (A, ), (C, ). No conclusion changes. | 5,760 |
| PCA size | 32, 128 or 768 instead of 64 RAD-DINO components (37% of the variance). 128: within 0.008; 32 or 768: down to . 64 components adequate for RAD-DINO with LR. | 720 |
| Age, history | Own-admission age; prior-diagnosis flags alone. Age: within 0.001 of the released age, at the all-positive level; flags: (A), (C) over the released EHR. The history gain comes from the diagnoses. | 240 |
Medical Image Benchmarks
Medical image benchmarks compare AI methods on curated clinical images, tasks, and evaluation protocols to measure accuracy, robustness, and clinical usefulness.
See all 369 papers