Organizations: Faculty of Computer Science and Engineering, Ho Chi Minh City University of Technology (HCMUT), VNU-HCM, Vietnam · Applied AI & Data Science Division, AK Technologies Company Limited, Ho Chi Minh City, Vietnam · Vietnam National University Ho Chi Minh City, Vietnam
Benchmarks of class-imbalance strategies for multimodal chest X-ray (CXR) and electronic health record (EHR) prediction are often evaluated on record-level splits of cohorts in which one patient contributes several radiographs. We audit such a benchmark on 29,083 MIMIC-CXR radiographs of 10,479 patients with eight diagnosis targets. First, the record-split benchmark was misleading: a lookup table that copies the label of a development-set radiograph with the same EHR vector nearly matched the best model, and on unseen patients every model fell to the level of predicting every case positive. Second, tracing the labels back to MIMIC-IV explained why: each patient's labels are keyword matches on the diagnosis codes of a single admission, copied, with one EHR vector per patient, to all of the patient's radiographs, so about a third of the radiographs carry labels that differ from the codes of their own admission. Third, under patient-separated evaluation with admission-aligned and rule-refined labels, the initial conclusions do not hold: in the evaluated configurations no imbalance strategy raised the observed mean macro F2 over no augmentation by more than 0.01, tuned tree ensembles showed no consistent advantage over tuned logistic regression, and the released EHR variables stayed at the level of the all-positive baseline, whereas the diagnostic history available at imaging time and image embeddings exceeded it, including, with rule-refined labels, a CheXpert-trained encoder that never saw these radiographs; RAD-DINO had seen 96.6% of them during self-supervised pretraining. Adding either encoder to the diagnostic history gave no statistically significant F2 gain; exploratory AUROC gains were significant with RAD-DINO but not with DenseNet. We provide reusable diagnostics, a leakage-aware protocol and a checklist.
Figures & tables
Released labels (R)
Positives (%)
Cons.
A: prev. %
C: prev. %
Target
Radiogr.
Patients
Prev. %
own-neg.
later
(%)
(patients)
(patients)
Pneumonia
7,980
4,318
27.44
33.4
18.4
89.5
30.84 (7,014)
28.03 (6,434)
Lung cancer
7,787
2,822
26.78
13.1
8.4
14.4
26.83 (3,372)
3.54 (518)
Fracture
2,078
918
7.15
40.2
24.0
75.6
7.27 (1,563)
5.12 (1,222)
Bronchitis
1,485
678
5.11
40.5
15.4
100.0
5.74 (1,231)
5.74 (1,231)
Arthritis
1,191
428
4.10
33.2
26.4
99.5
4.11 (700)
4.07 (691)
TABLE I: Released labels ( N = 29,083 radiographs, 10,479 patients) and label audit. Own-neg.: released positives that are negative in the radiograph’s own admission; later: the subset whose label came from a later admission; cons.: positive patients whose source admission has a code consistent with the rule-refined (C) definition; A, C: prevalence (% of radiographs) and positive patients of the admission-aligned and rule-refined labels (Section III-C ).
Protocol
Folds (rows)
Labels
Pat.
Label
EHR
P1
record
R
–
–
–
P2
patient
R
✓
–
–
P3
patient (1 row/patient)
R
✓
–
–
P1-A
record
A
–
✓
–
P4, P4-C
patient
A, C
✓
✓
–
P5
first-image year
R, A
✓
A only
–
TABLE II: Evaluation protocols. Pat.: patients separated across training, validation and test. Label, EHR: from the radiograph’s own admission (✓) or one value per patient (–); ∘ : presumably own admission (Section III-C ), possibly measured after the radiograph. P4-H and P4-C-H replace the released EHR values by age and prior diagnoses known at imaging time.
Representation
F2
F2 95% CI
Δ [95% CI]
PR-AUC
AUROC
Brier
P1 record split
EHR only
0.791 ± 0.012
[0.766, 0.815]
+0.509 ∗ [+0.484, +0.533]
0.842 ± 0.010
0.960 ± 0.005
0.025 ± 0.001
Image: 11 scores
0.283 ± 0.003
[0.278, 0.289]
+0.001 [ − 0.004, +0.007]
0.118 ± 0.004
0.569 ± 0.007
0.163 ± 0.031
Fusion: 11 scores
0.560 ± 0.002
[0.556, 0.564]
+0.278 ∗ [+0.274, +0.282]
0.540 ± 0.016
0.834 ± 0.007
0.062 ± 0.002
All-positive
0.282
0.500
EHR lookup (hit 78.4%)
0.788 ± 0.008
+0.506
TABLE III: Outer-test performance, averaged over the 8 targets within each fold (macro) and then over the 5 folds: mean ± SD; configuration selected on inner validation. Δ : F2 minus all-positive F2. 95% CIs and ∗ ( p<0.05/17 , Bonferroni over the 17 rows) from the corrected resampled t -test over the 5 fold-level values; CIs are not adjusted for multiplicity. Brier: lower is better; predicting the training prevalence scores 0.077 (P1, P2) and 0.085 (P3). Italic: reference baselines without learning.
Fig. 1: Paired change in test F2 vs. no augmentation (same classifier, target and fold) for EHR, image (11 scores, DenseNet, RAD-DINO) and fusion inputs. Points: mean of five fold-level differences averaged over targets, classifiers and representations; bars: 95% CI (corrected resampled t -test).
Representation
F2
F2 95% CI
Δ [95% CI]
PR-AUC
AUROC
P1-A record split, admission-aligned labels
EHR only
0.421 ± 0.013
[0.395, 0.448]
+0.131 ∗ [+0.112, +0.149]
0.287 ± 0.015
0.719 ± 0.012
Image: 11 scores
0.290 ± 0.003
[0.285, 0.296]
− 0.000 [ − 0.010, +0.009]
0.132 ± 0.003
0.567 ± 0.010
All-positive / lookup (hit 78.4%)
0.291 / 0.355
P4 patient-grouped, admission-aligned labels
EHR only
0.294 ± 0.007
[0.281, 0.308]
+0.004 [ − 0.008, +0.015]
0.140 ± 0.005
0.579 ± 0.008
TABLE IV: Admission-aligned (A) and rule-refined (C) labels: macro outer-test performance (8 targets). P1-A, P4, P4-C: mean ± SD over 5 folds; 95% CIs and ∗ ( p<0.05/14 , Bonferroni over the 14 unmarked model rows) from the corrected resampled t -test over the fold-level values; ‡ CheXpert-only DenseNet control, tested in its own Holm-corrected family (Table V ), not with ∗ ; image and EHR-fusion rows use the original features, history-fusion rows the re-extracted ones (Section V-F ). P5: single calendar-time split, 95% CIs from 1,000 patient-level bootstrap resamples (no SD); † bootstrap CI of Δ excludes 0. Italic: baselines without learning.
Contrast
Labels
Δ F2 [95% CI]
p
pH
DenseNet image − all-positive
A
+0.007 [ − 0.000, +0.013]
0.052
0.193
C
+0.022 [+0.012, +0.032]
0.004
0.025
DenseNet + EHR − all-positive
A
+0.013 [+0.000, +0.027]
0.048
0.193
C
+0.026 [+0.020, +0.031]
< 0.001
0.001
RAD-DINO image − DenseNet image
A
+0.019 [+0.002, +0.035]
0.034
0.180
C
+0.032 [+0.005, +0.058]
0.030
0.180
TABLE V: Encoder control: the eight planned F2 comparisons (macro outer-test F2 over 5 folds, corrected resampled t -test); pH : Holm-adjusted within this family.
Check
Setting, result and conclusion
Fits
Tuned models
Depth 3 or 5, minimum leaf 20 or 50, 100–1,000 trees or rounds; LR C∈{1,0.1,0.01} ; no augmentation. P1: depth 3 removes the tree advantage (EHR F2: RF 0.788 → 0.366, ET 0.793 → 0.315; LR 0.313), yet validation picks unlimited depth in 75 of 80 choices. Tuned trees vs. tuned LR: −0.017 to +0.005 ; 1,000 boosting rounds overfit (down to −0.057 ). No consistent tree advantage.
10,560
Joint tuning
P4-C; for each strategy, LR ( C∈{1,0.1,0.01} ) or LightGBM (four settings) chosen on inner validation. vs. no augmentation: EHR +0.006 to +0.007 , fusion −0.009 to +0.001 (all p≥0.24 ; CIs up to +0.023 ); tuned LightGBM vs. tuned LR −0.020 to −0.003 . No significant or consistent strategy gain within this grid.
2,240
Aligned subset
The 10,479 radiographs whose own admission supplied the labels (and presumably the EHR values); laboratory and vital-sign values may postdate the radiograph, so this evaluates admission alignment, not prediction-time validity. EHR +0.017 over all-positive; RAD-DINO fusion vs. EHR +0.009 ( p=0.24 ); C labels: RAD-DINO vs. EHR +0.028 ( p=0.004 ). EHR stays weak.
4,080
Frontal only
AP/PA radiographs only (20,294). F2 vs. all views −0.001 to +0.015 ; EHR at the all-positive level; RAD-DINO vs. EHR +0.013 (R, p=0.36 ), +0.029 (A, p=0.017 ), +0.050 (C, p=0.011 ). No conclusion changes.
5,760
PCA size
32, 128 or 768 instead of 64 RAD-DINO components (37% of the variance). 128: within 0.008; 32 or 768: down to −0.018 . 64 components adequate for RAD-DINO with LR.
720
Age, history
Own-admission age; prior-diagnosis flags alone. Age: within 0.001 of the released age, at the all-positive level; flags: +0.136 (A), +0.180 (C) over the released EHR. The history gain comes from the diagnoses.
240
TABLE VI: Robustness checks (Section V-F ), patient-separated protocols unless stated. F2: macro outer-test F2; unadjusted p-values from the corrected resampled t -test. Fits: model fits of each check (34,272 in total); full grids are listed in the configuration file of the code release.
Medical image benchmarks compare AI methods on curated clinical images, tasks, and evaluation protocols to measure accuracy, robustness, and clinical usefulness.
Machine learning holds promise for advancing clinical decision support, yet it remains unclear when multimodal learning truly helps in practice, particularly under modality missingness and fairness constraints. In this work, we conduct a systematic benchmark of multimodal fusion between Electronic Health Records (EHR) and chest X-rays (CXR) on standardized cohorts from MIMIC-IV and MIMIC-CXR, aiming to answer four fundamental questions: when multimodal fusion improves clinical prediction, how different fusion strategies compare, how robust existing methods are to missing modalities, and whether multimodal models achieve algorithmic fairness. Our study reveals several key insights. Multimodal fusion improves performance when modalities are complete, with gains concentrating in diseases that require complementary information from both EHR and CXR. While cross-modal learning mechanisms capture clinically meaningful dependencies beyond simple concatenation, the rich temporal structure of EHR introduces strong modality imbalance that architectural complexity alone cannot overcome. Under realistic missingness, multimodal benefits rapidly degrade unless models are explicitly designed to handle incomplete inputs. Moreover, multimodal fusion does not inherently improve fairness, with subgroup disparities mainly arising from unequal sensitivity across demographic groups. To support reproducible and extensible evaluation, we further release a flexible benchmarking toolkit that enables plug-and-play integration of new models and datasets. Together, this work provides actionable guidance on when multimodal learning helps, when it fails, and why, laying the foundation for developing clinically deployable multimodal systems that are both effective and reliable. The open-source toolkit can be found at https://github.com/jakeykj/CareBench.
Kejing Yin, Haizhou Xu, Wenfang Yao +5
Department of Computer Science, Hong Kong Baptist University · Division of Artificial Intelligence, Lingnan University · School of Nursing, The Hong Kong Polytechnic University
Long-tailed class distributions pose a significant challenge for multi-label chest X-ray (CXR) classification, where rare but clinically important findings are severely underrepresented. In this work, we present a systematic empirical evaluation of loss functions, CNN backbone architectures and post-training strategies on the CXR-LT 2026 benchmark, comprising approximately 143K images with 30 disease labels from PadChest. Our experiments demonstrate that LDAM with deferred re-weighting (LDAM-DRW) consistently outperforms standard BCE and asymmetric losses for rare class recognition. Amongst the architectures evaluated, ConvNeXt-Large achieves the best single-model performance with 0.5220 mAP and 0.3765 F1 on our development set, whilst classifier re-training and test-time augmentation further improve ranking metrics. On the official test leaderboard, our submission achieved 0.3950 mAP, ranking 5th amongst all 68 participating teams with total of 1528 submissions. We provide a candid analysis of the development-to-test performance gap and discuss practical insights for handling class imbalance in clinical imaging settings. Code is available at https://github.com/Nikhil-Rao20/Long_Tail.
Nikhileswara Rao Sulake
Department of Computer Science and Engineering, Rajiv Gandhi University of Knowledge Technologies, Nuzvid, India
Chest radiography is a widely used imaging modality for thoracic disease diagnosis, yet its conventional interpretation remains time-consuming and heavily dependent on expert knowledge. While deep learning has improved diagnostic efficiency through automated feature extraction, challenges such as class imbalance and the localization of multiple co-existing pathologies remain unsolved. In this paper, inspired by the strength of Convolutional Block Attention Module (CBAM) in feature refinement and the capability of CNN blocks in feature extraction, we propose a strategy to integrate CBAM into traditional CNN blocks to enhance performance in multi-label classification tasks. Our method achieves a mean AUC of 0.8695 on ChestXray14 dataset, outperforming several state-of-the-art baselines.Our source code is available at: https://github.com/NNNguyenDuyyy/FETC_CBAM_Enhanced_CNN.git
Department of Artificial Intelligence FPT University Ho Chi Minh City, Vietnam · Department of Computing Fundamental FPT University Ho Chi Minh City, Vietnam