cs.LGMar 3, 2026
SaveAn Empirical Analysis of Calibration and Selective Prediction in Multimodal Clinical Condition Classification
Organizations: New York University · New York University Abu Dhabi · University of Toronto
Abstract
As artificial intelligence systems move toward clinical deployment, ensuring reliable prediction behavior is fundamental for safety-critical decision-making tasks. One proposed safeguard is selective prediction, where models can defer uncertain predictions to human experts for review. In this work, we empirically evaluate the reliability of uncertainty-based selective prediction in multilabel clinical condition classification using multimodal ICU data. Across a range of state-of-the-art unimodal and multimodal models, we find that selective prediction can substantially degrade performance despite strong standard evaluation metrics. This failure is driven by severe class-dependent miscalibration, whereby models assign high uncertainty to correct predictions and low uncertainty to incorrect ones, particularly for underrepresented clinical conditions. Our results show that commonly used aggregate metrics can obscure these effects, limiting their ability to assess selective prediction behavior in this setting. Taken together, our findings characterize a task-specific failure mode of selective prediction in multimodal clinical condition classification and highlight the need for calibration-aware evaluation to provide strong guarantees of safety and robustness in clinical AI.
Figures & tables
Figure 1: Selective Prediction Can Serve as a Fail-Safe Mechanism in Safety-Critical Healthcare Settings. a) A model outputs prediction confidence/uncertainty scores; low-confidence cases are rejected and referred to an expert, yielding coverage-controlled evaluation and diagnosis. b) Miscalibration causes predictions that are mostly correct to have high uncertainty (low confidence) and incorrect predictions to have high confidence. Theoretical selective prediction curves show the difference between a well-calibrated model, random selection, and a severely miscalibrated model, showing improving, flat, and degrading performance with increasing coverage. Clinically, overconfidence risks missed or delayed diagnoses, while underconfidence triggers unnecessary rejections and shifts excess workload to clinicians.
| Model | AUROC | AUPRC | Selective AUROC | Selective AUPRC | |
| EHR (LSTM) ( Hochreiter and Schmidhuber, 1997 ) | 0.681 0.007 | 0.371 0.010 | 0.625 0.115 | 0.292 0.091 | 1.42 0.335 |
| CXR (ResNet-34) ( He et al., 2015 ) | 0.654 0.013 | 0.350 0.012 | 0.643 0.095 | 0.342 0.077 | 9.41 0.826 |
| DrFuse ( Yao et al., 2024 ) | 0.726 0.007 | 0.418 0.010 | 0.677 0.139 | 0.352 0.116 | 2.54 1.15 |
| MeTra ( Khader et al., 2023 ) | 0.707 0.010 | 0.399 0.015 | 0.661 0.125 | 0.333 0.104 | 2.32 0.83 |
| MedFuse ( Hayat et al., 2022 ) | 0.739 0.005 | 0.449 0.008 | 0.705 0.136 | 0.396 0.112 | 2.08 0.725 |
| MedFuse + Temp. Scaling ( Guo et al., 2017 ) | 0.739 0.005 | 0.449 0.008 | 0.705 0.135 | 0.398 0.110 | 1.97 0.60 |
Table 1: Performance Across Conditions. Aggregate metrics suggest that multimodal learning (MedFuse) outperforms unimodal baselines. However, these results mask substantial condition-specific variability. Particularly, standard calibration measures (ECE) are deceptively low, concealing severe and consistent miscalibration across individual clinical conditions. This condition-specific variability, motivates the class-level and selective prediction analysis presented in our study. Other multimodal architectures (DrFuse, MeTra) achieve similar aggregate performance, yet, we show that they exhibit comparable calibration failures despite more complex fusion mechanisms. We report improvement gains over EHR given its widespread use as the foundation for most clinical prediction models. ( Best , Second Best ).
Figure 2: How Does Calibration Influence Selective Prediction Behavior? (a) Selective AUROC, selective AUPRC, and class-stratified ECE across 25 clinical conditions for MedFuse as low-confidence predictions are rejected. (b) Relationship between positive-class calibration error (ECE c=1 AUC) and selective AUROC/AUPRC across conditions and random seeds, showing worse selective performance with higher ECE c=1 (Spearman’s rank, . Shaded region: variance across seeds.)
Figure 3: Are Class-dependent Calibration Effects Consistent Across Multimodal Architectures? Across MedFuse, DrFuse, and MeTRA, higher calibration error in the underrepresented class (i.e., positive class in this task) is consistently associated with lower selective AUROC and AUPRC across conditions. For all architectures and metrics, the null hypothesis of no association between stratified ECE and selective AUC is rejected, indicating that architectural complexity alone does not mitigate calibration-driven selective prediction failures. (Spearman’s rank , )
Figure 4: Can Loss-Upweighting Mitigate Observed Calibration Failures? Across architectures loss-upweighting reduces calibration error for the underrepresented class, but the negative relationship between stratified ECE and selective AUROC/AUPRC persists across conditions. For all architectures and metrics, the null hypothesis of no association between ECE and selective AUC is rejected (Spearman’s rank, ).
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Clinical Condition | Organ | Modality | Train Set Prevalence | ||
| CXR | EHR | Positive | Negative | ||
| 1 Acute and unspecified renal failure | Kidneys | - | ✓ | 32.32% (2,532) | 67.68% (5,309) |
| 2 Acute cerebrovascular disease | Brain (Cerebrovascular system) | - | ✓ | 7.99% (625) | 92.01% (7,216) |
| 3 Acute myocardial infarction | Heart (Coronary arteries) | - | ✓ | 8.57% (675) | 91.43% (7,166) |
| 4 Cardiac dysrhythmias | Heart (Conduction system) | - | ✓ | 37.36% (2,942) | 62.64% (4,899) |
| 5 Chronic kidney disease | Kidneys | ✓ | ✓ | 23.34% (1,824) | 76.66% (6,017) |
Table A.1 : Clinical Conditions and Dataset Characteristics. Overview of the 25 conditions analyzed in this study, including their associated organ/system, primary diagnostic modality (CXR or EHR), and the prevalence of positive and negative samples in the training set.
| Clinical Condition | EHR | CXR | MedFuse | ||||||||||||
| AUROC | AUPRC | Selective AUROC | Selective AUPRC | AUROC | AUPRC | Selective AUROC | Selective AUPRC | AUROC | AUPRC | Selective AUROC | Selective AUPRC | ||||
| 1 | 0.724 0.003 | 0.564 0.010 | 0.772 0.100 | 0.627 0.100 | 1.62 0.52 | 0.639 0.009 | 0.453 0.014 | 0.684 0.081 | 0.503 0.075 | 12.77 0.98 | 0.761 0.005 | 0.589 0.008 | 0.804 0.111 | 0.634 0.109 | 2.31 0.97 |
| 2 | 0.873 0.005 | 0.459 0.010 | 0.569 0.180 | 0.085 0.138 | 1.08 0.14 | 0.650 0.009 | 0.139 0.009 | 0.577 0.121 | 0.084 0.057 | 3.99 0.31 | 0.856 0.010 | 0.432 0.019 | 0.560 0.176 | 0.075 0.116 | 0.88 0.24 |
| 3 | 0.693 0.014 | 0.175 0.016 | 0.617 0.150 | 0.089 0.070 | 0.73 0.07 | 0.640 0.015 | 0.153 0.017 | 0.567 0.102 | 0.092 0.037 | 4.35 0.42 | 0.713 0.005 | 0.209 0.010 | 0.633 0.207 | 0.083 0.049 | 0.76 0.23 |
| 4 | 0.598 0.004 | 0.462 0.012 | 0.618 0.069 | 0.491 0.065 | 1.15 0.67 | 0.647 0.009 | 0.534 0.013 | 0.683 0.077 | 0.598 0.077 | 12.50 0.48 | 0.694 0.006 | 0.565 0.010 | 0.727 0.079 | 0.610 0.071 | 2.61 1.65 |
| 5 | 0.696 0.006 | 0.402 0.012 | 0.642 0.113 | 0.244 0.113 | 2.13 0.45 | 0.662 0.021 | 0.378 0.019 | 0.696 0.085 | 0.409 0.078 | 15.37 0.37 | 0.770 0.003 | 0.519 0.010 | 0.787 0.177 | 0.521 0.169 | 2.34 2.04 |
Table C.2 : Do Multimodal Models Improve or Degrade Selective Prediction Performance in this Task? Condition-level performance of standard and selective prediction metrics. MedFuse generally improves AUROC, AUPRC and selective AUROC, but calibration gains are very inconsistent compared to EHR. (Dark-bold: , Wilcoxon signed-rank test, 5 seeds; Light-bold: highest mean, not significant)
| Clinical Condition | MedFuse | DrFuse | MeTra | ||||||||||||
| AUROC | AUPRC | Selective AUROC | Selective AUPRC | AUROC | AUPRC | Selective AUROC | Selective AUPRC | AUROC | AUPRC | Selective AUROC | Selective AUPRC | ||||
| 1 | 0.761 0.005 | 0.589 0.008 | 0.804 0.111 | 0.634 0.109 | 2.31 0.97 | 0.732 0.004 | 0.551 0.008 | 0.767 0.104 | 0.585 0.113 | 4.06 1.21 | 0.722 0.009 | 0.541 0.011 | 0.759 0.083 | 0.573 0.071 | 2.73 0.39 |
| 2 | 0.856 0.010 | 0.432 0.019 | 0.560 0.176 | 0.075 0.116 | 0.88 0.24 | 0.899 0.004 | 0.460 0.013 | 0.584 0.272 | 0.118 0.166 | 1.06 0.44 | 0.900 0.004 | 0.490 0.022 | 0.602 0.253 | 0.112 0.172 | 1.13 0.32 |
| 3 | 0.713 0.005 | 0.209 0.010 | 0.633 0.207 | 0.083 0.049 | 0.76 0.23 | 0.697 0.013 | 0.166 0.010 | 0.637 0.169 | 0.090 0.069 | 2.21 0.58 | 0.717 0.010 | 0.197 0.014 | 0.610 0.158 | 0.085 0.058 | 1.98 1.12 |
| 4 | 0.694 0.006 | 0.565 0.010 | 0.727 0.079 | 0.610 0.071 | 2.61 1.65 | 0.650 0.012 | 0.520 0.013 | 0.692 0.084 | 0.547 0.089 | 3.42 1.66 | 0.628 0.011 | 0.500 0.008 | 0.648 0.076 | 0.529 0.079 | 3.59 1.65 |
| 5 | 0.770 0.003 | 0.519 0.010 | 0.787 0.177 | 0.521 0.169 | 2.34 2.04 | 0.726 0.006 | 0.448 0.009 | 0.693 0.127 | 0.334 0.152 | 3.59 2.23 | 0.709 0.017 | 0.414 0.019 | 0.668 0.101 | 0.299 0.105 | 2.68 1.31 |
Table C.3 : Condition-level Performance of Evaluation Metrics Across Multimodal Architectures. MedFuse, DrFuse, and MeTra achieve broadly comparable discrimination performance across conditions, with MedFuse consistently attaining the highest AUROC/AUPRC values, while all models exhibit elevated calibration error. Shading and bolding are applied within each model to compare standard AUC metrics against their selective AUC counterparts for the same condition. Under correct uncertainty estimation, selective AUROC/AUPRC should be statistically significantly higher than standard metrics, reflecting effective rejection of uncertain predictions. Instead, we observe no consistent improvement across architectures: standard AUC metrics frequently dominate their selective counterparts, indicating that selective prediction fails to provide systematic gains. These results further suggest that increased architectural complexity alone does not improve reliability or resolve calibration-driven failures in this task. (Dark-bold: , Wilcoxon signed-rank test, 5 seeds; Light-bold: highest mean, not significant)
| Clinical Condition | MedFuse (Loss-Upweighting) | DrFuse (Loss-Upweighting) | MeTra (Loss-Upweighting) | ||||||||||||
| AUROC | AUPRC | Selective AUROC | Selective AUPRC | AUROC | AUPRC | Selective AUROC | Selective AUPRC | AUROC | AUPRC | Selective AUROC | Selective AUPRC | ||||
| 1 | 0.761 0.003 | 0.588 0.008 | 0.756 0.088 | 0.672 0.065 | 3.09 0.86 | 0.731 0.004 | 0.547 0.007 | 0.717 0.090 | 0.628 0.081 | 4.58 0.98 | 0.715 0.008 | 0.533 0.012 | 0.721 0.088 | 0.618 0.077 | 3.35 1.48 |
| 2 | 0.871 0.006 | 0.431 0.010 | 0.872 0.093 | 0.491 0.069 | 2.38 0.66 | 0.895 0.005 | 0.473 0.010 | 0.878 0.153 | 0.547 0.135 | 4.44 1.50 | 0.897 0.007 | 0.478 0.034 | 0.871 0.111 | 0.546 0.078 | 3.79 0.79 |
| 3 | 0.716 0.003 | 0.205 0.005 | 0.744 0.167 | 0.260 0.097 | 9.58 2.12 | 0.693 0.007 | 0.175 0.007 | 0.687 0.118 | 0.205 0.073 | 13.34 6.55 | 0.699 0.027 | 0.182 0.012 | 0.694 0.143 | 0.205 0.070 | 10.09 3.99 |
| 4 | 0.692 0.005 | 0.563 0.007 | 0.650 0.084 | 0.631 0.060 | 2.89 1.26 | 0.621 0.005 | 0.492 0.005 | 0.589 0.073 | 0.550 0.066 | 4.58 2.03 | 0.630 0.009 | 0.496 0.010 | 0.595 0.073 | 0.543 0.068 | 1.93 0.65 |
| 5 | 0.760 0.006 | 0.496 0.012 | 0.805 0.094 | 0.599 0.113 | 4.60 1.52 | 0.718 0.002 | 0.437 0.006 | 0.726 0.085 | 0.522 0.088 | 5.90 2.72 | 0.696 0.015 | 0.402 0.013 | 0.678 0.095 | 0.467 0.080 | 4.98 3.28 |
Table C.5 : Condition-level Performance of Evaluation Metrics Across Multimodal Architectures under Loss-Upweighting. Although the performance of all three models is comparable, MedFuse outperforms both DrFuse and MeTra across different conditions and evaluation metrics, with the downside that all models present high ECE scores and the difference between models is not statistically significant. This further confirms that complex architectures do not necessarily improve predictive performance and do not solve the reliability issues observed in this clinical task. (Dark-bold: , Wilcoxon signed-rank test, 5 seeds; Light-bold: highest mean, not significant)
| Clinical Condition | MedFuse (Loss-Upweighting) | DrFuse (Loss-Upweighting) | MeTra (Loss-Upweighting) | ||||||
| 1 | 3.09 0.86 | 3.81 1.13 | 5.01 1.96 | 4.58 0.98 | 4.57 0.84 | 6.32 1.96 | 3.35 1.48 | 7.19 4.48 | 7.36 2.53 |
| 2 | 2.38 0.66 | 10.82 1.46 | 2.10 0.76 | 4.44 1.50 | 9.75 2.09 | 4.69 1.66 | 3.79 0.79 | 5.78 1.66 | 4.41 0.85 |
| 3 | 9.58 2.12 | 12.14 1.93 | 11.33 2.29 | 13.34 6.55 | 11.71 2.50 | 15.62 7.08 | 10.09 3.99 | 18.20 13.24 | 12.53 5.48 |
| 4 | 2.89 1.26 | 7.52 2.45 | 5.32 5.21 | 4.58 2.03 | 7.31 2.60 | 11.83 4.69 | 1.93 0.65 | 11.11 5.16 | 8.07 3.52 |
| 5 | 4.60 1.52 | 5.05 2.19 | 5.40 2.97 | 5.90 2.72 | 5.60 2.70 | 8.21 4.41 | 4.98 3.28 | 12.78 12.05 | 9.92 7.48 |
Table C.6 : Condition-level Performance of ECE Stratification Across Multimodal Architectures under Loss-Upweighting. Class-stratified calibration metrics under loss upweighting show reduced positive-class ECE for many conditions, but it still remains dominant across architectures. No single multimodal model consistently achieves the lowest ECE across conditions, indicating that loss upweighting mitigates but does not resolve class-dependent calibration errors. (Dark-bold: , Wilcoxon signed-rank test, 5 seeds; Light-bold: highest mean, not significant)
| Clinical Condition | EHR | CXR | MedFuse | ||||||
| 1 | 2.32 0.48 | 57.80 0.36 | 27.38 0.09 | 14.39 1.15 | 56.87 3.22 | 29.87 3.00 | 3.22 0.91 | 54.09 2.26 | 25.83 2.03 |
| 2 | 1.02 0.21 | 66.59 0.37 | 6.23 0.14 | 5.87 0.47 | 86.92 2.62 | 6.84 1.62 | 1.29 0.26 | 67.96 1.40 | 5.40 0.18 |
| 3 | 1.37 0.12 | 88.27 0.33 | 8.22 0.06 | 6.75 1.33 | 85.30 3.50 | 8.38 2.50 | 1.46 0.30 | 86.93 0.14 | 8.05 0.25 |
| 4 | 2.37 0.50 | 60.63 0.13 | 36.44 0.12 | 14.13 1.07 | 52.81 4.49 | 32.66 4.52 | 3.93 1.67 | 55.81 3.90 | 32.44 3.51 |
| 5 | 2.65 0.37 | 70.95 0.28 | 21.08 0.07 | 17.08 0.70 | 59.96 1.82 | 22.65 1.99 | 3.64 1.62 | 61.94 4.35 | 18.95 3.07 |
Table C.9 : Adaptive ECE for Unimodal Models. Aggregate and class-stratified Adaptive Expected Calibration Error (AECE) are reported for unimodal baselines. Positive-class AECE remains substantially higher than negative-class AECE, confirming that minority-class miscalibration persists when calibration error is estimated using adaptive equal-frequency bins. (Dark-bold: , Wilcoxon signed-rank test, 5 seeds; Light-bold: highest mean, not significant)
| Clinical Condition | MedFuse | DrFuse | MeTra | ||||||
| 1 | 3.22 0.91 | 54.09 2.26 | 25.83 2.03 | 5.16 1.20 | 55.57 3.44 | 26.30 2.71 | 3.01 0.39 | 59.40 1.02 | 26.08 1.02 |
| 2 | 1.29 0.26 | 67.96 1.40 | 5.40 0.18 | 1.92 1.05 | 60.49 4.83 | 6.21 1.11 | 2.02 0.62 | 59.96 5.55 | 6.44 1.01 |
| 3 | 1.46 0.30 | 86.93 0.14 | 8.05 0.25 | 2.37 0.35 | 88.15 2.27 | 7.83 1.71 | 2.11 1.13 | 84.75 2.92 | 9.34 1.95 |
| 4 | 3.93 1.67 | 55.81 3.90 | 32.44 3.51 | 4.82 2.29 | 58.41 5.60 | 34.07 4.56 | 5.16 1.72 | 59.83 4.55 | 34.46 4.44 |
| 5 | 3.64 1.62 | 61.94 4.35 | 18.95 3.07 | 4.64 2.03 | 66.72 4.71 | 19.40 3.56 | 3.36 1.01 | 70.00 1.94 | 20.06 1.60 |
Table C.10 : Adaptive ECE for Multimodal Models. Aggregate and class-stratified Adaptive Expected Calibration Error (AECE) are reported for standard multimodal architectures. The positive class exhibits substantially higher AECE than the negative class across MedFuse, DrFuse, and MeTra, reinforcing that multimodal performance gains do not eliminate class-dependent calibration errors. (Dark-bold: , Wilcoxon signed-rank test, 5 seeds; Light-bold: highest mean, not significant)
| Clinical Condition | MedFuse (Loss-Upweighting) | DrFuse (Loss-Upweighting) | MeTra (Loss-Upweighting) | ||||||
| 1 | 14.47 1.33 | 38.49 1.01 | 39.54 1.49 | 14.91 1.51 | 40.19 1.19 | 40.42 1.72 | 13.67 3.13 | 43.47 3.99 | 40.70 2.76 |
| 2 | 15.85 1.07 | 28.20 1.19 | 19.61 1.08 | 15.46 3.11 | 26.76 3.93 | 18.51 3.20 | 21.47 3.66 | 22.96 1.76 | 25.26 3.93 |
| 3 | 30.73 2.15 | 43.34 2.27 | 38.31 2.15 | 35.58 5.53 | 41.28 5.99 | 43.24 5.51 | 31.64 7.15 | 46.78 8.88 | 39.66 6.98 |
| 4 | 11.02 2.99 | 43.65 2.86 | 44.38 3.15 | 12.31 1.80 | 46.40 1.75 | 48.18 1.84 | 9.66 2.90 | 48.86 3.10 | 45.35 2.94 |
| 5 | 19.10 2.90 | 39.06 2.45 | 37.57 3.07 | 22.74 3.28 | 41.06 3.14 | 42.39 3.37 | 21.17 5.49 | 45.53 6.87 | 42.35 5.08 |
Table C.11 : Adaptive ECE for Multimodal Models under Loss-Upweighting. Aggregate and class-stratified Adaptive Expected Calibration Error (AECE) are reported for multimodal architectures trained with loss upweighting. Loss upweighting reduces positive-class AECE relative to standard multimodal models, but it redistributes calibration error toward the negative class, indicating that this mitigation does not fully resolve class-dependent miscalibration. (Dark-bold: , Wilcoxon signed-rank test, 5 seeds; Light-bold: highest mean, not significant)
| Clinical Condition | MedFuse (Loss-Upweighting) | DrFuse (Loss-Upweighting) | MeTra (Loss-Upweighting) | ||||||
| 1 | 20.22 0.47 | 19.19 1.02 | 20.71 1.16 | 21.30 0.55 | 20.82 1.21 | 21.52 1.34 | 21.35 0.81 | 22.66 3.41 | 20.73 2.78 |
| 2 | 10.68 0.51 | 17.41 0.75 | 10.10 0.61 | 10.46 1.44 | 16.47 2.92 | 10.00 1.77 | 12.87 1.95 | 11.96 1.37 | 12.94 2.23 |
| 3 | 20.79 1.45 | 23.15 2.02 | 20.55 1.80 | 24.24 4.12 | 21.09 5.20 | 24.56 5.03 | 20.63 4.35 | 25.50 8.98 | 20.14 5.71 |
| 4 | 22.49 0.73 | 21.98 2.67 | 22.81 2.78 | 24.15 0.56 | 22.95 1.58 | 24.88 1.85 | 23.48 0.51 | 25.52 3.00 | 22.23 2.64 |
| 5 | 20.05 1.22 | 20.14 2.46 | 20.02 2.35 | 22.08 1.49 | 20.60 2.84 | 22.54 2.81 | 21.79 2.29 | 23.77 6.22 | 21.17 4.94 |
Table C.14 : Brier Score for Multimodal Models under Loss-Upweighting. Aggregate and class-stratified Brier scores are reported for multimodal architectures trained with loss upweighting. Loss upweighting reduces positive-class Brier loss but increases negative-class and aggregate Brier loss, suggesting that the intervention shifts calibration error across classes rather than fully resolving the reliability gap. (Dark-bold: , Wilcoxon signed-rank test, 5 seeds; Light-bold: highest mean, not significant)
Figure C.1 : Unimodal CXR Stratified Calibration and Selective Prediction Across Clinical Conditions. For unimodal CXR, stratified positive-class ECE exhibits substantial variability across seeds and conditions (top), and AUROC/AUPRC remain below multimodal MedFuse performance. Across conditions, higher positive-class ECE AUC is associated with lower selective AUC (middle; Spearman’s rank ). Negative-class ECE AUC shows no clear monotonic relationship with selective AUC (bottom).
Figure C.2 : Unimodal EHR Stratified Calibration and Selective Prediction Across Clinical Conditions. Positive-class ECE is elevated for several conditions, indicating overconfidence on underrepresented positive cases. Compared to CXR, stratified ECE and AUROC/AUPRC show lower variability across seeds and conditions, yielding more stable predictions that closely mirror MedFuse behavior. Positive-class ECE AUC remains negatively associated with selective AUC, while negative-class ECE shows no clear monotonic relationship with selective performance.
Figure C.3 : MedFuse Stratified Calibration and Selective Prediction Across Clinical Conditions. Across conditions, higher positive-class ECE AUC consistently predicts lower selective AUC, indicating that overconfidence in positive predictions drives instability in selective evaluation. In contrast, negative-class ECE AUC remains low for most conditions and shows no clear monotonic relationship with selective AUC, suggesting that miscalibration is primarily driven by overconfident positive predictions rather than negative-class errors.
Figure C.4 : DrFuse Stratified Calibration and Selective Prediction Across Clinical Conditions. DrFuse exhibits elevated positive-class ECE across several conditions, indicating persistent overconfidence on underrepresented positive cases. Stratified ECE and selective AUROC/AUPRC closely mirror MedFuse trends, with higher positive-class ECE AUC associated with degraded selective performance. As in other models, negative-class ECE shows no clear monotonic relationship with selective AUC, suggesting that increased fusion complexity alone does not resolve calibration-driven reliability failures.
Figure C.5 : MeTRA Stratified Calibration and Selective Prediction Across Clinical Conditions. MeTRA shows class-dependent calibration behavior similar to MedFuse and DrFuse, with elevated positive-class ECE indicating overconfidence on underrepresented positives. Positive-class ECE AUC is negatively associated with selective AUC, while selective AUROC/AUPRC exhibit higher variance across seeds, consistent with other multimodal models, indicating that architectural complexity alone does not resolve calibration-driven selective instability.
Figure D1 : Comparative analysis for Acute Cerebrovascular Disease. For this condition, EHR predominantly drives the discriminative performance of MedFuse, while CXR shows poor positive-class calibration and contributes to the oscillatory behavior of selective AUROC. Applying label-weighted loss reduces early-threshold miscalibration in positive predictions but does not fully correct calibration at higher thresholds, leaving the overall selective prediction trend largely unchanged.
Figure D2 : Comparative analysis for Conduction Disorders. For this condition, the CXR model shows substantial seed-dependent variability, achieving higher mean performance but poorer calibration consistency than EHR. MedFuse integrates both modalities to improve selective AUROC and stabilize performance, though positive-class ECE remains variable. Label-weighted loss further reduces seed variance and attenuates positive-class ECE at higher confidence thresholds, yielding smoother and more reliable selective behavior, but with similar performance drops at the final thresholds.
Figure D3 : Comparative analysis for Respiratory Shock. For this condition, unimodal EHR and CXR achieve comparable baseline performance, with EHR showing slightly lower positive-class ECE at early thresholds. As rejection increases, however, EHR’s AUROC quickly deteriorates while CXR remains uniformly undercalibrated, and MedFuse largely inherits these calibration inconsistencies. Label-weighted loss substantially improves both stability and calibration, suppressing positive-class ECE spikes and producing a clearer monotonic gain in selective AUROC. This condition provides one of the strongest examples of how simple loss upweighting can directly reduce positive-class miscalibration and enhance the reliability of uncertainty estimates.
Figure D4 : Linear Regression of AUROC Against Stratified ECE. Relationship between AUROC and both positive- and negative-class ECE across 25 conditions. Regression slopes demonstrate a consistent, albeit not conclusive, negative association, indicating that poorer calibration modestly reduces discriminative performance in terms of AUROC.
Figure D5 : Linear Regression of AUPRC Against Stratified ECE. Relationship between AUPRC and class-stratified ECE across 25 conditions. The correlation between ECE and AUPRC exhibits a clear, monotonic decline, confirming via another approach, that high positive-class miscalibration strongly predicts lower precision-recall performance under selective prediction.