SMILE: Self-Explainable Multimodal Information Bottleneck for Medical Diagnosis
Authors: Yuqing Yang, Alexander Schmatz, Zhaozhao Ma, Changkyu Choi, Robert Jenssen, Shujian Yu
Organizations: Independent Researcher · Leiden Institute of Advanced Computer Science (LIACS), Leiden University, Leiden, The Netherlands · Quantitative Data Analytics Group, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands · School of Applied and Creative Computing, Purdue University, West Lafayette, IN, USA · Department of Informatics, University of Oslo, Oslo, Norway · Machine Learning Group, UiT — The Arctic University of Norway, Tromsø, Norway
Explainability is increasingly seen as a crucial requirement in AI-based medical diagnosis, particularly in safety-critical clinical decision-making. Most existing explainability methods in healthcare operate in a post-hoc manner and are predominantly designed for unimodal data, which limits their applicability in increasingly prevalent multimodal diagnostic settings. This paper addresses the problem of self-explainable multimodal diagnosis by formulating it within the information bottleneck (IB) framework. We propose a unified learning paradigm that jointly optimizes predictive performance and modality-specific explainability by identifying the most informative elements inside each modality that contribute to diagnostic decisions. To enable tractable and stable optimization, we employ a matrix-based Renyi's α-order entropy functional under the assumption of sufficiently expressive encoders. Extensive experiments on representative medical datasets spanning heterogeneous modalities demonstrate that the proposed method consistently achieves strong diagnostic performance, including an absolute accuracy improvement of 9.1 percentage points on the iCTCF dataset. Moreover, the learned explanations provide transparent and modality-aware insights into feature relevance, thereby improving both the explainability and generalization.
Figures & tables
Figure 1: Overview of the proposed SMILE framework. We design a modality-specific explainer to identify the top- k most informative features within each modality that influence diagnostic decisions. All selected features {Em}m=1M serve as the final explanations and are directly used for prediction, ensuring consistency between the explanations and the model decisions.
Figure 2: A 2D toy example illustrating how minimizing mutual information I(X;X~) and entropy H(X~) aids in feature selection.
Major Depressive Disorder: 1300 / Healthy Controls: 1128 (830 / 771 after QC)
Table 1: Dataset characteristics. Note that we use a prognosis task for iCTCF rather than a diagnostic task to identify infection, to better showcase explanations.
BRCA
ROSMAP
Method
ACC
WeightedF1
MacroF1
ACC
F1
AUC
DIABLO [ 55 ]
64.2 ± 0.9
53.4 ± 1.7
36.9 ± 1.7
74.2 ± 2.4
75.5 ± 2.5
83.0 ± 2.5
MOGONET [ 10 ]
82.9 ± 1.8
82.5 ± 1.7
77.4 ± 1.7
81.5 ± 2.3
82.1 ± 1.2
87.4 ± 1.2
TMC [ 56 ]
84.2 ± 0.5
84.4 ± 0.9
80.6 ± 0.9
82.5 ± 0.9
82.3 ± 0.6
88.5 ± 0.6
Dynamic [ 11 ]
87.1 ± 0.5
87.4 ± 0.6
83.5 ± 0.5
81.7 ± 1.5
82.3 ± 1.5
90.0 ± 1.2
DMIB [ 12 ]
86.0 ± 0.7
86.0 ± 0.8
81.6 ± 0.9
84.9 ± 1.8
85.3 ± 1.7
91.6 ± 0.7
Table 2: Performance of various multimodal methods on BRCA and ROSMAP datasets. The best results are in bold, and the second-best results are underlined.
ACC
AUC
JW 0.5
JW 0.6
De-COVID19-Net [ 57 ]
72.4
77.3
69.8
69.2
HoFN+SCResNet [ 58 ]
68.4
80.2
70.1
71.6
HUST-19 [ 50 ]
83.3
92.1
–
–
DMIB [ 12 ]
73.2
82.3
72.1
71.7
SMILE (ours)
92.4 ± 0.5
95.6 ± 0.6
84.3 ± 0.5
82.2 ± 0.3
Table 3: Performance of various multimodal methods on iCTCF dataset. SMILE results are reported as the mean and standard deviation across 10 runs of 5-fold cross-validation.
ACC
F1-score
AUC
MCC
DER [ 59 ]
56.3
41.3
66.8
17.4
MoNIG [ 60 ]
63.6
64.8
67.9
27.0
MIB [ 32 ]
55.9
70.1
77.7
21.3
MEIB [ 61 ]
52.2
68.5
82.2
6.7
SMILE (ours)
67.4 ± 2.3
69.7 ± 2.6
73.5 ± 3.6
35.0 ± 4.6
Table 4: Performance comparison of various multimodal methods on the REST-meta-MDD dataset (%). Best results are highlighted in bold. Our proposed SMILE achieves compelling performance in terms of all four metrics.
DuelRes
DuelRes (SMILE)
Res-DEN
Res-DEN (SMILE)
Kappa
65.4 ± 1.2
69.2 ± 0.8
70.1 ± 0.5
72.4 ± 0.5
Table 5: Performance of different strategies on Glaucoma Grading dataset. To compare directly with pre-trained models used for the classification task [ 51 ] , we use them as encoders.
Figure 4: The top-5 biomarkers among the k=30 features selected by SMILE are shown for two datasets (top row: BRCA ; bottom row: ROSMAP ) across three modalities: (a) mRNA expression, (b) DNA methylation, and (c) miRNA expression.
Figure 5: iCTCF visualizations on two modalities. (a) Qualitative examples of CT scan in the 2D montage: each explanation contains k=30 patches. The left rows present severe patients’ samples; the right rows present mild patients’ samples. (b) Qualitative examples of clinical information: normalized results from 150 test samples, k=20 features selected.
Figure 6: Glaucoma Grading visualizations on two modalities. (a) Qualitative examples of color fundus ( k=100 ); (b) Qualitative examples of OCT slices ( k=100 )
Figure 7: REST-meta-MDD biomarker visualizations on two modalities. (a) HC ( k=25 ); (b) MDD ( k=25 ). Top: sMRI subvolumes on axial slices. Bottom left: selected rs-fMRI edges on the cortical surface, nodes colored by AAL-116 ROI. Bottom right: circular connectome of the same edges with ROI labels. Color encodes selection frequency. Descriptive aggregates, not statistical comparisons; one run, whose posterior structural pattern recurred in 6 of 10 runs.
Figure 8: Qualitative examples of explanations generated by LIME, SHAP, and SMILE on different datasets. For CT scans in iCTCF dataset, we select k=30 with the patch size of 70×70 ; for OCT slices in Glaucoma Grading dataset, k=100 with the patch size of 4×4 .
Dataset
Fidelity (average confidence drop[%] ↓ )
Consistency (maximum sensitivity ↓ )
LIME
SHAP
SMILE
LIME
SHAP
SMILE
iCTCF
36.4 ± 2.1
48.1 ± 1.8
30.2 ± 0.9
0.48
0.37
0.22
Glaucoma
12.0 ± 0.8
15.1 ± 1.4
9.2 ± 0.6
0.39
0.41
0.27
Table 6: Comparison between built-in (ours) and post-hoc explanations. A lower value indicates better performance for metrics.
Dataset
Modality
w/o Xm
w/ Er
BRCA/ ROSMAP
mRNA
80.4 ± 1.1/78.6 ± 1.8
84.7 ± 1.9/82.3 ± 2.0
DNA meth
84.2 ± 0.7/81.2 ± 1.4
85.0 ± 1.5/81.8 ± 1.8
miRNA
83.9 ± 0.9/79.0 ± 1.5
84.3 ± 1.5/82.1 ± 1.7
mRNA+DNA meth
68.4 ± 1.4/63.7 ± 1.8
77.8 ± 4.2/75.4 ± 5.2
mRNA+miRNA
66.7 ± 1.8/61.5 ± 2.0
75.9 ± 4.7/71.1 ± 5.8
miRNA+DNA meth
71.4 ± 1.1/65.8 ± 1.3
79.6 ± 3.9/78.3 ± 4.9
Table 7: Ablation study results on different datasets. The modality mentioned refers to the object we operate on.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: iCTCF visualizations on 2D HRCT montages. Panels (a) and (b) show representative severe and mild patients, respectively. The highlighted regions indicate the HRCT patches selected by SMILE as prognostic evidence.
Figure 10: Glaucoma Grading visualizations. Panels (a) and (b) show additional representative cases under the same multimodal setting. SMILE highlights selected regions on color fundus images and OCT slices, providing modality-specific evidence for glaucoma-grade prediction.
Computer Vision, Institute of Cognitive Science, Osnabrück University, Germany. · Department of Mathematics, Rhodes University, South Africa. · National Institute for Theoretical and Computational Sciences (NITheCS), Stellenbosch, 7600, South Africa. +2