cs.CVMar 2, 2026
SaveBridging the gap between Performance and Interpretability: An Explainable Disentangled Multimodal Framework for Cancer Survival Prediction
Organizations: AI Technology for Life, Department of Information and Computing Sciences, Department of Biology, Utrecht University, Utrecht, The Netherlands · Department of Metabolic Diseases, Wilhelmina Children’s Hospital, University Medical Center Utrecht, Utrecht, The Netherlands · Regenerative Medicine Center Utrecht, Utrecht, The Netherlands · Computational Pathology, Department of Pathology, The Netherlands Cancer Institute, Amsterdam, The Netherlands · Visualization and Graphics, Department of Information and Computing Sciences, Utrecht University, Utrecht, The Netherlands · Lead contact
Abstract
While multimodal survival prediction models are increasingly more accurate, their complexity often reduces interpretability, limiting insight into how different data sources influence predictions. To address this, we introduce DIMAFx, an explainable multimodal framework for cancer survival prediction that produces disentangled, interpretable modality-specific and modality-shared representations from histopathology whole-slide images and transcriptomics data. Across multiple cancer cohorts, DIMAFx achieves state-of-the-art performance and improved representation disentanglement. Leveraging its interpretable design and SHapley Additive exPlanations, DIMAFx systematically reveals key multimodal interactions and the biological information encoded in the disentangled representations. In breast cancer survival prediction, the most predictive features contain modality-shared information, including one capturing solid tumor morphology contextualized primarily by late estrogen response, where higher-grade morphology aligned with pathway upregulation and increased risk, consistent with known breast cancer biology. Key modality-specific features capture microenvironmental signals from interacting adipose and stromal morphologies. These results show that multimodal models can overcome the traditional trade-off between performance and explainability, supporting their application in precision medicine.
Figures & tables
Figure 1: Overview of DIMAFx . Our proposed framework encodes each modality with interpretable features, then applies disentangled attention fusion and aggregation to produce four disentangled representations. The representations are used for the downstream tasks, including survival prediction. Both unimodal and multimodal feature importance is assessed using SHAP values, and the attention matrices from the self- and cross-attention layers provide insights into the intra- and inter-modal interactions.
Figure 2: Kaplan–Meier survival curves for DIMAFx, showing high- and low-risk test patient groups stratified by the median predicted risk score, with corresponding hazard ratios and p-values.
Figure 3: Unimodal interpretability analysis of DIMAFx. (A) SHAP values for the top unimodal features, ranked by mean absolute SHAP value and colored by predicted log2 risk. (B) Visualization of the five most important WSI prototype features. Per feature, the cardinality (c) of the prototype is displayed, indicating their average frequency across all test samples, and two patches representing the prototype are shown, one associated with a high-risk prediction and one with a low-risk prediction. W9: High-risk patch shows solid sheets of tumor cells with dark, hyperchromatic nuclei infiltrating adipocytes (arrows); low-risk patch shows occasional tubule formation (arrowheads) and lower nuclear grade. W0 : High-risk patch exhibits solid growth, marked pleomorphism, vesicular chromatin, and tumor necrosis (arrows); low-risk patch shows occasional gland formation (arrowheads), more uniform cells, and lower nuclear atypia. W10 : High-risk patch shows solid tumor nests with marked atypia (arrowheads) in stroma containing few immune cells; low-risk patch shows tubular structures (arrows) with low nuclear atypia and sparse immune cells. W3 : High-risk patch shows solid tumor nests with basophilic vacuolated cytoplasm, pleomorphic nuclei (arrowheads), and apoptotic cells; low-risk patch shows gland formation (arrow) with lower nuclear atypia. W8 : High-risk patch shows solid nests with abundant cytoplasm, vesicular nuclei (arrowheads), and marked atypia; low-risk patch shows solid growth with lower nuclear atypia (arrowheads), suggesting lower histologic grade. (C) Visualization of the five most important transcriptomic pathway features, illustrating how the model-assigned feature risk (SHAP values) varies with the mean log-transformed RSEM-normalized pathway expression.
Figure 4: Normalized SHAP values for modality-shared versus modality-specific features showing that shared features generally contribute more to survival prediction. WSI-derived features are colored red, and transcriptomic pathway-derived features are colored blue. The dotted line indicates when the SHAP of the modality-specific feature is equal to the SHAP of the modality-shared feature.
Figure 5: Multimodal interpretability analysis of DIMAFx for the Shared W8 feature, representing a solid-pattern tumor WSI feature contextualized by transcriptomic pathway information. (A) The most highly attended pathway features. (B) Interaction between the W8 and R13 features across all test samples. Each point represents a test case, with SHAP values for R13 (x-axis) and W8 (y-axis) reflecting the model-assigned feature risk, and is colored by the final predicted risk. (C) Visualization of the W8 and R13 features for four representative cases. The ridge plots show the frequency distributions of log transformed RSEM normalized gene expression values for the “R13: Estrogen response late” pathway in the highlighted case (green) and averaged over all train samples (gray). For the “W8: Tumor, solid pattern, abundant cytoplasm” feature, we show a representative patch and the cardinality (c) of this morphological prototype from each case. Case 168 : the W8 prototype shows a tumor with a solid growth pattern with focal gland formation (arrow), large vacuolated cytoplasm (arrowheads), and high-grade nuclear atypia with conspicuous nucleoli. Case 76 : tumor cells forming solid sheets with vague tubule formation (arrowheads) in a collagenous stroma, with scant cytoplasm. Case 52 : tumor cells forming solid sheets with vesicular chromatin and conspicuous nucleoli (arrowheads). Case 132 : glandular growth pattern (arrows) with more uniform cells and lower nuclear atypia.
Figure 6: Multimodal interpretability analysis of DIMAFx into two multimodal features. (A, B) “Specific W7”, an adipose/stroma WSI-specific feature; showing (A) the most highly attended WSI features by W7, and (B) the interaction between the “W7: Adipose stroma” and “W4: Adipose stroma”, plotting the SHAP value of the two features. Each sample is colored by the final predicted log2 risk. (C, D) “Shared R48”, KRAS signaling down pathway contextualized by WSI features; showing (C) the most highly attended WSI features, and (D) the interaction between the “R48: KRAS signaling down” and “W0: Tumor”, plotting the SHAP value of the two features, colored by the final predicted risk.
Figure 7: Unimodal feature extraction framework for DIMAFx, for the WSIs (top) and the gene expression data (bottom).
| Modality | Model | Avg. | BRCA | BLCA | LUAD | KIRC |
|---|---|---|---|---|---|---|
| Clinical | CoxPH 9 | 0.496 | 0.491±0.113 | 0.528±0.073 | 0.463±0.067 | 0.502±0.111 |
| WSI | ABMIL 24 | 0.606 | 0.624±0.083 | 0.541±0.077 | 0.570±0.043 | 0.688±0.135 |
| PANTHER 53 | 0.650 | 0.627±0.071 | 0.605±0.066 | 0.604±0.016 | 0.763±0.138 | |
| DIMAFx | 0.653 | 0.699±0.048 | 0.581±0.064 | 0.600±0.024 | 0.733±0.124 | |
| Transcript | MLP | 0.654 | 0.692±0.078 | 0.672±0.059 | 0.603±0.049 | 0.650±0.148 |
| omics | Pathway MLP | 0.673 | 0.685±0.045 | 0.663±0.054 | 0.630±0.068 | 0.712±0.075 |
Table 1: DSS test results measured by C-index (mean std) for each TCGA cancer cohort and the overall average (Avg). The best and second-best performances are denoted by bold and underlined , respectively.
| Modality | Model | Avg. | BRCA | BLCA | LUAD | KIRC |
|---|---|---|---|---|---|---|
| Clinical | CoxPH 9 | 0.562 | 0.611±0.192 | 0.576±0.111 | 0.551±0.089 | 0.509±0.116 |
| WSI | ABMIL 24 | 0.593 | 0.593±0.144 | 0.528±0.062 | 0.554±0.061 | 0.698±0.113 |
| PANTHER 53 | 0.618 | 0.543±0.155 | 0.573±0.089 | 0.586±0.038 | 0.771±0.128 | |
| DIMAFx | 0.619 | 0.627±0.095 | 0.526±0.088 | 0.588±0.040 | 0.736±0.115 | |
| Transcript | MLP | 0.576 | 0.559±0.073 | 0.561±0.086 | 0.496±0.118 | 0.686±0.113 |
| omics | Pathway MLP | 0.590 | 0.533±0.088 | 0.576±0.114 | 0.494±0.121 | 0.755±0.073 |
Table 2: DSS test results measured by C-index IPCW (mean std) for each TCGA cancer cohort and overall average (Avg). The best and second-best performances are indicated by bold and underlined , respectively.
| BRCA | BLCA | LUAD | KIRC | Avg. | ||
|---|---|---|---|---|---|---|
| PIBD 69 | D1 | 0.324±0.022 | 0.444±0.010 | 0.444±0.033 | 0.531±0.034 | 0.436 |
| D2 | 0.924±0.027 | 0.898±0.076 | 0.960±0.019 | 0.930±0.023 | 0.928 | |
| Total | 0.624±0.021 | 0.671±0.037 | 0.702±0.022 | 0.731±0.012 | 0.682 | |
| DIMAF 15 | D1 | 0.244±0.043 | 0.401±0.029 | 0.339±0.046 | 0.525±0.089 | 0.377 |
| D2 | 0.469±0.050 | 0.672±0.068 | 0.609±0.029 | 0.931±0.031 | 0.670 | |
| Total | 0.356±0.045 | 0.537±0.048 | 0.474±0.018 | 0.728±0.053 | 0.524 |
Table 3: Disentanglement test results measured by DC over the different folds (mean std). We denote the best and second-best total disentanglement by bold and underlined , respectively.
Figure S1: Kaplan–Meier survival curves for DIMAFx , showing high- and low-risk patient groups stratified by the median predicted risk score, with corresponding hazard ratios and p-values.
Figure S2: Kaplan–Meier survival curves for DIMAF, showing high- and low-risk patient groups stratified by the median predicted risk score, with corresponding hazard ratios and p-values.
Figure S3: Kaplan–Meier survival curves for MMP, showing high- and low-risk patient groups stratified by the median predicted risk score, with corresponding hazard ratios and p-values.
Figure S4: Kaplan–Meier survival curves for SurvPath, showing high- and low-risk patient groups stratified by the median predicted risk score, with corresponding hazard ratios and p-values.
Figure S5: Kaplan–Meier survival curves for PIBD, showing high- and low-risk patient groups stratified by the median predicted risk score, with corresponding hazard ratios and p-values.
| BRCA | BLCA | LUAD | KIRC | Avg. | ||
|---|---|---|---|---|---|---|
| PIBD | D1 | 0.334±0.024 | 0.216±0.055 | 0.246±0.116 | 0.183±0.063 | 0.245 |
| D2 | 0.159±0.052 | 0.367±0.086 | 0.354±0.072 | 0.328±0.021 | 0.302 | |
| Total | 0.246±0.034 | 0.291±0.051 | 0.300±0.068 | 0.256±0.022 | 0.273 | |
| DIMAF | D1 | 0.073±0.027 | 0.083±0.020 | 0.090±0.038 | 0.069±0.034 | 0.079 |
| D2 | 0.060±0.018 | 0.051±0.009 | 0.042±0.007 | 0.042±0.006 | 0.049 | |
| Total | 0.066±0.009 | 0.067±0.012 | 0.066±0.020 | 0.056±0.015 | 0.064 |
Table S1: Disentanglement test results measured by the orthogonal score over the different folds (mean std). We denote the best and second-best total disentanglement by bold and underlined , respectively.
| BRCA | BLCA | LUAD | KIRC | Avg. | ||
|---|---|---|---|---|---|---|
| DIMAF | Specific | 0.185±0.038 | 0.273±0.038 | 0.257±0.027 | 0.302±0.043 | 0.254 |
| Shared | 0.815±0.038 | 0.727±0.038 | 0.743±0.027 | 0.698±0.043 | 0.746 | |
| DIMAFx | Specific | 0.543±0.028 | 0.569±0.022 | 0.561±0.027 | 0.603±0.034 | 0.569 |
| Shared | 0.457±0.028 | 0.431±0.022 | 0.439±0.027 | 0.397±0.034 | 0.431 | |
| DIMAFx | Specific | 0.168±0.025 | 0.267±0.023 | 0.235±0.017 | 0.313±0.054 | 0.246 |
| Shared | 0.832±0.025 | 0.733±0.023 | 0.765±0.017 | 0.687±0.054 | 0.754 |
Table S2: Normalized SHAP values of the disentangled representations over the different folds (mean std).
Figure S6: Visualization of the “W7: Adipose stroma” and “W4: Adipose stroma” prototype features for four representative cases, showing fibrous and fibroadipose tissue. Case 58 : the W7 patch shows adipose tissue with histiocytic infiltration (arrows), suggestive of fat necrosis, while the W4 patch shows fibroadipose tissue without apparent abnormality. Case 13 : the W7 patch demonstrates fibroadipose tissue with activated fibroblasts (arrowheads) and a mild chronic inflammatory infiltrate; the W4 patch shows fibrous tissue fragments without apparent abnormality. Case 15 : the W7 patch shows fibroadipose tissue with unremarkable findings, and the W4 patch shows fibrous tissue. Case 195 : the W7 patch demonstrates fibroadipose tissue with a blood vessel apparent within the fibrous tissue (arrow), whereas the W4 patch shows detached collagen fibers within the stroma.
Figure S7: Visualization of the “R48: KRAS signaling down” and “W0: Tumor” features for three representative cases. The ridge plots show the frequency distributions of log transformed RSEM normalized gene expression values for the “R48: KRAS signaling down” pathway in the highlighted case (green) and averaged over all train samples (gray). For the “W0: Tumor” feature, we show the prototype cardinality (c) and a representative patch for each case. Case 37 : the W0 prototype has a relatively low cardinality and shows tumor cells infiltrating singly within stroma and exhibiting hyperchromatic (darkly stained), atypical nuclei (arrowheads). This is consistent with a high-grade morphology. The KRAS signaling down pathway distribution seems slightly upregulated. Case 179 : W0 shows atypical tumor cell infiltration forming small nests (arrows) with high-grade nuclear atypia. This is a high-grade tumor with a high frequency. Despite this, the corresponding SHAP value indicates a decrease in risk to the model’s prediction, and the case was assigned a low-risk score. Here, the pathway distribution is very similar to the baseline distribution. Case 38 : W0 shows tumor cells forming tubular structures (arrowheads) with intermediate nuclear atypia and hyperchromasia. This feature exhibits an intermediate-grade morphology. The R48 feature shows a slightly downregulated pathway distribution, which was assigned a relatively high SHAP value.
| Feature name | Short description |
|---|---|
| R0: TNFA signaling via NFKB | Genes regulated by NF-kB in response to TNF |
| R1: Hypoxia | Genes up-regulated in response to low oxygen levels (hypoxia) |
| R2: Cholesterol homeostasis | Genes involved in cholesterol homeostasis |
| R3: Mitotic spindle | Genes important for mitotic spindle assembly |
| R4: WNT beta catenin signaling | Genes up-regulated by activation of WNT signaling through accumulation of beta catenin CTNNB1 |
| R5: TGF beta signaling | Genes up-regulated in response to TGFB1 |
Table S3: Transcriptomic pathway features annotations from the Molecular Signatures Database (MSigDB) hallmark gene set collection, part 1.
| Feature name | Short description |
|---|---|
| R25: MTORC1 signaling | Genes up-regulated through activation of mTORC1 complex |
| R26: E2F targets | Genes encoding cell cycle related targets of E2F transcription factors |
| R27: MYC targets v1 | A subgroup of genes regulated by MYC - version 1 (v1) |
| R28: MYC targets v2 | A subgroup of genes regulated by MYC - version 2 (v2) |
| R29: Epithelial mesenchymal transition | Genes defining epithelial-mesenchymal transition, as in wound healing, fibrosis and metastasis |
| R30: Inflammatory response | Genes defining inflammatory response |
Table S4: Transcriptomic pathway features annotations from the Molecular Signatures Database (MSigDB) hallmark gene set collection, part 2.
| Train set | Test set | |||
|---|---|---|---|---|
| Feature | Annotation | Card. | Annotation | Card. |
| W0 | Tumor | 0.0951 | Tumor | 0.062689 |
| W1 | Tissue edge, stroma | 0.0501 | – | 0.000018 |
| W2 | Tissue edge, stroma tumor | 0.0212 | Tissue edge, stroma tumor | 0.000033 |
| W3 | Tumor | 0.0773 | Tumor | 0.009016 |
| W4 | Adipose stroma | 0.0620 | Adipose stroma | 0.098484 |
Table S5: WSI feature annotations for BRCA fold 2, showing the prototype annotation and the cardinality (Card.) of each prototype feature for the train and test set. For a given annotation with multiple morphologies ( and ), a notation of indicates that is strongly dominant relative to , whereas denotes that remains dominant, but is more prevalent compared to with .