DIGOM: Disentangled Interaction Graphs for Outcome prediction in Multimodal healthcare
Authors: Hakima Laribi, Olivier Lefebvre, Dan Poenaru, Martin Vallières
Organizations: Department of Computer Science, Université de Sherbrooke, 2500 Boulevard de l’Université, Sherbrooke, Québec, J1K 2R1, Canada · Department of Pediatric Surgery, Montreal Children’s Hospital, 1001 Boulevard Décarie, Montreal, Québec, H4A 3J1, Canada · Centre for Outcomes Research and Evaluation (CORE), Research Institute of the McGill University Health Centre, 5252 Boulevard de Maisonneuve Ouest, Montreal, Québec, H4A 3S5, Canada · Medical Physics Unit, Department of Oncology, McGill University, 1001 Boulevard Décarie, Montreal, Québec, H4A 3J1, Canada
Multimodal healthcare data contain relational information that extends beyond individual patient observations. Effective clinical outcome prediction therefore requires not only integrating information from multiple modalities, but also modeling the relationships between them. Standard fusion approaches do not explicitly capture these interactions, while existing graph-based approaches typically model only a subset of them. We propose DIGOM (Disentangled Interaction Graphs for Outcome prediction in Multimodal healthcare), a graph-based framework that jointly models three complementary interaction types: Intra-patient cross-modal, Cross-patient same-modal, and Cross-patient cross-modal interactions. DIGOM learns a sparse multiplex graph with disentangled views, each encoding interactions between pairs of modalities across patients. Discrete adjacency matrices with patient-adaptive connectivity are sampled via a differentiable mechanism, enabling end-to-end learning and allowing the model to select informative relationships without imposing a fixed number of neighbors. A regularization objective maintains overall graph sparsity, while message passing integrates this relational context into patient representations for outcome prediction. We evaluate DIGOM on two multimodal healthcare datasets, MIMIC-IV and NIH-TB, across five clinical prediction tasks. DIGOM achieves the highest AUROC on all five tasks while using fewer connections than existing multi-interaction approaches. Ablation studies reveal that each interaction type's contribution varies across tasks, highlighting the need to adapt relational structures per task. DIGOM also demonstrates greater robustness to missing modalities, suggesting learned relationships provide complementary information when modalities are unavailable. Post-hoc analyses further reveal task- and outcome-specific connectivity patterns in the learned graphs.
Figures & tables
Figure 1: Illustration of the three interaction types modeled by DIGOM and their role in clinical reasoning. (a) Existing methods typically capture only one of Intra-patient cross-modal or Cross-patient same-modal interactions, whereas DIGOM jointly models both along with Cross-patient cross-modal interactions, capturing a richer relational structure across patients and modalities. (b) A clinical case illustrating how a newly admitted patient (green) can benefit from information from a previously seen patient (red) with similar clinical findings who experienced in-hospital mortality. Shared observations help identify the related patient, while their remaining clinical information and outcome provide additional context for understanding the current patient’s condition and anticipating their outcome.
Figure 2: Overview of the DIGOM framework. ( Top ) The patient-adaptive sparse graph learning module produces two latent projections H and Z for each modality, computes pairwise similarity matrices, and samples discrete adjacency matrices to construct a sparse multimodal multi-interaction graph organized into M2 disentangled views, each encoding directed interactions from a source modality to a target modality across all patients. Views are grouped by target modality, such that each group contains all incoming interactions for a given modality. ( Bottom ) The multimodal multi-interaction message passing module propagates information across the learned graph for L layers. At each layer, representations of target modality ml are updated by aggregating neighboring representations from all M source modality views and combining them with the node’s own representation. The resulting enriched modality representations are concatenated and passed through a fully connected layer to produce the final outcome prediction Y^ . The model is jointly optimized using the cross-entropy loss LCE , the soft-HGR correlation loss LsHGR , and the density regularization loss Ldensity .
Method
Intra-patient
Cross-patient
Cross-patient
cross-modal
same-modal
cross-modal
Early Fusion
✗
✗
✗
Joint Fusion
✗
✗
✗
Late Fusion
✗
✗
✗
STGNN
✗
✓
✗
SGNN
✗
✓
✗
Table 1: Interaction types modeled by the considered baselines and DIGOM. ✓ indicates that the interaction type is explicitly modeled, ∼ indicates that it is captured but not modeled separately, and ✗ indicates that it is not modeled.
Figure 3: Evaluation protocol of each model based on nested five-fold cross-validation. The inner loops are used for hyperparameter optimization, and the same data splits are used for all models.
MIMIC-IV
NIH-TB
48h Mortality
48h LOS
30d Readmission
TB Outcome
TB Mortality
Early Fusion
0.870±0.02∗
0.802±0.02†
0.675±0.01∗
0.726±0.01∗
0.734±0.01∗
Joint Fusion
0.871±0.01∗
0.813±0.02
0.670±0.02∗
0.745±0.01†
0.741±0.02∗
Late Fusion
0.865±0.01∗
0.789±0.02∗
0.672±0.01∗
0.728±0.02∗
0.745±0.02†
STGNN
0.887±0.01∗
0.814±0.01
0.675±0.01
–
–
SGNN
0.883±0.01∗
0.813±0.01
0.683±0.01
0.735±0.01∗
0.766±0.02
Table 2: AUROC comparison across the five prediction tasks. Results are reported as mean ± standard deviation across the five outer testing folds. For the multiclass tuberculosis treatment outcome, AUROC is macro-averaged over the five classes. The best-performing model is shown in bold , and the second-best is underlined . Statistical significance was assessed using a one-sided Wilcoxon signed-rank test [ 50 ] to determine whether each baseline significantly underperformed DIGOM. ∗ and † indicate p-value<0.05 and 0.05≤p-value<0.1 , respectively. From left to right, the tasks are 48-hour ICU mortality risk, 48-hour ICU length of stay, 30-day readmission risk, tuberculosis treatment outcome, and mortality risk after tuberculosis treatment.
48M
48LOS
30DR
TBM
Early Fusion
0.053
0.014
0.011
0.025
Joint Fusion
0.057
−0.005
0.018
0.043
Late Fusion
0.051
0.035
0.027
0.007
STGNN
0.024
−0.001
0.008
–
SGNN
0.018
−0.009
0.009
−0.017
MMGL
0.304
0.004
0.036
0.019
Table 3: Net Reclassification Improvement (NRI) of DIGOM over each baseline. NRI measures whether DIGOM more appropriately reclassifies patients across risk categories compared with each baseline. Positive NRI values indicate improved overall reclassification, whereas negative values indicate worse reclassification. Since NRI is defined for binary classification, the multiclass tuberculosis treatment outcome is not included in this analysis. The decision threshold is selected to maximize the difference between sensitivity and the false positive rate on the training set. 48M : 48-hour ICU mortality risk, 48LOS : 48-hour ICU length of stay, 30DR : 30-day readmission risk, TBM : mortality risk after tuberculosis treatment.
Figure 4: AUROC of DIGOM trained with a single interaction type at a time, and with no interactions, across all five prediction tasks. Each bar reports the mean AUROC over five cross-validation folds, with error bars indicating the standard deviation. The dashed teal line marks the performance of the proposed DIGOM model, in which all three interaction types are learned jointly, and serves as a reference. The contribution of each interaction type varied across tasks, with no single type consistently providing the greatest benefit.
Figure 5: AUROC of the selected baselines and DIGOM when one modality is masked at a time at inference time, across all five prediction tasks. For each masked modality, continuous columns are imputed using the column mean and categorical columns using the column median. Each spoke of the radar chart corresponds to a modality, and each line represents a model evaluated after masking that modality. Graph-based methods modeling multiple interaction types generally maintained higher performance under missing modalities than conventional fusion methods or methods restricted to a single interaction type.
Figure 6: Distribution of the average number of connections per patient in the graphs learned by DIGOM, with and without the density loss, across four prediction tasks. The distribution without the density loss could not be obtained for 30-day readmission risk due to the memory cost of constructing a dense graph for its large cohort; its distribution with the density loss is shown separately in Figure B.1 . The density loss produced substantially sparser graphs across all tasks, while Table B.2 shows that predictive performance remained largely unaffected, indicating that many connections can be pruned without loss of useful information.
Figure 7: Average connectivity between source and target modality pairs in the graphs learned by DIGOM across all five prediction tasks, averaged over the five test folds. Each cell reports the fraction of patients connected from the source modality to the target modality. Connectivity patterns varied substantially across tasks, with higher connectivity concentrated on CT scan, demographic, and clinical modalities for the tuberculosis tasks, and on radiology notes for 48-hour ICU mortality, reflecting task-specific differences in how the model retrieves relational information across patients and modalities.
Figure 8: Average connection probability between source and target modality pairs in the graphs learned by DIGOM across all five prediction tasks, averaged over the five test folds. Each cell reports the mean probability assigned to the edges selected between the source and target modality prior to sampling. Modality pairs with higher connectivity tended to exhibit lower connection probabilities, suggesting that the model selects connections with greater confidence when it can identify a small targeted set of informative relationships, and resorts to a broader but less certain set when the relevant connections are harder to identify.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
48M
48LOS
30DR
Temporal
0.663±0.02
0.692±0.01
0.611±0.01
Average
0.661±0.02
0.680±0.03
0.574±0.01
Weighted average
0.625±0.03
0.700±0.01
0.578±0.02
First
0.632±0.03
0.629±0.02
0.564±0.01
Last
0.639±0.03
0.688±0.02
0.576±0.01
Appendix
Table A.1 : AUROC comparison across different aggregation strategies for chest X-ray imaging across the three prediction tasks in MIMIC-IV. The best strategy is shown in bold , and the second-best is underlined . Results are reported as mean ± standard deviation across the five inner testing folds. Temporal aggregates the image embeddings using a GRU, Average computes their unweighted average, Weighted average assigns higher weights to embeddings of images closer to the prediction time, and First and Last use only the first and last embeddings, respectively. 48M : 48-hour ICU mortality risk, 48LOS : 48-hour ICU length of stay, 30DR : 30-day readmission risk.
48M
48LOS
30DR
Temporal
0.716±0.02
0.747±0.02
0.641±0.01
Average
0.718±0.01
0.751±0.01
0.634±0.01
Weighted average
0.697±0.03
0.736±0.02
0.633±0.01
First
0.634±0.03
0.629±0.01
0.613±0.01
Last
0.677±0.01
0.702±0.02
0.603±0.02
Appendix
Table A.2 : AUROC comparison across different aggregation strategies for radiology notes across the three prediction tasks in MIMIC-IV. The best strategy is shown in bold , and the second-best is underlined . Results are reported as mean ± standard deviation across the five inner testing folds. Temporal aggregates the note embeddings using a GRU, Average computes their unweighted average, Weighted average assigns higher weights to embeddings of notes closer to the prediction time, and First and Last use only the first and last embeddings, respectively. 48M : 48-hour ICU mortality risk, 48LOS : 48-hour ICU length of stay, 30DR : 30-day readmission risk.
48h Mortality
48h LOS
30d Readmission
TB Outcome
TB Mortality
DIGOM w/ density loss
7.42%
4.06%
3.01%
9.65%
4.99%
DIGOM w/o density loss
50.62%
52.63%
–
50.03%
39.82%
MaxCorrMGNN top- k
5.94%
6.33%
0.99%
5.01%
4.23%
MaxCorrMGNN
–
–
–
62.68%
63.48%
Appendix
Table B.1 : Average connectivity rate for DIGOM and MaxCorrMGNN across prediction tasks, with and without graph density constraints. For MaxCorrMGNN, connectivity is reported as the maximum between the two propagation graphs A⋅C and C⋅A , where A captures Cross-patient same-modal similarities and C captures cross-modal similarities. Note that the top- k sparsity constraint is applied independently to A and C . Thus, the products A⋅C and C⋅A yield a different effective sparsity, which is why the reported connectivity for MaxCorrMGNN with top- k does not directly reflect k . “–” indicates that the condition could not be evaluated due to memory constraints from the large cohort size.
Figure B.1 : Distribution of the average number of connections per patient in the graphs learned by DIGOM for 30-day readmission risk prediction. The density loss effectively constrains graph connectivity despite the large cohort size, producing a sparse graph concentrated around the target density ρ .
w/o density loss
with density loss
48M
0.895±0.01
0.901±0.01
48LOS
0.816±0.02
0.816±0.02
TBO
0.752±0.01
0.761±0.01
TBM
0.758±0.04
0.771±0.02
Appendix
Table B.2 : Effect of the density loss in DIGOM on AUROC. Results are reported as mean ± standard deviation across the five outer testing folds. The 30-day readmission task could not be evaluated without the density loss due to the prohibitive memory cost of constructing a dense graph for its large cohort. 48M : 48-hour ICU mortality risk, 48LOS : 48-hour ICU length of stay, TBO : tuberculosis treatment outcome, and TBM : mortality risk after tuberculosis treatment.
Figure B.2 : Average connectivity between source and target modality pairs in the graphs learned by DIGOM, reported separately for the negative and positive classes across the four binary prediction tasks, averaged over the five test folds. For 48-hour ICU mortality and tuberculosis mortality risk, connectivity is higher in the positive class for the most connected modality pairs, suggesting that the model retrieves information from a broader set of patients when predicting the more clinically severe outcome.
Figure B.3 : Average connectivity between source and target modality pairs in the graphs learned by DIGOM, reported separately for each of the five tuberculosis treatment outcome classes, averaged over the five test folds. Connectivity is highest for the Died, Completed, and Cured classes, particularly for CT scan and demographic modalities, suggesting that the model retrieves information from a broader set of patients when predicting these outcomes.
Model
48h Mortality
48h LOS
30d Readmission
TB Outcome
TB Mortality
DIGOM w/ density loss
13.77
11.55
105.97
0.83
0.78
DIGOM w/o density loss
44.32
45.91
–
1.91
1.98
MaxCorrMGNN top- k
11.46
12.13
32.78
0.65
0.56
MaxCorrMGNN
–
–
–
5.89
6.01
Appendix
Table B.3 : Average inference time (in seconds) on the test set for DIGOM and MaxCorrMGNN across prediction tasks, with and without graph density constraints. Times are averaged across the five outer test folds. “–” indicates that the condition could not be evaluated due to memory constraints from the large cohort size.
Model
TB Outcome
TB Mortality
Joint Fusion
0.702±0.01
–
Late Fusion
–
0.723±0.02
SGNN
0.721±0.01
0.734±0.02
MMGL
0.705±0.01
0.718±0.02
MaxCorrMGNN
0.723±0.01
0.707±0.02
DIGOM (ours)
0.728±0.01
0.727±0.02
Appendix
Table B.4 : AUROC on the tuberculosis tasks when models are retrained without the demographic modality. Only the models included in the missing-modality analysis are reported; conventional fusion is therefore represented by its best-performing variant for each task. Results are reported as mean ± standard deviation across the five outer testing folds.
Hyperparameter
Values
Learning rate
[0.0001,0.01]
Positive class weight
[0.1,0.9]
GRU hidden size
{64,128,256,512,1024}
ρ (target density)
{0.01,0.02,0.03,0.04,0.05} ∗
λ1
0.001
λ2
[0.01,0.5]
Appendix
Table C.1 : Hyperparameter search ranges used by DIGOM. Ranges are shared across tasks unless otherwise noted.
Model
Hyperparameter
Values
Common to all models
Learning rate
[0.0001,0.01]
Positive class weight †
[0.1,0.9]
Early Fusion
Dropout
[0,0.2]
GRU hidden size
{64,128,256,512,1024}
MLP layers
[1,4]
MLP hidden size
{64,128,256,512,1024}
Appendix
Table C.2 : Hyperparameter search ranges used for each baseline model. Ranges are shared across tasks unless otherwise noted.
Electronic health record (EHR) data are inherently multimodal, and leveraging multiple modalities can improve predictive performance. However, most existing approaches rely on deep fusion, which obscures how individual modalities contribute to predictions and limits the interpretability of multimodal reasoning. We propose an explicit multimodal routing framework for clinical prediction that enables interpretable, robust, and auditable reasoning across three EHR modalities: structured longitudinal variables (L), clinical notes (N), and chest X-rays (I). Our model constructs discrete unimodal, directional bimodal, and trimodal routes to capture both individual modality signals and asymmetric cross-modal interactions. To audit multimodal reasoning and assess robustness, we introduce inference-time route masking, which simulates missing modalities and reweights the remaining routes without retraining. We analyze changes in performance and routing weights under these scenarios to understand model decision-making. We evaluate our framework on multi-label phenotype prediction (K = 25) and binary ICU mortality prediction using trimodal patient stays from MIMIC-IV, revealing systematic differences in modality reliance across clinical condition groups. Overall, our framework offers a transparent, auditable, and practical approach to multimodal clinical prediction, providing interpretability, robustness, and insights into how different data sources drive model decisions.
Multimodal graphs, where nodes contain heterogeneous features such as images and text, are increasingly common in real-world applications. Effectively learning on such graphs requires both adaptive intra-modal message passing and efficient inter-modal aggregation. However, most existing approaches to multimodal graph learning are typically extended from conventional graph neural networks and rely on static structures or dense attention, which limit flexibility and expressive node embedding learning. In this paper, we propose a novel multimodal graph representation learning framework with Dynamic information Pathways (DiP). By introducing modality-specific pseudo nodes, DiP enables dynamic message routing within each modality via proximity-guided pseudo-node interactions and captures inter-modality dependence through efficient information pathways in a shared state space. This design achieves adaptive, expressive, and sparse message propagation across modalities with linear complexity. We conduct the link prediction and node classification tasks to evaluate performance and carry out full experimental analyses. Extensive experiments across multiple benchmarks demonstrate that DiP consistently outperforms baselines.
Xiaobin Hong, Mingkai Lin, Xiaoli Wang +2
State Key Laboratory for Novel Software Technology, Nanjing University · Nanjing Forest University, College of Information Science and Technology · Zhejiang Provincial Seaport Investment & Operation Group Co. Ltd
Integrating diverse data modalities --- such as clinical notes, laboratory results, and medical imaging --- is essential for advancing clinical decision-making. While Large Language Models (LLMs) have shown remarkable performance in processing unstructured clinical text, their limited capacity to incorporate non-text modalities hinders their broader utility in healthcare applications. Here, we introduce M2G-LLM (Multimodal MedGraph-LLM), a novel framework that enhances LLMs with multimodal integration and alignment via Graph Neural Networks (GNNs). Our approach models temporal relationships between patient visits, propagates information across clinically similar patients, and aligns heterogeneous data sources to construct enriched multimodal context vectors. These vectors are injected into the intermediate layers of the LLM, enabling joint reasoning over textual and non-textual modalities. We evaluate M2G-LLM on the MIMIC-IV and MIMIC-CXR datasets, demonstrating improvements in clinical prediction tasks over strong baseline models. Our results highlight the promise of combining the language understanding of LLMs with the relational reasoning capabilities of GNNs for comprehensive, multimodal healthcare analysis.
Inyoung Choi, Sukwon Yun, Jiayi Xin +3
University of Pennsylvania, Philadelphia, PA, USA · University of North Carolina at Chapel Hill, Chapel Hill, NC, USA