Trimodal Deep Learning for Glioma Survival Prediction: A Feasibility Study Integrating Histopathology, Gene Expression, and MRI

Iain Swift; JingHua Ye

cs.CVMar 31, 2026

Trimodal Deep Learning for Glioma Survival Prediction: A Feasibility Study Integrating Histopathology, Gene Expression, and MRI

Authors: Iain Swift, JingHua Ye

Organizations: Department of Computer Science, Munster Technological University, Cork, Ireland

Abstract

Multimodal deep learning has improved prognostic accuracy for brain tumours by integrating histopathology and genomic data, yet the contribution of volumetric MRI within unified survival frameworks remains unexplored. This pilot study extends a bimodal framework by incorporating Fluid Attenuated Inversion Recovery (FLAIR) MRI from BraTS2021 as a third modality. Using the TCGA-GBMLGG cohort (664 patients), we evaluate three unimodal models, nine bimodal configurations, and three trimodal configurations across early, late, and joint fusion strategies. In this small cohort setting, trimodal early fusion achieves an exploratory Composite Score (CS = 0.854), with a controlled $Δ$ CS of +0.011 over the bimodal baseline on identical patients, though this difference is not statistically significant (p = 0.250, permutation test). MRI achieves reasonable unimodal discrimination (CS = 0.755) but does not substantially improve bimodal pairs, while providing measurable uplift in the three-way combination. All MRI containing experiments are constrained to 19 test patients, yielding wide bootstrap confidence intervals (e.g. [0.400,1.000]) that preclude definitive conclusions. These findings provide preliminary evidence that a third imaging modality may add prognostic value even with limited sample sizes, and that additional modalities require sufficient multimodal context to contribute effectively.

Figures & tables

Experiment	Train	Test	GBM%
FFPE Only	465	117	36.2% §
RNA Only	389	120	34.2% §
MRI Only	143	19	68.5%
FFPE + RNA	376	114	15.7%
RNA + MRI	47	19	25.5%
FFPE + MRI	58	19	15.8%

TABLE I: Data Splits by Modality Combination

Model	Fusion	Test CS
Unimodal
FFPE	—	0.773
RNA	—	0.836
MRI	—	0.755 ‡
Bimodal
FFPE + RNA	Early	0.806

TABLE II: Composite Score (CS) Across All Configurations

† Restricted to same 47/19 patient split as trimodal.
Baseline	+ MRI	$\Delta$ CS	Note
FFPE (0.773)	FFPE+MRI (0.781)	+0.008	Uncontrolled ‡
RNA (0.836)	RNA+MRI (0.808)	$-$ 0.028	Uncontrolled ‡
FFPE+RNA (0.806)	Trimodal (0.854)	+0.048	Uncontrolled
FFPE+RNA † (0.843)	Trimodal (0.854)	+0.011	Controlled
‡ Different patient subsets and sample sizes; $\Delta$ CS confounded

TABLE III: Impact of Adding MRI (Early Fusion). The controlled

\Delta

CS comparison uses identical patients to isolate MRI’s contribution.

Fig. 1: Composite Score (CS) across all configurations, grouped by modality combination and fusion strategy. Hatched bars (

\ddagger

) indicate experiments with

n

\leq

19 test patients where IBS is imprecise. Trimodal early fusion achieves the highest CS (0.854).

Model	Test CI	Parameters	Train CI
Bilinear	0.819	537K	0.979
Cross-Attention	0.814	1.84M	0.975
Gated	0.807	3.18M	0.954
Baseline MLP	0.801	$\sim$ 4.2M	$\sim$ 0.95

TABLE IV: Attention-Based Fusion Models (FFPE + RNA Early Fusion)

Cancer Survival Prediction

Cancer survival prediction estimates patient outcomes and time to events from clinical, molecular, and medical-imaging data.

See all 92 papers

Jun 12, 2026

Trimodal Glioma Representation Alignment via Volumetric Contrastive Learning

Glioma grading and survival prediction require the integration of heterogeneous information collected at different spatial and biological scales. Histopathology describes tissue morphology, mRNA expression captures molecular activity, and magnetic resonance imaging provides a non-invasive view of tumor extent and radiological heterogeneity. Existing glioma prognosis models often combine only two of these sources, while their alignment objectives remain mostly pairwise. This paper introduces GLORIA, a novel trimodal framework for GLioma Omics - Radiology - hIstopathology Alignment. GLORIA processes whole-slide image regions, gene-expression profiles, and 3D MRI volumes through modality-specific encoders, projects them into a shared latent space, and aligns them with a Gramian contrastive loss that measures the volume spanned by the three modality embeddings. The aligned representations are fused through a cross-modal gating module and optimized jointly for three-class glioma grading and overall survival prediction. We evaluate GLORIA on a matched TCGA-GBM/LGG and BraTS21 cohort, comprising 132 patients with all three modalities. On the shared trimodal test set, GLORIA improves over the bimodal WSI-mRNA baseline in all the metrics considered.

Denise Marini, Eleonora Grassucci, Danilo Comminiello

Dept. of Information Engineering, Electronics, and Telecommunications, Sapienza University of Rome, Italy

Cancer Survival Prediction Cross-Modal Alignment Oncology Multimodal Survival Analysis Multimodal Contrastive Learning Multimodal Fusion

Ask PDF

Jun 9, 2026

Multimodal Brain Tumour Classification Using Feature Fusion

Clinicians diagnose brain tumors by synthesizing patient symptoms, medical history, and quantitative imaging data from modalities such as MRI and CT scans into a unified clinical judgement. However, most deep learning models rely on MRI/CT images alone, failing to replicate the clinicians multimodal reasoning. We explore a two-branch multimodal network combining raw MRI scans with 91 extracted radiomic features (intensity, texture, shape, and boundary descriptors) to classify brain tumors into glioma, meningioma, pituitary, and no-tumor. A pre-trained CNN backbone encodes the image stream, whereas a dedicated MLP encodes the radiomic stream. Both streams are fused via concatenation, gated, or bidirectional cross-modal attention strategies. Across nine experimental runs on a balanced 7,200 image dataset, all multimodal configurations outperform unimodal baselines with gated fusion achieving the best accuracy of 96.13%.

Wajih ul Islam, Muhammad Yaqoob, Javed Ali Khan +1

School of Physics, Engineering and Computer Science, University of Hertfordshire, UK

Multimodal Classification Medical Image Classification Oncology Brain Tumor Classification Multimodal Fusion

Ask PDF

Mar 31, 2026

Quantifying Cross-Modal Interactions in Multimodal Glioma Survival Prediction via InterSHAP: Evidence for Additive Signal Integration

Multimodal deep learning for cancer prognosis is commonly assumed to benefit from synergistic cross-modal interactions, yet this assumption has not been directly tested in survival prediction settings. This work adapts InterSHAP, a Shapley interaction index-based metric, from classification to Cox proportional hazards models and applies it to quantify cross-modal interactions in glioma survival prediction. Using TCGA-GBM and TCGA-LGG data (n=575), we evaluate four fusion architectures combining whole-slide image (WSI) and RNA-seq features. Our central finding is an inverse relationship between predictive performance and measured interaction: architectures achieving superior discrimination (C-index 0.64

\to

0.82) exhibit equivalent or lower cross-modal interaction (4.8%

\to

3.0%). Variance decomposition reveals stable additive contributions across all architectures (WSI

{\approx}

40%, RNA

{\approx}

55%, Interaction

{\approx}

4%), indicating that performance gains arise from complementary signal aggregation rather than learned synergy. These findings provide a practical model auditing tool for comparing fusion strategies, reframe the role of architectural complexity in multimodal fusion, and have implications for privacy-preserving federated deployment.

Iain Swift, JingHua Ye, Ruairi O'Reilly

Department of Computer Science, Munster Technological University, Cork, Ireland

Multimodal Fusion Multimodal Survival Analysis Cancer Survival Prediction Multimodal Model Interpretability Shapley Value Attribution

Ask PDF

Abstract

Cancer Survival Prediction

Explore similar work

Trimodal Glioma Representation Alignment via Volumetric Contrastive Learning

Multimodal Brain Tumour Classification Using Feature Fusion

Quantifying Cross-Modal Interactions in Multimodal Glioma Survival Prediction via InterSHAP: Evidence for Additive Signal Integration