Organizations: School of Computer Science and Technology, Tongji University, 1239 Siping Road, Shanghai 200092, China · College of Computer Science and Artificial Intelligence, Fudan University, 2005 Songhu Road, Shanghai 200438, China
Multimodal survival prediction jointly analyzes histopathological whole-slide images (WSIs), genomic profiles, and clinical variables to stratify patients by risk. However, existing approaches often inadequately account for granularity differences across modalities, particularly in spatial organization and feature dimensionality. Within pathology, aggregating WSI patch features without preserving their spatial arrangement can obscure tissue architecture; across modalities, directly concatenating low-dimensional clinical variables with high-dimensional visual and genomic features may limit the contribution of clinical information to prediction. To address these challenges, we propose HGASurv, which introduces a hierarchical WSI encoding module and a clinical-guided graph propagation module for granularity-aware multimodal integration. The hierarchical WSI encoding module maps WSI patch features onto a spatial grid using their original coordinates and applies parallel convolutions to construct representations at patch, local-neighborhood, and regional scales. Genomic signature embeddings then query these scale-specific WSI representations through cross-modal co-attention, producing signature-conditioned morphological features. The clinical-guided graph propagation module uses clinical variables to define patient-similarity edges and propagates fused image-genomic node embeddings with graph attention, integrating clinical information through topology rather than feature concatenation. Across three TCGA cohorts, HGASurv achieved the strongest overall C-index performance, with ablation studies supporting the complementary contributions of the hierarchical WSI encoding and clinical-guided graph propagation modules. This design shows that matching representation and integration mechanisms to modality-specific granularity improves multimodal survival prediction.
Figures & tables
Figure 1: Overview of HGASurv. (1) Hierarchical whole-slide image (WSI) encoding preserves patch spatial organization and represents morphology at patch, local-neighborhood, and regional scales. (2) Genomic signatures guide scale-wise pathology aggregation through co-attention, yielding the fused image–genomic embedding Zi . (3) Clinical variables define patient relationships rather than node features; graph attention propagates Zi over this topology to obtain zi′ for discrete-time survival prediction.
Method
Year
Modality
TCGA-GBMLGG
TCGA-BLCA
TCGA-UCEC
SNN ( Klambauer et al., 2017 )
2017
Genomic
0.805±0.013
0.567±0.067
0.627±0.054
SNNTrans ( Klambauer et al., 2017 )
2017
Genomic
0.804±0.039
0.630±0.021
0.621±0.056
AttnMIL ( Ilse et al., 2018 )
2018
Pathology
0.759±0.040
0.523±0.043
0.665±0.029
DeepAttnMISL ( Yao et al., 2020 )
2020
Pathology
0.736±0.036
0.573±0.034
0.694±0.075
CLAM_SB ( Lu et al., 2021 )
2021
Pathology
0.753±0.041
0.543±0.040
0.674±0.057
CLAM_MB ( Lu et al., 2021 )
2021
Pathology
0.752±0.043
0.542±0.021
0.655±0.045
Table 1: C-index comparison of HGASurv with state-of-the-art deep learning methods on three TCGA cohorts. Values are the mean ± sample standard deviation across five folds. The best result is shown in bold and the second-best is underlined. P, G, and C denote pathology, genomics, and clinical variables, respectively.
Figure 2: Kaplan–Meier survival curves stratified by the final HGASurv out-of-fold risk predictions on (a) TCGA-GBMLGG, (b) TCGA-BLCA, and (c) TCGA-UCEC.
Configuration
Hier. WSI
Clinical Variables
GBMLGG
BLCA
UCEC
Full HGASurv
✓
✓
0.854±0.023
0.692±0.032
0.721±0.098
w/o Clinical Variables
✓
×
0.803±0.045
0.655±0.037
0.616±0.076
w/o Hierarchical WSI
×
✓
0.806±0.032
0.566±0.027
0.660±0.072
w/o both modules
×
×
0.787±0.034
0.587±0.036
0.597±0.107
Table 2: Component ablation of HGASurv across three TCGA cohorts.
Clinical integration
GBMLGG
BLCA
UCEC
No clinical variables
0.803±0.045
0.655±0.037
0.616±0.076
Clinical feature concatenation
0.824±0.046
0.664±0.026
0.666±0.102
Clinical-guided graph propagation (Ours)
0.854±0.023
0.692±0.032
0.721±0.098
Table 3: C-index comparison of clinical integration configurations. Values are the mean ± sample standard deviation across five folds. The best result in each cohort is shown in bold.
Figure 3: t-distributed stochastic neighbor embedding (t-SNE) of pathology, genomic, fused image–genomic, and clinical-graph-refined representations in TCGA-GBMLGG, TCGA-BLCA, and TCGA-UCEC (top to bottom). Colors denote predicted high- and low-risk groups.
Graph topology
GBMLGG
BLCA
UCEC
Self-loop graph
0.822±0.034
0.681±0.029
0.681±0.071
Random graph
0.820±0.042
0.681±0.032
0.666±0.067
Clinical graph (Ours)
0.854±0.023
0.692±0.032
0.721±0.098
Table 4: C-index comparison of patient-graph topology controls. Values are the mean ± sample standard deviation across five folds. The best result in each cohort is shown in bold.
Multimodal survival models can combine complementary prognostic information from whole-slide images and genomic profiles, but effective fusion remains challenging amid external cohort shift and computational complexity. To address these challenges, we propose MIST, multimodal survival prediction with genomic-guided histology attention. MIST represents genomic features as tokens and allows them to query compact foundation-model-derived histology context tokens before survival prediction. This design enriches molecular information with histology context rather than merging separately encoded modalities only at the final stage. Training combines discrete-time survival prediction with genomic feature masking, WSI dropout, and paired WSI-genomics contrastive alignment. Across four external evaluations in colon, renal, lung, and glioblastoma cohorts, MIST improves external C-index over standard fusion baselines in the primary comparisons. These results support genomic-guided histology attention as a compact and effective strategy for multimodal oncology outcome prediction. Our code is available at https://github.com/samiyavuuz/MIST .
Muhammet Sami Yavuz, Sabri Mustafa Kahya, Richard R. Chen +2
AI for Image-Guided Diagnosis and Therapy, School of Medicine and Health, Technical University of Munich (TUM), Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · Technical University of Munich (TUM), Munich, Germany +2
The integrative analysis of histopathological Whole-Slide Images (WSIs) and transcriptomic profiles holds significant promise for cancer survival prediction. However, existing methods typically project multi-modal features directly into a shared latent space without explicit alignment, leading to the entanglement of mismatched morphological cues and molecular signals. Furthermore, current fusion strategies often treat the extreme spatial heterogeneity of WSIs uniformly, lacking mechanisms to adaptively prioritize clinically relevant tissue scales for individual patients. To address these limitations, we propose an Anchor-driven Multi-modal Multi-scale Expert Selection (AM2ES) framework for survival prediction. Specifically, we present an Anchor-driven Multi-modal Fusion (AMF) module, which introduces learnable semantic anchors as cross-modal mediators to bridge the semantic gap by enforcing a structurally regularized alignment between transcriptomic features and multi-scale pathology representations. Built upon this aligned semantic space, we further design a Hierarchical Mixture-of-Experts (H-MoE) selection module to decouple the hierarchical prognostic selection process. Mimicking the pathologist's diagnostic workflow, H-MoE performs (i) Intra-scale Expert Filtering to discriminatively identify salient tumor regions within each magnification, and (ii) Inter-scale Hierarchy Routing to dynamically weight and select the most informative resolution levels. Extensive experiments on multiple TCGA cancer cohorts demonstrate that our AM2ES achieves state-of-the-art performance while offering fine-grained interpretability by visualizing how specific molecular pathways drive the expert routing decisions across tissue scales. The code will be released at https://github.com/taozh2017/AM2ES.
Tao Zhou, Ying Hu, Huazhu Fu +3
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China · Institute of Advanced Intelligence and Computing (IAIC), Agency for Science, Technology and Research (A*STAR), Singapore 138632 · School of Computer Science and Engineering, Southeast University, Nanjing 211189, China +2
We propose a novel multimodal deep learning framework for patient-level survival prediction, which integrates whole-slide histology features, RNA-seq expression profiles, and clinical variables. Our architecture combines an ABMIL module~\cite{ilse2018attention} for slide-level representation with feedforward encoders for RNA and clinical data. These embeddings are then integrated through low-rank bilinear cross-modal fusion~\cite{liu2018efficient} to model conditional interactions across modalities while controlling parameter growth. The model outputs continuous risk scores that are subsequently mapped to survival times using a nonparametric calibration procedure based on the Kaplan--Meier estimator~\cite{kaplan1958nonparametric}. By decomposing multimodal reasoning into independent pairwise interactions, the proposed fusion design promotes structural interpretability and parameter efficiency compared with full tensor and hierarchical fusion strategies. Experiments on the CHIMERA challenge dataset demonstrate improved predictive performance over concatenation-based baselines and competitive generalization on hidden evaluation cohorts. These results indicate that the proposed framework is a promising approach for multimodal survival prediction in HR-NMIBC. The implementation is publicly available at https://github.com/hassancpu/ChimeraChallenge2025_Task_3.
Hassan Keshvarikhojasteh, Josien P. W. Pluim, Mitko Veta
Department of Biomedical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands