Large Language Models (LLMs) have displayed impressive capabilities in handling tasks that require few demonstration examples, making them effective few-shot learners. Despite their potential, LLMs face challenges when it comes to addressing complex real-world tasks that involve multiple modalities or reasoning steps. For example, predicting cancer patients' survival period based on clinical data, cell slides, and genomics poses significant logistical complexities. Although several approaches have been proposed to tackle these challenges, they often fall short in achieving promising performance due to their inability to consider all modalities simultaneously or account for missing modalities, variations in modalities, and the integration of multi-modal data, ultimately compromising their effectiveness. This thesis proposes a novel approach for multi-modal tumor survival prediction to address these limitations. Taking inspiration from recent advancements in LLMs, particularly Mixture of Experts (MoE)-based models, a graph-guided MoE framework is introduced. This framework utilizes a graph structure to manage the predictions effectively and combines multiple models to enhance predictive power. Rather than training a single foundation model for end-to-end survival prediction, the approach leverages a MOE-guided ensemble to manage model callings as tools automatically. By leveraging the strengths of existing models and guiding them through a MOE framework, the aim is to achieve better performance and more accurate predictions in complex real-world tasks. Experiments and analysis on the TCGA-LUAD dataset show improved performance over the individual modal and vanilla ensemble models.
Figures & tables
Figure 2.1 : General Few Shot Learning Approach
Figure 2.2 : General Mixture of Experts Approach
Covariates
Group
Patients (N = 522)
Survival time
902.51 ± 892.15
Vital status
Alive
334 (63.98%)
Dead
188 (36.02%)
Stage
I
280 (53.64%)
II
130 (24.90%)
III
86 (16.48%)
Table 2.1 : Summary of Covariates for Patients (LUAD)
Figure 2.3 : Tissue Slides of LUAD Data
Figure 5Figure 6
Figure 3.3 : Graph based - Mixture of Experts Framework
Figure 3.4 : Graph Attention Network
Clinical Records of TCGA-LUAD (N=470)
Attribute
Description
Race
Race of patients
Age at Index
Age of patients at diagnosis
Gender
Gender of patients
Pathological stage
Cancer stage based on pathology reports
Morphology
Cellular structure and characteristics
Table 4.1 : Details of TCGA-LUAD Dataset
Diagnostic Slide Transformation
Transformation
Details
Slide Transformation
.svs to .png
Mean
(0.485,0.456,0.406)
Standard Deviation
(0.229,0.224,0.225)
Resize
224×224 pixels
Table 4.2 : Diagnostic Slide Transformation
Type
Method
LUAD
Individual Models
Lasso
0.650 ± 0.017
HGCN
0.633 ± 0.020
CTransPath
0.611 ± 0.085
PSSLM
0.642 ± 0.027
Ensemble
Naive
0.665 ± 0.020
Mixture of Experts
Dense
0.703 ± 0.022
Table 5.1 : The results of (CI ↑ ) of Individual Models, comparison with Ensemble and MoE based model.
Figure 5.1 : a) HGCN, b) Lasso
Figure 5.2 : a) CTransPath, b) PSSLM
Figure 5.3 : a) Ensemble, b) Mixture of Experts(MOE)
Multimodal survival prediction, a crucial yet challenging task, demands the integration of multimodal medical data (\eg Whole Slide Images (WSIs) and Genomic Profiles) to achieve accurate prognostic modeling. Given the inherent heterogeneity across modalities, the feature decoupling-fusion paradigm has emerged as a dominant approach. However, these methods have the following shortcomings: (1) fail to reduce the redundant information of modality features before decoupling, which negatively affects the feature decoupling and fusion effect;(2) lack the ability to model the fine-grained relationships of the features and capture the local information interactions between intra- and inter-modality features. To address these issues, we propose a Hierarchical Decoupling-Fusion Mixture-of-Experts (HDMoE) framework with two levels of MoE and Random Feature Reorganization (RFR) modules.In the first-level MoE, shared experts and routed experts are employed to remove redundant information and extract fine-grained specific features within each modality, while the second-level MoE facilitates fine-grained inter-modality feature decoupling. Besides, we design two RFR modules following each level of MoE to finely fuse intra- and inter-modality features, which can help the model capture more fine-grained relationships between modalities. Extensive experimental results on our private Liver Cancer (LC) and three TCGA public datasets confirm the effectiveness of our proposed method. Codes are available at https://github.com/ZJUMAI/HDMoE.
Huayi Wang, Haochao Ying, Yuyang Xu +5
Zhejiang University Hangzhou, China · Xinjiang University Urumqi, China · Hangzhou City University Hangzhou, China +1
We study multimodal survival analysis integrating clinical text, tabular covariates, and genomic profiles using locally deployable large language models (LLMs). As many institutions face tight computational and privacy constraints, this setting motivates the use of lightweight, on-premises models. Our approach jointly estimates calibrated survival probabilities and generates concise, evidence-grounded prognosis text via teacher-student distillation and principled multimodal fusion. On a TCGA cohort, it outperforms standard baselines, avoids reliance on cloud services and associated privacy concerns, and reduces the risk of hallucinated or miscalibrated estimates that can be observed in base LLMs.
Accurate survival prediction from multimodal medical data is essential for precision oncology, yet clinical deployment faces a persistent challenge: modalities are frequently incomplete due to cost constraints, technical limitations, or retrospective data availability. While recent methods attempt to address missing modalities through feature alignment or joint distribution learning, they fundamentally lack explicit modeling of the unique contributions of each modality as opposed to the information derivable from other modalities. We propose MUST (Modality-Specific representation-aware Transformer), a novel framework that explicitly decomposes each modality's representation into modality-specific and cross-modal contextualized components through algebraic constraints in a learned low-rank shared subspace. This decomposition enables precise identification of what information is lost when a modality is absent. For the truly modality-specific information that cannot be inferred from available modalities, we employ conditional latent diffusion models to generate high-quality representations conditioned on recovered shared information and learned structural priors. Extensive experiments on five TCGA cancer datasets demonstrate that MUST achieves state-of-the-art performance with complete data while maintaining robust predictions in both missing pathology and missing genomics conditions, with clinically acceptable inference latency.
Kyungwon Kim, Dosik Hwang
Yonsei University · Korea Institute of Science and Technology