PRIME: Prototype-Driven Multimodal Pretraining for Cancer Prognosis with Missing Modalities

Kai Yu; Shuang Zhou; Yiran Song; Zaifu Zhan; Jie Peng; Kaixiong Zhou; Tianlong Chen; Feng Xie

cs.LGApr 5, 2026

PRIME: Prototype-Driven Multimodal Pretraining for Cancer Prognosis with Missing Modalities

Authors: Kai Yu, Shuang Zhou, Yiran Song, Zaifu Zhan, Jie Peng, Kaixiong Zhou, Tianlong Chen, Feng Xie,

Organizations: Division of Computational Health Sciences, Department of Surgery, University of Minnesota, Minneapolis, MN, USA. · University of North Carolina at Chapel Hill, Chapel Hill, NC, USA. · North Carolina State University, Raleigh, NC, USA. · Centre for Innovation & Precision Eye Health, Department of Ophthalmology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore. ·

Abstract

Multimodal self-supervised pretraining offers a promising route to cancer prognosis by integrating histopathology whole-slide images, gene expression, and pathology reports, yet most existing approaches require fully paired and complete inputs. In practice, clinical cohorts are fragmented and often miss one or more modalities, limiting both supervised fusion and scalable multimodal pretraining. We propose PRIME, a missing-aware multimodal self-supervised pretraining framework that learns robust and transferable representations from partially observed cohorts. PRIME maps heterogeneous modality embeddings into a unified token space and introduces a shared prototype memory bank for latent-space semantic imputation via patient-level consensus retrieval, producing structurally aligned tokens without reconstructing raw signals. Two complementary pretraining objectives: inter-modality alignment and post-fusion consistency under structured missingness augmentation, jointly learn representations that remain predictive under arbitrary modality subsets. We evaluate PRIME on The Cancer Genome Atlas with label-free pretraining on 32 cancer types and downstream 5-fold evaluation on five cohorts across overall survival prediction, 3-year mortality classification, and 3-year recurrence classification. PRIME achieves the best macro-average performance among all compared methods, reaching 0.653 C-index, 0.689 AUROC, and 0.637 AUROC on the three tasks, respectively, while improving robustness under test-time missingness and supporting parameter-efficient and label-efficient adaptation. These results support missing-aware multimodal pretraining as a practical strategy for prognosis modeling in fragmented clinical data settings.

Figures & tables

Fig. 1: Overview of the proposed PRIME framework for multimodal cancer prognosis with missing modalities. Left (Sample Pre-processing): Frozen encoders extract token embeddings from WSIs, RNA-seq, and pathology reports. Middle (Missing Modality Imputation): In a unified token space, a shared sequence-level prototype memory bank imputes missing modalities via patient-level consensus retrieval, followed by modality-specific refinement. Right (Missing-aware Pretraining): Two complementary objectives, inter-modality contrastive alignment and post-fusion consistency under modality/token dropout with prototype mixtures, jointly drive self-supervised pretraining.

Cancer	$N_{\mathrm{tri}}$	OS	3yOS	3yRec
UCEC	499	80 / 419	62 / 203	99 / 176
LUAD	436	157 / 279	118 / 114	159 / 86
LGG	434	95 / 339	64 / 126	121 / 96
BRCA	855	116 / 739	56 / 357	69 / 340
BLCA	346	161 / 185	147 / 74	145 / 61

TABLE I: Per-cancer cohort statistics for downstream evaluation using tri-modal complete cases.

N_{\mathrm{tri}}

denotes the tri-modal cohort size. OS entries are reported as event/cens . 3yOS and 3yRec entries are reported as pos/neg after excluding patients censored before 3 years; for 3yRec, excluded cases are those censored before 3 years without a recorded event.

Methods	Modalities			UCEC	LUAD	LGG	BRCA	BLCA	Avg
Methods	Img	RNA	Text	UCEC	LUAD	LGG	BRCA	BLCA	Avg
Img-only	✓			0.668 $\pm$ 0.083	0.599 $\pm$ 0.056	0.670 $\pm$ 0.100	0.615 $\pm$ 0.087	0.547 $\pm$ 0.060	0.620 $\pm$ 0.052
RNA-only		✓		0.598 $\pm$ 0.094	0.510 $\pm$ 0.057	0.764 $\pm$ 0.056	0.498 $\pm$ 0.065	0.526 $\pm$ 0.043	0.579 $\pm$ 0.110
Text-only			✓	0.676 $\pm$ 0.066	0.553 $\pm$ 0.028	0.703 $\pm$ 0.097	0.556 $\pm$ 0.030	0.532 $\pm$ 0.025	0.604 $\pm$ 0.079
ABMIL [ 34 ]	✓			0.637 $\pm$ 0.049	0.604 $\pm$ 0.066	0.692 $\pm$ 0.056	0.617 $\pm$ 0.092	0.537 $\pm$ 0.035	0.617 $\pm$ 0.056
SNN [ 35 ]		✓		0.571 $\pm$ 0.070	0.520 $\pm$ 0.065	0.719 $\pm$ 0.073	0.466 $\pm$ 0.103	0.536 $\pm$ 0.053	0.563 $\pm$ 0.096

TABLE II: Task-1 overall survival performance (C-index) on five cancer cohorts. Values are mean

\pm

std over 5-fold CV within each cohort; Avg denotes the mean

\pm

std across cohorts. Best and second-best results are highlighted in bold and underline .

Methods	Task-2 Avg	Task-3 Avg
Img-only	0.640 $\pm$ 0.070	0.572 $\pm$ 0.087
RNA-only	0.587 $\pm$ 0.099	0.561 $\pm$ 0.117
Text-only	0.619 $\pm$ 0.070	0.608 $\pm$ 0.071
ABMIL [ 34 ]	0.638 $\pm$ 0.064	0.591 $\pm$ 0.080
SNN [ 35 ]	0.594 $\pm$ 0.110	0.568 $\pm$ 0.107
Early	0.655 $\pm$ 0.082	0.609 $\pm$ 0.060

TABLE III: Average AUROC across five cancer cohorts for Task-2 (3-year mortality) and Task-3 (3-year recurrence). Values are mean

\pm

std across cohorts. Best and second-best methods are highlighted within each task.

Fig. 2: Kaplan–Meier survival curves based on predicted risk scores for five TCGA cohorts. Each panel reports the log-rank $p$ -value and the hazard ratio (HR) from a univariate Cox model comparing high-risk vs. low-risk groups.

Task	Method	Full	LI	LR	LT	OI	OR	OT
Task-1	Scratch (FT)	0.623 $\pm$ 0.072	0.608 $\pm$ 0.069	0.631 $\pm$ 0.068	0.607 $\pm$ 0.051	0.607 $\pm$ 0.050	0.540 $\pm$ 0.085	0.606 $\pm$ 0.066
	Pretrained (LP)	0.653 $\pm$ 0.086	0.612 $\pm$ 0.086	0.641 $\pm$ 0.086	0.619 $\pm$ 0.090	0.603 $\pm$ 0.058	0.564 $\pm$ 0.084	0.611 $\pm$ 0.077
	Pretrained (LP+Missing)	0.653 $\pm$ 0.089	0.623 $\pm$ 0.085	0.639 $\pm$ 0.085	0.636 $\pm$ 0.087	0.611 $\pm$ 0.055	0.571 $\pm$ 0.063	0.610 $\pm$ 0.089
Task-2	Scratch (FT)	0.656 $\pm$ 0.064	0.611 $\pm$ 0.052	0.666 $\pm$ 0.064	0.621 $\pm$ 0.073	0.620 $\pm$ 0.072	0.486 $\pm$ 0.051	0.611 $\pm$ 0.052
	Pretrained (LP)	0.689 $\pm$ 0.110	0.670 $\pm$ 0.118	0.671 $\pm$ 0.097	0.665 $\pm$ 0.088	0.638 $\pm$ 0.077	0.618 $\pm$ 0.112	0.641 $\pm$ 0.101
	Pretrained (LP+Missing)	0.679 $\pm$ 0.117	0.679 $\pm$ 0.118	0.670 $\pm$ 0.104	0.676 $\pm$ 0.089	0.642 $\pm$ 0.077	0.627 $\pm$ 0.118	0.646 $\pm$ 0.107

TABLE IV: Robustness to missing modalities on three downstream tasks. Models are trained on full tri-modal data, with missingness introduced only at test time. Full denotes all modalities; LI/LR/LT and OI/OR/OT denote leave-one-out and only-one settings for Image/RNA/Text, respectively. Values are mean

\pm

std over 5-fold CV.

Fig. 3: Label-efficient downstream transfer performance on Task-1 (overall survival; C-index). The x-axis denotes the fraction of labeled training samples retained from the full training split. Left: unimodal baselines. Right: multimodal fusion baselines.

Variant	Components / Objective				Downstream performance
Variant	Pretrained Data (missing+full modality)	Prototypes	$L_{\mathrm{align}}$	$L_{\mathrm{fusion}}$	T1 (C-index)	T2 (AUROC)	T3 (AUROC)
w/o prototypes (token=0)	$\checkmark$	$\times$	$\checkmark$	$\checkmark$	0.646 $\pm$ 0.082	0.669 $\pm$ 0.123	0.595 $\pm$ 0.108
w/o missing modality	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	0.640 $\pm$ 0.074	0.656 $\pm$ 0.116	0.629 $\pm$ 0.087
Full method (Pretrained LP)	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	0.653 $\pm$ 0.086	0.689 $\pm$ 0.110	0.629 $\pm$ 0.098
w/o $L_{\mathrm{align}}$ ( $\lambda=0$ )	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	0.614 $\pm$ 0.072	0.625 $\pm$ 0.087	0.626 $\pm$ 0.065
w/o $L_{\mathrm{fusion}}$ ( $\lambda=1$ )	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	0.589 $\pm$ 0.068	0.624 $\pm$ 0.129	0.609 $\pm$ 0.093

TABLE V: Ablation on key components of our pretraining framework. All variants use the same downstream protocol (pretrained + LP) and are evaluated on the five-cohort average (mean

\pm

std over 5-fold CV). Full method uses missing-aware pretraining, prototypes, and the combined objective

\mathcal{L}=\lambda\mathcal{L}_{\mathrm{align}}+(1-\lambda)\mathcal{L}_{\mathrm{fusion}}

. “w/o

\mathcal{L}_{\mathrm{align}}

” corresponds to

\lambda=0

, and “w/o

\mathcal{L}_{\mathrm{fusion}}

” corresponds to

\lambda=1

Fig. 4: Sensitivity of the model to prototype bank size $K_{c}$ (left) and loss weight coefficient $\lambda$ (right).

Multimodal Pretraining

Multimodal pretraining learns reusable representations by training models on paired or mixed data from multiple modalities, such as text, images, audio, and signals.

See all 182 papers

MIST: Multimodal Survival Prediction with Genomic-Guided Histology Attention
Sep 18, 2026Muhammet Sami Yavuz, Sabri Mustafa Kahya, Richard R. Chen +2Cancer Survival Prediction Multimodal Survival Analysis
AI for Image-Guided Diagnosis and Therapy, School of Medicine and Health, Technical University of Munich (TUM), Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · Technical University of Munich (TUM), Munich, Germany +2
ProtoPathway: Biologically Structured Prototype-Pathway Fusion for Multimodal Cancer Survival Prediction
May 20, 2026Amaya Gallagher-Syed, Costantino Pitzalis, Myles J. Lewis +2Cancer Survival Prediction Oncology
Queen Mary University of London
Together, Then Apart: Balancing Alignment and Distinctiveness for Multimodal Survival Analysis
Nov 22, 2025Wenjing Liu, Qin Ren, Wen Zhang +2Cancer Survival Prediction Survival Analysis
Stony Brook University, Stony Brook, NY, USA · Johns Hopkins University, Baltimore, MD, USA · Brookhaven National Laboratory, Upton, NY, USA

Sep 18, 2026J/K move · Enter open · S save

MIST: Multimodal Survival Prediction with Genomic-Guided Histology Attention

Muhammet Sami Yavuz, Sabri Mustafa Kahya, Richard R. Chen, Jana Lipkova+1

AI for Image-Guided Diagnosis and Therapy, School of Medicine and Health, Technical University of Munich (TUM), Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · Technical University of Munich (TUM), Munich, Germany+2

Cancer Survival Prediction Multimodal Survival Analysis Multimodal Fusion

Open paper AskPDF arXiv

Multimodal survival models can combine complementary prognostic information from whole-slide images and genomic profiles, but effective fusion remains challenging amid external cohort shift and computational complexity. To address these challenges, we propose MIST, multimodal survival prediction with genomic-guided histology attention. MIST represents genomic features as tokens and allows them to query compact foundation-model-derived histology context tokens before survival prediction. This design enriches molecular information with histology context rather than merging separately encoded modalities only at the final stage. Training combines discrete-time survival prediction with genomic feature masking, WSI dropout, and paired WSI-genomics contrastive alignment. Across four external evaluations in colon, renal, lung, and glioblastoma cohorts, MIST improves external C-index over standard fusion baselines in the primary comparisons. These results support genomic-guided histology attention as a compact and effective strategy for multimodal oncology outcome prediction. Our code is available at https://github.com/samiyavuuz/MIST .

Abstract

Multimodal Pretraining

Explore similar work

MIST: Multimodal Survival Prediction with Genomic-Guided Histology Attention

ProtoPathway: Biologically Structured Prototype-Pathway Fusion for Multimodal Cancer Survival Prediction

Together, Then Apart: Balancing Alignment and Distinctiveness for Multimodal Survival Analysis