Real-World Multi-Modal and Longitudinal Lung Cancer Dataset

Rita Cordeiro Mendes; Maria Rita Fonseca Verdelho; Carlos Santiago; Catarina Barata

eess.IVSep 4, 2026

Real-World Multi-Modal and Longitudinal Lung Cancer Dataset

Authors: Rita Cordeiro Mendes, Maria Rita Fonseca Verdelho, Carlos Santiago, Catarina Barata

Organizations: Institute for Systems and Robotics, Instituto Superior Técnico, Portugal

Abstract

Multi-modal learning has demonstrated strong potential in medical applications by integrating heterogeneous data sources such as medical imaging, clinical records, and genomics to improve predictive performance and support clinical decision-making. However, advances in this area are often constrained by two key challenges: the limited availability of well-curated, ready-to-use datasets that accurately reflect real-world conditions, where medical data are frequently collected inconsistently and are often incomplete; and the inherent difficulty of integrating heterogeneous data modalities. In this work, we introduce a newly curated multi-center, multi-modal, and longitudinal dataset designed to support the evaluation of a wide range of learning pipelines under realistic conditions. The dataset comprises a total of 1,365 lung cancer patients and has three imaging modalities (whole-slide images, CT scans, and PET scans), structured clinical data, transcriptomic, and longitudinal follow-up and treatment information. For each imaging modality the dataset contains more than one instance. Moreover, the dataset exhibits substantial and non-uniform missingness across modalities, making it well-suited for studying robust multi-modal fusion strategies. We further provide both uni-modal and multi-modal benchmarks on the task of 12-month overall survival prediction, disease-specific survival, as well as longitudinal benchmark of hazard prediction under severe missing data. Our results show that, despite high levels of missingness, integrating complementary modalities consistently improves predictive performance over uni-modal approaches, highlighting the value of multi-modal fusion in realistic clinical settings. The dataset and benchmark code are available at https://github.com/ritacmendes/MMIST-LUNG.

Figures & tables

Modality	Patient Number	Modality Missingness	Alive @12months	Deceased @12months
CLIN	1365	0	1169 (86%)	196 (14%)
TRNSC	1284	6%	1107 (86%)	177 (14%)
FUP	880	35%	777 (88%)	103 (12%)
CHEM	277	80%	245 (88%)	32 (12%)
RAD	923	32%	804 (87%)	119 (13%)
SURG	291	79%	250 (86%)	41 (14%)

Table 1: Distribution of patients per modality in the lung dataset.

Figure 1: Kaplan-Meier survival plot, stratified by histological subtype.

Figure 2: Benchmark pipeline overview. In the example, PET is signaled as missing modality. In our experiments, the classifier (CLF) can be either MLP or XGBoost.

	Multi-modal Fusion		Modality Configuration
Classifier	Early	Late	CLIN	ALL
MLP	Avg	-	56.74 $\pm$ 4.39	69.01 $\pm$ 1.95
	Max	-		69.03 $\pm$ 1.85
	SA	-		69.93 $\pm$ 2.51
	-	Avg		70.35 $\pm$ 3.12
XGB	Avg	-	69.07 $\pm$ 0.79	61.47 $\pm$ 2.37

Table 2: Benchmarking results for disease-specific survival prediction. CLIN corresponds to uni-modal clinical results. ALL includes CLIN, WSI, Transcriptomics, CT and PET data. Bold highlights best BAcc.

	Multi-modal Fusion		Modality Configuration
Classifier	Early	Late	CLIN	ALL
MLP	Avg	-	54.77 $\pm$ 3.15	68.39 ± 2.26
	Max	-		67.46 ± 2.09
	SA	-		64.66 ± 2.11
	-	Avg		69.93 ± 3.17
XGB	Avg	-	66.44 ± 1.12	52.24 ± 2.30

Table 3: Results of the study where AJCC tumor stage was set to missing in the disease-specific survival prediction task. Bold highlights best BAcc.

	Multi-modal Fusion		Modality Configuration
Classifier	Early	Late	CLIN	ALL
MLP	Avg	-	56.67 ± 4.19	64.84 $\pm$ 1.81
	Max	-		65.42 $\pm$ 3.17
	SA	-		62.93 $\pm$ 1.97
	-	Avg		63.97 $\pm$ 1.80
XGB	Avg	-	64.96 $\pm$ 1.25	56.33 $\pm$ 2.89

Table 4: Benchmarking results for overall survival prediction at 12 months. ALL includes CLIN, WSI, Transcriptomics, CT and PEt data. Bold highlights best BAcc.

	Multimodal Fusion		Modality Configuration
Classifier	Early	Late	CLIN	ALL
	Avg	-		64.05 ± 1.56
	Max	-		64.40 ± 2.90
	SA	-		61.07 ± 2.58
MLP	-	Avg	54.63 ± 4.80	62.68 ± 2.43
	Avg	-		48.74 ± 1.43

Table 5: Results for the study where AJCC tumor stage was set to missing in the binary survival at 12 months prediction task. Bold highlights best BAcc.

Figure 3: Predicted survival functions for a representative LUAD patient at diagnosis (t = 0) and after incorporation of longitudinal treatment events (t=27 days). The red dashed line denotes the observed event (death) time. Square points indicate new follow-up events in the patient timeline.

Multimodal Survival Analysis

Multimodal survival analysis combines imaging, molecular, and clinical data to predict when health events occur and how risk changes over time.

See all 47 papers

Design Choices for Multimodal Transformer Learning with Missing and Longitudinal Healthcare Data
Oct 10, 2026Rita Cordeiro Mendes, Maria Manuel Pereira, Carlos Santiago +1Learning with Missing Data Multimodal Fusion
Institute for Systems and Robotics, Instituto Superior Técnico, Lisbon, Portugal · ULS, Braga, Portugal
Multi-FRuGaL: Multimodal Flexible Redundancy-aware Decomposed Gated Learning for Cancer Diagnosis and Prognosis
Jun 5, 2026Sanket Kachole, Siddhesh Thakur, Shubham Innani +4Missing-Modality Learning Cancer Survival Prediction
Division of Computational Pathology, Department of Pathology and Laboratory Medicine, Indiana University School of Medicine, Indianapolis, IN, USA · IU Melvin and Bren Simon Comprehensive Cancer Center, Indianapolis, IN, USA · Departments of Biostatistics and Health Data Science; Radiology and Imaging Sciences; Neurological Surgery; Indiana University School of Medicine, Indianapolis, IN, USA +1
When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion
Feb 27, 2026Kejing Yin, Haizhou Xu, Wenfang Yao +5Multimodal Fusion Multimodal Model Evaluation
Department of Computer Science, Hong Kong Baptist University · Division of Artificial Intelligence, Lingnan University · School of Nursing, The Hong Kong Polytechnic University

Oct 10, 2026J/K move · Enter open · S save

Design Choices for Multimodal Transformer Learning with Missing and Longitudinal Healthcare Data

Rita Cordeiro Mendes, Maria Manuel Pereira, Carlos Santiago, Catarina Barata

Institute for Systems and Robotics, Instituto Superior Técnico, Lisbon, Portugal · ULS, Braga, Portugal

Learning with Missing Data Multimodal Fusion Missing-Modality Learning Multimodal Survival Analysis

Open paper AskPDF arXiv

Multimodal learning offers the potential to improve predictions by integrating complementary information from heterogeneous data sources. However, multimodal datasets are frequently incomplete, raising important questions about how models should handle missing information. Although recent approaches increasingly rely on Transformer-based architectures for multimodal fusion, incomplete multimodal data introduces challenges such as missing modalities and intra-modal missingness, which motivate key architectural design choices, such as tokenization granularity. In addition, many datasets are longitudinal, with observations collected across multiple time points. The sequential nature of the Transformer architectures provides a natural framework for modeling such data, but introduces additional considerations, such as how temporal information should be represented. In this study, we systematically investigate these design choices using three different cancer datasets. We compare modality-level and feature-level tokenization and evaluate attention masking and imputation as strategies for handling missing information, both naturally occurring in the datasets and synthetically induced. We further investigate approaches for incorporating temporal information in our training process and how to model it. Experiments are conducted with unimodal and multimodal configurations, as well as different levels of missingness for binary survival classification and multiple time encoding strategies for discrete-time survival prediction. Our results show that while multimodal outperforms unimodal results, missing data heavily impacts predictions and thus effective design choices such as tokenization granularity, and intra-missing handling is essential for reliable prediction.

Abstract

Multimodal Survival Analysis

Explore similar work

Design Choices for Multimodal Transformer Learning with Missing and Longitudinal Healthcare Data

Multi-FRuGaL: Multimodal Flexible Redundancy-aware Decomposed Gated Learning for Cancer Diagnosis and Prognosis

When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion