MultiSigBERT: Beyond Survival Analysis through Multimodal and Sequential Modeling in Oncology

Paul Minchella; Stéphane Chrétien; Guillaume Metzler; Loïc Verlingue; Rémi Vaucher

cs.LGAug 17, 2026

MultiSigBERT: Beyond Survival Analysis through Multimodal and Sequential Modeling in Oncology

Authors: Paul Minchella, Stéphane Chrétien, Guillaume Metzler, Loïc Verlingue, Rémi Vaucher

Organizations: Université Lumière Lyon 2, 69500 Bron, France · Léon Bérard Center, Lyon 69008, France · EPITA, Lyon 69007, France

Abstract

Machine learning has become an essential component of modern healthcare, where the integration of heterogeneous data sources offers unprecedented opportunities to improve clinical decision-making. Electronic Health Records (EHR) contain complementary information -- including narrative clinical reports, numerical measurements, and structured variables -- yet most survival models remain limited to a single modality or fail to exploit the temporal nature of patient trajectories. We propose MultiSigBERT, a unified framework for multimodal sequential survival modeling in oncology based on path signature representations. Here, narrative medical reports (free-text) are converted into sentence embeddings by extracting and averaging contextual word embeddings. These representations are then compressed via modality-specific PCA and concatenated with structured covariates to form joint temporal trajectories which are then encoded using the Signature transform, a tool from Rough Paths theory that efficiently captures higher-order temporal interactions across modalities without supervision needed. The computed Signature features are finally incorporated as high dimensional features into a LASSO-regularized Cox model to estimate individualized risk scores. The performance of our novel MultiSigBERT pipeline is illustrated on the analysis of a real-world oncology cohort from the Léon Bérard Center, comprising over 120,000 medical reports and structured records from more than 2,500 patients. The model achieves a concordance index of 0.743 (sd 0.029) on an independent test set, demonstrating the benefit of jointly modeling multimodal temporal dynamics together with patient-level geometric structure for survival prediction.

Figures & tables

Figure 1: Overview of the MultiSigBERT pipeline. Step 1: Multimodal representation. Narrative clinical reports are converted into sentence embeddings and combined with structured variables observed at times $t_{1},\dots,t_{N}$ to form multimodal vectors $v_{t}$ . Step 2: Landmark restriction. Only observations within the window $[\boldsymbol{\mathrm{L}}-\boldsymbol{\mathrm{w}},\boldsymbol{\mathrm{L}}]$ are retained to avoid temporal information leakage. Step 3: Signature encoding. The truncated sequence $(v_{t})_{t}$ is encoded via the path signature transform, producing a fixed-dimensional representation $\mathbb{S}_{i}$ capturing temporal interactions across modalities. Step 4: Survival modeling. A LASSO-regularized Cox model is fitted on the signature features to estimate individualized risk scores. Step 5: Evaluation. Performance is assessed using the C-index and the Brier score.

Panel of Figure 2 — Figure 2: Landmark design: features are constructed from the backward window $[\boldsymbol{\mathrm{L}}-\boldsymbol{\mathrm{w}},\boldsymbol{\mathrm{L}}]$ and survival is re-indexed by $R_{i}=E_{i}-\boldsymbol{\mathrm{L}}$ . The prediction task is to estimate the conditional survival probability $\mathbb{P}(T_{i}>\boldsymbol{\mathrm{L}}+\tau\mid\mathbb{X}_{i}(\boldsymbol{\mathrm{L}}))$ , using only information observed before the landmark time.

Scenario	$\boldsymbol{\lambda}$	C-index Val. $\uparrow$	IBS@3y $\downarrow$
Reports Only	$5.15$	$0.747\;(\mathrm{sd}\;0.027)$	$0.1280\;(\mathrm{sd}\;0.004)$
Reports & Structured Variables	$4.60$	$0.743\;(\mathrm{sd}\;0.025)$	$0.1280\;(\mathrm{sd}\;0.004)$

Table 1: Optimal regularization LASSO-parameter

\lambda

selected by validation for each modeling scenario. Performance is reported as the mean C-index on the validation sets with its standard deviation (higher is better) and the Integrated Brier Score at 3 years (IBS@3y, lower is better).

Model	C-index $\uparrow$ (Test)	IBS $\downarrow$ (Test)
MultiSigBERT (RO)	0.741 [0.724, 0.758]	0.1282 [0.1243, 0.1322]
DeepSurv (RO)	0.600 [0.583, 0.614]	0.1512 [0.1406, 0.1619]
CoxTime (RO)	0.646 [0.638, 0.654]	0.091 [0.081, 0.101]
CTVF (RO)	0.619 [0.594, 0.645]	0.207 [0.185, 0.229]
MultiSigBERT (RS)	0.743 [0.725, 0.761]	0.1339 [0.1293, 0.1385]
DeepSurv (RS)	0.650 [0.620, 0.676]	0.1490 [0.1386, 0.1593]

Table 2: Test-set performance under the landmark design (Train = 1,263 patients; Test = 1,264 patients). Values are reported as mean with 95% Jackknife confidence intervals in brackets. RO: Reports Only; RS: Reports + Sequential Structured Data.

\uparrow

indicates that higher values are better;

\downarrow

indicates that lower values are better.

Figure 3: Distribution of cancer types in the study cohort. The figure reports the number of patients for each cancer category. Breast cancer represents the largest group (782 patients), followed by gynecological (232), intestinal (227), and prostate cancers (201). The remaining categories correspond to less frequent tumor types, reflecting the heterogeneous composition of the oncology cohort.

Figure 4: Distribution of report types in the clinical corpus. The majority of documents correspond to consultation reports (83,898), followed by hospital stay reports (33,966). Additional categories include follow-up consultations during treatment (3,778), consultation reports with procedures (1,618), and discharge reports (789). This distribution reflects the predominance of outpatient consultations in routine oncology follow-up, while hospitalizations represent a smaller but clinically important subset of the longitudinal documentation.

Figure 5: Empirical distributions of the sequential structured clinical variables used in the study: peripheral oxygen saturation (PO), systolic blood pressure (TA), Karnofsky performance index (KAR), and pulse rate (PL). Histograms display relative frequencies computed over all available observations in the dataset. These variables correspond to longitudinal measurements recorded during patient follow-up and illustrate the heterogeneous sampling patterns and value ranges typical of real-world electronic health records.

Figure 6: Boxplots of the sequential structured clinical variables used in the study: peripheral oxygen saturation (PO), systolic blood pressure (TA), Karnofsky performance index (KAR), and pulse rate (PL). The plots summarize the distribution of values across all observations, showing the median, interquartile range, and potential outliers. These measurements are extracted from longitudinal electronic health records and illustrate the variability and occasional extreme values typical of real-world clinical data.

Variable	Count	NaN Ratio	Mean	Std	Min	Q1	Median	Q3	Max
PO (Weight)	5,391,366	0.316	66.38	16.97	6	56	66	77	164
TA (Blood Pressure)	2,028,745	0.743	118.88	21.44	0	105	117	132	998
KAR (Karnosky)	4,453,251	0.435	80.23	19.48	10	70	80	100	100
PL (Pulse Rate)	2,019,331	0.744	83.26	18.87	0	71	81	95	880

Table 3: Descriptive statistics of the sequential structured variables. The dataset contains

7{,}885{,}925

observations corresponding to

2{,}527

unique patients.

Multimodal Survival Analysis

Multimodal survival analysis combines imaging, molecular, and clinical data to predict when health events occur and how risk changes over time.

See all 46 papers

May 12, 2026

Attention-Based Multimodal Survival Prediction with Cross-Modal Bilinear Fusion

We propose a novel multimodal deep learning framework for patient-level survival prediction, which integrates whole-slide histology features, RNA-seq expression profiles, and clinical variables. Our architecture combines an ABMIL module~\cite{ilse2018attention} for slide-level representation with feedforward encoders for RNA and clinical data. These embeddings are then integrated through low-rank bilinear cross-modal fusion~\cite{liu2018efficient} to model conditional interactions across modalities while controlling parameter growth. The model outputs continuous risk scores that are subsequently mapped to survival times using a nonparametric calibration procedure based on the Kaplan--Meier estimator~\cite{kaplan1958nonparametric}. By decomposing multimodal reasoning into independent pairwise interactions, the proposed fusion design promotes structural interpretability and parameter efficiency compared with full tensor and hierarchical fusion strategies. Experiments on the CHIMERA challenge dataset demonstrate improved predictive performance over concatenation-based baselines and competitive generalization on hidden evaluation cohorts. These results indicate that the proposed framework is a promising approach for multimodal survival prediction in HR-NMIBC. The implementation is publicly available at https://github.com/hassancpu/ChimeraChallenge2025_Task_3.

Hassan Keshvarikhojasteh, Josien P. W. Pluim, Mitko Veta

Department of Biomedical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands

Representation Learning Cancer Survival Prediction Survival Analysis Oncology Multimodal Survival Analysis Multimodal Fusion

Ask PDF

Jun 17, 2026

ChronoSurv: A Clinical Pathway-Guided Graph Framework for Multimodal Survival Analysis

Accurate survival prediction is essential for personalized treatment planning in head and neck cancer, yet remains challenging due to the heterogeneous and high-dimensional nature of multimodal clinical data. While deep survival models have improved predictive performance over classical statistical approaches, existing methods typically rely on static fusion strategies or temporally agnostic modeling, limiting their ability to capture structured clinical workflows. In this work, we propose ChronoSurv, a heterogeneous hierarchical directed graph framework for multimodal survival analysis. ChronoSurv represents patient care as a progression-aware clinical trajectory using directed graphs aligned with key diagnostic steps. A hierarchical topology incorporates fine-grained, coarse, and global representations, further supporting flexible adaptation to missing modalities, while heterogeneous message passing models complex and asymmetric relationships across modalities and clinical steps. Experimental results on two public datasets demonstrate that ChronoSurv achieves state-of-the-art discriminative performance while maintaining statistically reliable calibration. Comprehensive ablation studies further confirm the contribution of each architectural component, highlighting the potential of trajectory-aware graph modeling for multimodal survival prediction.

Hugo Miccinilli, Theo Di Piazza

Université Paris-Saclay, CentraleSupélec, MICS, France · University of Lyon, INSA Lyon, CREATIS, France

Cancer Survival Prediction Heterogeneous GNNs Survival Analysis Oncology Multimodal Survival Analysis Multimodal Graph Learning

Ask PDF

Aug 4, 2026

CIGTSurv: Clinical Information Guided Tri-modal Survival Prediction with Local Prototype Association and Global Feature Alignment

Multimodal learning has significantly advanced survival prediction by integrating pathology images with genomic data. However, clinical information, despite its critical role in reflecting a patient' s overall health, remains underutilized due to its discrete, sparse, and low-dimensional nature. Furthermore, the inherent heterogeneity across these modalities pose significant challenges in modeling cross-modal interactions. In this paper, we propose CIGTSurv, a Clinical Information Guided Tri-modal framework for Survival prediction. Specifically, we first design a holistic text template and use pretrained foundation models to transform clinical tabular data into high-dimensional tokenized embeddings. Using clinical information as an anchor, we then introduce a dual-level interaction mechanism: 1) a local prototype association (LPA) module based on cross-attention to explicitly learn token-level correspondences between different modalities, and 2) a global feature alignment (GFA) loss based on Maximum Mean Discrepancy (MMD) to implicitly enhance cross-modal distribution consistency. Extensive experiments on five TCGA cancer cohorts demonstrate that CIGTSurv achieves state-of-the-art (SOTA) survival prediction performance. Our source code is publicly available at https://github.com/Daijing-ai/CIGT-Surv.git.

Jing Dai, Qibin Zhang, Weiwei Zhou +4

Cancer Hospital of Dalian University of Technology, Shenyang, China · School of Biomedical Engineering, Faculty of Medicine, Dalian University of Technology, Dalian, China · Institute of Pathology, Technical University of Munich, TUM School of Medicine and Health, Munich, Germany +2

Cancer Survival Prediction Survival Analysis Cross-Modal Alignment Oncology Multimodal Survival Analysis Multimodal Fusion

Ask PDF

Abstract

Multimodal Survival Analysis

Explore similar work

Attention-Based Multimodal Survival Prediction with Cross-Modal Bilinear Fusion

ChronoSurv: A Clinical Pathway-Guided Graph Framework for Multimodal Survival Analysis

CIGTSurv: Clinical Information Guided Tri-modal Survival Prediction with Local Prototype Association and Global Feature Alignment