MultiSigBERT: Beyond Survival Analysis through Multimodal and Sequential Modeling in Oncology

Paul Minchella; Stéphane Chrétien; Guillaume Metzler; Loïc Verlingue; Rémi Vaucher

cs.LGAug 17, 2026

MultiSigBERT: Beyond Survival Analysis through Multimodal and Sequential Modeling in Oncology

Authors: Paul Minchella, Stéphane Chrétien, Guillaume Metzler, Loïc Verlingue, Rémi Vaucher

Organizations: Université Lumière Lyon 2, 69500 Bron, France · Léon Bérard Center, Lyon 69008, France · EPITA, Lyon 69007, France

Abstract

Machine learning has become an essential component of modern healthcare, where the integration of heterogeneous data sources offers unprecedented opportunities to improve clinical decision-making. Electronic Health Records (EHR) contain complementary information -- including narrative clinical reports, numerical measurements, and structured variables -- yet most survival models remain limited to a single modality or fail to exploit the temporal nature of patient trajectories. We propose MultiSigBERT, a unified framework for multimodal sequential survival modeling in oncology based on path signature representations. Here, narrative medical reports (free-text) are converted into sentence embeddings by extracting and averaging contextual word embeddings. These representations are then compressed via modality-specific PCA and concatenated with structured covariates to form joint temporal trajectories which are then encoded using the Signature transform, a tool from Rough Paths theory that efficiently captures higher-order temporal interactions across modalities without supervision needed. The computed Signature features are finally incorporated as high dimensional features into a LASSO-regularized Cox model to estimate individualized risk scores. The performance of our novel MultiSigBERT pipeline is illustrated on the analysis of a real-world oncology cohort from the Léon Bérard Center, comprising over 120,000 medical reports and structured records from more than 2,500 patients. The model achieves a concordance index of 0.743 (sd 0.029) on an independent test set, demonstrating the benefit of jointly modeling multimodal temporal dynamics together with patient-level geometric structure for survival prediction.

Figures & tables

Figure 1: Overview of the MultiSigBERT pipeline. Step 1: Multimodal representation. Narrative clinical reports are converted into sentence embeddings and combined with structured variables observed at times $t_{1},\dots,t_{N}$ to form multimodal vectors $v_{t}$ . Step 2: Landmark restriction. Only observations within the window $[\boldsymbol{\mathrm{L}}-\boldsymbol{\mathrm{w}},\boldsymbol{\mathrm{L}}]$ are retained to avoid temporal information leakage. Step 3: Signature encoding. The truncated sequence $(v_{t})_{t}$ is encoded via the path signature transform, producing a fixed-dimensional representation $\mathbb{S}_{i}$ capturing temporal interactions across modalities. Step 4: Survival modeling. A LASSO-regularized Cox model is fitted on the signature features to estimate individualized risk scores. Step 5: Evaluation. Performance is assessed using the C-index and the Brier score.

Panel of Figure 2 — Figure 2: Landmark design: features are constructed from the backward window $[\boldsymbol{\mathrm{L}}-\boldsymbol{\mathrm{w}},\boldsymbol{\mathrm{L}}]$ and survival is re-indexed by $R_{i}=E_{i}-\boldsymbol{\mathrm{L}}$ . The prediction task is to estimate the conditional survival probability $\mathbb{P}(T_{i}>\boldsymbol{\mathrm{L}}+\tau\mid\mathbb{X}_{i}(\boldsymbol{\mathrm{L}}))$ , using only information observed before the landmark time.

Scenario	$\boldsymbol{\lambda}$	C-index Val. $\uparrow$	IBS@3y $\downarrow$
Reports Only	$5.15$	$0.747\;(\mathrm{sd}\;0.027)$	$0.1280\;(\mathrm{sd}\;0.004)$
Reports & Structured Variables	$4.60$	$0.743\;(\mathrm{sd}\;0.025)$	$0.1280\;(\mathrm{sd}\;0.004)$

Table 1: Optimal regularization LASSO-parameter

\lambda

selected by validation for each modeling scenario. Performance is reported as the mean C-index on the validation sets with its standard deviation (higher is better) and the Integrated Brier Score at 3 years (IBS@3y, lower is better).

Model	C-index $\uparrow$ (Test)	IBS $\downarrow$ (Test)
MultiSigBERT (RO)	0.741 [0.724, 0.758]	0.1282 [0.1243, 0.1322]
DeepSurv (RO)	0.600 [0.583, 0.614]	0.1512 [0.1406, 0.1619]
CoxTime (RO)	0.646 [0.638, 0.654]	0.091 [0.081, 0.101]
CTVF (RO)	0.619 [0.594, 0.645]	0.207 [0.185, 0.229]
MultiSigBERT (RS)	0.743 [0.725, 0.761]	0.1339 [0.1293, 0.1385]
DeepSurv (RS)	0.650 [0.620, 0.676]	0.1490 [0.1386, 0.1593]

Table 2: Test-set performance under the landmark design (Train = 1,263 patients; Test = 1,264 patients). Values are reported as mean with 95% Jackknife confidence intervals in brackets. RO: Reports Only; RS: Reports + Sequential Structured Data.

\uparrow

indicates that higher values are better;

\downarrow

indicates that lower values are better.

Figure 3: Distribution of cancer types in the study cohort. The figure reports the number of patients for each cancer category. Breast cancer represents the largest group (782 patients), followed by gynecological (232), intestinal (227), and prostate cancers (201). The remaining categories correspond to less frequent tumor types, reflecting the heterogeneous composition of the oncology cohort.

Figure 4: Distribution of report types in the clinical corpus. The majority of documents correspond to consultation reports (83,898), followed by hospital stay reports (33,966). Additional categories include follow-up consultations during treatment (3,778), consultation reports with procedures (1,618), and discharge reports (789). This distribution reflects the predominance of outpatient consultations in routine oncology follow-up, while hospitalizations represent a smaller but clinically important subset of the longitudinal documentation.

Figure 5: Empirical distributions of the sequential structured clinical variables used in the study: peripheral oxygen saturation (PO), systolic blood pressure (TA), Karnofsky performance index (KAR), and pulse rate (PL). Histograms display relative frequencies computed over all available observations in the dataset. These variables correspond to longitudinal measurements recorded during patient follow-up and illustrate the heterogeneous sampling patterns and value ranges typical of real-world electronic health records.

Figure 6: Boxplots of the sequential structured clinical variables used in the study: peripheral oxygen saturation (PO), systolic blood pressure (TA), Karnofsky performance index (KAR), and pulse rate (PL). The plots summarize the distribution of values across all observations, showing the median, interquartile range, and potential outliers. These measurements are extracted from longitudinal electronic health records and illustrate the variability and occasional extreme values typical of real-world clinical data.

Variable	Count	NaN Ratio	Mean	Std	Min	Q1	Median	Q3	Max
PO (Weight)	5,391,366	0.316	66.38	16.97	6	56	66	77	164
TA (Blood Pressure)	2,028,745	0.743	118.88	21.44	0	105	117	132	998
KAR (Karnosky)	4,453,251	0.435	80.23	19.48	10	70	80	100	100
PL (Pulse Rate)	2,019,331	0.744	83.26	18.87	0	71	81	95	880

Table 3: Descriptive statistics of the sequential structured variables. The dataset contains

7{,}885{,}925

observations corresponding to

2{,}527

unique patients.

Multimodal Survival Analysis

Multimodal survival analysis combines imaging, molecular, and clinical data to predict when health events occur and how risk changes over time.

See all 46 papers

Attention-Based Multimodal Survival Prediction with Cross-Modal Bilinear Fusion
May 12, 2026Hassan Keshvarikhojasteh, Josien P. W. Pluim, Mitko VetaRepresentation Learning Cancer Survival Prediction
Department of Biomedical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands
ChronoSurv: A Clinical Pathway-Guided Graph Framework for Multimodal Survival Analysis
Jun 17, 2026Hugo Miccinilli, Theo Di PiazzaCancer Survival Prediction Heterogeneous GNNs
Université Paris-Saclay, CentraleSupélec, MICS, France · University of Lyon, INSA Lyon, CREATIS, France
CIGTSurv: Clinical Information Guided Tri-modal Survival Prediction with Local Prototype Association and Global Feature Alignment
Aug 4, 2026Jing Dai, Qibin Zhang, Weiwei Zhou +4Cancer Survival Prediction Survival Analysis
Cancer Hospital of Dalian University of Technology, Shenyang, China · School of Biomedical Engineering, Faculty of Medicine, Dalian University of Technology, Dalian, China · Institute of Pathology, Technical University of Munich, TUM School of Medicine and Health, Munich, Germany +2

May 12, 2026J/K move · Enter open · S save

Attention-Based Multimodal Survival Prediction with Cross-Modal Bilinear Fusion

Hassan Keshvarikhojasteh, Josien P. W. Pluim, Mitko Veta

Department of Biomedical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands

Representation Learning Cancer Survival Prediction Survival Analysis Oncology Multimodal Survival Analysis Multimodal Fusion

Open paper AskPDF arXiv

We propose a novel multimodal deep learning framework for patient-level survival prediction, which integrates whole-slide histology features, RNA-seq expression profiles, and clinical variables. Our architecture combines an ABMIL module~\cite{ilse2018attention} for slide-level representation with feedforward encoders for RNA and clinical data. These embeddings are then integrated through low-rank bilinear cross-modal fusion~\cite{liu2018efficient} to model conditional interactions across modalities while controlling parameter growth. The model outputs continuous risk scores that are subsequently mapped to survival times using a nonparametric calibration procedure based on the Kaplan--Meier estimator~\cite{kaplan1958nonparametric}. By decomposing multimodal reasoning into independent pairwise interactions, the proposed fusion design promotes structural interpretability and parameter efficiency compared with full tensor and hierarchical fusion strategies. Experiments on the CHIMERA challenge dataset demonstrate improved predictive performance over concatenation-based baselines and competitive generalization on hidden evaluation cohorts. These results indicate that the proposed framework is a promising approach for multimodal survival prediction in HR-NMIBC. The implementation is publicly available at https://github.com/hassancpu/ChimeraChallenge2025_Task_3.

Abstract

Multimodal Survival Analysis

Explore similar work

Attention-Based Multimodal Survival Prediction with Cross-Modal Bilinear Fusion

ChronoSurv: A Clinical Pathway-Guided Graph Framework for Multimodal Survival Analysis

CIGTSurv: Clinical Information Guided Tri-modal Survival Prediction with Local Prototype Association and Global Feature Alignment