Multimodal Survival Analysis with Locally Deployable Large Language Models

Moritz Gögl; Christopher Yau

cs.LGMar 23, 2026

Multimodal Survival Analysis with Locally Deployable Large Language Models

Authors: Moritz Gögl, Christopher Yau

Organizations: University of Oxford

Abstract

We study multimodal survival analysis integrating clinical text, tabular covariates, and genomic profiles using locally deployable large language models (LLMs). As many institutions face tight computational and privacy constraints, this setting motivates the use of lightweight, on-premises models. Our approach jointly estimates calibrated survival probabilities and generates concise, evidence-grounded prognosis text via teacher-student distillation and principled multimodal fusion. On a TCGA cohort, it outperforms standard baselines, avoids reliance on cloud services and associated privacy concerns, and reduces the risk of hallucinated or miscalibrated estimates that can be observed in base LLMs.

Figures & tables

Figure 1: Overview of the proposed framework. A compact causal LLM encodes the input text into hidden embeddings used by a survival model (here, CoxPH) and produces verbalized survival estimates with explanatory text. Covariates and gene expression are fused either early (A) or late (B).

Figure 2: Teacher pipeline for constructing the student target. (1) Numeric prompting at 1/3/5 years from the input text; extract survival probabilities, and fit a parametric curve (exponential by default) to obtain the 3 - year value. (2) Explanation prompting conditioned on the text and the rounded 3 - year percentage; the explanation plus a marked probability sentence form the student’s training target.

Model/Configuration								Hidden-state		Verbalized		Combined
Name	Text ∗	Cov	GE	Survival	Fusion	Pretrain	CC	$C^{\text{td}}\!\uparrow$	IBS $\!\downarrow$	$C^{\text{td}}\!\uparrow$	IBS $\!\downarrow$	$C^{\text{td}}\!\uparrow$	IBS $\!\downarrow$
BERTSurv				CoxPH	Early	—	—	0.691	0.150	—	—	—	—
Cov-only				Discrete	—	—	—	0.665	0.149	—	—	—	—
				CoxPH	—	—	—	0.668	0.149	—	—	—	—
GE-only				Discrete	—	—	—	0.734	0.139	—	—	—	—
				CoxPH	—	—	—	0.751	0.135	—	—	—	—

Table 1: Performance across configurations and baselines.

Panel of Figure 3 — Figure 3: Qualitative example: original report (left) and generated assessment (right). Consistent evidence spans are highlighted in the same color.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure A.1: Additional qualitative example: original report (left) and student-generated assessment (right). Consistent evidence spans are highlighted with the same color; missing but clinically relevant information is framed in orange. The explanation is reasonable and faithful to the report, but the 3-year survival probability (65%) is clearly over-optimistic.

Panel of Figure 6 — Figure A.2: Two negative examples of student-generated assessments. Both outputs exhibit degraded English fluency; the right example further fails to provide an explicit verbalized probability.

Model	Component	Parameter	Value
\Block [draw=gray, fill=gray!15]28-1 Student Model	LLM	LLM Name	DeepSeek-R1 Distill Qwen-1.5B
		Freezing	All but last 18 transformer layers
		Precision	bfloat16
		Attention dropout	0.1
		Learning rate	5e-5
	Autoencoder	Latent dim	128

Appendix

Table C.1: Hyperparameters of student model, teacher model, and baselines.

Multimodal Survival Analysis

Multimodal survival analysis combines imaging, molecular, and clinical data to predict when health events occur and how risk changes over time.

See all 46 papers

Towards end-to-end LLM-based censoring-aware survival analysis
May 25, 2026Yishu Wei, Hexin Dong, Yi Lin +3Survival Analysis Mortality Prediction
Department of Population Health Science, Weill Cornell Medicine, New York, 10065, US · Department of Medicine, Weill Cornell Medicine, New York, 10065, US
From Hazard Functions to Language Space: Cox-Supervised Distillation of Survival Risk into a Large Language Model
Jun 8, 2026Nicholas I-Hsien Kuo, Blanca Gallego, Louisa JormSurvival Analysis Language Model Calibration
Centre for Big Data Research in Health, the University of New South Wales, Sydney, Australia
CIGTSurv: Clinical Information Guided Tri-modal Survival Prediction with Local Prototype Association and Global Feature Alignment
Aug 4, 2026Jing Dai, Qibin Zhang, Weiwei Zhou +4Cancer Survival Prediction Survival Analysis
Cancer Hospital of Dalian University of Technology, Shenyang, China · School of Biomedical Engineering, Faculty of Medicine, Dalian University of Technology, Dalian, China · Institute of Pathology, Technical University of Munich, TUM School of Medicine and Health, Munich, Germany +2

May 25, 2026J/K move · Enter open · S save

Towards end-to-end LLM-based censoring-aware survival analysis

Yishu Wei, Hexin Dong, Yi Lin, Jiahe Qian+2

Department of Population Health Science, Weill Cornell Medicine, New York, 10065, US · Department of Medicine, Weill Cornell Medicine, New York, 10065, US

Survival Analysis Mortality Prediction Medical LLMs

Open paper AskPDF arXiv

Objective: Survival analysis is central to medical prediction, yet large language models (LLMs) are rarely used as end-to-end survival models because censoring prevents straightforward supervised fine-tuning. Here we present LLMSurvival, a framework that enables censoring-aware survival analysis with unmodified LLMs operating directly on tabular clinical data. Materials and Methods: LLMSurvival reformulates time-to-event prediction as pairwise ranking among comparable subjects, and derives test-time risk by aggregating comparisons against anchor individuals from the training cohort. Results: Across two clinical tasks (ICU mortality prediction in MIMIC-IV and fragility fracture prediction in a NewYork-Presbyterian/Weill Cornell Medicine cohort), LLMSurvival improves overall concordance over Cox proportional hazards modeling by 3.1% for ICU mortality and 0.5% for fracture risk, 2.1% on average for ICU mortality and 2.8% for fracture risk over three established deep learning survival models. Discussion: The results show that survival modeling with censoring can be made compatible with LLM fine-tuning through comparison-based reformulation. The framework demonstrates high portability and superior performance over expert curated scores like SAPS-II and FRAX scores across diverse clinical context. Furthermore, the framework supports local deployment, as compact, publicly available base models provide sufficient performance. Conclusion: The LLMSurvival framework serves as a proof of concept for an integrated, censoring-conscious approach to survival analysis via LLMs.

Abstract

Multimodal Survival Analysis

Explore similar work

Towards end-to-end LLM-based censoring-aware survival analysis

From Hazard Functions to Language Space: Cox-Supervised Distillation of Survival Risk into a Large Language Model

CIGTSurv: Clinical Information Guided Tri-modal Survival Prediction with Local Prototype Association and Global Feature Alignment