SYNPRED: A Synergistic Approach to Multimodal Learning for Clinical Prediction

Ivana Janíčková; Yen Y. Tan; Thomas H. Helbich; Philipp Seeböck; Julieta Di Marco; Zsuzsanna Bago-Horvath; Ulrike Heber; Thomas Spiegel

cs.LGOct 13, 2026

SYNPRED: A Synergistic Approach to Multimodal Learning for Clinical Prediction

Authors: Ivana Janíčková, Yen Y. Tan, Thomas H. Helbich, Philipp Seeböck, Julieta Di Marco, Zsuzsanna Bago-Horvath, Ulrike Heber, Thomas Spiegel, +1 more

Organizations: Computational Imaging Research Lab, Department of Biomedical Imaging and Image-guided Therapy, Medical University of Vienna, Austria · Comprehensive Center for Artificial Intelligence in Medicine, Medical University of Vienna, Austria · Christian Doppler Laboratory for Machine Learning Driven Precision Imaging, Department of Biomedical Imaging and Image-guided Therapy, Medical University of Vienna, Austria · Department of Obstetrics and Gynecology, Medical University of Vienna, Austria · Division of General and Pediactric Radiology, Department of Biomedical Imaging and Image-guided Therapy, Medical University of Vienna, Austria · Department of Pathology, Medical University of Vienna, Austria

Abstract

Multimodal medical data captures complementary aspects of disease. It provides a comprehensive basis for diagnosis and the identification of biological associations linked to progression or treatment response. Established contrastive learning approaches exploit shared information across modalities, but fall short in exploiting complementary, modality-specific signals relevant for a task. Here, we propose a synergistic multimodal variational autoencoder (VAE) framework that separates task-relevant unimodal and multimodal information from instance-specific variation task-irrelevant, allowing complementary signals across modalities to be leveraged for prediction. The method yields benefits even if modalities are missing at test time. We evaluate the method on a treatment response prediction task using medical imaging and RNA data. Empirical comparison with alignment-based and multimodal VAE state-of-the-art approaches demonstrates improved multimodal prediction performance, and consistent benefits for prediction from a single modality after multimodal training. The full code can be found here: https://github.com/cirmuw/SYNPRED.

Figures & tables

Figure 1: Method Overview Each modality-specific encoder maps the input to a factorised Gaussian posterior, decomposing the latent space into shared task-relevant (TR-fusion), modality-specific task-relevant (TR-uni), and instance-specific (IS) components with posterior parameters ( $\mu,\sigma$ ) and sampled latents ( $z$ ). TR-fusion posteriors of each modality are combined via a mixture-of-experts (MoE). Task-conditioned KL regularisation ( $\mathcal{L}_{\text{KL-task}}$ ) is applied to TR-fusion and TR-uni posteriors, while a KL term ( $\mathcal{L}_{\text{KL-inst}}$ ) regularises IS posteriors. The sampled latents $z$ are concatenated and decoded using a reconstruction loss ( $\mathcal{L}_{\text{rec}}$ ). For prediction, embeddings are combined using a product-of-experts (PoE).

Single-Modal Approaches
Method	IMG-only	RNA-only	Multimodal
Method	AUROC	AUROC	AUROC
IMG VAE	0.549 [0.508,0.590]	—	—
RNA VAE	—	0.656 [0.614,0.702]	—
Multimodal Approaches
concat ${\ddagger}$	—	—	0.644 [0.630,0.657]

Table 1: Comparison with SOTA: Mean AUROC and 95% confidence interval results for image/RNA-only, and multimodal evaluation.

{\dagger}

and

{\ddagger}

indicate complete modality requirement during pretraining and fine-tuning respectively. All baselines were finetuned, for SYNPRED (lin.prob.) is linear probing; (ft) is finetuning.

Ablation			IMG-only	RNA-only	Multimodal
$z^{uni}$	$z^{fusion}$	$z^{inst}$	AUROC	AUROC	AUROC
✗	✗	✗	0.524 [0.488,0.559]	0.643 [0.606,0.680]	0.646 [0.611,0.682]
✓	✓	✗	0.555 [0.505,0.606]	0.674 [0.626,0.723]	0.680 [0.632,0.730]
✓	✗	✓	0.603 [0.508,0.671]	0.706 [0.669,0.748]	0.719 [0.693,0.760]
✗	✓	✓	0.524 [0.468,0.573]	0.647 [0.604,0.681]	0.658 [0.611,0.670]
✓	✓	✓	0.638 [0.601,0.673]	0.706 [0.666,0.741]	0.725 [0.682,0.757]

Table 2: Ablation study of latent components. Mean AUROC (95% CI) for image-only, RNA-only, and multimodal prediction using linear probing, comparing a non-disentangled baseline and models with ablated task-relevant ( uni , fusion ) and instance-specific ( inst ) latent components.

Panel of Figure 2 — Figure 2: Explainability evaluation. ( Left ) DICE overlap between saliency maps and ground-truth segmentations. ( Right ) Qualitative saliency visualizations. The proposed method achieves the highest overlap and more coherent localization of task-relevant regions.

Explore similar work

Context-driven Missing-Modality Learning for Robust Medical Diagnosis with Image-Tabular Data
May 25, 2026Tianling Liu, Lequan Yu, Tong Han +1Missing-Modality Learning Multimodal Robustness
College of Intelligence and Computing, Tianjin University, Tianjin 300350, China. · Department of Statistics and Actuarial Science, School of Computing and Data Science, The University of Hong Kong, Hong Kong. · Department of Radiology, Tianjin Huanhu Hospital, Tianjin 300350, China. +2
Characterizing the Predictive Impact of Modalities with Supervised Latent-Variable Modeling
Feb 19, 2026Divyam Madaan, Sumit Chopra, Kyunghyun ChoMissing-Modality Learning Multimodal Learning
New York University. · New York University Grossman School of Medicine · Genentech +1
MMAP: Multimodal Missing-Aware Pretraining for Longitudinal Alzheimer's Prediction
Sep 22, 2026Fiona Kekwick, Matthew Baugh, Bernhard Kainz +2Missing-Modality Learning Cross-Modal Alignment
Department of Computing, Imperial College London, London, UK · FAU Erlangen-Nürnberg, Erlangen, Germany · Department of Brain Sciences, Imperial College London, London, UK +1

May 25, 2026J/K move · Enter open · S save

Context-driven Missing-Modality Learning for Robust Medical Diagnosis with Image-Tabular Data

Tianling Liu, Lequan Yu, Tong Han, Liang Wan

College of Intelligence and Computing, Tianjin University, Tianjin 300350, China. · Department of Statistics and Actuarial Science, School of Computing and Data Science, The University of Hong Kong, Hong Kong. · Department of Radiology, Tianjin Huanhu Hospital, Tianjin 300350, China.+2

Missing-Modality Learning Multimodal Robustness Medical Diagnosis Cross-Modal Alignment

Open paper AskPDF arXiv

While multimodal data integrating diverse imaging and clinical tabular records is crucial for accurate medical diagnosis, the arbitrary absence of specific modalities is prevalent in clinical practice, severely degrading the performance of multimodal models. Existing methods either discard missing modalities, leading to information loss, or struggle to synthesize them without capturing complex inter-modal dependencies. To address these limitations, we propose a novel Context-driven Missing-Modality Learning (CMML) framework, which sequentially performs modality synthesis and semantic alignment to achieve robust diagnosis under arbitrary missing conditions. Specifically, we design a Cascade Residual Transformer-based Autoencoder (CRTA) that leverages learnable context tokens acting as dataset-level semantic prior to capture inter-modal dependencies and synthesize key missing representations. These representations are further enriched by modality-specific memory banks. To resolve the discrepancy between original available and synthesized representations, we transform the learned context tokens into instance-adaptive semantic references by infusing multimodal representations from the CRTA's outputs. This reference guides the alignment of heterogeneous modality representations into a unified space, where class-aware contrastive refinement is finally applied to explore discriminative diagnostic cues. Extensive evaluations on skin lesion (Derm7pt), ocular disease (ODIR), and meningioma (MEN) datasets demonstrate that CMML significantly outperforms state-of-the-art (SOTA) methods, yielding AVG AUC improvements of 1.26%, 0.97%, and 1.32%, respectively.

Abstract

Explore similar work

Context-driven Missing-Modality Learning for Robust Medical Diagnosis with Image-Tabular Data

Characterizing the Predictive Impact of Modalities with Supervised Latent-Variable Modeling

MMAP: Multimodal Missing-Aware Pretraining for Longitudinal Alzheimer's Prediction