Enabling Vision and Cross-Modal Learning for Multimodal Stroke Recurrence Prediction: An Interpretable Two-Step Framework

Christian Gapp; Elias Tappeiner; Martin Welk; Karl Fritscher; Stephanie Mangesius; Constantin Eisenschink; Philipp Deisl; Michael Knoflach

cs.CVSep 10, 2026

Enabling Vision and Cross-Modal Learning for Multimodal Stroke Recurrence Prediction: An Interpretable Two-Step Framework

Authors: Christian Gapp, Elias Tappeiner, Martin Welk, Karl Fritscher, Stephanie Mangesius, Constantin Eisenschink, Philipp Deisl, Michael Knoflach,

Organizations: Institute of Biomedical Image Analysis UMIT TIROL – Private University for Health Sciences and Health Technology, Eduard-Wallnöfer-Zentrum 1, 6060 Hall in Tirol, Austria · VASCage – Centre on Clinical Stroke Research, 6020 Innsbruck, Austria · Department of Radiology, Medical University of Innsbruck, 6020 Innsbruck, Austria · Neuroimaging Research Core Facility, Medical University of Innsbruck, 6020 Innsbruck, Austria ·

Abstract

Multimodal stroke recurrence prediction requires effective integration of heterogeneous clinical and imaging data, yet modality imbalance often causes models to over-rely on dominant modalities and underutilize complementary information. While self-supervised pretraining and selective parameter freezing are commonly employed to improve representation learning and fine-tuning stability, their effect on modality contributions and cross-modal behavior in multimodal medical models remains largely unexplored. In this work, we investigate whether image pretraining on 3D CTA scans reduces modality imbalance and improves cross-modal integration for stroke recurrence prediction, a clinically critical task we recently addressed. To this end, two multimodal neural networks are pretrained in a self-supervised manner and subsequently fine-tuned using two distinct freezing strategies. Their performance and modality utilization are compared against both the baseline model from our previous work and models trained entirely from scratch in this study. Our results demonstrate that self-supervised pretraining enables more effective utilization of the multimodal image-tabular dataset, outperforming both the prior baseline and all non-pretrained models. Notably, the best-performing Vision Transformer based neural network successfully overcomes unimodal collapse. Synergy analysis reveals significant interactions between vision and both gender and CHD, suggesting clinically relevant patterns for stroke recurrence. Overall, our findings demonstrate that self-supervised pretraining and strategic fine-tuning support more balanced modality utilization and enable meaningful cross-modal interactions. Code is publicly available at https://github.com/ChristianGappGit/SSL_Pretraining.

Figures & tables

characteristic	attribute	n	%
gender	men	79	66.4
	women	40	33.6
heart disease	CHD only	24	20.2
	PAD only	20	16.8
	CHD + PAD	6	5.0
	none	69	58.0

Table 1: Baseline characteristics of the study population (N = 119).

Panel of Figure 1 — Figure 1: ResNetMLP: Vision model: ResNet34, Tabular model: MLP with one hidden layer. Fusion model: MLP with one hidden layer.

Figure 3: Visualization of parameter schedule during progressive layer unfreezing over 250 fine-tuning epochs for ResNet and ViT backbones. The training is divided into four stages, indicated by shaded background regions. Step curves show the cumulative number of trainable parameters (in millions) over time. ViT starts with more trainable parameters due to its larger vision head (see stage 1), whereas ResNet releases more parameters in stages 2-4, as larger layers are affected there.

Panel of Figure 4 — Figure 4: Pretraining L1 loss curves over epochs for training and validation across both architectures.

XSRD-Net	Training Setup	train AUC	test AUC	$mc_{v}$
ResNetMLP	from scratch – baseline	$0.570$	$0.549$	79.46%
	pretrained frozen	$0.409$	$0.653$	93.31%
	pretrained temp. frozen	$0.695$	$\mathbf{0.715}$	92.20%
ViTMLP	from scratch – baseline	$0.617$	$0.722$	0 0.13%
	pretrained frozen	$0.805$	$\mathbf{0.743}$	28.39%
	pretrained temp. frozen	$0.626$	$0.729$	0 5.53%

Table 2: Pretraining effects on performance for ResNetMLP and ViTMLP. The last column shows the modality contribution of vision

mc_{v}

. The whole analysis is done with the test dataset. Highlighted in green: overall best model.

XSRD-Net	Training Setup	vision	tabular
		3D CTA	age	gender	CHD	PAD
ResNetMLP	from scratch – baseline	79.46%	8.09%	6.96%	0.68%	4.81%
	pretrained frozen	93.31%	2.21%	3.13%	0.31%	1.04%
	pretrained temp. frozen	92.20%	2.57%	3.91%	0.40%	0.92%
ViTMLP	from scratch – baseline	0 0.13%	12.95%	75.68%	4.59%	6.65%
	pretrained frozen	28.39%	8.73%	55.60%	3.03%	4.25%

Table 3: Detailed modality contributions for RFS predictions using the test dataset. Green highlighted: modality contributions for overall best model from Table 2 . Marked in red: unimodal collapse of ViTMLP trained from scratch.

ViTMLP model (pretrained frozen)
	3D CTA	age	gender	CHD	PAD
MC	0.284	0.087	0.556	0.030	0.043
$s_{ij}$	$\times$	$-0.148$	$+0.337$	$+0.105$	$+0.002$
$s_{ij\mathrm{(relapses)}}$	$\times$	$-0.436$	$+1.068$	$+0.300$	$+0.004$

ResNetMLP model (pretrained temp. frozen)

Table 4: Modality contribution (MC) [ 4 ] and synergy scores [ 6 ] for RFS prediction across 3D CTA (

i

) and individual tabular features (

j

) using the best models for each architecture on the test set.

s_{ij\mathrm{(relapses)}}

is computed on relapse cases only. Green highlighted: significant synergy.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Occlusion sensitivity on 3D CTA scans for a true positive relapse example TP 1 (RFS = 107 days), using a mask size of [8,8,10] (25,088 inference passes). (a) ResNetMLP (pretrained, temp. frozen; visual importance 82.40%), (b) ViTMLP (pretrained, frozen; visual importance 54.28%): original images (top) and saliency maps (bottom). Slices are transversal (top view), coronal (back view), and sagittal (bottom view); yellow indicates high importance. ResNetMLP focuses on the left carotid region, while ViTMLP shifts to an upper-central intracranial area. Vision dominates in ResNetMLP, whereas gender is more influential in ViTMLP (35.94% vs. 8.50%), suggesting stronger reliance on gender-linked visual cues.

Multimodal Learning

Multimodal learning studies how models combine and reason over different data types, such as images, text, audio, signals, and structured records.

See all 338 papers

Vision-Core Guided Contrastive Learning for Balanced Multi-modal Prognosis Prediction of Stroke
May 14, 2026Liren Chen, Lidong Sun, Mingyan Huang +5Cross-Modal Alignment Clinical Outcome Prediction
School of Information Science and Engineering, East China University of Science and Technology, Shanghai, 200237, PR China
BrainAnytime: Anatomy-Aware Cross-Modal Pretraining for Brain Image Analysis with Arbitrary Modality Availability
May 13, 2026Guangqian Yang, Tong Ding, Wenlong Hou +4Missing-Modality Learning Masked Autoencoders
Department of Biomedical Engineering, The Hong Kong Polytechnic University, Hong Kong SAR, China · Department of Technology Management for Innovation, The University of Tokyo, Japan · Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR, China
Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
May 14, 2026Tan Pan, Shuhao Mei, Yixuan Sun +8Representation Learning Multimodal Robustness
Fudan University, China · Shanghai Academy of AI for Science, China · Bioinformatics Institute (BII), Agency for Science, Technology and Research (A*STAR), Singapore +2

May 14, 2026J/K move · Enter open · S save

Vision-Core Guided Contrastive Learning for Balanced Multi-modal Prognosis Prediction of Stroke

Liren Chen, Lidong Sun, Mingyan Huang, Junzhe Tang+4

School of Information Science and Engineering, East China University of Science and Technology, Shanghai, 200237, PR China

Cross-Modal Alignment Clinical Outcome Prediction Multimodal Contrastive Learning Multimodal Fusion

Open paper AskPDF arXiv

Deep learning and multi-modal fusion have demonstrated transformative potential in medical diagnosis by integrating diverse data sources. However, accurate prognosis for ischemic stroke remains challenging due to limitations in existing multi-modal approaches. First, current methods are predominantly confined to dual-modal fusion, lacking a framework that effectively integrates the trifecta of medical images, structured clinical data, and unstructured text. Second, they often fail to establish deep bidirectional interactions between modalities; To address these critical gaps, this paper proposes a novel tri-modal fusion model for ischemic stroke prognosis. Our approach first enriches the data representation by employing a Large Language Model (LLM) to automatically generate semi-structured diagnostic text from brain MRIs. This process not only addresses the scarcity of expert annotations but also serves as a regularized semantic enhancement, improving multimodal fusion robustness. Furthermore, we design a core component termed the Vision-Conditioned Dual Alignment Fusion Module (VDAFM), which strategically uses visual features as a conditional prior to guide fine-grained interaction with the generated text. This module achieves a dynamic and profound fusion through a dual semantic alignment loss, effectively mitigating modal heterogeneity. Extensive experiments on a real-world clinical dataset demonstrate that our model achieves state-of-the-art performance.

Abstract

Multimodal Learning

Explore similar work

Vision-Core Guided Contrastive Learning for Balanced Multi-modal Prognosis Prediction of Stroke

BrainAnytime: Anatomy-Aware Cross-Modal Pretraining for Brain Image Analysis with Arbitrary Modality Availability

Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging