Enabling Vision and Cross-Modal Learning for Multimodal Stroke Recurrence Prediction: An Interpretable Two-Step Framework
Authors: Christian Gapp, Elias Tappeiner, Martin Welk, Karl Fritscher, Stephanie Mangesius, Constantin Eisenschink, Philipp Deisl, Michael Knoflach, Astrid E. Grams, Elke R. Gizewski, Rainer Schubert
Organizations: Institute of Biomedical Image Analysis UMIT TIROL – Private University for Health Sciences and Health Technology, Eduard-Wallnöfer-Zentrum 1, 6060 Hall in Tirol, Austria · VASCage – Centre on Clinical Stroke Research, 6020 Innsbruck, Austria · Department of Radiology, Medical University of Innsbruck, 6020 Innsbruck, Austria · Neuroimaging Research Core Facility, Medical University of Innsbruck, 6020 Innsbruck, Austria · Department of Neurology, Medical University of Innsbruck, 6020 Innsbruck, Austria
Multimodal stroke recurrence prediction requires effective integration of heterogeneous clinical and imaging data, yet modality imbalance often causes models to over-rely on dominant modalities and underutilize complementary information. While self-supervised pretraining and selective parameter freezing are commonly employed to improve representation learning and fine-tuning stability, their effect on modality contributions and cross-modal behavior in multimodal medical models remains largely unexplored. In this work, we investigate whether image pretraining on 3D CTA scans reduces modality imbalance and improves cross-modal integration for stroke recurrence prediction, a clinically critical task we recently addressed. To this end, two multimodal neural networks are pretrained in a self-supervised manner and subsequently fine-tuned using two distinct freezing strategies. Their performance and modality utilization are compared against both the baseline model from our previous work and models trained entirely from scratch in this study. Our results demonstrate that self-supervised pretraining enables more effective utilization of the multimodal image-tabular dataset, outperforming both the prior baseline and all non-pretrained models. Notably, the best-performing Vision Transformer based neural network successfully overcomes unimodal collapse. Synergy analysis reveals significant interactions between vision and both gender and CHD, suggesting clinically relevant patterns for stroke recurrence. Overall, our findings demonstrate that self-supervised pretraining and strategic fine-tuning support more balanced modality utilization and enable meaningful cross-modal interactions. Code is publicly available at https://github.com/ChristianGappGit/SSL_Pretraining.
Figures & tables
characteristic
attribute
n
%
gender
men
79
66.4
women
40
33.6
heart disease
CHD only
24
20.2
PAD only
20
16.8
CHD + PAD
6
5.0
none
69
58.0
Table 1: Baseline characteristics of the study population (N = 119).
Figure 1: ResNetMLP: Vision model: ResNet34, Tabular model: MLP with one hidden layer. Fusion model: MLP with one hidden layer.
Figure 3: Visualization of parameter schedule during progressive layer unfreezing over 250 fine-tuning epochs for ResNet and ViT backbones. The training is divided into four stages, indicated by shaded background regions. Step curves show the cumulative number of trainable parameters (in millions) over time. ViT starts with more trainable parameters due to its larger vision head (see stage 1), whereas ResNet releases more parameters in stages 2-4, as larger layers are affected there.
Figure 4: Pretraining L1 loss curves over epochs for training and validation across both architectures.
XSRD-Net
Training Setup
train AUC
test AUC
mcv
ResNetMLP
from scratch – baseline
0.570
0.549
79.46%
pretrained frozen
0.409
0.653
93.31%
pretrained temp. frozen
0.695
0.715
92.20%
ViTMLP
from scratch – baseline
0.617
0.722
0 0.13%
pretrained frozen
0.805
0.743
28.39%
pretrained temp. frozen
0.626
0.729
0 5.53%
Table 2: Pretraining effects on performance for ResNetMLP and ViTMLP. The last column shows the modality contribution of vision mcv . The whole analysis is done with the test dataset. Highlighted in green: overall best model.
XSRD-Net
Training Setup
vision
tabular
3D CTA
age
gender
CHD
PAD
ResNetMLP
from scratch – baseline
79.46%
8.09%
6.96%
0.68%
4.81%
pretrained frozen
93.31%
2.21%
3.13%
0.31%
1.04%
pretrained temp. frozen
92.20%
2.57%
3.91%
0.40%
0.92%
ViTMLP
from scratch – baseline
0 0.13%
12.95%
75.68%
4.59%
6.65%
pretrained frozen
28.39%
8.73%
55.60%
3.03%
4.25%
Table 3: Detailed modality contributions for RFS predictions using the test dataset. Green highlighted: modality contributions for overall best model from Table 2 . Marked in red: unimodal collapse of ViTMLP trained from scratch.
ViTMLP model (pretrained frozen)
3D CTA
age
gender
CHD
PAD
MC
0.284
0.087
0.556
0.030
0.043
sij
×
−0.148
+0.337
+0.105
+0.002
sij(relapses)
×
−0.436
+1.068
+0.300
+0.004
ResNetMLP model (pretrained temp. frozen)
Table 4: Modality contribution (MC) [ 4 ] and synergy scores [ 6 ] for RFS prediction across 3D CTA ( i ) and individual tabular features ( j ) using the best models for each architecture on the test set. sij(relapses) is computed on relapse cases only. Green highlighted: significant synergy.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Occlusion sensitivity on 3D CTA scans for a true positive relapse example TP 1 (RFS = 107 days), using a mask size of [8,8,10] (25,088 inference passes). (a) ResNetMLP (pretrained, temp. frozen; visual importance 82.40%), (b) ViTMLP (pretrained, frozen; visual importance 54.28%): original images (top) and saliency maps (bottom). Slices are transversal (top view), coronal (back view), and sagittal (bottom view); yellow indicates high importance. ResNetMLP focuses on the left carotid region, while ViTMLP shifts to an upper-central intracranial area. Vision dominates in ResNetMLP, whereas gender is more influential in ViTMLP (35.94% vs. 8.50%), suggesting stronger reliance on gender-linked visual cues.
Deep learning and multi-modal fusion have demonstrated transformative potential in medical diagnosis by integrating diverse data sources. However, accurate prognosis for ischemic stroke remains challenging due to limitations in existing multi-modal approaches. First, current methods are predominantly confined to dual-modal fusion, lacking a framework that effectively integrates the trifecta of medical images, structured clinical data, and unstructured text. Second, they often fail to establish deep bidirectional interactions between modalities; To address these critical gaps, this paper proposes a novel tri-modal fusion model for ischemic stroke prognosis. Our approach first enriches the data representation by employing a Large Language Model (LLM) to automatically generate semi-structured diagnostic text from brain MRIs. This process not only addresses the scarcity of expert annotations but also serves as a regularized semantic enhancement, improving multimodal fusion robustness. Furthermore, we design a core component termed the Vision-Conditioned Dual Alignment Fusion Module (VDAFM), which strategically uses visual features as a conditional prior to guide fine-grained interaction with the generated text. This module achieves a dynamic and profound fusion through a dual semantic alignment loss, effectively mitigating modal heterogeneity. Extensive experiments on a real-world clinical dataset demonstrate that our model achieves state-of-the-art performance.
Liren Chen, Lidong Sun, Mingyan Huang +5
School of Information Science and Engineering, East China University of Science and Technology, Shanghai, 200237, PR China
Clinical diagnostic workups typically follow a modality escalation pathway: after initial clinical evaluation, clinicians begin with routine structural imaging (e.g., MRI), selectively add sequences such as FLAIR or T2 to refine the differential, and reserve molecular imaging (e.g., amyloid-PET) for cases that remain uncertain after standard evaluation. Consequently, patients are observed with heterogeneous and often incomplete modality subsets. However, most current AI models assume fixed data modalities as the model inputs. In this paper, we present BrainAnytime, a unified pretraining framework pretrained on 34,899 3D brain scans from five datasets that support brain image analysis under arbitrary modality availability spanning multi-sequence MRI and amyloid-PET. A single model accepts whatever imaging is available, from a lone T1 scan to a full multimodal workup. Pretraining learns structural-molecular correspondences between MRI and PET via cross-modal distillation (RCMD) and prioritizes disease-vulnerable anatomy via atlas-guided curriculum masking (PACM), all within a shared 3D masked autoencoder (Multi-MAE3D). Across four downstream tasks and five clinically motivated modality settings, BrainAnytime largely outperforms modality-specific models, missing-modality baselines, and large-scale brain MRI pretrained foundation models on most modality settings. Notably, it surpasses the strongest missing-modality baselines with relative improvements of 6.2% and 7.0% in average accuracy on CN vs. AD and CN vs. MCI classification, respectively. Code is available at https://github.com/SDH-Lab/BrainAnytime.
Guangqian Yang, Tong Ding, Wenlong Hou +4
Department of Biomedical Engineering, The Hong Kong Polytechnic University, Hong Kong SAR, China · Department of Technology Management for Innovation, The University of Tokyo, Japan · Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR, China
Self-supervised pre-training methods in medical imaging typically treat each individual as an isolated instance, learning representations through augmentation-based objectives or masked reconstruction. They often do not adequately capitalize on a key characteristic of physiological features: anatomical structures maintain consistent spatial relationships across individuals (instances), such as the thalamus being medial to the basal ganglia, regardless of variations in brain size, shape, or pathology. We propose leveraging this cross-instance topological consistency as a supervisory signal. The challenge arises from the inherent variability in medical imaging, which can differ significantly across instances and modalities. To tackle this, we focus on two alignment regimes. (i) Intra-instance: with pixel-level correspondences available, a cross-modal triplet objective explicitly preserves local neighborhood topology. (ii) Inter-instance: without such supervision, we derive pseudo-correspondences to control partial neighborhood alignment and prevent topology collapse across modalities. We validate our approach across 7 downstream multi-modal tasks, achieving average improvements of 1.1% and 5.94% in segmentation and classification tasks, respectively, and demonstrating significantly better robustness when modalities are missing at test time.
Tan Pan, Shuhao Mei, Yixuan Sun +8
Fudan University, China · Shanghai Academy of AI for Science, China · Bioinformatics Institute (BII), Agency for Science, Technology and Research (A*STAR), Singapore +2