3D Modality-Aware Pre-training for Vision-Language Model in MRI Multi-organ Abnormality Detection
Authors: Haowen Zhu, Ning Yin, Xiaogen Zhou
Organizations: School of Computer Science and Engineering, Southeast University, China · Department of Medical Imaging, Suzhou Traditional Chinese Medicine Hospital, China · School of Electronic, Electrical Engineering and Physics, Fujian University of Technology
Vision-language models (VLMs) show strong potential for complex diagnostic tasks in medical imaging. However, applying VLMs to multi-organ medical imaging introduces two principal challenges: (1) modality-specific vision-language alignment and (2) cross-modal feature fusion. In this work, we propose MedMAP, a Medical Modality-Aware Pretraining framework that enhances vision-language representation learning in 3D MRI. MedMAP comprises a modality-aware vision-language alignment stage and a fine-tuning stage for multi-organ abnormality detection. During the pre-training stage, the modality-aware encoders implicitly capture the joint modality distribution and improve alignment between visual and textual representations. We then fine-tune the pre-trained vision encoders (while keeping the text encoder frozen) for downstream tasks. To this end, we curated MedMoM-MRI3D, comprising 7,392 3D MRI volume-report pairs spanning twelve MRI modalities and nine abnormalities tailored for various 3D medical analysis tasks. Extensive experiments on MedMoM-MRI3D demonstrate that MedMAP significantly outperforms existing VLMs in 3D MRI-based multi-organ abnormality detection. Our code is available at https://github.com/RomantiDr/MedMAP.
Figures & tables
Figure 1: Overview of the MedMAP framework. (a) Pre-training stage: The vision and text encoders are pretrained using modality-specific MRI volumes and reports. (b) Fine-tuning stage: The text encoder remains frozen, while a projector and the vision pipeline are trained. Additionally, the cross-modal semantic aggregation (CSA) module integrates pathological visual and textual tokens through structured cross-modal interactions.
Method
Liver (Multi-Class)
Brain (Binary)
Acc. (%)
AUC (%)
Acc. (%)
AUC (%)
Baseline [ 9 ]
82.86
75.15
81.86
81.33
MCPL [ 12 ]
87.87
86.49
88.21
84.53
MedCLIP [ 13 ]
85.53
84.21
85.84
83.64
PI-RADS [ 15 ]
86.24
85.46
87.83
85.61
MedMAP (Ours)
91.57
88.14
90.86
87.33
Table 1: Performance comparison with SOTA methods. Best results are in bold. Acc. is Accuracy.
Components
ACC (%)
Baseline
MAVLP
CCT
CSA
✓
82.86
✓
✓
84.22
✓
✓
✓
87.25
✓
✓
✓
✓
91.57
Table 2: Ablation study of MedMAP components on the MedMoM-MRI3D-Liver dataset. ACC is Accuracy.
Figure 2: Qualitative analysis of (i) CSA module and (ii) model interpretability.