A4-MR: Abnormality-aware Anatomical Alignment for MR

Noel D'souza; Zhijian Yang; Nishank Singla; Pranay Doshi; Krisztian Koos; Fei Tan; Bruno Nunes; Laszlo Rusko

cs.CVOct 9, 2026

A4-MR: Abnormality-aware Anatomical Alignment for MR

Authors: Noel D'souza, Zhijian Yang, Nishank Singla, Pranay Doshi, Krisztian Koos, Fei Tan, Bruno Nunes, Laszlo Rusko, +9 more

Organizations: GE Healthcare, Seattle, USA · GE Healthcare, Budapest, Hungary · Amazon, USA

Abstract

Recent advances in large-scale foundation models have enabled strong generalization across a wide range of clinical and research tasks in magnetic resonance imaging (MRI). However, existing vision-language MRI models remain limited in anatomy-specific localization and pathology-aware representation, particularly in real-world settings. We introduce A4-MR, a 3D vision-language foundation model trained on approximately 50,000 MRI studies comprising 491,000 series, roughly half of which are paired with radiology reports. A4-MR is designed to: (1) learn anatomy localization through large-scale pretraining with noisy labels; (2) align image and text representations at the anatomical level; and (3) learn distinct representations of healthy and abnormal anatomy. Across a diverse set of downstream tasks, A4-MR outperforms existing foundation models, including a +1.7 AUROC improvement over MedSigLIP for abnormality detection. Using a frozen encoder, A4-MR also surpasses an end-to-end task-specific MedNeXt model for knee segmentation by +1.4 Dice and performs comparably to the best task-specific model for abdominal organ segmentation on AMOS, within 0.4 Dice. Finally, without anatomical prompts, A4-MR localizes 3D anatomy comparably to MedSAM2 provided with ground-truth bounding boxes. Together, these results demonstrate that anatomy-level multimodal alignment and abnormality-aware representation learning can enable a single MRI foundation model to support localization, segmentation, and pathology-related tasks.

Figures & tables

Panel of Figure 1 — Figure 1: Overview. (a) Pretraining data: MRI studies per body region (bubble size on a log scale), MR sequence types, scanner vendors, and patient age and sex (all counted per study). (b) Weak anatomy boxes from segmentation models and anatomy-level findings extracted from reports by an LLM supervise a two-stage pretraining: self-supervised image (DINO student–teacher on 3D crops) and text (masked language modeling) encoders (stage 1), then localized, abnormality-aware image–text alignment (stage 2, Fig. 2 ). (c) Evaluation with a frozen encoder: retrieval and localization zero-shot; abnormality detection, segmentation and report generation with lightweight task-specific modules.

Panel of Figure 2 — Figure 2: Cross-modal localized alignment. A 3D ViT encoder and a DETR decoder detect anatomies in each MR volume. Each decoder output feeds three heads: class and 3D box, trained with the Hungarian-matched DETR set loss, and a projection that gives one embedding per detected anatomy. Similarly, report findings are extracted per anatomy from the radiology reports and are encoded by a text encoder. The abnormality-aware contrastive loss $\mathcal{L}_{\mathrm{CLIP}}$ uses pairwise target matrix where the same organ in two images is a positive pair only if both are normal (thalamus); an abnormal anatomy is positive only with its own finding (chiasma, right eye). The image-level CLS–text contrastive loss is omitted for clarity. (best in color)

	Entire Report				Organ Descriptions
Model	@1	@3	@5	mAP	@1	@3	@5	mAP
A 4 -MR (ours)	93.27	86.52	84.13	90.13	85.42	74.51	68.74	77.58
Decipher-MR	90.90	79.60	71.90	77.80	86.60	72.30	62.80	66.70
BiomedCLIP	81.90	69.80	64.40	70.90	66.40	48.80	41.00	51.70
MedImageInsight	78.00	65.10	58.10	71.70	73.00	57.30	50.50	60.70
No Abnormality	88.25	82.49	78.87	84.38	81.47	68.71	60.76	70.66

Table 1: Image-to-Text Performance on body region retrieval task. Bold: best; underlined: second best.

	Entire Report				Organ Descriptions
Model	@1	@3	@5	mAP	@1	@3	@5	mAP
A 4 -MR (ours)	96.49	94.60	92.70	88.32	86.93	82.58	79.53	78.42
Decipher-MR	94.60	91.40	89.70	78.10	84.50	78.80	75.10	73.80
BiomedCLIP	83.50	72.40	68.10	75.50	66.00	52.70	47.40	62.40
MedImageInsight	91.90	81.10	74.90	77.20	73.90	63.70	56.90	66.80
No Abnormality	97.30	91.89	87.84	81.16	79.92	71.28	65.65	72.24

Table 2: Text-to-Image Retrieval Performance. Bold: best; underlined: second best.

Figure 3: UMAP [ 24 ] plot of the full CirrMRI600+ dataset with A 4 -MR (ours) model embeddings. Colors indicate the health status of the sample. Shapes indicate the MRI pulse sequence of the image that we have found to be an important factor for the model.

	Abnormality detection (AUROC)
Model	10%	25%	50%	100%
A 4 -MR (ours)	94.7 $\pm$ 6.8	97.2 $\pm$ 2.1	98.8 $\pm$ 0.7	98.9 $\pm$ 0.3
Decipher-MR	86.8 $\pm$ 4.8	89.7 $\pm$ 3.5	92.4 $\pm$ 2.3	94.5 $\pm$ 0.6
BiomedParse	80.0 $\pm$ 8.1	87.2 $\pm$ 6.0	89.6 $\pm$ 2.7	90.7 $\pm$ 0.3
Vista3D CT-MR	78.9 $\pm$ 8.1	87.8 $\pm$ 3.7	92.2 $\pm$ 1.7	93.2 $\pm$ 0.7
MRI-CORE	69.4 $\pm$ 5.4	73.0 $\pm$ 4.7	71.9 $\pm$ 3.7	72.5 $\pm$ 0.7

Table 3: Comparison of models on CirrMRI600+ abnormality detection tasks with various data budgets (mean

\pm

std). All values are mean of 10 repeated runs with different random seeds. Bold: best; underlined: second best.

	CLS	Mean organ emb	Random PE	Abnormality loss	Liver emb	Pred crop CLS	GT crop CLS	Abnormality detection (AUROC)
Model								10%	25%	50%	100%
A 4 -MR (ours)	✓			✓				94.7 $\pm$ 6.8	97.2 $\pm$ 2.1	98.8 $\pm$ 0.7	98.9 $\pm$ 0.3
Mean organ		✓		✓				89.9 $\pm$ 5.7	94.6 $\pm$ 3.2	97.9 $\pm$ 1.3	99.4 $\pm$ 0.2
Random PE			✓	✓				80.5 $\pm$ 9.7	92.6 $\pm$ 4.8	96.5 $\pm$ 1.1	97.4 $\pm$ 0.7
Liver				✓	✓			87.7 $\pm$ 5.9	94.0 $\pm$ 3.1	97.0 $\pm$ 2.0	96.9 $\pm$ 0.5
Liver crop CLS				✓		✓		95.5 $\pm$ 3.7	96.8 $\pm$ 2.4	99.1 $\pm$ 0.6	99.7 $\pm$ 0.2

Table 4: Ablation studies on the CirrMRI600+ abnormality detection task (AUROC, mean

\pm

std over 10 runs). Checkmarks indicate experiment setup. A 4 -MR model results are repeated from Tab. 3 . Bold: best; underlined: second best. GT crop CLS rows use the ground-truth liver mask (upper bound) and are not ranked.

Model	AMOS	Dataset A (pelvis)	CirrMRI600+
A 4 -MR (ours)	85.3	87.6	96.5
Decipher-MR	85.7	84.3	96.3
nnUNet3D	85.6	88.7	84.8

Table 5: Segmentation Dice scores (%, higher is better). Bold: best; underlined: second best.

Figure 4: Qualitative assessment from an external clinical case never seen by the models during training. (A) 3D rendering of segmentation outputs from the reference standard and all models in axial views. (B), (C),(D)and (E) show different sagittal slices from the same case with the respective overlaid segmentations for femoral, patellar and tibial cartilages and the meniscus horns. In (B) and (C), strong agreement is observed on sagittal slices through the center of the structures. However, in peripheral slices (D) and (E), models begin to disagree, particularly for femoral cartilage and the meniscus. Cyan arrows show bigger missegmentation errors.

Model	Femoral Cartilage	Tibial Cartilage	Patellar Cartilage	Meniscus	Average
A 4 -MR (ours)	88.2	87.2	89.0	85.0	87.3
MedNeXt	86.6	85.0	87.4	84.4	85.9
VNet	86.4	85.0	88.0	84.1	85.9
nnUNet3D	79.9	79.2	86.8	79.7	81.4

Table 6: Knee segmentation performance (Dice in %, higher is better). Bold: best; underlined: second best.

Figure 5: Label efficiency on knee segmentation: mean Dice over femoral, tibial and patellar cartilage and the meniscus versus the number of labeled training volumes (log scale). A 4 -MR uses a frozen encoder; VNet is trained end-to-end. Curves (filled markers) are on the validation set. At the full training set, the inset compares these validation end points with test-set results from Table 6 (hollow markers for A 4 -MR and VNet), including MedNeXt and nnUNet3D, which were trained on the full set only.

Model	Bounding-box IoU
A 4 -MR (zero-shot)	66.3
No Abnormality (zero-shot)	65.0
MedSAM2 (1 box)	65.5
MedSAM2 (3 slices)	74.8
nnUNet3D (AMOS-trained)	82.4

Table 7: Mean 3D bounding-box IoU (%) on AMOS22 MR validation after excluding unreliable (image, organ) pairs (

n{=}247

). Scores are averaged over images per organ, then over the 13 organs. MedSAM2 is prompted with ground-truth boxes on one slice (1 box) or on three slices per organ (3 slices). nnUNet3D is trained on AMOS; all other models are evaluated zero-shot. Bold: best; underlined: second best among the models above the rule; the AMOS-trained reference is not ranked.

Panel of Figure 6 — Figure 6: Ground-truth and predicted 3D bounding boxes on AMOS22 MR validation for the volumes with the best, median and worst mean bounding-box IoU, averaged over all models and organs (columns), for five organs (rows). Each panel is the axial slice on which the ground-truth mask best fills its bounding box. The ground-truth mask (blue), the ground-truth box (white) and each model’s box (colored) are projected along the head–foot axis onto this slice. Numbers at the bottom of each panel give the bounding-box IoU per model (A 4 : A 4 -MR; NA: No Abnormality; M1, M3: MedSAM2 with one or three box prompts). The right adrenal gland is shown enlarged in an inset. Volumes whose shown organs contain stray label voxels (Supplementary Fig. 9 ) were not considered when selecting these examples.

	Lesion detection			Lesion attributes			Prostate
Model	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$	Size agreement $\uparrow$	Location ROUGE-F1 $\uparrow$	PI-RADS agreement $\uparrow$	Volume error (cc) $\downarrow$
Qwen2.5-VL-3B	45.00	55.33	47.60	52.05	31.34	66.94	1174.48
Qwen2.5-VL-32B	26.00	26.00	26.00	60.00	33.33	75.00	20.00
Decipher-MR + Qwen2.5-3B	53.00	49.02	49.44	64.88	27.88	83.78	20.32
A 4 -MR + Qwen2.5-3B	53.33	49.13	49.64	63.94	24.74	80.74	16.43

Table 8: Prostate Structured Report Generation results. All Qwen models are the Instruct variants. Bold: best; underlined: second best.

Figure 7: Example of LLM-based report processing for a brain MRI study (date jittered). Left: original radiology report. Middle: report processing prompt. Right: processed and restructured report with one entry per anatomy, its findings and an abnormality flag (Abnormal: Yes/No), followed by the conclusion.

#	Anatomy	#	Anatomy	#	Anatomy
1	spleen	43	gluteus_medius_right	85	pallidum_right
2	kidney_right	44	gluteus_minimus_left	86	thalamus_left
3	kidney_left	45	gluteus_minimus_right	87	thalamus_right
4	gallbladder	46	autochthon_left	88	hypothalamus_left
5	liver	47	autochthon_right	89	hypothalamus_right
6	stomach	48	iliopsoas_left	90	ventricle_3rd

Table 9: Anatomy classes supported by the localization decoder.

Figure 8: Architecture of the segmentation decoder

Decoder	Encoder (pre-training crop size)	Curriculum learning	Initialization	Overall Dice (%)
segresnet37	A 4 -MR (384)	No	coldstart	86.00
segresnet37	A 4 -MR (384)	Yes	coldstart	86.20
segresnet37	A 4 -MR (384)	Yes	warmstart	86.67
segresnet37	vit-b + DinoV3 (256)	Yes	coldstart	86.37
segresnet37	vit-L + DinoV3 (256)	Yes	coldstart	86.82

Table 10: Pelvis ablation: Overall Validation Dice (%) on single-split validation. Bold: best; underlined: second best.

Decoder	Encoder	Curriculum	Initialization	Urethra	Penile bulb	Seminal vesicles
segresnet37	A 4 -MR (384)	No	Coldstart	57.00	80.00	80.10
segresnet37	A 4 -MR (384)	Yes	Coldstart	56.02	86.55	79.50
segresnet37	A 4 -MR (384)	Yes	Warmstart	57.52	86.68	81.15
segresnet37	ViT-B + DINOv3 (256)	Yes	Coldstart	56.94	87.11	79.13
segresnet37	ViT-L + DINOv3 (256)	Yes	Coldstart	58.30	87.24	78.97

Table 11: Pelvis ablation: organ Dice (%), Part 1. Bold: best; underlined: second best.

Decoder	Encoder	Curriculum	Initialization	Prostate	Rectum	Bladder
segresnet37	A 4 -MR (384)	No	Coldstart	90.00	91.20	96.00
segresnet37	A 4 -MR (384)	Yes	Coldstart	89.80	91.63	96.06
segresnet37	A 4 -MR (384)	Yes	Warmstart	89.70	91.55	96.25
segresnet37	ViT-B + DINOv3 (256)	Yes	Coldstart	89.47	91.14	95.75
segresnet37	ViT-L + DINOv3 (256)	Yes	Coldstart	89.52	91.90	95.56

Table 12: Pelvis ablation: organ Dice (%), Part 2. Bold: best; underlined: second best.

Decoder	Encoder	Curriculum	Initialization	Bowel bag	Femoral head (L)	Femoral head (R)
segresnet37	A 4 -MR (384)	No	Coldstart	84.60	90.90	85.60
segresnet37	A 4 -MR (384)	Yes	Coldstart	84.17	90.09	85.31
segresnet37	A 4 -MR (384)	Yes	Warmstart	84.83	91.26	85.27
segresnet37	ViT-B + DINOv3 (256)	Yes	Coldstart	82.81	92.76	85.37
segresnet37	ViT-L + DINOv3 (256)	Yes	Coldstart	83.50	94.09	85.24

Table 13: Pelvis ablation: organ Dice (%), Part 3. Bold: best; underlined: second best.

Decoder	Encoder	Curriculum	Initialization	Pelvis body	Hydrogel
segresnet37	A 4 -MR (384)	No	Coldstart	96.80	93.70
segresnet37	A 4 -MR (384)	Yes	Coldstart	95.56	93.47
segresnet37	A 4 -MR (384)	Yes	Warmstart	94.72	94.41
segresnet37	ViT-B + DINOv3 (256)	Yes	Coldstart	96.16	93.40
segresnet37	ViT-L + DINOv3 (256)	Yes	Coldstart	96.36	94.30

Table 14: Pelvis ablation: organ Dice (%), Part 4. Bold: best; underlined: second best.

Decoder	Encoder	LR	Initialization	Overall Dice (%)
segresnet37	A 4 -MR (384)	$1\!\times\!10^{-3}$	Coldstart	83.90
segresnet37	ViT-B + DINOv3 (256)	$1\!\times\!10^{-3}$	Coldstart	83.86
segresnet37	ViT-L + DINOv3 (256)	$1\!\times\!10^{-3}$	Coldstart	84.24
segresnet37 †	Decipher-MR (256)	$1\!\times\!10^{-3}$	Coldstart	83.76
segresnet37 †	A 4 -MR (384)	$1\!\times\!10^{-3}$	Coldstart	83.75
segresnet37	A 4 -MR (384)	$1\!\times\!10^{-4}$	Warmstart	84.19

Table 15: AMOS ablation: overall Dice (%) on single-split validation. Overall is computed excluding background, prostate/uterus, and gall bladder. Bold: best; underlined: second best.

Decoder	Encoder	LR	Init.	Spleen	R. Kidney	L. Kidney
segresnet37	A 4 -MR (384)	$1\!\times\!10^{-3}$	Cold	95.37	95.10	95.58
segresnet37	ViT-B + DINOv3 (256)	$1\!\times\!10^{-3}$	Cold	95.51	94.49	95.42
segresnet37	ViT-L + DINOv3 (256)	$1\!\times\!10^{-3}$	Cold	95.93	95.42	95.68
segresnet37 †	Decipher-MR (256)	$1\!\times\!10^{-3}$	Cold	95.86	95.52	95.49
segresnet37 †	A 4 -MR (384)	$1\!\times\!10^{-3}$	Cold	95.87	95.38	95.55
segresnet37	A 4 -MR (384)	$1\!\times\!10^{-4}$	Warm	95.68	95.19	95.65

Table 16: AMOS organ Dice (%), Part 1. Bold: best; underlined: second best.

Decoder	Encoder	LR	Init.	Gall Bladder	Esophagus	Liver
segresnet37	A 4 -MR (384)	$1\!\times\!10^{-3}$	Cold	75.84	77.80	97.04
segresnet37	ViT-B + DINOv3 (256)	$1\!\times\!10^{-3}$	Cold	75.48	77.83	97.07
segresnet37	ViT-L + DINOv3 (256)	$1\!\times\!10^{-3}$	Cold	79.39	76.23	96.99
segresnet37 †	Decipher-MR (256)	$1\!\times\!10^{-3}$	Cold	75.63	77.17	96.76
segresnet37 †	A 4 -MR (384)	$1\!\times\!10^{-3}$	Cold	73.97	77.59	97.06
segresnet37	A 4 -MR (384)	$1\!\times\!10^{-4}$	Warm	78.44	77.04	97.11

Table 17: AMOS organ Dice (%), Part 2. Bold: best; underlined: second best.

Decoder	Encoder	LR	Init.	Stomach	Aorta	Postcava
segresnet37	A 4 -MR (384)	$1\!\times\!10^{-3}$	Cold	89.31	91.83	89.40
segresnet37	ViT-B + DINOv3 (256)	$1\!\times\!10^{-3}$	Cold	89.64	92.31	89.42
segresnet37	ViT-L + DINOv3 (256)	$1\!\times\!10^{-3}$	Cold	88.03	92.04	89.29
segresnet37 †	Decipher-MR (256)	$1\!\times\!10^{-3}$	Cold	89.93	92.16	89.42
segresnet37 †	A 4 -MR (384)	$1\!\times\!10^{-3}$	Cold	89.98	91.93	89.24
segresnet37	A 4 -MR (384)	$1\!\times\!10^{-4}$	Warm	90.56	91.94	89.83

Table 18: AMOS organ Dice (%), Part 3. Bold: best; underlined: second best.

Decoder	Encoder	LR	Init.	Pancreas	R. Adrenal	L. Adrenal
segresnet37	A 4 -MR (384)	$1\!\times\!10^{-3}$	Cold	86.17	60.97	65.80
segresnet37	ViT-B + DINOv3 (256)	$1\!\times\!10^{-3}$	Cold	86.72	60.85	62.94
segresnet37	ViT-L + DINOv3 (256)	$1\!\times\!10^{-3}$	Cold	86.12	62.17	65.62
segresnet37 †	Decipher-MR (256)	$1\!\times\!10^{-3}$	Cold	85.41	58.97	65.09
segresnet37 †	A 4 -MR (384)	$1\!\times\!10^{-3}$	Cold	86.31	60.21	63.51
segresnet37	A 4 -MR (384)	$1\!\times\!10^{-4}$	Warm	85.87	60.62	65.25

Table 19: AMOS organ Dice (%), Part 4. Bold: best; underlined: second best.

Decoder	Encoder	LR	Init.	Duodenum
segresnet37	A 4 -MR (384)	$1\!\times\!10^{-3}$	Cold	70.51
segresnet37	ViT-B + DINOv3 (256)	$1\!\times\!10^{-3}$	Cold	72.45
segresnet37	ViT-L + DINOv3 (256)	$1\!\times\!10^{-3}$	Cold	72.27
segresnet37 †	Decipher-MR (256)	$1\!\times\!10^{-3}$	Cold	71.48
segresnet37 †	A 4 -MR (384)	$1\!\times\!10^{-3}$	Cold	72.17
segresnet37	A 4 -MR (384)	$1\!\times\!10^{-4}$	Warm	71.30

Table 20: AMOS organ Dice (%), Part 5. Bold: best; underlined: second best.

	Dice (%) $\uparrow$		mHD (mm) $\downarrow$
Structure	No bottleneck	Bottleneck	No bottleneck	Bottleneck
Femoral cartilage	78.8 $\pm$ 9.3	78.3 $\pm$ 9.4	0.519 $\pm$ 0.682	0.540 $\pm$ 0.733
Tibial cartilage	78.1 $\pm$ 9.8	77.7 $\pm$ 9.6	0.581 $\pm$ 0.767	0.589 $\pm$ 0.798
Patellar cartilage	80.0 $\pm$ 11.1	80.0 $\pm$ 10.8	0.586 $\pm$ 1.601	0.592 $\pm$ 1.516
Menisci	77.8 $\pm$ 12.9	77.7 $\pm$ 12.2	0.916 $\pm$ 2.131	0.849 $\pm$ 1.925
All	78.7 $\pm$ 10.9	78.4 $\pm$ 10.6	0.651 $\pm$ 1.436	0.642 $\pm$ 1.345

Table 21: Decoder size ablation on KneeSeg with the frozen stage-1 encoder: SegResNet decoder without and with bottleneck blocks. Dice (%, higher is better) and mean Hausdorff distance (mHD, mm, lower is better), mean

\pm

std over volumes. Bold: better of the two.

Figure 9: Ground-truth labels in AMOS with stray voxels disconnected from the organ (red, circled). The bounding box of all label voxels (white) is much larger than the box of the organ itself (green). Blue: organ on the shown slice (filled) and its outline projected over all slices. Each panel shows the full slice through the stray voxels. These (image, organ) pairs were excluded from the evaluation.

Model	All pairs ( $n{=}259$ )	After exclusion ( $n{=}247$ )	$\Delta$
A 4 -MR (zero-shot)	65.1	66.3	+1.2
No Abnormality	63.8	65.0	+1.2
MedSAM2 (1 box)	64.6	65.5	+0.9
MedSAM2 (3 slices)	74.0	74.8	+0.8
nnUNet3D (AMOS-trained)	81.0	82.4	+1.4

Table 22: Mean 3D bounding-box IoU (%) on AMOS22 MR validation, before and after excluding unreliable (image, organ) pairs. Scores are averaged over images per organ, then over the 13 organs. MedSAM2 is prompted with ground-truth boxes on one slice (1 box) or on three slices per organ (3 slices). nnUNet3D is trained on AMOS; all other models are evaluated zero-shot. Bold: best; underlined: second best among the models above the rule; the AMOS-trained reference is not ranked.

Organ	$n$	A 4 -MR (ours)	No Abnormality	MedSAM2 (1 box)	MedSAM2 (3 slices)	nnUNet3D (AMOS)
Spleen	20	74.5	72.4	92.3	94.3	94.8
Kidney R	20	76.4	74.8	96.0	96.1	93.7
Kidney L	18	75.8	75.4	96.1	96.6	98.3
Gallbladder	18	52.0	52.0	70.1	75.4	68.8
Esophagus	20	61.2	59.8	38.4	54.0	79.3
Liver	20	81.1	79.2	90.1	90.1	97.3

Table 23: Per-organ bounding-box IoU (%) on AMOS MR validation after excluding unreliable (image, organ) pairs;

n

is the number of images per organ. MedSAM2 is prompted with ground-truth boxes on one slice (1 box) or three slices per organ (3 slices). The nnUNet3D is trained on AMOS. Bold: best; underlined: second best per organ; the AMOS-trained nnUNet3D is a reference and is not ranked.

	PICAI-Crop			PICAI
Model	10%	25%	100%	10%	25%	100%
A 4 -MR (256)	$70.7\pm 4.4$	$71.2\pm 4.7$	$74.1\pm 4.1$	$59.5\pm 5.8$	$63.2\pm 5.8$	$68.1\pm 5.1$
A 4 -MR (384)	$68.1\pm 4.6$	$70.0\pm 4.9$	$73.6\pm 4.1$	$58.3\pm 5.9$	$62.3\pm 5.6$	$66.7\pm 4.8$
A 4 -MR (1st stage)	$68.6\pm 4.7$	$71.7\pm 3.8$	$75.2\pm 4.2$	$61.6\pm 5.9$	$65.8\pm 5.5$	$69.6\pm 4.6$

Table 24: PICAI Disease Classification (Cropped & Whole Images), AUROC (%, mean

\pm

std). Bold: best; underlined: second best.

	PICAI-Crop			PICAI
Model	10%	25%	100%	10%	25%	100%
A 4 -MR (256)	$5.35\pm 0.37$	$5.11\pm 0.36$	$4.87\pm 0.33$	$4.91\pm 0.31$	$4.66\pm 0.33$	$4.34\pm 0.29$
A 4 -MR (384)	$5.32\pm 0.35$	$5.10\pm 0.34$	$4.86\pm 0.31$	$5.06\pm 0.32$	$4.80\pm 0.31$	$4.46\pm 0.29$
A 4 -MR (1st stage)	$5.15\pm 0.32$	$4.86\pm 0.29$	$4.53\pm 0.27$	$4.69\pm 0.29$	$4.41\pm 0.28$	$4.09\pm 0.26$

Table 25: PICAI Age Prediction using cropped and whole-prostate images (MAE

\pm

std, lower is better). Bold: best; underlined: second best.

Explore similar work

3D Modality-Aware Pre-training for Vision-Language Model in MRI Multi-organ Abnormality Detection
Feb 27, 2026Haowen Zhu, Ning Yin, Xiaogen ZhouMedical VLMs Vision-Language Pretraining
School of Computer Science and Engineering, Southeast University, China · Department of Medical Imaging, Suzhou Traditional Chinese Medicine Hospital, China · School of Electronic, Electrical Engineering and Physics, Fujian University of Technology
Anatomy Contextualized Adaption of CT Foundation Models
Jul 29, 2026Roshan Kenia, Stephanie L McNamara, William LotterMedical Imaging Foundation Models VLM Adaptation
Department of Biomedical Informatics, Harvard Medical School, Boston, MA · Department of Data Science, Dana-Farber Cancer Institute, Boston, MA · Department of Radiology, Massachusetts General Hospital, Boston, MA +2
How Good are Foundation Models in Longitudinal MRI Disease Progression Reasoning?
Aug 13, 2026Wafa Al Ghallabi, Ritesh Thawkar, Sara Ghaboura +6Radiology VLMs Medical Image Analysis
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Department of Health Abu Dhabi, Abu Dhabi, UAE · Fatima College of Health Sciences, Abu Dhabi, UAE +1

Feb 27, 2026J/K move · Enter open · S save

3D Modality-Aware Pre-training for Vision-Language Model in MRI Multi-organ Abnormality Detection

Haowen Zhu, Ning Yin, Xiaogen Zhou

School of Computer Science and Engineering, Southeast University, China · Department of Medical Imaging, Suzhou Traditional Chinese Medicine Hospital, China · School of Electronic, Electrical Engineering and Physics, Fujian University of Technology

Medical VLMs Vision-Language Pretraining Medical Image Anomaly Detection Modality Gap in VLMs 3D Medical Imaging

Open paper AskPDF arXiv

Vision-language models (VLMs) show strong potential for complex diagnostic tasks in medical imaging. However, applying VLMs to multi-organ medical imaging introduces two principal challenges: (1) modality-specific vision-language alignment and (2) cross-modal feature fusion. In this work, we propose MedMAP, a Medical Modality-Aware Pretraining framework that enhances vision-language representation learning in 3D MRI. MedMAP comprises a modality-aware vision-language alignment stage and a fine-tuning stage for multi-organ abnormality detection. During the pre-training stage, the modality-aware encoders implicitly capture the joint modality distribution and improve alignment between visual and textual representations. We then fine-tune the pre-trained vision encoders (while keeping the text encoder frozen) for downstream tasks. To this end, we curated MedMoM-MRI3D, comprising 7,392 3D MRI volume-report pairs spanning twelve MRI modalities and nine abnormalities tailored for various 3D medical analysis tasks. Extensive experiments on MedMoM-MRI3D demonstrate that MedMAP significantly outperforms existing VLMs in 3D MRI-based multi-organ abnormality detection. Our code is available at https://github.com/RomantiDr/MedMAP.

Abstract

Explore similar work

3D Modality-Aware Pre-training for Vision-Language Model in MRI Multi-organ Abnormality Detection

Anatomy Contextualized Adaption of CT Foundation Models

How Good are Foundation Models in Longitudinal MRI Disease Progression Reasoning?