OrthoDiffusion: A Generalizable Multi-Task Diffusion Foundation Model for Musculoskeletal MRI Interpretation

Tian Lan; Lei Xu; Zimu Yuan; Shanggui Liu; Jiajun Liu; Jiaxin Liu; Weilai Xiang; Hongyu Yang

cs.CVFeb 24, 2026

OrthoDiffusion: A Generalizable Multi-Task Diffusion Foundation Model for Musculoskeletal MRI Interpretation

Authors: Tian Lan, Lei Xu, Zimu Yuan, Shanggui Liu, Jiajun Liu, Jiaxin Liu, Weilai Xiang, Hongyu Yang, +3 more

Organizations: Center for Applied Statistics and School of Statistics, Renmin University of China, Beijing, China. · Department of Sports Medicine, Peking University Third Hospital, Institute of Sports Medicine of Peking University, Beijing, China. · Beijing Key Laboratory of Research and Translation for Drugs and Medical Devices in Precision Diagnosis and Treatment of Sports Injuries, Beijing, China. · Engineering Research Center of Sports Trauma Treatment Technology and Devices, Ministry of Education, Beijing, China. · State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China. · School of Computer Science and Engineering, Beihang University, Beijing, China. · Institute of Artificial Intelligence, Beihang University, Beijing, China.

Abstract

Musculoskeletal disorders represent a significant global health burden and are a leading cause of disability worldwide. While MRI is essential for accurate diagnosis, its interpretation remains exceptionally challenging. Radiologists must identify multiple potential abnormalities within complex anatomical structures across different imaging planes, a process that requires significant expertise and is prone to variability. We developed OrthoDiffusion, a unified diffusion-based foundation model designed for multi-task musculoskeletal MRI interpretation. The framework utilizes three orientation-specific 3D diffusion models, pre-trained in a self-supervised manner on 15,948 unlabeled knee MRI scans, to learn robust anatomical features from sagittal, coronal, and axial views. These view-specific representations are integrated to support diverse clinical tasks, including anatomical segmentation and multi-label diagnosis. Our evaluation demonstrates that OrthoDiffusion achieves excellent performance in the segmentation of 11 knee structures and the detection of 8 knee abnormalities. The model exhibited remarkable robustness across different clinical centers and MRI field strengths, consistently outperforming traditional supervised models. Notably, in settings where labeled data was scarce, OrthoDiffusion maintained high diagnostic precision using only 10% of training labels. Furthermore, the anatomical representations learned from knee imaging proved highly transferable to other joints, achieving strong diagnostic performance across 11 diseases of the ankle and shoulder. These findings suggest that diffusion-based foundation models can serve as a unified platform for multi-disease diagnosis and anatomical segmentation, potentially improving the efficiency and accuracy of musculoskeletal MRI interpretation in real-world clinical workflows.

Figures & tables

Figure 1: Overview of the OrthoDiffusion framework. a , Multi-planar musculoskeletal MRI acquisition, including sagittal, coronal, and axial views, with anisotropic resolution across slices. Dataset construction and task composition across joints are illustrated. b , Unconditional 3D diffusion pretraining using a 3D U-Net noise predictor on large-scale knee MRI data. c , Feature extraction from intermediate diffusion representations at selected timesteps and bottleneck blocks, followed by pooling and multi-label classification. Feature-level and label-level fusion strategies are applied to integrate sagittal, coronal, and axial representations. d , Anatomical segmentation pipeline using diffusion representations coupled with a lightweight segmentation head. e , Multimodal fusion strategy integrating MRI diffusion representations with structured electronic health record (EHR) data for diagnosis. f , Diagnostic performance (AUROC) across 19 musculoskeletal abnormalities involving the knee, ankle, and shoulder joints, compared with baseline CNN models.

Task	Joint	Structure / Abnormality (Abbreviation)
Segmentation	Knee	Femur
		Tibia
		Patella
		Femoral cartilage (FC)
		Medial tibial cartilage (MTC)
		Lateral tibial cartilage (LTC)

Table 1: Unified definitions of anatomical structures and abnormalities for musculoskeletal MRI tasks. Abbreviations are defined separately for segmentation and classification.

Orientation	Methods	Femur	Tibia	Patella	FC	MTC	LTC	PC	MM	LM	ACL	PCL
Sagittal	3D-Unet	91.91	90.67	82.25	64.45	50.27	45.54	65.59	50.71	53.13	46.49	51.40
	UNETR	83.32	73.90	57.05	44.84	40.99	26.85	48.54	30.12	21.14	30.74	15.30
	Ours (FT, FT-Opt-Sag)	93.02	92.54	84.58	58.50	63.76	63.07	64.13	60.69	56.03	53.67	58.34
Coronal	3D-Unet	90.87	90.54	–	61.12	68.51	64.63	–	63.49	58.40	45.72	42.19
	UNETR	87.30	84.37	–	58.87	57.61	49.50	–	46.87	43.86	38.96	46.66
	Ours (FT, FT-Opt-Cor)	92.91	92.82	–	64.21	69.73	69.63	–	68.37	65.33	64.13	60.06

Table 2: Segmentation performance of OrthoDiffusion under less than 30% labeled data (Dice Similarity Coefficient, %) for 11 knee anatomical structures on sagittal and coronal MRI scans. Entries marked as “–” indicate anatomical structures that are not visible in the coronal view (e.g., Patella and PC). Boldface indicates the best performance under the same setting in each column.

Figure 2: Qualitative and quantitative evaluation of OrthoDiffusion for knee anatomical segmentation. a , Label efficiency of OrthoDiffusion for knee anatomical segmentation on the Sagittal plane , illustrated by representative Dice Similarity Coefficient curves for selected anatomical structures under progressively reduced supervision. b , Qualitative multi-planar segmentation results. Representative sagittal and coronal knee MRI slices showing the original input images, corresponding model-predicted anatomical segmentations (e.g., femur, tibia, cartilage, etc.), and manual ground-truth annotations, illustrating accurate delineation of key joint structures across imaging planes.

Dataset	Modality	Methods	ACL	PCL	MM	LM	MCL	LCL	PD	EFFU	Macro-AUC
Center A+B+ C+D+E+F+G	MRI-only	3D-Unet	94.35	84.76	70.91	68.76	83.44	79.13	94.67	85.03	82.63
		3D-ResNet-18	94.91	85.00	74.87	70.74	83.71	76.63	96.17	83.21	83.15
		Ours (LP, LP-Opt)	95.15	91.26	81.37	76.23	89.08	82.58	98.85	86.34	87.61
		Ours (FT, FT-Opt)	97.04	93.73	87.24	85.02	91.91	85.77	99.26	86.35	90.79
	EHR-only	EHR-only	52.37	53.22	48.77	50.19	54.24	62.02	57.88	55.99	54.33
	EHR + MRI	Ours (LP, LP-Opt)	95.24	91.23	81.40	76.38	89.24	84.29	98.84	86.84	87.93

Table 3: Comparison of per-class AUROC (%) for the eight-label knee injury prediction task using different modalities (MRI-only, EHR-only, and MRI+EHR) on the Center A+B+C+D+E+F+G test set. Boldface indicates the best performance under the same setting for each column.

Field Strength	Methods	ACL	PCL	MM	LM	MCL	LCL	PD	EFFU	Macro-AUC
1.5T	3D-Unet	94.06	84.06	70.60	71.42	84.72	82.76	91.44	88.13	83.40
	3D-ResNet-18	93.76	85.50	75.13	71.75	86.35	82.44	94.84	85.64	84.43
	Ours (FT, FT-Opt)	96.87	94.99	88.49	83.50	93.66	89.88	98.78	88.96	91.89
3T	3D-Unet	94.57	85.31	71.06	67.95	83.18	77.24	95.78	84.28	82.42
	3D-ResNet-18	95.76	85.91	74.80	70.51	81.14	75.24	97.03	82.98	82.92
	Ours (FT, FT-Opt)	97.29	93.99	87.07	85.36	91.11	84.54	99.40	85.51	90.53

Table 4: Comparisons of per-disease AUROC (%) for the eight-label knee injury prediction task on the Center A+B+C+D+E+F+G test set with different magnetic field strengths. Boldface indicates the best result under the same setting in each column.

Figure 3: Label efficiency, cross-anatomy transferability, and robustness of OrthoDiffusion. a , Diagnostic performance (AUROC) of OrthoDiffusion on eight knee abnormalities as a function of the proportion of labeled training data, demonstrating label-efficient learning under limited supervision. b , Cross-anatomy generalization performance of OrthoDiffusion on ankle and shoulder abnormality diagnosis, transferred from a diffusion backbone pretrained exclusively on knee MRI. c , Robustness of OrthoDiffusion to variations in MRI magnetic field strength (1.5T and 3.0 T), evaluated on knee MRI acquired at different field strengths.

Figure 4: Orientation-specific fusion expert-weights predicted by MPAE reflect clinical imaging practice. Sankey diagrams showing label-specific fusion weights assigned by the MPAE module across axial, sagittal, and coronal planes for representative (a) knee, (b) shoulder, and (c) ankle abnormalities in a single patient. The relative thickness of each flow indicates the contribution of each imaging orientation to the final prediction, revealing plane preferences that align with established clinical reading protocols for different orientations and pathologies. Representative MRI slices from each orientation are shown for reference.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 1 — Figure 1: Macro-averaged AUROC of the eight-label knee injury prediction task under linear probing on the Center A+B+C test set, based on SAP-processed features extracted from 3D-UNet blocks at different diffusion timesteps across MRI planes.

Characteristic	Training set	Test set	Validation set
Total MRI scans	715	119	60
Centers	2	2	2
Scanner manufacturer
GE	432 (60.42%)	75 (63.03%)	41 (68.33%)
Siemens	–	3 (2.52%)	1 (1.67%)
Philips	–	–	–

Appendix

Table 1: Summary statistics of knee MRI datasets for sagittal anatomical segmentation.

Characteristic	Training set	Test set	Validation set
Total MRI scans	802	134	66
Centers	2	2	2
Scanner manufacturer
GE	582 (72.57%)	85 (63.43%)	38 (57.58%)
Siemens	21 (2.62%)	3 (2.24%)	5 (7.58%)
Philips	–	–	–

Appendix

Table 2: Summary statistics of knee MRI datasets for coronal anatomical segmentation.

Characteristic	Training set	Test set	Validation set
Total MRI scans	2178	256	128
Centers	3	3	3
Scanner manufacturer
GE	741 (34.02%)	85 (33.20%)	31 (24.22%)
Siemens	865 (39.72%)	114 (44.53%)	94 (73.44%)
Philips	–	–	–

Appendix

Table 4: Summary statistics of ankle MRI datasets for abnormality classification.

Characteristic	Training set	Test set	Validation set
Total MRI scans	7167	1193	597
Centers	5	5	5
Scanner manufacturer
GE	2028 (28.30%)	531 (44.51%)	1 (0.17%)
Siemens	923 (12.88%)	–	256 (42.88%)
Philips	–	–	–

Appendix

Table 5: Summary statistics of shoulder MRI datasets for abnormality classification.

Orientation	Pooling	ACL	PCL	MM	LM	MCL	LCL	PD	EFFU
Sagittal	GAP	68.83	56.85	60.21	61.51	75.04	70.72	82.73	78.89
	GLP	92.88	84.42	73.26	66.26	82.00	75.68	95.07	81.93
	SAP	94.76	91.04	75.99	70.54	82.08	74.89	98.81	84.76
Coronal	GAP	74.61	58.33	65.69	61.10	78.96	73.90	84.89	81.84
	GLP	91.71	70.61	70.18	67.26	81.04	75.67	94.37	83.84
	SAP	95.00	83.03	78.94	74.72	87.38	77.53	98.20	83.65

Appendix

Table 6: Ablation study of pooling strategies for the eight-label knee injury prediction task across three orientations on the Center A+B+C test set. To ensure fair comparison across orientations, all features are extracted from block=mid_2 at timestep=30 . Results are reported as per-disease AUROC (%). Boldface indicates the best result under the same setting in each column.

Orientation	Pooling	ACL	PCL	MM	LM	MCL	LCL	PD	EFFU	CP	CR	CF1	OP	OR	OF1
Sagittal	GAP	73.10	12.90	62.95	59.36	33.40	22.71	20.03	73.49	44.63	36.98	35.32	62.56	62.57	62.57
	GLP	94.39	50.03	76.59	64.71	44.69	29.34	61.86	77.80	69.96	46.76	50.99	72.08	66.51	69.19
	SAP	95.94	73.29	79.72	69.11	46.31	28.85	86.25	81.99	69.26	62.31	64.91	73.49	72.04	72.76
Coronal	GAP	77.32	14.06	68.54	58.66	39.35	25.35	20.47	77.22	48.39	38.05	38.37	65.59	62.21	63.86
	GLP	93.00	22.79	73.09	66.47	44.89	24.84	49.99	79.96	68.40	43.61	46.61	71.66	65.09	68.22
	SAP	96.16	50.06	82.84	73.84	63.94	29.86	77.90	80.08	68.60	60.70	63.81	74.84	72.66	73.74

Appendix

Table 7: Ablation study of pooling strategies for the eight-label knee injury prediction task across three orientations on the Center A+B+C test set. To ensure fair comparison across orientations, all features are extracted from block=mid_2 at timestep=30 . Results are reported as per-disease average precision and overall multi-label metrics (%). Boldface indicates the best result under the same setting in each column.

Methods	ACL	PCL	MM	LM	MCL	LCL	PD	EFFU
Simple Concat	96.62	92.47	81.32	77.24	89.98	82.38	99.49	86.08
Linear Concat	96.52	92.46	81.05	76.94	89.67	81.26	99.42	85.50
Linear Add	96.54	92.47	80.93	76.86	89.62	81.14	99.42	85.27
Cross Attention	96.54	92.22	80.63	76.38	89.43	80.11	99.19	84.50
MPAE	96.54	92.08	80.86	76.06	89.70	82.29	99.46	86.12

Appendix

Table 8: Comparison of different representation fusion strategies for the eight-label knee injury prediction task on the Center A+B+C test dataset. For timestep–block selection settings, both are block=mid_2 and timestep=30 . Results are reported in terms of per-disease AUROC(%). Boldface indicates the best result in each column.

Methods	ACL	PCL	MM	LM	MCL	LCL	PD	EFFU	CP	CR	CF1	OP	OR	OF1
Simple Concat	97.32	76.97	85.18	76.79	69.04	36.00	92.30	83.40	75.24	66.39	70.22	78.62	74.73	76.63
Linear Concat	97.23	77.32	84.91	76.73	68.12	34.06	92.19	82.93	73.67	67.53	70.22	77.09	75.39	76.23
Linear Add	97.24	76.88	84.76	76.67	67.89	33.91	91.80	82.75	73.31	67.28	69.93	76.92	75.26	76.08
Cross Attention	97.29	75.77	84.45	76.12	68.05	33.08	90.47	81.76	72.48	67.78	69.83	76.17	75.67	75.92
MPAE	97.27	75.12	84.45	75.66	67.77	35.18	91.62	83.58	76.28	62.85	67.89	78.70	73.80	76.17

Appendix

Table 9: Comparison of different representation fusion strategies for the eight-label knee injury prediction task on the Center A+B+C knee dataset. For timestep–block selection settings, both are block=mid_2 and timestep=30 . Results are reported in terms of per-disease AP and overall multi-label metrics (%). Boldface indicates the best result in each column.

Task	Anatomy	Setting	Sagittal		Coronal		Axial
Task	Anatomy	Setting	Timestep	Block	Timestep	Block	Timestep	Block
Classification	Knee	LP-Opt	50	mid_0	100	mid_2	50	mid_0
	Knee	FT-Opt	150	mid_0	50	mid_2	200	mid_0
	Ankle	FT-A-Opt	300	mid_0	100	mid_1	30	mid_0
	Shoulder	FT-S-Opt	200	mid_2	30	mid_2	100	mid_1
Segmentation	Knee	FT-Opt-Sag/Cor	50	mid_2	50	mid_2	–	–

Appendix

Table 10: Task-specific optimal diffusion timestep and network block selections for different musculoskeletal tasks and anatomical sites.

Dataset	Methods	ACL	PCL	MM	LM	MCL	LCL	PD	EFFU	Macro-AUC
Center A+B+C	3D-Unet	96.28	88.01	72.68	69.56	84.99	79.77	95.94	86.09	84.17
	3D-ResNet-18	96.96	87.56	74.90	70.81	85.00	77.82	97.49	84.41	84.37
	Ours (LP, LP-Opt)	96.53	92.43	81.04	77.73	90.37	82.84	99.53	86.02	88.31
	Ours (FT, FT-Opt)	98.06	94.35	87.31	85.62	92.97	85.72	99.63	86.39	91.26
Center D+E+F+G	3D-Unet	71.21	66.88	73.35	63.49	73.60	65.38	86.06	72.92	71.61
	3D-ResNet-18	76.95	74.38	72.85	64.98	71.55	66.32	90.29	71.71	73.63

Appendix

Table 11: Comparisons of per-disease AUROC (%) for the eight-label knee injury prediction task on the Center A-C or D-G test dataset. Boldface indicates the best result under the same setting in each column.

Methods	ACL	PCL	MM	LM	MCL	LCL	PD	EFFU	CP	CR	CF1	OP	OR	OF1
3D-Unet	97.14	63.96	76.55	68.80	49.67	30.22	68.18	83.15	68.52	56.39	59.98	73.54	70.88	72.18
3D-Resnet-18	97.70	65.11	79.31	68.65	54.79	29.15	79.76	81.66	68.64	62.35	63.61	71.08	76.35	73.62
Ours (LP, LP-Opt)	97.14	78.41	85.23	77.54	69.88	37.18	92.36	82.73	75.44	67.76	71.11	78.03	75.20	76.59
Ours (FT, FT-Opt)	98.49	83.38	90.23	85.93	75.79	40.37	93.74	83.49	79.48	72.92	75.90	82.36	79.34	80.82

Appendix

Table 12: Comparisons of per-disease AP and overall multi-label metrics (%) for the eight-label knee injury prediction task on the Center A+B+C test dataset. Boldface indicates the best result in each column.

Methods	ACL	PCL	MM	LM	MCL	LCL	PD	EFFU	CP	CR	CF1	OP	OR	OF1
3D-Unet	81.15	32.21	86.10	72.63	52.68	45.46	55.89	95.70	69.35	45.33	53.62	77.85	54.96	64.43
3D-Resnet-18	83.73	33.96	86.59	71.53	50.77	45.84	71.58	95.54	70.25	46.70	53.66	79.57	57.68	66.88
Ours (LP, LP-Opt)	86.06	55.13	94.00	78.25	60.59	54.69	85.25	96.70	79.97	51.23	60.81	82.62	60.24	69.68
Ours (FT, FT-Opt)	87.60	61.76	95.36	85.24	68.63	57.98	84.79	96.40	83.61	56.82	66.18	86.61	65.10	74.33

Appendix

Table 13: Comparisons of per-disease AP and overall multi-label metrics (%) for the eight-label knee injury prediction task on the Center D–G test dataset. Boldface indicates the best result in each column.

Field Strength	Methods	ACL	PCL	MM	LM	MCL	LCL	PD	EFFU	CP	CR	CF1	OP	OR	OF1
1.5T	3D-Unet	95.09	55.03	70.32	66.30	58.32	44.45	63.02	86.98	68.35	55.93	59.82	73.17	64.25	68.42
	3D-ResNet-18	94.55	53.57	76.06	68.19	61.24	46.35	69.50	84.43	67.59	60.24	62.83	73.38	66.14	69.57
	Ours (FT, FT-Opt)	97.66	77.42	89.70	83.84	83.09	68.25	89.00	87.03	83.01	71.81	76.63	82.83	76.73	79.67
3T	3D-Unet	96.30	61.21	77.08	67.72	46.42	24.18	68.26	82.41	67.33	58.63	60.89	72.46	70.90	71.67
	3D-ResNet-18	97.08	63.17	81.07	70.41	49.33	23.17	74.87	81.35	65.95	59.92	61.98	74.50	69.01	71.65
	Ours (FT, FT-Opt)	98.14	83.63	90.62	86.29	71.19	35.60	93.29	84.18	79.12	70.33	74.28	82.77	77.97	80.29

Appendix

Table 14: Comparisons of per-disease AP, overall multi-label metrics (%) on the eight-label knee injury prediction task on the A+B+C+D+E+F+G test dataset with different magnetic field strengths. Boldface indicates the best result under the same setting in each column.

Methods	ATFL	CFL	ATR	OLT	CP	CR	CF1	OP	OR	OF1
3D-Unet	91.94	89.68	32.61	42.37	67.02	49.94	45.77	84.06	78.01	80.92
3D-Resnet-18	93.04	90.00	29.45	54.55	61.45	69.77	63.55	73.93	87.04	79.95
Ours (FT, FT-A-Opt)	95.73	93.49	40.90	72.26	85.46	60.82	60.59	83.36	83.52	83.44

Appendix

Table 15: Comparisons of per-disease AP, overall multi-label metrics (%) for the four-label ankle injury prediction task on the test dataset. Boldface indicates the best result in each column.

Methods	SSP	ISP	SSC	LHBT injury	AC	LHBT effusion	SASD	CP	CR	CF1	OP	OR	OF1
3D-Unet	99.19	80.19	65.05	54.97	74.47	17.17	92.87	65.58	60.19	61.37	79.73	75.86	77.75
3D-Resnet-18	99.32	80.32	66.85	55.59	75.53	18.27	92.59	64.37	63.69	63.08	77.73	79.69	78.70
Ours (FT, FT-S-Opt)	99.54	84.10	74.27	63.20	82.76	30.73	94.06	72.94	66.66	67.86	81.00	79.57	80.28

Appendix

Table 16: Comparisons of per-disease AP, overall multi-label metrics (%) for the seven-label shoulder injury prediction task on the test dataset. Boldface indicates the best result in each column.

Methods	ATFL	CFL	ATR	OLT
3D-Unet	65.31	68.93	74.88	61.06
3D-Resnet-18	66.62	69.16	75.78	68.54
Ours (FT, FT-A-Opt)	77.52	77.86	82.00	80.43

Appendix

Table 17: Comparisons of per-disease AUROC(%) for the four-label ankle injury prediction task on the test dataset. Boldface indicates the best result in each column.

Methods	SSP	ISP	SSC	LHBT injury	AC	LHBT effusion	SASD
3D-Unet	81.72	74.59	73.66	78.25	86.00	65.82	80.25
3D-Resnet-18	83.48	74.92	74.97	78.40	86.30	67.03	79.01
Ours (FT, FT-S-Opt)	88.49	79.08	79.54	81.49	91.03	75.77	83.38

Appendix

Table 18: Comparisons of per-disease AUROC(%) for the seven-label shoulder injury prediction task on the test dataset. Boldface indicates the best result in each column.

Explore similar work

Apr 13, 2026

MosaicMRI: A Diverse Dataset and Benchmark for Raw Musculoskeletal MRI

Deep learning underpins a wide range of applications in MRI, including reconstruction, artifact removal, and segmentation. However, progress has been driven largely by public datasets focused on brain and knee imaging, shaping how models are trained and evaluated. As a result, careful studies of the reliability of these models across diverse anatomical settings remain limited. In this work, we introduce MosaicMRI, a large and diverse collection of fully sampled raw musculoskeletal (MSK) MR measurements designed for training and evaluating machine-learning--based methods. MosaicMRI is the largest open-source raw MSK MRI dataset to date, comprising 3,150 volumes and 89,830 slices, including a 3,096-volume 1.5~~T core dataset and a 54-volume 0.55~~T low-field set. The dataset offers substantial diversity in volume orientation (e.g., axial, sagittal), imaging contrasts (e.g., PD, T1, T2), anatomies (e.g., spine, knee, hip, ankle, and others), numbers of acquisition coils, and field strengths. Using accelerated reconstruction as a testbed, we study scaling, robustness, and data selection under realistic MSK distribution shifts. Across E2E-VarNet, U-Net, and ViT baselines, mixed-anatomy training improves over anatomy-specific training, showing that the benefit of anatomical diversity is not architecture-specific. Zero-shot experiments on the 0.55~T low-field set highlight that using more training data improves average reconstruction under field-strength shift. Controlled subset experiments further suggest that these gains come from transferable cross-anatomy structure, rather than anatomy or contrast matching alone.

Paula Arguello, Berk Tinaz, Mohammad Shahab Sepehri +2

Department of Computer Science, University of Southern California · USC Center on AI Foundations for the Sciences · Department of Electrical and Computer Engineering, University of Southern California +1

Magnetic Resonance Imaging Medical Image Benchmarks Cross-Domain Generalization Accelerated MRI Reconstruction

Ask PDF

Oct 9, 2026

A4-MR: Abnormality-aware Anatomical Alignment for MR

Recent advances in large-scale foundation models have enabled strong generalization across a wide range of clinical and research tasks in magnetic resonance imaging (MRI). However, existing vision-language MRI models remain limited in anatomy-specific localization and pathology-aware representation, particularly in real-world settings. We introduce A4-MR, a 3D vision-language foundation model trained on approximately 50,000 MRI studies comprising 491,000 series, roughly half of which are paired with radiology reports. A4-MR is designed to: (1) learn anatomy localization through large-scale pretraining with noisy labels; (2) align image and text representations at the anatomical level; and (3) learn distinct representations of healthy and abnormal anatomy. Across a diverse set of downstream tasks, A4-MR outperforms existing foundation models, including a +1.7 AUROC improvement over MedSigLIP for abnormality detection. Using a frozen encoder, A4-MR also surpasses an end-to-end task-specific MedNeXt model for knee segmentation by +1.4 Dice and performs comparably to the best task-specific model for abdominal organ segmentation on AMOS, within 0.4 Dice. Finally, without anatomical prompts, A4-MR localizes 3D anatomy comparably to MedSAM2 provided with ground-truth bounding boxes. Together, these results demonstrate that anatomy-level multimodal alignment and abnormality-aware representation learning can enable a single MRI foundation model to support localization, segmentation, and pathology-related tasks.

Noel D'souza, Zhijian Yang, Nishank Singla +14

GE Healthcare, Seattle, USA · GE Healthcare, Budapest, Hungary · Amazon, USA

Medical Imaging Foundation Models Radiology VLMs Magnetic Resonance Imaging Medical Image Localization 3D Medical Image Segmentation Medical Image Anomaly Detection

Ask PDF

Sep 24, 2026

Multi-Resolution Feature Fusion U-Net for Magnetic Resonance Imaging Segmentation

The segmentation of anatomical structures in medical images and particularly in MRI scans, is essential for clinical diagnosis and monitoring disease progression. While Deep Learning (DL) architectures, such as U-Net and its extensions are very effective in medical image segmentation tasks, they often struggle with preserving fine-grained details and global contextual information. This is especially challenging for MRI data segmentation, where anatomical structures are characterized by irregular boundaries and variations in shape, contrast, and scale. To address this challenge, we propose a novel DL architecture for MRI segmentation across different anatomical structures. Specifically, the architecture introduces a module, named Multi-Resolution Feature Fusion (MRFF), that can be easily integrated into any U-Net-like architecture. The MRFF is integrated in all levels of an encode-decoder structure, along with attention mechanisms and skip connections to extract features at multiple resolutions, enabling the model to capture both fine-grained details and global contextual information. We evaluate the MRFFU-Net on two publicly available benchmark MRI datasets of different anatomical targets; one for Cerebrospinal Fluid (CSF) segmentation in spinal MR scans, and one for left atrium cardiac segmentation, from the Medical Segmentation Decathlon (MSD) challenge. Experimental results indicate that MRFFU-Net outperforms state-of-the-art models across multiple evaluation metrics, demonstrating its effectiveness in MRI segmentation.

Eirini Cholopoulou, Dimitrios E. Diamantis, Dimitris K. Iakovidis

Department of Computer Science and Biomedical University of Thessaly Lamia, Greece

Medical Image Segmentation U-Net Multi-Scale Feature Fusion Magnetic Resonance Imaging

Ask PDF