Cross-Modal Ultrasound-MRI Learning for Fetal Brain Ventricular Volumetry and Abnormality Screening

Yuhao Huang; Yuanji Zhang; Yuhuan Lu; Dong Ni; P. Ellen Grant; Davood Karimi

eess.IVAug 14, 2026

Cross-Modal Ultrasound-MRI Learning for Fetal Brain Ventricular Volumetry and Abnormality Screening

Authors: Yuhao Huang, Yuanji Zhang, Yuhuan Lu, Dong Ni, P. Ellen Grant, Davood Karimi

Organizations: Department of Radiology, Boston Children’s Hospital, and Harvard Medical School, Boston, USA · Medical Ultrasound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China · Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong, China · Shenzhen Luohu People’s Hospital, Shenzhen, China · School of Artificial Intelligence, Shenzhen University, Shenzhen, China · School of Biomedical Engineering and Informatics, Nanjing Medical University, Nanjing, China

Abstract

Assessment of ventriculomegaly (VM) on fetal brain ultrasound relies primarily on measuring lateral ventricular atrial width on standard planes, which is operator-dependent and may not fully reflect the overall ventricular enlargement. Fetal brain MRI provides more reliable volumetric information but is costly and less accessible for routine use. To address these limitations, we propose VIFBA, an ultrasound video-based framework for fetal brain assessment that predicts MRI-derived lateral ventricular volume, classifies VM severity, and identifies potential non-VM fetal brain abnormalities. Our contribution is three-fold. First, we introduce a joint-embedding predictive architecture (JEPA)-inspired tube latent prediction objective that leverages spatio-temporal coherence in ultrasound videos to enhance representation learning. Second, we develop a contrastive cross-modal alignment strategy that transfers structural information from MRI to ultrasound during training, while requiring ultrasound alone at inference. Third, we augment VIFBA with a training-free vision-language model and retrieval augmentation to verify uncertain predictions and identify potential non-VM fetal brain abnormalities. We validated VIFBA on a large dataset comprising 857 cases (3,196 videos) with paired fetal brain ultrasound and MRI examinations. On held-out test data, VIFBA achieved an MAE of 0.5909 mL and Pearson correlation coefficient of 0.9907 for ventricular volume regression, 0.9400 accuracy for VM severity classification, and an F1 score of 0.7764 for multi-abnormality classification, substantially outperforming single-task baselines, video-based strong competitors, and state-of-the-art foundation models. By enabling MRI-informed volumetric assessment from routine ultrasound alone, VIFBA offers a practical and potentially broadly deployable pathway toward accurate and affordable prenatal brain screening.

Figures & tables

Fig. 1: Overview of the training pipeline of our proposed method.

Fig. 2: Testing pipeline of our proposed method, including case-level testing, and VLM verification and non-VM brain abnormality alerting.

Fig. 3: Volume calculation overview in our data preprocessing stage.

Fig. 4: Details of our dataset. (a) bilateral ventricle volume between fetal subjects in the Developing Human Connection Project [ 69 ] (segmentation results generated using DrawEM [ 70 , 71 ] and quality-controlled by the dHCP team) and ours (segmented by deep models [ 31 , 33 ] ). Together, the scatter plot analysis and expert visual inspection provide further validation of the accuracy of the automatic segmentation method. (b) Statistical distribution of bilateral ventricle volume in non-VM and mild, moderate, and severe VMs across different gestational age groups (weeks). (c) Distribution of case counts across six abnormality categories (A1–A6, details refer to Section IV-A ) among different groups, including non-VM abnormal brain and mild, moderate, and severe VMs. (d) The UpSet plot illustrates the set size of each category (VM and A1–A6) and the quantitative distribution of their complex intersections.

Level	Group	MSE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	Pearson $\uparrow$	Spearman $\uparrow$
Video	G1	0.2531	0.5031	0.4390	0.9535	0.9295
	G2	0.7298	0.8543	0.7338	0.9618	0.8809
	G3	2.5167	1.5864	1.4416	0.9208	0.8684
	G4	7.0826	2.6613	2.2956	0.9611	1.0000
	G5	7.6917	2.7734	2.5630	0.9479	1.0000
	Overall	1.6476	1.2836	0.9364	0.9793	0.9403

TABLE I: Regression performance across different groups using video-level output aggregation and case-level feature aggregation. Red subscripts denote the absolute performance gains of case-level feature aggregation over video-level output aggregation. G1: Healthy brain, G2: Non-VM abnormal brain, G3: Mild VM, G4: Moderate VM, and G5: Severe VM.

	MSE	RMSE	MAE	Pearson	Spearman
I3D	12.6789	3.5607	2.7929	0.8470	0.6547
R(2+1)D	13.0733	3.6157	2.5684	0.8243	0.6678
TimeSformer	11.8981	3.4494	2.6758	0.8906	0.7331
InternVideo2	11.4605	3.3853	2.5386	0.9510	0.8318
VideoMamba	12.0456	3.4707	2.6615	0.8752	0.7496
V-JEPA2	10.8888	3.2998	2.6546	0.8821	0.6937

TABLE II: Method Comparison on the regression task.

	Acc	Pre	Rec	F1	AUC
I3D	0.7000	0.6361	0.7611	0.6610	0.8899
R(2+1)D	0.7000	0.6264	0.7086	0.6450	0.8659
TimeSformer	0.7300	0.6462	0.7196	0.6666	0.9006
InternVideo2	0.7500	0.6605	0.7545	0.6762	0.9030
VideoMamba	0.7300	0.6498	0.7424	0.6670	0.8996
V-JEPA2	0.7700	0.7247	0.7604	0.7208	0.9199

TABLE III: Method Comparison on the classification task.

Fig. 5: Comparisons with statistical significance analysis between VIFBA and competitors on (a) regression and (b) classification tasks. Each cell reports the $p$ -value and the mean gain $\Delta$ of VIFBA.

Fig. 6: Group-wise detailed regression performance, including normal brain (n=45), non-VM abnormal brain (n=24), mild VM (n=19), moderate VM (n=7), and severe VM (n=5). Y-axis represents lateral ventricular volume (mL). For each group, cases are sorted by the ground-truth lateral ventricular volume ( green circles ). The blue circles indicate the video-level predictions, while the red circles show the case-level output.

Method & Module			Regression					Classification
Method & Module			MSE	RMSE	MAE	Pearson	Spearman	Acc	Pre	Rec	F1	AUC
Baseline (Reg)			9.9199	3.1496	2.6122	0.9033	0.7648	/	/	/	/	/
Baseline (Cls)			/	/	/	/	/	0.7900	0.7387	0.8256	0.7568	0.9186
Baseline (Reg+Cls)			6.3064	2.5112	2.0827	0.9707	0.9304	0.8200	0.7310	0.8423	0.7569	0.9495
TLP	CMTA	VLM	MSE	RMSE	MAE	Pearson	Spearman	Acc	Pre	Rec	F1	AUC
✓	✗	✗	2.1778	1.4757	1.0739	0.9882	0.9179	0.8800	0.8002	0.8890	0.8302	0.9601

TABLE IV: Ablation study on our VIFBA framework.

VIFBA	Regression					Classification
VIFBA	MSE	RMSE	MAE	Pearson	Spearman	Acc	Pre	Rec	F1	AUC
PMC-CLIP	1.1804	1.0864	0.8970	0.9893	0.9239	0.8900	0.8398	0.8917	0.8537	0.9296
BiomedCLIP	1.0475	1.0234	0.8300	0.9915	0.9466	0.9100	0.8652	0.9122	0.8849	0.9572
USFM	0.8702	0.9329	0.7904	0.9965	0.9704	0.9400	0.8878	0.9300	0.9053	0.9717
EchoCare	0.9217	0.9600	0.8049	0.9905	0.9400	0.9300	0.8793	0.8523	0.8580	0.9399
Ultrasound-CLIP	0.9013	0.9494	0.7994	0.9942	0.9376	0.9300	0.8747	0.9550	0.9048	0.9717

TABLE V: Comparison of VIFBA with different foundation model-based encoders.

Fig. 7: Pairwise statistical results among ablation variants on (a) regression and (b) classification metrics. Each cell reports the $p$ -value for testing whether the row method significantly outperforms the column method. Boxes with black borders indicate statistically significant improvements ( $p<0.05$ ). M1: Baseline (Reg)/(Cls), M2: Baseline (Reg+Cls), M3: TLP, M4: CMTA, M5: VLM, M6: TLP+CMTA, M7: TLP+VLM, M8: CMTA+VLM, M9: VIFBA.

Fig. 8: Performance variation with different numbers of selected videos per case. Full denotes using all available videos for case-level feature aggregation.

Method		Pre	Rec	F1	HL	EMR
VIFBA-CLS-S1		0.4243	0.6299	0.4575	0.1250	0.3200
VIFBA-CLS-S2		0.4762	0.7036	0.5084	0.1110	0.4100
VIFBA	Claude-Haiku-4.5	0.6129	0.7836	0.6673	0.0630	0.6900
	Gemini-3.1-Pro	0.6517	0.8027	0.7053	0.0530	0.7000
	Qwen3.5-Plus	0.6485	0.7847	0.6861	0.0590	0.7100
	GPT-5.4	0.6333	0.8346	0.6941	0.0610	0.7100

TABLE VI: Comparison of VIFBA with baselines and different VLMs on the multi-label classification task. S1: one-stage classification over 10 categories (normal brain, mild/moderate/severe VM, and A1–A6); S2: two-stage classification that first identifies abnormal brains and then determines their specific diseases.

Fig. 9: Multi-seed performance comparison of VIFBA with different foundation encoders. E1–E6 represent PMC-CLIP, BiomedCLIP, USFM, EchoCare, Ultrasound-CLIP, and FetalCLIP, respectively. Statistical significance is denoted as * ( $p<0.05$ ), ** ( $p<0.01$ ), *** ( $p<0.001$ ), and ns (not significant).

Fig. 10: Visualization of selected test cases. For each case, the figure shows example frames from the input ultrasound videos, an axial slice from the reconstructed MRI volume, and ventricular segmentation results obtained from the MRI based on BOUNTI and FeTA segmentors. Note that, to facilitate visualization of the ventricular shape, we have arbitrarily rotated the ventricular segmentation masks for each case. In addition, we show the ground-truth (GT, based on the segmentations/reports and checked by an expert) and VIFBA-predicted (pred) ventricular volumes and diagnostic labels.

Fig. 11: Two typical examples showing the VM verification and multi-abnormality alerting process.

Explore similar work

WASP: Weakly Aligned Spatiotemporal Pairs for Fetal Brain MRI-Ultrasound Learning
Oct 3, 2026Francesco Correnti, Gabriele Magrini, Marco Mistretta +6Cross-Modal Alignment Neuroimaging
University of Florence, Italy · University of Trento, Italy · Meyer Children’s Hospital IRCCS, Florence, Italy
MVSegNet: A Lightweight Boundary-Aware Network for Fetal Lateral Ventricle Segmentation and Atrial Width Estimation in Prenatal Ultrasound
Jun 5, 2026Arafat Hossain SayemUltrasound Image Segmentation Ultrasound Imaging
Automated Fetal Brain MRI Biometry in Healthy and Pathological Cases
Aug 16, 2026Ema Masterl, Tina Vipotnik Vesnaver, Nejc Šubič +1Brain MRI Medical Image Localization
Faculty of Medicine, University of Ljubljana, Ljubljana, Slovenia · University Medical Centre Ljubljana, Ljubljana, Slovenia · Faculty of Electrical Engineering, University of Ljubljana, Ljubljana, Slovenia

Oct 3, 2026J/K move · Enter open · S save

WASP: Weakly Aligned Spatiotemporal Pairs for Fetal Brain MRI-Ultrasound Learning

Francesco Correnti, Gabriele Magrini, Marco Mistretta, Niccolò Biondi+5

University of Florence, Italy · University of Trento, Italy · Meyer Children’s Hospital IRCCS, Florence, Italy

Cross-Modal Alignment Neuroimaging Ultrasound Imaging Magnetic Resonance Imaging

Open paper AskPDF arXiv

Magnetic Resonance Imaging (MRI) is widely regarded as the optimal sensor for fetal brain analysis due to its superior soft-tissue contrast and anatomical detail. However, its high cost and operational burden make it invasive and difficult to obtain at scale. Ultrasound (US), in contrast, is cheap, safe, and routinely acquired, and as a result it has produced substantially larger datasets and a growing ecosystem of pretrained models. This asymmetry raises a natural question: Can we teach a US-only model to understand fetal MRI from only a limited set of examples? The standard recipe, training a foundation model on subject-to-subject paired MRI-US scans, is not viable since no such paired fetal dataset is publicly available. In this paper we address this gap with Weakly Aligned Spatiotemporal Pairs (WASP), a framework that formulates cross-modal correspondence as an entropic Optimal Transport problem driven by clinical metadata, in particular Gestational Age (GA) and diagnostic planes, enabling the fitting of a lightweight alignment module that lifts MRI representations into the US latent space, without fine-tuning the backbone. Empirically, WASP yields its largest gains when MRI is unseen by the model during pretraining (on USFM, GA estimation error drops from 21.9 to 17.4 days and standard plane classification accuracy climbs from 61.9% to 69.0%), while providing smaller, backbone-dependent refinements for backbones pretrained on both modalities (e.g., BioMedParse GA estimation error from 6.5 to 6.0 days and SAM-Med2d plane accuracy from 83.3% to 88.1%). Code is available at https://github.com/miccunifi/WASP.

Abstract

Explore similar work

WASP: Weakly Aligned Spatiotemporal Pairs for Fetal Brain MRI-Ultrasound Learning

MVSegNet: A Lightweight Boundary-Aware Network for Fetal Lateral Ventricle Segmentation and Atrial Width Estimation in Prenatal Ultrasound

Automated Fetal Brain MRI Biometry in Healthy and Pathological Cases