Cross-Modal Ultrasound-MRI Learning for Fetal Brain Ventricular Volumetry and Abnormality Screening
Authors: Yuhao Huang, Yuanji Zhang, Yuhuan Lu, Dong Ni, P. Ellen Grant, Davood Karimi
Organizations: Department of Radiology, Boston Children’s Hospital, and Harvard Medical School, Boston, USA · Medical Ultrasound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China · Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong, China · Shenzhen Luohu People’s Hospital, Shenzhen, China · School of Artificial Intelligence, Shenzhen University, Shenzhen, China · School of Biomedical Engineering and Informatics, Nanjing Medical University, Nanjing, China
Assessment of ventriculomegaly (VM) on fetal brain ultrasound relies primarily on measuring lateral ventricular atrial width on standard planes, which is operator-dependent and may not fully reflect the overall ventricular enlargement. Fetal brain MRI provides more reliable volumetric information but is costly and less accessible for routine use. To address these limitations, we propose VIFBA, an ultrasound video-based framework for fetal brain assessment that predicts MRI-derived lateral ventricular volume, classifies VM severity, and identifies potential non-VM fetal brain abnormalities. Our contribution is three-fold. First, we introduce a joint-embedding predictive architecture (JEPA)-inspired tube latent prediction objective that leverages spatio-temporal coherence in ultrasound videos to enhance representation learning. Second, we develop a contrastive cross-modal alignment strategy that transfers structural information from MRI to ultrasound during training, while requiring ultrasound alone at inference. Third, we augment VIFBA with a training-free vision-language model and retrieval augmentation to verify uncertain predictions and identify potential non-VM fetal brain abnormalities. We validated VIFBA on a large dataset comprising 857 cases (3,196 videos) with paired fetal brain ultrasound and MRI examinations. On held-out test data, VIFBA achieved an MAE of 0.5909 mL and Pearson correlation coefficient of 0.9907 for ventricular volume regression, 0.9400 accuracy for VM severity classification, and an F1 score of 0.7764 for multi-abnormality classification, substantially outperforming single-task baselines, video-based strong competitors, and state-of-the-art foundation models. By enabling MRI-informed volumetric assessment from routine ultrasound alone, VIFBA offers a practical and potentially broadly deployable pathway toward accurate and affordable prenatal brain screening.
Figures & tables
Fig. 1: Overview of the training pipeline of our proposed method.
Fig. 2: Testing pipeline of our proposed method, including case-level testing, and VLM verification and non-VM brain abnormality alerting.
Fig. 3: Volume calculation overview in our data preprocessing stage.
Fig. 4: Details of our dataset. (a) bilateral ventricle volume between fetal subjects in the Developing Human Connection Project [ 69 ] (segmentation results generated using DrawEM [ 70 , 71 ] and quality-controlled by the dHCP team) and ours (segmented by deep models [ 31 , 33 ] ). Together, the scatter plot analysis and expert visual inspection provide further validation of the accuracy of the automatic segmentation method. (b) Statistical distribution of bilateral ventricle volume in non-VM and mild, moderate, and severe VMs across different gestational age groups (weeks). (c) Distribution of case counts across six abnormality categories (A1–A6, details refer to Section IV-A ) among different groups, including non-VM abnormal brain and mild, moderate, and severe VMs. (d) The UpSet plot illustrates the set size of each category (VM and A1–A6) and the quantitative distribution of their complex intersections.
Level
Group
MSE ↓
RMSE ↓
MAE ↓
Pearson ↑
Spearman ↑
Video
G1
0.2531
0.5031
0.4390
0.9535
0.9295
G2
0.7298
0.8543
0.7338
0.9618
0.8809
G3
2.5167
1.5864
1.4416
0.9208
0.8684
G4
7.0826
2.6613
2.2956
0.9611
1.0000
G5
7.6917
2.7734
2.5630
0.9479
1.0000
Overall
1.6476
1.2836
0.9364
0.9793
0.9403
TABLE I: Regression performance across different groups using video-level output aggregation and case-level feature aggregation. Red subscripts denote the absolute performance gains of case-level feature aggregation over video-level output aggregation. G1: Healthy brain, G2: Non-VM abnormal brain, G3: Mild VM, G4: Moderate VM, and G5: Severe VM.
MSE
RMSE
MAE
Pearson
Spearman
I3D
12.6789
3.5607
2.7929
0.8470
0.6547
R(2+1)D
13.0733
3.6157
2.5684
0.8243
0.6678
TimeSformer
11.8981
3.4494
2.6758
0.8906
0.7331
InternVideo2
11.4605
3.3853
2.5386
0.9510
0.8318
VideoMamba
12.0456
3.4707
2.6615
0.8752
0.7496
V-JEPA2
10.8888
3.2998
2.6546
0.8821
0.6937
TABLE II: Method Comparison on the regression task.
Acc
Pre
Rec
F1
AUC
I3D
0.7000
0.6361
0.7611
0.6610
0.8899
R(2+1)D
0.7000
0.6264
0.7086
0.6450
0.8659
TimeSformer
0.7300
0.6462
0.7196
0.6666
0.9006
InternVideo2
0.7500
0.6605
0.7545
0.6762
0.9030
VideoMamba
0.7300
0.6498
0.7424
0.6670
0.8996
V-JEPA2
0.7700
0.7247
0.7604
0.7208
0.9199
TABLE III: Method Comparison on the classification task.
Fig. 5: Comparisons with statistical significance analysis between VIFBA and competitors on (a) regression and (b) classification tasks. Each cell reports the p -value and the mean gain Δ of VIFBA.
Fig. 6: Group-wise detailed regression performance, including normal brain (n=45), non-VM abnormal brain (n=24), mild VM (n=19), moderate VM (n=7), and severe VM (n=5). Y-axis represents lateral ventricular volume (mL). For each group, cases are sorted by the ground-truth lateral ventricular volume ( green circles ). The blue circles indicate the video-level predictions, while the red circles show the case-level output.
Method & Module
Regression
Classification
MSE
RMSE
MAE
Pearson
Spearman
Acc
Pre
Rec
F1
AUC
Baseline (Reg)
9.9199
3.1496
2.6122
0.9033
0.7648
/
/
/
/
/
Baseline (Cls)
/
/
/
/
/
0.7900
0.7387
0.8256
0.7568
0.9186
Baseline (Reg+Cls)
6.3064
2.5112
2.0827
0.9707
0.9304
0.8200
0.7310
0.8423
0.7569
0.9495
TLP
CMTA
VLM
MSE
RMSE
MAE
Pearson
Spearman
Acc
Pre
Rec
F1
AUC
✓
✗
✗
2.1778
1.4757
1.0739
0.9882
0.9179
0.8800
0.8002
0.8890
0.8302
0.9601
TABLE IV: Ablation study on our VIFBA framework.
VIFBA
Regression
Classification
MSE
RMSE
MAE
Pearson
Spearman
Acc
Pre
Rec
F1
AUC
PMC-CLIP
1.1804
1.0864
0.8970
0.9893
0.9239
0.8900
0.8398
0.8917
0.8537
0.9296
BiomedCLIP
1.0475
1.0234
0.8300
0.9915
0.9466
0.9100
0.8652
0.9122
0.8849
0.9572
USFM
0.8702
0.9329
0.7904
0.9965
0.9704
0.9400
0.8878
0.9300
0.9053
0.9717
EchoCare
0.9217
0.9600
0.8049
0.9905
0.9400
0.9300
0.8793
0.8523
0.8580
0.9399
Ultrasound-CLIP
0.9013
0.9494
0.7994
0.9942
0.9376
0.9300
0.8747
0.9550
0.9048
0.9717
TABLE V: Comparison of VIFBA with different foundation model-based encoders.
Fig. 7: Pairwise statistical results among ablation variants on (a) regression and (b) classification metrics. Each cell reports the p -value for testing whether the row method significantly outperforms the column method. Boxes with black borders indicate statistically significant improvements ( p<0.05 ). M1: Baseline (Reg)/(Cls), M2: Baseline (Reg+Cls), M3: TLP, M4: CMTA, M5: VLM, M6: TLP+CMTA, M7: TLP+VLM, M8: CMTA+VLM, M9: VIFBA.
Fig. 8: Performance variation with different numbers of selected videos per case. Full denotes using all available videos for case-level feature aggregation.
Method
Pre
Rec
F1
HL
EMR
VIFBA-CLS-S1
0.4243
0.6299
0.4575
0.1250
0.3200
VIFBA-CLS-S2
0.4762
0.7036
0.5084
0.1110
0.4100
VIFBA
Claude-Haiku-4.5
0.6129
0.7836
0.6673
0.0630
0.6900
Gemini-3.1-Pro
0.6517
0.8027
0.7053
0.0530
0.7000
Qwen3.5-Plus
0.6485
0.7847
0.6861
0.0590
0.7100
GPT-5.4
0.6333
0.8346
0.6941
0.0610
0.7100
TABLE VI: Comparison of VIFBA with baselines and different VLMs on the multi-label classification task. S1: one-stage classification over 10 categories (normal brain, mild/moderate/severe VM, and A1–A6); S2: two-stage classification that first identifies abnormal brains and then determines their specific diseases.
Fig. 9: Multi-seed performance comparison of VIFBA with different foundation encoders. E1–E6 represent PMC-CLIP, BiomedCLIP, USFM, EchoCare, Ultrasound-CLIP, and FetalCLIP, respectively. Statistical significance is denoted as * ( p<0.05 ), ** ( p<0.01 ), *** ( p<0.001 ), and ns (not significant).
Fig. 10: Visualization of selected test cases. For each case, the figure shows example frames from the input ultrasound videos, an axial slice from the reconstructed MRI volume, and ventricular segmentation results obtained from the MRI based on BOUNTI and FeTA segmentors. Note that, to facilitate visualization of the ventricular shape, we have arbitrarily rotated the ventricular segmentation masks for each case. In addition, we show the ground-truth (GT, based on the segmentations/reports and checked by an expert) and VIFBA-predicted (pred) ventricular volumes and diagnostic labels.
Fig. 11: Two typical examples showing the VM verification and multi-abnormality alerting process.
Magnetic Resonance Imaging (MRI) is widely regarded as the optimal sensor for fetal brain analysis due to its superior soft-tissue contrast and anatomical detail. However, its high cost and operational burden make it invasive and difficult to obtain at scale. Ultrasound (US), in contrast, is cheap, safe, and routinely acquired, and as a result it has produced substantially larger datasets and a growing ecosystem of pretrained models. This asymmetry raises a natural question: Can we teach a US-only model to understand fetal MRI from only a limited set of examples? The standard recipe, training a foundation model on subject-to-subject paired MRI-US scans, is not viable since no such paired fetal dataset is publicly available. In this paper we address this gap with Weakly Aligned Spatiotemporal Pairs (WASP), a framework that formulates cross-modal correspondence as an entropic Optimal Transport problem driven by clinical metadata, in particular Gestational Age (GA) and diagnostic planes, enabling the fitting of a lightweight alignment module that lifts MRI representations into the US latent space, without fine-tuning the backbone. Empirically, WASP yields its largest gains when MRI is unseen by the model during pretraining (on USFM, GA estimation error drops from 21.9 to 17.4 days and standard plane classification accuracy climbs from 61.9% to 69.0%), while providing smaller, backbone-dependent refinements for backbones pretrained on both modalities (e.g., BioMedParse GA estimation error from 6.5 to 6.0 days and SAM-Med2d plane accuracy from 83.3% to 88.1%). Code is available at https://github.com/miccunifi/WASP.
Francesco Correnti, Gabriele Magrini, Marco Mistretta +6
University of Florence, Italy · University of Trento, Italy · Meyer Children’s Hospital IRCCS, Florence, Italy
Fetal ventriculomegaly is assessed by measuring the atrial width of the lateral ventricle in prenatal ultrasound. Accurate segmentation is essential for this measurement, but acoustic shadowing, speckle noise, and poor contrast make it difficult. We developed MVSegNet, a lightweight encoder-decoder network combining multi-scale feature extraction and boundary-aware refinement. The model was trained and evaluated on 584 expert-annotated transventricular ultrasound frames using a 70/15/15 split. Performance was compared against six segmentation baselines using overlap, boundary, and measurement metrics. MVSegNet achieved a Dice score of 80.79%, IoU of 68.47%, Hausdorff distance of 4.07 mm, and atrial width mean absolute error of 3.40 mm. The model contains 2.31 million parameters and runs at 165.6 frames per second on an NVIDIA T4 GPU. MVSegNet outperformed all evaluated baselines on boundary and measurement metrics while maintaining low computational cost, supporting its use in automated fetal ultrasound analysis.
Automated biometric analysis of fetal brain MRI enables reproducible, observer-independent quantitative assessment, yet existing methods are often restricted to few measurements or evaluated only on healthy cases. We assemble and evaluate an automated biometric analysis pipeline that localizes 22 anatomical landmarks on NeSVoR-reconstructed 3D volumes and derives 11 clinically relevant measurements spanning supratentorial, ventricular, cerebellar, and midline structures. We compare two landmark localization models, H3DE-Net and SCN, on a heterogeneous cohort of 122 acquisitions (both healthy controls and range pathologies). Localization accuracy was assessed with a linear mixed-effects model, agreement with normative growth trajectories with calibrated centile charts, and diagnostic utility with a decision tree classifying VM severity. H3DE-Net achieved significantly lower localization error than SCN across all landmarks (mean 1.36 mm vs. 3.58 mm in HC and 1.90 mm vs. 4.13 mm in PC; p < 0.001), and outperformed a GA-based regression baseline in 7 of 11 measurements. H3DE-Net measurements yielded higher classification AUC in every diagnostic group, with the clearest advantage in separating healthy controls from VM. Decision tree thresholds for ventricular width fell near the clinical 10 mm and 15 mm cut-offs used to define and grade VM.
Ema Masterl, Tina Vipotnik Vesnaver, Nejc Šubič +1
Faculty of Medicine, University of Ljubljana, Ljubljana, Slovenia · University Medical Centre Ljubljana, Ljubljana, Slovenia · Faculty of Electrical Engineering, University of Ljubljana, Ljubljana, Slovenia