Organizations: Medical Intelligence and Robotic Cognition (MIRoC) Lab, Department of Mechanical Engineering, The University of Hong Kong, Hong Kong SAR, China · Chair for Computer-Aided and Medical Procedures (CAMP), Tehcnical University of Munich, Munich, Germany · Department of Medical Ultrasonics, The First Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China · Shenzhen Luohu People’s Hospital, Shenzhen, China · Department of Medical Ultrasonics, Guangdong Provincial People’s Hospital, Guangzhou, China
Objective: To develop a unified vision-language model for prompt-conditioned quantitative ultrasound measurement that addresses the mismatch between continuous measurement targets and discrete token-based numerical prediction. Methods: This article presents HAD-VLM, a hybrid autoregressive-diffusion framework that combines autoregressive text generation with continuous measurement prediction within a shared backbone. Visual prompt tuning adapts the vision encoder to ultrasound images. A diffusion process predicts a five-component representation comprising caliper endpoint coordinates and measurement length. A post-processing strategy combining the proposed coordinate-length consistency mechanism with ensemble prediction is introduced to refine measurement estimates. We evaluate four echocardiographic measurement tasks on EchoNet-LVH, with additional assessments of medical question answering and X-ray vertebral measurement. Results: HAD-VLM achieves mean absolute errors of 1.36, 2.77, 3.05, and 1.16 mm for IVSd, LVIDd, LVIDs, and LVPWd, respectively, reducing errors by 21.4-37.6% relative to the Vicuna-based autoregressive baseline. It also outperforms fine-tuned Qwen3-VL and direct MLP regression across all four tasks. Medical question answering performance is preserved on the evaluated dataset. For X-ray vertebral measurement, HAD-VLM reduces mean absolute error from 11.4 to 6.7 pixels. Conclusion: Integrating autoregressive language generation with diffusion-based continuous prediction improves quantitative medical image measurement while retaining prompt-conditioned task flexibility. Significance: HAD-VLM provides a unified approach to combining language interaction with anatomically grounded numerical prediction for quantitative medical image analysis.
Figures & tables
Fig. 1 : An illustration of the mismatch between continuous anatomical measurements and discrete token-based numerical prediction.
Fig. 2 : Overview of HAD-VLM for prompt-conditioned quantitative ultrasound measurement. The framework comprises (a) an prompt-adapted vision encoder, (b) a projector, (c) a hybrid LLM, and (d) a post-processing module. It integrates autoregressive text generation with diffusion-based continuous measurement prediction via a 5-tuple measurement representation and a coordinate-length consistency mechanism.
Fig. 3 : Illustration of the coordinate-length consistency mechanism with a threshold selection based on validation-set discrepancies.
Models
IVSd (MAD: 1.20)
LVIDd (MAD: 2.97)
LVIDs (MAD: 3.60)
LVPWd (MAD: 1.16)
MAE ↓
RMSE ↓
R2↑
r↑
MAE ↓
RMSE ↓
R2↑
r↑
MAE ↓
RMSE ↓
R2↑
r↑
MAE ↓
RMSE ↓
R2↑
r↑
VLM (Vicuna)
1.82
2.54
0.25
0.62
3.62
5.00
0.60
0.82
3.88
4.99
0.65
0.82
1.86
8.43
-12.55
0.15
Fine-tuned Qwen3-VL
1.68
2.44
0.32
0.60
3.71
4.87
0.60
0.79
3.82
5.01
0.64
0.81
1.47
1.95
0.28
0.57
VLM + MLP Regression
1.41
2.00
0.54
0.74
3.00
3.95
0.74
0.86
3.42
4.52
0.71
0.85
1.28
1.67
0.47
0.71
HAD-VLM (ours)
1.36
1.96
0.56
0.75
2.77
3.62
0.78
0.89
3.05
4.00
0.77
0.88
1.16
1.54
0.55
0.74
All MAE, RMSE, and MAD values are reported in mm. Bold values indicate the best reported result for each metric.
TABLE I : Quantitative comparison on four EchoNet-LVH measurement tasks, including IVSd, LVIDd, LVIDs, and LVPWd (MAE: mean absolute error; RMSE: root mean squared error; r : Pearson correlation coefficient; R2 : coefficient of determination; MAD: mean absolute deviation reflecting expert measurement variability [ 15 ] ).
Fig. 4 : Qualitative visualization of three representative prediction patterns.
Split
IVSd
LVIDd
LVIDs
LVPWd
Total
Training set
10265
10312
9859
10254
40690
Validation set
310
309
284
310
1213
Test set
1163
1170
1154
1164
4651
TABLE II : EchoNet-LVH dataset split for the measurement tasks.
Models
IVSd (MAD: 1.20)
LVIDd (MAD: 2.97)
LVIDs (MAD: 3.60)
LVPWd (MAD: 1.16)
MAE ↓
RMSE ↓
R2↑
r↑
MAE ↓
RMSE ↓
R2↑
r↑
MAE ↓
RMSE ↓
R2↑
r↑
MAE ↓
RMSE ↓
R2↑
r↑
w/o prompt tuning
1.93
2.80
0.11
0.41
5.07
6.64
0.26
0.58
5.60
7.40
0.22
0.54
1.58
2.08
0.18
0.48
w/o cons. w/o ens.
1.49
2.14
0.47
0.69
3.04
3.98
0.75
0.87
3.31
4.29
0.74
0.86
1.31
1.73
0.42
0.67
w/ cons. w/o ens.
1.46
2.07
0.51
0.72
2.96
3.85
0.75
0.87
3.34
4.41
0.72
0.85
1.23
1.60
0.52
0.73
HAD-VLM (ours)
1.36
1.96
0.56
0.75
2.77
3.62
0.78
0.89
3.05
4.00
0.77
0.88
1.16
1.54
0.55
0.74
w/: with, w/o: without, cons.: coordinate-length consistency, ens.: ensembling strategy. All MAE, RMSE, and MAD values are reported in mm.
TABLE III : Ablation results across four EchoNet-LVH measurement tasks: IVSd, LVIDd, LVIDs, and LVPWd. (MAE: mean absolute error; RMSE: root mean squared error; r : Pearson correlation coefficient; MAD: mean absolute deviation reflecting expert measurement variability [ 15 ] . All MAE, RMSE, and MAD values are reported in mm.)
Task
Training
MAE ↓
RMSE ↓
R2↑
r↑
IVSd
Single-task
1.44
2.01
0.54
0.74
HAD-VLM (ours)
1.36
1.96
0.56
0.75
LVIDd
Single-task
3.69
4.84
0.61
0.80
HAD-VLM (ours)
2.77
3.62
0.78
0.89
LVIDs
Single-task
3.69
4.89
0.66
0.83
HAD-VLM (ours)
3.05
4.00
0.77
0.88
TABLE IV : Comparison of single-task and multi-task training on EchoNet-LVH .
Fig. 5 : Illustration of superior endplate length measurements for the Left: (a) C4 and Right: (b) C5 vertebrae.
Models
F1 ↑
BERT P ↑
BERT R ↑
BERT F1 ↑
VLM (vicuna)
0.298
0.855
0.885
0.869
HAD-VLM (ours)
0.320
0.873
0.891
0.882
Bold values indicate the best reported result for each metric.
TABLE V : Evaluation results on the medical QA dataset [ 30 ] .
Models
Numerical prediction
Absolute errors
VLM (vicuna)
Discrete tokens
11.4 ± 93.9
HAD-VLM (ours)
Continuous diffusion
6.7 ± 5.7
Absolute errors are reported in mean ± std pixels.
Bold values indicate the best reported result.
TABLE VI : Measurement results on the CSXA -derived X-ray dataset.
Large vision-language models (LVLMs) have achieved strong performance across many medical imaging tasks, yet their application to ultrasound remains limited due to its inherent complexity and variability. In this work, we revisit what is truly needed to enable real-world ultrasound understanding. Instead of introducing complex architectures or elaborate training strategies, we show that data scale and clinically faithful data alignment are the key factors. We construct a large-scale dataset of 1.5M real-world ultrasound examinations, containing 17.7M images, multi-organ coverage, and paired uncurated clinical reports. Crucially, we organize the data at the examination level, aligning multiple images with their corresponding reports to reflect real clinical workflows. We then fine-tune a standard LVLM using low-rank adaptation (LoRA) on this dataset without task-specific modifications. Surprisingly, this simple recipe already leads to strong performance across diverse ultrasound understanding tasks, outperforming prior methods designed with more complex pipelines. Beyond these results, we present model and data scaling analyses that provide insights into the role of scale in ultrasound LVLMs.
Bingcong Yan, Chunlei Li, Jingliang Hu +3
1MedAI Technology (Wuxi) Co. Ltd. · 2Technical University of Munich
Automated radiology report generation is key for reducing radiologist workload and improving diagnostic consistency, yet generating accurate reports for 3D medical imaging remains challenging. Existing vision-language models face two limitations: they do not leverage segmentation-pretrained encoders, and they inject visual features only at the input layer of language models, losing multi-scale information. We propose U-VLM, which enables hierarchical vision-language modeling in both training and architecture: (1) progressive training from segmentation to classification to report generation, and (2) multi-layer visual injection that routes U-Net encoder features to corresponding language model layers. Each training stage can leverage different datasets without unified annotations. U-VLM achieves state-of-the-art performance on CT-RATE (F1: 0.414 vs 0.258, BLEU-mean: 0.349 vs 0.305) and AbdomenAtlas 3.0 (F1: 0.624 vs 0.518 for segmentation-based detection) using only a 0.1B decoder trained from scratch, demonstrating that well-designed vision encoder pretraining outweighs the benefits of 7B+ pre-trained language models. Ablation studies show that progressive pretraining significantly improves F1, while multi-layer injection improves BLEU-mean. Code is available at https://github.com/yinghemedical/U-VLM.
Pengcheng Shi, Minghui Zhang, Kehan Song +3
Medical Image Insights Co. Ltd., Shanghai, China · Shanghai Jiao Tong University, Shanghai, China
Vision-Language Models (VLMs) have significantly advanced medical visual question answering, yet their performance in ultrasound remains suboptimal. In clinical practice, sonographers explicitly focus on lesion regions to formulate reports, though diagnostic interpretations sometimes vary due to inherent subjectivity. However, existing VLMs are not explicitly structured to interactively zoom into lesions prior to diagnosis; moreover, they typically treat annotations as unbiased ground truths, failing to account for their inherent subjectivity and ambiguity. In this paper, we propose a framework specifically designed to consider the sonographer's cognitive workflow. We first introduce a structured Zoom-then-Diagnose paradigm, which replicates the interactive search process to enable lesion-focused reasoning. Furthermore, within the Group Relative Policy Optimization (GRPO) framework, we introduce an uncertainty-aware reward derived from stochastic group-wise rollouts to estimate prediction consistency as a proxy for model confidence. Together, these two components encourage the model to reinforce accurate predictions on clear cases while remaining cautious under ambiguity. Experiments across liver, breast, and thyroid datasets show that our framework improves lesion localization by 39.3%, demonstrating that our model has learned the ability to actively look closer and diagnose.
Yue Zhou, Erxuan Wu, Yikang Sun +5
Computer Aided Medical Procedures (CAMP), TU Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · Zhongshan Hospital, Fudan University, China +1