Beyond Discrete Tokens: Preserving Numerical Continuity in Medical Vision-Language Models

Dianye Huang; Jiamei Guo; Yue Zhou; Mingde Li; Yi Xiong; Wei Wang; Nassir Navab; Zhongliang Jiang

cs.CVOct 23, 2026

Beyond Discrete Tokens: Preserving Numerical Continuity in Medical Vision-Language Models

Authors: Dianye Huang, Jiamei Guo, Yue Zhou, Mingde Li, Yi Xiong, Wei Wang, Nassir Navab, Zhongliang Jiang

Organizations: Medical Intelligence and Robotic Cognition (MIRoC) Lab, Department of Mechanical Engineering, The University of Hong Kong, Hong Kong SAR, China · Chair for Computer-Aided and Medical Procedures (CAMP), Tehcnical University of Munich, Munich, Germany · Department of Medical Ultrasonics, The First Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China · Shenzhen Luohu People’s Hospital, Shenzhen, China ·

Abstract

Objective: To develop a unified vision-language model for prompt-conditioned quantitative ultrasound measurement that addresses the mismatch between continuous measurement targets and discrete token-based numerical prediction. Methods: This article presents HAD-VLM, a hybrid autoregressive-diffusion framework that combines autoregressive text generation with continuous measurement prediction within a shared backbone. Visual prompt tuning adapts the vision encoder to ultrasound images. A diffusion process predicts a five-component representation comprising caliper endpoint coordinates and measurement length. A post-processing strategy combining the proposed coordinate-length consistency mechanism with ensemble prediction is introduced to refine measurement estimates. We evaluate four echocardiographic measurement tasks on EchoNet-LVH, with additional assessments of medical question answering and X-ray vertebral measurement. Results: HAD-VLM achieves mean absolute errors of 1.36, 2.77, 3.05, and 1.16 mm for IVSd, LVIDd, LVIDs, and LVPWd, respectively, reducing errors by 21.4-37.6% relative to the Vicuna-based autoregressive baseline. It also outperforms fine-tuned Qwen3-VL and direct MLP regression across all four tasks. Medical question answering performance is preserved on the evaluated dataset. For X-ray vertebral measurement, HAD-VLM reduces mean absolute error from 11.4 to 6.7 pixels. Conclusion: Integrating autoregressive language generation with diffusion-based continuous prediction improves quantitative medical image measurement while retaining prompt-conditioned task flexibility. Significance: HAD-VLM provides a unified approach to combining language interaction with anatomically grounded numerical prediction for quantitative medical image analysis.

Figures & tables

Fig. 1 : An illustration of the mismatch between continuous anatomical measurements and discrete token-based numerical prediction.

Fig. 2 : Overview of HAD-VLM for prompt-conditioned quantitative ultrasound measurement. The framework comprises (a) an prompt-adapted vision encoder, (b) a projector, (c) a hybrid LLM, and (d) a post-processing module. It integrates autoregressive text generation with diffusion-based continuous measurement prediction via a 5-tuple measurement representation and a coordinate-length consistency mechanism.

Fig. 3 : Illustration of the coordinate-length consistency mechanism with a threshold selection based on validation-set discrepancies.

All MAE, RMSE, and MAD values are reported in mm. Bold values indicate the best reported result for each metric.
Models	IVSd (MAD: 1.20)				LVIDd (MAD: 2.97)				LVIDs (MAD: 3.60)				LVPWd (MAD: 1.16)
Models	MAE $\downarrow$	RMSE $\downarrow$	$R^{2}\uparrow$	$r\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	$R^{2}\uparrow$	$r\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	$R^{2}\uparrow$	$r\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	$R^{2}\uparrow$	$r\uparrow$
VLM (Vicuna)	1.82	2.54	0.25	0.62	3.62	5.00	0.60	0.82	3.88	4.99	0.65	0.82	1.86	8.43	-12.55	0.15
Fine-tuned Qwen3-VL	1.68	2.44	0.32	0.60	3.71	4.87	0.60	0.79	3.82	5.01	0.64	0.81	1.47	1.95	0.28	0.57
VLM + MLP Regression	1.41	2.00	0.54	0.74	3.00	3.95	0.74	0.86	3.42	4.52	0.71	0.85	1.28	1.67	0.47	0.71
HAD-VLM (ours)	1.36	1.96	0.56	0.75	2.77	3.62	0.78	0.89	3.05	4.00	0.77	0.88	1.16	1.54	0.55	0.74

TABLE I : Quantitative comparison on four EchoNet-LVH measurement tasks, including IVSd, LVIDd, LVIDs, and LVPWd (MAE: mean absolute error; RMSE: root mean squared error;

r

: Pearson correlation coefficient;

R^{2}

: coefficient of determination; MAD: mean absolute deviation reflecting expert measurement variability [ 15 ] ).

Fig. 4 : Qualitative visualization of three representative prediction patterns.

Split	IVSd	LVIDd	LVIDs	LVPWd	Total
Training set	10265	10312	9859	10254	40690
Validation set	310	309	284	310	1213
Test set	1163	1170	1154	1164	4651

TABLE II : EchoNet-LVH dataset split for the measurement tasks.

w/: with, w/o: without, cons.: coordinate-length consistency, ens.: ensembling strategy. All MAE, RMSE, and MAD values are reported in mm.
Models	IVSd (MAD: 1.20)				LVIDd (MAD: 2.97)				LVIDs (MAD: 3.60)				LVPWd (MAD: 1.16)
Models	MAE $\downarrow$	RMSE $\downarrow$	$R^{2}\uparrow$	$r\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	$R^{2}\uparrow$	$r\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	$R^{2}\uparrow$	$r\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	$R^{2}\uparrow$	$r\uparrow$
w/o prompt tuning	1.93	2.80	0.11	0.41	5.07	6.64	0.26	0.58	5.60	7.40	0.22	0.54	1.58	2.08	0.18	0.48
w/o cons. w/o ens.	1.49	2.14	0.47	0.69	3.04	3.98	0.75	0.87	3.31	4.29	0.74	0.86	1.31	1.73	0.42	0.67
w/ cons. w/o ens.	1.46	2.07	0.51	0.72	2.96	3.85	0.75	0.87	3.34	4.41	0.72	0.85	1.23	1.60	0.52	0.73
HAD-VLM (ours)	1.36	1.96	0.56	0.75	2.77	3.62	0.78	0.89	3.05	4.00	0.77	0.88	1.16	1.54	0.55	0.74

TABLE III : Ablation results across four EchoNet-LVH measurement tasks: IVSd, LVIDd, LVIDs, and LVPWd. (MAE: mean absolute error; RMSE: root mean squared error;

r

: Pearson correlation coefficient; MAD: mean absolute deviation reflecting expert measurement variability [ 15 ] . All MAE, RMSE, and MAD values are reported in mm.)

Task	Training	MAE $\downarrow$	RMSE $\downarrow$	$R^{2}\uparrow$	$r\uparrow$
IVSd	Single-task	1.44	2.01	0.54	0.74
IVSd	HAD-VLM (ours)	1.36	1.96	0.56	0.75
LVIDd	Single-task	3.69	4.84	0.61	0.80
LVIDd	HAD-VLM (ours)	2.77	3.62	0.78	0.89
LVIDs	Single-task	3.69	4.89	0.66	0.83
LVIDs	HAD-VLM (ours)	3.05	4.00	0.77	0.88

TABLE IV : Comparison of single-task and multi-task training on EchoNet-LVH .

Fig. 5 : Illustration of superior endplate length measurements for the Left: (a) C4 and Right: (b) C5 vertebrae.

Bold values indicate the best reported result for each metric.
Models	F1 $\uparrow$	BERT P $\uparrow$	BERT R $\uparrow$	BERT F1 $\uparrow$
VLM (vicuna)	0.298	0.855	0.885	0.869
HAD-VLM (ours)	0.320	0.873	0.891	0.882

TABLE V : Evaluation results on the medical QA dataset [ 30 ] .

Models	Numerical prediction	Absolute errors
VLM (vicuna)	Discrete tokens	11.4 $\pm$ 93.9
HAD-VLM (ours)	Continuous diffusion	6.7 $\pm$ 5.7
Absolute errors are reported in mean $\pm$ std pixels.
Bold values indicate the best reported result.

TABLE VI : Measurement results on the CSXA -derived X-ray dataset.

Medical Image Analysis

Medical image analysis uses computational methods to interpret clinical images, measure anatomy and disease, and support diagnosis and treatment.

See all 292 papers

Towards Real-World Ultrasound Understanding: Large Vision-Language Models from Multi-Image Examinations with Long-Form Reports
Jul 2, 2026Bingcong Yan, Chunlei Li, Jingliang Hu +3VLM Adaptation Ultrasound Imaging
1MedAI Technology (Wuxi) Co. Ltd. · 2Technical University of Munich
U-VLM: Hierarchical Vision Language Modeling for Report Generation
Feb 28, 2026Pengcheng Shi, Minghui Zhang, Kehan Song +3Radiology Report Generation Vision-Language Models
Medical Image Insights Co. Ltd., Shanghai, China · Shanghai Jiao Tong University, Shanghai, China
Look-Closer-Then-Diagnose: Confidence-Aware Ultrasound VQA via Active Zooming
May 20, 2026Yue Zhou, Erxuan Wu, Yikang Sun +5Visual Question Answering Uncertainty Estimation for VLA Models
Computer Aided Medical Procedures (CAMP), TU Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · Zhongshan Hospital, Fudan University, China +1

Jul 2, 2026J/K move · Enter open · S save

Towards Real-World Ultrasound Understanding: Large Vision-Language Models from Multi-Image Examinations with Long-Form Reports

Bingcong Yan, Chunlei Li, Jingliang Hu, Yilei Shi+2

1MedAI Technology (Wuxi) Co. Ltd. · 2Technical University of Munich

VLM Adaptation Ultrasound Imaging Medical VLMs

Open paper AskPDF arXiv

Large vision-language models (LVLMs) have achieved strong performance across many medical imaging tasks, yet their application to ultrasound remains limited due to its inherent complexity and variability. In this work, we revisit what is truly needed to enable real-world ultrasound understanding. Instead of introducing complex architectures or elaborate training strategies, we show that data scale and clinically faithful data alignment are the key factors. We construct a large-scale dataset of 1.5M real-world ultrasound examinations, containing 17.7M images, multi-organ coverage, and paired uncurated clinical reports. Crucially, we organize the data at the examination level, aligning multiple images with their corresponding reports to reflect real clinical workflows. We then fine-tune a standard LVLM using low-rank adaptation (LoRA) on this dataset without task-specific modifications. Surprisingly, this simple recipe already leads to strong performance across diverse ultrasound understanding tasks, outperforming prior methods designed with more complex pipelines. Beyond these results, we present model and data scaling analyses that provide insights into the role of scale in ultrasound LVLMs.

Abstract

Medical Image Analysis

Explore similar work

Towards Real-World Ultrasound Understanding: Large Vision-Language Models from Multi-Image Examinations with Long-Form Reports

U-VLM: Hierarchical Vision Language Modeling for Report Generation

Look-Closer-Then-Diagnose: Confidence-Aware Ultrasound VQA via Active Zooming