UltraG-Bench: A Multi-task Benchmark for assessing Large Vision-Language Models on Pixel-level Evidence Grounding in Ultrasound

Quanhao Zhu; Bo Xu; Rui Lin; Chenyuan Wang; Yu Shao; Boling Zhu; Jiuyan Sun; Liang Zhao

cs.CVSep 25, 2026

UltraG-Bench: A Multi-task Benchmark for assessing Large Vision-Language Models on Pixel-level Evidence Grounding in Ultrasound

Authors: Quanhao Zhu, Bo Xu, Rui Lin, Chenyuan Wang, Yu Shao, Boling Zhu, Jiuyan Sun, Liang Zhao,

Organizations: Dalian University of Technology · RMIT University

Abstract

Ultrasound is one of the most widely used medical imaging modalities, and recent large vision-language models(VLMs) have shown increasing capabilities in ultrasound image understanding. However, these models fail to provide pixel-level visual evidence aligned with their semantic predictions, and their fine-grained grounding capability in ultrasound remains largely unclear. We introduce UltraG-Bench, a large-scale multi-task benchmark for evaluating pixel-level evidence grounding in ultrasound. UltraG-Bench is built by annotating 40 public ultrasound segmentation datasets spanning 13 anatomical categories, and comprises three progressive tasks: instruction-guided segmentation, evidence-grounded VQA, and evidence-grounded report generation, with 331125, 666779, and 138832 annotations, respectively. Comprehensive evaluation of 14 state-of-the-art models reveals a substantial gap between semantic understanding and fine-grained pixel-level localization. We further propose UltraG-Agent, which combines the semantic reasoning capabilities of a VLM with the ultrasound-specific segmentation capability of UltraSAM3. Experiments show that UltraG-Agent substantially improves both semantic prediction and pixel-level visual grounding. Our dataset and code are available at https://github.com/zhuqh19/UltraG-Bench.

Figures & tables

Figure 1: Overview of UltraG-Bench, including 13 anatomical categories, benchmark statistics, and representative examples of the three tasks: Instruction-guided Segmentation, Evidence-grounded VQA, and Evidence-grounded Report Generation.

Benchmark	Modality	Scale	Spatial Evidence	Region Grounding	Instr.-Seg.	Grounded VQA	Grounded Report	Coverage
MS-CXR	CXR	1,162 pairs	Bounding Box	✓	✗	✗	✗	Chest
Chest ImaGenome	CXR	242,072 images	Bounding Box	✓	✗	✗	✗	Chest
PadChest-GR	CXR	4,555 studies	Bounding Box	✓	✗	✗	✓	Chest
GEMeX	CXR	1.61M VQA pairs	Bounding Box	✓	✗	✓	✗	Chest
GEMeX-RMCoT	CXR	202K instances	Bounding Box	✓	✗	✓	✗	Chest
MIMIC-ILS	CXR	1.1M pairs	Pixel Mask	✓	✓	✗	✗	Chest

Table 1: Comparison of UltraG-Bench with representative medical evidence-grounded datasets.

Figure 2: Construction pipeline of the UltraG-Bench, including public data collection, progressive task design, evidence-constrained annotation, automated validation, and expert review.

Closed-source MLLMs
Model	Instruction-guided Seg.		Evidence-grounded VQA		Evidence-grounded Report Gen.
Model	IoU $\uparrow$	Dice $\uparrow$	Acc. $\uparrow$	G-Acc. $\uparrow$	ROUGE-L $\uparrow$	SemAcc $\uparrow$
Claude-4.5	0.1544	0.2180	0.3327	0.0214	0.1334	0.3281
GPT-5.4	0.2653	0.3718	0.3631	0.0609	0.2264	0.3577
Open-source General MLLMs
Qwen3-VL-4B	0.1069	0.1537	0.2197	0.0137	0.0793	0.1899

Table 2: Experimental Results on UltraG-Bench. The best, second-best, and third-best results among all models are highlighted in green , blue , and yellow , respectively. For UltraG-Agent,

I

Q

, and

L

refer to InternVL3.5-8B, Qwen3-VL-8B, and Lingshu-7B, while

US3

and

S3

refer to UltraSAM3 and SAM3, respectively.

Figure 4: Fine-grained organ-level comparison across 13 anatomical categories for the three tasks in UltraG-Bench. UltraG-Agent variants with UltraSAM3 show consistently strong performance across diverse organs.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Organ	Dataset	Train Images	Train Masks	Test Images	Test Masks
Abdomen	AbdomenUS [ 33 ]	784	3,079	198	781
Breast	BrEast [ 34 ]	204	402	52	102
Breast	BUID [ 35 ]	185	370	47	94
Breast	BUSI [ 36 ]	624	1,056	156	274
Breast	BUS_BRA [ 37 ]	1,500	3,000	375	750
Breast	BUS_DatasetB [ 38 ]	130	260	33	66

Appendix

Table 3: Statistics of the 40 ultrasound datasets retained in the Grounded-US annotations.

Figure 5: Anatomical-category distributions of the three tasks in UltraG-Bench across 13 anatomical categories.

Closed-source MLLMs
Model	Presence $\uparrow$	Category $\uparrow$	Location $\uparrow$	Size $\uparrow$
Claude-4.5	0.8789	0.3871	0.0067	0.0001
GPT-5.4	0.8139	0.4824	0.0305	0.0418
Open-source General-purpose MLLMs
Qwen3-VL-4B	0.5231	0.2328	0.0001	0.0065
MiniCPM-V 2.6	0.7070	0.2978	0.0002	0.0020

Appendix

Table 4: Detailed comparison of the four semantic attributes used in Semantic Accuracy on UltraG-Bench. The best, second-best, and third-best results among all models are highlighted in green , blue , and yellow , respectively. For UltraG-Agent,

I

Q

, and

L

refer to InternVL3.5-8B, Qwen3-VL-8B, and Lingshu-7B, while

US3

and

S3

refer to UltraSAM3 and SAM3, respectively.

Figure 6: Breast Instruction-guided Segmentation Case Study.

Figure 7: Fetal Instruction-guided Segmentation Case Study.

Figure 8: Thyroid Evidence-grounded VQA Case Study.

Figure 9: Ovarian Evidence-grounded VQA Case Study.

Figure 10: Prostate Evidence-grounded Report Generation Case Study.

Figure 11: Liver Evidence-grounded Report Generation Case Study.

Figure 12: Abdomen Instruction-guided Segmentation Error Analysis.

Figure 13: Lung Evidence-grounded VQA Error Analysis.

Figure 14: Cardiac Evidence-grounded Report Generation Error Analysis.

Medical VLMs

Medical vision-language models connect clinical images with text for diagnosis, reporting, question answering, and image grounding.

See all 475 papers

UltraBench 2: Towards Robust Evaluation of Vision Foundation Models on Ultrasound
Sep 23, 2026Ashwath Radhachandran, Adam Tupper, Christian Gagné +1Ultrasound Image Segmentation Medical Imaging Foundation Models
Bioengineering Department, University of California, Los Angeles · Institut Intelligence et Données (IID), Université Laval Mila – Quebec AI Institute · Radiological Sciences Department, UCLA David Geffen School of Medicine
Towards Real-World Ultrasound Understanding: Large Vision-Language Models from Multi-Image Examinations with Long-Form Reports
Jul 2, 2026Bingcong Yan, Chunlei Li, Jingliang Hu +3VLM Adaptation Ultrasound Imaging
1MedAI Technology (Wuxi) Co. Ltd. · 2Technical University of Munich
UltraVR: A Diagnostic Ultra-Resolution Image-VQA Benchmark for Evidence-Grounded Reasoning
Jun 4, 2026Gexin Huang, Yanting Yang, Myeongkyun Kang +6VLM Evaluation Visual Question Answering
University of British Columbia, Vancouver, BC, Canada · Vector Institute, Toronto, ON, Canada · The Hong Kong Polytechnic University, Hong Kong SAR, China +1

Sep 23, 2026J/K move · Enter open · S save

UltraBench 2: Towards Robust Evaluation of Vision Foundation Models on Ultrasound

Ashwath Radhachandran, Adam Tupper, Christian Gagné, William Speier

Bioengineering Department, University of California, Los Angeles · Institut Intelligence et Données (IID), Université Laval Mila – Quebec AI Institute · Radiological Sciences Department, UCLA David Geffen School of Medicine

Ultrasound Image Segmentation Medical Imaging Foundation Models Vision Foundation Models Medical Image Classification Ultrasound Imaging Medical Image Benchmarks

Open paper AskPDF arXiv

Benchmarking is an increasingly critical part of research in machine learning and the domains where it is applied, including healthcare. Yet, despite the steady development of new ultrasound foundation models in recent years, the development of well-designed benchmarks to evaluate them has lagged behind. This deficiency has led to fragmented and inconsistent evaluations of competing models, making it difficult to measure progress. To address this issue, we introduce UltraBench 2, a comprehensive benchmark with wide anatomical and task coverage, and a focus on standardization, reproducibility, and ease-of-use. Using this benchmark, we compare existing vision foundation models for ultrasound image analysis. Our analyses demonstrate that ultrasound-specific pretraining still leads on classification, but that state-of-the-art general-purpose models have drawn level on segmentation.

Abstract

Medical VLMs

Explore similar work

UltraBench 2: Towards Robust Evaluation of Vision Foundation Models on Ultrasound

Towards Real-World Ultrasound Understanding: Large Vision-Language Models from Multi-Image Examinations with Long-Form Reports

UltraVR: A Diagnostic Ultra-Resolution Image-VQA Benchmark for Evidence-Grounded Reasoning