ReXSonoVQA: A Video QA Benchmark for Procedure-Centric Ultrasound Understanding

Xucheng Wang; Xiaoman Zhang; Sung Eun Kim; Ankit Pal; Pranav Rajpurkar

cs.CVApr 13, 2026

ReXSonoVQA: A Video QA Benchmark for Procedure-Centric Ultrasound Understanding

Authors: Xucheng Wang, Xiaoman Zhang, Sung Eun Kim, Ankit Pal, Pranav Rajpurkar

Organizations: Department of Biomedical Informatics, Harvard Medical School, Boston, MA

Abstract

Ultrasound acquisition requires skilled probe manipulation and real-time adjustments. Vision-language models (VLMs) could enable autonomous ultrasound systems, but existing benchmarks evaluate only static images, not dynamic procedural understanding. We introduce ReXSonoVQA, a video QA benchmark with 514 video clips and 514 questions (249 MCQ, 265 free-response) targeting three competencies: Action-Goal Reasoning, Artifact Resolution & Optimization, and Procedure Context & Planning. Zero-shot evaluation of Gemini 3 Pro, Qwen3.5-397B, LLaVA-Video-72B, and Seed 2.0 Pro shows VLMs can extract some procedural information, but troubleshooting questions remain challenging with minimal gains over text-only baselines, exposing limitations in causal reasoning. ReXSonoVQA enables developing perception systems for ultrasound training, guidance, and robotic automation.

Figures & tables

Figure 1: A ReXSonoVQA example: Type 3 (Procedure Context & Planning, Free-Response) question requiring identification of the screening objective and anatomical transition during a transverse sweep. Gemini 3 Pro correctly identifies the anatomical transition (tendon to muscle) but fails to specify the correct screening objective, receiving a partial score (1/2). More MCQs and free-responses examples see Appendix B .

Benchmark	Modality	# Items	Diagnosis	US- Specific	Multi- Organ	Dynamic Reasoning	MCQ	Free- Response	Blind Baseline
MedMCQA	Text	193,155	✓	✗	✓	✗	✓	✗	✗
GMAI-MMBench	Image	26,000	✓	✗	✓	✗	✓	✗	✗
OmniMedVQA	Image	127,995	✓	✗	✓	✗	✓	✗	✗
VQA-RAD	Image	3,515	✓	✗	✓	✗	✓	✓	✗
ReXVQA	Image	696,000	✓	✗	✗	✗	✓	✗	✗
ReXrank	Image	600,000+	✓	✗	✗	✗	✗	✓	✗

Table 1: Comparison of ReXSonoVQA with existing medical vision-language benchmarks. Dynamic Reasoning : Requires understanding of temporal processes, actions, and causal relationships. Diagnosis : Evaluates pathology identification or diagnostic interpretation. Blind Baseline : Includes text-only evaluation to verify questions require visual input. ✓ indicates present; ✗ indicates absent.

Figure 2: End-to-end pipeline for constructing ReXSonoVQA: (1) Task Definition, (2) Data Curation & Ground Truth Construction, (3) Prompt Refinement and Quality Control Loop, and (4) Benchmark Construction & Evaluation.

Figure 3: Example of video preprocessing. We crop the original video frame to retain only the ultrasound image stream, excluding surrounding content.

Panel of Figure 4 — Figure 4: Examples from ReXSonoVQA. Left: Time-aligned procedural events derived from instructional narration. Right: A clip-grounded question-answer item targeting action-goal reasoning.

Figure 5: ReXSonoVQA dataset composition (514 items). Distribution of questions across clinical categories by task type.

Figure 6: Multi-model zero-shot MCQ accuracy (%, left) and free-response mean score (0–2 rubric, right) by task type under paired evaluation settings. Dashed red line marks 25% random chance for MCQ.

Figure 7: Multi-model zero-shot MCQ accuracy (%, left) and free-response mean score (0–2 rubric, right) by video duration under paired evaluation settings. Dashed red line marks 25% random chance for MCQ.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure A1: Prompt template used for QA generation.

Panel of Figure 10 — Figure A2: Prompt template used for QA generation (continued).

Panel of Figure 11 — Figure A3: Prompt template used for MCQ distractor refinement.

Panel of Figure 12 — Figure A4: Prompt template used for model inference.

Panel of Figure 13 — Figure A5: Prompt template used for LLM-as-a-judge.

Figure A6: Exploratory evaluation. Demonstrated why GPT-5.2 and Claude Opus 4.5 were excluded from our benchmark’s baseline. Without native video support, both models hallucinate completely unrelated anatomical structures (colon/intussusception and vein/thrombosis) when provided with sequential images, scoring 0/2. In contrast, Gemini 3 Pro with native video understanding correctly identifies the anatomical transition, achieving 1/2. This stark difference motivated our focus on models with native video support.

Figure A7: Case Study 1 (Free-Response, Type 1: Action–Goal Reasoning). Gemini 3 Pro correctly identifies the probe maneuver and acquisition target.

Figure A8: Case Study 2 (Free-Response, Type 2: Artifact Resolution & Optimization). Gemini 3 Pro hallucinates a compression maneuver instead of the actual optimization action.

Figure A9: Case Study 3 (MCQ, Type 1: Action–Goal Reasoning). Gemini 3 Pro misinterprets compression behavior due to incorrect anatomy identification, leading to the wrong action–goal selection.

Figure A10: Case Study 4 (MCQ, Type 2: Artifact Resolution & Optimization). Gemini 3 Pro fails to identify the color bleeding artifact and the corresponding optimization cue.

Figure A11: Case Study 5: (MCQ, Type 3: Procedure Context & Planning) fails to recognize the vertebral acoustic shadow as the key landmark for protocol context.

	Vid. Correct	Vid. Incorrect	Total
Blind Correct	97	8	105
Blind Incorrect	68	76	144
Total	165	84	249

Appendix

Table A1: Gemini 3 Pro: MCQ cross-setting outcome table (

n=249

). Rows = blind (text-only) outcome; columns = video-informed outcome.

	2 pt	1 pt (W.C.)	1 pt (W.V.)	0 pt	Total
2 pt	15	2	1	1	19
1 pt (W.C.)	7	4	2	8	21
1 pt (W.V.)	13	1	3	6	23
0 pt	64	22	18	98	202
Total	99	29	24	113	265

Appendix

Table A2: Gemini 3 Pro: Free-response cross-setting outcome table (

n=265

). Rows = blind outcome; columns = video-informed outcome. W.C. = wrong conclusion; W.V. = wrong visual evidence.

	Vid. Correct	Vid. Incorrect	Total
Blind Correct	45	60	105
Blind Incorrect	29	115	144
Total	74	175	249

Appendix

Table A3: LLaVA-Video-72B: MCQ cross-setting outcome table (

n=249

	2 pt	1 pt (W.C.)	1 pt (W.V.)	0 pt	Total
2 pt	0	2	6	11	19
1 pt (W.C.)	1	1	3	16	21
1 pt (W.V.)	4	2	3	14	23
0 pt	2	16	16	168	202
Total	7	21	28	209	265

Appendix

Table A4: LLaVA-Video-72B: Free-response cross-setting outcome table (

n=265

	Vid. Correct	Vid. Incorrect	Total
Blind Correct	66	39	105
Blind Incorrect	72	72	144
Total	138	111	249

Appendix

Table A5: Qwen3.5-397B: MCQ cross-setting outcome table (

n=249

	2 pt	1 pt (W.C.)	1 pt (W.V.)	0 pt	Total
2 pt	13	1	1	4	19
1 pt (W.C.)	7	0	3	11	21
1 pt (W.V.)	9	2	5	7	23
0 pt	45	13	23	121	202
Total	74	16	32	143	265

Appendix

Table A6: Qwen3.5-397B: Free-response cross-setting outcome table (

n=265

	Vid. Correct	Vid. Incorrect	Total
Blind Correct	65	40	105
Blind Incorrect	65	79	144
Total	130	119	249

Appendix

Table A7: Seed 2.0 Pro: MCQ cross-setting outcome table (

n=249

	2 pt	1 pt (W.C.)	1 pt (W.V.)	0 pt	Total
2 pt	13	1	0	5	19
1 pt (W.C.)	7	4	1	9	21
1 pt (W.V.)	10	1	4	8	23
0 pt	50	18	23	111	202
Total	80	24	28	133	265

Appendix

Table A8: Seed 2.0 Pro: Free-response cross-setting outcome table (

n=265

Medical VQA

Medical visual question answering studies how AI systems answer clinical questions using medical images and related patient or report data.

See all 175 papers

UltraG-Bench: A Multi-task Benchmark for assessing Large Vision-Language Models on Pixel-level Evidence Grounding in Ultrasound
Sep 25, 2026Quanhao Zhu, Bo Xu, Rui Lin +7Medical VLMs Medical Image Grounding
Dalian University of Technology · RMIT University
Look-Closer-Then-Diagnose: Confidence-Aware Ultrasound VQA via Active Zooming
May 20, 2026Yue Zhou, Erxuan Wu, Yikang Sun +5Visual Question Answering Uncertainty Estimation for VLA Models
Computer Aided Medical Procedures (CAMP), TU Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · Zhongshan Hospital, Fudan University, China +1
Towards Real-World Ultrasound Understanding: Large Vision-Language Models from Multi-Image Examinations with Long-Form Reports
Jul 2, 2026Bingcong Yan, Chunlei Li, Jingliang Hu +3VLM Adaptation Ultrasound Imaging
1MedAI Technology (Wuxi) Co. Ltd. · 2Technical University of Munich

Sep 25, 2026J/K move · Enter open · S save

UltraG-Bench: A Multi-task Benchmark for assessing Large Vision-Language Models on Pixel-level Evidence Grounding in Ultrasound

Quanhao Zhu, Bo Xu, Rui Lin, Chenyuan Wang+6

Dalian University of Technology · RMIT University

Medical VLMs Medical Image Grounding Medical Image Benchmarks Ultrasound Image Segmentation

Open paper AskPDF arXiv

Ultrasound is one of the most widely used medical imaging modalities, and recent large vision-language models(VLMs) have shown increasing capabilities in ultrasound image understanding. However, these models fail to provide pixel-level visual evidence aligned with their semantic predictions, and their fine-grained grounding capability in ultrasound remains largely unclear. We introduce UltraG-Bench, a large-scale multi-task benchmark for evaluating pixel-level evidence grounding in ultrasound. UltraG-Bench is built by annotating 40 public ultrasound segmentation datasets spanning 13 anatomical categories, and comprises three progressive tasks: instruction-guided segmentation, evidence-grounded VQA, and evidence-grounded report generation, with 331125, 666779, and 138832 annotations, respectively. Comprehensive evaluation of 14 state-of-the-art models reveals a substantial gap between semantic understanding and fine-grained pixel-level localization. We further propose UltraG-Agent, which combines the semantic reasoning capabilities of a VLM with the ultrasound-specific segmentation capability of UltraSAM3. Experiments show that UltraG-Agent substantially improves both semantic prediction and pixel-level visual grounding. Our dataset and code are available at https://github.com/zhuqh19/UltraG-Bench.

Abstract

Medical VQA

Explore similar work

UltraG-Bench: A Multi-task Benchmark for assessing Large Vision-Language Models on Pixel-level Evidence Grounding in Ultrasound

Look-Closer-Then-Diagnose: Confidence-Aware Ultrasound VQA via Active Zooming

Towards Real-World Ultrasound Understanding: Large Vision-Language Models from Multi-Image Examinations with Long-Form Reports