Medical Image Grounding

Follow this topic Ask about this topic

Medical image grounding studies how to connect clinical language—such as a finding in a report or an answer to a question—to the region, slice, or structure that provides visual evidence. This can make image-based diagnoses and reports easier to verify, and can reveal when a model gives a plausible answer without actually reading the image. Common approaches align reports with regions using weak supervision or boxes and masks, add anatomical priors and region prompts, or train vision-language models with grounding objectives, tools, and reinforcement learning. Benchmarks span chest X-rays, CT, MRI, ultrasound, and other modalities, but a central challenge remains: strong semantic answers do not guarantee precise localization, especially in 3D, across views, or under distribution shift.

Latest papers

82 papers9 papers in the last four weeks, up 13% on the four weeks before. 0.1% of all new papers.

Auditing Frontier Vision-Language Models for Trustworthy Medical VQA: Grounding Failures, Format Collapse, and Domain Adaptation
Apr 30, 2026Xupeng Chen, Binbin Shi, Chenqian Le +5VLM Evaluation Healthcare
New York University, New York, USA · Tsinghua University, Beijing, China · Columbia University, New York, USA +1
Dynamic Decision Learning: Test-Time Evolution for Abnormality Grounding in Rare Diseases
Apr 27, 2026Jun Li, Mingxuan Liu, Jiazhen Pan +4Medical Imaging Test-Time Adaptation
Technical University of Munich · Munich Center for Machine Learning · University of Trento +3
Region-Grounded Report Generation for 3D Medical Imaging: A Fine-Grained Dataset and Graph-Enhanced Framework
Apr 20, 2026Cong Huy Nguyen, Son Dinh Nguyen, Guanlin Li +8Radiology Report Generation Medical Report Generation
AI4LIFE, Hanoi University of Science and Technology, Vietnam · SAMOVAR, Télécom SudParis, Institut Polytechnique de Paris, France · Military Central Hospital, Vietnam
Beyond a Single Frame: Multi-Frame Spatially Grounded Reasoning Across Volumetric MRI
Apr 17, 2026Lama Moukheiber, Caleb M. Yeung, Haotian Xue +33D Spatial Reasoning 3D Medical Imaging
Georgia Institute of Technology · Harvard University · Georgetown University
MApLe: Multi-instance Alignment of Diagnostic Reports and Large Medical Images
Apr 15, 2026Felicia Bader, Philipp Seeböck, Anastasia Bartashova +2Radiology VLMs Vision-Language Alignment
Computational Imaging Research Lab, Department of Biomedical Imaging and Image-guided Therapy, Medical University of Vienna, Austria · Comprehensive Center for Artificial Intelligence in Medicine, Medical University of Vienna, Austria · Medical Anomaly Detection (MANO) Group, Computational Imaging Research (CIR), Department of Biomedical Imaging and Image-guided Therapy, Medical University of Vienna, Austria +1
Medical thinking with multiple images
Apr 14, 2026Zonghai Yao, Benlu Wang, Yifan Zhang +8Medical VQA Multimodal Reasoning
Manning College of Information and Computer Sciences, UMass Amherst · Center for Healthcare Organization and Implementation Research, VA Bedford Health Care · Department of Computer Science, Yale University +2
MedP-CLIP: Medical CLIP with Region-Aware Prompt Integration
Apr 13, 2026Jiahui Peng, He Yao, Jingwen Li +9Promptable Image Segmentation Multimodal Pretraining
aSichuan University, Chengdu, 610065, China · bXinjiang University, Urumqi, 830046, China · cAlibaba Group, Hangzhou, 310023, China +3
Enhancing Fine-Grained Spatial Grounding in 3D CT Report Generation via Discriminative Guidance
Apr 12, 2026Chenyu Wang, Weicheng Dai, Han Liu +2Radiology Report Generation CT Report Generation
Boston University, Boston, MA, USA · Siemens Healthineers, Princeton, NJ, USA
Enhancing Medical Visual Grounding via Knowledge-guided Spatial Prompts
Apr 2, 2026Yifan Gao, Tao Zhou, Yi Zhou +3Medical Image Grounding Medical Image Localization
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China · School of Computer Science and Engineering, Southeast University, Nanjing 211189, China · Yong Yoo Lin School of Medicine, National University of Singapore, Singapore +1
MedLoc-R1: Performance-Aware Curriculum Reward Scheduling for GRPO-Based Medical Visual Grounding
Mar 30, 2026Guangjing Yang, Ziyuan Qin, Chaoran Zhang +6Medical Image Grounding Reward Shaping
Beijing University of Posts and Telecommunications · Emory University · Peking University +1
MedOpenClaw and MedFlowBench: Auditing Medical Agents in Full-Study Workflows
Mar 25, 2026Weixiang Shen, Chengzhi Shen, Yanzhu Hu +12Medical VLMs Medical Image Benchmarks
Technical University of Munich (TUM) · TUM University Hospital · LMU Munich +6
LoFi: Location-Aware Fine-Grained Representation Learning for Chest X-ray
Mar 19, 2026Myeongkyun Kang, Yanting Yang, Xiaoxiao LiMedical Image Representation Learning Medical VLMs
The University of British Columbia, Vancouver, BC V6T 1Z4, Canada · Vector Institute, Toronto, ON M5G 0C6, Canada
LoGSAM: Parameter-Efficient Cross-Modal Grounding for MRI Segmentation
Mar 18, 2026Mohammad Robaitul Islam Bhuiyan, Sheethal Bhat, Melika Qahqaie +4Image Segmentation Medical Imaging Foundation Models
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Siemens Healthineers, Erlangen, Germany
Medical Image Spatial Grounding with Semantic Sampling
Mar 15, 2026Andrew Seohwan Yu, Mohsen Hariri, Kunio Nakamura +3Medical VLMs Medical Image Grounding
Case Western Reserve University, Cleveland OH 44106, USA · Cleveland Clinic, Cleveland OH 44106, USA
How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images
Mar 15, 2026Guimeng Liu, Tianze Yu, Somayeh Ebrahimkhani +3Medical VLMs Medical Image Grounding
Singapore University of Technology and Design, Singapore · Department of Neurology, National Neuroscience Institute, Singapore · Duke-NUS Medical School, Singapore
Location-Aware Pretraining for Medical Difference Visual Question Answering
Mar 5, 2026Denis Musinguzi, Caren Han, Prasenjit MitraMedical VQA Vision-Language Pretraining
Department of Electrical and Computer Engineering, Carnegie Mellon University, Kigali, Rwanda · University of Melbourne, Melbourne, Australia.
GroundedSurg: A Multi-Procedure Benchmark for Language-Conditioned Surgical Tool Segmentation
Mar 1, 2026Tajamul Ashraf, Abrar Ul Riyaz, Wasif Tak +4Medical Image Grounding Referring Image Segmentation
King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Gaash Research Lab, National Institute of Technology Srinagar, India · Thapar Institute of Engineering and Technology, India +1
EMAD: Evidence-Centric Grounded Multimodal Diagnosis for Alzheimer's Disease
Feb 22, 2026Qiuhui Chen, Xuancheng Yao, Zhenglei Zhou +2Medical VLMs Explainable Medical Image Analysis
East China University of Science and Technology · Shanghai Jiao Tong University · Tencent
Ophiuchus: Incentivizing Tool-augmented "Think with Images" for Joint Medical Segmentation, Understanding and Reasoning
Dec 16, 2025Yankai Jiang, Yujie Zhang, Peng Zhang +5Image Segmentation Tool Use in VLMs
Zhejiang University · Shanghai Artificial Intelligence Laboratory · Shanghai Innovation Institute +5
Generalised Medical Phrase Grounding
Nov 30, 2025Wenjun Zhang, Shekhar S. Chandra, Aaron NicolsonRadiology VLMs Healthcare
The University of Queensland, Brisbane, Australia · Australian e-Health Research Centre, CSIRO Health and Biosecurity, Australia
Anatomy-Grounded Weakly Supervised Prompt Tuning for Chest X-ray Latent Diffusion Models
Jun 12, 2025Konstantinos Vilouras, Ilias Stogiannidis, Junyu Yan +2Diffusion Model Alignment Prompt Tuning
School of Engineering, University of Edinburgh · Canon Medical Research Europe Ltd.
MultiViewDx: Evidence-Linked Multi-View Clinical Diagnosis
Oct 19, 2024Junda Wang, Zonghai Yao, Yujan Ting +5Healthcare Medical Image Analysis
United Imaging Intelligence, MA, USA · Manning College of Information and Computer Sciences, University of Massachusetts Amherst, MA, USA · Department of Medicine, University of Massachusetts Medical School, Worcester, MA, USA +1

Apr 30, 2026J/K move · Enter open · S save

Auditing Frontier Vision-Language Models for Trustworthy Medical VQA: Grounding Failures, Format Collapse, and Domain Adaptation

Xupeng Chen, Binbin Shi, Chenqian Le, Qifu Yin+4

New York University, New York, USA · Tsinghua University, Beijing, China · Columbia University, New York, USA+1

VLM Evaluation Healthcare Medical VQA Medical VLMs Medical Image Grounding

Open paper AskPDF arXiv

Deploying vision-language models (VLMs) in clinical settings demands auditable behavior under realistic failure conditions, yet the failure landscape of frontier VLMs on specialized medical inputs is poorly characterized. We audit five recent frontier and grounding-aware VLMs (Gemini~~2.5~~Pro, GPT-5, o3, GLM-4.5V, Qwen~~2.5~~VL) on Medical VQA along two trust-relevant axes. Perception: all models localize anatomical and pathological targets poorly -- the best model reaches only 0.23 mean IoU and 19.1% [email protected] -- and exhibit clinically dangerous laterality confusion. Pipeline integration: a self-grounding pipeline, where the same model localizes then answers, degrades VQA accuracy for every model -- driven by both inaccurate localization and format-compliance failures under the two-step prompt (parse failure rises to 70%--99% for Gemini and GPT-5 on VQA-RAD). Replacing predicted boxes with ground-truth annotations recovers and improves VQA accuracy, consistent with the failure residing in the perception module rather than in the decomposition itself. These observational findings identify grounding quality as a primary trustworthiness bottleneck in our SLAKE bounding-box setting. As a complementary fine-tuning follow-up, supervised fine-tuning of Qwen~~2.5~~VL on combined Med-VQA training data attains the highest reported SLAKE open-ended recall (85.5%) among comparable methods, suggesting that the VQA-level gap is tractable with domain adaptation; whether this also closes the perception/trustworthiness bottleneck is left to future work.