Medical VQA

VQA: Visual Question Answering

Medical visual question answering (MedVQA) asks models to interpret images such as X-rays, scans, pathology slides, and surgical videos in response to clinical questions, supporting tasks from finding specific abnormalities to integrating evidence across views or time. Common approaches adapt vision-language models with medical image-question datasets, while newer methods add region grounding, clinical knowledge, stepwise reasoning, or tools for measurements and image navigation. Benchmarks increasingly test spatial, temporal, multilingual, and multi-image reasoning, as well as whether answers actually depend on the visual evidence. A central challenge remains that high answer accuracy can coexist with shortcut use, hallucinations, poor calibration, or weak generalization across modalities and clinical settings.

Latest papers

175 papers19 papers in the last four weeks, up 46% on the four weeks before. 0.1% of all new papers.

How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images
Mar 15, 2026Guimeng Liu, Tianze Yu, Somayeh Ebrahimkhani +3Medical VLMs Medical Image Grounding
Singapore University of Technology and Design, Singapore · Department of Neurology, National Neuroscience Institute, Singapore · Duke-NUS Medical School, Singapore
Step-CoT: Stepwise Visual Chain-of-Thought for Medical Visual Question Answering
Mar 14, 2026Lin Fan, Yafei Ou, Zhipeng Deng +8Medical VQA Visual CoT
Southwest Jiaotong University · RIKEN · Hokkaido University +4
Beyond Medical Diagnostics: How Medical Multimodal Large Language Models Think in Space
Mar 14, 2026Quoc-Huy Trinh, Xi Ding, Yang Liu +73D Spatial Reasoning Multimodal Large Language Models
TemporalDoRA: Temporal PEFT for Robust Surgical Video Question Answering
Mar 10, 2026Luca Carlini, Chiara Lena, Cesare Hassan +4Video QA Parameter-Efficient Fine-Tuning
Dipartimento di Elettronica, Informazione e Bioingegneria (DEIB), Politecnico di Milano, Italy · IRCCS Humanitas Research Hospital, Italy · UCL Hawkes Institute and Department of Computer Science, University College London, UK +1
Med-Evo: Test-time Self-evolution for Medical Multimodal Large Language Models
Mar 8, 2026Dunyuan Xu, Xikai Yang, Juzheng Miao +3Multimodal Large Language Models Test-Time Adaptation of VLMs
Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China · Institute of Medical Intelligence and XR, The Chinese University of Hong Kong, Hong Kong, China
Deep Expert Injection for Anchoring Retinal VLMs with Domain-Specific Knowledge
Mar 7, 2026Shuai Lu, Meng Wang, Jia Guo +6Vision-Language Models Medical VLMs
Beijing Institute of Technology, Beijing, China · Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore · National University of Singapore, Singapore +2
SUREON: A Benchmark and Vision-Language-Model for Surgical Reasoning
Mar 6, 2026Alejandra Perez, Anita Rau, Lee White +4Surgical Video Understanding Medical VQA
Intuitive Surgical Inc., Sunnyvale, United States
Location-Aware Pretraining for Medical Difference Visual Question Answering
Mar 5, 2026Denis Musinguzi, Caren Han, Prasenjit MitraMedical VQA Vision-Language Pretraining
Department of Electrical and Computer Engineering, Carnegie Mellon University, Kigali, Rwanda · University of Melbourne, Melbourne, Australia.
Beyond Accuracy: Evaluating Visual Grounding In Multimodal Medical Reasoning
Mar 3, 2026Anas Zafar, Leema Krishna Murali, Ashish VashistVisual Grounding Medical VQA
The University of Texas MD Anderson Cancer Center · Cohere Labs Community · Eisai Inc. +1
Better Eyes, Better Thoughts: Why Vision Chain-of-Thought Fails in Medicine
Mar 2, 2026Yuan Wu, Zongxian Yang, Jiayu Qian +5Medical VLMs VLM Reasoning
Department of Computer Science, City University of Hong Kong (Dongguan) · College of Computing and Data Science (CCDS), Nanyang Technological University · School of Computer Science, Northwestern Polytechnical University
CARE: Towards Clinical Accountability in Multi-Modal Medical Reasoning with an Evidence-Grounded Agentic Framework
Mar 2, 2026Yuexi Du, Jinglu Wang, Shujie Liu +2Medical VLMs Medical Reasoning
Microsoft Research Asia · Department of Biomedical Engineering, Yale University · Department of Radiology & Biomedical Imaging, Yale University
When Does RL Help Medical VLMs? Disentangling Vision, SFT, and RL Gains
Mar 1, 2026Ahmadreza Jeddi, Kimia Shaban, Negin Baghbanzadeh +4Medical VLMs RL for VLMs
University of Toronto, Canada · Vector Institute, Canada · KITE Research Institute, University Health Network, Canada +1
ClinCoT: Clinical-Aware Visual Chain-of-Thought for Medical Vision Language Models
Mar 1, 2026Xiwei Liu, Yulong Li, Xinlin Zhuang +5Vision-Language Models Medical VLMs
Mohamed bin Zayed University of Artificial Intelligence · East China Normal University
DUCX: Decomposing Unfairness in Tool-Using Chest X-ray Agents
Feb 28, 2026Zikang Xu, Ruinan Jin, Xiaoxiao LiAlgorithmic Fairness Algorithmic Bias
Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Anhui, China · The University of British Columbia, Vancouver, BC V6Z 1Z4, Canada · Vector Institute, Toronto, ON M5G 1M1, Canada
Act Like a Pathologist: Tissue-Aware Whole Slide Image Reasoning
Feb 28, 2026Wentao Huang, Weimin Lyu, Peiliang Lou +8Pathology VQA Medical VQA
Stony Brook University, NY, USA · Mayo Clinic, MN, USA · Harvard Medical School, MA, USA +1
OPGAgent: An Agent for Auditable Dental Panoramic X-ray Interpretation
Feb 28, 2026Zhaolin Yu, Litao Yang, Ben Babicka +7Dental Imaging Medical Imaging
AIM for Health Lab, Faculty of Information Technology, Monash University, Melbourne, Australia · Faculty of Information Technology, Monash University, Melbourne, Australia · Curae Health, Melbourne, Australia +4
Mechanistically Guided LoRA Improves Paraphrase Consistency in Medical Vision-Language Models
Feb 24, 2026Binesh Sadanandan, Vahid BehzadanMedical VLMs Medical VQA
SAIL Lab, University of New Haven, CT, USA
LUMEN: Longitudinal Multi-Modal Radiology Model for Prognosis and Diagnosis
Feb 24, 2026Zhifan Jiang, Dong Yang, Vishwesh Nath +7Radiology VLMs Medical VQA
Sheikh Zayed Institute for Pediatric Surgical Innovation, Children’s National Hospital, Washington DC, USA · Nvidia Corporation, Santa Clara, CA, USA · ETSI Telecomunicación, Universidad Politécnica de Madrid, Madrid, Spain +1
Hepato-LLaVA: An Expert MLLM with Sparse Topo-Pack Attention for Hepatocellular Pathology Analysis on Whole Slide Images
Feb 23, 2026Yuxuan Yang, Zhonghao Yan, Yi Zhang +6Pathology VQA Medical VLMs
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing 100876, China · Department of Pathology, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing 100021, China
3DMedAgent: Unified Perception-to-Understanding for 3D Medical Analysis
Feb 20, 2026Ziyue Wang, Linghan Cai, Chang Han Low +83D Medical Imaging Medical VLMs
National University of Singapore · Microsoft Research · TUD Dresden University of Technology +2
CT-Bench: A Benchmark for Multimodal Lesion Understanding in Computed Tomography
Feb 16, 2026Qingqing Zhu, Qiao Jin, Tejas S. Mathai +10Medical Image Benchmarks Computed Tomography
MedAD-R1: Consistency-Reinforced Policy Optimization for Interpretable Medical Anomaly Detection
Feb 1, 2026Haitao Zhang, Yingying Wang, Jiaxiang Wang +5Healthcare Group Relative Policy Optimization
School of Informatics, Xiamen University · Institute of Artificial Intelligence, Xiamen University · Zhejiang Expressway Co., Ltd. +3
Ophiuchus: Incentivizing Tool-augmented "Think with Images" for Joint Medical Segmentation, Understanding and Reasoning
Dec 16, 2025Yankai Jiang, Yujie Zhang, Peng Zhang +5Image Segmentation Tool Use in VLMs
Zhejiang University · Shanghai Artificial Intelligence Laboratory · Shanghai Innovation Institute +5
Echo-CoPilot: A Multiple-Perspective Agentic Framework for Reliable Echocardiography Interpretation
Dec 6, 2025Moein Heidari, Ali Mehrabian, Mohammad Amin Roohi +5Medical VQA Echocardiography
The University of British Columbia, Vancouver, BC, Canada · Rutgers Cancer Institute, New Brunswick, NJ, USA
MultiViewDx: Evidence-Linked Multi-View Clinical Diagnosis
Oct 19, 2024Junda Wang, Zonghai Yao, Yujan Ting +5Healthcare Medical Image Analysis
United Imaging Intelligence, MA, USA · Manning College of Information and Computer Sciences, University of Massachusetts Amherst, MA, USA · Department of Medicine, University of Massachusetts Medical School, Worcester, MA, USA +1

Mar 15, 2026J/K move · Enter open · S save

How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images

Guimeng Liu, Tianze Yu, Somayeh Ebrahimkhani, Lin Zhi Zheng Shawn+2

Singapore University of Technology and Design, Singapore · Department of Neurology, National Neuroscience Institute, Singapore · Duke-NUS Medical School, Singapore

Medical VLMs Medical Image Grounding VLM Evaluation Medical VQA

Open paper AskPDF arXiv

Generalist multimodal large language models (MLLMs) have achieved impressive performance across a wide range of vision-language tasks. However, their performance on medical tasks, particularly in zero-shot settings where generalization is critical, remains suboptimal. A key research gap is the limited understanding of why medical MLLMs underperform in medical image interpretation. In this work, we present a pioneering systematic investigation into the visual grounding capabilities of state-of-the-art medical MLLMs. To disentangle visual grounding from semantic grounding, we design VGMED, a novel evaluation dataset developed with expert clinical guidance, explicitly assessing the visual grounding capability of medical MLLMs. We introduce new quantitative metrics and conduct detailed qualitative analyses. Our study across eight state-of-the-art (SOTA) medical MLLMs validates that they often fail to ground their predictions in clinically relevant image regions. We note that this finding is specific to medical image analysis; in contrast, prior work has shown that MLLMs are capable of grounding their predictions in the correct image regions when applied to natural scene images. Motivated by these findings, we propose VGRefine, a simple yet effective inference-time method that refines attention distribution to improve visual grounding in medical settings. Our approach achieves SOTA performance across 6 diverse Med-VQA benchmarks (over 110K VQA samples from 8 imaging modalities) without requiring additional training or external expert models. Overall, our work, for the first time, systematically validates inadequate visual grounding as one of the key contributing factors for medical MLLMs' under-performance. Additional experiments are included in the Supp.