Medical Visual Question Answering

Also known as VQA

Medical visual question answering (MedVQA) asks models to interpret images such as X-rays, scans, pathology slides, and surgical videos in response to clinical questions, supporting tasks from finding specific abnormalities to integrating evidence across views or time. Common approaches adapt vision-language models with medical image-question datasets, while newer methods add region grounding, clinical knowledge, stepwise reasoning, or tools for measurements and image navigation. Benchmarks increasingly test spatial, temporal, multilingual, and multi-image reasoning, as well as whether answers actually depend on the visual evidence. A central challenge remains that high answer accuracy can coexist with shortcut use, hallucinations, poor calibration, or weak generalization across modalities and clinical settings.

Latest papers

175 papers19 papers in the last four weeks, up 46% on the four weeks before. 0.1% of all new papers.

GRASP: Group-Adaptive Hybrid RL-SFT Post-Training for Medical VQA without Reasoning Annotations
Oct 25, 2026Wanqi Dong, Wanyue Dong, Jingze Ge +1Medical Visual Question Answering Medical VLMs
National University of Singapore · Shandong Provincial Daizhuang Hospital
Conviction Without Evidence: Verdict Stability Is Independent of Visual Grounding in Medical Vision--Language Models
Oct 25, 2026Federico Felizzi, Francesco Andrea Causio, Bianca Destro Castaniti +17Medical VLMs Visual Evidence Sensitivity in VLMs
SIIAM, Rome, Italy · Dept. of Computer Science, University of Pennsylvania, USA · UCSC, Rome, Italy +4
MIS-VQA: An Annotation-Driven Surgical Vision-Language Dataset for Complex Surgical Understanding
Oct 25, 2026Mohamed Hamdy, Iheb Zouari, Fatmaelzahraa Ahmed +6Medical Visual Question Answering Surgical Video Understanding
Department of Computer Science and Engineering, Qatar University, Doha, Qatar · Hamad Medical Corporation, Doha, Qatar
Learning When to Collaborate: Selective Multi-Agent Medical Reasoning via Uncertainty-Aware Routing
Oct 24, 2026Jiacheng Hou, Xueliang Cui, Dan Lu +1Multi-Agent Reasoning Medical Visual Question Answering
SIGS, Tsinghua University · Southern University of Science and Technology · Shenzhen University of Advanced Technology, China +2
Enhancing RAG-based Medical VQA with Gated Multimodal Representations
Oct 16, 2026Daniela Jimenez-Pano, Hugo Jair Escalante, Delia Irazú Hernández-FaríasGated Multimodal Fusion Medical Visual Question Answering
LabTL - INAOE, Puebla, Mexico · Institute for AAII, UTEP, El Paso, TX, USA · Department of Computer Science, UTEP, El Paso, TX, USA
When Evidence Vanishes: Distilling Visual-Evidence Dependency in Medical Vision-Language Models
Oct 16, 2026Qizhen Lan, Yiling Ma, Ruiqi Chen +8Medical VLMs Visual Evidence Sensitivity in VLMs
The University of Texas Health Science Center at Houston · Yale University · University of Michigan +3
MedSTP: Preserving Subspace and Transition Evidence for Medical VQA Token Pruning
Oct 15, 2026Mengchen Fan, Qizhen Lan, Xi Xiao +6Visual Token Pruning Medical Visual Question Answering
University of Alabama at Birmingham · UTHealth Houston · Northeastern University +1
PlanPath: Exploration-Guided Orchestration for Patient-Level Pathology Reasoning
Oct 13, 2026Jiusong Ge, Mingzhao He, Qidong Liu +8Pathology Visual Question Answering Medical Visual Question Answering
Xi’an Jiaotong University · University of Cambridge · A*STAR +1
Seeing Through the Forgery Is Not Enough: Regressive Sycophancy under Fabricated Evidence in Medical Vision-Language Models
Oct 13, 2026Sadif Ahmed, Soumya Swagata Biswas, Rifat Shahriyar +1Sycophancy in VLMs Medical VLMs
Bangladesh University of Engineering and Technology
PathPilot: An Adaptive Question-Conditioned Multi-Magnification Navigation Agent for Whole-Slide Image Analysis
Oct 12, 2026Zhengyang Xu, Han Li, Jingsong Liu +5Whole-Slide Image Analysis Computational Pathology
Institute of Pathology, Technical University of Munich, Germany · Munich Center for Machine Learning, Munich, Germany · Chair of Computer Aided Medical Procedures, Technical University of Munich, Germany +4
HeiCo-FOCUS: A Clinically Grounded Dataset for Long-Context Video Understanding
Oct 7, 2026Leon Mayer, Lucas Luttner, Patrick Godau +42Long-Video Understanding Surgical Video Understanding
A doctrine-grounded visual question answering dataset for Tactical Combat Casualty Care
Oct 5, 2026Junseob Kim, Jade Chng, Ayman Ali +5Medical Visual Question Answering Video Question Answering
Department of Electrical and Computer Engineering, Duke University, Durham, NC, USA · Department of Biomedical Engineering, Duke University, Durham, NC, USA · Department of Surgery, Duke University, Durham, NC, USA +3
MedPrune: Topology-Efficient Multimodal Multi-Agent Communication Evolution for Medical VQA Tasks
Oct 5, 2026Jiuheng Wan, Runze Li, Chen Chen +5Communication Topology Optimization Medical Visual Question Answering
School of Computer Science and Information Engineering, Hefei University of Technology · Nanjing University · Guangdong University of Finance and Economics +2
BrainTRACE: Tracing Longitudinal, Multimodal, and Volumetric Evidence in Brain MRI Clinical Reasoning
Oct 5, 2026Qizhen Lan, Mengchen Fan, Hang Zhang +5VLM Evaluation 3D Medical Imaging
University of Texas Health Science Center at Houston · University of Alabama at Birmingham · University of Pittsburgh +3
Localization Lens for Improving Medical Vision-Language Models
Oct 3, 2026Hasan Farooq, Murtaza Taj, Mehwish Nasim +1Contrastive Learning for VLMs Medical VLMs
Lahore University of Management Sciences, Lahore, Pakistan · The University of Western Australia, Perth, Australia · Information Technology University of the Punjab, Lahore, Pakistan
CineMR: Tool-Integrated Vision-Language Reasoning for Quantitative Cardiac MRI Assessment
Oct 1, 2026Kunyang Li, Hai Nguyen, Joshua Lowe +6Tool Use in VLMs Medical Visual Question Answering
Institute for Artificial Intelligence, University of Central Florida · Department of Clinical Sciences, College of Medicine, University of Central Florida · Nemours Cardiac Center, Nemours Children’s Hospital, Florida +1
Learning Where to Look: Anatomical Grounding and Guided Attention for Cardiac MRI Vision-Language Models
Sep 30, 2026Bangwei Guo, Xiao Chen, Boris Mailhe +8Medical Visual Question Answering Medical Image Grounding
Rutgers University, NJ, USA · United Imaging Intelligence, Boston, MA, USA · University of Texas Southwestern Medical Center, TX, USA +1
CRAFT: Causal Responsibility and Failure Tracing in Medical Vision Language Models
Sep 30, 2026Chunzheng Zhu, Jiaqi Zeng, Hongbo Zhao +3Abstention in VLMs Vision-Language Models
Hunan University · University of Hong Kong
See, Measure, and Reason: Learning Visually Grounded Reasoning in Pathology
Sep 28, 2026Chengyang Zhang, Wenchuan Zhang, Bo Li +8Computational Pathology Medical Visual Question Answering
College of Computer Science, Sichuan University · Department of Pathology and Institute of Clinical Pathology, West China Hospital, Sichuan University · Department of Computer Science, School of Computing, National University of Singapore +1
EyeVQA: Benchmarking Ophthalmic Vision-Language Models from Recognition to Spatial Grounding
Sep 26, 2026Gujie Shao, Zixun Xie, Xuechun Xing +7VLM Evaluation Medical Visual Question Answering
Contamination, Prior, or Evidence? Decomposing and Training Evidence Use in Whole-Slide Vision-Language Models
Sep 26, 2026Wenhao Zhang, Zhongliang Zhou, Shiyuan Zhang +6Pathology Visual Question Answering Medical VLMs
University of Virginia, Charlottesville · Merck & Co., Inc.
From Reward Signal to Visual Utility: A Controlled Audit of Medical VLM Post-Training
Sep 25, 2026Wang JingxinVLM Post-Training Vision-Language Models
Institute of Neuroscience, CAS
Representation-guided in-context learning for medical image interpretation with multimodal large language models
Sep 21, 2026Minda Zhao, Fangyu Hu, Yan Luo +8Medical Image Analysis Multimodal Large Language Models
Harvard AI and Robotics Lab, Schepens Eye Research Institute of Massachusetts Eye and Ear, Harvard Medical School, Boston, MA, USA · Department of Ophthalmology, Massachusetts Eye and Ear, Harvard Medical School, Boston, MA, USA · Department of Ophthalmology, National Taiwan University Hospital, Taipei City, Taiwan +6
A Unified Vision-Language Model for PSMA PET/CT Report Generation, Visual Question Answering, and Lesion Segmentation
Sep 14, 2026Yang Xing, Jiong Wu, Savas Ozdemir +11Radiology Report Generation Medical Visual Question Answering
1J. Crayton Pruitt Family Department of Biomedical Engineering, University of Florida, Gainesville, FL, USA · Department of Radiology, University of Florida, Jacksonville, FL, USA · Department of Applied Mathematics & Statistics, Stony Brook University, Stony Brook, NY, USA +3
Option-Aware Retrieval and Task-Specific VLM Adaptation for Medical VQA
Sep 14, 2026Tristan Kirscher, Niklas C. Koser, Soren PirkVLM Adaptation Multiple-Choice Question Answering
ICube, Institut Strauss · CAU
STSG-VQA: Evidence-Grounded Temporal Question Answering from Surgical Spatio-Temporal Scene Graphs
Sep 8, 2026Jing Li, Duygu SarikayaMedical Visual Question Answering Question Answering
School of Computer Science, University of Leeds, UK
A visual large language foundational model for medical image recognition using clinician-contributed online resources
Sep 7, 2026Lingxuan Hou, Yuhua Xie, Yue Hu +14Medical Visual Question Answering Medical VLMs
College of Biomedical Engineering, Sichuan University, Chengdu, Sichuan, China · School of Engineering, Computer & Mathematical Sciences, Auckland University of Technology, Auckland, New Zealand · Guang’anmen Hospital, China Academy of Chinese Medical Sciences, Beijing, China +2
Separating Capability from Confidence: Grounded Dual-State Calibration for GRPO-Trained Medical Vision-Language Models
Sep 6, 2026Yangyang Xie, Ke Hao, Jiaqi Liu +2Confidence Estimation for VLMs VLM Calibration
Shanghai Jiao Tong University, Shanghai, China · Medical Image Insights Co. Ltd., Shanghai, China
MedProb: Probing Internal Representations of Vision-Language Models for Medical Question Answering
Sep 3, 2026Erfan Nourbakhsh, Ke Yang, Anthony RiosPosition Bias in VLMs Medical Visual Question Answering
The University of Texas at San Antonio
MedQA-MM: Shortcuts Behind Medical Visual Reasoning
Sep 3, 2026Benlu Wang, Yifan Zhang, Jiaqing Yu +7Medical Visual Question Answering Shortcut Learning
University of Massachusetts Amherst · 4Yale University · University of Massachusetts Lowell +6

Oct 25, 2026J/K move · Enter open · S save

GRASP: Group-Adaptive Hybrid RL-SFT Post-Training for Medical VQA without Reasoning Annotations

Wanqi Dong, Wanyue Dong, Jingze Ge, Mehul Motani

National University of Singapore · Shandong Provincial Daizhuang Hospital

Medical Visual Question Answering Medical VLMs RL for VLMs

Open paper AskPDF arXiv

Adapting general-purpose vision-language models (VLMs) to medical visual question answering (VQA) is valuable for helping clinicians read images and make diagnoses, but annotated data are scarce and provide only short answers, without reasoning annotations. Supervised fine-tuning (SFT) fits these labels but overfits to them and amplifies their biases, eroding out-of-domain (OOD) generalization. Reinforcement learning (RL) better preserves generalization but never uses the gold answer as a training target: GRPO-style algorithms learn from the contrast between correct and incorrect answers within a group of sampled responses, so all-correct or all-wrong groups give no correctness signal. Existing SFT-RL hybrids either apply SFT and RL in separate stages or switch between them per question, or need reasoning traces from experts or stronger models, which medical VQA data lack. We propose GRASP, which combines RL and SFT in every update using only short answers: RL lets the model explore its own reasoning, SFT teaches the gold answer after that reasoning to supply the direct supervision RL lacks, and their balance adapts to how well the model answers each question. Tuning only about 1% of the parameters with LoRA, GRASP outperforms all compared training objectives both in domain and on OOD open-ended questions across Qwen3-VL-8B, InternVL3-8B and Gemma 3-4B. On an OOD benchmark spanning eight imaging modalities, GRASP-trained InternVL3-8B and Qwen3-VL-8B beat all 10 compared general and medical VLMs in open-ended accuracy, including models trained on up to 2,600 times more data or with over 250 times more trainable parameters. We release MedTerse, a dataset of 12.9K short-answer questions, and our code at https://github.com/sea-comet/GRASP-MedVQA.