Medical Visual Question Answering

Also known as VQA

Medical visual question answering (MedVQA) asks models to interpret images such as X-rays, scans, pathology slides, and surgical videos in response to clinical questions, supporting tasks from finding specific abnormalities to integrating evidence across views or time. Common approaches adapt vision-language models with medical image-question datasets, while newer methods add region grounding, clinical knowledge, stepwise reasoning, or tools for measurements and image navigation. Benchmarks increasingly test spatial, temporal, multilingual, and multi-image reasoning, as well as whether answers actually depend on the visual evidence. A central challenge remains that high answer accuracy can coexist with shortcut use, hallucinations, poor calibration, or weak generalization across modalities and clinical settings.

Latest papers

175 papers19 papers in the last four weeks, up 46% on the four weeks before. 0.1% of all new papers.

FM $^2$ : Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging
Jul 15, 2026Shengchao Chen, Ting ShuMultimodal Federated Learning Cross-Modal Alignment
School of Artificial Intelligence, Shenzhen University Shenzhen, China · Australian AI Institute, University of Technology Sydney Sydney, Australia
Auditing Data Leakage in Whole-Slide Image Multimodal Benchmarks
Jul 14, 2026Wenhao Zhang, Zhongliang Zhou, John Kang +1VLM Evaluation Data Leakage
University of Virginia · Merck & Co., Inc.
An Empirical Analysis of Continual Learning for Heterogeneous Medical Visual Question Answering
Jul 13, 2026Mai A. Shaaban, Tausifa Jan Saleem, Alaa Mohamed +3Medical Visual Question Answering Continual Learning for VLMs
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Department of Mathematics and Computer Science, Faculty of Science, Alexandria University, Alexandria, Egypt
IRIS: An Intelligent Vision-Language System for Ocular Surface Diseases via Topic Tree and Scene-Driven VQA Generation
Jul 5, 2026Hao Wei, Wenjin Qi, Dasen Dai +2Visual Question Answering Vision-Language Models
The Chinese University of Hong Kong, Shatin, Hong Kong, China · Westlake University, Xihu District, Hangzhou, Zhejiang Province, China
Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs
Jul 4, 2026Anas Zafar, Leema Krishna Murali, Siddhant Bharadwaj +2VLM Evaluation Counterfactual Evaluation
The University of Texas MD Anderson Cancer Center · Eisai Inc. · Cohere Labs Community +1
Breaking Failure Cascades: Step-Aware Reinforcement Learning for Medical Multimodal Reasoning
Jun 30, 2026Junha Jung, Minbyul Jeong, Suhyeon Lim +5Process Reward Models Medical Visual Question Answering
Korea University · AIGEN Sciences · Upstage AI +3
Token-Sparse Medical Multimodal Reasoning via Dual-Stream Reinforcement Learning
Jun 30, 2026Kaitao Chen, Weiqian Zhao, Jiamin Wu +6Efficient Multimodal Inference Medical Visual Question Answering
1Fudan University · 2Shanghai Artificial Intelligence Laboratory · 3Shanghai Jiao Tong University +1
Synergistic Perception-Reasoning Governance: Grounding Medical MLLMs with Verifiable Anatomical Evidence
Jun 30, 2026Rui Hao, Qiankun Li, Junyuan Mao +4Radiology Report Generation Medical Visual Question Answering
Huazhong University of Science and Technology, Wuhan, China · Imperial Global Singapore, Imperial College London, London, United Kingdom · Nanyang Technological University, Singapore, Singapore +2
Aloe-Vision: Robust Vision-Language Models for Healthcare
Jun 25, 2026Jaume Guasch-Martí, Enrique Lopez-Cuena, Martín Suárez-Fernández +3VLM Evaluation VLM Robustness
Barcelona Supercomputing Center (BSC)
Just how sure are you? Improving Verbalized Uncertainty Calibration in Medical VQA
Jun 25, 2026Eren Senoglu, Federico Toschi, Nicolo Brunello +2Medical Visual Question Answering Language Model Calibration
Politecnico di Milano
SurgAtlas: A Large-Scale Surgical Video-Language Dataset with 2,391 Hours of Open and Minimally Invasive Surgery
Jun 24, 2026Filippos Bellos, Andre S. Gala-Garza, Miaowei Wang +8Medical Visual Question Answering Surgical Video Understanding
University of Michigan Ann Arbor, Michigan, USA · University of Edinburgh Edinburgh, United Kingdom · Vanderbilt University Medical Center Nashville, Tennessee, USA +3
Teach-to-Reason: Competition-Guided Reasoning with a Self-Improving Teacher
Jun 24, 2026Xiao Han, Hao Liu, Zhimin Bao +5Medical Visual Question Answering Medical VLMs
1AIRC, Midea Group
Multilingual Hematology Visual Question Answering Dataset
Jun 24, 2026Hajra Malik, Hafiza Tooba Aftab, Abdul Rehman +2Medical Visual Question Answering Medical VLMs
Information Technology University, Lahore · King Edward Medical University, Lahore
Scalable Training of Spatially Grounded 2D Vision-Language Models for Radiology
Jun 18, 2026Yusuf Salcan, Simon Ging, Robin Tibor Schirrmeister +4Radiology Report Generation Radiology VLMs
Computer Vision Group, University of Freiburg, Germany · CRIION-AI Lab, Freiburg, Germany · Adaptive & Agentic AI +3
Evaluation of Medical Vision Language Models HuluMed and MedGemma, and general purpose chatbots Gemma 3, ChatGPT Plus, and Claude Pro on real previously unseen wound images
Jun 16, 2026Yunzhe Xue, Mohammed Saim Ahmed Quadri, Neal Panse +2VLM Evaluation Medical Visual Question Answering
Department of Data Science, New Jersey Institute of Technology, Newark, NJ, USA · Department of Computer Science, New Jersey Institute of Technology, Newark, NJ, USA · Vascular and Endovascular Surgery, Robert Wood Johnson Hospital, New Brunswick, NJ, USA
Vision-language models for chest radiography do not always need the image
Jun 16, 2026Mahshad Lotfinia, Sebastian Ziegelmayer, Lisa Adams +4VLM Evaluation Medical Visual Question Answering
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Department of Diagnostic and Interventional Radiology, TUM University Clinic, School of Medicine and Health, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany · Lab for AI in Medicine, RWTH Aachen University, Aachen, Germany +1
Enhancing Pathological VLMs with Cross-scale Reasoning
Jun 16, 2026Chi Phan, Tianyi Zhang, Qiaochu Xue +5Visual Question Answering Computational Pathology
Department of Electrical and Computer Engineering, National University of Singapore, Singapore · PuzzleLogic Pte Ltd, Singapore · Department of Pathology, Fujian Medical University Cancer Hospital & Fujian Cancer Hospital, Fuzhou, China
Training LLMs with Reinforcement Learning over Digital Twin Representations for Reasoning-Intensive Surgical VideoQA
Jun 15, 2026Yiqing Shen, Han Zhang, Mathias UnberathRL for Language Model Reasoning Medical Visual Question Answering
Johns Hopkins University, Baltimore, MD, USA
Uncertainty Is Not a Safety Net for Clinical VQA, but Can It Anticipate Model Failure?
Jun 15, 2026Arnisa Fazla, Alberto Testoni, Ameen Abu-Hanna +2VLM Evaluation VLM Robustness
Department of Medical Informatics, Amsterdam University Medical Center, University of Amsterdam, Amsterdam, The Netherlands. · Amsterdam Public Health, Methodology, Amsterdam, The Netherlands. · MaiNLP, Center for Information and Language Processing, LMU Munich, Germany. +1
Adapt or Calibrate? How Local Labels Buy Certified Coverage in Medical VQA
Jun 14, 2026Reza Khanmohammadi, Kundan Thind, Mohammad M. GhassemiVLM Evaluation Confidence Estimation for VLMs
Michigan State University East Lansing, MI, USA · Henry Ford Health Detroit, MI, USA
Object Tokens as a Bridge Between Segmentation and Visual Question Answering in Robotic Surgery
Jun 14, 2026Yiping Li, Ronald de Jong, Romy van Jaarsveld +5Medical Visual Question Answering Surgical Video Understanding
Department of Biomedical Engineering, Eindhoven University of Technology, The Netherlands · Department of Surgery, University Medical Center Utrecht, The Netherlands · Department of Electrical Engineering, Eindhoven University of Technology, The Netherlands
OpenMedQ: Broad Open Pretraining for Medical Vision-Language Models
Jun 11, 2026Ibrahim Gulluk, Max Van Puyvelde, Olivier GevaertMedical Visual Question Answering Medical Image Classification
Department of Electrical Engineering, Stanford University · Department of Biomedical Data Science, Stanford University School of Medicine · Department of Mathematical Modelling, Statistics and Bioinformatics, Ghent University
OpenMedReason: Scientific Reasoning Supervision for Medical Vision-Language Models
Jun 10, 2026Negin Baghbanzadeh, Pritam Sarkar, Michael Colacci +6Medical Visual Question Answering Medical VLMs
York University · Vector Institute · University of British Columbia +5
UniReason-Med: A Shared Grounded Reasoning Interface for 2D-to-3D Transfer in Medical VQA
Jun 10, 2026Mengzhuo Chen, Yan Shu, Chi Liu +4Medical Visual Question Answering VLM Reasoning
IQuest Research
Measuring Pretraining Contamination in Public Medical Vision-Language Benchmarks
Jun 8, 2026Bruce Changlong Xu, Lan Wu, Alexander RyuVLM Evaluation Medical Visual Question Answering
Stanford University Stanford, CA, USA · University of California, Berkeley Berkeley, CA, USA · Mayo Clinic Rochester, MN, USA
Noise-Aware Visual Representation Learning for Medical Visual Question Answering
Jun 4, 2026I Putu Adi Pratama, Bahadorreza Ofoghi, Atul Sajjanhar +1VLM Robustness Denoising Autoencoders
Deakin University, 221 Burwood Highway, Burwood, Victoria 3125, Australia
Attention Consistent Longitudinal Medical Visual Question Answering Guided by Vision Foundation Models
Jun 3, 2026Jialin Wu, Qianru Zhang, Georges El Fakhri +1Vision Foundation Model Adaptation Medical Visual Question Answering
Yale Biomedical Imaging Institute New Haven, CT 06510
Does Language Shift Break Medical Vision-Language Models? Indonesian Radiology Visual Question Answering Case Study
Jun 2, 2026Pieter Christy Yan Yudhistira, Dzaki Rafif Malik, Novanto YudistiraVLM Evaluation VLM Robustness
Intelligent System Laboratory, Faculty of Computer Science Brawijaya University, Malang, Indonesia
Automated Report-Derived Oncology VQA Benchmark for Evaluating Vision-Language Models on 3D Medical Imaging
Jun 1, 2026Bo Liu, Hanxue Gu, Xiangru Li +6VLM Evaluation Medical Imaging
UCSF–UC Berkeley Joint Graduate Program in Bioengineering · Department of Radiology, UCSF · Department of Radiation Oncology, UCSF
Ask4VG: Risk-Aware Question Selection for Reducing Prior-Driven Answers in Medical VQA
May 31, 2026Xiaorong Zhu, Qiang Li, Zibo Xu +2Medical Visual Question Answering VLM Hallucination Mitigation
School of Microelectronics, Tianjin University, Tianjin 300072, China · DISI, University of Trento, Trento, Italy · School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China

Jul 15, 2026J/K move · Enter open · S save

FM $^2$ : Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging

Shengchao Chen, Ting Shu

School of Artificial Intelligence, Shenzhen University Shenzhen, China · Australian AI Institute, University of Technology Sydney Sydney, Australia

Multimodal Federated Learning Cross-Modal Alignment Medical Visual Question Answering Medical Image Classification Multimodal Foundation Models Federated Learning+1

Open paper AskPDF arXiv

Building foundation models for medical imaging requires pooling data across institutions, yet privacy regulations prohibit centralized aggregation. Existing Federated Foundation Models either fine-tune natural-image models with poor medical-domain transfer, or train from scratch within a single modality, lacking the flexibility to unify tasks. We identify an under-explored challenge, Imaging Modality Heterogeneity, where clients operate under two structural regimes: Overlapped (shared modalities with heterogeneous label distributions) and Non-overlapped (fully disjoint modalities per client). We propose FM

^2

, a unified framework that trains the core backbone from scratch to preserve medical domain fidelity while optionally incorporating biomedical pretrained encoders for vision-language alignment. FM

^2

equips each client with dual Mixture-of-Experts modules (a Class-wise MoE for personalized category knowledge and a Domain-wise MoE for shared cross-modality representations), coupled with a Heterogeneous Modality Alignment (HMA) regularizer that explicitly aligns modality-specific expert parameters, admitting provable

O(1/\sqrt{T})

convergence and generalization guarantees. FM

^2

further incorporates Caption-Enhanced Learning (CEL), where locally retained GPT-4o-generated captions serve as a textual semantic bridge enabling representation transfer across clients with disjoint modalities, and demonstrates extensibility to Federated Medical VQA. Experiments on our MIMH benchmark (classification and CEL) and real-world medical VQA datasets confirm consistent superiority over state-of-the-art federated baselines and strong out-of-modality generalization across all three tasks.