Medical VLMs

VLM: Vision-Language Model

Medical vision-language models study how to link images such as X-rays, CT scans, pathology slides, and surgical videos with clinical language. Common approaches include CLIP-style image-text pretraining, adapting large multimodal language models with medical data, and adding region-level grounding, segmentation, or retrieval of clinical evidence. They are evaluated on tasks such as report generation on MIMIC-CXR, visual question answering on VQA-RAD and SLAKE, and volumetric CT or whole-slide pathology analysis. A central challenge is ensuring that answers genuinely use relevant visual evidence and remain calibrated under prompt changes, rare findings, and shifts between hospitals or populations.

Latest papers

479 papers51 papers in the last four weeks, up 21% on the four weeks before. 0.3% of all new papers.

Iterative Multimodal Retrieval-Augmented Generation for Medical Question Answering
Apr 30, 2026Xupeng Chen, Binbin Shi, Chenqian Le +5Multimodal RAG Retrieval-Augmented Generation
New York University, New York, USA · Tsinghua University, Beijing, China · Independent Researcher +1
Auditing Frontier Vision-Language Models for Trustworthy Medical VQA: Grounding Failures, Format Collapse, and Domain Adaptation
Apr 30, 2026Xupeng Chen, Binbin Shi, Chenqian Le +5VLM Evaluation Healthcare
New York University, New York, USA · Tsinghua University, Beijing, China · Columbia University, New York, USA +1
CheXthought: A global multimodal dataset of clinical chain-of-thought reasoning and visual attention for chest X-ray interpretation
Apr 29, 2026Sonali Sharma, Jin Long, George Shih +7Chest X-Ray Classification Medical Image Analysis
Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Stanford, CA, USA · Department of Radiology, Stanford University, Stanford, CA, USA · Department of Pediatrics, Stanford University School of Medicine, Stanford, California, USA +6
MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolution
Apr 29, 2026Chunzheng Zhu, Jiaqi Zeng, Junyu Jiang +2Memory-Augmented Language Models Medical Image Analysis
Hunan University, Changsha, China
Toward Multimodal Conversational AI for Age-Related Macular Degeneration
Apr 28, 2026Ran Gu, Benjamin Hou, Mélanie Hébert +5Medical Diagnosis Medical VQA
Division of Intramural Research, NaMonal Library of Medicine, NaMonal InsMtutes of Health, Bethesda, Maryland, USA · 2. Division of Epidemiology and Clinical ApplicaMons, NaMonal Eye InsMtute, NaMonal InsMtutes of Health, Bethesda, Maryland, USA · 1. Division of Intramural Research, NaMonal Library of Medicine, NaMonal InsMtutes of Health, Bethesda, Maryland, USA
Learning from Medical Entity Trees: An Entity-Centric Medical Data Engineering Framework for MLLMs
Apr 28, 2026Jianghang Lin, Haihua Yang, Deli Yu +6Training Data Curation Multimodal QA
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, China · ByteDance · Northeastern University
Sum-of-Checks: Structured Reasoning for Surgical Safety with Large Vision-Language Models
Apr 24, 2026Weiqiu You, Cassandra Goldberg, Amin Madani +2Endoscopy Surgical Video Understanding
Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA, USA. · Department of Surgery, University of Toronto, Toronto, ON, Canada. · Surgical Artificial Intelligence Research Academy, University Health Network, Toronto, ON, Canada. +1
Weighting What Matters: Boosting Sample Efficiency in Medical Report Generation via Token Reweighting
Apr 22, 2026Alexander Weers, Daniel Rueckert, Martin J. MentenMedical Report Generation Healthcare
TUM School of Computation, Information and Technology, Technical University of Munich, DE · Munich Center for Machine Learning (MCML), DE · Department of Computing, Imperial College London, UK
InVitroVision: a Multi-Modal AI Model for Automated Description of Embryo Development using Natural Language
Apr 22, 2026Nicklas Neu, Thomas Ebner, Jasmin Primus +4VLM Adaptation Medical VLMs
Software Competence Center Hagenberg GmbH,Softwarepark 32a,Hagenberg, 4232, Upper Austria, Austria · Kinderwunsch Zentrum Kepler Universitätsklinikum,Krankenhausstraße 26, Linz, 4021 Upper Austria, Austria · Wunschkind Klinik Dr. Brunbauer, Ebendorferstraße 6/4 Vienna, 1010, Vienna, Austria
Infection-Reasoner: A Compact Vision-Language Model for Wound Infection Classification with Evidence-Grounded Clinical Reasoning
Apr 21, 2026Palawat Busaranuvong, Reza Saadati Fard, Emmanuel Agu +4Healthcare Medical Diagnosis
Worcester Polytechnic Institute, USA
REVEAL: Multimodal Vision-Language Alignment of Retinal Morphometry and Clinical Risks for Incident AD and Dementia Prediction
Apr 20, 2026Seowung Leem, Lin Gu, Chenyu You +2Alzheimer's Disease Medical Imaging
J. Crayton Pruitt Family Department of Biomedical Engineering, University of Florida, United2026 States · Research Institute of Electrical Communication, Tohoku University, Japan · Department of Applied Mathematics & Statistics, Stony Brook University, United StatesApr +1
ProtoCLIP: Prototype-Aligned Latent Refinement for Robust Zero-Shot Chest X-Ray Classification
Apr 20, 2026Florian Kittler, Sheethal Bhat, Andreas MaierVLM Robustness Chest X-Ray Classification
Friedrich-Alexander University Erlangen-Nuremberg, Erlangen 91054, Germany
Medical Image Understanding Improves Survival Prediction via Visual Instruction Tuning
Apr 20, 2026Xixi Liu, Jorge Lazo, Andreas Hallqvist +8Clinical Outcome Prediction Medical VLMs
Chalmers University of Technology, Gothenburg, Sweden
Can LLM-Generated Text Empower Surgical Vision-Language Pre-training?
Apr 20, 2026Chengan Che, Chao Wang, Jiayuan Huang +2Surgical Video Understanding Medical VLMs
Visual Understanding Research Group, Department of Informatics, King’s College London, UK
BioVLM: Routing Prompts, Not Parameters, for Cross-Modality Generalization in Biomedical VLMs
Apr 19, 2026Mainak Singha, Tanisha Gupta, Ankit Jha +3Cross-Modal Learning Prompt Learning
University of Trento, Italy · Carnegie Mellon University, USA · LNMIIT Jaipur, India +3
PBSBench: A Multi-Level Vision-Language Framework and Benchmark for Hematopathology Whole Slide Image Interpretation
Apr 19, 2026Yuanlong Wang, Weichi Chen, Adrian Rajab +4Computational Pathology Medical VQA
The Ohio State University · The Ohio State University Wexner Medical Center
When Background Matters: Breaking Medical Vision Language Models by Transferable Attack
Apr 19, 2026Akash Ghosh, Subhadip Baidya, Sriparna Saha +1VLM Robustness Adversarial Attacks on VLMs
Indian Institute of Technology Patna1 · Work done as Visiting Researcher at MBZUAI · MBZUAI3 +1
DREAM: Dynamic Retinal Enhancement with Adaptive Multi-modal Fusion for Expert Precision Medical Report Generation
Apr 19, 2026Nagur Shareef Shaik, Teja Krishna Cherukuri, Dong Hye YeMedical Imaging Medical Report Generation
Department of Computer Science, Georgia State University, Atlanta, GA USA
Bias-constrained multimodal intelligence for equitable and reliable clinical AI
Apr 18, 2026Cheng Li, Weijian Huang, Jiarun Liu +6Unified Multimodal Models Algorithmic Fairness
Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China. · Pengcheng Laboratory, Shenzhen, China. · University of Chinese Academy of Sciences, Beijing, China. +3
Expert-Annotated Embryo Image Dataset with Natural Language Descriptions for Evidence-Based Patient Communication in IVF
Apr 16, 2026Nicklas Neu, Thomas Ebner, Jasmin Primus +4Medical VLMs Explainable Medical Image Analysis
1Software Competence Center Hagenberg, Hagenberg, 4232, Austria · 2Kepler Universitätsklinikum, Kinderwunsch Zentrum, Linz, 4020, Austria · 3Wunschkind Klinik Dr. Brunbauer, Wien, 1010, Austria
MedRCube: A Multidimensional Framework for Fine-Grained and In-Depth Evaluation of MLLMs in Medical Imaging
Apr 15, 2026Zhijie Bao, Fangke Chen, Licheng Bao +4Multimodal Model Evaluation Medical VLMs
School of Data Science, Fudan University, China · Shanghai Innovation Institute, China · School of Integrated Circuits, Zhejiang University, China +2
CLIP Architecture for Abdominal CT Image-Text Alignment and Zero-Shot Learning: Investigating Batch Composition and Data Scaling
Apr 15, 2026Shivika, Kartik Bose, Pankaj GuptaMedical VLMs 3D Medical Imaging
Department of Radiodiagnosis and Imaging, PGIMER, Chandigarh, India-160012
Representation geometry shapes task performance in vision-language modeling for CT enterography
Apr 14, 2026Cristian Minoccheri, Emily Wittrup, Kayvan Najarian +1Medical VLMs Medical Image Classification
Gilbert S. Omenn Department of Computational Medicine and Bioinformatics, University of Michigan, 1109 Geddes Avenue, Ann Arbor, 48104, MI, USA. · Department of Emergency Medicine, University of Michigan, 1109 Geddes Avenue, Ann Arbor, 48104, MI, USA. · Department of Electrical Engineering and Computer Science, University of Michigan, 1109 Geddes Avenue, Ann Arbor, 48104, MI, USA. +1
MedConcept: Unsupervised Concept Discovery for Interpretability in Medical VLMs
Apr 13, 2026Md Rakibul Haque, KM Arefeen Sultan, Tushar Kataria +1Unsupervised Visual Concept Discovery VLM Interpretability
Kahlert School of Computing, University of Utah · Scientific Computing and Imaging Institute, University of Utah
Dialectic-Med: Mitigating Diagnostic Hallucinations via Counterfactual Adversarial Multi-Agent Debate
Apr 13, 2026Zhixiang Lu, Jionglong SuVLM Hallucination Mitigation Medical VLMs
Xi’an Jiaotong-Liverpool University
MedP-CLIP: Medical CLIP with Region-Aware Prompt Integration
Apr 13, 2026Jiahui Peng, He Yao, Jingwen Li +9Promptable Image Segmentation Multimodal Pretraining
aSichuan University, Chengdu, 610065, China · bXinjiang University, Urumqi, 830046, China · cAlibaba Group, Hangzhou, 310023, China +3
ReXSonoVQA: A Video QA Benchmark for Procedure-Centric Ultrasound Understanding
Apr 13, 2026Xucheng Wang, Xiaoman Zhang, Sung Eun Kim +2Medical VQA Video QA
Department of Biomedical Informatics, Harvard Medical School, Boston, MA
MedVeriSeg: Teaching LISA-Like Medical Segmentation Models to Verify Query Validity Without Extra Training
Apr 11, 2026Qinyue Tong, Xiaozhen Wang, Ziqian Lu +3Medical Image Segmentation Hallucination Detection in VLMs
School of Aeronautics and Astronautics, Zhejiang University, Hangzhou 310027, China · Southern Medical University, Guangzhou 510515, Guangdong, China · School of Computer Science and Technology (School of Artificial Intelligence), Zhejiang Sci-Tech University, Hangzhou 310018, Zhejiang, China +1
Improving Medical VQA through Trajectory-Aware Process Supervision
Apr 10, 2026Halil Ibrahim Gulluk, Olivier GevaertMedical VQA Medical VLMs
Electrical Engineering, Stanford University · Stanford Center For Biomedical Informatics Research
Is There Knowledge Left to Extract? Evidence of Fragility in Medically Fine-Tuned Vision-Language Models
Apr 10, 2026Oliver McLaughlin, Daniel Shubin, Carsten Eickhoff +3Medical VLMs VLM Reasoning
Brown University · University of Washington · University of Tübingen +1

Apr 30, 2026J/K move · Enter open · S save

Iterative Multimodal Retrieval-Augmented Generation for Medical Question Answering

Xupeng Chen, Binbin Shi, Chenqian Le, Jiaqi Zhang+4

New York University, New York, USA · Tsinghua University, Beijing, China · Independent Researcher+1

Multimodal RAG Retrieval-Augmented Generation Iterative RAG Healthcare Medical VLMs Medical QA+1

Open paper AskPDF arXiv

Medical retrieval-augmented generation (RAG) systems typically operate on text chunks extracted from biomedical literature, discarding the rich visual content (tables, figures, structured layouts) of original document pages. We propose MED-VRAG, an iterative multimodal RAG framework that retrieves and reasons over PMC document page images instead of OCR'd text. The system pairs ColQwen2.5 patch-level page embeddings with a sharded MapReduce LLM filter, scaling to ~350K pages while keeping Stage-1 retrieval under 30 ms via an offline coarse-to-fine index (C=8 centroids per page, ANN over centroids, exact two-way scoring on the top-R shortlist). A vision-language model (VLM) then iteratively refines its query and accumulates evidence in a memory bank across up to 3 reasoning rounds, with a single iteration costing ~15.9 s and the full three-round pipeline ~47.8 s on 4xA100. Across four medical QA benchmarks (MedQA, MedMCQA, PubMedQA, MMLU-Med), MEDVRAG reaches 78.6% average accuracy. Under controlled comparison with the same Qwen2.5-VL-32B backbone, retrieval contributes a +5.8 point gain over the no-retrieval baseline; we also note a +1.8 point edge over MedRAG + GPT-4 (76.8%), with the caveat that this is a cross-paper rather than head-to-head comparison. Ablations isolate +1.0 from page-image vs text-chunk retrieval, +1.5 from iteration, and +1.0 from the memory bank.