Medical VLMs

VLM: Vision-Language Model

Medical vision-language models study how to link images such as X-rays, CT scans, pathology slides, and surgical videos with clinical language. Common approaches include CLIP-style image-text pretraining, adapting large multimodal language models with medical data, and adding region-level grounding, segmentation, or retrieval of clinical evidence. They are evaluated on tasks such as report generation on MIMIC-CXR, visual question answering on VQA-RAD and SLAKE, and volumetric CT or whole-slide pathology analysis. A central challenge is ensuring that answers genuinely use relevant visual evidence and remain calibrated under prompt changes, rare findings, and shifts between hospitals or populations.

Latest papers

480 papers51 papers in the last four weeks, up 21% on the four weeks before. 0.3% of all new papers.

JMed48k: A Multi-Profession Japanese Medical Licensing Benchmark for Vision-Language Model Evaluation
May 21, 2026Yue Xun, Junyu Liu, Qian Niu +10VLM Evaluation Healthcare
The Hong Kong Polytechnic University · Kyoto University · The University of Tokyo +3
Universal CT Representations from Anatomy to Disease Phenotype through Agglomerative Pretraining
May 21, 2026Yuheng Li, Yuan Gao, Haoyu Dong +5Medical Imaging Foundation Models Medical VLMs
Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology and Emory University, Atlanta, GA 30332, USA · Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA 30322, USA · Department of Electrical and Computer Engineering, Duke University, Durham, NC 27705, USA +2
SurgOnAir: Hierarchy-Aware Real-Time Surgical Video Commentary
May 20, 2026Jingyi He, Yue Zhou, Long Bai +3Streaming Video Understanding Surgical Video Understanding
Computer Aided Medical Procedures (CAMP), TU Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · The Chinese University of Hong Kong, Hong Kong +1
VIHD: Visual Intervention-based Hallucination Detection for Medical Visual Question Answering
May 20, 2026Jiayi Chen, Benteng Ma, Zehui Liao +3Hallucination Detection in VLMs Healthcare
Department of Data Science & AI, Faculty of Information Technology, Monash University, Melbourne, VIC 3800, Australia · Hong Kong Polytechnic University, Hong Kong SAR, China · Alfred Health Radiology, Alfred Health, Melbourne, VIC 3004, Australia +1
MedExpMem: Adapting Experience Memory for Differential Diagnosis
May 20, 2026Qianhan Feng, Zhongzhen Huang, Yakun Zhu +4Radiology VLMs Healthcare
The Chinese University of Hong Kong · Shanghai Jiao Tong University · Shanghai Innovation Institue
HalluCXR: Benchmarking and Mitigating Hallucinations in Medical Vision-Language Models for Chest Radiograph Interpretation
May 19, 2026Haoyu Wang, Zitong LiVision-Language Models Audio-Language Model Hallucination Detection
Department of Biostatistics & Health Informatics, Institute of Psychiatry, Psychology & Neuroscience, King’s College London, London, UK
Rethinking Visual Attribution for Chest X-ray Reasoning in Large Vision Language Models
May 19, 2026Guangzhi Xiong, Qiao Jin, Sanchit Sinha +2Medical Visual Question Answering VLM Interpretability
University of Virginia · National Institutes of Health
Concept-Guided Noisy Negative Suppression for Zero-Shot Classification and Grounding of Chest X-Ray Findings
May 19, 2026Chenyu Lian, Hong-Yu Zhou, Chun-Ka Wong +1Contrastive Learning Chest X-Ray Classification
The Center for Smart Health, School of Nursing, the Hong Kong Polytechnic University, Hong Kong, China · Research Institute for Smart Ageing, the Hong Kong Polytechnic University,2026 Hong Kong, China · School of Biomedical Engineering, Tsinghua Medicine, Tsinghua University, Beijing, China +1
MAM-CLIP: Vision-Language Pretraining on Mammography Atlases for BI-RADS Classification
May 19, 2026Halil Ibrahim Gulluk, Olivier GevaertMedical Imaging Medical Image Classification
Department of Electrical Engineering Stanford University Stanford, CA 94305 · Biomedical Informatics Research (BMIR) Stanford University Stanford, CA 94304
MedFM-Robust: Benchmarking Robustness of Medical Foundation Models
May 18, 2026Xiangxiang Cui, Tianjin Huang, Yifang Wang +2Multimodal Robustness Neural Network Robustness
Beijing Normal University, China · University of Exeter, United Kingdom · University College London, United Kingdom +2
Geometry-Aware Uncertainty Coresets for Robust Visual In-Context Learning in Histopathology
May 18, 2026Franciskus Xaverius Erick, Johanna Paula Müller, Bernhard KainzVLM Robustness In-Context Learning
FAU Erlangen-Nürnberg, Erlangen, DE · Department of Computing, Imperial College London, London, UK
Wasserstein Equilibrium Decoding for Reliable Medical Visual Question Answering
May 18, 2026Luca Hagen, Johanna P. Müller, Weitong Zhang +2Healthcare Efficient VLM Inference
Friedrich-Alexander University Erlangen-Nürnberg, Erlangen, Germany · Imperial College London, London, United Kingdom · University College London, London, United Kingdom
How Good LLMs Are at Answering Bangla Medical Visual Questions? Dataset and Benchmarking
May 18, 2026Rafid Ahmed, Intesar Tahmid, Mir Sazzat Hossain +3VLM Evaluation Multilingual Language Model Evaluation
Penta Global Limited, Bangladesh · Center for Computational & Data Sciences, Independent University, Bangladesh
Segmentation, Detection and Explanation: A Unified Framework for CT Appearance Reasoning
May 15, 2026Yuyuan Liu, Can Peng, Yingyu Yang +3Image Segmentation Medical VLMs
Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford
BiomedAP: A Vision-Informed Dual-Anchor Framework with Gated Cross-Modal Fusion for Robust Medical Vision-Language Adaptation
May 15, 2026Huanyang Tong, Kai Liu, Fangjun Kuang +1VLM Robustness Prompt Learning
Wenzhou University · Wenzhou Business College
RoiMAM: Region-of-Interest Medical Attention Model for Efficient Vision-Language Understanding
May 15, 2026Jiayan Yang, Zhuoyu Wu, Wenqi FangHealthcare Efficient VLM Inference
Shenzhen Institutes of Advanced Technology, Chinese Academy of Science · CyPhi(ΨΦ) AI Research Lab, School of IT, Monash University, Malaysia Campus
DermAgent: A Self-Reflective Agentic System for Dermatological Image Analysis with Multi-Tool Reasoning and Traceable Decision-Making
May 14, 2026Yize Liu, Siyuan Yan, Ming Hu +5Medical Image Analysis LLM Agent Orchestration
AIM for Health Lab, Faculty of Information Technology, Monash University, Melbourne, Australia · Faculty of Information Technology, Monash University, Melbourne, Australia · University College London, Institute of Ophthalmology, London, United Kingdom
CA-GCL: Cross-Anatomy Global-Local Contrastive Learning for Robust 3D Medical Image Understanding
May 13, 2026Hanwen Zhang, Yao Liu, Die Dai +4VLM Robustness Medical Image Anomaly Detection
University of Electronic Science and Technology of China, Chengdu, China · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE
Verification Mirage: Mapping the Reliability Boundary of Self-Verification in Medical VQA
May 11, 2026Ruinan Jin, Beidi Zhao, Myeongkyun Kang +2VLM Evaluation Vision-Language Models
1The University of British Columbia · 2Vector Institute · 3Redmin University of China
Med-StepBench: A Hierarchical Reasoning Framework for Evaluating Hallucinations in Medical Vision-Language Models
May 11, 2026Minh Khoi Nguyen, Dai Lam Le, Amir Reza Jafari +8Healthcare Hallucination Detection
1AI4LIFE, Hanoi University of Science and Technology, Vietnam · 2SAMOVAR, Télécom SudParis, Institut Polytechnique de Paris, France · Military Central Hospital, Vietnam +1
LiteMedCoT-VL: Parameter-Efficient Adaptation for Medical Visual Question Answering
May 10, 2026Runze Ma, Shunbo Jia, Haonan Lyu +2CoT Distillation Healthcare
School of Information Technology Monash University Malaysia · Faculty of Innovation Engineering Macau University of Science and Technology · Department of Bioelectronics Faculty of Biomedical Engineering Shenzhen University of Advanced Technology +1
Cross-Modal Semantic-Enhanced Diffusion Framework for Diabetic Retinopathy Grading
May 10, 2026Yiqun WangDiabetic Retinopathy Grading Medical VLMs
Beijing Jiaotong University · School of Software Engineering Beijing Jiaotong University Beijing, China
CapCLIP: A Vision-Language Representation Alignment Approach for Wireless Capsule Endoscopy Analysis
May 8, 2026Haroon Wahab, Irfan Mehmood, Hassan UgailEndoscopy Medical Image Analysis
School of Computer Science, AI and Electronics Faculty of Engineering and Digital Technologies University of Bradford Bradford, UK · School of Management Faculty of Mgmt, Law & Social Sciences University of Bradford Bradford, UK · Centre for Visual Computing and Intelligent Systems University of Bradford Bradford, UK
MedVIGIL: Evaluating Trustworthy Medical VLMs Under Broken Visual Evidence
May 8, 2026Hanqi Jiang, Junhao Chen, Mingyu Kang +12VLM Evaluation VLM Robustness
University of Georgia · Chungbuk National University · Chungnam National University Hospital +5
Retina-RAG: Retrieval-Augmented Vision-Language Modeling for Joint Retinal Diagnosis and Clinical Report Generation
May 7, 2026Abdelrahman Zaian, Sheethal Bhat, Mohamed Abdalkader +1Diabetic Retinopathy Grading Medical Report Generation
Friedrich-Alexander-Universität, Germany
FairEnc: A Fair Vision-Language Model with Fair Vision and Text Encoders for Glaucoma Detection
May 6, 2026Mohamed Elhabebe, Ayman El-Baz, Qing LiuAlgorithmic Fairness Algorithmic Bias
Center for Machine Vision and Signal Analysis, University of Oulu, Finland · Department of Bioengineering, University of Louisville, USA · The Department of Physics and Engineering, UiT The Arctic University of Norway, Norway
MIRAGE: Retrieval and Generation of Multimodal Images and Texts for Medical Education
May 6, 2026Miguel Diaz Benito, Cecilia Diana Albelda, Alvaro Garcia Martin +3Multimodal IR Medical Education
Universidad Autónoma de Madrid, 28049 Madrid, Spain
Scientific Domain Knowledge Improves Vision-Language Fundus Models
May 4, 2026Verena Jasmin Hallitschke, Carsten Eickhoff, Philipp BerensVision-Language Pretraining Healthcare
1Hertie Institute for AI in Brain Health, University of Tübingen, Tübingen, Germany · Institute for Applied Medical Informatics, University of Tübingen, Tübingen, Germany · 2Tübingen AI Center, Tübingen, Germany
Ultrasound Vision-Language Alignment via Contrastive Learning
May 4, 2026Zhuoyang Lyu, Yiyang Zhang, Tongxin Wang +1Cross-Modal Alignment Ultrasound Imaging
Department of Biostatistics Harvard T. H. Chan School of Public Health Boston, MA, USA
Exploring Prompt Alignment with Clinical Factors in Zero-Shot Segmentation VLMs for NSCLC Tumor Segmentation
May 2, 2026Suraj Pai, Thibault Heintz, Cosmin Ciausu +3VLM Robustness Image Segmentation
Artificial Intelligence in Medicine Program, Mass General Brigham, Harvard Medical School, USA

May 21, 2026J/K move · Enter open · S save

JMed48k: A Multi-Profession Japanese Medical Licensing Benchmark for Vision-Language Model Evaluation

Yue Xun, Junyu Liu, Qian Niu, Xinyi Wang+9

The Hong Kong Polytechnic University · Kyoto University · The University of Tokyo+3

VLM Evaluation Healthcare Medical Visual Question Answering Medical VLMs Medical Image Benchmarks

Open paper AskPDF arXiv

We introduce JMed48k, a multi-profession Japanese healthcare licensing benchmark for evaluating vision-language models. Built from official PDF materials released by the Japanese Ministry of Health, Labour and Welfare, JMed48k contains 48,862 exam questions and 20,142 images from 11 national licensing examinations between 2005 and 2025, with visual content annotated under an 8-type taxonomy. From this corpus, we derive JMed48k-Eval, a recent five-year evaluation subset with 12,484 scored questions, including 9,905 text-only questions and 2,579 questions with images. We evaluate 21 proprietary, open-source, and medical-specific models, reporting text-only and with-image performance separately. Because these subsets contain different questions, we further introduce a paired image-removal audit that evaluates questions with images before and after removing visual content to explore four answer-transition states. The audit shows that proprietary and open source models gain substantially from images, whereas medical-specific systems show limited observable use of visual evidence, with many correct answers persisting after image removal. Even among proprietary models, the net image-removal effect varies sevenfold across professions, from +5.7 points on Physician questions to +39.8 points on Public Health Nurse questions. We release JMed48k to support reproducible, profession-stratified evaluation of vision-language models in medical licensing settings.