Medical VLMs

VLM: Vision-Language Model

Ask about this topic

Overview

Medical vision-language models study how to link images such as X-rays, CT scans, pathology slides, and surgical videos with clinical language. Common approaches include CLIP-style image-text pretraining, adapting large multimodal language models with medical data, and adding region-level grounding, segmentation, or retrieval of clinical evidence. They are evaluated on tasks such as report generation on MIMIC-CXR, visual question answering on VQA-RAD and SLAKE, and volumetric CT or whole-slide pathology analysis. A central challenge is ensuring that answers genuinely use relevant visual evidence and remain calibrated under prompt changes, rare findings, and shifts between hospitals or populations.

Latest papers

CardsList

479 papers51 papers in the last four weeks, up 21% on the four weeks before. 0.3% of all new papers.

  1. MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding

    Mar 24, 2026Basit Alawode, Arif Mahmood, Muaz Khalifa Al-Radi +6Whole-Slide Image AnalysisMedical VLMs

    Department of Computer Science, Khalifa University of Science and Technology, UAE. · Information Technology University, Pakistan. · KAU, KSA. +1

  2. To Agree or To Be Right? The Grounding-Sycophancy Tradeoff in Medical Vision-Language Models

    Mar 23, 2026OFM Riaz Rahman Aranya, Kevin DesaiSycophancy in VLMsMedical VLMs

    Department of Computer Science, The University of Texas at San Antonio

  3. A vision-language model and platform for temporally mapping surgery from video

    Mar 23, 2026Dani KiyassehSurgical Video UnderstandingMedical VLMs

    Halsted AI, Sunnyvale, California, 94086 USA

  4. MARCUS: An agentic, multimodal vision-language model for cardiac diagnosis and management

    Mar 23, 2026Jack W O'Sullivan, Mohammad Asadi, Lennart Elbe +10Medical VLMsMultimodal Understanding

    Division of Cardiology, Department of Medicine, Stanford University, CA, USA · Department of Biomedical Data Science, Stanford University, CA, USA · Department of Electrical Engineering, Stanford University, CA, USA +7

  5. Deterministic Hallucination Detection in Medical VQA via Confidence-Evidence Bayesian Gain

    Mar 23, 2026Mohammad Asadi, Tahoura Nedaee, Jack W. O'Sullivan +2Hallucination Detection in VLMsHallucination Detection

    Department of Electrical Engineering, Stanford University, CA, USA · Department of Biology, Stanford University, CA, USA · Division of Cardiology, Department of Medicine, Stanford University, CA, USA +3

  6. MIRAGE: The Illusion of Visual Understanding

    Mar 23, 2026Mohammad Asadi, Jack W. O'Sullivan, Fang Cao +5VLM EvaluationFaithfulness in VLMs

    Department of Electrical Engineering, Stanford University, CA, USA · Division of Cardiology, Department of Medicine, Stanford University, CA, USA · Department of Biomedical Data Science, Stanford University, CA, USA +3

  7. SleepVLM: A Rule-Grounded Vision-Language Model for Auditable Sleep Staging

    Mar 22, 2026Guifeng Deng, Pan Wang, Mengfan Niu +9Sleep Stage ClassificationMedical VLMs

    Affiliated Mental Health Center & Hangzhou Seventh People’s Hospital, School of Brain Science and Brain Medicine, and Liangzhu Laboratory, Zhejiang University School of Medicine, Hangzhou 310058, China · College of Biomedical Engineering & Instrument Science, Zhejiang University, Hangzhou 310058, China · Department of Psychiatry and Mental Health, Wenzhou Medical University, Wenzhou 325035, Zhejiang Province, China +2

  8. When Minor Edits Matter: LLM-Driven Prompt Attack for Medical VLM Robustness in Ultrasound

    Mar 22, 2026Yasamin Medghalchi, Milad Yazdani, Amirhossein Dabiriaghdam +7VLM RobustnessAdversarial Attacks on VLMs

    University of British Columbia, Vancouver, Canada

  9. SkinCLIP-VL: Consistency-Aware Vision-Language Learning for Multimodal Skin Cancer Diagnosis

    Mar 22, 2026Zhixiang Lu, Shijie Xu, Kaicheng Yan +6Medical VLMsVLM Adaptation

    Xi’an Jiaotong-Liverpool University, China · University of Liverpool, United Kingdom

  10. Consistent but Dangerous: Per-Sample Safety Classification Reveals False Reliability in Medical Vision-Language Models

    Mar 22, 2026Binesh Sadanandan, Vahid BehzadanVLM EvaluationMedical VLMs

    SAIL Lab, University of New Haven, West Haven, CT, USA

  11. Clinical Cognition Alignment for Gastrointestinal Diagnosis with Multimodal LLMs

    Mar 21, 2026Huan Zheng, Yucheng Zhou, Tianyi Yan +6EndoscopyMedical Diagnosis

    SKL-IOTSC, CIS, University of Macau · 2Shanghai Jiao Tong University · 3COWARobot Co. Ltd. +1

  12. Chain-of-Adaptation: Surgical Vision-Language Adaptation with Reinforcement Learning

    Mar 20, 2026Jiajie Li, Chenhui Xu, Meihuan Liu +1VLM AdaptationMedical VLMs

    University at Buffalo Buffalo, NY, USA

  13. HiPath: Hierarchical Vision-Language Alignment for Structured Pathology Report Prediction

    Mar 20, 2026Ruicheng Yuan, Zhenxuan Zhang, Anbang Wang +5Computational PathologyMedical Report Generation

    College of Computer Science and Electronic Engineering, Hunan University, Changsha, Hunan, China · Department of Bioengineering and Imperial-X, Imperial College London, London, UK · Department of Pathology, Xiangtan Maternal and Child Health Hospital, Xiangtan, Hunan, China +1

  14. ReXInTheWild: A Unified Benchmark for Medical Photograph Understanding

    Mar 19, 2026Oishi Banerjee, Sung Eun Kim, Alexandra N. Willauer +9Medical VLMsMedical Image Benchmarks

    Department of Biomedical Informatics, Harvard Medical School, Boston, MA · National Strategic Technology Research Institute, Seoul National University Hospital, Seoul, Republic of Korea · Department of Medicine, Division of Gastroenterology, Massachusetts General Hospital, Boston, MA +3

  15. Gastric-X: A Multimodal Multi-Phase Benchmark Dataset for Advancing Vision-Language Models in Gastric Cancer Analysis

    Mar 19, 2026Sheng Lu, Hao Chen, Rui Yin +3Medical VLMsMedical Image Benchmarks

    Ruijin Hospital · University of Cambridge · Nanjing First Hospital +2

  16. Instruction-Free Tuning of Large Vision Language Models for Medical Instruction Following

    Mar 19, 2026Myeongkyun Kang, Soopil Kim, Xiaoxiao Li +1Medical VLMsLarge Vision-Language Models

    Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, BC V6T 1Z4, Canada · Department of Robotics and Mechatronics Engineering, Daegu Gyeongbuk Institute of Science and Technology (DGIST), Daegu 42988, Republic of Korea · Division of Intelligent Robot, Daegu Gyeongbuk Institute of Science and Technology (DGIST), Daegu 42988, Republic of Korea +2

  17. LoFi: Location-Aware Fine-Grained Representation Learning for Chest X-ray

    Mar 19, 2026Myeongkyun Kang, Yanting Yang, Xiaoxiao LiMedical Image Representation LearningMedical VLMs

    The University of British Columbia, Vancouver, BC V6T 1Z4, Canada · Vector Institute, Toronto, ON M5G 0C6, Canada

  18. VGS-Decoding: Visual Grounding Score Guided Decoding for Hallucination Mitigation in Medical VLMs

    Mar 19, 2026Govinda Kolli, Adinath Madhavrao Dukre, Behzad Bozorgtabar +2VLM Hallucination MitigationMedical VLMs

    MBZUAI, Abu Dhabi, UAE · Rajiv Gandhi University of Knowledge Technologies, India · Aarhus University, Aarhus, Denmark +1

  19. CoDA: Exploring Chain-of-Distribution Attacks and Post-Hoc Token-Space Repair for Medical Vision-Language Models

    Mar 19, 2026Xiang Chen, Fangfang Yang, Chunlei Meng +6Medical Image Quality AssessmentVLM Robustness

  20. MedQ-UNI: Toward Unified Medical Image Quality Assessment and Restoration via Vision-Language Modeling

    Mar 19, 2026Jiyao Liu, Junzhi Ning, Wanying Qu +4Medical Image Quality AssessmentMedical Image Analysis

    Fudan University, Shanghai, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China

  21. Mind the Rarities: Can Rare Skin Diseases Be Reliably Diagnosed via Diagnostic Reasoning?

    Mar 19, 2026Yang Liu, Jiyao Yang, Hongjin Zhao +10Rare Disease DiagnosisMedical VLMs

    Carnegie Mellon University · University of Alabama at Birmingham · Australian National University +3

  22. Anatomy-Guided Vision-Language Learning with Angular Prototype Separation for Multi-Label Video Capsule Endoscopy Classification Under Class Imbalance

    Mar 18, 2026Podakanti Satyajith Chary, Nagarajan GanapathyMedical ImagingTemporal Video Understanding

    Department of Engineering Science, IIT Hyderabad · Department of Biomedical Engineering, IIT Hyderabad

  23. Concept-to-Pixel: Prompt-Free Universal Medical Image Segmentation

    Mar 18, 2026Haoyun Chen, Fenghe Tang, Wenxin Ma +1Medical Image SegmentationCross-Modality Medical Image Segmentation

    School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China (USTC), Hefei, Anhui 230026, China · Center for Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE), Suzhou Institute for Advanced Research, USTC, Suzhou, Jiangsu 215123, China · Jiangsu Provincial Key Laboratory of Multimodal Digital Twin Technology, Suzhou Jiangsu, 215123, China +1

  24. DiffVP: Differential Visual Semantic Prompting for LLM-Based CT Report Generation

    Mar 18, 2026Yuhe Tian, Kun Zhang, Haoran Ma +4CT Report GenerationRadiology Report Generation

    Department of Electronic Engineering and Information Science, School of Information Science and Technology, University of Science and Technology of China (USTC), Hefei, Anhui 230026, China · Center for Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE), Suzhou Institute for Advanced Research, University of Science and Technology of China (USTC), Suzhou, Jiangsu 215123, China · Jiangsu Provincial Key Laboratory of Multimodal Digital Twin Technology, University of Science and Technology of China (USTC), Suzhou, Jiangsu 215123, China +2

  25. DeepCORO-CLIP: A Multi-View Foundation Model for Comprehensive Coronary Angiography Video-Text Analysis and External Validation

    Mar 18, 2026Sarra Harrabi, Yichen Wu, Geoffrey H. Tison +6Coronary AngiographyMedical Imaging Foundation Models

    Division of Cardiology, Department of Medicine, Montreal Heart Institute, 5000 Belanger Street, QC, H1T 1C8 Canada · HeartWise.ai, 5000 Belanger Street, QC, H1T 1C8 Canada · McGill, Faculty of Engineering, Montreal, Quebec, Canada. +7

  26. MedSAD-CLIP: Supervised CLIP with Token-Patch Cross-Attention for Medical Anomaly Detection and Segmentation

    Mar 18, 2026Thuy Truong Tran, Minh Kha Do, Phuc Nguyen Duy +1Medical Image Anomaly DetectionMedical VLMs

    Singapore Management University · La Trobe University · Vietnam National University, Hanoi

  27. On the Cone Effect and Modality Gap in Medical Vision-Language Embeddings

    Mar 18, 2026David Restrepo, Miguel L Martins, Chenwei Wu +5Modality Gap in VLMsMultimodal Learning

    CentraleSupélec, Université Paris-Saclay, France · University of Porto, Portugal · University of Michigan, USA +3

  28. ACE-LoRA: Graph-Attentive Context Enhancement for Parameter-Efficient Adaptation of Medical Vision-Language Models

    Mar 17, 2026M. Arda Aydın, Melih B. Yilmaz, Aykut Koç +1VLM AdaptationMedical VLMs

    Bilkent University · National Magnetic Resonance Research Center (UMRAM)

  29. IOSVLM: A 3D Vision-Language Model for Unified Dental Diagnosis from Intraoral Scans

    Mar 17, 2026Huimin Xiong, Zijie Meng, Tianxiang Hu +33D VLMsMedical VLMs

    ZJU-UIUC Institute, Zhejiang University, Haining, 314400, China · Stomatology Hospital, School of Stomatology, Zhejiang University School of Medicine, Hangzhou, 310058, China · Angelalign Research Institute, Angel Align Inc., Shanghai, 200011, China

  30. InViC: Intent-aware Visual Cues for Medical Visual Question Answering

    Mar 17, 2026Zhisong Wang, Ziyang Chen, Zanting Ye +3Medical VQAMedical VLMs

    National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, School of Computer Science and Engineering, Northwestern Polytechnical University, Xi’an 710072, China · Southern Medical University · Westlake University