Medical VLMs

VLM: Vision-Language Model

Ask about this topic

Overview

Medical vision-language models study how to link images such as X-rays, CT scans, pathology slides, and surgical videos with clinical language. Common approaches include CLIP-style image-text pretraining, adapting large multimodal language models with medical data, and adding region-level grounding, segmentation, or retrieval of clinical evidence. They are evaluated on tasks such as report generation on MIMIC-CXR, visual question answering on VQA-RAD and SLAKE, and volumetric CT or whole-slide pathology analysis. A central challenge is ensuring that answers genuinely use relevant visual evidence and remain calibrated under prompt changes, rare findings, and shifts between hospitals or populations.

Latest papers

CardsList

479 papers51 papers in the last four weeks, up 21% on the four weeks before. 0.3% of all new papers.

  1. PathGLS: Evaluating Pathology Vision-Language Models without Ground Truth through Multi-Dimensional Consistency

    Mar 17, 2026Minbing Chen, Zhu Meng, Fei SuVLM EvaluationMedical VLMs

    Beijing University of Posts and Telecommunications

  2. Medical Image Spatial Grounding with Semantic Sampling

    Mar 15, 2026Andrew Seohwan Yu, Mohsen Hariri, Kunio Nakamura +3Medical VLMsMedical Image Grounding

    Case Western Reserve University, Cleveland OH 44106, USA · Cleveland Clinic, Cleveland OH 44106, USA

  3. A comprehensive multimodal dataset and benchmark for ulcerative colitis scoring in endoscopy

    Mar 15, 2026Noha Ghatwary, Jiangbei Yue, Ahmed Elgendy +8Medical Image BenchmarksEndoscopy

    Computer Engineering Department, Arab Academy for Science and Technology,Smart Village, Giza, Egypt · AI in Medicine and Surgery group, School of Computer Science, University of Leeds, LS2 9JT, Leeds, United Kingdom · Department of Electrical and Computer Engineering, Queen’s University, Canada +5

  4. How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images

    Mar 15, 2026Guimeng Liu, Tianze Yu, Somayeh Ebrahimkhani +3Medical VLMsMedical Image Grounding

    Singapore University of Technology and Design, Singapore · Department of Neurology, National Neuroscience Institute, Singapore · Duke-NUS Medical School, Singapore

  5. EviAgent: Evidence-Driven Agent for Radiology Report Generation

    Mar 14, 2026Tuoshi Qi, Shenshen Bu, Yingfei Xiang +1Radiology Report GenerationMedical VLMs

    School of Computer Science and Engineering, Sun Yat-sen University · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Sangfor Technologies, Shenzhen, Guangdong, China

  6. Surg-R1: A Hierarchical Reasoning Foundation Model for Scalable and Interpretable Surgical Decision Support with Multi-Center Clinical Validation

    Mar 12, 2026Jian Jiang, Chenxi Lin, Yiming Gu +24Medical VLMsSurgical Video Understanding

    Global College, Shanghai Jiao Tong University, Shanghai, China · Institute for Medical Engineering & Science, Massachusetts Institute of Technology, Cambridge, MA, USA · Department of Biomedical Engineering and Department of Electrical and Computer Engineering, National University of Singapore, Singapore +11

  7. Human Knowledge Integrated Multi-modal Learning for Single Source Domain Generalization

    Mar 12, 2026Ayan Banerjee, Kuntal Thakur, Sandeep GuptaDomain GeneralizationCross-Domain Generalization

    Impact Lab, Arizona State University

  8. MedPruner: Training-Free Hierarchical Token Pruning for Efficient 3D Medical Image Understanding in Vision-Language Models

    Mar 12, 2026Shengyuan Liu, Zanting Ye, Yunrui Lin +6Efficient VLM Inference3D Medical Imaging

    Chinese University of Hong Kong · Westlake University · Southern Medical University +4

  9. Diabetic Retinopathy Grading with CLIP-based Ranking-Aware Adaptation:A Comparative Study on Fundus Image

    Mar 12, 2026Sungjun ChoDiabetic Retinopathy GradingRetinal Disease Classification

    Hong Kong University of Science and Technology (HKUST), Hong Kong, China

  10. Evidential learning driven Breast Tumor Segmentation with Stage-divided Vision-Language Interaction

    Mar 11, 2026Jingxing Zhong, Qingtao Pan, Xuchang Zhou +2Medical Image SegmentationText-Guided Image Segmentation

    Center for Precision Medicine, The First Affiliated Hospital of Xiamen University, School of Medicine, Xiamen University, Xiamen, China; Department of Laboratory Medicine, The First Affiliated Hospital of Xiamen University, School of Medicine, Xiamen University, Xiamen, 361003, China · Maynooth International Engineering College, Fuzhou University, Fuzhou, 350108, China · Department of Computer and Data Science, Case Western Reserve University, Cleveland, Ohio, USA, Cleveland, 44106, Ohio, USA +2

  11. PathMem: Toward Cognition-Aligned Memory Transformation for Pathology MLLMs

    Mar 10, 2026Jinyue Li, Yuci Liang, Qiankun Li +7Memory-Augmented VLMsMedical VLMs

    University of Science and Technology of China · Shenzhen University · Nanyang Technological University +2

  12. TubeMLLM: A Foundation Model for Topology Knowledge Exploration in Vessel-like Anatomy

    Mar 10, 2026Yaoyu Liu, Minghui Zhang, Xin You +2Medical VLMsTopology-Preserving Image Segmentation

    Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, China · Department of Automation, Shanghai Jiao Tong University, Shanghai, China

  13. MedKCO: Medical Vision-Language Pretraining via Knowledge-Driven Cognitive Orchestration

    Mar 10, 2026Chenran Zhang, Ruiqi Wu, Tao Zhou +1Medical VLMsVision-Language Pretraining

    School of Computer Science and Engineering, Southeast University, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Ministry of Education, China · School of Computer Science and Engineering, Nanjing University of Science and Technology, China

  14. Meissa: Multi-modal Medical Agentic Intelligence

    Mar 9, 2026Yixiong Chen, Xinyi Bai, Yue Pan +2Medical AI AgentsMedical LLMs

    Johns Hopkins University, Baltimore, MD, USA · Cornell University, Ithaca, NY, USA

  15. Vision-Language Models Encode Clinical Guidelines for Concept-Based Medical Reasoning

    Mar 9, 2026Mohamed Harmanani, Bining Long, Zhuoxin Guo +6Explainable Medical Image AnalysisMedical Reasoning

    Queen’s University · Vector Institute · McMaster University +1

  16. BioGait-VLM: A Tri-Modal Vision-Language-Biomechanics Framework for Interpretable Clinical Gait Assessment

    Mar 9, 2026Erdong Chen, Yuyang Ji, Jacob K. Greenberg +5Gait AnalysisMedical VLMs

    Department of Computer Science, Drexel University · Department of Neurological Surgery, Washington University · University of California, Berkeley

  17. MedQ-Deg: A Multidimensional Benchmark for Evaluating MLLMs Across Medical Image Quality Degradations

    Mar 8, 2026Jiyao Liu, Junzhi Ning, Chenglong Ma +14MLLM EvaluationMedical VLMs

    Fudan University, Shanghai, China · Shanghai AI Lab, Shanghai, China · Shanghai Jiaotong University, Shanghai, China +1

  18. Prompt-Based Caption Generation for Single-Tooth Dental Images Using Vision-Language Models

    Mar 8, 2026Anastasiia Sukhanova, Aiden Taylor, Julian Myers +5Dental ImagingImage Captioning

    Marshall University · West Virginia State University

  19. PromptGate Client Adaptive Vision Language Gating for Open Set Federated Active Learning

    Mar 7, 2026Adea Nesturi, David Dueñas Gaviria, Jiajun Zeng +1Active LearningFederated Learning

    University of Bonn, University Hospital Bonn, Clinic for Diagnostic and Interventional Radiology, 53127 Bonn, Germany

  20. Deep Expert Injection for Anchoring Retinal VLMs with Domain-Specific Knowledge

    Mar 7, 2026Shuai Lu, Meng Wang, Jia Guo +6Vision-Language ModelsMedical VLMs

    Beijing Institute of Technology, Beijing, China · Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore · National University of Singapore, Singapore +2

  21. TrajPred: Trajectory-Conditioned Joint Embedding Prediction for Surgical Instrument-Tissue Interaction Recognition in Vision-Language Models

    Mar 7, 2026Jiajun Cheng, Xiaofan Yu, Subarna Tripathi +2Medical VLMs

  22. SUREON: A Benchmark and Vision-Language-Model for Surgical Reasoning

    Mar 6, 2026Alejandra Perez, Anita Rau, Lee White +4Surgical Video UnderstandingMedical VQA

    Intuitive Surgical Inc., Sunnyvale, United States

  23. OralGPT-Plus: Learning to Use Visual Tools via Reinforcement Learning for Panoramic X-ray Analysis

    Mar 6, 2026Yuxuan Fan, Jing Hao, Hong Chen +5Medical Image AnalysisMedical VLMs

    The Hong Kong University of Science and Technology (GZ) · School of Computer Science, Peking University · Faculty of Dentistry, The University of Hong Kong +3

  24. K-MaT: Knowledge-Anchored Manifold Transport for Cross-Modal Prompt Learning in Medical Imaging

    Mar 6, 2026Jiajun Zeng, Shadi AlbarqouniCross-Modal LearningMedical VLMs

    University of Bonn, University Hospital Bonn, Clinic for Diagnostic and Interventional Radiology, 53127 Bonn, Germany

  25. DERM-3R: A Resource-Efficient Multimodal Agents Framework for Dermatologic Diagnosis and Treatment in Real-World Clinical Settings

    Mar 6, 2026Jirui Dai, Zhendong Wang, Chongjing Wang +7Traditional Chinese MedicineDermatology

  26. Evaluating GPT-5 as a Multimodal Clinical Reasoner: A Landscape Commentary

    Mar 5, 2026Alexandru Florea, Shansong Wang, Mingzhe Hu +5Clinical ReasoningMultimodal QA

    Department of Radiation Oncology, Winship Cancer Institute, Emory University School of Medicine · Department of Biomedical Engineering, Georgia Institute of Technology

  27. Thinking with Gaze: Sequential Eye-Tracking as Visual Reasoning Supervision for Medical VLMs

    Mar 5, 2026Yiwei Li, Yifan Zhou, Huaqin Zhao +6Radiology VLMsMedical VLMs

    School of Computing, University of Georgia · Massachusetts General Hospital, Harvard Medical School · Department of Biomedical Engineering, New Jersey Institute of Technology

  28. Logit-Level Uncertainty Quantification in Vision-Language Models for Histopathology Image Analysis

    Mar 3, 2026Betul Yurdem, Ferhat Ozgur Catak, Murat Kuzlu +1Uncertainty QuantificationMedical VLMs

    Department of Electrical and Electronics Engineering, Izmir Bakircay University, 35665 Izmir, Turkey · Department of Electrical Engineering and Computer Science, University of Stavanger, 4021 Stavanger, Norway · Batten College of Engineering and Technology, Old Dominion University, Norfolk, VA 23529, USA

  29. Semi-Supervised Few-Shot Adaptation of Vision-Language Models

    Mar 3, 2026Julio Silva-Rodríguez, Ender KonukogluVLM AdaptationFew-Shot Image Classification

    Computer Vision Lab, ETH Zurich, Zurich, Switzerland

  30. Intelligent Pathological Diagnosis of Gestational Trophoblastic Diseases via Visual-Language Deep Learning Model

    Mar 3, 2026Yuhang Liu, Yueyang Cang, Wenge Que +12Computational PathologyHistopathology Image Classification

    Department of Automation, Tsinghua University, Beijing, China · College of Materials Science and Engineering, Donghua University, Shanghai, China · Department of Pathology, Third Affiliated Hospital of Zhengzhou University, Zhengzhou, China +6