Medical VLMs

VLM: Vision-Language Model

Ask about this topic

Overview

Medical vision-language models study how to link images such as X-rays, CT scans, pathology slides, and surgical videos with clinical language. Common approaches include CLIP-style image-text pretraining, adapting large multimodal language models with medical data, and adding region-level grounding, segmentation, or retrieval of clinical evidence. They are evaluated on tasks such as report generation on MIMIC-CXR, visual question answering on VQA-RAD and SLAKE, and volumetric CT or whole-slide pathology analysis. A central challenge is ensuring that answers genuinely use relevant visual evidence and remain calibrated under prompt changes, rare findings, and shifts between hospitals or populations.

Latest papers

CardsList

479 papers51 papers in the last four weeks, up 21% on the four weeks before. 0.3% of all new papers.

  1. Aligning Medical Vision Language Models at the Capability Boundary with Entropy Adaptive Rubric Grounding

    Oct 29, 2026Hongyu Bai, Jiaqi Zeng, Chunzheng ZhuMedical VLMsReasoning for VLMs

    Shanghai Jiao Tong University Shanghai, China · Hunan University Changsha, China

  2. Visually Irrelevant Hallucinations in Medical Vision-Language Models

    Oct 28, 2026Ke Zou, Nan Zhou, Tao Yang +9VLM HallucinationHallucination Detection in VLMs

    K. Zou is with the Department of Ophthalmology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore 119228. · N. Zhou, T. Yang, M. Liu, L. He, and H. Chen are with the College of Computer Science, Sichuan University, Chengdu 610065, China. · Z. Liao is with the Faculty of Information Technology, Monash University, Clayton, Victoria 3800, Australia. +5

  3. Knowledge-structured and evidence-aware semantic self-distillation for glomerular pathology foundation model

    Oct 28, 2026Qiming He, Jing Li, Zihan Wang +24Pathology Foundation ModelsMedical VLMs

    Fuzhou University Affiliated Provincial Hospital, Interdisciplinary Institute for Medical Engineering, Fuzhou University, Fuzhou, China. · Medical Optical Technology R&D Center, Research Institute of Tsinghua, Pearl River Delta, Guangzhou, China. · Department of Pathology, School of Basic Medicine and Xijing Hospital, Fourth Military Medical University, Xi’an, China. +5

  4. Evaluating Federated Vision-Language and Vision-Only Models for Privacy-Aware Breast Cancer Imaging

    Oct 26, 2026Malithi Mithsara Wanniarachchi Kankanamge, Ning Yang, Sree Bash Debnath +2Medical ImagingMammography Image Classification

    School of Computing, Southern Illinois University, 1230 Lincoln Drive, Carbondale, 62901, IL, USA. · Department of Radiation Oncology, Stritch School of Medicine, Loyola University, 1032 W. Sheridan Rd, Chicago, 60660, IL, USA. · Department of Radiology, University of Texas Southwestern Medical Center, Dallas, 75390, TX, USA.

  5. GRASP: Group-Adaptive Hybrid RL-SFT Post-Training for Medical VQA without Reasoning Annotations

    Oct 25, 2026Wanqi Dong, Wanyue Dong, Jingze Ge +1Medical VQAMedical VLMs

    National University of Singapore · Shandong Provincial Daizhuang Hospital

  6. Conviction Without Evidence: Verdict Stability Is Independent of Visual Grounding in Medical Vision--Language Models

    Oct 25, 2026Federico Felizzi, Francesco Andrea Causio, Bianca Destro Castaniti +17Medical VLMsVisual Evidence Sensitivity in VLMs

    SIIAM, Rome, Italy · Dept. of Computer Science, University of Pennsylvania, USA · UCSC, Rome, Italy +4

  7. MIS-VQA: An Annotation-Driven Surgical Vision-Language Dataset for Complex Surgical Understanding

    Oct 25, 2026Mohamed Hamdy, Iheb Zouari, Fatmaelzahraa Ahmed +6Medical VQASurgical Video Understanding

    Department of Computer Science and Engineering, Qatar University, Doha, Qatar · Hamad Medical Corporation, Doha, Qatar

  8. Meta-CLIP: Metadata-Conditioned Contrastive Learning for Zero-Shot Disease Inference

    Oct 24, 2026Haozhe Luo, Nirosh Sivanesan, Shelley Zixin Shu +2Medical VLMsContrastive Learning for VLMs

    University of Bern, Bern, Switzerland · ARTORG Center for Biomedical Engineering Research, University of Bern, Bern, Switzerland

  9. Beyond Discrete Tokens: Preserving Numerical Continuity in Medical Vision-Language Models

    Oct 23, 2026Dianye Huang, Jiamei Guo, Yue Zhou +5Medical Image AnalysisMedical VLMs

    Medical Intelligence and Robotic Cognition (MIRoC) Lab, Department of Mechanical Engineering, The University of Hong Kong, Hong Kong SAR, China · Chair for Computer-Aided and Medical Procedures (CAMP), Tehcnical University of Munich, Munich, Germany · Department of Medical Ultrasonics, The First Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China +2

  10. Multimodal Generative AI Assists Comprehensive Echocardiographic Interpretation

    Oct 23, 2026Yi Qin, Marawan Elbatel, Shanshan Song +28EchocardiographyMedical VLMs

    Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China · Guangdong Cardiovascular Institute, Guangdong Provincial People’s Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, Guangdong Province, China · Li Ka Shing Institute of Health Sciences, Faculty of Medicine, The Chinese University of Hong Kong, Hong Kong SAR, China +19

  11. A Visual Grounding Benchmark for Retinal OCT with Multimodal Large Language Models

    Oct 22, 2026Ronald Fecso, José Morano, Botond Fazekas +6Medical VLMsMedical Image Grounding

    Institute of Artificial Intelligence, Center for Medical Data Science, Medical University of Vienna, Vienna, Austria · Department of Ophthalmology and Optometry, Medical University of Vienna, Vienna, Austria

  12. Expandable Modality Experts for Medical Vision-Language Learning

    Oct 21, 2026DM BAPPY, Eunsu Kim, Hanjoo Lee +2Medical VLMsContinual Learning for VLMs

    Korea University · The Catholic University of Korea

  13. Gated Prototype Transfer for Zero-Support Medical Vision Language Model

    Oct 20, 2026Sumaiya Fazal, Rizwan Qureshi, Laleh Behjat +1VLM AdaptationMedical VLMs

    FAST-NUCES · Massachusetts General Hospital, Harvard University · University of Calgary +1

  14. CPVB-RG: Cross-Prior Visual Bridging for Prior-Aware Chest X-Ray Report Generation

    Oct 20, 2026Sike Xiang, Shuang Chen, Julie Cox +1Radiology Report GenerationMedical VLMs

    Department of Computer Science, Durham University, UK · Department of Radiology, County Durham and Darlington NHS Foundation Trust, UK

  15. VGL-stGen: Learning a shared multimodal model for lung histology and spatial transcriptomics

    Oct 20, 2026Haenara Shin, Dongjoo Lee, Sungwoo Bae +3Gene Expression Prediction from Histopathology ImagesMedical VLMs

    Portrai, Inc., Seoul, Republic of Korea

  16. Reliability-Guided Vision-Language Decoding with Text-Aware Skip Refinement for Medical Image Segmentation

    Oct 20, 2026Sivan Talaei, Fatemeh Daneshfar, Abdulhady Abas Abdullah +1Medical VLMsMedical Image Segmentation

    Department of Computer Engineering, University of Kurdistan, Sanandaj, Kurdistan, Iran · Artificial Intelligence and Innovation Centre, University of Kurdistan, Erbil, Kurdistan, Iraq · Center for Machine Vision and Signal Analysis, University of Oulu, Oulu, Finland

  17. Beyond the Evidence Boundary: Behavioral Mirage Under Progressive Degradation in Medical VLMs

    Oct 20, 2026Edric Castel C. Hao, Rez Samantha Z. FlorescaMedical VLMsVLM Robustness

    Ateneo de Manila Senior High School

  18. MMedAgent-Plus: Scaling Multimodal Medical Agents across Modalities, Tasks and Workflows

    Oct 19, 2026Ruoxin Wang, Yuanting Pan, Binxu Li +13Medical AI AgentsTool-Using Vision-Language Agents

    Duke University · Stanford University · Yale University +7

  19. What Language Adds, and at What Cost: Four Controlled Probes of Pathology Vision-Language Models

    Oct 19, 2026Wenhao Zhang, Zhongliang Zhou, Yi Ping +2Medical VLMsVLM Evaluation

    School of Data Science, University of Virginia, Charlottesville, VA 22903, USA · Merck & Co., Inc., Rahway, NJ 07065, USA

  20. XGLASS: X-Ray, Gaze and Language Assisted Slice Sequencing for CT Volumes

    Oct 19, 2026Michael Glöckler, Timo RopinskiMedical Image AnalysisMedical VLMs

    Ulm University, Germany

  21. ViCoR: Estimating Visual Necessity via Counterfactual Residuals for Multimodal Medical Data Selection

    Oct 18, 2026Haotian Gan, Yudong Li, Xintian Li +2Medical VLMsVLM Robustness

    Tsinghua University · The University of Hong Kong · China Agricultural University

  22. Enhancing RAG-based Medical VQA with Gated Multimodal Representations

    Oct 16, 2026Daniela Jimenez-Pano, Hugo Jair Escalante, Delia Irazú Hernández-FaríasGated Multimodal FusionMedical VQA

    LabTL - INAOE, Puebla, Mexico · Institute for AAII, UTEP, El Paso, TX, USA · Department of Computer Science, UTEP, El Paso, TX, USA

  23. When Evidence Vanishes: Distilling Visual-Evidence Dependency in Medical Vision-Language Models

    Oct 16, 2026Qizhen Lan, Yiling Ma, Ruiqi Chen +8Medical VLMsVisual Evidence Sensitivity in VLMs

    The University of Texas Health Science Center at Houston · Yale University · University of Michigan +3

  24. MedSTP: Preserving Subspace and Transition Evidence for Medical VQA Token Pruning

    Oct 15, 2026Mengchen Fan, Qizhen Lan, Xi Xiao +6Visual Token PruningMedical VQA

    University of Alabama at Birmingham · UTHealth Houston · Northeastern University +1

  25. Multimodal Medical Image Parsing with Task-Specific Low-Rank Adapters

    Oct 14, 2026Taha Sabir, Alberto Montero Molina, Thinh Quang Ly +3Medical Image AnalysisVLM Adaptation

    Department of Computer Science, Gettysburg College, Gettysburg, PA 17325, USA · Higher School of Computing (ESCOM), Instituto Politécnico Nacional (IPN), Mexico City 07738, Mexico · Department of Computer Science and Engineering, Michigan State University, East Lansing, MI 48824, USA +2

  26. Seeing Through the Forgery Is Not Enough: Regressive Sycophancy under Fabricated Evidence in Medical Vision-Language Models

    Oct 13, 2026Sadif Ahmed, Soumya Swagata Biswas, Rifat Shahriyar +1Sycophancy in VLMsMedical VLMs

    Bangladesh University of Engineering and Technology

  27. Beyond Sparse Captions: Aligning Slide-Level Text and Patch-Level Vision in Pathology

    Oct 12, 2026Daniel Shao, Dongmin Bang, Luca L. Weishaupt +6Computational PathologyMedical VLMs

    MIT · MGB AI Institute · Harvard University +1

  28. X-Ragent: Tool-Augmented Reinforcement Learning for Evidence-Grounded Chest X-Ray Reasoning

    Oct 12, 2026Yi Wang, Zhenlong Yuan, Yuchan Guo +9Medical VLMsMedical Imaging

    Institute of Computing Technology · University of Chinese Academy of Sciences · Carnegie Mellon University +6

  29. ClinEdge: An Expert-Graded Benchmark of Real Patient Edge Cases Where Frontier Multimodal Models Fail

    Oct 11, 2026Anurag Sharma, Aditya Sharma, Sapna BaalwanMultimodal Model EvaluationMedical Image Benchmarks

    Orthopaedic Surgeon · FutrBridge · Independent Researcher

  30. Text-Guided Medical Image Segmentation via Residual Aggregation and Dense Alignment Representations

    Oct 10, 2026Jie Gui, Hang Tu, Yunfeng Dou +3Text-Guided Image SegmentationPromptable Image Segmentation

    Key Laboratory of Intelligent Computing and Signal Processing, Ministry of Education, Anhui University, Hefei, China · School of Artificial Intelligence, Anhui University, Hefei, China · School of Computer Science and Technology, Anhui University, Hefei, China +1