Medical VLMs

VLM: Vision-Language Model

Medical vision-language models study how to link images such as X-rays, CT scans, pathology slides, and surgical videos with clinical language. Common approaches include CLIP-style image-text pretraining, adapting large multimodal language models with medical data, and adding region-level grounding, segmentation, or retrieval of clinical evidence. They are evaluated on tasks such as report generation on MIMIC-CXR, visual question answering on VQA-RAD and SLAKE, and volumetric CT or whole-slide pathology analysis. A central challenge is ensuring that answers genuinely use relevant visual evidence and remain calibrated under prompt changes, rare findings, and shifts between hospitals or populations.

Latest papers

479 papers51 papers in the last four weeks, up 21% on the four weeks before. 0.3% of all new papers.

Learning to Read Where to Look: Disease-Aware Vision-Language Pretraining for 3D CT
Mar 2, 2026Simon Ging, Philipp Arnold, Sebastian Walter +6Medical Image Retrieval Medical VLMs
Computer Vision Group, University of Freiburg, Germany · Adaptive & Agentic AI · Adaptive & Agentic AI (A3) Lab, Aarhus University, Denmark +5
Better Eyes, Better Thoughts: Why Vision Chain-of-Thought Fails in Medicine
Mar 2, 2026Yuan Wu, Zongxian Yang, Jiayu Qian +5Medical VLMs VLM Reasoning
Department of Computer Science, City University of Hong Kong (Dongguan) · College of Computing and Data Science (CCDS), Nanyang Technological University · School of Computer Science, Northwestern Polytechnical University
CARE: Towards Clinical Accountability in Multi-Modal Medical Reasoning with an Evidence-Grounded Agentic Framework
Mar 2, 2026Yuexi Du, Jinglu Wang, Shujie Liu +2Medical VLMs Medical Reasoning
Microsoft Research Asia · Department of Biomedical Engineering, Yale University · Department of Radiology & Biomedical Imaging, Yale University
When Does RL Help Medical VLMs? Disentangling Vision, SFT, and RL Gains
Mar 1, 2026Ahmadreza Jeddi, Kimia Shaban, Negin Baghbanzadeh +4Medical VLMs RL for VLMs
University of Toronto, Canada · Vector Institute, Canada · KITE Research Institute, University Health Network, Canada +1
ClinCoT: Clinical-Aware Visual Chain-of-Thought for Medical Vision Language Models
Mar 1, 2026Xiwei Liu, Yulong Li, Xinlin Zhuang +5Vision-Language Models Medical VLMs
Mohamed bin Zayed University of Artificial Intelligence · East China Normal University
GroundedSurg: A Multi-Procedure Benchmark for Language-Conditioned Surgical Tool Segmentation
Mar 1, 2026Tajamul Ashraf, Abrar Ul Riyaz, Wasif Tak +4Medical Image Grounding Referring Image Segmentation
King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Gaash Research Lab, National Institute of Technology Srinagar, India · Thapar Institute of Engineering and Technology, India +1
MedGPT-oss: Training a General-Purpose Vision-Language Model for Biomedicine
Mar 1, 2026Kai Zhang, Zhengqing Yuan, Cheng Peng +10Healthcare Multimodal Pretraining
Department of Computer Science and Engineering, Lehigh University · Department of Computer Science and Engineering, University of Notre Dame · Department of Health Outcomes & Biomedical Informatics, University of Florida +3
Act Like a Pathologist: Tissue-Aware Whole Slide Image Reasoning
Feb 28, 2026Wentao Huang, Weimin Lyu, Peiliang Lou +8Pathology VQA Medical VQA
Stony Brook University, NY, USA · Mayo Clinic, MN, USA · Harvard Medical School, MA, USA +1
LLM-Bootstrapped Targeted Finding Guidance for Factual MLLM-based Medical Report Generation
Feb 28, 2026Cunyuan Yang, Dejuan Song, Xiaotao Pang +7Medical Report Generation Radiology Report Generation
Zhejiang University · The Second Affiliated Hospital Zhejiang University School of Medicine · Hangzhou Pu Jian Medical Technology Co., Ltd.
3D Modality-Aware Pre-training for Vision-Language Model in MRI Multi-organ Abnormality Detection
Feb 27, 2026Haowen Zhu, Ning Yin, Xiaogen ZhouMedical VLMs Vision-Language Pretraining
School of Computer Science and Engineering, Southeast University, China · Department of Medical Imaging, Suzhou Traditional Chinese Medicine Hospital, China · School of Electronic, Electrical Engineering and Physics, Fujian University of Technology
MediX-R1: Open Ended Medical Reinforcement Learning
Feb 26, 2026Sahal Shaji Mullappilly, Mohammed Irfan Kurpath, Omair Mohamed +5RL for VLMs Medical VLMs
Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI) · Jubilee Mission Medical College and Research Institute · JJM Medical College
PRIMA: Pre-Training with Risk-Integrated Image--Metadata Alignment for Medical Diagnosis with LLM-Based Feature Aggregation
Feb 26, 2026Yiqing Wang, Chunming He, Ziyun Yang +5Cross-Modal Alignment Multimodal Pretraining
Department of Biomedical Engineering, Duke University 101 Science Drive, Durham, NC 27705 USA · Department of Ophthalmology and Visual Sciences, University of Michigan 1000 Wall Street, Ann Arbor, MI, USA
Cytoarchitecture in Words: Weakly Supervised Vision-Language Modeling for Human Brain Microscopy
Feb 26, 2026Matthew Sutton, Katrin Amunts, Timo Dickscheid +1Medical VLMs Vision-Language Models
Institute of Neuroscience and Medicine (INM-1), Research Centre Jülich, Jülich, Germany · Helmholtz AI, Research Centre Jülich, Jülich, Germany · C´ecile & Oskar Vogt Institute for Brain Research, University Hospital Düsseldorf, Düsseldorf, Germany +1
Modeling Expert AI Diagnostic Alignment via Immutable Inference Snapshots
Feb 26, 2026Dimitrios P. Panagoulias, Evangelia-Aikaterini Tsichrintzi, Georgios Savvidis +1Dermatology Clinical Decision Support
Department of Informatics, University of Piraeus, Karaoli ke Dimitriou 80, Piraeus, 18534, Greece · Department of Research & Development, Noetiv PC, Valaoritou 18, Athens, 10671, Greece · Department of Dermatology, Dermacen SA, Valaoritou 18, Athens, 10671, Greece
A data- and compute-efficient chest X-ray foundation model beyond aggressive scaling
Feb 26, 2026Chong Wang, Yabin Zhang, Yunhe Gao +9Medical Imaging Foundation Models Medical VLMs
Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, CA, USA. · Department of Radiology, Stanford University, Stanford, CA, USA. · Department of Computer Science, Stanford University, Stanford, CA, USA. +3
BiCLIP: Bidirectional and Consistent Language-Image Processing for Robust Medical Image Segmentation
Feb 25, 2026Saivan Talaei, Fatemeh Daneshfar, Abdulhady Abas Abdullah +1Multimodal Robustness Medical VLMs
Department of Computer Engineering, University of Kurdistan, Sanandaj, Kurdistan, Iran · Artificial Intelligence and Innovation Center, University of Kurdistan, Erbil, Kurdistan, Iraq · Center for Machine Vision and Signal Analysis, University of Oulu, Oulu, Finland
MedTri: A Platform for Structured Medical Report Normalization to Enhance Vision-Language Pretraining
Feb 25, 2026Yuetan Chu, Xinhua Ma, Xinran Jin +2Medical VLMs Vision-Language Pretraining
King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia · Faculty of Computing, Harbin, China.
Brain3D: Brain Report Automation via Inflated Vision Transformers in 3D
Feb 25, 2026Mariano Barone, Francesco Di Serio, Giuseppe Riccio +4Radiology Report Generation Medical VLMs
University of Naples Federico II, Department of Electrical Engineering and Information Technology, Italy · Northwestern University, Dept. of Computer Science, McCormick School of Engineering and Applied Science, United States · Pegaso University, Department of Information Science and Technology, Italy
MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models
Feb 25, 2026Boqi Chen, Xudong Liu, Jiachuan Peng +5Medical Diagnosis Medical VLMs
ETH Zurich · MBZUAI · Amazon +4
SigVLP: Sigmoid Volume-Language Pre-Training for Self-Supervised CT-Volume Adaptive Representation Learning
Feb 25, 2026Jiayi Wang, Hadrien Reynaud, Ibrahim Ethem Hamamci +4Medical VLMs Medical Image Representation Learning
Friedrich-Alexander University Erlangen-Nürnberg, Germany · Department of Quantitative Biomedicine, University of Zurich, Switzerland · ETH AI Center, ETH Zurich, Switzerland +2
SurGo-R1: Benchmarking and Modeling Contextual Reasoning for Operative Zone in Surgical Video
Feb 25, 2026Guanyi Qin, Xiaozhen Wang, Zhu Zhuo +7Surgical Video Understanding Medical VLMs
National University of Singapore, Singapore · Guangzhou Research Translation and Innovation Institute, National University of Singapore, China · Southern Medical University, China
Mechanistically Guided LoRA Improves Paraphrase Consistency in Medical Vision-Language Models
Feb 24, 2026Binesh Sadanandan, Vahid BehzadanMedical VLMs Medical VQA
SAIL Lab, University of New Haven, CT, USA
PSF-Med: Measuring and Explaining Paraphrase Sensitivity in Medical Vision Language Models
Feb 24, 2026Binesh Sadanandan, Vahid BehzadanMedical VLMs VLM Evaluation
SAIL Lab, University of New Haven, West Haven, CT, USA.
MUSE: Harnessing Precise and Diverse Semantics for Few-Shot Whole Slide Image Classification
Feb 24, 2026Jiahao Xu, Sheng Huang, Xin Zhang +3Computational Pathology Medical VLMs
School of Big Data & Software Engineering, Chongqing University · School of Computer Science & Technology, Chongqing University
MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation
Feb 23, 2026Taha Koleilat, Hojat Asgariandehkordi, Omid Nejati Manzari +3Medical Image Segmentation Text-Guided Image Segmentation
Concordia University, Montreal, Canada
Closing the gap in multimodal medical representation alignment
Feb 23, 2026Eleonora Grassucci, Giordano Cicchetti, Danilo ComminielloModality Gap in VLMs Vision-Language Alignment
Dept. of Information Engineering, Electronics, and Telecommunications, Sapienza University of Rome, Italy
CT-Flow: Orchestrating CT Interpretation Workflow with Model Context Protocol Servers
Feb 23, 2026Yannian Gu, Xizhuo Zhang, Linjie Mu +4Medical Image Analysis Medical VLMs
Qing Yuan Research Institute, Shanghai Jiao Tong University, Shanghai, China · Sensetime Research, Shanghai, China · Shanghai Innovation Institute, Shanghai, China
Hepato-LLaVA: An Expert MLLM with Sparse Topo-Pack Attention for Hepatocellular Pathology Analysis on Whole Slide Images
Feb 23, 2026Yuxuan Yang, Zhonghao Yan, Yi Zhang +6Pathology VQA Medical VLMs
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing 100876, China · Department of Pathology, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing 100021, China
EMAD: Evidence-Centric Grounded Multimodal Diagnosis for Alzheimer's Disease
Feb 22, 2026Qiuhui Chen, Xuancheng Yao, Zhenglei Zhou +2Medical VLMs Explainable Medical Image Analysis
East China University of Science and Technology · Shanghai Jiao Tong University · Tencent
Similarity-as-Evidence: Calibrating Overconfident VLMs for Interpretable and Label-Efficient Medical Active Learning
Feb 21, 2026Zhuofan Xie, Zishan Lin, Jinliang Lin +3Active Learning VLM Calibration
School of Electronic Technology and Engineering, Xiamen University, Xiamen, China · School of Informatics, Xiamen University, Xiamen, China · School of Engineering, Case Western Reserve University, Cleveland, USA

Mar 2, 2026J/K move · Enter open · S save

Learning to Read Where to Look: Disease-Aware Vision-Language Pretraining for 3D CT

Simon Ging, Philipp Arnold, Sebastian Walter, Hani Alnahas+5

Computer Vision Group, University of Freiburg, Germany · Adaptive & Agentic AI · Adaptive & Agentic AI (A3) Lab, Aarhus University, Denmark+5

Medical Image Retrieval Medical VLMs 3D VLMs Medical Image Localization Medical Image Classification

Open paper AskPDF arXiv

Recent 3D CT vision-language models align volumes with reports via contrastive pretraining, but typically rely on limited public data and provide only coarse global supervision. We train a 3D CT vision-language model on 98k report-volume pairs (50k patients) collected at a single hospital, combined with public datasets, using SigLIP-style contrastive pretraining together with prompt-based disease supervision in the shared vision-text embedding space. On CT-RATE, our model achieves state-of-the-art text-to-image retrieval (R@10 31.5 vs. 22.2) and competitive disease classification (AUC 83.8 vs. 83.8), with consistent results on Rad-ChestCT (AUC 77.0 vs. 77.3). We further observe that radiologists routinely reference specific images within their reports (e.g., ``series X, image Y''), linking textual descriptions to precise axial locations. We automatically mine 262k such snippet-slice pairs and introduce the task of intra-scan snippet localization -- predicting the axial depth referred to by a text snippet -- reducing mean absolute error to 36.3 mm at 12 mm feature resolution, compared with 67.0 mm for the best baseline. Adding this localization objective leaves retrieval and classification broadly unchanged within confidence bounds, yielding a single unified model for retrieval, classification, and intra-scan grounding.