Medical VLMs

VLM: Vision-Language Model

Medical vision-language models study how to link images such as X-rays, CT scans, pathology slides, and surgical videos with clinical language. Common approaches include CLIP-style image-text pretraining, adapting large multimodal language models with medical data, and adding region-level grounding, segmentation, or retrieval of clinical evidence. They are evaluated on tasks such as report generation on MIMIC-CXR, visual question answering on VQA-RAD and SLAKE, and volumetric CT or whole-slide pathology analysis. A central challenge is ensuring that answers genuinely use relevant visual evidence and remain calibrated under prompt changes, rare findings, and shifts between hospitals or populations.

Latest papers

479 papers51 papers in the last four weeks, up 21% on the four weeks before. 0.3% of all new papers.

3DMedAgent: Unified Perception-to-Understanding for 3D Medical Analysis
Feb 20, 2026Ziyue Wang, Linghan Cai, Chang Han Low +83D Medical Imaging Medical VLMs
National University of Singapore · Microsoft Research · TUD Dresden University of Technology +2
LATA: Laplacian-Assisted Transductive Adaptation for Conformal Uncertainty in Medical VLMs
Feb 19, 2026Behzad Bozorgtabar, Dwarikanath Mahapatra, Sudipta Roy +3Conformal Prediction VLM Calibration
Aarhus University (A3 Lab) · Khalifa University · Jio Institute +2
Automated Histopathology Report Generation via Pyramidal Feature Extraction and the UNI Foundation Model
Feb 18, 2026Ahmet Halici, Ece Tugba Cebeci, Musa Balci +2Medical Report Generation Medical VLMs
ViseurAI
OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysis
Feb 18, 2026Tianwei Lin, Zhongwei Qiu, Wenqiao Zhang +12Radiology VLMs Medical VLMs
Zhejiang University · DAMO Academy, Alibaba Group · Hupan Lab
CT-Bench: A Benchmark for Multimodal Lesion Understanding in Computed Tomography
Feb 16, 2026Qingqing Zhu, Qiao Jin, Tejas S. Mathai +10Medical Image Benchmarks Computed Tomography
Uncertainty-Aware Vision-Language Segmentation for Medical Imaging
Feb 16, 2026Aryan Das, Tanishq Rachamalla, Koushik Biswas +2Cross-Modality Medical Image Segmentation Medical VLMs
VIT Bhopal · SAHE, Andhra Pradesh · IIIT Delhi +2
CytoCLIP: Learning Cytoarchitectural Characteristics in Developing Human Brain Using Contrastive Language Image Pre-Training
Jan 18, 2026Pralaypati Ta, Sriram Venkatesaperumal, Keerthi Ram +1Medical VLMs Histopathology Image Classification
Sudha Gopalakrishnan Brain Centre, Indian Institute of Technology, Madras, Chennai, India. · Department of Electrical Engineering, Indian Institute of Technology, Madras, Chennai, India.
MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentation
Jan 14, 2026Yang Xing, Jiong Wu, Savas Ozdemir +4Interactive Medical Image Segmentation Unified Multimodal Models
Department of Biomedical Engineering, University of Florida, Gainesville, FL, USA · Department of Radiology, University of Florida, Jacksonville, FL, USA · Research Computing, University of Florida, Gainesville, FL, USA +2
M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding
Jan 13, 2026Juntao Jiang, Jiangning Zhang, Yali Bi +7Medical VLMs Multimodal CoT Reasoning
1Zhejiang University · University of Science and Technology of China · 3East China Normal University +2
HERO: Hierarchical Evidential Reasoning Optimization for Radiology Report Generation via Reason-then-Summarize
Jan 6, 2026Kun Zhao, Guodong Liu, Hui Ji +7Radiology Report Generation Medical VLMs
University of Pittsburgh · Eli and Lilly Company · The University of Manchester +1
MedGemma vs GPT-4: Open-Source and Proprietary Zero-shot Medical Disease Classification from Images
Dec 29, 2025Md. Sazzadul Islam Prottasha, Nabil Walid RafiMedical Image Classification Medical VLMs
Department of Information and Communication Technology, Bangladesh University of Professionals, Dhaka, Bangladesh
A Tool Bottleneck Framework for Clinically-Informed and Interpretable Medical Image Understanding
Dec 24, 2025Christina Liu, Alan Q. Wang, Joy Hsu +2Tool Use in VLMs Medical Image Classification
California Institute of Technology, Pasadena, CA 91125 · Stanford University, Stanford, CA 94305
PathFLIP: Fine-grained Language-Image Pretraining for Versatile Computational Pathology
Dec 19, 2025Fengchun Liu, Songhan Jiang, Linghan Cai +2Computational Pathology Vision-Language Grounding
Harbin Institute of Technology, Shenzhen, School of Computer Science and Technology · National University of Singapore, Department of Electronic and Computer Engineering
Ophiuchus: Incentivizing Tool-augmented "Think with Images" for Joint Medical Segmentation, Understanding and Reasoning
Dec 16, 2025Yankai Jiang, Yujie Zhang, Peng Zhang +5Image Segmentation Tool Use in VLMs
Zhejiang University · Shanghai Artificial Intelligence Laboratory · Shanghai Innovation Institute +5
MedInsightBench: Evaluating Medical Analytics Agents Through Multi-Step Insight Discovery in Multimodal Medical Data
Dec 15, 2025Zhenghao Zhu, Chuxue Cao, Sirui Han +4Data Analysis Agents Multimodal Model Evaluation
The Hong Kong University of Science and Technology, Hong Kong, China · ByteDance, China
6 Fingers, 1 Kidney: Natural Adversarial Medical Images Reveal Critical Weaknesses of Vision-Language Models
Dec 3, 2025Leon Mayer, Piotr Kalinowski, Caroline Ebersbach +6VLM Evaluation VLM Robustness
German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Germany. · Medical Faculty, Heidelberg University, Germany. · Faculty of Mathematics and Computer Science, Heidelberg University, Germany. +9
Skin-R1: Clinical Knowledge-Guided Dermatological Diagnosis Using Vision-Language Models
Nov 18, 2025Zehao Liu, Weijieying Ren, Jipeng Zhang +4VLM Robustness Medical Diagnosis
Pennsylvania State University, University Park, PA 16802, USA · Stanford University, Stanford, CA 94305, USA · Hong Kong University of Science and Technology, Hong Kong, China
From Classification to Cross-Modal Understanding: Leveraging Vision-Language Models for Fine-Grained Renal Pathology
Nov 15, 2025Zhenhao Guo, Rachit Saluja, Tianyu Shi +13Medical VLMs VLM Adaptation
New York University, New York, NY 10012, USA · Cornell Tech, New York, NY 10044, USA · Vanderbilt University, Nashville, TN, 37235, USA +4
Cross-Task Generalization in Handwriting-Based Alzheimer's Screening via Vision Language Adaptation
Nov 8, 2025Changqing Gong, Huafeng Qin, Mounim A. El-YacoubiAlzheimer's Disease VLM Adaptation
Telecom SudParis, Institut Polytechnique de Paris, 91120 Palaiseau, France · School of Computer Science and Information Engineering, Chongqing Technology and Business University, Chongqing 400067, China
Foundation Models in Medical Image Analysis: A Systematic Review and Meta-Analysis
Oct 19, 2025Praveenbalaji Rajendran, Mojtaba Safari, Wenfeng He +4Medical Imaging Foundation Models Vision Foundation Models
Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA 30322 · Department of Computer Science and Informatics, Emory University, Atlanta, GA 30322
BiomedXPro: Prompt Optimization for Explainable Diagnosis with Biomedical Vision Language Models
Oct 17, 2025Kaushitha Silva, Mansitha Eashwara, Sanduni Ubayasiri +2Medical VLMs Explainable Medical Image Analysis
University of Peradeniya Peradeniya, Sri Lanka · RMIT University Melbourne, Australia
RAU: Reference-based Anatomical Understanding with Vision Language Models
Sep 26, 2025Yiwei Li, Yikang Liu, Jiaqi Guo +7Image Segmentation Medical VLMs
United Imaging Intelligence, Boston, MA, USA · School of Computing, University of Georgia, Athens, GA, USA · Department of Electrical and Computer Engineering, Northwestern University, Evanston, IL, USA
Bridging the Gap in Ophthalmic AI: MM-Retinal-Reason Dataset and OphthaReason Model toward Dynamic Multimodal Reasoning
Aug 22, 2025Ruiqi Wu, Yuang Yao, Na Su +11RL for VLMs Medical Image Analysis
School of Computer Science and Engineering, Southeast University, Nanjing, China · Department of Ophthalmology, The First Affiliated Hospital of Nanjing Medical University, Nanjing, China · School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China
NEARL: Interacted Query Adaptation with Orthogonal Regularization for Medical Vision-Language Understanding
Aug 6, 2025Zelin Peng, Yichen Zhao, Yu Huang +5Cross-Modal Learning VLM Adaptation
MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University · Department of Radiology, The First Affiliated Hospital, School of Medicine, Zhejiang University · Mohamed bin Zayed University of Artificial Intelligence +1
Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning
Jun 8, 2025Weiwen Xu, Hou Pong Chan, Long Li +15Medical Report Generation Multimodal Pretraining
LASA Team · DAMO Academy, Alibaba Group
SurgXBench: Explainable Vision-Language Model Benchmark for Surgery
May 16, 2025Jiajun Cheng, Xianwu Zhao, Sainan Liu +5VLM Evaluation Surgical Video Understanding
Arizona State University · Intel Labs · University of California, Merced +2
Synergistic Vision-Language Reinforcement Enables Scalable On-Demand Analysis across Diverse Clinical Tasks
May 6, 2025Haonan Wang, Jiaji Mao, Lehan Wang +11Image Segmentation Medical Imaging Foundation Models
Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR · Department of Radiology, Guangdong Provincial Key Laboratory of Malignant Tumor Epigenetics and Gene Regulation, Sun Yat-Sen Memorial Hospital, Sun Yat-Sen University, Guangzhou 510120, China · Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR
Clinician-Friendly Foundation Models for Ophthalmic Image Diagnostics without Fine-Tuning or Technical Barriers
Apr 22, 2025Meng Wang, Tian Lin, Qingshan Hou +69Medical Imaging Foundation Models Ophthalmic Image Analysis
A Large-Scale Vision-Language Dataset Derived from Open Scientific Literature to Advance Biomedical Generalist AI
Mar 26, 2025Alejandro Lozano, Min Woo Sun, James Burgess +16Medical VLMs
Stanford University · Mayo Clinic

Feb 20, 2026J/K move · Enter open · S save

3DMedAgent: Unified Perception-to-Understanding for 3D Medical Analysis

Ziyue Wang, Linghan Cai, Chang Han Low, Haofeng Liu+7

National University of Singapore · Microsoft Research · TUD Dresden University of Technology+2

3D Medical Imaging Medical VLMs 3D VQA Medical VQA LLM Agents

Open paper AskPDF arXiv

3D CT analysis spans a continuum from low-level perception to high-level clinical understanding. Existing 3D-oriented analysis methods adopt either isolated task-specific modeling or task-agnostic end-to-end paradigms to produce one-hop outputs, impeding the systematic accumulation of perceptual evidence for downstream reasoning. In parallel, recent multimodal large language models (MLLMs) exhibit improved visual perception and can integrate visual and textual information effectively, yet their predominantly 2D-oriented designs fundamentally limit their ability to perceive and analyze volumetric medical data. To bridge this gap, we propose 3DMedAgent, a unified agent that enables 2D MLLMs to perform general 3D CT analysis without 3D-specific fine-tuning. 3DMedAgent coordinates heterogeneous visual and textual tools through a flexible MLLM agent, progressively decomposing complex 3D analysis into tractable subtasks that transition from global to regional views, from 3D volumes to informative 2D slices, and from visual evidence to structured textual representations. Central to this design, 3DMedAgent maintains a long-term structured memory that aggregates intermediate tool outputs and supports query-adaptive, evidence-driven multi-step reasoning. We further introduce the DeepChestVQA benchmark for evaluating unified perception-to-understanding capabilities in 3D thoracic imaging. Experiments across over 40 tasks demonstrate that 3DMedAgent consistently outperforms general, medical, and 3D-specific MLLMs, highlighting a scalable path toward general-purpose 3D clinical assistants.Code and data are available at https://github.com/jinlab-imvr/3DMedAgent.