Medical VLMs

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

22 papers in the last four weeks, up 22% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 266

All topics

Clinical Cognition Alignment for Gastrointestinal Diagnosis with Multimodal LLMs
Mar 21, 2026Huan Zheng, Yucheng Zhou, Tianyi Yan +6Endoscopy Medical Diagnosis
SKL-IOTSC, CIS, University of Macau · 2Shanghai Jiao Tong University · 3COWARobot Co. Ltd. +1
HiPath: Hierarchical Vision-Language Alignment for Structured Pathology Report Prediction
Mar 20, 2026Ruicheng Yuan, Zhenxuan Zhang, Anbang Wang +5Computational Pathology Medical Report Generation
College of Computer Science and Electronic Engineering, Hunan University, Changsha, Hunan, China · Department of Bioengineering and Imperial-X, Imperial College London, London, UK · Department of Pathology, Xiangtan Maternal and Child Health Hospital, Xiangtan, Hunan, China +1
Medical Image Spatial Grounding with Semantic Sampling
Mar 15, 2026Andrew Seohwan Yu, Mohsen Hariri, Kunio Nakamura +3Medical VLMs Medical Image Grounding
Case Western Reserve University, Cleveland OH 44106, USA · Cleveland Clinic, Cleveland OH 44106, USA
MedPruner: Training-Free Hierarchical Token Pruning for Efficient 3D Medical Image Understanding in Vision-Language Models
Mar 12, 2026Shengyuan Liu, Zanting Ye, Yunrui Lin +6Efficient VLM Inference 3D Medical Imaging
Chinese University of Hong Kong · Westlake University · Southern Medical University +4
MedGPT-oss: Training a General-Purpose Vision-Language Model for Biomedicine
Mar 1, 2026Kai Zhang, Zhengqing Yuan, Cheng Peng +10Healthcare Multimodal Pretraining
Department of Computer Science and Engineering, Lehigh University · Department of Computer Science and Engineering, University of Notre Dame · Department of Health Outcomes & Biomedical Informatics, University of Florida +3
PRIMA: Pre-training with Risk-integrated Image-Metadata Alignment for Medical Diagnosis via LLM
Feb 26, 2026Yiqing Wang, Chunming He, Ming-Chen Lu +4Cross-Modal Alignment Multimodal Pretraining
Department of Biomedical Engineering, Duke University, Durham, NC, USA · Department of Ophthalmology and Visual Sciences, University of Michigan, Ann Arbor, MI, USA
BiCLIP: Bidirectional and Consistent Language-Image Processing for Robust Medical Image Segmentation
Feb 25, 2026Saivan Talaei, Fatemeh Daneshfar, Abdulhady Abas Abdullah +1Multimodal Robustness Medical VLMs
Department of Computer Engineering, University of Kurdistan, Sanandaj, Kurdistan, Iran · Artificial Intelligence and Innovation Center, University of Kurdistan, Erbil, Kurdistan, Iraq · Center for Machine Vision and Signal Analysis, University of Oulu, Oulu, Finland
CytoCLIP: Learning Cytoarchitectural Characteristics in Developing Human Brain Using Contrastive Language Image Pre-Training
Jan 18, 2026Pralaypati Ta, Sriram Venkatesaperumal, Keerthi Ram +1Medical VLMs Histopathology Image Classification
Sudha Gopalakrishnan Brain Centre, Indian Institute of Technology, Madras, Chennai, India. · Department of Electrical Engineering, Indian Institute of Technology, Madras, Chennai, India.
MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentation
Jan 14, 2026Yang Xing, Jiong Wu, Savas Ozdemir +4Interactive Medical Image Segmentation Unified Multimodal Models
Department of Biomedical Engineering, University of Florida, Gainesville, FL, USA · Department of Radiology, University of Florida, Jacksonville, FL, USA · Research Computing, University of Florida, Gainesville, FL, USA +2
M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding
Jan 13, 2026Juntao Jiang, Jiangning Zhang, Yali Bi +7Medical VLMs Multimodal CoT Reasoning
1Zhejiang University · University of Science and Technology of China · 3East China Normal University +2
A Tool Bottleneck Framework for Clinically-Informed and Interpretable Medical Image Understanding
Dec 24, 2025Christina Liu, Alan Q. Wang, Joy Hsu +2Tool Use in VLMs Medical Image Classification
California Institute of Technology, Pasadena, CA 91125 · Stanford University, Stanford, CA 94305
PathFLIP: Fine-grained Language-Image Pretraining for Versatile Computational Pathology
Dec 19, 2025Fengchun Liu, Songhan Jiang, Linghan Cai +2Computational Pathology Vision-Language Grounding
Harbin Institute of Technology, Shenzhen, School of Computer Science and Technology · National University of Singapore, Department of Electronic and Computer Engineering
Ophiuchus: Incentivizing Tool-augmented "Think with Images" for Joint Medical Segmentation, Understanding and Reasoning
Dec 16, 2025Yankai Jiang, Yujie Zhang, Peng Zhang +5Image Segmentation Tool Use in VLMs
Zhejiang University · Shanghai Artificial Intelligence Laboratory · Shanghai Innovation Institute +5
6 Fingers, 1 Kidney: Natural Adversarial Medical Images Reveal Critical Weaknesses of Vision-Language Models
Dec 3, 2025Leon Mayer, Piotr Kalinowski, Caroline Ebersbach +6VLM Evaluation VLM Robustness
German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Germany. · Medical Faculty, Heidelberg University, Germany. · Faculty of Mathematics and Computer Science, Heidelberg University, Germany. +9
Skin-R1: Clinical Knowledge-Guided Dermatological Diagnosis Using Vision-Language Models
Nov 18, 2025Zehao Liu, Weijieying Ren, Jipeng Zhang +4VLM Robustness Medical Diagnosis
Pennsylvania State University, University Park, PA 16802, USA · Stanford University, Stanford, CA 94305, USA · Hong Kong University of Science and Technology, Hong Kong, China
Cross-Task Generalization in Handwriting-Based Alzheimer's Screening via Vision Language Adaptation
Nov 8, 2025Changqing Gong, Huafeng Qin, Mounim A. El-YacoubiAlzheimer's Disease VLM Adaptation
Telecom SudParis, Institut Polytechnique de Paris, 91120 Palaiseau, France · School of Computer Science and Information Engineering, Chongqing Technology and Business University, Chongqing 400067, China
RAU: Reference-based Anatomical Understanding with Vision Language Models
Sep 26, 2025Yiwei Li, Yikang Liu, Jiaqi Guo +7Image Segmentation Medical VLMs
United Imaging Intelligence, Boston, MA, USA · School of Computing, University of Georgia, Athens, GA, USA · Department of Electrical and Computer Engineering, Northwestern University, Evanston, IL, USA
Bridging the Gap in Ophthalmic AI: MM-Retinal-Reason Dataset and OphthaReason Model toward Dynamic Multimodal Reasoning
Aug 22, 2025Ruiqi Wu, Yuang Yao, Na Su +11Medical Image Analysis Multimodal Reasoning
School of Computer Science and Engineering, Southeast University, Nanjing, China · Department of Ophthalmology, The First Affiliated Hospital of Nanjing Medical University, Nanjing, China · School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China
NEARL: Interacted Query Adaptation with Orthogonal Regularization for Medical Vision-Language Understanding
Aug 6, 2025Zelin Peng, Yichen Zhao, Yu Huang +5Cross-Modal Learning VLM Adaptation
MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University · Department of Radiology, The First Affiliated Hospital, School of Medicine, Zhejiang University · Mohamed bin Zayed University of Artificial Intelligence +1
Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning
Jun 8, 2025Weiwen Xu, Hou Pong Chan, Long Li +15Medical Report Generation Multimodal Pretraining
LASA Team · DAMO Academy, Alibaba Group
SurgXBench: Explainable Vision-Language Model Benchmark for Surgery
May 16, 2025Jiajun Cheng, Xianwu Zhao, Sainan Liu +5VLM Evaluation Surgical Video Understanding
Arizona State University · Intel Labs · University of California, Merced +2
Synergistic Vision-Language Reinforcement Enables Scalable On-Demand Analysis across Diverse Clinical Tasks
May 6, 2025Haonan Wang, Jiaji Mao, Lehan Wang +11Image Segmentation Medical Imaging Foundation Models
Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR · Department of Radiology, Guangdong Provincial Key Laboratory of Malignant Tumor Epigenetics and Gene Regulation, Sun Yat-Sen Memorial Hospital, Sun Yat-Sen University, Guangzhou 510120, China · Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR
Clinician-Friendly Foundation Models for Ophthalmic Image Diagnostics without Fine-Tuning or Technical Barriers
Apr 22, 2025Meng Wang, Tian Lin, Qingshan Hou +69Medical Imaging Foundation Models Medical Image Analysis
A Large-Scale Vision-Language Dataset Derived from Open Scientific Literature to Advance Biomedical Generalist AI
Mar 26, 2025Alejandro Lozano, Min Woo Sun, James Burgess +16Medical VLMs
Stanford University · Mayo Clinic
Solve the Missing First Step: Can VLMs Standardize Raw Heterogeneous Medical Data?
Date pendingXin Chen, Dongliang Xu, Cunhao Zhu +5Medical VLMs Medical Image Benchmarks
Shandong University · *Corresponding authors. · Stanford University
Slot2Text: Object-Centric Visual Tokenization for Efficient and Spatially Traceable Surgical MLLMs
Date pendingGuiqiu Liao, Matjaz Jogan, Daniel A. HashimotoImage Tokenization Healthcare
GRASP Laboratory, University of Pennsylvania · PCASO Laboratory, Department of Surgery, University of Pennsylvania · Department of Computer and Information Science, University of Pennsylvania

Mar 21, 2026, cs.CVJ/K move · Enter open · S save

Clinical Cognition Alignment for Gastrointestinal Diagnosis with Multimodal LLMs

Huan Zheng, Yucheng Zhou, Tianyi Yan, Dubing Chen+5

SKL-IOTSC, CIS, University of Macau · 2Shanghai Jiao Tong University · 3COWARobot Co. Ltd.+1

Endoscopy Medical Diagnosis Medical VLMs Visual Grounding RL for VLMs

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have demonstrated remarkable potential in medical image analysis. However, their application in gastrointestinal endoscopy is currently hindered by two critical limitations: the misalignment between general model reasoning and standardized clinical cognitive pathways, and the lack of causal association between visual features and diagnostic outcomes. In this paper, we propose a novel Clinical-Cognitive-Aligned (CogAlign) framework to address these challenges. First, we endow the model with rigorous clinical analytical capabilities by constructing the hierarchical clinical cognition dataset and employing Supervised Fine-Tuning (SFT). Unlike conventional approaches, this strategy internalizes the hierarchical diagnostic logic of experts, ranging from anatomical localization and morphological evaluation to microvascular analysis, directly into the model. Second, to eliminate visual bias, we provide a theoretical analysis demonstrating that standard supervised tuning inevitably converges to spurious background correlations. Guided by this insight, we propose a counterfactual-driven reinforcement learning strategy to enforce causal rectification. By generating counterfactual normal samples via lesion masking and optimizing through clinical-cognition-centric rewards, we constrain the model to strictly ground its diagnosis in causal lesion features. Extensive experiments demonstrate that our approach achieves State-of-the-Art (SoTA) performance across multiple benchmarks, significantly enhancing diagnostic accuracy in complex clinical scenarios.

Momentum

Related topics

Latest papers 266