Medical VLMs

VLM: Vision-Language Model

Medical vision-language models study how to link images such as X-rays, CT scans, pathology slides, and surgical videos with clinical language. Common approaches include CLIP-style image-text pretraining, adapting large multimodal language models with medical data, and adding region-level grounding, segmentation, or retrieval of clinical evidence. They are evaluated on tasks such as report generation on MIMIC-CXR, visual question answering on VQA-RAD and SLAKE, and volumetric CT or whole-slide pathology analysis. A central challenge is ensuring that answers genuinely use relevant visual evidence and remain calibrated under prompt changes, rare findings, and shifts between hospitals or populations.

Latest papers

479 papers51 papers in the last four weeks, up 21% on the four weeks before. 0.3% of all new papers.

Universal CT Representations from Anatomy to Disease Phenotype through Agglomerative Pretraining
May 21, 2026Yuheng Li, Yuan Gao, Haoyu Dong +5Medical Imaging Foundation Models Medical VLMs
Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology and Emory University, Atlanta, GA 30332, USA · Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA 30322, USA · Department of Electrical and Computer Engineering, Duke University, Durham, NC 27705, USA +2
SurgOnAir: Hierarchy-Aware Real-Time Surgical Video Commentary
May 20, 2026Jingyi He, Yue Zhou, Long Bai +3Streaming Video Understanding Surgical Video Understanding
Computer Aided Medical Procedures (CAMP), TU Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · The Chinese University of Hong Kong, Hong Kong +1
VIHD: Visual Intervention-based Hallucination Detection for Medical Visual Question Answering
May 20, 2026Jiayi Chen, Benteng Ma, Zehui Liao +3Hallucination Detection in VLMs Healthcare
Department of Data Science & AI, Faculty of Information Technology, Monash University, Melbourne, VIC 3800, Australia · Hong Kong Polytechnic University, Hong Kong SAR, China · Alfred Health Radiology, Alfred Health, Melbourne, VIC 3004, Australia +1
MedExpMem: Adapting Experience Memory for Differential Diagnosis
May 20, 2026Qianhan Feng, Zhongzhen Huang, Yakun Zhu +4Radiology VLMs Healthcare
The Chinese University of Hong Kong · Shanghai Jiao Tong University · Shanghai Innovation Institue
HalluCXR: Benchmarking and Mitigating Hallucinations in Medical Vision-Language Models for Chest Radiograph Interpretation
May 19, 2026Haoyu Wang, Zitong LiVision-Language Models Audio-Language Model Hallucination Detection
Department of Biostatistics & Health Informatics, Institute of Psychiatry, Psychology & Neuroscience, King’s College London, London, UK
Rethinking Visual Attribution for Chest X-ray Reasoning in Large Vision Language Models
May 19, 2026Guangzhi Xiong, Qiao Jin, Sanchit Sinha +2Medical VQA VLM Interpretability
University of Virginia · National Institutes of Health
Concept-Guided Noisy Negative Suppression for Zero-Shot Classification and Grounding of Chest X-Ray Findings
May 19, 2026Chenyu Lian, Hong-Yu Zhou, Chun-Ka Wong +1Contrastive Learning Chest X-Ray Classification
The Center for Smart Health, School of Nursing, the Hong Kong Polytechnic University, Hong Kong, China · Research Institute for Smart Ageing, the Hong Kong Polytechnic University,2026 Hong Kong, China · School of Biomedical Engineering, Tsinghua Medicine, Tsinghua University, Beijing, China +1
MAM-CLIP: Vision-Language Pretraining on Mammography Atlases for BI-RADS Classification
May 19, 2026Halil Ibrahim Gulluk, Olivier GevaertMedical Imaging Medical Image Classification
Department of Electrical Engineering Stanford University Stanford, CA 94305 · Biomedical Informatics Research (BMIR) Stanford University Stanford, CA 94304
MedFM-Robust: Benchmarking Robustness of Medical Foundation Models
May 18, 2026Xiangxiang Cui, Tianjin Huang, Yifang Wang +2Multimodal Robustness Neural Network Robustness
Beijing Normal University, China · University of Exeter, United Kingdom · University College London, United Kingdom +2
Geometry-Aware Uncertainty Coresets for Robust Visual In-Context Learning in Histopathology
May 18, 2026Franciskus Xaverius Erick, Johanna Paula Müller, Bernhard KainzVLM Robustness In-Context Learning
FAU Erlangen-Nürnberg, Erlangen, DE · Department of Computing, Imperial College London, London, UK
Wasserstein Equilibrium Decoding for Reliable Medical Visual Question Answering
May 18, 2026Luca Hagen, Johanna P. Müller, Weitong Zhang +2Healthcare Efficient VLM Inference
Friedrich-Alexander University Erlangen-Nürnberg, Erlangen, Germany · Imperial College London, London, United Kingdom · University College London, London, United Kingdom
How Good LLMs Are at Answering Bangla Medical Visual Questions? Dataset and Benchmarking
May 18, 2026Rafid Ahmed, Intesar Tahmid, Mir Sazzat Hossain +3VLM Evaluation Multilingual Language Model Evaluation
Penta Global Limited, Bangladesh · Center for Computational & Data Sciences, Independent University, Bangladesh
Segmentation, Detection and Explanation: A Unified Framework for CT Appearance Reasoning
May 15, 2026Yuyuan Liu, Can Peng, Yingyu Yang +3Image Segmentation Medical VLMs
Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford
BiomedAP: A Vision-Informed Dual-Anchor Framework with Gated Cross-Modal Fusion for Robust Medical Vision-Language Adaptation
May 15, 2026Huanyang Tong, Kai Liu, Fangjun Kuang +1VLM Robustness Prompt Learning
Wenzhou University · Wenzhou Business College
RoiMAM: Region-of-Interest Medical Attention Model for Efficient Vision-Language Understanding
May 15, 2026Jiayan Yang, Zhuoyu Wu, Wenqi FangHealthcare Efficient VLM Inference
Shenzhen Institutes of Advanced Technology, Chinese Academy of Science · CyPhi(ΨΦ) AI Research Lab, School of IT, Monash University, Malaysia Campus
DermAgent: A Self-Reflective Agentic System for Dermatological Image Analysis with Multi-Tool Reasoning and Traceable Decision-Making
May 14, 2026Yize Liu, Siyuan Yan, Ming Hu +5Medical Image Analysis LLM Agent Orchestration
AIM for Health Lab, Faculty of Information Technology, Monash University, Melbourne, Australia · Faculty of Information Technology, Monash University, Melbourne, Australia · University College London, Institute of Ophthalmology, London, United Kingdom
CA-GCL: Cross-Anatomy Global-Local Contrastive Learning for Robust 3D Medical Image Understanding
May 13, 2026Hanwen Zhang, Yao Liu, Die Dai +4VLM Robustness Medical Image Anomaly Detection
University of Electronic Science and Technology of China, Chengdu, China · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE
Verification Mirage: Mapping the Reliability Boundary of Self-Verification in Medical VQA
May 11, 2026Ruinan Jin, Beidi Zhao, Myeongkyun Kang +2VLM Evaluation Vision-Language Models
1The University of British Columbia · 2Vector Institute · 3Redmin University of China
Med-StepBench: A Hierarchical Reasoning Framework for Evaluating Hallucinations in Medical Vision-Language Models
May 11, 2026Minh Khoi Nguyen, Dai Lam Le, Amir Reza Jafari +8Healthcare Hallucination Detection
1AI4LIFE, Hanoi University of Science and Technology, Vietnam · 2SAMOVAR, Télécom SudParis, Institut Polytechnique de Paris, France · Military Central Hospital, Vietnam +1
LiteMedCoT-VL: Parameter-Efficient Adaptation for Medical Visual Question Answering
May 10, 2026Runze Ma, Shunbo Jia, Haonan Lyu +2CoT Distillation Healthcare
School of Information Technology Monash University Malaysia · Faculty of Innovation Engineering Macau University of Science and Technology · Department of Bioelectronics Faculty of Biomedical Engineering Shenzhen University of Advanced Technology +1
Cross-Modal Semantic-Enhanced Diffusion Framework for Diabetic Retinopathy Grading
May 10, 2026Yiqun WangDiabetic Retinopathy Grading Medical VLMs
Beijing Jiaotong University · School of Software Engineering Beijing Jiaotong University Beijing, China
CapCLIP: A Vision-Language Representation Alignment Approach for Wireless Capsule Endoscopy Analysis
May 8, 2026Haroon Wahab, Irfan Mehmood, Hassan UgailEndoscopy Medical Image Analysis
School of Computer Science, AI and Electronics Faculty of Engineering and Digital Technologies University of Bradford Bradford, UK · School of Management Faculty of Mgmt, Law & Social Sciences University of Bradford Bradford, UK · Centre for Visual Computing and Intelligent Systems University of Bradford Bradford, UK
MedVIGIL: Evaluating Trustworthy Medical VLMs Under Broken Visual Evidence
May 8, 2026Hanqi Jiang, Junhao Chen, Mingyu Kang +12VLM Evaluation VLM Robustness
University of Georgia · Chungbuk National University · Chungnam National University Hospital +5
Retina-RAG: Retrieval-Augmented Vision-Language Modeling for Joint Retinal Diagnosis and Clinical Report Generation
May 7, 2026Abdelrahman Zaian, Sheethal Bhat, Mohamed Abdalkader +1Diabetic Retinopathy Grading Medical Report Generation
Friedrich-Alexander-Universität, Germany
FairEnc: A Fair Vision-Language Model with Fair Vision and Text Encoders for Glaucoma Detection
May 6, 2026Mohamed Elhabebe, Ayman El-Baz, Qing LiuAlgorithmic Fairness Algorithmic Bias
Center for Machine Vision and Signal Analysis, University of Oulu, Finland · Department of Bioengineering, University of Louisville, USA · The Department of Physics and Engineering, UiT The Arctic University of Norway, Norway
MIRAGE: Retrieval and Generation of Multimodal Images and Texts for Medical Education
May 6, 2026Miguel Diaz Benito, Cecilia Diana Albelda, Alvaro Garcia Martin +3Multimodal IR Medical Education
Universidad Autónoma de Madrid, 28049 Madrid, Spain
Scientific Domain Knowledge Improves Vision-Language Fundus Models
May 4, 2026Verena Jasmin Hallitschke, Carsten Eickhoff, Philipp BerensVision-Language Pretraining Healthcare
1Hertie Institute for AI in Brain Health, University of Tübingen, Tübingen, Germany · Institute for Applied Medical Informatics, University of Tübingen, Tübingen, Germany · 2Tübingen AI Center, Tübingen, Germany
Ultrasound Vision-Language Alignment via Contrastive Learning
May 4, 2026Zhuoyang Lyu, Yiyang Zhang, Tongxin Wang +1Cross-Modal Alignment Ultrasound Imaging
Department of Biostatistics Harvard T. H. Chan School of Public Health Boston, MA, USA
Exploring Prompt Alignment with Clinical Factors in Zero-Shot Segmentation VLMs for NSCLC Tumor Segmentation
May 2, 2026Suraj Pai, Thibault Heintz, Cosmin Ciausu +3VLM Robustness Image Segmentation
Artificial Intelligence in Medicine Program, Mass General Brigham, Harvard Medical School, USA
Echo-α: Large Agentic Multimodal Reasoning Model for Ultrasound Interpretation
Apr 30, 2026Jing Zhang, Wentao Jiang, Tao Huang +8Medical Image Analysis Multimodal Reasoning
School of Computer Science, Wuhan University, China · The Central Hospital of Wuhan, China · Taizhou Hospital of Zhejiang Province, China +1

May 21, 2026J/K move · Enter open · S save

Universal CT Representations from Anatomy to Disease Phenotype through Agglomerative Pretraining

Yuheng Li, Yuan Gao, Haoyu Dong, Yuxiang Lai+4

Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology and Emory University, Atlanta, GA 30332, USA · Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA 30322, USA · Department of Electrical and Computer Engineering, Duke University, Durham, NC 27705, USA+2

Medical Imaging Foundation Models Medical VLMs Continued Pretraining Medical Image Representation Learning Computed Tomography

Open paper AskPDF arXiv

Computed tomography (CT) is a central to three-dimensional medical imaging, yet CT-based artificial intelligence remains fragmented across task-specific models for segmentation, classification, registration, and report analysis. Here we present FlexiCT, a family of CT foundation models trained by agglomerative continual pretraining on 266,227 CT volumes from 56 publicly available datasets, forming a large-scale public resource for CT representation learning. FlexiCT uses agglomerative pretraining across three stages: two-dimensional axial pretraining, three-dimensional anatomical pretraining and report-guided semantic alignment. This training strategy supports slice-level, volume-level and vision-language analysis. Across five downstream task families (segmentation, classification, registration, vision-language understanding and clinical retrieval), FlexiCT matches or exceeds prior task-specific approaches on multiple benchmarks. Its embeddings further organize CT scans along gradients associated with various tumor stages, suggesting that CT foundation models can capture imaging features relevant to disease phenotype characterization. Project page and code are available at: https://ricklisz.github.io/flexict.github.io and https://github.com/ricklisz/FlexiCT.