Medical VLMs

VLM: Vision-Language Model

Medical vision-language models study how to link images such as X-rays, CT scans, pathology slides, and surgical videos with clinical language. Common approaches include CLIP-style image-text pretraining, adapting large multimodal language models with medical data, and adding region-level grounding, segmentation, or retrieval of clinical evidence. They are evaluated on tasks such as report generation on MIMIC-CXR, visual question answering on VQA-RAD and SLAKE, and volumetric CT or whole-slide pathology analysis. A central challenge is ensuring that answers genuinely use relevant visual evidence and remain calibrated under prompt changes, rare findings, and shifts between hospitals or populations.

Latest papers

479 papers51 papers in the last four weeks, up 21% on the four weeks before. 0.3% of all new papers.

MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering
Apr 10, 2026Suyang Xi, Songtao Hu, Yuxiang Lai +4Medical VQA Medical VLMs
Department of Radiation Oncology and Winship Cancer Institute, Emory University School of Medicine, Atlanta, 30322, GA, USA · Department of Biostatistics and Bioinformatics, Emory University, Atlanta, 30322, GA, USA
Predictive Entropy as a Joint Screen for Error and Paraphrase Instability in Medical Vision-Language Models
Apr 10, 2026Binesh Sadanandan, Vahid BehzadanMedical VLMs VLM Evaluation
SAIL Lab, University of New Haven, West Haven, CT, USA
Towards Responsible Multimodal Medical Reasoning via Context-Aligned Vision-Language Models
Apr 9, 2026Sumra Khan, Sagar Chhabriya, Aizan Zafar +5VLM Hallucination Mitigation Medical VLMs
Department of Computer Science, Salim Habib University, Karachi, Pakistan · Computer Science, Institute of Business Administration Sukkur, Pakistan · Center for Research in Computer Vision, University of Central Florida, USA +2
Fundus-R1: Training a Fundus-Reading MLLM on Public Data
Apr 9, 2026Yuchuan Deng, Qijie Wei, Kaiheng Qian +6Medical VLMs Medical Imaging
Renmin University of China, China · The Hong Kong University of Science and Technology, Hong Kong SAR, China · Zhejiang Gongshang University, China
MedVR: Annotation-Free Medical Visual Reasoning via Agentic Reinforcement Learning
Apr 9, 2026Zheng Jiang, Heng Guo, Chengyu Fang +4VLM Reasoning Visual Reasoning
Tsinghua University · DAMO Academy, Alibaba Group · Hupan Lab, 310023, Hangzhou, China +2
Evidence-Based Actor-Verifier Reasoning for Echocardiographic Agents
Apr 7, 2026Peng Huang, Yiming Wang, Yineng Chen +9Echocardiography Medical VLMs
University at Albany, SUNY · Southwest Jiaotong University · Purdue University +3
MedLayBench-V: A Large-Scale Benchmark for Expert-Lay Semantic Alignment in Medical Vision Language Models
Apr 7, 2026Han Jang, Junhyeok Lee, Heeseong Eum +1Healthcare Medical VLMs
Seoul National University · Department of Radiology, Seoul National University Hospital · The Advanced Imaging and Computational Neuroimaging (AICON) Laboratory +2
Temporal Inversion for Learning Interval Change in Chest X-Rays
Apr 6, 2026Hanbin Ko, Kyungmin Jeon, Doowoong Choi +1Longitudinal Medical Image Analysis Medical VLMs
Interdisciplinary Program in Bioengineering, Seoul National University Graduate School · Integrated Major in Innovative Medical Science, Seoul National University Graduate School · Dept. of Radiology, Seoul National University Hospital +2
Visual Instruction-Finetuned Language Model for Versatile Brain MR Image Tasks
Apr 3, 2026Jonghun Kim, Sinyoung Ra, Hyunjin ParkMedical VLMs Multimodal Instruction Tuning
Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, Korea · Department of Artificial Intelligence, Sungkyunkwan University, Suwon, Korea
Ultrasound-CLIP: Semantic-Aware Contrastive Pre-training for Ultrasound Image-Text Understanding
Apr 2, 2026Jiayun Jin, Haolong Chai, Xueying Huang +7Medical VLMs Medical Image Representation Learning
Hangzhou City University · Hong Kong Baptist University · Zhejiang University +3
Sparse Spectral LoRA: Routed Experts for Medical VLMs
Apr 1, 2026Omid Nejati Manzari, Hojat Asgariandehkordi, Taha Koleilat +2Medical VLMs Vision-Language Models
Concordia University, Montreal, Canada
An Approach to Enriching Surgical Video Datasets for Fine-Grained Spatial-Temporal Understanding of Vision-Language Models
Apr 1, 2026Lennart Maack, Alexander SchlaeferMedical VQA Surgical Video Understanding
Hamburg University of Technology, Germany
KG-CMI: Knowledge graph enhanced cross-Mamba interaction for medical visual question answering
Apr 1, 2026Xianyao Zheng, Hong Yu, Hui Cui +7Medical VQA Medical VLMs
School of Software, Northwestern Polytechnical University, Shaanxi, China · Tianjin Central Hospital of Gynecology Obstetrics, Tianjin, China · Department of Computer Science and Information Technology, La Trobe University, Melbourne, Australia +6
VLM-in-the-Loop: A Plug-In Quality Assurance Module for ECG Digitization Pipelines
Apr 1, 2026Jiachen Li, Shihao Li, Soovadeep Bakshi +2Electrocardiography Tool Use
The University of Texas at Austin
SurgTEMP: Temporal-Aware Surgical Video Question Answering with Text-guided Visual Memory for Laparoscopic Cholecystectomy
Mar 31, 2026Shi Li, Vinkle Srivastav, Shih-Min Yin +7Medical VQA Surgical Video Understanding
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France · IHU Strasbourg, Strasbourg, France · Department of General Surgery, Kaohsiung Chang Gung Memorial Hospital, Chang Gung University College of Medicine, Kaohsiung, Taiwan +1
Calibrated Confidence Expression for Radiology Report Generation
Mar 31, 2026David Bani-Harouni, Chantal Pellegrini, Julian Lüers +6Radiology Report Generation Medical VLMs
Computer Aided Medical Procedures, Technical University of Munich, Germany · Munich Center for Machine Learning (MCML), Germany · Department of Diagnostic and Interventional Radiology, TUM Klinikum rechts der Isar, Germany +3
Prompts Without Evidence: How Neuroimaging Mentions Shift Clinical Vision-Language Model Predictions
Mar 30, 2026Doan Nam Long Vu, Simone BalloccuMedical VLMs Prompt Sensitivity
Natural Language Processing for Expert Domains (ExpNLP), Technical University of Darmstadt
Synergizing Discriminative Exemplars and Self-Refined Experience for MLLM-based In-Context Learning in Medical Diagnosis
Mar 29, 2026Wenkai Zhao, Zipei Wang, Mengjie Fang +3Multimodal ICL Medical VLMs
CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences · School of Software, North University of China
A Comparative Study in Surgical AI: Potential and Limitations of Data, Compute, and Scaling
Mar 28, 2026Kirill Skobelev, Eric Fithian, Yegor Baranovski +9Surgical Video Understanding Language Model Scaling
Center for Applied AI, Chicago Booth, Chicago, IL, USA · 2Surgical Data Science Collective, Washington D.C., USA · 3Children’s National Hospital, Washington D.C., USA +1
MEDIC-AD: Towards Medical Vision-Language Model's Clinical Intelligence
Mar 28, 2026Woohyeon Park, Jaeik Kim, Sunghwan Steve Cho +7Medical VLMs Medical Image Anomaly Detection
AIDAS Laboratory, Seoul National University · Samsung Changwon Hospital · Samsung Medical Center +1
SkinGPT-X: A Self-Evolving Collaborative Multi-Agent System for Transparent and Trustworthy Dermatological Diagnosis
Mar 27, 2026Zhangtianyi Chen, Yuhao Shen, Florensia Widjaja +6Medical Diagnosis Dermatology
School of Data Science, The Chinese University of Hong Kong, Shenzhen · Department of Dermatology, Tianjin Institute of Integrative Dermatology, Tianjin Academy of Traditional Chinese Medicine Affiliated Hospital, Tianjin 300120, China · Department of Dermatology, Beijing AnZhen Hospital, Capital Medical University, Beijing 100029, China..
Seeing Like Radiologists: Context- and Gaze-Guided Vision-Language Pretraining for Chest X-rays
Mar 27, 2026Kang Liu, Zhuoqi Ma, Siyu Liang +5Radiology VLMs Medical VLMs
Xidian University, Xi’an, China · Wuhan University, Wuhan, China
FairLLaVA: Fairness-Aware Parameter-Efficient Fine-Tuning for Large Vision-Language Assistants
Mar 27, 2026Mahesh Bhosale, Abdul Wasi, Shantam Srivastava +5Multimodal Large Language Models Medical VLMs
University at Buffalo · University of Kashmir · Accenture +1
Photon: Speedup Volume Understanding with Efficient Multimodal Large Language Models
Mar 26, 2026Chengyu Fang, Heng Guo, Zheng Jiang +3Medical VLMs 3D VLMs
Tsinghua University · DAMO Academy, Alibaba Group · Hupan Lab +1
NeuroVLM-Bench: Evaluation of Vision-Enabled Large Language Models for Clinical Reasoning in Neurological Disorders
Mar 25, 2026Katarina Trojachanec Dineva, Stefan Andonov, Ilinka Ivanoska +5Medical VLMs Multimodal Model Evaluation
Faculty of computer science and engineering, Ss. Cyril and Methodius University, Skopje, North Macedonia.
MedOpenClaw and MedFlowBench: Auditing Medical Agents in Full-Study Workflows
Mar 25, 2026Weixiang Shen, Chengzhi Shen, Yanzhu Hu +12Medical VLMs Medical Image Benchmarks
Technical University of Munich (TUM) · TUM University Hospital · LMU Munich +6
RVLM: Recursive Vision-Language Models with Adaptive Depth
Mar 25, 2026Nicanor Mayumu, Zeenath Khan, Melodena Stephens +2VLM Reasoning Medical VLMs
Department of Computer Science, University of Wollongong in Dubai, Dubai Knowledge Park, Dubai, UAE · Department of Academic Affairs, Mohammed Bin Rashid School of Government, City Walk, Building B02, Dubai, UAE
VOLMO: Versatile and Open Large Models for Ophthalmology
Mar 25, 2026Zhenyue Qin, Younjoon Chung, Elijah Lee +16Medical VLMs Medical LLMs
Department of Biomedical Informatics & Data Science, Yale University · Ray and Stephanie Lane Computational Biology Department, Carnegie Mellon University · Yong Loo Lin School of Medicine, National University of Singapore +4
MedObvious: Exposing the Medical Moravec's Paradox in VLMs via Clinical Triage
Mar 24, 2026Ufaq Khan, Umair Nawaz, L D M S S Teja +5VLM Evaluation Medical VLMs
Mohamed bin Zayed University of Artificial Intelligence, UAE · National Institute of Technology, Silchar · Birmingham City University, UK
When Models Learn to Ask Why: Adaptive Causal Reasoning for Trustworthy Medical Vision-Language Models
Mar 24, 2026Jianxin Lin, Chunzheng Zhu, Peter J. Kneuertz +2Medical VLMs Causal Reasoning in Language Models
The Ohio State University · Hunan University · Amazon

Apr 10, 2026J/K move · Enter open · S save

MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering

Suyang Xi, Songtao Hu, Yuxiang Lai, Wangyun Dan+3

Department of Radiation Oncology and Winship Cancer Institute, Emory University School of Medicine, Atlanta, 30322, GA, USA · Department of Biostatistics and Bioinformatics, Emory University, Atlanta, 30322, GA, USA

Medical VQA Medical VLMs Latent Visual Reasoning RL for VLMs

Open paper AskPDF arXiv

Medical vision--language models (VLMs) have shown strong potential for medical visual question answering (VQA), yet their reasoning remains largely text-centric: images are encoded once as static context, and subsequent inference is dominated by language. This paradigm is fundamentally limited in clinical scenarios, where accurate answers often depend on subtle, localized visual evidence that cannot be reliably preserved in static embeddings. We propose MedLVR, a latent visual reasoning framework that introduces an explicit visual evidence state into autoregressive decoding. Instead of relying solely on text-based intermediate reasoning, MedLVR interleaves a short latent reasoning segment within the decoder by reusing hidden states as continuous latent steps, enabling iterative preservation and refinement of query-relevant visual evidence before answer generation. To support effective visual supervision, we adopt a two-stage training strategy: region of interest (ROI)-supervised fine-tuning aligns latent states with clinically relevant image evidence, and Visual-Latent Policy Optimization (VLPO) further optimizes latent reasoning and answer generation under outcome-level rewards. Experiments on OmniMedVQA and five external medical VQA benchmarks show that MedLVR consistently outperforms recent reasoning baselines and improves the average score over the Qwen2.5-VL-7B backbone from 48.3% to 53.4%. These results show that latent visual reasoning provides an effective mechanism for preserving diagnostically relevant visual evidence and improving the reliability of medical VQA.