Medical Image Grounding

Ask about this topic

Overview

Medical image grounding studies how to connect clinical language—such as a finding in a report or an answer to a question—to the region, slice, or structure that provides visual evidence. This can make image-based diagnoses and reports easier to verify, and can reveal when a model gives a plausible answer without actually reading the image. Common approaches align reports with regions using weak supervision or boxes and masks, add anatomical priors and region prompts, or train vision-language models with grounding objectives, tools, and reinforcement learning. Benchmarks span chest X-rays, CT, MRI, ultrasound, and other modalities, but a central challenge remains: strong semantic answers do not guarantee precise localization, especially in 3D, across views, or under distribution shift.

Latest papers

CardsList

80 papers9 papers in the last four weeks, up 13% on the four weeks before. 0.1% of all new papers.

  1. Aligning Medical Vision Language Models at the Capability Boundary with Entropy Adaptive Rubric Grounding

    Oct 29, 2026Hongyu Bai, Jiaqi Zeng, Chunzheng ZhuMedical VLMsReasoning for VLMs

    Shanghai Jiao Tong University Shanghai, China · Hunan University Changsha, China

  2. PathCowork: Learning from real-world pathology practice to build AI copilots

    Oct 29, 2026Sheng Wang, Jialu Yao, Maria V. Rossi +22Medical AI AgentsComputational Pathology

    Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA · Department of Electrical and Systems Engineering, School of Engineering and Applied Science, University of Pennsylvania, Philadelphia, PA, USA · Department of Pathology and Laboratory Medicine, Hospital of the University of Pennsylvania, Philadelphia, PA, USA +2

  3. A Visual Grounding Benchmark for Retinal OCT with Multimodal Large Language Models

    Oct 22, 2026Ronald Fecso, José Morano, Botond Fazekas +6Medical VLMsMedical Image Grounding

    Institute of Artificial Intelligence, Center for Medical Data Science, Medical University of Vienna, Vienna, Austria · Department of Ophthalmology and Optometry, Medical University of Vienna, Vienna, Austria

  4. APED-CXR: Adaptive Privileged Evidence Distillation for Chest X-Ray Grounding with Image-Only Inference

    Oct 21, 2026Md Shihabul Islam Shovo, Nishi Kanta Paul, Jamil FayyadMedical Image GroundingAdaptive Knowledge Distillation

    NIMISHES Lab, Dhaka, Bangladesh · American University of Sharjah, Sharjah, UAE

  5. Carving Radiology Reports at Their Evidential Joints: Anatomical Region Sets as Generation Units

    Oct 21, 2026Jinwei Yang, Junhao Chen, Jianwei Zhao +3Radiology Report GenerationMedical Image Grounding

    University of Electronic Science and Technology of China

  6. PACE-MoE: Adaptive Domain Sampling and Domain-Specific LoRA for Medical Visual Grounding

    Oct 20, 2026Zhenxuan Wang, Jisheng Dang, Ronghao Lin +6Medical Image GroundingAdaptive Sampling

    School of Information Science & Engineering, Lanzhou University, Lanzhou 730000, China · College of Computer Science and Software Engineering, Shenzhen University, Shenzhen 518060, China · School of Medical Technology, Beijing Institute of Technology, Beijing 100081, China +2

  7. Text-Guided Medical Image Segmentation via Residual Aggregation and Dense Alignment Representations

    Oct 10, 2026Jie Gui, Hang Tu, Yunfeng Dou +3Text-Guided Image SegmentationPromptable Image Segmentation

    Key Laboratory of Intelligent Computing and Signal Processing, Ministry of Education, Anhui University, Hefei, China · School of Artificial Intelligence, Anhui University, Hefei, China · School of Computer Science and Technology, Anhui University, Hefei, China +1

  8. ΔΔRepresentation: Geometry Supervised Representation Learning of Phenotypes via Counterfactual Reasoning for Medical VLMs

    Oct 7, 2026Hao Wang, Qiwei Zeng, Jinghao Lin +6Medical Image Representation LearningMedical VLMs

    The University of Sydney · Shanghai Jiao Tong University · Northeastern University +1

  9. BrainTRACE: Tracing Longitudinal, Multimodal, and Volumetric Evidence in Brain MRI Clinical Reasoning

    Oct 5, 2026Qizhen Lan, Mengchen Fan, Hang Zhang +5VLM Evaluation3D Medical Imaging

    University of Texas Health Science Center at Houston · University of Alabama at Birmingham · University of Pittsburgh +3

  10. Scaling 3D Visual Grounding in Abdominal CT

    Oct 2, 2026Sam Church, Danyal Maqbool, Joshua D. Warner +4Medical Image Grounding3D Visual Grounding

    University of Wisconsin–Madison, Department of Computer Sciences, Madison, WI USA · University of Wisconsin–Madison, Department of Radiology, Madison, WI USA

  11. Learning Where to Look: Anatomical Grounding and Guided Attention for Cardiac MRI Vision-Language Models

    Sep 30, 2026Bangwei Guo, Xiao Chen, Boris Mailhe +8Medical VQAMedical Image Grounding

    Rutgers University, NJ, USA · United Imaging Intelligence, Boston, MA, USA · University of Texas Southwestern Medical Center, TX, USA +1

  12. UltraG-Bench: A Multi-task Benchmark for assessing Large Vision-Language Models on Pixel-level Evidence Grounding in Ultrasound

    Sep 25, 2026Quanhao Zhu, Bo Xu, Rui Lin +7Medical VLMsMedical Image Grounding

    Dalian University of Technology · RMIT University

  13. Anatomical Grounding and Leakage-Aware Multimodal Contrastive Learning for Alzheimer's Disease Classification from Structural MRI

    Sep 14, 2026Paul-Gabriel Nicolae, Irina Georgiana MocanuData LeakageAlzheimer's Disease Classification

    Computer Science and Engineering Department, Faculty of Automatic Control and Computers, National University of Science and Technology POLITEHNICA Bucharest, Romania

  14. Concept-Grounded Reasoning with Prompt-Driven Localization for Interpretable Structured Report Generation

    Sep 14, 2026Xinyue Xu, Hongbin Lin, Juangui Xu +6Radiology Report GenerationConcept Bottleneck Models

    The Hong Kong University of Science and Technology · The Chinese University of Hong Kong · The Hong Kong University of Science and Technology (Guangzhou) +3

  15. MetaStructAtlas: A Grounded 3D Vision-Language Dataset and Benchmark for Functional and Structural Reasoning in Whole-Body PET/CT

    Sep 3, 2026Chenguang Zheng, Le Xue, Yichi Zhang +8PET/CT ImagingMedical VLMs

    Huashan Hospital & Human Phenome Institute, Fudan University, Shanghai, China · Shanghai Academy of Artificial Intelligence for Science, Shanghai, China · Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, China +1

  16. AlphaRAD: Grounded Zero-Shot Classification in Chest Radiology via αα-Corrected Binary Cross Entropy and Factorized Latent Supervision

    Sep 1, 2026Jianzhong You, Yuan Gao, Chris McIntoshChest X-Ray ClassificationMedical Image Classification

    Peter Munk Cardiac Centre, University Health Network (UHN), Toronto, Canada · Ted Rogers Centre for Heart Research, Toronto, Canada · Vector Institute, Toronto, Canada +5

  17. Instance-Guided Report Anchoring for Text-Free 3D Abnormality Segmentation in Chest CT

    Aug 31, 2026Zhenyu Bu, Haoyan Ding, Chushu Shen +7Medical Image Segmentation3D Medical Image Segmentation

    Siemens Medical Solutions USA, Inc., Malvern, PA, USA · The Ohio State University, Columbus, OH, USA · University of California, Los Angeles, Los Angeles, CA, USA +2

  18. SlideBank: A Persistent Hierarchical Evidence Bank for Consistent Whole-Slide Reasoning

    Aug 31, 2026Beidi Zhao, Gexin Huang, Ciro Zhang +6Memory-Augmented VLMsMedical VQA

    University of British Columbia · Vector Institute · Harvard University +3

  19. CheXGround: Anatomical Region Tokens for Grounded Longitudinal Chest X-ray Interpretation

    Aug 31, 2026Adonay Demewez Gebremedhin, Wessam Shehieb, Sara Alansari +4Radiology Report GenerationRadiology VLMs

    Ajman University Ajman, United Arab Emirates · University of Birmingham Birmingham, United Kingdom · Khalifa University Abu Dhabi, United Arab Emirates +1

  20. From Reasoning to Pixels: Grounded Medical Multimodal LLMs for VQA and Segmentation

    Aug 27, 2026Haowen Gu, Gensheng Pei, Junzhu Mao +3Medical VLMsMedical Image Grounding

    Nanjing University of Science and Technology, Nanjing, China · State Key Laboratory of Intelligent Manufacturing of Advanced Construction Machinery, Nanjing, China · Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, Korea

  21. PerFact: Perception-Derived Fact Prompting for 3D Brain MRI Report Generation

    Aug 18, 2026Jianyu Sun, Zhenxuan Zhang, Guang Yang +1Radiology Report GenerationMedical VLMs

    Department of Bioengineering, Imperial College London, London, UK · Imperial-X, Imperial College London, London, UK · School of Biomedical Engineering & Imaging Sciences, King’s College London, London, UK

  22. Counterfactual Anatomy-guided Spatial-Temporal Decoding for Annotation-Free Hallucination Mitigation in Medical VLMs

    Aug 18, 2026Yifan Lu, Adinath Dukre, Abhijit Das +4VLM Hallucination MitigationMedical VLMs

    Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE

  23. MedPixel: A Unified Pixel-Language Model for Medical Reasoning and Segmentation

    Aug 10, 2026Haoyu Yang, Meixing Shi, Zengjie Chen +5Medical VLMsVLM Reasoning

    Zhejiang University · Fudan University · Kuaishou +2

  24. EliSeg: Verified Target Construction for Report-Grounded Abnormality Segmentation

    Aug 7, 2026Chengyi Peng, Haoyu Yang, Meixing Shi +2Text-Guided Image SegmentationMedical Image Segmentation

    1Zhejiang University · 2Shanghai Artificial Intelligence Laboratory

  25. Beyond Fluency: A Clinical Benchmark and Anomaly-Enhanced Baseline for Spine MRI Report Generation

    Aug 7, 2026Bruno Palau, Franziska Vogt, Daria Laslo +4VLM EvaluationRadiology Report Generation

    ETH Zurich, Switzerland · 2Swiss Institute of Bioinformatics (SIB), Switzerland

  26. LocAnyMed: Vision-Language Grounding for Multimodal Medical Images

    Aug 4, 2026Zihan Wang, Tong Liu, Zhiwei Wang +6Medical VLMsMedical Image Grounding

    School of Computer Science, Wuhan University, Wuhan, China · College of Computing and Data Science, Nanyang Technological University, Singapore · Australian Artificial Intelligence Institute, School of Computer Science, University of Technology Sydney, Sydney, Australia +1

  27. Slot2Text: Object-Centric Visual Tokenization for Efficient and Spatially Traceable Surgical MLLMs

    Aug 2, 2026Guiqiu Liao, Matjaz Jogan, Daniel A. HashimotoImage TokenizationHealthcare

    GRASP Laboratory, University of Pennsylvania · PCASO Laboratory, Department of Surgery, University of Pennsylvania · Department of Computer and Information Science, University of Pennsylvania

  28. Location-Aware Fine-Grained Representation Learning for Medical Vision Foundation Models

    Aug 2, 2026Myeongkyun Kang, Yanting Yang, Xiaoxiao LiMedical Imaging Foundation ModelsMedical Image Grounding

    The University of British Columbia · Vector Institute

  29. Semantically Calibrated Evidence Composition for CT Vision-Language Learning

    Jul 31, 2026Guoliang You, Haifan Gong, Xiaomeng ChuMedical VLMsMedical Image Grounding

    University of Pennsylvania · 2The Chinese University of Hong Kong, Shenzhen · 3Yale University

  30. RadPRISM: Schema-stratified radiology-report supervision for concept-disentangled image representations and visual grounding

    Jul 31, 2026Fabian Drexel, Marlene Fritzsche, Era Stambollxhiu +15Chest X-Ray ClassificationDisentangled Representation Learning

    Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM) and TUM University Hospital, Munich, Germany · Technical University of Munich, School of Medicine and Health, TUM University Hospital, Klinikum rechts der Isar, Department of Diagnostic and Interventional Radiology, Munich, Germany · Department of Radiology, Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, Berlin, Germany +5