VLM Hallucination

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, down 25% on the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 58

All topics

Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy
May 20, 2026Yutong Xie, Zhenglin Hua, Ran Wang +3Vision-Language Models Hallucination in Language Models
School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Artificial Intelligence, Shenzhen University, Shenzhen, China · National Engineering Laboratory for Big Data Systems Computing Technology, Shenzhen University, Shenzhen, China +3
VIHD: Visual Intervention-based Hallucination Detection for Medical Visual Question Answering
May 20, 2026Jiayi Chen, Benteng Ma, Zehui Liao +3Hallucination Detection in VLMs Healthcare
Department of Data Science & AI, Faculty of Information Technology, Monash University, Melbourne, VIC 3800, Australia · Hong Kong Polytechnic University, Hong Kong SAR, China · Alfred Health Radiology, Alfred Health, Melbourne, VIC 3004, Australia +1
HalluCXR: Benchmarking and Mitigating Hallucinations in Medical Vision-Language Models for Chest Radiograph Interpretation
May 19, 2026Haoyu Wang, Zitong LiVision-Language Models Audio-Language Model Hallucination Detection
Department of Biostatistics & Health Informatics, Institute of Psychiatry, Psychology & Neuroscience, King’s College London, London, UK
MHSA: A Lightweight Framework for Mitigating Hallucinations via Steered Attention in LVLMs
May 14, 2026Wei Ding, Yilin Li, Yudong Zhang +4Vision-Language Models LLM Hallucination Mitigation
Tsinghua University Beijing, China · Tsinghua University, Tencent Beijing, China · Tencent Beijing, China +2
Do We Really Need External Tools to Mitigate Hallucinations? SIRA: Shared-Prefix Internal Reconstruction of Attribution
May 14, 2026Tian Qin, Junzhe Chen, Yuqing Shi +3Efficient VLM Inference Hallucination in Language Models
Tsinghua University · The University of Sydney · Stanford University +1
Reducing Hallucination in Vision-Language Models via Stage-wise Preference Optimization under Distribution Shift
May 13, 2026Qinwu XuVLM Robustness Hallucination in Language Models
Meta AI
Mitigating Action-Relation Hallucinations in LVLMs via Relation-aware Visual Enhancement
May 12, 2026Zhenxin Qin, Qiang Li, Qingzhuo Wang +3VLM Robustness VLM Hallucination
Tongji University, Shanghai, China
When Looking Is Not Enough: Visual Attention Structure Reveals Hallucination in MLLMs
May 12, 2026Fanpu Cao, Xin Zou, Xuming Hu +1Hallucination Detection in VLMs Hallucination in Language Models
Thrust of Artificial Intelligence, HKUST (Guangzhou), China · Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China
Not Blind but Silenced: Rebalancing Vision and Language via Adversarial Counter-Commonsense Equilibrium
May 11, 2026Qingxin Xiao, Peilin Zhao, Yangyang Zhao +2VLM Robustness Hallucination in Language Models
South China University of Technology · Institute for Super Robotics (Huangpu) · Shanghai Jiao Tong University +1
Vocabulary Hijacking in LVLMs: Unveiling Critical Attention Heads by Excluding Inert Tokens to Mitigate Hallucination
May 11, 2026Yangneng Chen, Junlin Li, Weijun Yao +4Hallucination in Language Models VLM Hallucination
1Harbin Institute of Technology (Shenzhen), China · 2Huawei Technologies Co., Ltd. · 3The Hong Kong Polytechnic University +1
When Relations Break: Analyzing Relation Hallucination in Vision-Language Model Under Rotation and Noise
May 6, 2026Philip Wootaek Shin, Ajay Narayanan Sridhar, Sivani Devarapalli +3VLM Robustness VLM Hallucination
The Pennsylvania State University
MedScribe: Clinically Grounded CT Reporting through Agentic Workflows
May 3, 2026Giuseppe A. Orlando, Paolo Papotti, Maria A. Zuluaga +2Radiology Report Generation Healthcare
Inria Centre at Université Côte d’Azur Sophia Antipolis, France · EURECOM Biot, France · Dep. of Nuclear Medicine, Centre Antoine Lacassagne Nice, France Université Côte d’Azur, CNRS, Inserm, iBV
Query-Dependent Use of Generated Descriptions for Reliable Visual Question Answering
May 3, 2026Zeshang Li, Shuoyang ZhangVisual Question Answering Hallucination in Language Models
University of International Relations
Delineating Knowledge Boundaries for Honest Large Vision-Language Models
Apr 29, 2026Junru Song, Yimeng Hu, Yijing Chen +4VLM Robustness Vision-Language Models
Shandong University
Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models
Apr 28, 2026Chengsheng Zhang, Chenghao Sun, Xinyan Jiang +2Vision-Language Models Hallucination in Language Models
University of Science and Technology of China · Shanghai Advanced Research Institute, Chinese Academy of Sciences, Shanghai, China · University of Chinese Academy of Sciences, Beijing, China
Retrieval-Guided Generation for Safer Histopathology Image Captioning
Apr 27, 2026Md. Enamul Hoq, Wataru Uegami, Saghir Alfasly +6Multimodal RAG Computational Pathology
Kimia Lab, Department of Artificial Intelligence & Informatics, Mayo Clinic, Rochester, MN, USA · Lane Department of Computer Science and Electrical Engineering, West Virginia University, Morgantown, WV, USA · Department of Computer Science and Engineering, Princeton University, Princeton, NJ, USA +2
Aligning with Your Own Voice: Self-Corrected Preference Learning for Hallucination Mitigation in LVLMs
Apr 27, 2026Byeonggeuk Lim, JungMin Yun, Junehyoung Kwon +2Vision-Language Models Hallucination in Language Models
Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University · Department of Artificial Intelligence, Chung-Ang University
When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMs
Apr 23, 2026Pegah Khayatan, Jayneel Parekh, Arnaud Dapogny +3VLM Evaluation LLM Hallucination Mitigation
1ISIR, Sorbonne Universit´e, Paris, France · 2Valeo.ai, Paris, France
Reinforcing 3D Understanding in Point-VLMs via Geometric Reward Credit Assignment
Apr 23, 2026Jingkun Chen, Ruoshi Xu, Mingqi Gao +23D Spatial Reasoning VLM Hallucination
Northwestern Polytechnical University China · Southern University of Science and Technology China · The University of Sheffield United Kingdom +2
Mitigating Hallucinations in Large Vision-Language Models without Performance Degradation
Apr 22, 2026Xingyu Zhu, Junfeng Fang, Shuo Wang +4Vision-Language Models VLM Adaptation
MoE Key Lab of BIPC, University of Science and Technology of China · National University of Singapore
LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Models
Apr 20, 2026Zhiyuan Jiang, Weihao Hong, Xinlei Guan +8VLM Evaluation Hallucination Detection in VLMs
Kean University, USA · North Dakota State University, Fargo, ND, USA · McGill University, Montreal, Quebec, Canada +1
Mitigating Multimodal Hallucination via Phase-wise Self-reward
Apr 20, 2026Yu Zhang, Chuyang Sun, Kehai Chen +3Vision-Language Models Hallucination in Language Models
Harbin Institute of Technology · Peng Cheng Laboratory · Shenzhen, China
When Text Hijacks Vision: Benchmarking and Mitigating Text Overlay-Induced Hallucination in Vision Language Models
Apr 19, 2026Cui Yakun, Xingqun Qi, TianTian Geng +3VLM Evaluation VLM Hallucination
The Hong Kong University of Science and Technology · Hong Kong, China · University of Birmingham
A Progressive Training Strategy for Embodied Vision-Language Models to Mitigate Spatio-Temporal Hallucinations
Apr 12, 2026Xiaoda Yang, Shuai Yang, Can Wang +9VLM Adaptation VLM Hallucination
The Hong Kong University of Science and Technology · The Chinese University of Hong Kong
Overconfidence and Calibration in Medical VQA: Empirical Findings and Hallucination-Aware Mitigation
Apr 2, 2026Ji Young Byun, Young-Jin Park, Jean-Philippe Corbeil +1Vision-Language Models VLM Hallucination
Johns Hopkins University, School of Medicine · Massachusetts Institute of Technology · Microsoft Healthcare & Life Sciences
Revealing Multi-View Hallucination in Large Vision-Language Models
Mar 25, 2026Wooje Park, Insu Lee, Soohyun Kim +4Visual Question Answering Hallucination in Language Models
Seoul National University · Sungkyunkwan University
LEAD: Layer-wise Expert-aligned Decoding for Faithful Radiology Report Generation
Feb 4, 2026Ruixiao Yang, Yuanhe Tian, Di Dong +3Radiology Report Generation Language Model Decoding
Beijing Institute of Technology · Zhongguancun Academy · Zhongguancun Institute of Artificial Intelligence +1
SAVER: Mitigating Hallucinations in Large Vision-Language Models via Style-Aware Visual Early Revision
Aug 5, 2025Zhaoxu Li, Chenqi Kong, Yi Yu +6VLM Evaluation LLM Hallucination Mitigation
ROSE Lab, Interdisciplinary Graduate Programme, Nanyang Technological University, Singapore · ROSE Lab, School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · City University of Hong Kong, Hong Kong SAR +3

May 20, 2026, cs.CVJ/K move · Enter open · S save

Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy

Yutong Xie, Zhenglin Hua, Ran Wang, Wing W. Y. Ng+2

School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Artificial Intelligence, Shenzhen University, Shenzhen, China · National Engineering Laboratory for Big Data Systems Computing Technology, Shenzhen University, Shenzhen, China+3

Vision-Language Models Hallucination in Language Models VLM Hallucination Large Vision-Language Models VLM Hallucination Mitigation

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) have shown remarkable performance on a wide range of vision-language tasks. Despite this progress, they are still prone to hallucination, generating responses that are inconsistent with visual content. In this work, we find that LVLMs tend to hallucinate when they pay insufficient attention to the correct visual evidence and gradually forget it during the generation process. We empirically find that although LVLMs overall attend insufficiently to visual evidence, they exhibit sensitivity to the correct visual evidence in specific layers, with notable inter-layer discrepancy. Motivated by this observation, we propose a novel hallucination mitigation method that enhances visual evidence based on Inter-Layer Visual Attention Discrepancy (ILVAD). Specifically, we obtain the attention weights from early generated tokens to visual tokens across layers and identify the tokens that are repeatedly activated as visual evidence, forming a saliency map. We then enhance attention to visual evidence during generation through the saliency map to reduce visual forgetting. In addition, we leverage the saliency map to obtain attention scores of generated text to visual evidence, in order to select and emphasize text tokens that are strongly grounded in visual evidence. Our method is training-free and plug-and-play. Multiple benchmark evaluations conducted on five recently released models show that our method can consistently mitigate hallucinations in different LVLMs over various architectures. Code is available at https://github.com/ytx-ML/ILVAD.

Momentum

Related topics

Latest papers 58