Object Hallucination in VLMs

VLM: Vision-Language Model

This topic studies when vision-language models mention objects that are absent, miss visible objects, or misread relationships in images and videos. Researchers evaluate these failures with captioning, question-answering, and grounding benchmarks such as CHAIR, POPE, and AMBER, and use controlled tests to separate weak perception from language-prior effects. Mitigation approaches include training-free contrastive decoding and attention or representation steering, as well as region-based verification, preference tuning, and reinforcement learning. A central challenge is reducing unsupported claims without suppressing correct details or harming fluency, while ensuring improvements hold across models, tasks, and evaluation methods.

Latest papers

128 papers17 papers in the last four weeks, up 89% on the four weeks before. 0.1% of all new papers.

Improving Vision-language Models with Perception-centric Process Reward Models
Apr 27, 2026Yingqian Min, Kun Zhou, Yifan Li +6Process Reward Models Object Hallucination in VLMs
Gaoling School of Artificial Intelligence, Renmin University of China. · Bytedance. · University of California, San Diego. +1
Global Context or Local Detail? Adaptive Visual Grounding for Hallucination Mitigation
Apr 27, 2026Yubo Jiang, Xin Yang, Abudukelimu Wuerkaixi +7Hallucination in Language Models Object Hallucination in VLMs
School of Astronautics, Beihang University, Beijing 102206, China · Longcat Interaction Team, Meituan, Beijing 100102, China · Tianmushan Laboratory, Beihang University, Hangzhou 311115, China
MIRAGE: A Micro-Interaction Relational Architecture for Grounded Exploration in Multi-Figure Artworks
Apr 26, 2026Jui-Cheng Chiu, Yu-Chao Wang, Shengyang Luo +4Visual Spatial Reasoning Object Hallucination in VLMs
Purdue University
R-CoV: Region-Aware Chain-of-Verification for Alleviating Object Hallucinations in LVLMs
Apr 22, 2026Jiahao Xie, Alessio Tonioni, Nathalie Rauschmayr +2Hallucination Detection in VLMs Vision-Language Models
Max Planck Institute for Informatics, SIC · VIA Research Center · Google
Breaking the Illusion: When Positive Meets Negative in Multimodal Decoding
Apr 22, 2026Yubo Jiang, Yitong An, Xin Yang +7Vision-Language Models Hallucination in Language Models
School of Astronautics, Beihang University, Beijing 102206, China · Longcat Interaction Team, Meituan, Beijing 100102, China · Tianmushan Laboratory, Beihang University, Hangzhou 311115, China
VCE: A zero-cost hallucination mitigation method of LVLMs via visual contrastive editing
Apr 21, 2026Yanbin Huang, Yisen Li, Guiyao Tie +6Large Vision-Language Models Object Hallucination in VLMs
1Huazhong University of Science and Technology · Institute of Artificial Intelligence (TeleAI), China Telecom
LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Models
Apr 20, 2026Zhiyuan Jiang, Weihao Hong, Xinlei Guan +8VLM Evaluation Hallucination Detection in VLMs
Kean University, USA · North Dakota State University, Fargo, ND, USA · McGill University, Montreal, Quebec, Canada +1
DO-Bench: An Attributable Benchmark for Diagnosing Object Hallucination in Vision-Language Models
Apr 18, 2026JiYang Wang, Jiawei Chen, Mengqi Xiao +3VLM Evaluation Hallucination Detection in VLMs
Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University · Zhongguancun Academy
Chain-of-Thought Degrades Visual Spatial Reasoning Capabilities of Multimodal LLMs
Apr 17, 2026Sai Srinivas Kancheti, Aditya Sanjiv Kanade, Vineeth N. Balasubramanian +1Multimodal Model Evaluation Visual Spatial Reasoning
IIT, Hyderabad · Microsoft Research India
See Fair, Speak Truth: Equitable Attention Improves Grounding and Reduces Hallucination in Vision-Language Alignment
Apr 10, 2026Mohammad Anas Azeez, Ankan Deria, Zohaib Hasan Siddiqui +5VLM Hallucination Mitigation Object Hallucination in VLMs
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · King Fahd University of Petroleum and Minerals, Dhahran, Saudi Arabia · Macquarie University, Sydney, Australia +2
3D-VCD: Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive Decoding
Apr 9, 2026Makanjuola Ogunleye, Eman Abdelrahman, Ismini LourentzouVLM Hallucination Mitigation Object Hallucination in VLMs
Virginia Tech · University of Illinois Urbana-Champaign
HaloProbe: Bayesian Detection and Mitigation of Object Hallucinations in Vision-Language Models
Apr 7, 2026Reihaneh Zohrabi, Hosein Hasani, Akshita Gupta +3Hallucination Detection in VLMs Object Hallucination in VLMs
Multimodal AI Lab, Technical University of Darmstadt, Darmstadt, Germany · Department of Computer Engineering, Sharif University of Technology, Tehran, Iran
Beyond the Global Scores: Fine-Grained Token Grounding as a Robust Detector of LVLM Hallucinations
Apr 6, 2026Tuan Dung Nguyen, Minh Khoi Ho, Qi Chen +8Hallucination Detection in VLMs LLM Hallucination Detection
Hanoi University of Science and Technology · Australian Institute for Machine Learning, University of Adelaide · Mohamed bin Zayed University of Artificial Intelligence +2
Focus Matters: Phase-Aware Suppression for Hallucination in Vision-Language Models
Apr 4, 2026Sohyeon Kim, Sang Yeon Yoon, Kyeongbo KongObject Hallucination in VLMs VLM Hallucination Mitigation
Pusan National University · Pukyong National University
Reinforcing Consistency in Video MLLMs with Structured Rewards
Apr 1, 2026Yihao Quan, Zeru Shi, Jinman Zhao +1Video-Language Models Object Hallucination in VLMs
Rutgers University · University of Toronto
ACT Now: Preempting LVLM Hallucinations via Adaptive Context Integration
Apr 1, 2026Bei Yan, Yuecong Min, Jie Zhang +2Attention Intervention in VLMs VLM Hallucination Mitigation
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
First Logit Boosting: Visual Grounding Method to Mitigate Object Hallucination in Large Vision-Language Models
Apr 1, 2026Jiwoo Ha, Jongwoo Baek, Jinhyun SoVLM Hallucination Mitigation Object Hallucination in VLMs
DGIST EECS
Visual Attention Drifts,but Anchors Hold:Mitigating Hallucination in Multimodal Large Language Models via Cross-Layer Visual Anchors
Mar 26, 2026Chengxu Yang, Jingling Yuan, Chuang Hu +1Object Hallucination in VLMs VLM Hallucination Mitigation
School of Computer Science and Artificial Intelligence, Wuhan University of Technology, China · School of Computer Science, Wuhan University, China
Mitigating Object Hallucinations in LVLMs via Attention Imbalance Rectification
Mar 25, 2026Han Sun, Qin Li, Peixin Wang +1Object Hallucination in VLMs VLM Hallucination Mitigation
Shanghai Key Laboratory of Trustworthy Computing, East China Normal University
Segmentation-Based Attention Entropy: Detecting and Mitigating Object Hallucinations in Large Vision-Language Models
Mar 17, 2026Jiale Song, Jiaxin Luo, Xue-song Tang +2Object Hallucination in VLMs VLM Hallucination Mitigation
School of Information and Intelligent Science, Donghua University, Shanghai, China
Do Not Leave a Gap: Hallucination-Free Object Concealment in Vision-Language Models
Mar 16, 2026Amira Guesmi, Muhammad ShafiqueAdversarial Attacks on VLMs VLM Robustness
Engineering Division, New York University Abu Dhabi, UAE
One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination
Mar 11, 2026Zhan Fa, Yue Duan, Jian Zhang +2VLM Hallucination Mitigation Object Hallucination in VLMs
National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Computer Science and Engineering, Southeast University, China
Overthinking Causes Hallucination: Tracing Confounder Propagation in Vision Language Models
Mar 8, 2026Abin Shoby, Ta Duc Huy, Tuan Dung Nguyen +6Hallucination Detection in VLMs Object Hallucination in VLMs
Australian Institute for Machine Learning, University of Adelaide · Hanoi University of Science and Technology · Mohamed bin Zayed University of Artificial Intelligence
HiPP-Prune: Hierarchical Preference-Conditioned Structured Pruning for Vision-Language Models
Mar 6, 2026Lincen Bai, Hedi Tabia, Raul Santos-RodriguezLVLM Pruning Neural Network Pruning
Université Paris-Saclay · University of Bristol
AdaIAT: Adaptively Increasing Attention to Generated Text to Alleviate Hallucinations in LVLM
Mar 5, 2026Li'an Zhong, Ziqiang He, Jibin Zheng +3Attention Intervention in VLMs VLM Hallucination Mitigation
Sun Yat-Sen University · Foshan University · University of British Columbia
HulluEdit: Single-Pass Evidence-Consistent Subspace Editing for Mitigating Hallucinations in Large Vision-Language Models
Feb 26, 2026Yangguang Lin, Quan Fang, Yufei Li +3VLM Hallucination Mitigation Object Hallucination in VLMs
Beijing University of Posts and Telecommunications, Beijing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · Beijing Jiaotong University, Beijing, China
Beyond Dominant Patches: Spatial Credit Redistribution For Grounded Vision-Language Models
Feb 25, 2026Niamul Hassan Samin, Md Arifur Rahman, Abdullah Ibne Hanif Arean +2VLM Hallucination Mitigation Object Hallucination in VLMs
The KOW Company · University of Dhaka · American International University Bangladesh (AIUB)
NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors
Feb 25, 2026Lingfeng Ren, Weihao Yu, Runpeng Yu +1Object Hallucination in VLMs VLM Hallucination Mitigation
National University of Singapore, Singapore · Peking University Shenzhen Graduate School, China
Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models
Feb 25, 2026Jianghao Yin, Qin Chen, Kedi Chen +3VLM Hallucination Mitigation Large Vision-Language Models
East China Normal University
See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs
Feb 25, 2026Yongchang Zhang, Oliver Ma, Tianyi Liu +2Multimodal Reasoning VLM Reasoning
Southeast University · AIIA, Ministry of Education, China · University of Oxford

Apr 27, 2026J/K move · Enter open · S save

Improving Vision-language Models with Perception-centric Process Reward Models

Yingqian Min, Kun Zhou, Yifan Li, Yuhuan Wu+5

Gaoling School of Artificial Intelligence, Renmin University of China. · Bytedance. · University of California, San Diego.+1

Process Reward Models Object Hallucination in VLMs VLM Reasoning Visually Grounded Reasoning RL for VLMs

Open paper AskPDF arXiv

Recent advancements in reinforcement learning with verifiable rewards (RLVR) have significantly improved the complex reasoning ability of vision-language models (VLMs). However, its outcome-level supervision is too coarse to diagnose and correct errors within the reasoning chain. To this end, we propose Perceval, a process reward model (PRM) that enables token-level error grounding, which can extract image-related claims from the response and compare them one by one with the visual evidence in the image, ultimately returning claims that contain perceptual errors. Perceval is trained with perception-intensive supervised training data. We then integrate Perceval into the RL training process to train the policy models. Specifically, compared to traditional GRPO, which applies sequence-level advantages, we apply token-level advantages by targeting penalties on hallucinated spans identified by Perceval, thus enabling fine-grained supervision signals. In addition to augmenting the training process, Perceval can also assist VLMs during the inference stage. Using Perceval, we can truncate the erroneous portions of the model's response, and then either have the model regenerate the response directly or induce the model to reflect on its previous output. This process can be repeated multiple times to achieve test-time scaling. Experiments show significant improvements on benchmarks from various domains across multiple reasoning VLMs trained with RL, highlighting the promise of perception-centric supervision as a general-purpose strategy. For test-time scaling, it also demonstrates consistent performance gains over other strategies, such as major voting. Our code and data will be publicly released at https://github.com/RUCAIBox/Perceval.