Object Hallucination in VLMs

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, level with the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 86

All topics

Evolution of Accuracy and Visual-Cognitive Errors in a Decade of Vision-Language AI Models
Jul 10, 2026Shravan Murlidaran, Miguel P. EcksteinVLM Evaluation Image Captioning
aPsychological & Brain Sciences, University of California, Santa Barbara, , Santa Barbara, 93106, California, USA · bDepartment of Computer Science, University of California, Santa Barbara, , Santa Barbara, 93106, California, USA · cDepartment of Electrical and Computer Engineering, University of California, Santa Barbara, , Santa Barbara, 93106, California, USA
SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineering
Jul 5, 2026Kai Tang, Jinhao You, Bohua Zhang +6Large Vision-Language Models Object Hallucination in VLMs
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Nanyang Technological University · University of Pennsylvania +4
ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMs
Jun 30, 2026Zhiyuan Yao, Zheren Fu, Zhixiao Zheng +3Cross-Attention Direct Preference Optimization
University of Science and Technology of China, Hefei, China · State Key Laboratory of Communication Content Cognition, People’s Daily Online, China · Huawei Technologies Ltd., China
See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs
Jun 29, 2026Yuqing Lei, Wenbo Lyu, Yingjun Du +3Visual Attention Large Vision-Language Models
University of Chinese Academy of Sciences · University of Amsterdam · United Imaging Healthcare Co., Ltd.
FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Models
Jun 28, 2026Yichen Guo, Kai Tang, Fenglai Lin +5Vision-Language Models Large Vision-Language Models
1Nanyang Technological University · 2State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · 3Tsinghua University
HKVLM: Faithful Query--Region Binding for Frozen-Detector Visual Grounding
Jun 27, 2026Bo MaSelective Prediction Cross-Modal Alignment
Auckland University of Technology
Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decoding
Jun 25, 2026Liu Yu, Can Chen, Ping Kuang +3Object Hallucination in VLMs Faithfulness in VLMs
School of Information and Software Engineering, University of Electronic Science and Technology of China, Chengdu, Sichuan, China · School of Computer Science, University of Auckland, Auckland, New Zealand
Vision-driven Preference Synthesis for Mitigating Hallucinations in VLMs
Jun 24, 2026Yunhun Nam, Jongheon JeongSynthetic Data Generation Object Hallucination in VLMs
Korea University
Spectral Query-Key Product Weight Steering for Training-Free VLM Hallucination Mitigation
Jun 18, 2026Karn Tiwari, Varnith Chordia, Prathosh A PObject Hallucination in VLMs VLM Hallucination Mitigation
1Indian Institute of Science, Bengaluru · 2Snap Research
Look Again Before You Abstain:Budgeted Conformal Evidence Acquisition for Reliable Vision-Language Model
Jun 15, 2026Jian Xu, Delu Zeng, John Paisley +1Hallucination Detection in VLMs Active Perception
RIKEN iTHEMS · RIKEN AIP · South China University of Technology +1
Visual Para-Thinker++: A Single-Policy Multi-Agent Framework for Visual Reasoning
Jun 8, 2026Haoran Xu, Hongyu Wang, Yifei Gao +4Object Hallucination in VLMs Multi-Agent Collaboration
Zhejiang University · Hunan University · Tianjin University +3
HorusEye: Language as Dynamic Attention for Emergency Visual Analysis
Jun 2, 2026Armel YaraVLM Robustness Vision-Language Models
P $^2$ -DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimization
Jun 2, 2026Ruipeng Zhang, Zhihao Li, Haozhang Yuan +2VLM Robustness Direct Preference Optimization
Guangdong Provincial Key Laboratory of Computational AI Models and Cognitive Intelligence, School of Computer Science & Engineering, South China University of Technology · Pazhou Lab, Guangzhou, China · Engineering Research Center of the Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human, Guangzhou, China
Mitigating Manifold Departure: Uncertainty-Aware Subspace Rectification for Trustworthy MLLM Decoding
May 31, 2026Yingxuan Zhuang, Jingxiao Yang, Miao Pan +7Object Hallucination in VLMs VLM Hallucination Mitigation
Ningbo Global Innovation Center, Zhejiang University
YARD: Y-Architecture Register Decoding for Efficient Hallucination Mitigation in Large Vision-Language Models
May 29, 2026Ting Chen, Geng Li, Guohao Chen +5Efficient VLM Inference Object Hallucination in VLMs
Guangdong University of Technology · Nanyang Technological University · Shenzhen TENCLASS Technology Co., Ltd.
CFCamo: A Counterfactual Detect-or-Abstain Framework for Camouflaged Object Detection
May 28, 2026Suhang Li, Osamu Yoshie, Yuya IeiriSelective Prediction Object Hallucination in VLMs
The authors are with the Graduate School of Information, Production and Systems, Waseda University, Fukuoka, Japan.
Mitigating Hallucination in Vision-Language Models through Barrier-Regulated Adaptive Closed-form Steering
May 28, 2026Soumyadeep Jana, Pulkit Mittal, Sanasam Ranbir SinghHallucination in Language Models Large Vision-Language Models
Indian Institute of Technology Guwahati, India
VCap: Hypergeometric Rewards for Weak-to-Strong Visual Captioning
May 27, 2026Xingyu Lu, Jinpeng Wang, Yi-Fan Zhang +13Image Captioning Object Hallucination in VLMs
1Tsinghua Shenzhen International Graduate School · 2Harbin Institute of Technology, Shenzhen · 3Chinese Academy of Sciences +1
Rethinking Visual Neglect: Steering via Context-Preference for MLLM Hallucination Mitigation
May 27, 2026Jingwen Wu, Xijun Zhang, Ge SongLLM Hallucination Mitigation Hallucination in Language Models
School of Computer and Electronic Information, Nanjing Normal University, China
Advancing Creative Physical Intelligence in Large Multimodal Models
May 25, 2026Cheng Qian, Hyeonjeong Ha, Jiayu Liu +10Multimodal Grounding Object Hallucination in VLMs
1UIUC · 2Amazon
Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation
May 24, 2026Yangneng Chen, Jing LiVision-Language Models VLM Adaptation
1Harbin Institute of Technology, Shenzhen, China.
Universal Boosts, Specific Suppressors: Sparse Autoencoder Steering of Medical Vision-Language Models
May 24, 2026Farhad Nooralahzadeh, Benjamin Gundersen, Nicolas Deperrois +7Radiology Report Generation Healthcare
University of Zurich and University Hospital of Zurich, Switzerland · Zurich University of Applied Sciences, Switzerland · Kobe University, Japan +3
Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration
May 24, 2026Yuanzhi Xu, Qian Gao, Jun Fan +4VLM Robustness Vision-Language Models
Qilu University of Technology (Shandong Academy of Sciences) Jinan, China · China Telecom Digital Intelligence Technology Co, Ltd Jinan, China · Shenyang Aerospace University Shenyang, China +1
Correcting Visual Blur Induced by Attention Distraction to Reduce Hallucinations: Algorithm and Theory
May 23, 2026Quanjiang Li, Zhiming Liu, Wei Luo +2LLM Hallucination Mitigation Hallucination in Language Models
National University of Defense Technology, China · Harbin Institute of Technology (Shenzhen), China · Xi’an Jiaotong University, China.
CHASD: Language Increment-Calibrated Contrastive Decoding against Hallucination in LVLMs
May 22, 2026Xiaoyi Huang, Kejia Zhang, Zhiming LuoHallucination Detection in VLMs Efficient VLM Inference
Institute of Artificial Intelligence, Xiamen University · Department of Artificial Intelligence, Xiamen University
Transcoders Trace Visual Grounding and Hallucinations in Vision-Language Models
May 21, 2026Dimitrios Damianos, Leon Voukoutis, Georgios Skyrianos +2Object Hallucination in VLMs VLM Interpretability
Institute of Language and Speech Processing, Athena Research Center Athens, Greece
Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens
May 20, 2026Meng Shen, Minghao Wu, Deepu RajanVision-Language Models Hallucination in Language Models
Nanyang Technological University · Monash University
Pseudocode-Guided Structured Reasoning for Automating Reliable Inference in Vision-Language Models
May 19, 2026Weicong Ni, Tianbao Jiang, Linlin WangVLM Robustness Object Hallucination in VLMs
All authors are with East China Normal University, Shanghai, China.
ClaimDiff-RL: Fine-Grained Caption Reinforcement Learning through Visual Claim Comparison
May 19, 2026Tianle Li, Xuyang Shen, Yan Ma +7VLM Evaluation Reward Modeling
1The Chinese University of Hong Kong · 2MiniMax
Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models
May 18, 2026Xinpeng Dong, Min Zhang, Kairong Han +3Object Hallucination in VLMs Multimodal Large Language Models
Zhejiang University · East China Normal University · Zhejiang University of Science and Technology

Jul 10, 2026, cs.CVJ/K move · Enter open · S save

Evolution of Accuracy and Visual-Cognitive Errors in a Decade of Vision-Language AI Models

Shravan Murlidaran, Miguel P. Eckstein

aPsychological & Brain Sciences, University of California, Santa Barbara, , Santa Barbara, 93106, California, USA · bDepartment of Computer Science, University of California, Santa Barbara, , Santa Barbara, 93106, California, USA · cDepartment of Electrical and Computer Engineering, University of California, Santa Barbara, , Santa Barbara, 93106, California, USA

VLM Evaluation Image Captioning Object Hallucination in VLMs

Open paper AskPDF arXiv

Vision language models (VLMs) have made remarkable progress in visual reasoning during the last decade. Most evaluations have used simple scenes (MS-COCO) that do not showcase complex human interactions or behaviors, only a handful of non-curated human descriptions as a benchmark, and have not focused on understanding the model's error types. Here, we introduce the Complex Social Behavior (CSB) dataset, containing 100 images depicting complex social interactions/behaviors. We analyze the progression of scene descriptions over a decade (2017-2025) of VLMs (four pre-Multimodal Large Language Models, MLLMs, and five MLLMs). We evaluate the accuracy of the models and 20 human descriptions relative to a gold standard on the CSB dataset and on a sample from MS-COCO. We analyzed five visual-cognitive error types: object detection, recognition, hallucination, scene understanding, and spatial dependence. The CSB dataset showed a more pronounced improvement than MS-COCO in scene description accuracy, with pre-MLLMs achieving much lower accuracy than the bottom-ranked human descriptions and MLLMs attaining accuracies similar to the top-ranked human descriptions. We show that MLLMs have eliminated the gap in scene description accuracy between simpler MS-COCO scenes and scenes depicting complex behaviors (CSB). MLLMs have almost eliminated all error types in our tested datasets, except for occasionally relying on different image regions for scene descriptions than humans do (spatial dependence error). We also show that detection, recognition, and hallucination errors have the highest impact on scene description accuracy. Together, our findings provide a more thorough evaluation of how visual language models have advanced over the last decade.

Momentum

Related topics

Latest papers 86