Object Hallucination

Object hallucination, the tendency of large vision-language models (LVLMs) to generate descriptions containing objects not present in the input image, is a significant challenge hindering their reliability. Current research focuses on understanding the root causes of this phenomenon, exploring attention mechanisms within transformer-based architectures and investigating the roles of visual encoders and language decoders. This research aims to develop methods, both training-based and training-free, to mitigate hallucinations, improving the accuracy and trustworthiness of LVLMs for various applications, such as image captioning and visual question answering. Improved evaluation metrics are also being developed to more accurately assess and compare different approaches to reducing object hallucination.

Papers

November 28, 2023

Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, Lidong Bing
Large Vision Language Model Contrastive Language Image Mitigating Hallucination Visual Input Contrastive Decoding Object Hallucination

October 9, 2023

Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models
Holy Lovenia, Wenliang Dai, Samuel Cahyawijaya, Ziwei Ji, Pascale Fung
Vision Language Model Visual Question Answering Visual Question Negation Detection Object Hallucination

October 3, 2023

HallE-Control: Controlling Object Hallucination in Large Multimodal Models
Bohan Zhai, Shijia Yang, Chenfeng Xu, Sheng Shen, Kurt Keutzer, Chunyuan Li, Manling Li
External Control Large Multimodal Model Multi Granularity Object Hallucination Visual Argument

October 1, 2023

Analyzing and Mitigating Object Hallucination in Large Vision-Language Models
Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, Huaxiu Yao
Large Vision Language Model Vision Language Task Mitigating Hallucination Hallucination Evaluation LLM Hallucination Object Hallucination

May 17, 2023

Evaluating Object Hallucination in Large Vision-Language Models
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, Ji-Rong Wen
Large Vision Language Model Multimodal Task Hallucination Evaluation Object Hallucination

October 14, 2022

Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training
Wenliang Dai, Zihan Liu, Ziwei Ji, Dan Su, Pascale Fung
Vision Language Hallucination Evaluation Language Model Hallucination Object Hallucination Large Scale Vision Language

December 2, 2021

Consensus Graph Representation Learning for Better Grounded Image Captioning
Wenqiao Zhang, Haochen Shi, Siliang Tang, Jun Xiao, Qiang Yu, Yueting Zhuang
Image Captioning Visual Grounding Captioning Model Object Hallucination Consensus Graph Visual Captioning Model