Object Hallucination

Object hallucination, the tendency of large vision-language models (LVLMs) to generate descriptions containing objects not present in the input image, is a significant challenge hindering their reliability. Current research focuses on understanding the root causes of this phenomenon, exploring attention mechanisms within transformer-based architectures and investigating the roles of visual encoders and language decoders. This research aims to develop methods, both training-based and training-free, to mitigate hallucinations, improving the accuracy and trustworthiness of LVLMs for various applications, such as image captioning and visual question answering. Improved evaluation metrics are also being developed to more accurately assess and compare different approaches to reducing object hallucination.

Papers

October 30, 2024

Unified Triplet-Level Hallucination Evaluation for Large Vision-Language Models
Junjie Wu, Tsz Ting Chung, Kai Chen, Dit-Yan Yeung
Full Model Hallucination Evaluation Object Hallucination Hallucination Benchmark Large Vision Language Relation Hallucination

October 21, 2024

Mitigating Object Hallucination via Concentric Causal Attention
Yun Xing, Yiheng Li, Ivan Laptev, Shijian Lu
Large Vision Language Model Mitigating Hallucination Causal Attention Object Hallucination Hallucination Benchmark

October 9, 2024

From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models
Yuying Shang, Xinyi Zeng, Yutao Zhu, Xiao Yang, Zhengwei Fang, Jingyuan Zhang, Jiawei Chen, Zinan Liu, Yu Tian
Large Vision Language Model Tetromino Pixel K TOKEN Visual Feature Visual Input Visual Encoder Object Hallucination Hallucination Dataset

October 6, 2024

DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination
Xuan Gong, Tianshi Ming, Xinpeng Wang, Zhihua Wei
Attention Mechanism Large Vision Language Model Visual Encoder Adaptive Token Object Hallucination

October 4, 2024

Investigating and Mitigating Object Hallucinations in Pretrained Vision-Language (CLIP) Models
Yufang Liu, Tao Ji, Changzhi Sun, Yuanbin Wu, Aimin Zhou
Full Model Large Vision Language Model Single CLIP Mitigating Hallucination Object Hallucination

September 14, 2024

ODE: Open-Set Evaluation of Hallucinations in Multimodal Large Language Models
Yahan Tu, Rui Hu, Jitao Sang
Multimodal Large Language Model Content Hallucination Open Set Object Hallucination Hallucination Rate ODE Sampler

September 2, 2024

Understanding Multimodal Hallucination with Parameter-Free Representation Alignment
Yueqian Wang, Jianxin Liang, Yuxuan Wang, Huishuai Zhang, Dongyan Zhao
Neural Representation Image Representation Representation Alignment Object Hallucination Multimodal Hallucination

July 8, 2024

Multi-Object Hallucination in Vision-Language Models
Xuweiyi Chen, Ziqiao Ma, Xuejun Zhang, Sihan Xu, Shengyi Qian, Jianing Yang, David F. Fouhey, Joyce Chai
Vision Language Model Large Vision Language Model Linear Probing Visual Hallucination Object Hallucination

June 20, 2024

Does Object Grounding Really Reduce Hallucination of Large Vision-Language Models?
Gregor Geigle, Radu Timofte, Goran Glavaš
Large Vision Language Model Visual Question Answering Content Hallucination Object Grounding LLM Hallucination Object Hallucination

June 18, 2024

June 12, 2024

Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Models
Chun-Yi Kuan, Wei-Ping Huang, Hung-yi Lee
Yes No Question Challenge Task Audio Understanding Object Hallucination Auditory Perception Audio Captioning Model Audio Task

May 28, 2024

Data-augmented phrase-level alignment for mitigating object hallucination
Pritam Sarkar, Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan Ö. Arık, Tomas Pfister
Multimodal Large Language Model Mitigating Hallucination Object Hallucination Contrastive Tuning

April 22, 2024

VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models
Haoyi Qiu, Wenbo Hu, Zi-Yi Dou, Nanyun Peng
Large Language Model Large Vision Language Model Benchmark Dataset Content Hallucination Higher Coverage Rate Faithfulness Test Object Hallucination Error Vector Assisted Learning

April 3, 2024

ALOHa: A New Measure for Hallucination in Captioning Models
Suzanne Petryk, David M. Chan, Anish Kachinthaya, Haodi Zou, John Canny, Joseph E. Gonzalez, Trevor Darrell
Content Hallucination Multimodal Pre New Measure Object Hallucination

March 1, 2024

HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding
Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, Jiawei Zhou
Fine Grained Large Vision Language Model Vision Language Task Mitigating Hallucination Multimodal Context Object Hallucination Contrast Adaptation

February 23, 2024

Seeing is Believing: Mitigating Hallucination in Large Vision-Language Models via CLIP-Guided Decoding
Ailin Deng, Zhirui Chen, Bryan Hooi
Large Vision Language Model Mitigating Hallucination Plain Sight Language Model Hallucination Object Hallucination

February 18, 2024

Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Models
Junfei Wu, Qiang Liu, Ding Wang, Jinghao Zhang, Shu Wu, Liang Wang, Tieniu Tan
Large Vision Language Model Closed Loop Detection Model Hallucination Detection Object Hallucination

February 15, 2024

EFUF: Efficient Fine-grained Unlearning Framework for Mitigating Hallucinations in Multimodal Large Language Models
Shangyu Xing, Fei Zhao, Zhen Wu, Tuo An, Weihao Chen, Chunhui Li, Jianbing Zhang, Xinyu Dai
Multimodal Large Language Model Human Annotation Mitigating Hallucination Object Hallucination Alignment Algorithm

February 13, 2024

Mitigating Object Hallucination in Large Vision-Language Models via Classifier-Free Guidance
Linxi Zhao, Yihe Deng, Weitong Zhang, Quanquan Gu
Large Vision Language Model Mitigating Hallucination Vision Foundation Model Visual Context Classifier Free Guidance Object Hallucination

Object Hallucination

Papers

Unified Triplet-Level Hallucination Evaluation for Large Vision-Language Models

Mitigating Object Hallucination via Concentric Causal Attention

From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models

DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination

Investigating and Mitigating Object Hallucinations in Pretrained Vision-Language (CLIP) Models

ODE: Open-Set Evaluation of Hallucinations in Multimodal Large Language Models

Understanding Multimodal Hallucination with Parameter-Free Representation Alignment

Multi-Object Hallucination in Vision-Language Models

Does Object Grounding Really Reduce Hallucination of Large Vision-Language Models?

AGLA: Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention

Do More Details Always Introduce More Hallucinations in LVLM-based Image Captioning?

Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Models

Data-augmented phrase-level alignment for mitigating object hallucination

VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models

ALOHa: A New Measure for Hallucination in Captioning Models

HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding

Seeing is Believing: Mitigating Hallucination in Large Vision-Language Models via CLIP-Guided Decoding

Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Models

EFUF: Efficient Fine-grained Unlearning Framework for Mitigating Hallucinations in Multimodal Large Language Models

Mitigating Object Hallucination in Large Vision-Language Models via Classifier-Free Guidance