cs.CVAug 5, 2025

IKOD: Image-Attention-Guided Key-Value Merging against Object Hallucination in the Decoding of Large Vision-Language Models

Authors: Jiabing Yang, Chenhang Cui, Yixiang Chen, Tao Yu, Yan Huang, Liang Wang

Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences · NLPR, Institute of Automation, Chinese Academy of Sciences · National University of Singapore

Abstract

Large vision-language models (LVLMs) exhibit strong visual understanding and language generation capabilities yet remain prone to object hallucination, producing plausible descriptions unsupported by the input image. Analyzing generation dynamics across eight diverse LVLMs, we observe that cross-modal visual attention steadily decays as generation unfolds because appended text tokens continuously expand the softmax denominator; consequently, hallucinated objects consistently appear later and under lower image attention than grounded mentions. Unlike existing training-free remedies that rely on input perturbations or external models, we propose IKOD (Image-attention-guided Key-value merging against Object hallucination in Decoding), a collaborative decoder that directly compresses the competing text context within the native key-value (KV) cache. At each step, IKOD keeps visual tokens intact and merges low-attention text entries into a compact auxiliary cache, fusing the resulting image-focused distribution with the base branch to correct ungrounded predictions while preserving linguistic fluency. An adaptive variant (IKOD-A) further schedules collaboration strength online from the measured attention decay under a single universal configuration. Extensive experiments across hallucination and general multimodal benchmarks on eight diverse LVLMs demonstrate that IKOD and IKOD-A consistently mitigate object hallucination while preserving object recall and general multimodal capabilities at minimal inference cost. Code is available at https://github.com/jiabingyang01/IKOD.

Figures & tables

Explore similar work

Feb 25, 2026

NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors

Object hallucination is a critical issue in Large Vision-Language Models (LVLMs), where outputs include objects that do not appear in the input image. A natural question arises from this phenomenon: Which component of the LVLM pipeline primarily contributes to object hallucinations? The vision encoder to perceive visual information, or the language decoder to generate text responses? In this work, we strive to answer this question through designing a systematic experiment to analyze the roles of the vision encoder and the language decoder in hallucination generation. Our observations reveal that object hallucinations are predominantly associated with the strong priors from the language decoder. Based on this finding, we propose a simple and training-free framework, No-Language-Hallucination Decoding, NoLan, which refines the output distribution by dynamically suppressing language priors, modulated based on the output distribution difference between multimodal and text-only inputs. Experimental results demonstrate that NoLan effectively reduces object hallucinations across various LVLMs on different tasks. For instance, NoLan achieves substantial improvements on POPE, enhancing the accuracy of LLaVA-1.5 7B and Qwen-VL 7B by up to 6.45 and 7.21, respectively. The code is publicly available at: https://github.com/lingfengren/NoLan.
Aug 7, 2026

Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination

Large Vision-Language Models (LVLMs) often suffer from object hallucination, generating objects that are absent from the image. Prior work largely attributes this to insufficient visual attention. However, we find that both real and hallucinated objects receive equally strong visual attention in the model's mid-to-late layers, suggesting that the key issue may not be how much the model attends, but what it attends to and why. To this end, we decode the visual features of high-attention regions using Logit Lens, and observe that regions corresponding to real objects can be correctly decoded to the target object tokens, whereas those for hallucinated objects cannot. Building on this, we identify two hallucination mechanisms: (i) visual uncertainty, triggered by semantically similar or confusable regions; masking these regions eliminates the hallucination. (ii) contextual prior, triggered by strong co-occurrence priors; even when the initially attended region is masked, the hallucination persists and attention drifts to other regions. Based on these findings, we propose a simple yet effective training-free Detect-Mitigate framework comprising a Logit-Lens Consistency Check to detect hallucination and targeted remedies: High-Attention Regions Masking (HARM) for visual uncertainty hallucination, and Visual Evidence Enhanced Decoding (VEED) for contextual prior hallucination. Our approach achieves state-of-the-art results on multiple hallucination benchmarks. Code will be available.
Apr 1, 2026

First Logit Boosting: Visual Grounding Method to Mitigate Object Hallucination in Large Vision-Language Models

Recent Large Vision-Language Models (LVLMs) have demonstrated remarkable performance across various multimodal tasks that require understanding both visual and linguistic inputs. However, object hallucination -- the generation of nonexistent objects in answers -- remains a persistent challenge. Although several approaches such as retraining and external grounding methods have been proposed to mitigate this issue, they still suffer from high data costs or structural complexity. Training-free methods such as Contrastive Decoding (CD) are more cost-effective, avoiding additional training or external models, but still suffer from long-term decay, where visual grounding weakens and language priors dominate as the generation progresses. In this paper, we propose First Logit Boosting (FLB), a simple yet effective training-free technique designed to alleviate long-term decay in LVLMs. FLB stores the logit of the first generated token and adds it to subsequent token predictions, effectively mitigating long-term decay of visual information. We observe that FLB (1) sustains the visual information embedded in the first token throughout generation, and (2) suppresses hallucinated words through the stabilizing effect of the ``The'' token. Experimental results show that FLB significantly reduces object hallucination across various tasks, benchmarks, and backbone models. Notably, it causes negligible inference overhead, making it highly applicable to real-time multimodal systems. Code is available at https://github.com/jiwooha20/FLB