cs.CVAug 25, 2026

Targeting the Attention Heads Behind Object Hallucination in LLaVA

Authors: Armaan Sandhu, Abhilasha Senapati, Hima Kammachi

Organizations: University of Massachusetts Amherst

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Vision-language models such as LLaVA-1.5-7B often hallucinate objects absent from the image when generating captions. We ask whether an interpretability diagnosis of this failure can guide a targeted fix, and we measure what that fix actually changes. We rank attention heads by how much their image attention drops around hallucinated object words, then screen the shortlist by ablating candidate heads and measuring the change in hallucination-token log probability, yielding a 32-head set. We restrict two interventions to these heads: a head-sliced LoRA adapter and an inference-time grounding controller. On 400 held-out COCO images, the combined method lowers CHAIRs (the fraction of captions with a hallucinated object) from 0.370 to 0.230 and CHAIRi (the fraction of hallucinated object mentions) from 0.156 to 0.096 (p < 0.001, paired sign-flip tests). Two controls sharpen attribution. A random-head LoRA control, matched layer-for-layer and trained identically, performs no better than the matched baseline on a separate 200-image control split, supporting the role of head selection rather than LoRA capacity. Under fixed decoding budgets, the CHAIR reduction persists and grows with budget (23% at 64 tokens to 58% at 128), arguing against a pure max-token or truncation artifact, although the method remains shorter and more conservative. The resulting behavior reduces unsupported object mentions while also lowering object recall (0.78 to 0.70). We present a diagnosis-to-intervention pipeline for object hallucination, and, more importantly, a controlled account of what acting on the diagnostic signal actually does: it localizes intervention sites with real, non-random leverage, reported as a behavioral profile rather than a single score.

Figures & tables

Figure 1: Diagnosis-to-intervention pipeline. Head selection identifies hallucination-linked heads, the LoRA stage adapts only those head slices, and the grounding controller is applied on top over the same head set.
MethodCHAIRsCHAIRiAvg. len.
Baseline0.3700 [0.3225, 0.4175]0.1558 [0.1327, 0.1800]61.1
LoRA0.2650 [0.2225, 0.3075]0.1043 [0.0857, 0.1245]27.6
Grounding only0.3100 [0.2650, 0.3575]0.1407 [0.1174, 0.1654]62.1
LoRA + Grounding (Ours)0.2300 [0.1900, 0.2700]0.0958 [0.0761, 0.1163]29.6
Table 1: Main 400-image held-out comparison with bootstrap 95% confidence intervals. Lower CHAIRs and CHAIRi are better.
MethodΔ\Delta CHAIRsΔ\Delta CHAIRi
LoRA-0.1050 [-0.1575, -0.0550], p=0.0002p=0.0002-0.0515 [-0.0738, -0.0295], p=0.0001p=0.0001
Grounding only-0.0600 [-0.1125, -0.0075], p=0.0287p=0.0287-0.0151 [-0.0386, +0.0082], p=0.2096p=0.2096
LoRA + Grounding (Ours)-0.1400 [-0.1900, -0.0900], p=0.0001p=0.0001-0.0601 [-0.0837, -0.0369], p=0.0001p=0.0001
Table 2: Paired deltas versus the baseline on the same 400 held-out images. Negative deltas are better. P-values use paired sign-flip permutation tests with 10000 resamples.
Figure 2: Hallucination reduction is accompanied by shorter captions and lower object recall. This tradeoff is central to evaluating actionability.
BudgetMethodCHAIRsCHAIRiAvg. len.
64Baseline0.2800 [0.2375, 0.3225]0.1066 [0.0863, 0.1266]49.5
64SPIN0.3025 [0.2575, 0.3475]0.1216 [0.1000, 0.1445]48.8
64VCD0.2900 [0.2475, 0.3350]0.1301 [0.1069, 0.1559]48.8
64LoRA0.22250.094028.4
64Grounding only0.26750.105349.3
64LoRA + Grounding (Ours)0.21500.090927.9
Table 3: All methods at fixed decoding budgets on the same 400 held-out images. Baseline, SPIN, and VCD include bootstrap 95% confidence intervals; our conditions report point estimates, with bootstrap intervals for these fixed-budget runs left to appendix evaluation. Lower CHAIRs and CHAIRi are better. LoRA + Grounding (Ours) outperforms both training-free baselines at every budget; the relative reduction over the matched baseline grows from 23% at 64 tokens to 58% at 128 tokens.
Figure 3: Caption-level CHAIR versus decoding budget on 400 held-out COCO images. Baseline CHAIRs rises sharply (0.28 →\to 0.37 →\to 0.51) as the budget grows, while LoRA + Grounding (Ours) stays near 0.22 at all budgets. The relative reduction grows from 23% at 64 tokens to 58% at 128, indicating the baseline becomes progressively miscalibrated to the image under larger budgets while the targeted intervention does not.
Figure 4: Adapter-scale sweep on 200 held-out images. Grounding control improves CHAIRs across LoRA strengths, but does not resolve the separate length-matching and head-selection questions.
Figure 5: Qualitative comparison on a held-out kitchen image. Green object mentions are supported by COCO annotations; red mentions are counted as hallucinated by CHAIR. The combined LoRA + grounding method suppresses the oven hallucination while retaining grounded objects, but yields a shorter, more conservative caption.
More in this topic

Object Hallucination in VLMs

Object hallucination in vision-language models is the production of object claims unsupported by visual evidence.

See all 128 papers

Explore similar work

CardsList
Jul 30, 2026

Hallucinations Leave a Grounding Signature:Verifier-Guided Decoding for Selective Object Correction

Large vision-language models (LVLMs) often hallucinate objects that are absent from an image. Despite recent progress, existing mitigation methods still lack reliable object-level grounding diagnostics and therefore tend to apply coarse-grained interventions, which can impair visual understanding, shorten responses, and reduce coverage of genuinely grounded objects. The key challenge is thus to detect, during generation, whether each emerging object mention is supported by reliable visual evidence, so that hallucination can be mitigated selectively. Yet output confidence reflects next-token plausibility rather than visual support, allowing language priors to make absent objects appear certain. We show that the missing diagnostic evidence is encoded in an Intrinsic Grounding Signature (IGS), a distributed signed attention pattern that remains informative for such confident hallucinations. Based on IGS, we propose Verifier-Guided Decoding (VGD), a decoding framework in which a lightweight verifier examines each emerging object mention, rolls back the KV cache when the mention is identified as high risk, suppresses the object and its synonyms, and regenerates the affected continuation. Because VGD intervenes only on object mentions identified as high risk, it reduces object hallucination while preserving the model's original visual understanding and grounded object coverage. Experiments on CHAIR and AMBER-G show that VGD achieves state-of-the-art object hallucination reduction: at @rec90, it cuts AMBER-G CHAIR by 43.6% while retaining 99.6% of grounded-object coverage, and reduces CHAIR-MSCOCO CHAIRi_i/CHAIRs_s by 37.0%/30.4% without shortening captions.

Lei Yang, Xinze Liu, Dayan Wu +7

Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · Department of Applied Mathematics and Statistics, Johns Hopkins University, Baltimore, MD, USA

Hallucination Detection in VLMsLLM Hallucination MitigationObject Hallucination in VLMsVision-Language GroundingVLM Hallucination Mitigation
AskPDF
May 24, 2026

Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration

The generation of factually incorrect objects, commonly known as object hallucination, remains a persistent challenge in Large Vision-Language Models (LVLMs). Current approaches to address this issue - ranging from expensive data-driven fine-tuning and high-latency contrastive decoding to rigid attention head truncation - frequently compromise either computational efficiency or the continuity of the model's feature space. To overcome these limitations, we introduce a novel, training-free inference strategy that operates as a region-aware adaptive weighting mechanism to dynamically correct semantic drift without relying on abrupt heuristic truncations. By computing an outlier-resistant statistical midpoint across various attention heads, we establish a stable anchor for reliable visual representations. We then utilize the inter-head disagreement mapped across regions to dynamically determine intervention budgets, gently suppressing hallucination-inducing attention paths through a continuous penalty modulation. This recalibration process effectively rectifies visual-semantic misalignments while fully preserving generative fluency and language priors. Comprehensive evaluations on standard multimodal benchmarks, including CHAIR, POPE, and MME, reveal that our strategy substantially curtails both instance- and sentence-level hallucinations. The results demonstrate state-of-the-art performance against contemporary baselines, confirming our method's efficiency and algorithmic robustness. Our code will be public.

Yuanzhi Xu, Qian Gao, Jun Fan +4

Qilu University of Technology (Shandong Academy of Sciences) Jinan, China · China Telecom Digital Intelligence Technology Co, Ltd Jinan, China · Shenyang Aerospace University Shenyang, China +1

VLM RobustnessVision-Language ModelsHallucination in Language ModelsObject Hallucination in VLMsVLM Hallucination MitigationAttention Intervention in VLMs+1
AskPDF
Jun 29, 2026

See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs

Large Vision-Language Models (LVLMs) excel at multimodal tasks but remain prone to object hallucinations. Prior training-free remedies often uniformly strengthen visual signals, which may also amplify irrelevant regions and introduce spurious evidence, harming fluency. We propose Context-aware Attention Intervention (CAI), a training-free inference-time mechanism that enforces a see only when needed principle via two-axis selectivity: where to look and when to intervene. At each decoding step, CAI derives token-specific visual relevance from early-layer representations to localize semantically aligned regions, and applies a conservative, entropy- and depth-gated attention tilt only for uncertainty-spiking tokens in deeper layers where visual grounding degrades, leaving confident tokens and irrelevant regions largely unchanged. This targeted intervention strengthens visual grounding while preserving linguistic fluency, and it yields consistent improvements even without contrastive decoding, which remains optional as an auxiliary bias-suppression module. Extensive experiments across multiple LVLM backbones and benchmarks show that CAI achieves state-of-the-art hallucination mitigation, and our analysis characterizes CAI as a KL-minimal attention reweighting with bounded interference under inactive gates or small tilts. Code is available at https://github.com/Iris1946/CAI.

Yuqing Lei, Wenbo Lyu, Yingjun Du +3

University of Chinese Academy of Sciences · University of Amsterdam · United Imaging Healthcare Co., Ltd.

Visual AttentionLarge Vision-Language ModelsObject Hallucination in VLMsVLM Hallucination MitigationAttention Intervention in VLMs
AskPDF