Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMs

Guangtao Lyu; Qi Liu; Chenghao Xu; Jiexi Yan; Muli Yang; Xueting Li; Fen Fang; Cheng Deng

cs.CVFeb 17, 2026

Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMs

Authors: Guangtao Lyu, Qi Liu, Chenghao Xu, Jiexi Yan, Muli Yang, Xueting Li, Fen Fang, Cheng Deng

Organizations: School of Electronic Engineering, Xidian University, Xi’an, China · College of Computer and Information, Hohai University, Nanjing, China · School of Computer Science and Technology, Xidian University, Xi’an, China · Institute for Infocomm Research, A*STAR, Singapore

Abstract

LVLMs have achieved strong multimodal reasoning capabilities but remain prone to hallucinations, producing outputs inconsistent with visual inputs or user instructions. Existing training-free methods, including contrastive decoding and auxiliary expert models, which incur several times more computational overhead and may introduce potential interference, as well as static internal signal enhancement, are often vulnerable to the attention sink phenomenon. We find that internal Positive Attention Dynamics (PAD) in LVLMs naturally reveal semantically core visual regions under the distortions of attention sinks. Based on this, we propose Positive Attention Dynamics Enhancement (PADE), a training-free attention intervention that constructs a PAD map to identify semantically core visual regions, applies per-head Median Absolute Deviation Scaling to adaptively control the intervention strength, and leverages System-Token Compensation to maintain attention to complex user instructions and support long-term output consistency. Experiments on multiple LVLMs and benchmarks show that PADE improves visual grounding and reduces hallucinations, validating the effectiveness of leveraging internal attention dynamics for reliable multimodal reasoning.

Figures & tables

Figure 1: Comparison of hallucination mitigation paradigms. (a) Contrastive decoding methods. (b) Auxiliary expert methods. (c) Static internal signal methods. (d) Ours PADE: internal positive attention dynamics.

Figure 2: Static versus dynamic internal attention signals. Static mean attention is dominated by attention sinks, while Positive Attention Dynamics (PAD) more reliably highlight semantically core visual regions.

Panel of Figure 3 — Figure 3: Attention analysis of LLaVA-1.5-7B (left) and 13B (right). Top: the attention ratio of different token types (System, Vision, Instruction, Output). Bottom: heatmap visualizations of attention distributions, including (a) static attention from uniformly sampled layers, (b) layer-averaged static attention, and (c) positive attention dynamics.

Figure 4: Overview of our PADE. PADE identifies semantically core visual regions via Positive Attention Dynamics (PAD) and selectively enhances them in the target layer, with Median Absolute Deviation Scaling for adaptively controlling the intervention strength and System-Token Compensation to preserve attention for understanding complex instructions and ensuring consistent long-term generation.

Setup	Method	LLaVA-1.5		InstructBLIP		Qwen-VL		LLaVA-1.5-13B		LLaVA-Next
Setup	Method	Accuracy $\uparrow$	F1 $\uparrow$	Accuracy $\uparrow$	F1 $\uparrow$	Accuracy $\uparrow$	F1 $\uparrow$	Accuracy $\uparrow$	F1 $\uparrow$	Accuracy $\uparrow$	F1 $\uparrow$
Random	Vanilla	84.63	84.99	83.33	83.57	85.17	83.00	83.27	84.27	84.23	82.14
	VCD	84.57	85.02	84.60	84.49	84.69	82.91	83.47	84.52	83.48	81.36
	PAI	85.12	85.64	83.82	83.98	85.63	83.74	84.06	84.98	83.76	81.52
	VAF	85.64	85.82	85.12	85.28	85.93	84.08	84.32	85.28	84.84	82.64
	VAR	86.12	86.56	85.62	85.71	86.54	84.66	84.82	85.76	85.12	83.26

Table 1: Results on the POPE (Accuracy and F1).

\uparrow

indicates that higher is better. Best results are bolded .

Method	LLaVA-1.5		InstructBLIP		Qwen-VL		LLaVA-1.5-13B		LLaVA-Next
Method	CHAIR S $\downarrow$	CHAIR I $\downarrow$	CHAIR S $\downarrow$	CHAIR I $\downarrow$	CHAIR S $\downarrow$	CHAIR I $\downarrow$	CHAIR S $\downarrow$	CHAIR I $\downarrow$	CHAIR S $\downarrow$	CHAIR I $\downarrow$
Vanilla	55.1	16.4	57.4	17.6	52.1	16.7	50.4	14.7	30.2	10.9
VCD	54.4	16.6	60.7	18.0	50.4	17.2	49.6	14.3	29.8	10.6
M3ID	56.6	15.8	62.3	18.2	49.8	17.4	—	—	—	—
Woodpecker	57.6	16.7	60.8	17.6	51.8	16.3	—	—	—	—
HALC	51.0	14.8	53.8	15.7	49.6	15.4	—	—	—	—

Table 2: Results on CHAIR (Max Token 128).

\downarrow

denotes lower is better. – denotes unavailable results.

Methods	VizWiz	MME			LLaVA-Wild	MM-Vet
Methods	Accuracy $\uparrow$	Perception $\uparrow$	Cognition $\uparrow$	Overall $\uparrow$	Average $\uparrow$	Total $\uparrow$
Vanilla	50.00	1508.97	355.71	1864.68	64.80	31.1
VCD	44.90	1515.01	357.86	1872.87	63.21	30.2
ICD	37.62	1306.91	287.86	1594.77	56.90	25.9
OPERA	50.76	1473.62	310.71	1784.34	64.31	32.0
INTER	48.77	1502.35	336.18	1838.53	61.70	30.9

Table 3: Results on multiple general vision language benchmarks.

\uparrow

indicates that higher is better.

Methods	fACC $\uparrow$	qACC $\uparrow$	${easy}$ A $\uparrow$	${hard}$ A $\uparrow$
Vanilla	17.9	8.13	36.0	36.7
VCD	13.9	11.4	33.0	34.7
ICD	13.9	8.35	36.9	33.5
OPERA	16.2	5.49	37.6	35.4
INTER	15.8	8.21	36.9	34.5
V-ITI	17.9	10.27	36.6	37.0

Table 4: Results on the HallusionBench.

Method	CHAIR ( $\downarrow$ )	Cover ( $\uparrow$ )	Hall ( $\downarrow$ )	Cog ( $\downarrow$ )
Vanilla	7.8	51.0	36.4	4.2
VCD	7.5	50.8	36.2	4.1
OPERA	7.3	49.6	32.0	3.5
DoLA	7.6	51.6	36.0	4.0
PAI	7.4	49.9	33.2	3.7
PADE (ours)	7.1	51.8	31.4	3.4

Table 5: Results on AMBER Generative Subset.

Model	Components	CHAIR S $\downarrow$	CHAIR I $\downarrow$
LLaVA-1.5-7B	PADE	48.6	13.7
	w/o MAD	54.9	16.3
	w/o STC	49.2	14.0
LLaVA-1.5-13B	PADE	47.9	12.8
	w/o MAD	50.3	14.5
	w/o STC	48.6	13.3

Table 6: Ablation results on the proposed components.

Panel of Figure 5 — Figure 5: Ablation results on the intervention layer and strength $\lambda$ of LLaVA-1.5-7B (top) and 13B (bottom).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Direction hallucination in open-ended long-term generation. Although the model attends to the dog throughout generation, it incorrectly describes the running direction as towards the water, while the dog is in fact moving away from it. A substantial portion of attention is absorbed by several semantically irrelevant sink tokens, especially in later layers, diluting the focus on the dog region. Positive Attention Dynamics (PAD) captures consistent attention changes concentrated around the dog region across layers. When applied at later layers, PADE selectively reinforces this core region after attention diffusion, correcting the directional hallucination.

Figure 7: Object existence hallucination in a cluttered scene. The image contains multiple objects, including several cups and a visually dominant bottle. Due to the presence of multiple salient objects, visual attention becomes fragmented, and smaller target objects such as the cups receive substantially less attention than the large bottle. Although both the cups and the bottle are attended to at intermediate layers, attention gradually diffuses away from semantically relevant regions, and sink tokens increasingly dominate the final-layer attention map, causing the cup regions to become inconspicuous. PAD captures the transient attention concentration on the cup regions before diffusion occurs, enabling PADE to re-emphasize these small but semantically critical regions in later layers and correctly recognize the existence of cups, thereby mitigating the hallucinated judgment.

Figure 8: Object existence hallucination for a small target. The phone occupies a relatively small visual region, resulting in weak and sparse attention signals. In later layers, attention disperses from the phone region and becomes highly imbalanced, with sink tokens dominating the distribution. Despite the absence of strong static attention, PAD identifies consistent attention dynamics localized around the phone region. By reinforcing this region at later layers, PADE restores visual grounding and alleviates the hallucination.

Figure 9: Object existence hallucination under occlusion and prompt variation. The bag is partially occluded and occupies a small visible area, leading to weak static attention signals. As attention diffuses in later layers, sink tokens dominate the distribution, resulting in hallucinated responses. PAD captures coherent attention dynamics around the hand and bag regions across layers and enables PADE to selectively enhance these regions in the target layer. This figure shares the same image as Figure 8 but uses a different user instruction (bag vs. phone). Both prompts exhibit similar sink-dominated attention patterns and hallucination behaviors, indicating that such failures arise from intrinsic LVLM attention dynamics rather than prompt-specific artifacts. PADE consistently leverages attention evolution to recover semantically core regions and mitigate hallucinations across prompts.

Figure 10: Comparison of static mean attention and positive attention dynamics during the generation of different tokens. (a) Original model output. (b) Static mean attention. (c) Positive Attention Dynamics (PAD). (d) Output after applying PADE, where semantically core regions are enhanced via PAD. Across all tokens, PAD more reliably highlights semantically core visual regions than static attention. In particular, when generating tokens corresponding to key objects (e.g., “apple”) and attributes (e.g., “red”), attention changes are most strongly concentrated on the apple region. These results demonstrate that PAD effectively captures semantically relevant visual areas and validate the effectiveness of PADE in reinforcing core visual regions for reliable multimodal understanding.

Attention Intervention in VLMs

See all 16 papers

See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs
Jun 29, 2026Yuqing Lei, Wenbo Lyu, Yingjun Du +3Visual Attention Large Vision-Language Models
University of Chinese Academy of Sciences · University of Amsterdam · United Imaging Healthcare Co., Ltd.
Look Clearly Before Answering: Mitigating Hallucinations in LVLMs via Saliency-Driven Perceptual Realignment
Jul 18, 2026Pengxu Chen, Yao Zhu, Guangming Zhu +4Large Vision-Language Models VLM Hallucination Mitigation
Xidian University Xi’an, China · Tsinghua University Beijing, China · Shanghai Road Transport Development Center Shanghai, China +1
Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy
May 20, 2026Yutong Xie, Zhenglin Hua, Ran Wang +3Vision-Language Models Hallucination in Language Models
School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Artificial Intelligence, Shenzhen University, Shenzhen, China · National Engineering Laboratory for Big Data Systems Computing Technology, Shenzhen University, Shenzhen, China +3

Jun 29, 2026J/K move · Enter open · S save

See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs

Yuqing Lei, Wenbo Lyu, Yingjun Du, Xiantong Zhen+2

University of Chinese Academy of Sciences · University of Amsterdam · United Imaging Healthcare Co., Ltd.

Visual Attention Large Vision-Language Models Object Hallucination in VLMs VLM Hallucination Mitigation Attention Intervention in VLMs

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) excel at multimodal tasks but remain prone to object hallucinations. Prior training-free remedies often uniformly strengthen visual signals, which may also amplify irrelevant regions and introduce spurious evidence, harming fluency. We propose Context-aware Attention Intervention (CAI), a training-free inference-time mechanism that enforces a see only when needed principle via two-axis selectivity: where to look and when to intervene. At each decoding step, CAI derives token-specific visual relevance from early-layer representations to localize semantically aligned regions, and applies a conservative, entropy- and depth-gated attention tilt only for uncertainty-spiking tokens in deeper layers where visual grounding degrades, leaving confident tokens and irrelevant regions largely unchanged. This targeted intervention strengthens visual grounding while preserving linguistic fluency, and it yields consistent improvements even without contrastive decoding, which remains optional as an auxiliary bias-suppression module. Extensive experiments across multiple LVLM backbones and benchmarks show that CAI achieves state-of-the-art hallucination mitigation, and our analysis characterizes CAI as a KL-minimal attention reweighting with bounded interference under inactive gates or small tilts. Code is available at https://github.com/Iris1946/CAI.

Abstract

Attention Intervention in VLMs

Explore similar work

See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs

Look Clearly Before Answering: Mitigating Hallucinations in LVLMs via Saliency-Driven Perceptual Realignment

Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy