EviAnchor: Mitigating Hallucinations in Large Vision-Language Models via Regional Visual Evidence Compensation

Sihang Jia; Shuliang Liu; Songbo Yang; Xuming Hu

cs.AIAug 29, 2026

EviAnchor: Mitigating Hallucinations in Large Vision-Language Models via Regional Visual Evidence Compensation

Authors: Sihang Jia, Shuliang Liu, Songbo Yang, Xuming Hu

Organizations: The Hong Kong University of Science and Technology (Guangzhou)

Abstract

Large vision-language models (LVLMs) frequently generate content unsupported by visual inputs. Preliminary experiments show that visual evidence is primarily incorporated into answer-side representations in early-to-middle decoder layers, while its direct influence progressively weakens in later layers. This attenuation suggests that visual evidence acquired earlier may be insufficiently utilized during subsequent generation. Based on this observation, we propose EviAnchor, a training-free and single-branch inference framework that preserves and reactivates visual evidence throughout generation. EviAnchor introduces Regional Evidence Anchor (REA) slots to progressively aggregate dense visual tokens into spatially structured representations. It then strengthens the current decision state's access to these visual anchors through decision-conditioned evidence routing, mitigating excessive dependence on textual context. Finally, the model resumes its native Transformer computation to integrate the retrieved visual evidence with question semantics and generation history. Experiments across POPE, CHAIR, and MMHal-Bench demonstrate consistent improvements in visual grounding.

Figures & tables

Panel of Figure 1 — Figure 1: Layer-wise causal analysis of visual information flow. (a) We pair images containing opposite visual facts and selectively retain text-to-visual attention either before or after each layer boundary. The resulting image-conditioned decision gap measures where visual information is transferred into answer-side representations. (b) We replace visual-token states within successive layer windows with those from a blank-image branch, while perturbing question-token states as a textual reference, and measure the resulting decision changes. Additional implementation details are provided in the supplementary material.

Figure 2: Qualitative illustration of insufficient cross-layer utilization of visual evidence. The native model generates unsupported objects highlighted in red. Ablating the original visual-token states in later layers leaves these hallucinated mentions nearly unchanged, indicating that the prediction has limited dependence on late-layer visual representations.

Figure 3: Overview of EviAnchor. The Regional Evidence Anchor aggregates visual patches into spatial REA slots, the Decision-Conditioned Evidence Routing reconnects the current generation state with these visual anchors, and the Native Semantic Recomposition restores the original Transformer computation for coherent response generation.

Method	LLaVA-1.5-7B		Qwen3-VL-4B		InstructBLIP-7B
Method	Acc. $\uparrow$	F1 $\uparrow$	Acc. $\uparrow$	F1 $\uparrow$	Acc. $\uparrow$	F1 $\uparrow$
Origin	76.46	80.41	85.37	85.63	77.93	80.11
VCD	77.79	80.70	89.11	89.23	80.93	81.47
ICD	77.62	80.63	88.74	88.84	80.41	81.39
CICD	78.69	81.51	88.96	88.97	81.59	83.09
ClearSight	77.23	80.51	88.67	88.56	79.37	81.57

Table 1: Performance evaluation on POPE (%). Best results are highlighted in bold .

Method	LLaVA-1.5-7B				Qwen3-VL-4B				InstructBLIP-7B
Method	CHAIRs $\downarrow$	CHAIRi $\downarrow$	Rec. $\uparrow$	Len.	CHAIRs $\downarrow$	CHAIRi $\downarrow$	Rec. $\uparrow$	Len.	CHAIRs $\downarrow$	CHAIRi $\downarrow$	Rec. $\uparrow$	Len.
Origin	58.20	17.06	76.88	101.866	47.00	12.73	67.06	242.152	48.60	14.69	73.15	101.550
VCD	57.00	17.70	76.88	102.116	48.40	10.93	67.68	241.214	57.20	16.00	72.53	96.824
ICD	56.40	17.21	76.41	101.742	47.60	11.18	67.31	240.886	55.80	15.62	72.87	98.316
CICD	48.80	14.84	75.57	102.406	48.40	10.89	67.06	240.054	48.00	13.06	71.35	99.642
ClearSight	52.40	15.50	77.04	98.526	51.20	10.91	67.25	241.528	51.20	14.10	73.83	100.830

Table 2: Performance evaluation on CHAIR (%). CHAIRs and CHAIRi measure sentence-level and instance-level object hallucinations, respectively. Best results are highlighted in bold .

Method	LLaVA-1.5		Qwen3-VL		InstructBLIP
Method	Score $\uparrow$	Rate $\downarrow$	Score $\uparrow$	Rate $\downarrow$	Score $\uparrow$	Rate $\downarrow$
Origin	2.64	58.3	4.04	40.6	2.06	64.4
VCD	2.25	63.5	4.17	37.5	2.08	65.6
ICD	2.31	57.3	4.26	36.5	2.18	61.5
CICD	2.66	58.3	4.17	36.5	2.26	60.4
ClearSight	2.52	57.3	4.00	39.6	2.12	63.5

Table 3: Performance evaluation on MMHal-Bench. Best results for each backbone are highlighted in bold .

Variant	POPE		CHAIR
Variant	Acc. $\uparrow$	F1 $\uparrow$	CHAIR ${}_{s}\downarrow$	CHAIR ${}_{i}\downarrow$	Rec. $\uparrow$
Origin	84.73	84.21	58.20	17.06	76.88
w/o Regional Anchoring	85.27	85.43	53.60	15.14	76.95
w/o Evidence Routing	85.57	85.83	52.20	14.78	76.90
w/o Native Recomposition	85.60	85.34	51.40	14.52	71.80
EviAnchor	86.37	86.76	48.80	14.10	77.38

Table 4: Component ablation results on POPE and CHAIR. w/o Regional Anchoring removes the REA slots and redistributes the transferred attention mass to the original visual tokens. w/o Evidence Routing disables textual perturbation and attention transfer. w/o Native Recomposition extends the routing intervention to the final decoder layer.

REA Design	POPE		CHAIR
REA Design	Acc. $\uparrow$	F1 $\uparrow$	CHAIR ${}_{s}\downarrow$	CHAIR ${}_{i}\downarrow$	Rec. $\uparrow$	Empty $\downarrow$
Global $1\times 1$ (1)	85.10	85.71	54.80	15.64	76.70	0
Spatial $2\times 2$ (4)	85.10	85.80	52.60	15.06	76.90	0
Spatial $3\times 3$ (9)	86.37	86.76	48.80	14.10	77.38	0
Spatial $4\times 4$ (16)	87.43	87.28	49.60	10.90	71.48	8
Spatial $4\times 4$ (16) †	86.04	86.37	45.20	13.28	77.26	0

Table 5: Ablation on the number of REA slots. Variants marked with

\dagger

and

\ddagger

use relaxation strengths

\lambda=0.1

and

\lambda=0

, respectively.

Figure 4: Progressive visual evidence aggregation during regional anchoring. The REA-to-Image curve shows the attention from REA slots to their corresponding image tokens, while the Image Tokens curve shows the current decision state’s attention to the original visual tokens.

Figure 5: Effect of evidence-routing length.

VLM Hallucination Mitigation

Vision-language model hallucination mitigation keeps generated claims grounded in visual evidence.

See all 228 papers

Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy
May 20, 2026Yutong Xie, Zhenglin Hua, Ran Wang +3Vision-Language Models Hallucination in Language Models
School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Artificial Intelligence, Shenzhen University, Shenzhen, China · National Engineering Laboratory for Big Data Systems Computing Technology, Shenzhen University, Shenzhen, China +3
Self-Correction Inside the Model: Leveraging Layer Attention to Mitigate Hallucinations in Large Vision Language Models
Feb 28, 2026April FuCross-Layer Attention Self-Correction in VLMs
Independent Researcher
SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineering
Jul 5, 2026Kai Tang, Jinhao You, Bohua Zhang +6Large Vision-Language Models Object Hallucination in VLMs
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Nanyang Technological University · University of Pennsylvania +4

May 20, 2026J/K move · Enter open · S save

Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy

Yutong Xie, Zhenglin Hua, Ran Wang, Wing W. Y. Ng+2

School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Artificial Intelligence, Shenzhen University, Shenzhen, China · National Engineering Laboratory for Big Data Systems Computing Technology, Shenzhen University, Shenzhen, China+3

Vision-Language Models Hallucination in Language Models VLM Hallucination Large Vision-Language Models VLM Hallucination Mitigation

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) have shown remarkable performance on a wide range of vision-language tasks. Despite this progress, they are still prone to hallucination, generating responses that are inconsistent with visual content. In this work, we find that LVLMs tend to hallucinate when they pay insufficient attention to the correct visual evidence and gradually forget it during the generation process. We empirically find that although LVLMs overall attend insufficiently to visual evidence, they exhibit sensitivity to the correct visual evidence in specific layers, with notable inter-layer discrepancy. Motivated by this observation, we propose a novel hallucination mitigation method that enhances visual evidence based on Inter-Layer Visual Attention Discrepancy (ILVAD). Specifically, we obtain the attention weights from early generated tokens to visual tokens across layers and identify the tokens that are repeatedly activated as visual evidence, forming a saliency map. We then enhance attention to visual evidence during generation through the saliency map to reduce visual forgetting. In addition, we leverage the saliency map to obtain attention scores of generated text to visual evidence, in order to select and emphasize text tokens that are strongly grounded in visual evidence. Our method is training-free and plug-and-play. Multiple benchmark evaluations conducted on five recently released models show that our method can consistently mitigate hallucinations in different LVLMs over various architectures. Code is available at https://github.com/ytx-ML/ILVAD.

Abstract

VLM Hallucination Mitigation

Explore similar work

Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy

Self-Correction Inside the Model: Leveraging Layer Attention to Mitigate Hallucinations in Large Vision Language Models

SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineering