Attention-space Contrastive Guidance for Efficient Hallucination Mitigation in LVLMs

Yujin Jo; Sangyoon Bae; Taesup Kim

cs.CVJan 20, 2026

Attention-space Contrastive Guidance for Efficient Hallucination Mitigation in LVLMs

Authors: Yujin Jo, Sangyoon Bae, Taesup Kim

Organizations: Seoul National University

Abstract

Hallucinations in large vision--language models (LVLMs) often arise when language priors dominate over visual evidence, leading to object misidentification and visually inconsistent descriptions. We address this problem by framing hallucination mitigation as contrastive guidance that steers generation toward visually grounded and semantically faithful text. We propose Attention-space Contrastive Guidance (ACG), a training-free, single-pass method that operates directly in self-attention layers, where hallucination-inducing cross-modal biases emerge. ACG constructs both image-conditioned and approximate text-only attention paths within a single forward pass, enabling efficient guidance before errors accumulate at the output layer. Because this masking-based surrogate can introduce approximation bias, we further apply a lightweight orthogonal projection that suppresses components aligned with the text-only path, yielding a more visually grounded correction. Experiments on CHAIR and POPE show that ACG improves faithfulness over existing training-free baselines while maintaining caption quality, reducing latency by up to $2\times$ compared to multi-pass contrastive decoding methods.

Figures & tables

Figure 1: Comparison of inference-time strategies for mitigating LVLM hallucinations. (a) Logit-level contrastive decoding, (b) hidden-state-level latent steering, (c) attention map intervention, and (d) the proposed Attention-space Contrastive Guidance (ACG).

Model	Method	Avg.	Random				Popular				Adversarial
Model	Method	Acc.	Acc.	Prec.	Rec.	F1	Acc.	Prec.	Rec.	F1	Acc.	Prec.	Rec.	F1
LLaVA [ 19 ]	Regular	84.83	89.37	89.66	89.00	89.33	86.00	84.05	88.87	86.39	79.13	74.39	88.87	80.98
	VCD [ 14 ]	85.38	88.63	91.96	84.67	88.16	85.97	86.93	84.67	85.78	81.53	79.67	84.67	82.09
	PAI [ 21 ]	84.91	89.30	89.54	89.00	89.27	86.13	84.26	88.87	86.50	79.30	74.59	88.87	81.11
	VISTA [ 18 ]	83.03	88.25	87.31	90.33	88.79	84.10	80.21	90.53	85.06	76.73	70.93	90.60	79.57
	Ours	86.03	88.30	95.92	80.00	87.24	86.57	92.22	79.87	85.60	83.23	85.63	79.87	82.65

Table 1: POPE results (%). We report overall average accuracy (Avg.) as well as accuracy (Acc.), precision (Prec.), recall (Rec.), and F1 under the Random , Popular , and Adversarial settings. Bold indicate the best performance in each column.

Model	Method	Max Tokens 128			Max Tokens 64
Model	Method	CHAIR s	CHAIR i	F1	CHAIR s	CHAIR i	F1
LLaVA [ 19 ]	Regular	56.2	18.3	70.6	25.2	8.3	68.5
	VCD	55.0	17.0	72.5	27.2	8.8	70.0
	PAI	25.6	7.6	75.9	13.8	4.5	72.4
	VISTA	31.0	10.5	76.6	23.6	7.6	74.3
	Ours	21	4.8	74.4	16.8	4.5	72.8

Table 2: CHAIR results (%) on LLaVA-1.5 and MiniGPT-4. Bold indicates the best CHAIR performance and underlines indicate the second-best in each column.

	MMHal ( $\uparrow$ )	MMMU ( $\uparrow$ )			MathVista ( $\uparrow$ )
Method		MC	Open	Total
Regular	1.94	37.54	3.77	35.56	22.6
Ours ( $\gamma$ =2)	2.12	38.49	9.43	36.78	23.7

Table 3: Performance on MMHal, MMMU, and MathVista.

Model	Method	CHAIR s	CHAIR i	F1
LLaVA-NeXT 7B	Regular	31.2	8.1	72.1
LLaVA-NeXT 7B	Ours	25.2	5.4	73.9
LLaVA-NeXT 13B	Regular	33.8	8.3	71.5
LLaVA-NeXT 13B	Ours	31.0	5.5	74.9

Table 4: Generalization results (%) on LLaVA-NeXT 7B and 13B under longer decoding. We evaluate CHAIR with max_new_tokens=512 . We use

\gamma=2.0

for both models.

Figure 2: Justifying the masked unconditional path. As visual information is degraded by Gaussian noise, hallucination increases, fidelity decreases, and the mean text-to-image (T2I) attention ratio shows an overall downward trend.

Method	$\gamma$	F1 $\uparrow$	CHAIR s $\downarrow$	CHAIR i $\downarrow$
ACG (w/ Ortho)	2.1	77.6	34.2	7.6
ACG (w/o Ortho)	1.2	77.4	38.8	9.7
ACG (w/ Ortho)	2.4	74.4	21.0	4.8
ACG (w/o Ortho)	1.3	74.0	30.4	8.8

Table 5: Ablation at matched F1. At similar object-level fidelity (F1), ACG (w/ Ortho) yields lower CHAIR s and CHAIR i than ACG (w/o Ortho).

Figure 3: Guidance-scale trade-off on LLaVA-1.5 (CHAIR, max 128). Increasing $\gamma$ reduces instance-level hallucination (blue; left axis), but overly large $\gamma$ degrades object-level fidelity (red; right axis). The dotted line marks the canonical setting $\gamma{=}2.4$ .

Layer Block	$\gamma$	CHAIR i (↓)	F1 (↑)	Len (↓)
All (1–32)	2.4	4.8	74.4	72.4
Early (1–8)	2.5	7.1	77.5	77.6
Early (1–8)	3.0	5.3	69.8	68.0
Early–Mid (9–16)	6.0	11.1	74.8	79.2
Early–Mid (9–16)	8.0	5.7	56.6	45.9
Mid–Late (17–24)	6.0	10.7	77.4	95.6

Table 6: Block-wise characterization on LLaVA-1.5 (CHAIR, max 128). We partition the decoder into contiguous layer blocks and report representative operating points from each

\gamma

sweep.

Method	Level	Passes	Per-image Latency (s) $\downarrow$	Per-word Latency (s) $\downarrow$	CHAIR i $\downarrow$
Regular	–	1-pass	2.81 (1.00 $\times$ )	0.03	18.3
VCD	Logit	2-pass	5.54 (1.97 $\times$ )	0.06	17.0
PAI	Logit+Attn	2-pass	6.42 (2.28 $\times$ )	0.07	7.6
VISTA	Latent	3-pass	5.55 (1.98 $\times$ )	0.07	10.5
ACG-Fast	Attention	1-pass	2.96 (1.05 $\times$ )	0.04	7.3
ACG-Full	Attention	1-pass	3.34 (1.19 $\times$ )	0.05	4.8

Table 7: Efficiency vs. faithfulness on LLaVA-1.5 (CHAIR, max 128). We compare latency, number of forward passes, and CHAIR i . All values are averaged over the evaluation set.

Figure 4: Comparison of responses from the baseline LLaVA-1.5 and LLaVA-1.5 with ACG. Hallucinated or incorrect content is shown in red , and accurate content in blue .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Table 8: ACG quantitative validation. (LLaVA-1.5, POPE random split)

$\gamma$	CHAIR s (↓)	CHAIR i (↓)	F1 (↑)	Len
1.0	47.4	12.8	77.8	91.6
1.3	48.6	12.9	78.1	89.4
1.5	44.8	11.6	78.1	88.2
1.7	42.4	10.3	79.1	84.9
1.9	36.4	8.6	79.3	83.5
2.0	33.0	8.0	79.0	82.1

Appendix

Table 9: Effect of guidance scale

\gamma

on CHAIR (max 128) for LLaVA-1.5. We report sentence-level hallucination (CHAIR s ), instance-level hallucination (CHAIR i ), F1, and average caption length (Len). We choose

\gamma=\mathbf{2.4}

as our operating point.

$\gamma$	CHAIR s (↓)	CHAIR i (↓)	F1 (↑)	Len
0.10	27.6	9.0	70.6	72.9
0.15	24.4	7.9	71.0	69.4
0.20	21.0	6.8	70.3	63.5
0.25	16.6	5.2	69.5	59.1
0.30	10.8	3.3	68.0	66.3
0.35	6.2	2.5	63.2	73.6

Appendix

Table 10: Effect of guidance scale

\gamma

on CHAIR (max 128) for MiniGPT-4. We again report CHAIR s , CHAIR i , F1, and average caption length (Len). We choose

\gamma=\mathbf{0.3}

as our operating point.

Table 11: Layer-block ablation on LLaVA-1.5 (CHAIR, max 128). We apply ACG only to a given layer block and report CHAIR s , CHAIR i , F1, and average caption length (Len).

$\gamma$	Avg	ATTR	ADV	COMP	COUNT	SPAT	ENV	HOL	OTH	Hallucination Rate
0 (Vanilla)	1.94	2.25	1.42	2.67	1.50	1.92	2.92	1.75	1.08	0.59
1	2.06	2.25	1.33	2.17	1.33	2.75	3.25	2.08	1.33	0.56
1.5	2.07	2.50	1.08	1.75	1.92	2.00	3.25	2.08	2.00	0.57
2	2.12	2.50	1.67	1.92	1.58	2.25	3.25	2.00	1.83	0.53
2.4	2.01	2.58	2.00	1.83	1.17	2.25	3.25	1.17	1.83	0.56

Appendix

Table 12: Category-wise informativeness and hallucination rate on MMHal.

Figure 5: Success examples on MMHal-Bench. In the first example, vanilla LLaVA-1.5 hallucinates a bright and sunny environment, whereas our method correctly infers that the cabin is dark and lit by artificial light. In the second example, vanilla misreads the runner’s bib number as 1019 , while our method outputs the correct number 1097 , matching the ground-truth answer.

Figure 6: Qualitative CHAIR examples comparing vanilla LLaVA-1.5, PAI, and our ACG method. For each caption, we highlight object tokens: blue tokens denote objects that are grounded in the image, while red tokens indicate hallucinated objects (e.g., sink , cup , table , knife ). Compared to the baselines, ACG removes spurious background objects and focuses on the truly visible entities (e.g., the toaster, glasses, paper, and scissors).

Panel of Figure 7 — Figure 7: Qualitative comparison on MMMU benchmark.

Panel of Figure 8 — Figure 8: Qualitative comparison on Mathvista benchmark.

VLM Hallucination Mitigation

Vision-language model hallucination mitigation keeps generated claims grounded in visual evidence.

See all 228 papers

See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs
Jun 29, 2026Yuqing Lei, Wenbo Lyu, Yingjun Du +3Visual Attention Large Vision-Language Models
University of Chinese Academy of Sciences · University of Amsterdam · United Imaging Healthcare Co., Ltd.
Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy
May 20, 2026Yutong Xie, Zhenglin Hua, Ran Wang +3Vision-Language Models Hallucination in Language Models
School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Artificial Intelligence, Shenzhen University, Shenzhen, China · National Engineering Laboratory for Big Data Systems Computing Technology, Shenzhen University, Shenzhen, China +3
Look Clearly Before Answering: Mitigating Hallucinations in LVLMs via Saliency-Driven Perceptual Realignment
Jul 18, 2026Pengxu Chen, Yao Zhu, Guangming Zhu +4Large Vision-Language Models VLM Hallucination Mitigation
Xidian University Xi’an, China · Tsinghua University Beijing, China · Shanghai Road Transport Development Center Shanghai, China +1

Jun 29, 2026J/K move · Enter open · S save

See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs

Yuqing Lei, Wenbo Lyu, Yingjun Du, Xiantong Zhen+2

University of Chinese Academy of Sciences · University of Amsterdam · United Imaging Healthcare Co., Ltd.

Visual Attention Large Vision-Language Models Object Hallucination in VLMs VLM Hallucination Mitigation Attention Intervention in VLMs

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) excel at multimodal tasks but remain prone to object hallucinations. Prior training-free remedies often uniformly strengthen visual signals, which may also amplify irrelevant regions and introduce spurious evidence, harming fluency. We propose Context-aware Attention Intervention (CAI), a training-free inference-time mechanism that enforces a see only when needed principle via two-axis selectivity: where to look and when to intervene. At each decoding step, CAI derives token-specific visual relevance from early-layer representations to localize semantically aligned regions, and applies a conservative, entropy- and depth-gated attention tilt only for uncertainty-spiking tokens in deeper layers where visual grounding degrades, leaving confident tokens and irrelevant regions largely unchanged. This targeted intervention strengthens visual grounding while preserving linguistic fluency, and it yields consistent improvements even without contrastive decoding, which remains optional as an auxiliary bias-suppression module. Extensive experiments across multiple LVLM backbones and benchmarks show that CAI achieves state-of-the-art hallucination mitigation, and our analysis characterizes CAI as a KL-minimal attention reweighting with bounded interference under inactive gates or small tilts. Code is available at https://github.com/Iris1946/CAI.

Abstract

VLM Hallucination Mitigation

Explore similar work

See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs

Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy

Look Clearly Before Answering: Mitigating Hallucinations in LVLMs via Saliency-Driven Perceptual Realignment