AdaIAT: Adaptively Increasing Attention to Generated Text to Alleviate Hallucinations in LVLM

Li'an Zhong; Ziqiang He; Jibin Zheng; Jin Li; Z. Jane Wang; Xiangui Kang

cs.CVMar 5, 2026

AdaIAT: Adaptively Increasing Attention to Generated Text to Alleviate Hallucinations in LVLM

Authors: Li'an Zhong, Ziqiang He, Jibin Zheng, Jin Li, Z. Jane Wang, Xiangui Kang

Organizations: Sun Yat-Sen University · Foshan University · University of British Columbia

Abstract

Hallucination has been a significant impediment to the development and application of current Large Vision-Language Models (LVLMs). To mitigate hallucinations, one intuitive and effective way is to directly increase attention weights to image tokens during inference. Although this effectively reduces the hallucination rate, it often induces repetitive descriptions. To address this, we first conduct an analysis of attention patterns and reveal that real object tokens tend to assign higher attention to the generated text than hallucinated ones. This inspires us to leverage the generated text, which contains instruction-related visual information and contextual knowledge, to alleviate hallucinations while maintaining linguistic coherence. We therefore propose Attention to Generated Text (IAT) and demonstrate that it significantly reduces the hallucination rate while avoiding repetitive descriptions. To prevent naive amplification from impairing the inherent prediction capabilities of LVLMs, we further explore Adaptive IAT (AdaIAT) that employs a layer-wise threshold to control intervention time and fine-grained amplification magnitude tailored to the characteristics of each attention head. Both analysis and experiments demonstrate the effectiveness of AdaIAT. Results of several LVLMs show that AdaIAT effectively alleviates hallucination (reducing hallucination rates $C_S$ and $C_I$ on LLaVA-1.5 by 35.8% and 37.1%, respectively) while preserving linguistic performance and prediction capability, achieving an attractive trade-off.

Figures & tables

Figure 1: Case illustration: The attention intervention methods amplify the attention to image tokens, thereby emphasizing visual information and mitigating hallucinations. However, the relatively low attention to the generated text causes the model to forget preceding utterances, leading to repetitive descriptions of the prominent object ‘clock tower’.

Figure 2: The mechanisms of mitigating hallucination based on Greedy in PAI and AdaIAT, with repeated descriptions annotated in red and hallucinations annotated in blue . While Greedy generates the hallucinated object “cars”, PAI enhances attention to image tokens with a fixed $\alpha$ to mitigate hallucination. However, it suffers from repeated subjects and monotonous, redundant language. In contrast, AdaIAT employs layer-wise thresholds to control the amplification and designs $\mathcal{M}^{(l,h)}$ for each attention head to adaptively enhance attention towards the generated text tokens, which produces accurate and hallucination-free captions.

Panel of Figure 3 — Figure 3: Visualization of the average per-token attention weights from text token $t_{n+1}$ to generated text tokens $T_{p}$ ( $\bar{\mathbf{A}}_{T_{p}}^{r}$ and $\bar{\mathbf{A}}_{T_{p}}^{h}$ ) and to image tokens $V$ ( $\bar{\mathbf{A}}_{V}^{r}$ and $\bar{\mathbf{A}}_{V}^{h}$ ), showing only layers 5–18 for clearer observation.

Figure 4: Trends of textual diversity

D_{1}

for different methods as the hallucination rate

C_{S}

decreases. The dashed line denotes the

D_{1}

of the original greedy decoding as a reference. Regions closer to the top-left indicate better performance.

Figure 5: The distributions of textual diversity

D_{1}

for captions generated by different methods using the LLaVA-1.5-7B, where a higher

D_{1}

corresponds to greater text diversity. The distribution is partially truncated for better visualization.

Figure 6: Characteristics of different heads on LLaVA-1.5-7B. Left: Visualization of the average per‑token attention weight to generated text for each layer and head, computed over the sets of real objects ( ${\mathbf{A}}_{T_{p}}^{r}$ ) and hallucinated objects ( ${\mathbf{A}}_{T_{p}}^{h}$ ). Right: The visualization of ${\mathbf{A}}_{T_{p}}^{r}/{\mathbf{A}}_{T_{p}}^{h}$ . Though ${\mathbf{A}}_{T_{p}}^{r}$ is generally higher than ${\mathbf{A}}_{T_{p}}^{h}$ , the fold‑difference varies across different heads.

Models	LLaVA-1.5-7B [ 20 ]				LLaVA-1.5-13B [ 20 ]				Janus-Pro-7B [ 9 ]				Qwen2.5-VL-7B [ 3 ]
Metrics	$C_{S}\downarrow$	$C_{I}\downarrow$	F1 $\uparrow$	$D_{1}\uparrow$	$C_{S}\downarrow$	$C_{I}\downarrow$	F1 $\uparrow$	$D_{1}\uparrow$	$C_{S}\downarrow$	$C_{I}\downarrow$	F1 $\uparrow$	$D_{1}\uparrow$	$C_{S}\downarrow$	$C_{I}\downarrow$	F1 $\uparrow$	$D_{1}\uparrow$
Sample	53.0	15.6	73.4	0.65	52.0	13.7	74.9	0.66	34.0	9.0	73.5	0.64	42.0	11.3	72.5	0.67
AdaIAT †	42.0	12.3	73.1	0.66	37.8	10.1	75.0	0.67	29.8	8.1	72.6	0.65	34.6	9.0	75.1	0.68
Greedy	49.0	13.3	77.9	0.60	47.8	12.5	78.9	0.60	25.8	6.7	76.8	0.62	33.6	8.3	76.5	0.65
PAI [ 21 ]	31.8	7.8	77.7	0.50	34.2	9.0	79.3	0.56	20.4	5.6	76.1	0.61	32.0	8.2	74.7	0.64
HGAI [ 14 ]	31.4	6.9	78.3	0.50	26.4	7.8	76.9	0.55	21.0	5.3	75.9	0.62	30.4	8.0	76.8	0.65

Table 1: CHAIR hallucination evaluation of various methods on three LVLMs. The best values (excluding Greedy) are highlighted in bold .

{\dagger}

indicates that the attention intervention is applied based on the Sample decoding.

	Greedy	PAI [ 21 ]	HGAI [ 14 ]	IAT	AdaIAT
$C_{o}\downarrow$	0.292	0.266	0.261	0.254	0.252
$D_{1}\uparrow$	0.61	0.51	0.53	0.61	0.61

Table 2: OpenCHAIR hallucination evaluation of different methods on LLaVA-1.5-7B.

Figure 7: Results of HalluBench in LLaVA-1.5-7B. Five aspects are analyzed: hallucinated sentences per image (HSPI), hallucinated words per image (HWPI), hallucinated sentences ratio (HSR), hallucinated words ratio (HWR), and Distinct-1 (

D_{1}

). Larger radar area indicates better performance.

	Greedy	PAI [ 21 ]	HGAI [ 14 ]	IAT	AdaIAT
$D_{1}\uparrow$	0.614	0.498	0.492	0.609	0.610
$B_{self}\downarrow$	0.058	0.242	0.247	0.090	0.071
$B_{r}\uparrow$	85.29	84.77	84.78	85.20	85.17
$B_{d}\uparrow$	59.48	56.80	56.61	58.99	58.29

Table 3: Evaluation results of the text quality for captions generated by different methods on LLaVA-1.5-7B.

	$\alpha$	0.6	0.7	0.8	0.9	1
IAT	$C_{S}\downarrow$	36.4	35.2	29.8	21.2	13.6
	$C_{I}\downarrow$	10.3	10.5	9.0	7.0	5.0
	F1 $\uparrow$	78.1	77.4	76.8	73.5	68.4
	$D_{1}\uparrow$	0.62	0.61	0.61	0.57	0.52
	$\alpha$	4	5	6	7	8
AdaIAT	$C_{S}\downarrow$	34.6	33.8	31.4	31.8	29.4

Table 4: Performances of IAT and AdaIAT under different amplification factor

\alpha

$\beta$	0.1	0.3	0.5	0.7	1	1.5
$C_{S}\downarrow$	32.2	32.6	31.4	33.0	32.6	34.4
$C_{I}\downarrow$	9.2	8.7	8.3	8.7	8.89	9.66
F1 $\uparrow$	79.1	79.0	79.4	78.8	78.5	78.1
$D_{1}\uparrow$	0.599	0.600	0.601	0.597	0.599	0.592

Table 5: Performances of AdaIAT under different balanced coefficient

\beta

	Layers	0-5	18-31	0-18	5-31	5-18
IAT	$C_{S}\downarrow$	24.0	26.0	4.0	1.1	29.8
	$C_{I}\downarrow$	7.1	8.1	3.3	1.8	9.0
	F1 $\uparrow$	75.3	73.8	48.5	30.3	76.8
	$D_{1}\uparrow$	0.582	0.604	0.163	0.036	0.606
AdaIAT	$C_{S}\downarrow$	45.2	43.8	27.4	29.0	31.4
	$C_{I}\downarrow$	12.9	11.9	7.1	8.6	8.3

Table 6: The impact of different layer schemes on the hallucination mitigation performance of IAT and AdaIAT.

Attention Intervention in VLMs

See all 15 papers

Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy
May 20, 2026Yutong Xie, Zhenglin Hua, Ran Wang +3Vision-Language Models Hallucination in Language Models
School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Artificial Intelligence, Shenzhen University, Shenzhen, China · National Engineering Laboratory for Big Data Systems Computing Technology, Shenzhen University, Shenzhen, China +3
See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs
Jun 29, 2026Yuqing Lei, Wenbo Lyu, Yingjun Du +3Visual Attention Large Vision-Language Models
University of Chinese Academy of Sciences · University of Amsterdam · United Imaging Healthcare Co., Ltd.
Look Clearly Before Answering: Mitigating Hallucinations in LVLMs via Saliency-Driven Perceptual Realignment
Jul 18, 2026Pengxu Chen, Yao Zhu, Guangming Zhu +4Large Vision-Language Models VLM Hallucination Mitigation
Xidian University Xi’an, China · Tsinghua University Beijing, China · Shanghai Road Transport Development Center Shanghai, China +1

May 20, 2026J/K move · Enter open · S save

Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy

Yutong Xie, Zhenglin Hua, Ran Wang, Wing W. Y. Ng+2

School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Artificial Intelligence, Shenzhen University, Shenzhen, China · National Engineering Laboratory for Big Data Systems Computing Technology, Shenzhen University, Shenzhen, China+3

Vision-Language Models Hallucination in Language Models VLM Hallucination Large Vision-Language Models VLM Hallucination Mitigation

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) have shown remarkable performance on a wide range of vision-language tasks. Despite this progress, they are still prone to hallucination, generating responses that are inconsistent with visual content. In this work, we find that LVLMs tend to hallucinate when they pay insufficient attention to the correct visual evidence and gradually forget it during the generation process. We empirically find that although LVLMs overall attend insufficiently to visual evidence, they exhibit sensitivity to the correct visual evidence in specific layers, with notable inter-layer discrepancy. Motivated by this observation, we propose a novel hallucination mitigation method that enhances visual evidence based on Inter-Layer Visual Attention Discrepancy (ILVAD). Specifically, we obtain the attention weights from early generated tokens to visual tokens across layers and identify the tokens that are repeatedly activated as visual evidence, forming a saliency map. We then enhance attention to visual evidence during generation through the saliency map to reduce visual forgetting. In addition, we leverage the saliency map to obtain attention scores of generated text to visual evidence, in order to select and emphasize text tokens that are strongly grounded in visual evidence. Our method is training-free and plug-and-play. Multiple benchmark evaluations conducted on five recently released models show that our method can consistently mitigate hallucinations in different LVLMs over various architectures. Code is available at https://github.com/ytx-ML/ILVAD.

Abstract

Attention Intervention in VLMs

Explore similar work

Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy

See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs

Look Clearly Before Answering: Mitigating Hallucinations in LVLMs via Saliency-Driven Perceptual Realignment