ACT Now: Preempting LVLM Hallucinations via Adaptive Context Integration

Bei Yan; Yuecong Min; Jie Zhang; Shiguang Shan; Xilin Chen

cs.CVApr 1, 2026

ACT Now: Preempting LVLM Hallucinations via Adaptive Context Integration

Authors: Bei Yan, Yuecong Min, Jie Zhang, Shiguang Shan, Xilin Chen

Organizations: State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

Abstract

Large Vision-Language Models (LVLMs) frequently suffer from severe hallucination issues. Existing mitigation strategies predominantly rely on isolated, single-step states to enhance visual focus or suppress strong linguistic priors. However, these static approaches neglect dynamic context changes across the generation process and struggles to correct inherited information loss. To address this limitation, we propose Adaptive Context inTegration (ACT), a training-free inference intervention method that mitigates hallucination through the adaptive integration of contextual information. Specifically, we first propose visual context exploration, which leverages spatio-temporal profiling to adaptively amplify attention heads responsible for visual exploration. To further facilitate vision-language alignment, we propose semantic context aggregation that marginalizes potential semantic queries to effectively aggregate visual evidence, thereby resolving the information loss caused by the discrete nature of token prediction. Extensive experiments across diverse LVLMs demonstrate that ACT significantly reduces hallucinations and achieves competitive results on both discriminative and generative benchmarks, acting as a robust and highly adaptable solution without compromising fundamental generation capabilities.

Figures & tables

Figure 1: Illustration of temporal evolution in cross-modal attention during generation. (Left) Visual attention typically surges before the model predicts the corresponding token. (Top Right) Attention heads show distinct behavioral patterns during decoding. (Bottom Right) Discretization of low-confidence predictions causes information loss, resulting in poor grounding and potential hallucinations.

Figure 2: Overview of the proposed ACT method. (Left) VCE amplifies dynamic heads to capture broader visual evidence. (Right) SCA marginalizes parallel textual hypotheses to preemptively resolve local linguistic uncertainty.

	POPE		MME					CHAIR
Method	Acc↑	F1↑	Exist.↑	Count↑	Pos.↑	Color↑	Total↑	$\text{C}_{S}$ ↓	$\text{C}_{I}$ ↓	Length
Dola [ 8 ]	83.1	80.2	180.1	127.4	119.3	154.6	594.1	57.0	15.2	97.5
VCD [ 18 ]	85.0	85.3	184.7	137.3	128.7	153.0	603.7	51.0	14.9	101.9
OPERA [ 11 ]	85.2	84.2	180.7	133.3	111.7	123.3	549.0	47.0	14.6	95.3
DOPRA [ 32 ]	84.3	84.6	185.7	138.3	120.7	133.0	577.7	46.3	13.8	96.1
HALC [ 4 ]	84.0	83.9	190.0	143.3	128.3	160.0	621.6	50.2	12.4	97.2

Table 1: Comparison with state-of-the-art hallucination mitigation methods on LLaVA-1.5-7B. The top-2 results are bolded and u nderlined, respectively.

	POPE	MME	AMBER-d	AMBER-g			CHAIR
Method	Acc↑	Total↑	Acc↑	CHAIR↓	Hal↓	Cog↓	$\text{C}_{S}$ ↓	$\text{C}_{I}$ ↓	F1↑
LLaVA-1.5-7B	85.2	638.3	71.6	7.5	35.9	4.3	51.4	15.2	75.8
+ACT	88.0 ( $\uparrow$ 2.8)	668.3 ( $\uparrow$ 30.0)	77.0 ( $\uparrow$ 5.4)	5.1 ( $\downarrow$ 2.4)	27.4 ( $\downarrow$ 8.5)	2.7 ( $\downarrow$ 1.6)	33.0 ( $\downarrow$ 18.4)	9.3 ( $\downarrow$ 5.9)	78.3 ( $\uparrow$ 2.5)
LLaVA-1.5-13B	83.9	636.7	68.9	7.2	34.2	3.8	52.6	13.9	77.5
+ACT	86.6 ( $\uparrow$ 2.7)	645.0 ( $\uparrow$ 8.3)	74.0 ( $\uparrow$ 5.1)	4.6 ( $\downarrow$ 2.6)	25.3 ( $\downarrow$ 8.9)	2.3 ( $\downarrow$ 1.5)	40.8 ( $\downarrow$ 11.8)	10.7 ( $\downarrow$ 3.2)	79.1 ( $\uparrow$ 1.6)
InternVL2-8B	85.6	701.7	71.2	8.3	68.2	8.1	49.6	10.3	73.9

Table 2: Evaluation results of the proposed ACT across different LVLM architectures.

	POPE	MME	AMBER-d	AMBER-g			CHAIR
Method	Acc↑	Total↑	Acc↑	CHAIR↓	Hal↓	Cog↓	$\text{CHAIR}_{S}$ ↓	$\text{CHAIR}_{I}$ ↓	F1↑
Baseline	85.2	638.3	71.6	7.5	35.9	4.3	51.4	15.2	75.8
+ SCA	85.2	638.3	71.6	7.5	33.1	4.1	46.8	13.8	76.5
+ VCE	88.0	668.3	77.0	5 .3	27.2	2 .8	3 5.4	9 .6	7 8.0
+ ACT	88.0	668.3	77.0	5.1	2 7.4	2.7	33.0	9.3	78.3

Table 3: Ablation of core modules in ACT.

	POPE	MME	AMBER-d	AMBER-g			CHAIR
Metric	Acc↑	Total↑	Acc↑	CHAIR↓	Hal↓	Cog↓	$\text{CHAIR}_{S}$ ↓	$\text{CHAIR}_{I}$ ↓	F1↑
Entropy Change	85.3	618.3	73.1	5.9	29.1	3.0	4 1.8	1 1.7	7 7.3
JSD	87.1	570.0	75.5	5.4	27.5	2 .5	46.8	12.8	77.1
SSIM	8 7.6	593.3	75.7	5.0	2 6.1	2.3	46.6	13.0	77.0
CoM Shift	87.3	6 35.0	7 6.2	6.0	29.1	2.8	51.0	14.2	76.1
STCS(Ours)	88.0	668.3	77.0	5 .1	27.4	2.7	33.0	9.3	78.3

Table 4: Comparison of different head profiling metrics in ACT.

Table 7

Panel of Figure 3 — Figure 3: Visualizations of ablation study results. (A) Impact of the guidance shift between dynamic and static heads. (B) Robustness to calibration set size and source domain. (C) Positional distribution of hallucinations in CHAIR evaluation.

Figure 4: Effect of the guidance scale of ACT on discriminative and generative tasks.

Figure 5: Qualitative comparison of visual attention maps between the baseline and our proposed ACT on LLaVA-1.5-7B.

Attention Intervention in VLMs

See all 15 papers

Beyond Visual Enhancement: Adaptive Multi-Context Steering to Mitigate LVLM Hallucinations
Oct 8, 2026Shuran Ma, JiaLe Li, Yuxin Dong +9VLM Hallucination Mitigation Large Vision-Language Models
Shanghai Jiao Tong University · Peking University · Beijing University of Chemical Technology +3
See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs
Jun 29, 2026Yuqing Lei, Wenbo Lyu, Yingjun Du +3Visual Attention Large Vision-Language Models
University of Chinese Academy of Sciences · University of Amsterdam · United Imaging Healthcare Co., Ltd.
AdaIAT: Adaptively Increasing Attention to Generated Text to Alleviate Hallucinations in LVLM
Mar 5, 2026Li'an Zhong, Ziqiang He, Jibin Zheng +3Attention Intervention in VLMs VLM Hallucination Mitigation
Sun Yat-Sen University · Foshan University · University of British Columbia

Oct 8, 2026J/K move · Enter open · S save

Beyond Visual Enhancement: Adaptive Multi-Context Steering to Mitigate LVLM Hallucinations

Shuran Ma, JiaLe Li, Yuxin Dong, Shan Zheng+8

Shanghai Jiao Tong University · Peking University · Beijing University of Chemical Technology+3

VLM Hallucination Mitigation Large Vision-Language Models Object Hallucination in VLMs Representation Steering

Open paper AskPDF arXiv

Hallucination remains a significant challenge in Large Vision-Language Models (LVLMs). Existing training-free methods generally mitigate hallucinations through contrastive decoding or visual enhancement, often increasing the relative influence of visual evidence during generation. This raises a fundamental question: Can LVLMs dynamically regulate the contributions of different context sources to suppress hallucinations? In this work, we investigate and quantify how LVLMs coordinate multiple context sources during decoding and examine how this intrinsic behavior can guide hallucination mitigation. We find that LVLMs exhibit an intrinsic vision-attending tendency that can guide adaptive visual steering, while textual contexts can also contribute to hallucination mitigation. Motivated by these findings, we propose AIMS (Adaptive Information Multi-source Steering), a lightweight training-free framework that adaptively coordinates visual, prefilled textual, and generated contexts during decoding. Specifically, AIMS constructs compact prototypes for the three context domains and estimates their affinities with the current query to determine head-wise steering weights. The resulting multi-source steering direction is applied to the query representation, enabling adaptive context integration without additional model training or auxiliary forward passes. Extensive experiments across multiple LVLMs and decoding strategies demonstrate that AIMS effectively mitigates object hallucination while maintaining competitive general-purpose multimodal capabilities.

Abstract

Attention Intervention in VLMs

Explore similar work

Beyond Visual Enhancement: Adaptive Multi-Context Steering to Mitigate LVLM Hallucinations

See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs

AdaIAT: Adaptively Increasing Attention to Generated Text to Alleviate Hallucinations in LVLM