Beyond Visual Enhancement: Adaptive Multi-Context Steering to Mitigate LVLM Hallucinations

Shuran Ma; JiaLe Li; Yuxin Dong; Shan Zheng; Qingyun Jiang; Xiang Chen; Qi Zhu; Deyi Ji

cs.CVOct 8, 2026

Beyond Visual Enhancement: Adaptive Multi-Context Steering to Mitigate LVLM Hallucinations

Authors: Shuran Ma, JiaLe Li, Yuxin Dong, Shan Zheng, Qingyun Jiang, Xiang Chen, Qi Zhu, Deyi Ji, +4 more

Organizations: Shanghai Jiao Tong University · Peking University · Beijing University of Chemical Technology · Nanjing University of Aeronautics and Astronautics · University of Science and Technology of China · Tsinghua University

Abstract

Hallucination remains a significant challenge in Large Vision-Language Models (LVLMs). Existing training-free methods generally mitigate hallucinations through contrastive decoding or visual enhancement, often increasing the relative influence of visual evidence during generation. This raises a fundamental question: Can LVLMs dynamically regulate the contributions of different context sources to suppress hallucinations? In this work, we investigate and quantify how LVLMs coordinate multiple context sources during decoding and examine how this intrinsic behavior can guide hallucination mitigation. We find that LVLMs exhibit an intrinsic vision-attending tendency that can guide adaptive visual steering, while textual contexts can also contribute to hallucination mitigation. Motivated by these findings, we propose AIMS (Adaptive Information Multi-source Steering), a lightweight training-free framework that adaptively coordinates visual, prefilled textual, and generated contexts during decoding. Specifically, AIMS constructs compact prototypes for the three context domains and estimates their affinities with the current query to determine head-wise steering weights. The resulting multi-source steering direction is applied to the query representation, enabling adaptive context integration without additional model training or auxiliary forward passes. Extensive experiments across multiple LVLMs and decoding strategies demonstrate that AIMS effectively mitigates object hallucination while maintaining competitive general-purpose multimodal capabilities.

Figures & tables

Figure 1 : Comparison of training-free hallucination mitigation methods for LVLMs. From left to right: contrastive decoding, visual enhancement, and our method that dynamically steers the query toward three context branches during generation.

Figure 2 : Visual attention dynamics during decoding. Results on 500 MSCOCO examples using Qwen2.5-VL-3B, grouped into five sequence-length bins. $T1$ and $T2$ mark the onset of visual description and semantic summarization, respectively.

Method	$C_{S}\downarrow$	$C_{I}\downarrow$	Recall $\uparrow$	F1 $\uparrow$	Length $\uparrow$
Baseline	56.0	9.96	74.05	74.10	404.19
$\text{Fixed-V}_{\alpha=0.05}$	51.6	9.48	74.11	74.99	350.88
$\text{Fixed-V}_{\alpha=0.15}$	39.0	9.30	70.18	75.29	199.70
Adaptive-V	40.2	8.33	71.53	75.08	292.41

Table 1 : Preliminary analysis of adaptive and multi-context steering. (a) Comparison of fixed and adaptive query steering toward the vision prototype. (b) Impact of different context branches under fixed-strength steering.

C_{S}

and

C_{I}

denote CHAIR S and CHAIR I , respectively.

Figure 3 : Overview of AIMS. Given vision, prefilled text, and generated-token key caches, AIMS constructs head-wise multi-context prototypes and adaptively weights them according to their affinity with the current query. The weighted prototypes form a dynamic steering target that adjusts the query representation before standard self-attention.

Method	LLaVA-1.5-7B
	Greedy					Beam Search					Nucleus
	$C_{S}$ $\downarrow$	$C_{I}$ $\downarrow$	R $\uparrow$	F1 $\uparrow$	Len $\uparrow$	$C_{S}$ $\downarrow$	$C_{I}$ $\downarrow$	R $\uparrow$	F1 $\uparrow$	Len $\uparrow$	$C_{S}$ $\downarrow$	$C_{I}$ $\downarrow$	R $\uparrow$	F1 $\uparrow$	Len $\uparrow$
Baseline	51.4	14.70	79.42	76.80	97.9	55.2	14.93	80.34	77.10	103.5	50.0	15.31	74.05	73.19	101.5
MemVR	49.8	14.20	78.64	76.81	93.8	53.0	14.30	79.54	77.53	95.7	52.6	15.80	72.91	72.73	93.6
OPERA	–	–	–	–	–	44.6	12.72	79.25	78.40	92.7	–	–	–	–	–
VCD	–	–	–	–	–	–	–	–	–	–	49.9	15.72	75.24	74.00	94.6

Table 2 : Evaluation results of different methods on CHAIR.

Method	Qwen2.5-VL-3B
	Greedy				Beam Search				Nucleus
	CHAIR $\downarrow$	Cover $\uparrow$	Hal $\downarrow$	Cog $\downarrow$	CHAIR $\downarrow$	Cover $\uparrow$	Hal $\downarrow$	Cog $\downarrow$	CHAIR $\downarrow$	Cover $\uparrow$	Hal $\downarrow$	Cog $\downarrow$
Baseline	8.2	69.6	52.7	5.7	6.8	68.3	44.7	4.9	9.9	70.3	63.7	7.4
VTI	6.9	65.0	43.8	4.3	5.8	63.8	35.1	3.9	8.5	65.9	51.6	5.4
AIMS	6.6	64.8	39.7	3.8	5.2	64.0	33.2	3.1	7.8	65.2	47.9	4.2
Method	Qwen3.5-9B

Table 3 : Evaluation results of different methods on AMBER-G.

Method	Qwen2.5-VL-3B
	Greedy			Beam Search			Nucleus
	Per $\uparrow$	Cog $\uparrow$	Sum $\uparrow$	Per $\uparrow$	Cog $\uparrow$	Sum $\uparrow$	Per $\uparrow$	Cog $\uparrow$	Sum $\uparrow$
Baseline	1591.38	612.86	2204.24	1558.17	537.14	2095.31	1466.52	513.93	1980.45
MemVR	1591.93	626.42	2218.36	1538.66	544.64	2083.30	1425.96	523.92	1949.88
VCD	–	–	–	–	–	–	1485.20	521.78	2006.98
VTI	1472.86	522.50	1995.36	1520.20	480.35	2000.55	1448.89	518.57	1967.46

Table 5 : Evaluation results of different methods on MME.

Method	Greedy					Beam Search					Nucleus
Method	$C_{S}$ $\downarrow$	$C_{I}$ $\downarrow$	R $\uparrow$	F1 $\uparrow$	Len $\uparrow$	$C_{S}$ $\downarrow$	$C_{I}$ $\downarrow$	R $\uparrow$	F1 $\uparrow$	Len $\uparrow$	$C_{S}$ $\downarrow$	$C_{I}$ $\downarrow$	R $\uparrow$	F1 $\uparrow$	Len $\uparrow$
AIMS RBF	42.8	8.85	71.51	75.31	301.5	38.4	7.87	73.67	77.40	311.2	51.0	9.74	70.56	72.94	339.9
AIMS NCS	40.2	8.33	71.53	75.08	292.4	38.4	7.84	72.84	77.02	325.9	45.6	9.64	70.75	73.74	335.3
AIMS KL	46.9	8.48	71.40	74.27	344.7	36.2	7.57	72.84	77.41	314.7	51.8	9.43	71.83	72.54	399.6
AIMS	40.8	9.07	71.69	74.66	284.2	38.2	7.67	71.84	76.27	325.2	44.8	9.33	70.56	73.62	327.6

Table 6 : Ablation of different affinity weighting functions on Qwen2.5-VL-3B.

Figure 5 : Ablation study of different context branches in AIMS.

Method	$\bm{\alpha}$	$(\bm{\tau_{V}},\bm{\tau_{P}},\bm{\tau_{G}})$	$C_{S}$ $\downarrow$	$C_{I}$ $\downarrow$	F1 $\uparrow$
AIMS	0.08	(1.5, 1.2, 1.1)	40.8	9.07	74.66
	0.05	(1.5, 1.2, 1.1)	49.8	8.95	75.51
	0.08	(1.0, 1.0, 1.0)	44.4	7.59	75.22

Table 8 : Ablation of

\alpha

and

\tau

Method	Peak Mem. (MB) $\downarrow$	Time (s/token) $\downarrow$
Baseline	7235.87	0.0437
MemVR	7239.53	0.0698
VCD	7387.08	0.0905
VTI	7482.31	0.2424
SID	7759.74	0.1075
AIMS	7236.11	0.0705

Table 9 : Comparison of computational overhead on Qwen2.5-VL-3B.

Figure 7 : Case study 1 on the CHAIR benchmark using Qwen3.5-9B with greedy decoding.

Figure 8 : Case study 2 on the CHAIR benchmark using Qwen3.5-9B with greedy decoding.

Figure 9 : Case study 1 on the CHAIR benchmark using Qwen2.5-VL-3B with greedy decoding.

Figure 10 : Case study 2 on the CHAIR benchmark using Qwen2.5-VL-3B with greedy decoding.

Figure 11 : Case study on the AMBER-G benchmark using Qwen2.5-VL-3B with greedy decoding.

Panel of Figure 12 — Figure 12 : Qualitative examples on MME using Qwen3.5-9B with nucleus sampling.

Explore similar work

See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs
Jun 29, 2026Yuqing Lei, Wenbo Lyu, Yingjun Du +3Visual Attention Large Vision-Language Models
University of Chinese Academy of Sciences · University of Amsterdam · United Imaging Healthcare Co., Ltd.
Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation
Jun 2, 2026Ruipeng Zhang, Zhihao Li, C. L. Philip Chen +1VLM Hallucination Large Vision-Language Models
Guangdong Provincial Key Laboratory of Computational AI Models and Cognitive Intelligence, School of Computer Science & Engineering, South China University of Technology, Guangzhou, China · Engineering Research Center of the Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human, Guangzhou, China · Pazhou Lab, Guangzhou, China.
Look Clearly Before Answering: Mitigating Hallucinations in LVLMs via Saliency-Driven Perceptual Realignment
Jul 18, 2026Pengxu Chen, Yao Zhu, Guangming Zhu +4Large Vision-Language Models VLM Hallucination Mitigation
Xidian University Xi’an, China · Tsinghua University Beijing, China · Shanghai Road Transport Development Center Shanghai, China +1

Jun 29, 2026, cs.CVJ/K move · Enter open · S save

See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs

Yuqing Lei, Wenbo Lyu, Yingjun Du, Xiantong Zhen+2

University of Chinese Academy of Sciences · University of Amsterdam · United Imaging Healthcare Co., Ltd.

Visual Attention Large Vision-Language Models Object Hallucination in VLMs VLM Hallucination Mitigation

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) excel at multimodal tasks but remain prone to object hallucinations. Prior training-free remedies often uniformly strengthen visual signals, which may also amplify irrelevant regions and introduce spurious evidence, harming fluency. We propose Context-aware Attention Intervention (CAI), a training-free inference-time mechanism that enforces a see only when needed principle via two-axis selectivity: where to look and when to intervene. At each decoding step, CAI derives token-specific visual relevance from early-layer representations to localize semantically aligned regions, and applies a conservative, entropy- and depth-gated attention tilt only for uncertainty-spiking tokens in deeper layers where visual grounding degrades, leaving confident tokens and irrelevant regions largely unchanged. This targeted intervention strengthens visual grounding while preserving linguistic fluency, and it yields consistent improvements even without contrastive decoding, which remains optional as an auxiliary bias-suppression module. Extensive experiments across multiple LVLM backbones and benchmarks show that CAI achieves state-of-the-art hallucination mitigation, and our analysis characterizes CAI as a KL-minimal attention reweighting with bounded interference under inactive gates or small tilts. Code is available at https://github.com/Iris1946/CAI.

Abstract

Explore similar work

See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs

Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation

Look Clearly Before Answering: Mitigating Hallucinations in LVLMs via Saliency-Driven Perceptual Realignment