Beyond Visual Enhancement: Adaptive Multi-Context Steering to Mitigate LVLM Hallucinations

Shuran Ma; JiaLe Li; Yuxin Dong; Shan Zheng; Qingyun Jiang; Xiang Chen; Qi Zhu; Deyi Ji

cs.CVOct 8, 2026

Beyond Visual Enhancement: Adaptive Multi-Context Steering to Mitigate LVLM Hallucinations

Authors: Shuran Ma, JiaLe Li, Yuxin Dong, Shan Zheng, Qingyun Jiang, Xiang Chen, Qi Zhu, Deyi Ji, +4 more

Organizations: Shanghai Jiao Tong University · Peking University · Beijing University of Chemical Technology · Nanjing University of Aeronautics and Astronautics · University of Science and Technology of China · Tsinghua University

Abstract

Hallucination remains a significant challenge in Large Vision-Language Models (LVLMs). Existing training-free methods generally mitigate hallucinations through contrastive decoding or visual enhancement, often increasing the relative influence of visual evidence during generation. This raises a fundamental question: Can LVLMs dynamically regulate the contributions of different context sources to suppress hallucinations? In this work, we investigate and quantify how LVLMs coordinate multiple context sources during decoding and examine how this intrinsic behavior can guide hallucination mitigation. We find that LVLMs exhibit an intrinsic vision-attending tendency that can guide adaptive visual steering, while textual contexts can also contribute to hallucination mitigation. Motivated by these findings, we propose AIMS (Adaptive Information Multi-source Steering), a lightweight training-free framework that adaptively coordinates visual, prefilled textual, and generated contexts during decoding. Specifically, AIMS constructs compact prototypes for the three context domains and estimates their affinities with the current query to determine head-wise steering weights. The resulting multi-source steering direction is applied to the query representation, enabling adaptive context integration without additional model training or auxiliary forward passes. Extensive experiments across multiple LVLMs and decoding strategies demonstrate that AIMS effectively mitigates object hallucination while maintaining competitive general-purpose multimodal capabilities.

Figures & tables

Figure 1 : Comparison of training-free hallucination mitigation methods for LVLMs. From left to right: contrastive decoding, visual enhancement, and our method that dynamically steers the query toward three context branches during generation.

Figure 2 : Visual attention dynamics during decoding. Results on 500 MSCOCO examples using Qwen2.5-VL-3B, grouped into five sequence-length bins. $T1$ and $T2$ mark the onset of visual description and semantic summarization, respectively.

Method	$C_{S}\downarrow$	$C_{I}\downarrow$	Recall $\uparrow$	F1 $\uparrow$	Length $\uparrow$
Baseline	56.0	9.96	74.05	74.10	404.19
$\text{Fixed-V}_{\alpha=0.05}$	51.6	9.48	74.11	74.99	350.88
$\text{Fixed-V}_{\alpha=0.15}$	39.0	9.30	70.18	75.29	199.70
Adaptive-V	40.2	8.33	71.53	75.08	292.41

Table 1 : Preliminary analysis of adaptive and multi-context steering. (a) Comparison of fixed and adaptive query steering toward the vision prototype. (b) Impact of different context branches under fixed-strength steering.

C_{S}

and

C_{I}

denote CHAIR S and CHAIR I , respectively.

Figure 3 : Overview of AIMS. Given vision, prefilled text, and generated-token key caches, AIMS constructs head-wise multi-context prototypes and adaptively weights them according to their affinity with the current query. The weighted prototypes form a dynamic steering target that adjusts the query representation before standard self-attention.

Method	LLaVA-1.5-7B
	Greedy					Beam Search					Nucleus
	$C_{S}$ $\downarrow$	$C_{I}$ $\downarrow$	R $\uparrow$	F1 $\uparrow$	Len $\uparrow$	$C_{S}$ $\downarrow$	$C_{I}$ $\downarrow$	R $\uparrow$	F1 $\uparrow$	Len $\uparrow$	$C_{S}$ $\downarrow$	$C_{I}$ $\downarrow$	R $\uparrow$	F1 $\uparrow$	Len $\uparrow$
Baseline	51.4	14.70	79.42	76.80	97.9	55.2	14.93	80.34	77.10	103.5	50.0	15.31	74.05	73.19	101.5
MemVR	49.8	14.20	78.64	76.81	93.8	53.0	14.30	79.54	77.53	95.7	52.6	15.80	72.91	72.73	93.6
OPERA	–	–	–	–	–	44.6	12.72	79.25	78.40	92.7	–	–	–	–	–
VCD	–	–	–	–	–	–	–	–	–	–	49.9	15.72	75.24	74.00	94.6

Table 2 : Evaluation results of different methods on CHAIR.

Method	Qwen2.5-VL-3B
	Greedy				Beam Search				Nucleus
	CHAIR $\downarrow$	Cover $\uparrow$	Hal $\downarrow$	Cog $\downarrow$	CHAIR $\downarrow$	Cover $\uparrow$	Hal $\downarrow$	Cog $\downarrow$	CHAIR $\downarrow$	Cover $\uparrow$	Hal $\downarrow$	Cog $\downarrow$
Baseline	8.2	69.6	52.7	5.7	6.8	68.3	44.7	4.9	9.9	70.3	63.7	7.4
VTI	6.9	65.0	43.8	4.3	5.8	63.8	35.1	3.9	8.5	65.9	51.6	5.4
AIMS	6.6	64.8	39.7	3.8	5.2	64.0	33.2	3.1	7.8	65.2	47.9	4.2
Method	Qwen3.5-9B

Table 3 : Evaluation results of different methods on AMBER-G.

Method	Qwen2.5-VL-3B
	Greedy			Beam Search			Nucleus
	Per $\uparrow$	Cog $\uparrow$	Sum $\uparrow$	Per $\uparrow$	Cog $\uparrow$	Sum $\uparrow$	Per $\uparrow$	Cog $\uparrow$	Sum $\uparrow$
Baseline	1591.38	612.86	2204.24	1558.17	537.14	2095.31	1466.52	513.93	1980.45
MemVR	1591.93	626.42	2218.36	1538.66	544.64	2083.30	1425.96	523.92	1949.88
VCD	–	–	–	–	–	–	1485.20	521.78	2006.98
VTI	1472.86	522.50	1995.36	1520.20	480.35	2000.55	1448.89	518.57	1967.46

Table 5 : Evaluation results of different methods on MME.

Method	Greedy					Beam Search					Nucleus
Method	$C_{S}$ $\downarrow$	$C_{I}$ $\downarrow$	R $\uparrow$	F1 $\uparrow$	Len $\uparrow$	$C_{S}$ $\downarrow$	$C_{I}$ $\downarrow$	R $\uparrow$	F1 $\uparrow$	Len $\uparrow$	$C_{S}$ $\downarrow$	$C_{I}$ $\downarrow$	R $\uparrow$	F1 $\uparrow$	Len $\uparrow$
AIMS RBF	42.8	8.85	71.51	75.31	301.5	38.4	7.87	73.67	77.40	311.2	51.0	9.74	70.56	72.94	339.9
AIMS NCS	40.2	8.33	71.53	75.08	292.4	38.4	7.84	72.84	77.02	325.9	45.6	9.64	70.75	73.74	335.3
AIMS KL	46.9	8.48	71.40	74.27	344.7	36.2	7.57	72.84	77.41	314.7	51.8	9.43	71.83	72.54	399.6
AIMS	40.8	9.07	71.69	74.66	284.2	38.2	7.67	71.84	76.27	325.2	44.8	9.33	70.56	73.62	327.6

Table 6 : Ablation of different affinity weighting functions on Qwen2.5-VL-3B.

Figure 5 : Ablation study of different context branches in AIMS.

Method	$\bm{\alpha}$	$(\bm{\tau_{V}},\bm{\tau_{P}},\bm{\tau_{G}})$	$C_{S}$ $\downarrow$	$C_{I}$ $\downarrow$	F1 $\uparrow$
AIMS	0.08	(1.5, 1.2, 1.1)	40.8	9.07	74.66
	0.05	(1.5, 1.2, 1.1)	49.8	8.95	75.51
	0.08	(1.0, 1.0, 1.0)	44.4	7.59	75.22

Table 8 : Ablation of

\alpha

and

\tau

Method	Peak Mem. (MB) $\downarrow$	Time (s/token) $\downarrow$
Baseline	7235.87	0.0437
MemVR	7239.53	0.0698
VCD	7387.08	0.0905
VTI	7482.31	0.2424
SID	7759.74	0.1075
AIMS	7236.11	0.0705

Table 9 : Comparison of computational overhead on Qwen2.5-VL-3B.

Figure 7 : Case study 1 on the CHAIR benchmark using Qwen3.5-9B with greedy decoding.

Figure 8 : Case study 2 on the CHAIR benchmark using Qwen3.5-9B with greedy decoding.

Figure 9 : Case study 1 on the CHAIR benchmark using Qwen2.5-VL-3B with greedy decoding.

Figure 10 : Case study 2 on the CHAIR benchmark using Qwen2.5-VL-3B with greedy decoding.

Figure 11 : Case study on the AMBER-G benchmark using Qwen2.5-VL-3B with greedy decoding.

Panel of Figure 12 — Figure 12 : Qualitative examples on MME using Qwen3.5-9B with nucleus sampling.

Explore similar work

Jun 29, 2026cs.CV

See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs

Large Vision-Language Models (LVLMs) excel at multimodal tasks but remain prone to object hallucinations. Prior training-free remedies often uniformly strengthen visual signals, which may also amplify irrelevant regions and introduce spurious evidence, harming fluency. We propose Context-aware Attention Intervention (CAI), a training-free inference-time mechanism that enforces a see only when needed principle via two-axis selectivity: where to look and when to intervene. At each decoding step, CAI derives token-specific visual relevance from early-layer representations to localize semantically aligned regions, and applies a conservative, entropy- and depth-gated attention tilt only for uncertainty-spiking tokens in deeper layers where visual grounding degrades, leaving confident tokens and irrelevant regions largely unchanged. This targeted intervention strengthens visual grounding while preserving linguistic fluency, and it yields consistent improvements even without contrastive decoding, which remains optional as an auxiliary bias-suppression module. Extensive experiments across multiple LVLM backbones and benchmarks show that CAI achieves state-of-the-art hallucination mitigation, and our analysis characterizes CAI as a KL-minimal attention reweighting with bounded interference under inactive gates or small tilts. Code is available at https://github.com/Iris1946/CAI.

Yuqing Lei, Wenbo Lyu, Yingjun Du +3

University of Chinese Academy of Sciences · University of Amsterdam · United Imaging Healthcare Co., Ltd.

Visual Attention Large Vision-Language Models Object Hallucination in VLMs VLM Hallucination Mitigation

Ask PDF

Jun 2, 2026cs.CV

Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation

Large vision language models (LVLMs) have made rapid advancements and are deployed across various applications, yet hallucinations remain a major challenge. Activation steering is appealing due to its minimal training overhead and controllability at inference time. However, we found that during autoregressive decoding, visual conditioning affects token prediction sparsely and locally across decoding steps, and many existing methods that average image-versus-no-image differences over the entire sequence dilute these critical signals, yielding low signal-to-noise ratio steering directions. Additionally, many existing methods apply a fixed steering strength, which misallocates the intervention budget, over-perturbs non-critical tokens, and can cause instability. To address these limitations, we propose Token-Level Visual-Sensitivity Steering (TLVS) for hallucination mitigation. Our approach first extracts token-level steering vectors and refines them, and then applies fine-grained, visual-sensitivity-adaptive steering only where it matters. This lightweight, plug-and-play mechanism requires only minimal training for calibration and can be applied across diverse vision-language models. It modulates the steering strength at each decoding step, selectively suppressing hallucination-prone spans while preserving evidence-grounded content. We evaluate TLVS on several benchmarks, including POPE, AMBER, CHAIR (COCO), MMHal, and HallusionBench, demonstrating consistent improvements over previous steering methods.

Ruipeng Zhang, Zhihao Li, C. L. Philip Chen +1

Guangdong Provincial Key Laboratory of Computational AI Models and Cognitive Intelligence, School of Computer Science & Engineering, South China University of Technology, Guangzhou, China · Engineering Research Center of the Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human, Guangzhou, China · Pazhou Lab, Guangzhou, China.

VLM Hallucination Large Vision-Language Models Activation Steering VLM Hallucination Mitigation

Ask PDF

Jul 18, 2026cs.CV

Look Clearly Before Answering: Mitigating Hallucinations in LVLMs via Saliency-Driven Perceptual Realignment

Large vision-language models (LVLMs) have demonstrated remarkable capabilities in multimodal understanding. However, they remain prone to hallucinations, generating responses that are inconsistent with the visual evidence. Existing mitigation methods largely address language-prior bias or cross-modal imbalance, while progressive visual degradation across perception and memory remains underexplored. In this work, we propose Saliency-Driven Perceptual Realignment (SDPR), a training-free framework that mitigates the degradation of visual awareness throughout inference. Specifically, we first introduce saliency-driven attention redistribution to release attention hijacked by non-semantic sink tokens, thereby recovering critical visual evidence. Second, we identify spatial distortion in the KV cache and propose saliency-driven cache alignment to preserve query-relevant visual features during generation. Finally, we introduce prior-constrained contrastive decoding to penalize unfaithful predictions induced by dominant language priors. Our proposed SDPR is robust against hallucinations due to its holistic alignment of visual awareness across the entire generative trajectory. Extensive experiments across diverse LVLM architectures show that SDPR outperforms state-of-the-art methods on both hallucination and general-purpose benchmarks, requiring no additional training and incurring minimal runtime overhead. The code is available \href{https://github.com/PengSyuChen/SDPR}{\color{blue}{here}}.

Pengxu Chen, Yao Zhu, Guangming Zhu +4

Xidian University Xi’an, China · Tsinghua University Beijing, China · Shanghai Road Transport Development Center Shanghai, China +1

Large Vision-Language Models VLM Hallucination Mitigation

Ask PDF