Locate-then-Sparsify: Attribution Guided Sparse Strategy for Visual Hallucination Mitigation

Tiantian Dang; Chao Bi; Shufan Shen; Jinzhe Liu; Qingming Huang; Shuhui Wang

cs.CVMar 17, 2026

Locate-then-Sparsify: Attribution Guided Sparse Strategy for Visual Hallucination Mitigation

Authors: Tiantian Dang, Chao Bi, Shufan Shen, Jinzhe Liu, Qingming Huang, Shuhui Wang

Organizations: State Key Lab. of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · School of Computer Science and Technology, University of Chinese Academy of Sciences

Abstract

Despite the significant advancements in Large Vision-Language Models (LVLMs), their tendency to generate hallucinations undermines reliability and restricts broader practical deployment. Among the hallucination mitigation methods, feature steering emerges as a promising approach that reduces erroneous outputs in LVLMs without increasing inference costs. However, current methods apply uniform feature steering across all layers. This heuristic strategy ignores inter-layer differences, potentially disrupting layers unrelated to hallucinations and ultimately leading to performance degradation on general tasks. In this paper, we propose Locate-Then-Sparsify for Feature Steering (LTS-FS), a plug-and-play framework which controls the steering intensity according to the hallucination relevance of each layer. We first construct a dataset comprising token-level and sentence-level hallucination cases. Based on this dataset, we introduce an attribution method based on causal interventions to quantify the hallucination relevance of each layer. With the attribution scores across layers, we propose a layerwise strategy that converts these scores into feature steering intensities for individual layers, enabling more precise adjustments specifically on hallucination-relevant layers. Extensive experiments across multiple LVLMs and benchmarks demonstrate that LTS-FS effectively mitigates hallucination while preserving strong performance. Codes are available at https://github.com/huttersadan/LTS-FS.

Figures & tables

Panel of Figure 1 — Figure 1: Current methods ( e.g. , Nullu [ 40 ] ) mitigate hallucinations by uniformly steering features across layers, which (a) alters feature distributions and (b) In contrast, we propose a layerwise steering framework LTS-FS, which mitigates hallucinations more effectively ( e.g. , on CHAIR) while minimally perturbing the feature distributions, preserving more generalization ability.

Figure 2: Hallucination examples at token level and sentence level.

Figure 3: Overview of our LTS-FS framework. First, we build a bi-granularity dataset with token-level and sentence-level hallucinations. Then, using the dataset, hallucination-relevant layers are identified via intervention-based attribution. Finally, a layerwise strategy is applied to control the feature steering intensity across layers according to the attribution scores.

Method	LLaVA-v1.5-7B				LLaVA-v1.5-13B				Qwen-VL2.5-7B
Method	C S $\downarrow$	C I $\downarrow$	Recall	Len.	C S $\downarrow$	C I $\downarrow$	Recall	Len.	C S $\downarrow$	C I $\downarrow$	Recall	Len.
Regular	53.0	13.9	77.2	98.0	40.8	9.5	77.2	111.8	27.0	7.4	61.6	120.6
VCD	55.2	16.7	77.5	89.2	39.2	9.2	79.1	108.2	26.2	7.6	61.2	120.3
AGLA	50.8	16.1	75.2	88.1	38.4	9.1	78.7	109.3	25.2	7.1	59.5	118.6
$\text{Nullu}^{*}$	50.2	13.7	76.9	93.3	38.0	9.4	74.5	105.8	27.4	7.7	60.7	121.6
$\text{VTI}^{*}$	47.4	13.9	76.2	88.9	36.3	9.2	75.9	94.4	25.5	7.1	61.6	121.3

Table 1: CHAIR results of various LVLMs on MSCOCO. Bold indicates the best performance. C S and C I mean lower hallucination. Recall and output length (Len.) serve as controls, indicating that reductions in C S /C I do not stem from suppressing objects or truncating responses. ∗ denotes the feature steering methods.

Method	LLaVA-v1.5-7B			LLaVA-v1.5-13B			Qwen-VL2.5-7B
Method	Popular	Random	Adversarial	Popular	Random	Adversarial	Popular	Random	Adversarial
Regular	77.52	85.37	70.13	78.40	81.91	71.07	83.31	85.32	80.17
VCD	79.09	86.55	71.48	79.38	82.27	71.73	83.19	85.94	80.56
AGLA	78.67	85.32	71.63	80.11	82.64	72.27	83.34	86.02	80.92
$\text{Nullu}^{*}$	79.42	86.35	71.57	80.88	83.24	72.43	83.06	85.82	80.74
$\text{VTI}^{*}$	77.03	84.84	69.40	79.22	84.08	71.77	82.74	85.49	80.19

Table 2: Comparison of the average accuracy under different settings (i.e., Random, Popular, Adversarial) with different baselines and our framework on POPE Bold indicates the best results, and underline means the second best. ∗ denotes the feature steering methods.

Method	LLaVA-v1.5-7B			LLaVA-v1.5-13B			Qwen-VL2.5-7B
Method	Popular	Random	Adversarial	Popular	Random	Adversarial	Popular	Random	Adversarial
Regular	80.71	86.47	75.85	81.30	83.85	76.47	81.68	83.54	78.93
VCD	81.23	87.16	76.04	82.01	83.76	75.76	81.95	83.88	79.51
AGLA	81.47	86.77	75.89	82.32	83.58	75.48	81.86	83.63	79.14
$\text{Nullu}^{*}$	81.67	86.28	76.17	82.97	84.73	77.04	81.27	83.73	79.32
$\text{VTI}^{*}$	80.40	86.08	75.42	81.83	83.82	76.80	80.88	83.37	78.70

Table 3: Comparison of the average F1 score under different settings (i.e., Random, Popular, Adversarial) with different baselines and our framework on POPE. Bold indicates the best results, and underline is the second best. ∗ denotes the feature steering methods.

Hallucination Level	C S	C I	POPE acc	POPE f1
Baseline (Nullu)	50.2	13.7	79.11	81.37
Token-level only	50.0	13.4	79.59	81.85
Sentence-level only	47.3	13.0	79.33	81.58
Both levels	46.8	13.5	79.92	82.02

Table 4: Ablation results for different levels of hallucination.

$r_{s}$	CHAIR S	CHAIR I	Recall	Length
0.0 (Regular)	53.0	13.9	77.2	98.0
0.3	49.5	14.2	76.2	95.7
0.5	46.8	13.5	76.6	93.2
0.7	47.6	13.0	76.6	97.0
0.9	49.1	13.3	75.7	96.8
Soft Gating	46.7	13.4	76.1	94.5

Table 5: Ablation results for different choices of

r_{s}

Panel of Figure 5 — Figure 5: Demonstration of our framework for hallucination mitigation on two examples of LLaVA-Bench using LLaVA-v1.5-7B.

Model	Method	Accuracy $\uparrow$	Detailedness $\uparrow$
LLaVA-1.5	Original	5.74	5.23
LLaVA-1.5	Nullu	6.46	5.51
	LTS-FS(Nullu)	6.96	6.23
Qwen-vl2.5	Original	6.06	5.54
Qwen-vl2.5	Nullu	6.37	5.68
	LTS-FS(Nullu)	6.59	6.07

Table 6: GPT-4V-aided evaluation of LLava-Bench.

Method	CHAIRs	POPE acc	details	CLAIR
Original	53.0	77.63	5.23	80.03
nullu	50.2	79.11	5.51	75.00
LTS-FS(nullu)	46.8	79.92	6.23	82.74
Soft Gating	46.7	79.5	6.26	83.64

Table 7: Trade-off between hallucination mitigation and general capability preservation.

Method	Existence	Count	Position	Color	Posters	Celebrity	Scene	Landmark	Artwork	OCR	Total
Regular	182	118	105	151	118	112	145	131	108	78	1248
Nullu	190	122	106	157	128	118	148	130	114	121	1334
LTS-FS(Nullu)	195	153	128	157	130	127	155	131	113	123	1412

Table 8: Results on all MME perception-related tasks.

Model	Method	Common Sense Reasoning	Numerical Calculation	Text Translation	Code Reasoning	Total
LLaVA-1.5-7B	Regular	110	50	50	71	281
	Nullu	113	59	75	77	324
	LTS-FS + Nullu	120	59	75	80	334

Table 9: Results on all MME recognition-related tasks.

Method	Preparation Cost	Inference Cost
Regular	–	1.31s
VCD	0s	3.14s
Nullu	30mins	1.37s
Ours	90mins	1.34s

Table 10: Time analysis comparison of different hallucination mitigation strategies. VCD represents a decoding-based method. Nullu represents a feature-steering-based method.

Settings	C S	C S	Recall	Length
Regular	53.0	13.9	77.2	98.0
w/o HI	52.0	14.0	76.9	97.4
w/o CI	48.2	13.6	77.1	95.7
w/o PI	47.6	13.7	76.9	94.3
LTS-FS(Nullu)	46.8	13.5	76.6	93.2

Table 11: Ablation study of indicators. HI, CI, PI respectively indicate hallucination indicator, cue indicator, and position indicator.

Settings	C S	C I	Acc	F1
Regular	53.0	13.9	75.47	79.83
MSCOCO→GQA	—	—	77.31	79.57
GQA→MSCOCO	49.5	13.2	—	—
Antidote→GQA	—	—	77.28	80.12
Antidote→MSCOCO	49.8	13.7	—	—
LTS-FS(Nullu)	46.8	13.5	77.15	80.63

Table 12: Results of the generalization test of our framework. We use LLaVA-v1.5-7B to conduct this experiment. C S and C I is the CHAIR S and CHAIR I under CHAIR benchmark. ACC and F1 mean the accuracy and F1 score in the GQA subset on POPE.

Figure 6: A sample generation based on CHAIR benchmark

Setting	Model	Method	Accuracy $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 Score $\uparrow$
Random	LLaVA-v1.5-7B	Regular	85.37	80.77	93.22	86.47
		VCD	86.55	84.02	90.69	87.16
		AGLA	85.32	83.56	91.34	86.77
		Nullu	86.35	84.36	91.09	86.28
		VTI	84.84	80.02	93.36	86.08
		LTS-FS(Nullu)	87.13	84.69	91.02	87.64

Table 13: Average POPE results with Random and Popular.

VLM Hallucination

Vision-language model hallucination is the production of claims unsupported by or in conflict with visual evidence.

See all 71 papers

Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation
Jun 2, 2026Ruipeng Zhang, Zhihao Li, C. L. Philip Chen +1VLM Hallucination Large Vision-Language Models
Guangdong Provincial Key Laboratory of Computational AI Models and Cognitive Intelligence, School of Computer Science & Engineering, South China University of Technology, Guangzhou, China · Engineering Research Center of the Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human, Guangzhou, China · Pazhou Lab, Guangzhou, China.
See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs
Jun 29, 2026Yuqing Lei, Wenbo Lyu, Yingjun Du +3Visual Attention Large Vision-Language Models
University of Chinese Academy of Sciences · University of Amsterdam · United Imaging Healthcare Co., Ltd.
Mitigating Entangled Steering in Large Vision-Language Models for Hallucination Reduction
Apr 9, 2026Yuanhong Zhang, Zhaoyang Wang, Xin Zhang +2VLM Hallucination VLM Hallucination Mitigation
Ministry of Education Key Laboratory of Intelligent Networks and Network Security, Xi‘an Jiaotong University, Xi‘an, China · Centre for Frontier AI Research, Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore

Jun 2, 2026J/K move · Enter open · S save

Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation

Ruipeng Zhang, Zhihao Li, C. L. Philip Chen, Tong Zhang

Guangdong Provincial Key Laboratory of Computational AI Models and Cognitive Intelligence, School of Computer Science & Engineering, South China University of Technology, Guangzhou, China · Engineering Research Center of the Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human, Guangzhou, China · Pazhou Lab, Guangzhou, China.

VLM Hallucination Large Vision-Language Models Activation Steering VLM Hallucination Mitigation

Open paper AskPDF arXiv

Large vision language models (LVLMs) have made rapid advancements and are deployed across various applications, yet hallucinations remain a major challenge. Activation steering is appealing due to its minimal training overhead and controllability at inference time. However, we found that during autoregressive decoding, visual conditioning affects token prediction sparsely and locally across decoding steps, and many existing methods that average image-versus-no-image differences over the entire sequence dilute these critical signals, yielding low signal-to-noise ratio steering directions. Additionally, many existing methods apply a fixed steering strength, which misallocates the intervention budget, over-perturbs non-critical tokens, and can cause instability. To address these limitations, we propose Token-Level Visual-Sensitivity Steering (TLVS) for hallucination mitigation. Our approach first extracts token-level steering vectors and refines them, and then applies fine-grained, visual-sensitivity-adaptive steering only where it matters. This lightweight, plug-and-play mechanism requires only minimal training for calibration and can be applied across diverse vision-language models. It modulates the steering strength at each decoding step, selectively suppressing hallucination-prone spans while preserving evidence-grounded content. We evaluate TLVS on several benchmarks, including POPE, AMBER, CHAIR (COCO), MMHal, and HallusionBench, demonstrating consistent improvements over previous steering methods.

Abstract

VLM Hallucination

Explore similar work

Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation

See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs

Mitigating Entangled Steering in Large Vision-Language Models for Hallucination Reduction