Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models

Jianghao Yin; Qin Chen; Kedi Chen; Jie Zhou; Xingjiao Wu; Liang He

cs.CVFeb 25, 2026

Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models

Authors: Jianghao Yin, Qin Chen, Kedi Chen, Jie Zhou, Xingjiao Wu, Liang He

Organizations: East China Normal University

Abstract

Large Vision-Language Models (LVLMs) exhibit outstanding performance on vision-language tasks but struggle with hallucination problems. Through in-depth analysis of LVLM activation patterns, we reveal two key findings: 1) truthfulness and visual perception capabilities predominantly engage different subsets of attention heads within the model architecture; and 2) truthfulness steering vectors vary significantly across different semantic contexts. Based on these observations, we propose Dynamic Multimodal Activation Steering, a training-free approach for hallucination mitigation. Our method constructs a semantic-based truthfulness steering vector database and computes visual perception steering vectors, enabling context-aware interventions during inference by dynamically selecting the most relevant steering vectors based on input semantic similarity and applying them to the most influential attention heads. We conduct comprehensive experiments across multiple models and datasets, demonstrating that our approach significantly enhances model performance, outperforming existing state-of-the-art methods.

Figures & tables

Panel of Figure 1 — Figure 1: Activation differences in LLaVAv1.5.

Figure 2: Overview of the DMAS framework.

Model	Method	Existence $\uparrow$	Count $\uparrow$	Position $\uparrow$	Color $\uparrow$	Total Scores $\uparrow$
LLaVAv1.5	Regular	175.67	124.67	114.00	151.00	565.33
	VCD	184.66	138.33	128.67	153.00	604.66
	OPERA	180.67	133.33	123.33	155.00	592.33
	VAF	195.00	158.33	128.33	155.00	636.67
	DECO	185.00	153.33	118.33	155.00	611.66
	DAMO	195.00	150.00	143.33	165.00	653.33

Table 1: Results on MME. The best results are shown in bold.

Dataset	Setting	Method	Accuracy $\uparrow$	Precision	Recall	F1 Score $\uparrow$
MSCOCO	LLaVAv1.5	Regular	81.38	88.04	72.78	79.65
		VCD	84.33	85.93	83.28	84.52
		OPERA	84.21	88.23	79.79	83.72
		VAF	86.90	89.43	83.77	86.47
		AGLA	85.82	93.78	76.83	84.44
		VTI	86.48	90.11	82.09	85.90

Table 2: Results on POPE. Best results are in bold, and second-best values are underlined.

Table 5

Panel of Figure 3 — Figure 3: Impact of key hyperparameters .

Figure 4: Effect of dynamic intervention.

Method	ScienceQA		ViQuAE
Method	LLaVAv1.5	QwenVL	LLaVAv1.5	QwenVL
Regular	52.75	46.41	43.38	50.09
VTI	51.46	-	42.29	-
ICT	52.95	-	42.47	-
Ours	62.27	48.04	56.00	54.08

Table 5: Generality on ScienceQA and ViQuAE.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Setting	Method	Accuracy $\uparrow$	Precision	Recall	F1 Score $\uparrow$
MSCOCO	Random	Regular	83.29	92.13	72.80	81.33
		VCD	87.73	91.42	83.28	87.16
		Ours	90.03	90.51	90.03	90.02
	Popular	Regular	81.88	88.93	72.80	80.06
		VCD	85.38	86.92	83.28	85.06
		Ours	87.33	89.16	85.00	87.03

Appendix

Table 6: Results on LLaVAv1.5. The best results are shown in bold.

Method	Acc $\uparrow$	F1 $\uparrow$	CHAIR $\downarrow$	Hal $\downarrow$	AMBER SCORE $\uparrow$
	Discriminative		Generative
Regular	67.4	71.2	11.6	47.7	79.80
VCD ( Leng et al., 2024 )	68.1	71.1	9.8	43.8	80.65
ICD ( Wang et al., 2024c )	70.3	73.4	8.8	38.7	82.3
IBD ( Zhu et al., 2025b )	69.2	72.2	9.8	42.2	81.2
DeFG ( Zhang et al., 2025 )	70.2	73.0	9.1	39.9	81.95

Appendix

Table 7: Results on AMBER. The best results are shown in bold.

Model	Method	Existence $\uparrow$	Count $\uparrow$	Position $\uparrow$	Color $\uparrow$	Total Scores $\uparrow$	$CHAIR_{S}$ $\downarrow$	$CHAIR_{I}$ $\downarrow$
		MME					CHAIR
LLaVAv1.5 7B	Regular	175.67	124.67	114.00	151.00	565.33	51.0	15.2
LLaVAv1.5 7B	Ours	195.00	158.33	133.33	173.33	659.99	30.8	11.4
LLaVA1.5 13B	Regular	185.00	131.67	95.00	175.00	586.67	45.0	11.8
LLaVA1.5 13B	Ours	185.00	158.33	103.33	180.00	626.66	38.0	10.8

Appendix

Table 8: Scalability Analysis Across Model Sizes on MME and CHAIR. The best results are shown in bold.

Figure 5: Effect of dynamic intervention.

Model	Method	MMVP	MM-Vet	IllusionVQA	HallusionBench
LLaVAv1.5	Vanilla	30.3	38.9	32.6	50.2
	ICT	31.0	33.4	33.1	52.7
	Ours	54.3	39.3	35.2	53.7
QwenVL	Vanilla	41.0	32.9	31.7	42.4
QwenVL	Ours	43.3	34.4	33.6	42.9

Appendix

Table 9: Experiments on more benchmarks.

Scale	Existence $\uparrow$	Count $\uparrow$	Position $\uparrow$	Color $\uparrow$	Accuracy $\uparrow$	Relation $\uparrow$	F1 $\uparrow$
	MME				MMVP	AMBER
100%	195.00	158.33	133.33	173.33	54.3	69.0	87.2
75%	195.00	158.33	133.33	168.33	52.0	69.0	87.2
50%	195.00	158.33	133.33	168.33	52.0	69.0	87.2
25%	195.00	158.33	133.33	168.33	53.0	69.0	87.1

Appendix

Table 10: Analysis of dataset scale for steering vectors construction.

Scale	Existence $\uparrow$	Count $\uparrow$	Position $\uparrow$	Color $\uparrow$	Accuracy $\uparrow$	Relation $\uparrow$	F1 $\uparrow$
	MME				MMVP	AMBER
Ours	195.00	158.33	133.33	173.33	54.3	69.0	87.2
POPE	195.00	158.33	123.33	153.33	52.3	65.1	86.0
PHD	195.00	158.33	133.33	173.33	53.7	68.3	86.9

Appendix

Table 11: Analysis of dataset composition for steering vectors construction.

Decoding Strategies	Method	Existence $\uparrow$	Count $\uparrow$	Position $\uparrow$	Color $\uparrow$	Total Scores $\uparrow$	Accuracy $\uparrow$
		MME					MMVP
Beam	Vanilla	190.00	153.33	113.33	148.33	604.99	29.7
Beam	Ours	191.67	156.66	135.00	171.66	654.99	55.1
Nucleus	Vanilla	190.00	147.78	123.89	150.55	612.22	32.3
Nucleus	Ours	190.00	150.00	130.00	159.44	629.44	53.1
Greedy	Vanilla	190.00	153.33	113.33	148.33	604.99	30.3

Appendix

Table 12: Experiments on different decoding strategies.

Model	Method	Total Scores $\uparrow$	$CHAIR_{S}$ $\downarrow$	$CHAIR_{I}$ $\downarrow$
		MME	CHAIR
Qwen2.5-VL	Vanilla	698.33	47.4	10
Qwen2.5-VL	Ours	718.33	43.2	9.5
InternVL3	Vanilla	698.33	28.4	7.6
InternVL3	Ours	713.33	26.4	6.9
Mantis	Vanilla	653.33	44.6	12.7

Appendix

Table 13: Experiments on more models.

	MME					CHAIR
Strategy	Existence $\uparrow$	Count $\uparrow$	Position $\uparrow$	Color $\uparrow$	Total Scores $\uparrow$	$CHAIR_{S}$ $\downarrow$	$CHAIR_{I}$ $\downarrow$
Random	195.00	158.33	133.33	173.33	659.99	31.2	10.9
Hard Negative	195.00	158.33	133.33	173.33	659.99	30.8	11.4

Appendix

Table 14: Ablation studies on hallucination objects selection strategies.

VLM Hallucination Mitigation

Vision-language model hallucination mitigation keeps generated claims grounded in visual evidence.

See all 228 papers

Mitigating Entangled Steering in Large Vision-Language Models for Hallucination Reduction
Apr 9, 2026Yuanhong Zhang, Zhaoyang Wang, Xin Zhang +2VLM Hallucination VLM Hallucination Mitigation
Ministry of Education Key Laboratory of Intelligent Networks and Network Security, Xi‘an Jiaotong University, Xi‘an, China · Centre for Frontier AI Research, Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore
Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating
May 20, 2026Zhe Cheng, Wenyu Chen, Fode Zhang +1VLM Robustness Hallucination in Language Models
Center of Statistical Research, School of Statistics and Data Science, Southwestern University of Finance and Economics, Chengdu, China. · Department of Biomedical Engineering, College of Design and Engineering, National University of Singapore, Singapore.
Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation
Jun 2, 2026Ruipeng Zhang, Zhihao Li, C. L. Philip Chen +1VLM Hallucination Large Vision-Language Models
Guangdong Provincial Key Laboratory of Computational AI Models and Cognitive Intelligence, School of Computer Science & Engineering, South China University of Technology, Guangzhou, China · Engineering Research Center of the Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human, Guangzhou, China · Pazhou Lab, Guangzhou, China.

Apr 9, 2026J/K move · Enter open · S save

Mitigating Entangled Steering in Large Vision-Language Models for Hallucination Reduction

Yuanhong Zhang, Zhaoyang Wang, Xin Zhang, Weizhan Zhang+1

Ministry of Education Key Laboratory of Intelligent Networks and Network Security, Xi‘an Jiaotong University, Xi‘an, China · Centre for Frontier AI Research, Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore

VLM Hallucination VLM Hallucination Mitigation Large Vision-Language Models

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) have achieved remarkable success across cross-modal tasks but remain hindered by hallucinations, producing textual outputs inconsistent with visual content. Existing methods mitigate hallucinations but often alter generation behavior, resulting in shorter outputs and shifted token distributions, especially in latent space steering approaches. We identify that this issue stems from entangled steering signals, where suppressing hallucinations inadvertently disrupts the model's intrinsic generation behavior. To address this, we propose MESA, an effective plug-and-play framework that performs controlled and selective latent intervention for hallucination mitigation. Specifically, MESA targets hallucination-relevant responses while preserving the model's original token distribution, enabling effective hallucination reduction without compromising generation behavior. Extensive experiments across diverse generative and discriminative benchmarks demonstrate that MESA consistently reduces hallucinations while better preserving generation behavior, outperforming prior methods across multiple LVLM families.

Abstract

VLM Hallucination Mitigation

Explore similar work

Mitigating Entangled Steering in Large Vision-Language Models for Hallucination Reduction

Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating

Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation