Mitigating Entangled Steering in Large Vision-Language Models for Hallucination Reduction

Yuanhong Zhang; Zhaoyang Wang; Xin Zhang; Weizhan Zhang; Joey Tianyi Zhou

cs.CVApr 9, 2026

Mitigating Entangled Steering in Large Vision-Language Models for Hallucination Reduction

Authors: Yuanhong Zhang, Zhaoyang Wang, Xin Zhang, Weizhan Zhang, Joey Tianyi Zhou

Organizations: Ministry of Education Key Laboratory of Intelligent Networks and Network Security, Xi‘an Jiaotong University, Xi‘an, China · Centre for Frontier AI Research, Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore

Abstract

Large Vision-Language Models (LVLMs) have achieved remarkable success across cross-modal tasks but remain hindered by hallucinations, producing textual outputs inconsistent with visual content. Existing methods mitigate hallucinations but often alter generation behavior, resulting in shorter outputs and shifted token distributions, especially in latent space steering approaches. We identify that this issue stems from entangled steering signals, where suppressing hallucinations inadvertently disrupts the model's intrinsic generation behavior. To address this, we propose MESA, an effective plug-and-play framework that performs controlled and selective latent intervention for hallucination mitigation. Specifically, MESA targets hallucination-relevant responses while preserving the model's original token distribution, enabling effective hallucination reduction without compromising generation behavior. Extensive experiments across diverse generative and discriminative benchmarks demonstrate that MESA consistently reduces hallucinations while better preserving generation behavior, outperforming prior methods across multiple LVLM families.

Figures & tables

Figure 1. Motivation of MESA. Existing methods produce entangled steering directions from stochastic perturbations, reducing hallucination but altering generation behavior (top). MESA instead induces hallucination-specific representations via controlled perturbations, resulting in a clean steering direction that enables selective mitigation while preserving generation behavior (bottom).

Panel of Figure 2 — Figure 2. Analysis of latent space steering effects on hallucination and generation behavior. (Left) Relationship between steering strength $\alpha$ , hallucination ( $CHAIR_{S}$ ), and output length for VTI and Nullu. (Right) Mechanistic analysis: VTI shifts EOS competition, while Nullu alters token frequency distribution.

Figure 3. Overview of MESA. MESA decomposes hallucination mitigation into three offline stages and an inference-time intervention. We first construct a distributional supervision set from original and degraded inputs (Stage I), then learn a controlled perturbation to induce hallucination-aware representations while preserving semantics (Stage II), and extract steering directions via PCA over induced representation shifts (Stage III). At inference, the learned directions are injected into hidden states to selectively correct hallucination without disrupting generation dynamics (Stage IV).

Method	LLaVA-v1.5				Qwen-VL
Method	$\mathrm{CHAIR}_{S}$ $\downarrow$	$\mathrm{CHAIR}_{I}$ $\downarrow$	Recall $\uparrow$	Avg. length $\uparrow$	$\mathrm{CHAIR}_{S}$ $\downarrow$	$\mathrm{CHAIR}_{I}$ $\downarrow$	Recall $\uparrow$	Avg. length $\uparrow$
Vanilla	48.80	13.40	77.70	98.70	38.70	14.30	72.70	82.40
VCD ( Leng et al., 2024 )	56.30	16.00	77.10	99.60	37.60	13.20	64.90	71.30
ICD ( Wang et al., 2024 )	59.50	15.80	76.30	103.7	49.90	13.80	71.90	101.1
VAF ( Yin et al., 2025 )	51.10	14.00	77.90	96.60	36.90	10.60	72.50	79.50
ICT ( Chen et al., 2025 )	51.00	14.90	76.70	101.20	38.10	11.40	74.40	85.10

Table 1. CHAIR hallucination evaluation results on different LVLMs with max new tokens set to 512.

Model	Method	MSCOCO						GQA
		Random		Popular		Adversarial		Random		Popular		Adversarial
		Accuracy $\uparrow$	F1 $\uparrow$	Accuracy $\uparrow$	F1 $\uparrow$	Accuracy $\uparrow$	F1 $\uparrow$	Accuracy $\uparrow$	F1 $\uparrow$	Accuracy $\uparrow$	F1 $\uparrow$	Accuracy $\uparrow$	F1 $\uparrow$
LLaVA-v1.5	Vanilla	85.53	88.80	83.20	84.28	75.60	78.63	84.96	86.25	73.30	77.93	67.4	74.31
	VCD ( Leng et al., 2024 )	87.73	87.16	85.38	85.06	80.88	81.33	86.65	86.99	80.73	82.24	76.09	78.78
	ICD ( Wang et al., 2024 )	85.23	83.52	83.73	82.00	81.46	80.32	85.90	85.29	78.56	78.79	75.90	76.89
	VAF ( Yin et al., 2025 )	87.36	85.94	86.46	85.09	83.86	82.71	89.00	88.63	81.16	82.00	79.10	80.41

Table 2. Comparison results on the POPE benchmark. Results are reported in terms of Accuracy (%) and F1 Score (%). The proposed method achieves consistent improvements over related methods, with the best results highlighted in bold.

Generative Task
Method	CHAIR $\downarrow$	Cover $\uparrow$	Hallucination $\downarrow$	Cognition $\downarrow$
Vanilla	10.6	50.9	36.4	4.1
VTI ( Liu et al., 2025 )	6.7	47.6	28.1	3.6
Nullu ( Yang et al., 2025 )	7.4	48.2	30.5	3.7
MESA	6.4	50.4	27.4	3.2
Discriminative Task

Table 3. Results on AMBER on LLaVA-v1.5.

Figure 4. Evaluation results of LLaVA-v1.5 on the AMBER benchmark for the discriminative task.

Metric	Vanilla	MESA	VTI ( Liu et al., 2025 )	MESA	Nullu ( Yang et al., 2025 )	MESA
Acc $\uparrow$	6.26	6.43	3.93	5.67	6.10	6.52
Det $\uparrow$	5.80	5.81	3.53	5.61	5.67	5.77

Table 4. Results on LLaVA-Bench.

Training Loss for Perturbation
Method	$\mathrm{CHAIR}_{S}$ $\downarrow$	$\mathrm{CHAIR}_{I}$ $\downarrow$	Recall $\uparrow$	Avg. length $\uparrow$
Vanilla	48.8	13.4	77.7	98.7
w $\mathcal{L}_{\text{preserve}}$	41.6	11.2	75.6	93.8
w $\mathcal{L}_{\text{hall}}$	46.4	12.2	79.3	104.4
MESA	31.0	8.6	75.8	93.5
Different Type of Perturbation

Table 5. Ablation of training loss and perturbation designs.

Panel of Figure 5 — Figure 5. Ablation of key parameters. (a) Effect of different perturbation weight combinations in the hallucination-guided loss $\mathcal{L}_{\text{hall}}$ for training perturbation $\delta$ . (b) Effect of varying penalty strength $\alpha$ on the steering direction.

Decoding	Method	$\mathrm{CHAIR}_{S}$ $\downarrow$	$\mathrm{CHAIR}_{I}$ $\downarrow$	Recall $\uparrow$	Avg. len $\uparrow$	Decoding	Method	$\mathrm{CHAIR}_{S}$ $\downarrow$	$\mathrm{CHAIR}_{I}$ $\downarrow$	Recall $\uparrow$	Avg. len $\uparrow$
Top P	Vanilla	52.4	15.4	76.2	98.5	Top P+Temp.	Vanilla	50.6	14.4	78.3	98.2
Top P	MESA	36.2	8.7	75.3	90.1	Top P+Temp.	MESA	39.6	10.1	75.2	88.8
Top K	Vanilla	52.8	16.0	74.5	102.9	Top P+Temp.	Vanilla	51.2	14.9	76.7	98.2
Top K	MESA	38.0	9.9	71.5	92.3	Top P+Temp.	MESA	39.8	9.2	74.6	89.8
Temp.	Vanilla	50.8	15.0	77.6	99.0	Greedy	Vanilla	48.8	13.4	77.7	98.7
Temp.	MESA	35.6	9.3	73.4	89.2	Greedy	MESA	31.0	8.6	75.8	93.5

Table 6. An ablation study of different decoding strategies.

Panel of Figure 6 — Figure 6. Further analysis of MESA. (a)-(b) Generation behavior on CHAIR with LLaVA-v1.5: MESA preserves the EOS margin and maintains the Zipfian token distribution, indicating minimal disruption to the original generation dynamics. (c)-(d) Word-level hallucination reduction on CHAIR and AMBER: MESA consistently suppresses high-frequency hallucinated object words compared to the baseline.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

GPT-4o Prompt

You are an AI designed to evaluate and score the performance of two AI assistants in describing a given image. Your primary focus is on the accuracy and detailedness of their descriptions. You will assess the accuracy by checking for hallucinations - any part of the description that is inconsistent with the image content. For detailedness, you will consider how rich the response is in necessary details, excluding any hallucinated parts. You will provide scores on a scale from 1 to 10 for each assistant separately, based on these criteria. After scoring, you will offer an explanation for your evaluation, ensuring it is free from bias and not influenced by the order of presentation of the responses. Input format: [Assistant 1] {Response 1} [End of Assistant 1] [Assistant 2] {Response 2} [End of Assistant 2] Output format: Accuracy: Scores of the two answers: Reason: Detailedness: Scores of the two answers: Reason:

Appendix

Table 7. The prompt used for GPT-4o-aided evaluation.

Setting	Method	LLaVA-v1.5				Qwen-VL
Setting	Method	Accuracy $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 Score $\uparrow$	Accuracy $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 Score $\uparrow$
Random	Vanilla	85.53	86.77	90.93	88.80	87.23	91.70	80.06	85.50
	VCD	87.73 (↑2.20)	91.42 (↑4.65)	83.28 (↓7.65)	87.16 (↓1.64)	88.63 (↑1.40)	94.64 (↑2.94)	81.91 (↑1.85)	87.81 (↑2.31)
	ICD	85.23 (↓0.30)	94.44 (↑7.67)	74.86 (↓16.07)	83.52 (↓5.28)	82.43 (↓4.80)	98.80 (↑7.10)	65.67 (↓14.39)	78.89 (↓6.61)
	VAF	87.36 (↑1.83)	96.82 (↑10.05)	77.26 (↓13.67)	85.94 (↓2.86)	86.80 (↓0.43)	98.50 (↑6.80)	74.73 (↓5.33)	84.98 (↓0.52)
	ICT	88.96 (↑3.43)	96.87 (↑10.10)	80.40 (↓10.53)	87.85 (↓0.95)	88.06 (↑0.83)	96.90 (↑5.20)	78.11 (↓1.95)	86.55 (↑1.05)

Appendix

Table 8. Performance Comparison under Different Settings on MSCOCO with POPE Evaluation

Setting	Method	LLaVA-v1.5				Qwen-VL
Setting	Method	Accuracy $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 Score $\uparrow$	Accuracy $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 Score $\uparrow$
Random	Vanilla	85.40	80.83	91.30	85.76	87.46	91.32	82.80	86.85
	VCD	86.15 (↑0.75)	85.18 (↑4.35)	87.53 (↓3.77)	86.34 (↑0.58)	89.22 (↑1.76)	90.77 (↓0.55)	87.32 (↑4.52)	89.01 (↑2.16)
	ICD	86.53 (↑1.13)	90.59 (↑9.76)	81.53 (↓9.77)	85.82 (↑0.06)	82.80 (↓4.66)	97.04 (↑5.72)	67.67 (↓15.13)	79.73 (↓7.12)
	VAF	88.73 (↑3.33)	91.91 (↑11.08)	84.93 (↓6.37)	88.28 (↑2.52)	87.06 (↓0.40)	93.78 (↑2.46)	80.73 (↓2.07)	86.76 (↓0.09)
	ICT	88.96 (↑3.56)	92.88 (↑12.05)	84.40 (↓6.90)	88.43 (↑2.67)	89.36 (↑1.90)	92.88 (↑1.56)	85.80 (↑3.00)	89.20 (↑2.35)

Appendix

Table 9. Performance Comparison under Different Settings on A-OKVQA with POPE Evaluation

Setting	Method	LLaVA-v1.5				Qwen-VL
Setting	Method	Accuracy $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 Score $\uparrow$	Accuracy $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 Score $\uparrow$
Random	Vanilla	84.96	79.44	94.33	86.25	74.76	98.43	50.33	66.60
	VCD	86.65 (↑1.69)	84.85 (↑5.41)	89.24 (↓5.09)	86.99 (↑0.74)	85.59 (↑10.83)	86.88 (↓11.55)	83.84 (↑33.51)	85.33 (↑18.73)
	ICD	85.90 (↑0.94)	89.10 (↑9.66)	81.80 (↓12.53)	85.29 (↓0.96)	86.20 (↑11.44)	95.55 (↓2.88)	75.93 (↑25.60)	84.62 (↑18.02)
	VAF	89.00 (↑4.04)	91.66 (↑12.22)	85.80 (↓8.53)	88.63 (↑2.38)	86.11 (↑11.35)	96.87 (↓1.56)	77.53 (↑27.20)	86.11 (↑19.51)
	ICT	89.26 (↑4.30)	92.93 (↑13.49)	85.00 (↓9.33)	88.78 (↑2.53)	85.33 (↑10.57)	95.95 (↓2.48)	78.13 (↑27.80)	86.12 (↑19.52)

Appendix

Table 10. Performance Comparison under Different Settings on GQA with POPE Evaluation

Method	Generative Task				Discriminative Task
Method	CHAIR $\downarrow$	Cover $\uparrow$	Hallucination $\downarrow$	Cognition $\downarrow$	Accuracy $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 Score $\uparrow$
Vanilla	10.6	50.9	36.4	4.1	71.4	81.9	73.1	77.2
VCD	8.9 (↓1.70)	51.8 (↑0.90)	41.9 (↑5.50)	4.3 (↑0.20)	73.9 (↑2.50)	83.8 (↑1.90)	75.2 (↑2.10)	79.3 (↑2.10)
ICD	9.3 (↓1.30)	50.5 (↓0.40)	41.3 (↑4.90)	4.9 (↑0.80)	74.1 (↑2.70)	73.2 (↓8.70)	96.2 (↑23.10)	83.1 (↑5.90)
VAF	10.9 (↑0.30)	33.0 (↓17.90)	28.1 (↓8.30)	3.7 (↓0.40)	69.8 (↓1.60)	94.8 (↑12.90)	57.6 (↓15.50)	71.7 (↓5.50)
ICT	7.0 (↓3.60)	48.0 (↓2.90)	28.4 (↓8.00)	3.5 (↓0.60)	79.5 (↑8.10)	84.7 (↑2.80)	84.4 (↑11.30)	84.5 (↑7.30)

Appendix

Table 11. Performance comparison on generative and discriminative tasks with AMBER Evaluation.

Figure 7. The throughput (tested on NVIDIA A800) v.s.

\mathrm{CHAIR}_{S}

on CHAIR benchmark.

Rank	$\mathrm{CHAIR}_{S}$ $\downarrow$	$\mathrm{CHAIR}_{I}$ $\downarrow$	Recall $\uparrow$	Avg. length $\uparrow$
1	31.0	8.6	75.8	93.5
2	30.8	8.6	75.8	93.4
3	31.6	8.7	75.8	93.5
4	30.1	8.2	76.2	93.4
5	31.2	8.5	76.1	93.7

Appendix

Table 12. Performance under different rank settings.

Panel of Figure 8 — Figure 8. Comparison of token probability distributions and representative examples under normal and hallucination-inducing conditions. (a)-(b): The cumulative probability mass over token ranks shows a consistent concentration pattern. (c)-(d): Token-level examples illustrate that Jigsaw-induced hallucination alters the probability distribution of top-ranked tokens.

Figure 9. Illustration of hallucination correction comparisons. Text highlighted in red bold denotes explicit hallucinations or factual errors, whereas blue bold highlights indicate milder but potentially questionable over-inferences.

VLM Hallucination

Vision-language model hallucination is the production of claims unsupported by or in conflict with visual evidence.

See all 71 papers

Mitigating Hallucinations in Large Vision-Language Models without Performance Degradation
Apr 22, 2026Xingyu Zhu, Junfeng Fang, Shuo Wang +4Vision-Language Models VLM Adaptation
MoE Key Lab of BIPC, University of Science and Technology of China · National University of Singapore
Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation
Jun 2, 2026Ruipeng Zhang, Zhihao Li, C. L. Philip Chen +1VLM Hallucination Large Vision-Language Models
Guangdong Provincial Key Laboratory of Computational AI Models and Cognitive Intelligence, School of Computer Science & Engineering, South China University of Technology, Guangzhou, China · Engineering Research Center of the Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human, Guangzhou, China · Pazhou Lab, Guangzhou, China.
Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models
Feb 25, 2026Jianghao Yin, Qin Chen, Kedi Chen +3VLM Hallucination Mitigation Large Vision-Language Models
East China Normal University

Apr 22, 2026J/K move · Enter open · S save

Mitigating Hallucinations in Large Vision-Language Models without Performance Degradation

Xingyu Zhu, Junfeng Fang, Shuo Wang, Beier Zhu+3

MoE Key Lab of BIPC, University of Science and Technology of China · National University of Singapore

Vision-Language Models VLM Adaptation Hallucination in Language Models VLM Hallucination Large Vision-Language Models Representation Disentanglement+1

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) exhibit powerful generative capabilities but frequently produce hallucinations that compromise output reliability. Fine-tuning on annotated data devoid of hallucinations offers the most direct solution, while its high computational cost motivates recent representation-based methods, which focus on mitigating hallucinatory components within hidden representations. Though efficient, we empirically observe that these methods degrade general generation capacity due to incomplete extraction of hallucination components and non-selective parameter updates. To address these limitations, we propose MPD, a dual-stage framework for mitigating hallucinations without performance degradation. Specifically, our MPD relies on two essential factors: (1) semantic-aware component disentanglement to extract pure hallucination components, and (2) interpretable parameter updates that selectively modify parameters most relevant to hallucination. Extensive experiments demonstrate that MPD achieves state-of-the-art performance, reducing hallucinations by 23.4% while maintaining 97.4% of general generative capability as evaluated on LLaVA-Bench and MME, with no additional computational cost.

Abstract

VLM Hallucination

Explore similar work

Mitigating Hallucinations in Large Vision-Language Models without Performance Degradation

Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation

Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models