Hallucination-aware intermediate representation edit in large vision-language models

Wei Suo; Hanzu Zhang; Lijun Zhang; Ji Ma; Peng Wang; Yanning Zhang

cs.CVMar 31, 2026

Hallucination-aware intermediate representation edit in large vision-language models

Authors: Wei Suo, Hanzu Zhang, Lijun Zhang, Ji Ma, Peng Wang, Yanning Zhang

Organizations: School of Computer Science, Northwestern Polytechnical University, China.

Abstract

Large Vision-Language Models have demonstrated exceptional performance in multimodal reasoning and complex scene understanding. However, these models still face significant hallucination issues, where outputs contradict visual facts. Recent research on hallucination mitigation has focused on retraining methods and Contrastive Decoding (CD) methods. While both methods perform well, retraining methods require substantial training resources, and CD methods introduce dual inference overhead. These factors hinder their practical applicability. To address the above issue, we propose a framework for dynamically detecting hallucination representations and performing hallucination-eliminating edits on these representations. With minimal additional computational cost, we achieve state-of-the-art performance on existing benchmarks. Extensive experiments demonstrate the effectiveness of our approach, highlighting its efficient and robust hallucination elimination capability and its powerful controllability over hallucinations. Code is available at https://github.com/ASGO-MM/HIRE

Figures & tables

Figure 1: Comparison of mainstream hallucination mitigation paradigms. (a) Retraining-based methods: Constructing hallucination-specific datasets and training frameworks. (b) Contrastive decoding methods: Comparing the original probability distribution with a perturbed one. (c) Our method: Editing intermediate representations of LVLMs.

Figure 2: Overview of HIRE. Our framework consists of two key components: the Editor, which learns semantic invariance and hallucinatory difference through contrastive learning, and the Router, which learns efficient editing strategies through DPO.

Max new tokens set to 512
Method	LLaVA-1.5 ( Liu et al., 2024b )			InstructBLIP ( Dai et al., 2023 )
	CHAIR S $\downarrow$	CHAIR I $\downarrow$	TFLOPs $\downarrow$	CHAIR S $\downarrow$	CHAIR I $\downarrow$	TFLOPs $\downarrow$
baseline	51.3	16.8	10.23	51.0	24.2	2.77
OPERA ( Huang et al., 2024 )	45.2	12.7	-	47.4	12.9	-
SID ( Huo et al., 2024 )	44.2	12.2	-	42.3	12.4	-
ICD ( Wang et al., 2024 )	47.4	13.9	20.63	46.3	15.3	5.64

Table 1: Evaluation results on the CHAIR benchmark with LLaVA-1.5.

Method	Random		Popular		Adversarial		ALL		TFLOPs $\downarrow$
	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	TFLOPs $\downarrow$
LLaVA-1.5-7B	83.77	81.94	82.57	80.86	79.77	78.47	82.04	80.42	8.13
Referenced Results
+HACL	88.59	88.70	87.84	87.36	86.54	85.73	87.66	87.26	-
+VTI	89.50	88.89	87.36	86.69	82.57	82.11	86.48	85.90	-
Comparable Results

Table 2: Evaluation results on the POPE benchmark on the MS COCO datasets with LLaVA-1.5 and InstructBLIP.

Figure 3: Control hallucination via $\alpha$ .

Model	Setting	Generative					Discriminative			AMBER $\uparrow$
Model	Setting	CHAIR $\downarrow$	Cover $\uparrow$	Hal $\downarrow$	Cog $\downarrow$	TFLOPs $\downarrow$	Acc. $\uparrow$	F1 $\uparrow$	TFLOPs $\downarrow$	AMBER $\uparrow$
LLaVA-1.5	baseline	8.0	44.5	31.0	2.2	9.89	67.00	71.10	8.07	81.58
	VCD	6.7	46.5	27.8	2.0	19.55	67.30	71.10	16.14	82.20
	M3ID	6.0	48.9	26.0	1.5	19.81	67.25	70.90	16.14	82.25
	AVISC	6.3	46.6	25.6	2.0	19.45	70.70	75.45	16.14	84.60
	Octopus	4.8	49.2	23.4	1.2	20.48	76.70	82.70	16.35	88.95

Table 3: Evaluation results on the AMBER benchmark with LLaVA-1.5 and InstructBLIP.

	Semantic encoder	Hallucinatory encoder	Router	$\text{CHAIR}_{S}$ $\downarrow$	$\text{CHAIR}_{I}$ $\downarrow$	TFLOPs $\downarrow$
1				51.3	16.8	10.23
2	✓			37.0	12.2	-
3		✓		48.6	13.0	-
4	✓	✓		30.4	9.5	16.23
5	✓	✓	✓	30.2	9.7	11.81

Table 4: Ablation study on the impact of encoder architecture and the Router module. Results show the effects on hallucination mitigation performance and computational cost.

Figure 4: Some examples of generative and discriminative tasks on the MSCOCO dataset, with hallucinated content highlighted in red and newly added correct content displayed in green.

Figure 5: Distribution of original, hallucinated, and edited representations.

Table 10

Method	MME $\uparrow$	SEED $\uparrow$
LLaVA-1.5 7B	1751.64	64.3
+Ours	1751.99	63.8

Table 7: Comparison of general capabilities on MME and SEED-Bench.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Edit layer	CHAIR ${}_{S}\downarrow$	CHAIR ${}_{I}\downarrow$
baseline	51.3	16.8
shallow (0 $\sim$ 9)	42.0	13.0
middle (10 $\sim$ 19)	36.0	10.7
deep (20 $\sim$ 29)	46.8	14.1
HIRE (0 $\sim$ 31)	30.2	9.7

Appendix

Table 8: Evaluating the impact of editing different layers on the CHAIR benchmark.

Router type	CHAIR S $\downarrow$	CHAIR I $\downarrow$	BLEU $\uparrow$	training time $\downarrow$
LLaVA-1.5	51.3	16.8	17.47	-
I (a unified router)	30.6	9.6	20.53	11h
II (layer-specific routers)	30.4	9.6	20.65	14h
III (initial embedding-based decision)	46.2	12.9	16.93	8h
Ours (first layer embedding-based decision)	30.2	9.7	20.59	8h

Appendix

Table 9: Performance of different router strategies evaluated by hallucination (CHAIR) and text quality (BLEU).

Method	CHAIR S $\downarrow$	CHAIR I $\downarrow$
LLaVA-1.5	51.3	16.8
Instruction perturbation	32.4	8.7
Visual perturbation	30.2	9.7

Appendix

Table 10: CHAIR evaluation of HIRE trained with different hallucination induction methods.

Figure 6: Performance of HIRE on CHAIR benchmark under different training data sizes.

Varying Learning Rate.
Setting	CHAIRs $\downarrow$	CHAIRi $\downarrow$
LLaVA-1.5-7B	51.3	16.8
+Ours (1e-3)	30.2	9.7
5e-4	32.4	12.1
5e-5	33.4	10.2
Varying Scaling Factor $\beta$ .

Appendix

Table 11: Analysis of training stability under different hyperparameters.

Model	CHAIR		AMBER
Model	CHAIR s $\downarrow$	CHAIR i $\downarrow$	CHAIR $\downarrow$	Cover. $\uparrow$	Cog. $\downarrow$	HalRate $\downarrow$
Qwen-2.5-VL	46.2	9.6	8.8	59.9	3.4	46.6
Qwen-2.5-VL+Ours	35.6	8.1	6.5	60.3	2.1	36.8
TinyLLaVA-1.5B	58.6	17.5	11.2	50.8	6.2	48.0
TinyLLaVA-1.5B+Ours	36.6	11.7	8.6	48.7	2.6	35.6
LLaVA-1.5-13B	43.8	12.3	6.3	51.2	3.1	30.9

Appendix

Table 12: Evaluation on CHAIR and AMBER across model types and scales

Dataset	Method	Random		Popular		Adversarial		All
Dataset	Method	Acc $\uparrow$	F1 $\uparrow$	Acc $\uparrow$	F1 $\uparrow$	Acc $\uparrow$	F1 $\uparrow$	Acc $\uparrow$	F1 $\uparrow$
A-OKVQA	LLaVA-1.5	84.93	84.07	80.90	80.64	74.80	75.59	80.21	80.10
	+VCD	85.53	85.12	81.17	81.46	75.03	76.72	80.58	81.10
	+M3ID	85.06	84.30	80.90	80.77	74.80	76.15	80.25	80.41
	+AVISC	87.33	86.14	85.03	84.03	79.27	79.16	83.88	83.11
	+Ours	88.90	89.20	86.50	86.52	78.07	79.70	84.49	85.14

Appendix

Table 13: More results on the GQA and A-OKVQA datasets of POPE.

Model	CHAIR S $\downarrow$	CHAIR I $\downarrow$
LLaVA-1.5	51.3	16.8
+Ours (trained on VG_100K)	30.6	10.5
+Ours (trained on MSCOCO)	30.2	9.7

Appendix

Table 14: Evaluation on the generalization ability of HIRE

	CHAIR S $\downarrow$	CHAIR I $\downarrow$
baseline	51.3	16.8
1	32.0	9.9
2	30.0	9.3
3	31.6	9.6
4	28.4	9.1
5	30.4	9.1

Appendix

Table 15: Evaluation on the stability of HIRE

Panel of Figure 9 — Figure 9: Some examples of generative tasks on the COCO dataset, with hallucinated content highlighted in red and newly added correct content displayed in blue.

Panel of Figure 10 — Figure 10: Some examples show that HIRE can amplify or mitigate hallucination by adjust the hyperparameter $\alpha$ .

Figure 11: Some examples of discriminative tasks on the COCO dataset.

VLM Hallucination Mitigation

Vision-language model hallucination mitigation keeps generated claims grounded in visual evidence.

See all 228 papers

HulluEdit: Single-Pass Evidence-Consistent Subspace Editing for Mitigating Hallucinations in Large Vision-Language Models
Feb 26, 2026Yangguang Lin, Quan Fang, Yufei Li +3VLM Hallucination Mitigation Object Hallucination in VLMs
Beijing University of Posts and Telecommunications, Beijing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · Beijing Jiaotong University, Beijing, China
Mitigating Hallucinations in Large Vision-Language Models without Performance Degradation
Apr 22, 2026Xingyu Zhu, Junfeng Fang, Shuo Wang +4Vision-Language Models VLM Adaptation
MoE Key Lab of BIPC, University of Science and Technology of China · National University of Singapore
Beyond Global Editing: Per-Instance Disentangled Subspaces for Training-Free Hallucination Mitigation in LVLMs
Aug 10, 2026Ali Cheraghian, Hamidreza Dastmalchi, Hamed Barzamini +4Large Vision-Language Models Model Editing
Macquarie University, Australia · Australian National University, Australia · York University, Canada +4

Feb 26, 2026J/K move · Enter open · S save

HulluEdit: Single-Pass Evidence-Consistent Subspace Editing for Mitigating Hallucinations in Large Vision-Language Models

Yangguang Lin, Quan Fang, Yufei Li, Jiachen Sun+2

Beijing University of Posts and Telecommunications, Beijing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · Beijing Jiaotong University, Beijing, China

VLM Hallucination Mitigation Object Hallucination in VLMs Large Vision-Language Models Representation Engineering

Open paper AskPDF arXiv

Object hallucination in Large Vision-Language Models (LVLMs) significantly hinders their reliable deployment. Existing methods struggle to balance efficiency and accuracy: they often require expensive reference models and multiple forward passes, or apply static edits that risk suppressing genuine visual evidence. To address this, we introduce HulluEdit, a single-pass, reference-free intervention framework. Our core innovation is orthogonal subspace editing: we decompose the hidden states of the model into orthogonal subspaces - visual evidence, conflicting priors, and residual uncertainty - enabling selective suppression of hallucinatory patterns without interfering with visual grounding. This approach mathematically guarantees that edits applied to the prior subspace leave the visual component entirely unaffected. Extensive experiments show that HulluEdit achieves state-of-the-art hallucination reduction on benchmarks including POPE and CHAIR across diverse architectures, while preserving general capabilities on MME and maintaining efficient inference. Our method consistently outperforms contrastive decoding and static subspace editing baselines, offering a new pathway toward more trustworthy LVLMs.

Abstract

VLM Hallucination Mitigation

Explore similar work

HulluEdit: Single-Pass Evidence-Consistent Subspace Editing for Mitigating Hallucinations in Large Vision-Language Models

Mitigating Hallucinations in Large Vision-Language Models without Performance Degradation

Beyond Global Editing: Per-Instance Disentangled Subspaces for Training-Free Hallucination Mitigation in LVLMs