HulluEdit: Single-Pass Evidence-Consistent Subspace Editing for Mitigating Hallucinations in Large Vision-Language Models

Yangguang Lin; Quan Fang; Yufei Li; Jiachen Sun; Junyu Gao; Jitao Sang

cs.CVFeb 26, 2026

HulluEdit: Single-Pass Evidence-Consistent Subspace Editing for Mitigating Hallucinations in Large Vision-Language Models

Authors: Yangguang Lin, Quan Fang, Yufei Li, Jiachen Sun, Junyu Gao, Jitao Sang

Organizations: Beijing University of Posts and Telecommunications, Beijing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · Beijing Jiaotong University, Beijing, China

Abstract

Object hallucination in Large Vision-Language Models (LVLMs) significantly hinders their reliable deployment. Existing methods struggle to balance efficiency and accuracy: they often require expensive reference models and multiple forward passes, or apply static edits that risk suppressing genuine visual evidence. To address this, we introduce HulluEdit, a single-pass, reference-free intervention framework. Our core innovation is orthogonal subspace editing: we decompose the hidden states of the model into orthogonal subspaces - visual evidence, conflicting priors, and residual uncertainty - enabling selective suppression of hallucinatory patterns without interfering with visual grounding. This approach mathematically guarantees that edits applied to the prior subspace leave the visual component entirely unaffected. Extensive experiments show that HulluEdit achieves state-of-the-art hallucination reduction on benchmarks including POPE and CHAIR across diverse architectures, while preserving general capabilities on MME and maintaining efficient inference. Our method consistently outperforms contrastive decoding and static subspace editing baselines, offering a new pathway toward more trustworthy LVLMs.

Figures & tables

Panel of Figure 1 — Figure 1: Comparison of (a) traditional LVLMs prone to object hallucinations versus (b) our HulluEdit method that mitigates hallucinations via orthogonal subspace decomposition.

Figure 2: Overview of HalluEdit. We estimate a visual subspace $U$ from weighted visual tokens, an orthogonal anti -prior subspace $P$ from the text cache, and retain the residual subspace $R$ as uncertainty that is softly regularized when editing $h$ .

Category	Method	LLaVA-1.5-7B		LLaVA-1.5-13B		Qwen-VL-Chat-7B
Category	Method	Accuracy $\uparrow$	F1 $\uparrow$	Accuracy $\uparrow$	F1 $\uparrow$	Accuracy $\uparrow$	F1 $\uparrow$
Random	Greedy	87.8	87.5	87.6	87.4	88.2	87.9
	VCD	88.4	87.7	88.9	87.8	89.1	88.4
	ICD	88.1	87.6	88.1	87.6	88.9	88.1
	VAF	89.6	89.3	90.1	89.9	90.0	89.7
	DeCo	89.1	89.8	82.2	84.9	87.8	87.2

Table 1: Object hallucination evaluation on POPE benchmark. HulluEdit achieves consistent improvements across all model architectures and evaluation splits, with particularly strong performance on the Adversarial subset where language priors are most confounding.

Method	LLaVA-1.5			MiniGPT-4			mPLUG-Owl2
Method	$\mathrm{CHAIR}_{i}\downarrow$	$\mathrm{CHAIR}_{s}\downarrow$	BLEU $\uparrow$	$\mathrm{CHAIR}_{i}\downarrow$	$\mathrm{CHAIR}_{s}\downarrow$	BLEU $\uparrow$	$\mathrm{CHAIR}_{i}\downarrow$	$\mathrm{CHAIR}_{s}\downarrow$	BLEU $\uparrow$
Greedy	7.08	20.40	15.72	12.20	32.40	14.57	8.62	22.90	15.01
Beam Search [ 8 ]	6.84	19.50	15.99	11.87	30.10	15.35	7.62	20.30	15.43
DoLa [ 5 ]	6.75	20.20	15.68	12.15	31.90	14.54	8.36	22.40	15.13
OPERA [ 10 ]	6.07	17.50	16.02	11.96	29.70	14.82	7.18	20.07	15.41
VCD [ 12 ]	7.28	20.30	14.53	12.64	29.00	14.42	8.68	22.80	15.14

Table 2: Caption hallucination evaluation on MSCOCO dataset. Our method establishes new state-of-the-art performance on LLaVA-1.5 and mPLUG-Owl2, demonstrating the effectiveness of orthogonal subspace decomposition in reducing both instance-level and sentence-level hallucinations.

Method	Existence $\uparrow$	Count $\uparrow$	Position $\uparrow$	Color $\uparrow$
LLaVA-1.5	181.67	118.33	104.44	152.78
Nullu	190.00	121.11	105.56	156.67
DeCo	175.00	128.33	98.33	125.00
HulluEdit	195.00	105.00	126.67	160.00

Table 3: MME fine-grained evaluation. HulluEdit improves object recognition (Existence, Position, Color) while trading off Count performance, supporting the separation of visual and language priors.

Figure 3: Decoding throughput comparison measured in tokens per second (TPS). HulluEdit achieves competitive inference speed, significantly faster than recent hallucination mitigation methods like OPERA and HALC while maintaining strong performance.

Figure 4: Qualitative comparison of object hallucination mitigation. An example image with captions generated by the Original model and our HulluEdit method. Hallucinated objects are highlighted in red.

Category	Variant	$\mathbf{CHAIR_{i}\downarrow}$	$\mathbf{CHAIR_{s}\downarrow}$
Full Model	$L_{a}{=}26$ , $L_{e}{=}\text{last}$	4.18	13.00
Layer Selection	$L_{a}{=}20$ , $L_{e}{=}\text{last}$	5.55	19.72
Layer Selection	$L_{a}{=}30$ , $L_{e}{=}\text{last}$	5.37	13.80
Single-layer	$L_{a}{=}L_{e}{=}\text{last}$	5.50	18.20
Components	uniform SVD	4.85	13.68
	w/o orth. complement	5.60	15.90

Table 4: Ablation studies of HulluEdit components on LLaVA-1.5-7B using the MSCOCO dataset. We report instance-level (

\mathrm{CHAIR}_{i}

) and sentence-level (

\mathrm{CHAIR}_{s}

) hallucination rates (lower is better).

L_{a}

: anchor layer,

L_{e}

: edit layer.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Anchor layer ( $l_{a}$ )	26
Evidence rank ( $r$ )	8
Prior rank ( $q$ )	5
Base visual ( $\kappa$ )	0.60
Base prior ( $\lambda_{0}$ )	0.26
Max suppression ( $\lambda_{\max}$ )	3.6

Appendix

Table 5: Hyperparameter configuration for HulluEdit on LLaVA-1.5-7B

Panel of Figure 5 — Figure 5: Qualitative comparison of object hallucination mitigation on LLaVA-1.5-7B. The baseline model (top) generates descriptions containing spurious objects (highlighted in red), while HulluEdit (bottom) produces outputs strictly aligned with visual evidence. These examples demonstrate our method’s ability to suppress conflicting linguistic priors while preserving accurate visual descriptions.

Method	CHAIR $\downarrow$	Cover $\uparrow$	Hal $\downarrow$	Cog $\downarrow$
Vanilla	8.2	48.9	34.3	4.0
DeCo	6.6	47.5	28.1	2.8
Ours	5.3	47.0	22.4	2.1

Appendix

Table 6: Results on AMBER benchmark.

\downarrow

indicates lower is better,

\uparrow

indicates higher is better.

Method	Overall	qAcc	fAcc	aAcc
Vanilla	27.6	13.6	20.5	48.8
QLoRA	25.2	13.2	16.2	46.2
Ours	30.1	16.9	21.4	52.1

Appendix

Table 7: Results on Hallu-Bench benchmark.

Model	Method	Total
LLaVA-1.5	Vanilla	23.6
LLaVA-1.5	DeCo	27.9
LLaVA-1.5	Ours	28.5

Appendix

Table 8: MMVet results demonstrating general capability preservation.

Figure 6: MME benchmark results on LLaVA-1.5. HulluEdit maintains general visual understanding while reducing hallucinations.

Model	Method	CHAIR s $\downarrow$	CHAIR i $\downarrow$	Recall $\uparrow$
Qwen2.5-VL	greedy	15.6	6.2	46.0
	Ours	13.2	6.04	45.2
Intern2.5-VL	greedy	15.4	5.4	51.6
	Ours	12.8	5.1	50.9

Appendix

Table 9: Generalization results on recent VLM architectures.

Variable	CHAIR s $\downarrow$	CHAIR i $\downarrow$	Recall $\uparrow$
Vanilla	20.4	7.08	54.6
$r$ = [4,8]	13.4 $\pm$ 0.4	4.21 $\pm$ 0.03	54.5 $\pm$ 0.3
$q$ = [4,8]	13.2 $\pm$ 0.2	4.20 $\pm$ 0.02	54.1 $\pm$ 0.2
$\kappa$ = [0.3,0.8]	13.3 $\pm$ 0.3	4.22 $\pm$ 0.04	54.2 $\pm$ 0.2

Appendix

Table 10: Hyperparameters sensitivity analysis on LLaVA-1.5-7B.

Figure 7: Distribution of activation changes in visual subspace (

U

) vs. anti-prior subspace (

P

). The magnitude of changes in the visual subspace is orders of magnitude smaller (

10^{-3}

) compared to the anti-prior subspace (

10^{1}

), validating minimal interference with visual evidence.

VLM Hallucination Mitigation

Vision-language model hallucination mitigation keeps generated claims grounded in visual evidence.

See all 228 papers

Beyond Global Editing: Per-Instance Disentangled Subspaces for Training-Free Hallucination Mitigation in LVLMs
Aug 10, 2026Ali Cheraghian, Hamidreza Dastmalchi, Hamed Barzamini +4Large Vision-Language Models Model Editing
Macquarie University, Australia · Australian National University, Australia · York University, Canada +4
Hallucination-aware intermediate representation edit in large vision-language models
Mar 31, 2026Wei Suo, Hanzu Zhang, Lijun Zhang +3VLM Hallucination Mitigation Vision-Language Models
School of Computer Science, Northwestern Polytechnical University, China.
HIME: Mitigating Object Hallucinations in LVLMs via Hallucination Insensitivity Model Editing
Feb 21, 2026Ahmed Akl, Abdelwahed Khamis, Ali Cheraghian +3Object Hallucination in VLMs VLM Hallucination Mitigation
School of Information and Communication Technology, Griffith University, Australia · Data61, CSIRO, Australia · School of Engineering, Macquarie University, Sydney, Australia

Aug 10, 2026J/K move · Enter open · S save

Beyond Global Editing: Per-Instance Disentangled Subspaces for Training-Free Hallucination Mitigation in LVLMs

Ali Cheraghian, Hamidreza Dastmalchi, Hamed Barzamini, Morteza Saberi+3

Macquarie University, Australia · Australian National University, Australia · York University, Canada+4

Large Vision-Language Models Model Editing VLM Hallucination Mitigation

Open paper AskPDF arXiv

Recent advances in large vision-language models (LVLMs) have enabled powerful multimodal reasoning by integrating visual encoders with large language models (LLMs). However, their reliability is frequently undermined by hallucinations, where generated text inaccurately describes the visual input. Although fine-tuning can mitigate this problem, it is computationally expensive and requires large, curated datasets, making training-free alternatives attractive. Among these, model editing is more promising than decoding-based approaches: decoding methods adapt outputs per input but introduce computational overhead and instability, whereas model editing modifies internal representations offline, providing a more efficient and stable solution. However, existing model-editing techniques typically rely on a single global subspace to correct hallucinations, treating all test samples identically and failing to capture diverse hallucination modes across inputs. To address this limitation, we propose a training-free hallucination mitigation framework for dynamic, per-instance suppression at test time. Our method first constructs a set of Disentangled Hallucination Subspaces, each isolating a distinct hallucination mode. During inference, the model adaptively calculates weights reflecting each input's relationship to these subspaces, guiding a dynamically combined projection that selectively suppresses the most probable hallucination directions while preserving image-grounded semantics. Extensive experiments across multiple vision-language benchmarks and LVLM families demonstrate consistent improvements, highlighting the robustness, generalizability, and efficiency of our approach.

Abstract

VLM Hallucination Mitigation

Explore similar work

Beyond Global Editing: Per-Instance Disentangled Subspaces for Training-Free Hallucination Mitigation in LVLMs

Hallucination-aware intermediate representation edit in large vision-language models

HIME: Mitigating Object Hallucinations in LVLMs via Hallucination Insensitivity Model Editing