HIME: Mitigating Object Hallucinations in LVLMs via Hallucination Insensitivity Model Editing

Ahmed Akl; Abdelwahed Khamis; Ali Cheraghian; Zhe Wang; Sara Khalifa; Kewen Wang

cs.CVFeb 21, 2026

HIME: Mitigating Object Hallucinations in LVLMs via Hallucination Insensitivity Model Editing

Authors: Ahmed Akl, Abdelwahed Khamis, Ali Cheraghian, Zhe Wang, Sara Khalifa, Kewen Wang

Organizations: School of Information and Communication Technology, Griffith University, Australia · Data61, CSIRO, Australia · School of Engineering, Macquarie University, Sydney, Australia

Abstract

Large Vision-Language Models (LVLMs) have demonstrated impressive multimodal understanding capabilities, yet they remain prone to object hallucination, where models describe non-existent objects or attribute incorrect factual information, raising serious concerns for reliable real-world deployment. While fine-tuning is a commonly adopted mitigation strategy, its high computational cost and practical difficulty motivate the need for training-free alternatives, among which model editing has recently emerged as a promising direction. However, indiscriminate editing risks disrupting the rich implicit knowledge encoded in pre-trained LVLMs, leading to a fundamental question: how much intervention is necessary at each layer to suppress hallucinations while preserving pre-trained knowledge? To address this question, we present a systematic analysis of LVLM decoders built on three widely used large language model backbones-Qwen, LLaMA, and Vicuna-revealing clear layer-wise differences in susceptibility to object hallucination. Building on these insights, we introduce the Hallucination Insensitivity Score (HIS), a principled metric that quantifies each layer's sensitivity to hallucination and provides guidance for targeted intervention. Leveraging HIS, we propose Hallucination Insensitivity Model Editing (HIME), a simple yet effective layer-adaptive weight editing approach that selectively modifies latent features to suppress hallucinations while preserving pre-trained knowledge. Extensive experiments demonstrate that HIME reduces hallucinations by an average of 61.8% across open-ended generation benchmarks, including CHAIR, MME, and GPT-4V-aided evaluation, without introducing additional parameters, inference-time latency, or computational overhead.

Figures & tables

Figure 1: Illustrates the phenomenon of knowledge distortion because of the fixed weight editing. On top, LVLM e.g. LLaVA-1.5 hallucinates non-existent objects, chair, couch , frequently co-occurring to bed. Left bottom, Fixed Weight Editing reduce hallucinated object, yet drop existent objects, bed . Right bottom, Our approach, HIME, mitigate hallucination, chair, couch , and preserve the pre-trained knowledge, bed .

Figure 2: HIME hallucination mitigation framework. Given an image with truthful and hallucinated captions, the Hallucination Insensitivity Score (HIS) is derived by contrasting attention distributions via KL divergence. During model editing , model weights are orthogonally projected into the truthful subspace and weighted by inverse HIS. The resulting Edited LVLM Decoder emphasises truthful representations while reducing hallucination sensitivity across layers.

Figure 3: Layer-wise distribution of the Hallucination Insensitivity Score (HIS) across four VLM backbones. The y-axis reports the HIS value (higher indicates greater robustness to hallucinations). Across four VLM decoders, we observe a highly structured, non-uniform depth profile. Across architectures, mid-depth layers exhibit consistently higher robustness, while late/early decoder layers show pronounced sensitivity to hallucinations. This recurring pattern suggests targeted editing of specific depth regions rather than global model intervention.

Method	LLaVA-1.5			MiniGPT-4			mPLUG-Owl2
	$\text{CHAIR}_{s}\downarrow$	$\text{CHAIR}_{i}\downarrow$	BLEU $\uparrow$	$\text{CHAIR}_{s}\downarrow$	$\text{CHAIR}_{i}\downarrow$	BLEU $\uparrow$	$\text{CHAIR}_{s}\downarrow$	$\text{CHAIR}_{i}\downarrow$	BLEU $\uparrow$
Greedy	$20.40_{\pm 2.80}$	$7.08_{\pm 0.33}$	$15.72_{\pm 0.10}$	$32.40_{\pm 2.20}$	$12.20_{\pm 0.42}$	$14.57_{\pm 0.11}$	$22.90_{\pm 0.90}$	$8.62_{\pm 0.11}$	$15.01_{\pm 0.24}$
Beam Search Freitag and Al-Onaizan (2017)	$19.50_{\pm 2.30}$	$6.84_{\pm 0.79}$	$15.99_{\pm 0.14}$	$30.10_{\pm 0.30}$	$11.87_{\pm 0.37}$	$15.35_{\pm 0.24}$	$20.30_{\pm 0.70}$	$6.92_{\pm 0.19}$	$15.43_{\pm 0.53}$
DoLa Chuang et al. (2024)	$20.20_{\pm 2.80}$	$6.55_{\pm 0.70}$	$15.58_{\pm 0.62}$	$31.00_{\pm 2.10}$	$12.15_{\pm 0.89}$	$14.54_{\pm 0.12}$	$22.40_{\pm 0.80}$	$7.36_{\pm 0.34}$	$15.13_{\pm 0.26}$
OPERA Huang et al. (2024)	$17.50_{\pm 0.50}$	$6.07_{\pm 0.33}$	$16.02_{\pm 0.12}$	$29.70_{\pm 0.40}$	$11.90_{\pm 0.36}$	$14.82_{\pm 0.05}$	$20.67_{\pm 0.72}$	$6.40_{\pm 0.18}$	$15.41_{\pm 0.18}$
VCD Leng et al. (2024)	$20.30_{\pm 1.10}$	$7.28_{\pm 1.40}$	$14.53_{\pm 0.01}$	$29.20_{\pm 0.80}$	$11.69_{\pm 0.11}$	$14.42_{\pm 0.01}$	$22.80_{\pm 0.80}$	$8.67_{\pm 0.17}$	$15.14_{\pm 0.13}$

Table 1: CHAIR evaluation results on MSCOCO dataset of LVLMs (LLaVA-1.5, MiniGPT-4 and mPLUG-Owl2) with different methods for mitigating OH. Lower

\text{CHAIR}_{s}

and

\text{CHAIR}_{i}

indicate less Object Hallucination. Higher BLEU generally represents higher captioning quality. We use 64 as the max token number in this experiment. Bold indicates the best result of all methods.

LVLMs	CHAIR S $\downarrow$	CHAIR I $\downarrow$	BLEU $\uparrow$
QWen2-VL-8B-Instruct(QWen2)	20.8	5.36	11.16
HIME	17.2	4.43	11.30
QWen3-VL-8B-Instruct(QWen3)	8.40	4.62	9.81
HIME	6.00	3.44	8.89

Table 2: Evaluating HIME with advanced and recent LVLMs using backbones other than Llama. Lower CHAIR s , and CHAIR i is better.

Model	Method	Existence	Count	Position	Color	Posters
LLaVA-1.5	Original	$181.67\pm 2.36$	$118.33\pm 12.47$	$104.44\pm 10.39$	$152.78\pm 5.67$	$117.23\pm 4.79$
LLaVA-1.5	Nullu	$190.00\pm 4.08$	$121.11\pm 7.74$	$105.56\pm 8.30$	156.67 $\pm$ 9.81	$127.55\pm 4.20$
LLaVA-1.5	HIME	195.00 $\pm$ 0.00	155.56 $\pm$ 4.81	123.33 $\pm$ 0.00	151.67 $\pm$ 5.77	130.27 $\pm$ 0.59
Model	Method	Celebrity	Scene	Landmark	Artwork	OCR
LLaVA-1.5	Original	$111.67\pm 3.90$	$144.83\pm 1.50$	$130.65\pm 5.26$	$108.92\pm 2.99$	$75.83\pm 5.89$
LLaVA-1.5	Nullu	$115.59\pm 6.60$	$147.92\pm 1.36$	$131.66\pm 1.09$	$113.00\pm 2.07$	$121.67\pm 8.25$

Table 3: Results on MME perception-related tasks. Bold indicates the best score per column (mean

\pm

sample SD).

Figure 4: Results from the baseline LLaVA and our approach HIME on the cognition tasks.

Figure 5: Ablation study evaluation results for Hallucination Insensitivity Score (HIS) on LLaVA-1.5. Lower CHAIR s , and CHAIR i is better, while higher BLUE is better.

Figure 6: Results from a GPT-4V–assisted evaluation of open-ended generation. Accuracy reflects how well the response matches the image, while Detailedness captures the richness of relevant detail. Both scores are reported on a 10-point scale. on LLaVA-1.5.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

$\{\ell\}$	C S	C I	BLEU	$k$	C S	C I	BLEU
5–32	18.6	6.44	15.15	2	20.2	6.53	15.58
10–32	18.2	5.95	15.16	5	13.8	4.56	15.82
20–32	13.8	4.56	15.82	10	17.00	5.39	15.65
30–32	15.6	4.99	15.62	32	15.8	5.5	14.29

Appendix

Table 4: Ablation study of the hyper-parameters in HIME . On the left side, we use

k=5

and change the layers range to be edited. On the right side, we use

l=20-32

and change the top-ranked vectors used for weight editing.

Parameter	Value
Do-sample	False
Num-beams (beam search)	3
Max new tokens (CHAIR)	64
Max new tokens (MME)	64
Max new tokens (LLaVA-Bench)	1024

Appendix

Table 5: Hyperparameters used for LVLM decoding.

Panel of Figure 7 — Figure 7: Cognition-task results for (a) mPLUG-Owl2 and (b) MiniGPT-4: baseline vs. HIME.

Figure 8: Illustration of hallucination correction by our proposed HIME with a sample from LLaVA-Bench using LLaVA-1.5-7B. We highlight the hallucinated objects from the original model in red .

Figure 10: Layer-wise distribution of the proposed normalised Hallucination Insensitivity Score (HIS) across five widely used LVLMs: mPLUG-Owl2, MiniGPT-4, LLaVA-1.5, Qwen2-vl, and Qwen3-VL. Despite architectural differences, all models exhibit similar response patterns: layers around the middle show lower HIS values, indicating higher susceptibility to hallucinations, while deeper/early layers are comparatively more robust. Highlighting HIS as a principled indicator for identifying hallucination-prone layers and guiding targeted model editing

Panel of Table 6 — Table 6: The prompt used to evaluate the performance of our proposed method using GPT-4V evaluator.

Table 7: Evaluation example for the image-description evaluator with GPT-4V.

Object Hallucination in VLMs

Object hallucination in vision-language models is the production of object claims unsupported by visual evidence.

See all 128 papers

Beyond Global Editing: Per-Instance Disentangled Subspaces for Training-Free Hallucination Mitigation in LVLMs
Aug 10, 2026Ali Cheraghian, Hamidreza Dastmalchi, Hamed Barzamini +4Large Vision-Language Models Model Editing
Macquarie University, Australia · Australian National University, Australia · York University, Canada +4
NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors
Feb 25, 2026Lingfeng Ren, Weihao Yu, Runpeng Yu +1Object Hallucination in VLMs VLM Hallucination Mitigation
National University of Singapore, Singapore · Peking University Shenzhen Graduate School, China
HulluEdit: Single-Pass Evidence-Consistent Subspace Editing for Mitigating Hallucinations in Large Vision-Language Models
Feb 26, 2026Yangguang Lin, Quan Fang, Yufei Li +3VLM Hallucination Mitigation Object Hallucination in VLMs
Beijing University of Posts and Telecommunications, Beijing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · Beijing Jiaotong University, Beijing, China

Aug 10, 2026J/K move · Enter open · S save

Beyond Global Editing: Per-Instance Disentangled Subspaces for Training-Free Hallucination Mitigation in LVLMs

Ali Cheraghian, Hamidreza Dastmalchi, Hamed Barzamini, Morteza Saberi+3

Macquarie University, Australia · Australian National University, Australia · York University, Canada+4

Large Vision-Language Models Model Editing VLM Hallucination Mitigation

Open paper AskPDF arXiv

Recent advances in large vision-language models (LVLMs) have enabled powerful multimodal reasoning by integrating visual encoders with large language models (LLMs). However, their reliability is frequently undermined by hallucinations, where generated text inaccurately describes the visual input. Although fine-tuning can mitigate this problem, it is computationally expensive and requires large, curated datasets, making training-free alternatives attractive. Among these, model editing is more promising than decoding-based approaches: decoding methods adapt outputs per input but introduce computational overhead and instability, whereas model editing modifies internal representations offline, providing a more efficient and stable solution. However, existing model-editing techniques typically rely on a single global subspace to correct hallucinations, treating all test samples identically and failing to capture diverse hallucination modes across inputs. To address this limitation, we propose a training-free hallucination mitigation framework for dynamic, per-instance suppression at test time. Our method first constructs a set of Disentangled Hallucination Subspaces, each isolating a distinct hallucination mode. During inference, the model adaptively calculates weights reflecting each input's relationship to these subspaces, guiding a dynamically combined projection that selectively suppresses the most probable hallucination directions while preserving image-grounded semantics. Extensive experiments across multiple vision-language benchmarks and LVLM families demonstrate consistent improvements, highlighting the robustness, generalizability, and efficiency of our approach.

Abstract

Object Hallucination in VLMs

Explore similar work

Beyond Global Editing: Per-Instance Disentangled Subspaces for Training-Free Hallucination Mitigation in LVLMs

NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors

HulluEdit: Single-Pass Evidence-Consistent Subspace Editing for Mitigating Hallucinations in Large Vision-Language Models