Fighting Hallucinations with Counterfactuals: Diffusion-Guided Perturbations for LVLM Hallucination Suppression

Hamidreza Dastmalchi; Aijun An; Ali Cheraghian; Hamed Barzamini

cs.CVMar 11, 2026

Fighting Hallucinations with Counterfactuals: Diffusion-Guided Perturbations for LVLM Hallucination Suppression

Authors: Hamidreza Dastmalchi, Aijun An, Ali Cheraghian, Hamed Barzamini

Organizations: York University Toronto, Canada · Macquarie University Sydney, Australia · Northern Illinois University Illinois, USA

Abstract

While large vision-language models (LVLMs) achieve strong performance on multimodal tasks, they frequently generate hallucinations -- unfaithful outputs misaligned with the visual input. To address this issue, we introduce CIPHER (Counterfactual Image Perturbations for Hallucination Extraction and Removal), a training-free method that suppresses vision-induced hallucinations via lightweight feature-level correction. Unlike prior training-free approaches that primarily focus on text-induced hallucinations, CIPHER explicitly targets hallucinations arising from the visual modality. CIPHER operates in two phases. In the offline phase, we construct OHC-25K (Object-Hallucinated Counterfactuals, 25,000 samples), a counterfactual dataset consisting of diffusion-edited images that intentionally contradict the original ground-truth captions. We pair these edited images with the unchanged ground-truth captions and process them through an LVLM to extract hallucination-related representations. Contrasting these representations with those from authentic (image, caption) pairs reveals structured, systematic shifts spanning a low-rank subspace characterizing vision-induced hallucination. In the inference phase, CIPHER suppresses hallucinations by projecting intermediate hidden states away from this subspace. Experiments across multiple benchmarks show that CIPHER significantly reduces hallucination rates while preserving task performance, demonstrating the effectiveness of counterfactual visual perturbations for improving LVLM faithfulness. Code and additional materials are available at https://hamidreza-dastmalchi.github.io/cipher-cvpr2026/.

Figures & tables

Figure 1: Given an image–caption pair $(\boldsymbol{I},\mathcal{C})$ , we generate a counterfactual image $\tilde{\boldsymbol{I}}$ using a diffusion model conditioned on a GPT-perturbed caption $\tilde{\mathcal{C}}$ via controlled perturbations. A vision–language model encodes both $(\boldsymbol{I},\mathcal{C})$ and $(\tilde{\boldsymbol{I}},\mathcal{C})$ , yielding features $\boldsymbol{h}$ and $\tilde{\boldsymbol{h}}$ . Their difference $\boldsymbol{\delta}=\tilde{\boldsymbol{h}}-\boldsymbol{h}$ captures the hallucination direction we later nullify at inference.

Figure 2: (a) Hallucinated image generation: given an image $\boldsymbol{I}_{i}$ and its ground-truth caption $\mathcal{C}_{i}$ , a GPT model generates a hallucinated caption $\tilde{\mathcal{C}}_{i}$ . The image is then encoded by the encoder $\mathcal{E}$ of the Stable Diffusion Model (SDM), and both forward and reverse diffusion steps are applied, conditioned on $\tilde{\mathcal{C}}_{i}$ , to produce hallucinated images $\tilde{\boldsymbol{I}}_{i,j}$ . (b) Estimating hallucination subspace: the LVLM encodes both hallucinated and ground-truth image–caption pairs to extract hidden states. Feature differences $\{\boldsymbol{\delta}_{\ell}^{(i)}\}_{i=1}^{M}$ are computed, stacked, and decomposed via SVD. The top $r$ right-singular vectors are retained in a hallucination basis bank for inference-time suppression.

Figure 3: Inference-time projection mechanism. Given an input image and instruction, the model generates text autoregressively. At each decoding step during generation, hidden states from selected layers are projected onto the subspace orthogonal to the corresponding hallucination space, using the hallucination basis bank obtained in the offline phase.

Method	LLaVA-1.5			MiniGPT-4			mPLUG-Owl2
	CHAIR ${}_{S}\downarrow$	CHAIR ${}_{I}\downarrow$	BLEU $\uparrow$	CHAIR ${}_{S}\downarrow$	CHAIR ${}_{I}\downarrow$	BLEU $\uparrow$	CHAIR ${}_{S}\downarrow$	CHAIR ${}_{I}\downarrow$	BLEU $\uparrow$
Greedy	20.40 ±2.80	7.08 ±0.33	15.72 ±0.10	32.40 ±2.20	12.20 ±0.42	14.57 ±0.11	22.90 ±0.90	8.62 ±0.11	15.01 ±0.24
Beam Search	19.50 ±2.30	6.84 ±0.79	15.99 ±0.14	30.10 ±0.30	11.87 ±0.37	15.35 ±0.24	20.30 ±0.70	7.62 ±0.19	15.43 ±0.05
DoLa (ICLR’24)	20.20 ±2.80	6.75 ±0.54	15.68 ±0.10	31.90 ±3.30	12.15 ±0.89	14.54 ±0.12	22.40 ±1.80	8.36 ±0.04	15.13 ±0.21
OPERA (CVPR’24)	17.50 ±0.50	6.07 ±0.32	16.02 ±0.02	29.70 ±0.30	11.96 ±0.29	14.82 ±0.05	20.07 ±2.07	7.18 ±0.39	15.41 ±0.12
VCD (CVPR’24)	20.30 ±1.10	7.28 ±0.10	14.53 ±0.01	29.00 ±2.80	12.64 ±1.19	14.42 ±0.01	22.80 ±0.80	8.68 ±0.17	15.14 ±0.13

Table 1: CHAIR and BLEU scores across LVLMs; lower CHAIR = less hallucination, higher BLEU = better fluency.

Method	LLaVA-1.5			MiniGPT-4			mPLUG-Owl2
	Accuracy $\uparrow$	Precision $\uparrow$	F score $\uparrow$	Accuracy $\uparrow$	Precision $\uparrow$	F score $\uparrow$	Accuracy $\uparrow$	Precision $\uparrow$	F score $\uparrow$
Greedy	79.14 $\pm$ 0.89	91.98 $\pm$ 0.82	90.45 $\pm$ 0.86	71.22 $\pm$ 1.27	93.72 $\pm$ 1.02	90.04 $\pm$ 1.23	76.46 $\pm$ 0.92	88.85 $\pm$ 1.15	87.29 $\pm$ 1.15
Beam Search	79.41 $\pm$ 0.69	92.52 $\pm$ 0.55	90.96 $\pm$ 0.59	71.65 $\pm$ 1.15	94.70 $\pm$ 0.60	90.97 $\pm$ 0.85	76.76 $\pm$ 1.02	90.28 $\pm$ 0.80	88.56 $\pm$ 0.87
DoLa (ICLR’24)	78.98 $\pm$ 0.56	91.66 $\pm$ 0.81	90.15 $\pm$ 0.79	71.28 $\pm$ 1.15	93.92 $\pm$ 0.83	90.22 $\pm$ 1.04	76.07 $\pm$ 1.09	88.54 $\pm$ 1.25	86.95 $\pm$ 1.27
OPERA (CVPR’24)	79.29 $\pm$ 0.32	92.25 $\pm$ 0.07	90.71 $\pm$ 0.11	70.48 $\pm$ 1.63	94.41 $\pm$ 1.11	90.66 $\pm$ 1.42	75.49 $\pm$ 1.29	91.23 $\pm$ 1.06	89.11 $\pm$ 1.17
VCD (CVPR’24)	78.01 $\pm$ 0.75	91.33 $\pm$ 0.88	89.69 $\pm$ 0.89	70.83 $\pm$ 1.83	92.31 $\pm$ 0.88	88.76 $\pm$ 1.29	75.49 $\pm$ 1.27	88.75 $\pm$ 1.56	87.02 $\pm$ 1.57

Table 2: Accuracy, precision, and F1 scores of different methods on the OPOPE benchmark, evaluated across LLaVA-1.5, MiniGPT-4, and mPLUG-Owl2.

Model	Method	Accuracy $\uparrow$	Detailedness $\uparrow$
LLaVA-1.5	Original	6.79 ±1.50	6.33 ±1.18
LLaVA-1.5	CIPHER	7.08 ±1.32	6.75 ±1.13
MiniGPT-4	Original	5.54 ±2.41	5.58 ±2.06
MiniGPT-4	CIPHER	6.16 ±1.99	6.64 ±1.88
mPLUG-Owl2	Original	6.48 ±2.12	6.26 ±1.62
mPLUG-Owl2	CIPHER	7.35 ±1.63	6.91 ±1.61

Table 3: GPT-4V evaluations on LLaVA-Bench indicate that CIPHER consistently enhances model outputs, yielding higher accuracy and more detailed answers.

Figure 5: LLaVA-Bench example: CIPHER reduces hallucinations and improves grounding.

Figure 6: Layer-wise linear probing performance comparing textual and diffusion-based visual hallucination perturbations. Textual perturbations exhibit moderate and unstable separability across layers, whereas diffusion-based visual perturbations produce consistently high accuracy, recall, and F1.

Metric	Greedy	Beam	DoLa	VCD	OPERA	HALC	Nullu	CIPHER
CHAIR S $\downarrow$ (%)	20.40	19.50	20.20	20.30	17.50	16.90	15.20	13.05
Throughput $\uparrow$ (items/s)	0.70	0.55	0.42	0.35	0.10	0.05	0.70	0.70

Table 4: Comparison of CHAIR S and throughput (items/s) for different mitigation methods, tested on LLaVA-7B with an NVIDIA A6000 GPU.

Figure 7: CHAIR S of CIPHER using hallucination subspaces derived from images perturbed at different diffusion steps.

Figure 8: Ablation study on subspace rank ( $r$ )

Text	Image	$\text{CHAIR}_{S}\downarrow$	$\text{CHAIR}_{I}\downarrow$	$\text{BLEU}\uparrow$
✓	✗	15.20	5.30	15.69
✗	✓	13.05	4.53	15.82
✓	✓	15.71	5.32	15.66

Table 5: Ablation study on the source of hallucination. ✓denotes hallucinated input and ✗ denotes ground-truth input.

Figure 9: CHAIR S of the original and CIPHER-refined models under increasing levels of Gaussian noise.

Parameters	Values
General Settings
Temperature	0
Number of Beams	3
Maximum New Tokens (CHAIR)	64
Maximum New Tokens (OPOPE)	256
Maximum New Tokens (LLaVA-Bench)	1024

Table 6: General and model-specific parameter settings.

Setting	Model	Method	Accuracy	Precision	Recall	F Score
Random	LLaVA-1.5	Greedy	81.52	98.41	64.07	96.42
		DoLa	81.38	98.11	64.00	96.14
		OPERA	81.62	98.57	64.17	96.58
		VCD	80.57	98.41	62.13	96.25
		HALC	79.58	98.21	60.27	95.89
		Nullu	81.18	98.05	63.63	96.05

Table 7: Performance comparison of various hallucination mitigation methods across different models and evaluation settings.

Figure 11: Effect of subspace rank $r$ on CHAIR S , CHAIR I , and BLEU for MiniGPT-4 on the CHAIR benchmark. Using $r=64$ yields the best performance, minimizing hallucinations with minimum BLEU degradation.

Figure 12: Effect of subspace rank $r$ on CHAIR S , CHAIR I , and BLEU for mPLUG-Owl2 on the CHAIR benchmark. Using $r=32$ yields the best performance, minimizing hallucinations with minimum BLEU degradation.

Layer Range	CHAIR S $\downarrow$	CHAIR I $\downarrow$
8–16	13.52	4.82
8–24	14.10	5.02
16–24	18.10	5.86
16–32	13.05	4.53
20–32	14.10	4.84
24–32	14.00	4.64

Table 8: Ablation study on the transformer layer range used for projection in VISTA.

Figure 14: Example of GPT-4V-based evaluation for image description.

VLM Hallucination

Vision-language model hallucination is the production of claims unsupported by or in conflict with visual evidence.

See all 71 papers

Attention-space Contrastive Guidance for Efficient Hallucination Mitigation in LVLMs
Jan 20, 2026Yujin Jo, Sangyoon Bae, Taesup KimVLM Hallucination Mitigation Object Hallucination in VLMs
Seoul National University
Look Clearly Before Answering: Mitigating Hallucinations in LVLMs via Saliency-Driven Perceptual Realignment
Jul 18, 2026Pengxu Chen, Yao Zhu, Guangming Zhu +4Large Vision-Language Models VLM Hallucination Mitigation
Xidian University Xi’an, China · Tsinghua University Beijing, China · Shanghai Road Transport Development Center Shanghai, China +1
Beyond Global Editing: Per-Instance Disentangled Subspaces for Training-Free Hallucination Mitigation in LVLMs
Aug 10, 2026Ali Cheraghian, Hamidreza Dastmalchi, Hamed Barzamini +4Large Vision-Language Models Model Editing
Macquarie University, Australia · Australian National University, Australia · York University, Canada +4

Jan 20, 2026J/K move · Enter open · S save

Attention-space Contrastive Guidance for Efficient Hallucination Mitigation in LVLMs

Yujin Jo, Sangyoon Bae, Taesup Kim

Seoul National University

VLM Hallucination Mitigation Object Hallucination in VLMs Faithfulness in VLMs Attention Intervention in VLMs Contrastive Decoding

Open paper AskPDF arXiv

Hallucinations in large vision--language models (LVLMs) often arise when language priors dominate over visual evidence, leading to object misidentification and visually inconsistent descriptions. We address this problem by framing hallucination mitigation as contrastive guidance that steers generation toward visually grounded and semantically faithful text. We propose Attention-space Contrastive Guidance (ACG), a training-free, single-pass method that operates directly in self-attention layers, where hallucination-inducing cross-modal biases emerge. ACG constructs both image-conditioned and approximate text-only attention paths within a single forward pass, enabling efficient guidance before errors accumulate at the output layer. Because this masking-based surrogate can introduce approximation bias, we further apply a lightweight orthogonal projection that suppresses components aligned with the text-only path, yielding a more visually grounded correction. Experiments on CHAIR and POPE show that ACG improves faithfulness over existing training-free baselines while maintaining caption quality, reducing latency by up to

2\times

compared to multi-pass contrastive decoding methods.

Abstract

VLM Hallucination

Explore similar work

Attention-space Contrastive Guidance for Efficient Hallucination Mitigation in LVLMs

Look Clearly Before Answering: Mitigating Hallucinations in LVLMs via Saliency-Driven Perceptual Realignment

Beyond Global Editing: Per-Instance Disentangled Subspaces for Training-Free Hallucination Mitigation in LVLMs