Do Not Leave a Gap: Hallucination-Free Object Concealment in Vision-Language Models

Amira Guesmi; Muhammad Shafique

cs.CRMar 16, 2026

Do Not Leave a Gap: Hallucination-Free Object Concealment in Vision-Language Models

Authors: Amira Guesmi, Muhammad Shafique

Organizations: Engineering Division, New York University Abu Dhabi, UAE

Abstract

Vision-language models (VLMs) have recently shown remarkable capabilities in visual understanding and generation, but remain vulnerable to adversarial manipulations of visual content. Prior object-hiding attacks primarily rely on suppressing or blocking region-specific representations, often creating semantic gaps that inadvertently induce hallucination, where models invent plausible but incorrect objects. In this work, we demonstrate that hallucination arises not from object absence per se, but from semantic discontinuity introduced by such suppression-based attacks. We propose a new class of background-consistent object concealment attacks, which hide target objects by re-encoding their visual representations to be statistically and semantically consistent with surrounding background regions. Crucially, our approach preserves token structure and attention flow, avoiding representational voids that trigger hallucination. We present a pixel-level optimization framework that enforces background-consistent re-encoding across multiple transformer layers while preserving global scene semantics. Extensive experiments on state-of-the-art vision-language models show that our method effectively conceals target objects while preserving up to $86\%$ of non-target objects and reducing grounded hallucination by up to $3\times$ compared to attention-suppression-based attacks.

Figures & tables

Figure 1: Overview of Background-Consistent Re-encoding (BCR) . A pixel-level perturbation $\delta$ is optimized to produce an adversarial image $\mathbf{x}^{adv}$ from a clean image $\mathbf{x}$ with a specified ROI. A frozen vision encoder extracts layer-wise hidden states, from which ROI and background tokens are identified via patch embedding. BCR enforces semantic continuity by (i) aligning ROI and background statistics, (ii) softly projecting ROI features onto background representations, and (iii) preserving background tokens between clean and adversarial images. A total variation regularizer encourages smooth perturbations. The objective is optimized across multiple transformer layers, while the language model remains frozen and is used only for evaluation.

Dataset	Method	Instruct-BLIP				BLIP2-T5				LLaVA
		C $\uparrow$	GP $\uparrow$	GH $\downarrow$	SD $\downarrow$	C $\uparrow$	GP $\uparrow$	GH $\downarrow$	SD $\downarrow$	C $\uparrow$	GP $\uparrow$	GH $\downarrow$	SD $\downarrow$
ImageNet	No Attack	0.00	1.00	0.00	0.00	0.00	1.00	0.00	0.00	0.00	1.00	0.00	0.00
	Masking	1.00	0.41	0.59	0.54	1.00	0.52	0.48	0.40	1.00	0.55	0.45	0.41
	PRM	0.66	0.38	0.62	0.44	0.70	0.31	0.69	0.41	0.73	0.34	0.66	0.39
	VIP	0.93	0.57	0.43	0.35	0.96	0.59	0.41	0.32	0.98	0.62	0.48	0.29
	BCR (Ours)	0.92	0.81	0.19	0.13	0.95	0.83	0.17	0.10	0.98	0.86	0.14	0.08

Table 1: Multi-model evaluation on ImageNet and COCO subsets. C : Concealment success, GP : Global Preservation, GH : Grounded Hallucination rate (lower is better), SD : Semantic Drift.

Figure 2: Qualitative comparison of object concealment attacks on InstructBLIP.

Figure 3: Failure cases of pixel-space obfuscation methods. When the target object is masked or blurred, the resulting visual artifacts introduce ambiguous signals that vision–language models attempt to explain. Despite the absence of the original object, the model generates hallucinated descriptions containing unrelated entities such as dogs or cows.

Model	Instruct-BLIP		BLIP2-T5		LLaVA
Method	SSIM $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	LPIPS $\downarrow$
PRM	0.93	0.06	0.93	0.06	0.92	0.07
VIP	0.71	0.34	0.804	0.175	0.65	0.32
BCR (Ours)	0.96	0.02	0.95	0.03	0.94	0.04

Table 2: Perceptual fidelity between clean and adversarial images. Higher SSIM and lower LPIPS indicate better visual similarity.

$\mathcal{L}_{\text{stat}}$	$\mathcal{L}_{\text{dict}}$	$\mathcal{L}_{\text{pres}}$	$\mathcal{L}_{\text{tv}}$	CS $\uparrow$	HR $\downarrow$	GP $\uparrow$
✓	✓	✓	✓	0.91	0.08	0.83
✓	✓	✓		0.90	0.09	0.81
✓	✓		✓	0.86	0.17	0.69
✓		✓	✓	0.79	0.21	0.74
	✓	✓	✓	0.72	0.28	0.66
VIP-style suppression baseline				0.88	0.42	0.51

Table 3: Ablation study of BCR components on Instruct-BLIP.

\checkmark

indicates the loss term is enabled. We report Concealment Success (CS

\uparrow

), Hallucination Rate (HR

\downarrow

), and Global Preservation (GP

\uparrow

Figure 4: Sensitivity analysis of targeted transformer layers. Early-layer optimization leaves object semantics largely intact, allowing the model to still recognize the target object. In contrast, targeting deeper layers removes the object semantics while preserving the surrounding scene context, resulting in successful concealment.

Targeted Layers	Concealment Success $\uparrow$	Hallucination Rate $\downarrow$
Early Layers	0.22	0.07
Middle Layers	0.41	0.11
Late Layers	0.91	0.18

Table 4: Quantitative sensitivity analysis of targeted transformer layers on Instruct-BLIP. Later-layer optimization yields stronger concealment and lower hallucination.

Figure 5: Additional qualitative comparisons between suppression-based attacks and BCR. While suppression-based methods often introduce hallucinated or semantically unrelated objects, BCR consistently removes the target object while preserving the overall scene structure and contextual elements.

Adversarial Attacks on VLMs

Adversarial attacks on vision-language models exploit images, text, and multimodal interactions to manipulate outputs, bypass safety, or degrade reliability.

See all 182 papers

Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration
May 24, 2026Yuanzhi Xu, Qian Gao, Jun Fan +4VLM Robustness Vision-Language Models
Qilu University of Technology (Shandong Academy of Sciences) Jinan, China · China Telecom Digital Intelligence Technology Co, Ltd Jinan, China · Shenyang Aerospace University Shenyang, China +1
Hallucinations Leave a Grounding Signature:Verifier-Guided Decoding for Selective Object Correction
Jul 30, 2026Lei Yang, Xinze Liu, Dayan Wu +7Hallucination Detection in VLMs LLM Hallucination Mitigation
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · Department of Applied Mathematics and Statistics, Johns Hopkins University, Baltimore, MD, USA
Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination
Aug 7, 2026Zichuan Wang, Songlin Yang, Bo Peng +4Hallucination Detection in VLMs Vision-Language Models
School of Artificial Intelligence, University of Chinese Academy of Sciences · 3Hong Kong University of Science and Technology · 2New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Science

May 24, 2026J/K move · Enter open · S save

Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration

Yuanzhi Xu, Qian Gao, Jun Fan, Guohui Ding+3

Qilu University of Technology (Shandong Academy of Sciences) Jinan, China · China Telecom Digital Intelligence Technology Co, Ltd Jinan, China · Shenyang Aerospace University Shenyang, China+1

VLM Robustness Vision-Language Models Hallucination in Language Models Object Hallucination in VLMs VLM Hallucination Mitigation Attention Intervention in VLMs+1

Open paper AskPDF arXiv

The generation of factually incorrect objects, commonly known as object hallucination, remains a persistent challenge in Large Vision-Language Models (LVLMs). Current approaches to address this issue - ranging from expensive data-driven fine-tuning and high-latency contrastive decoding to rigid attention head truncation - frequently compromise either computational efficiency or the continuity of the model's feature space. To overcome these limitations, we introduce a novel, training-free inference strategy that operates as a region-aware adaptive weighting mechanism to dynamically correct semantic drift without relying on abrupt heuristic truncations. By computing an outlier-resistant statistical midpoint across various attention heads, we establish a stable anchor for reliable visual representations. We then utilize the inter-head disagreement mapped across regions to dynamically determine intervention budgets, gently suppressing hallucination-inducing attention paths through a continuous penalty modulation. This recalibration process effectively rectifies visual-semantic misalignments while fully preserving generative fluency and language priors. Comprehensive evaluations on standard multimodal benchmarks, including CHAIR, POPE, and MME, reveal that our strategy substantially curtails both instance- and sentence-level hallucinations. The results demonstrate state-of-the-art performance against contemporary baselines, confirming our method's efficiency and algorithmic robustness. Our code will be public.

Abstract

Adversarial Attacks on VLMs

Explore similar work

Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration

Hallucinations Leave a Grounding Signature:Verifier-Guided Decoding for Selective Object Correction

Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination