cs.CVFeb 25, 2026

Beyond Dominant Patches: Spatial Credit Redistribution For Grounded Vision-Language Models

Authors: Niamul Hassan Samin, Md Arifur Rahman, Abdullah Ibne Hanif Arean, Juena Ahmed Noshin, Md Ashikur Rahman

Organizations: The KOW Company · University of Dhaka · American International University Bangladesh (AIUB)

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Vision-Language Models (VLMs) often hallucinate objects that are not present in the input image. We identify a contributing cause of this behavior, which we term spatial credit collapse: in early transformer layers, hidden-state activation concentrates on a small number of visual patches, suppressing surrounding contextual evidence and increasing reliance on language priors. Across seven models we observe a strong correlation between visual attention entropy and hallucination rate (r = -0.65, p < 0.001), suggesting that reduced spatial credit diversity contributes to hallucination. To address this issue we propose Spatial Credit Redistribution (SCR), a training-free inference-time method. SCR uses a lightweight two-pass procedure. A diagnostic pass identifies the top-K high-attention source patches and their spatial neighbors. A redistribution pass then scales each source by 1/lambda (~0.91) and injects a (lambda - 1) weighted copy of its hidden state into neighboring patches, restoring suppressed visual context without modifying model weights. Because the diagnostic pass is performed once per image and reused across the output sequence, the added latency is negligible (<0.5 ms per token for 100-token responses). We evaluate SCR across seven model configurations from four VLM families (Chameleon, LLaVA-1.5, Qwen-VL/Qwen2-VL, and InternVL2) on five benchmarks: POPE, CHAIR, MME, HallusionBench, and AMBER. SCR reduces POPE-Adversarial hallucination by 4.6-6.0 percentage points and CHAIR-s by 41-51 percent while preserving caption quality (CIDEr drop <=0.8). Compared with prior inference-time methods including OPERA, VCD, OA-VCD, DoLa, VLI, SID, and CRoPS, SCR achieves a better trade-off between hallucination reduction, generation quality, and latency.

Figures & tables

Figure 1: SCR methodology overview. (a) Full two-pass inference pipeline. (b) Spatial credit redistribution via 8-connected neighborhood structure.
ModelMethodHR ↓\downarrowAcc ↑\uparrowCIDEr ↑\uparrowΔ\Delta HR ↓\downarrow
Cham-7BVanilla19.5 ±\pm .880.2 ±\pm .7102.3 ±\pm 1.4-
OPERA16.8 ±\pm 1.082.8 ±\pm .7100.3 ±\pm 1.3−- 2.7
DoLa17.2 ±\pm 1.082.5 ±\pm .8101.8 ±\pm 1.4−- 2.3
VCD15.1 ±\pm 1.384.6 ±\pm 1.097.4 ±\pm 2.0−- 4.4
OA-VCD14.7 ±\pm 1.185.0 ±\pm .999.2 ±\pm 1.5−- 4.8
VLI †15.8 ±\pm .983.8 ±\pm .8100.6 ±\pm 1.1−- 3.7
Table 1: POPE-Adversarial HR (% ↓\downarrow ), Accuracy (% ↑\uparrow ), CIDEr ( ↑\uparrow ) for 7B models. Mean ±\pm std over 3 runs. Green = best HR; Blue = second best. Δ\Delta HR relative to Vanilla. Uniform-Smooth : same SCR operation with random (not attention-guided) source selection. † Concurrent preprint, author-reimplemented (no official code; see § 4 ).
ModelMethodHR ↓\downarrowAcc ↑\uparrowCIDEr ↑\uparrowΔ\Delta HR ↓\downarrow
Cham-30BVanilla18.2 ±\pm .781.5 ±\pm .6108.2 ±\pm 1.2-
OPERA15.4 ±\pm 1.084.2 ±\pm .6106.8 ±\pm 1.1−- 2.8
DoLa16.0 ±\pm .983.6 ±\pm .7107.6 ±\pm 1.1−- 2.2
VCD14.1 ±\pm 1.185.6 ±\pm .8103.8 ±\pm 1.7−- 4.1
OA-VCD13.7 ±\pm 1.086.1 ±\pm .7105.2 ±\pm 1.1−- 4.5
VLI †14.9 ±\pm .884.8 ±\pm .6106.2 ±\pm 1.0−- 3.3
Table 2: POPE-Adversarial HR (% ↓\downarrow ), Accuracy (% ↑\uparrow ), CIDEr ( ↑\uparrow ) for large-scale models. Mean ±\pm std over 3 runs. Color coding as Tab. 1 . On LLaVA-13B, our CRoPS † reproduction reaches the lowest HR but degrades CIDEr by 4.0 pp vs. SCR’s 0.7 pp; SCR Pareto-dominates on all three axes per our reproduction.
Figure 2: POPE-Adversarial HR for all seven VLM configurations. SCR reduces HR by ≈{\approx} 4.6-6.0 pp across all model families.
ModelRandomPopularAdversarial
Chameleon-7B12.5 →\to 8.515.8 →\to 10.919.5 →\to 13.5
Chameleon-30B11.2 →\to 7.814.5 →\to 9.418.2 →\to 12.6
LLaVA-1.5-7B11.8 →\to 8.615.0 →\to 10.418.8 →\to 13.1
LLaVA-1.5-13B10.5 →\to 7.413.8 →\to 10.017.5 →\to 12.8
Qwen-VL11.5 →\to 8.214.8 →\to 10.118.5 →\to 12.9
Qwen2-VL-7B10.2 →\to 6.813.5 →\to 9.217.0 →\to 11.4
Table 3: POPE split analysis (HR%) across model architectures (Vanilla →\to SCR). Adversarial split matches Tabs. 1 and 2 ; consistent gains across all splits and architectures.
ModelMethodCHAIR-s ↓\downarrowCHAIR-i ↓\downarrowCIDEr ↑\uparrow
Cham-7BVanilla12.3%8.5%102.3
VCD9.8%6.8%97.4
OA-VCD9.5%6.3%99.2
CRoPS †9.1%5.8%98.2
SCR7.1%4.8%101.7
Cham-30BVanilla10.8%7.2%108.2
Table 4: CHAIR on COCO captions (3,000 images, val2014). Lower CHAIR is better; higher CIDEr is better. Key baselines shown; full eight-baseline per-model comparison in Appendix B. Green = best per group; Blue = second best. SCR achieves the best CHAIR-s/i in five of seven configurations; on LLaVA-1.5-13B and InternVL2-7B, our CRoPS † reproduction achieves marginally lower CHAIR-s/i but degrades CIDEr by ≈4{\approx}4 pp, versus SCR’s ≤0.8{\leq}0.8 pp. SCR is the only method that simultaneously achieves competitive hallucination reduction and near-lossless generation quality across all configurations.
ModelMME-P ↑\uparrowHBench ↑\uparrowAMBER-O ↓\downarrowAMBER-A ↓\downarrowAMBER-R ↓\downarrow
Base →\to SCRBase →\to SCRBase →\to SCRBase →\to SCRBase →\to SCR
Cham-7B1285 →\to 135828.5 →\to 31.218.5 →\to 12.822.1 →\to 19.825.3 →\to 24.2
Cham-30B1342 →\to 141830.1 →\to 33.017.0 →\to 11.620.8 →\to 18.524.0 →\to 23.0
LLaVA-7B1510 →\to 157631.2 →\to 34.115.8 →\to 10.819.5 →\to 17.623.0 →\to 22.1
LLaVA-13B1588 →\to 164234.8 →\to 37.414.5 →\to 10.217.8 →\to 15.921.5 →\to 20.8
Qwen-VL1548 →\to 161233.5 →\to 36.015.5 →\to 10.818.6 →\to 16.522.8 →\to 21.4
Table 5: Extended benchmark evaluation (Vanilla  → \,\to\, SCR). MME-P = MME Perception ( ↑\uparrow ); HBench = HallusionBench aAcc (% ↑\uparrow ); AMBER-O/A/R = AMBER [ 34 ] object/attribute/relation hallucination rate (% ↓\downarrow ). SCR yields strong object-hallucination reductions (AMBER-O: −- 4.1-5.7 pp), moderate attribute gains (AMBER-A: −- 1.6-2.3 pp), and minimal relational change (AMBER-R: −- 0.7-1.4 pp), consistent with its spatial-grounding scope. InternVL2-7B follows the same pattern, confirming generalization to the latest model family.
ModelPearson rrpp -value
Chameleon-7B−0.68-0.682.4×10−52.4{\times}10^{-5}
Chameleon-30B−0.63-0.634.1×10−44.1{\times}10^{-4}
LLaVA-1.5-7B−0.72-0.728.7×10−68.7{\times}10^{-6}
LLaVA-1.5-13B−0.69-0.691.8×10−51.8{\times}10^{-5}
Qwen-VL−0.61-0.616.3×10−46.3{\times}10^{-4}
Qwen2-VL-7B−0.58-0.581.2×10−31.2{\times}10^{-3}
Table 6: Entropy-hallucination correlation (Pearson rr ) across all seven models. All correlations are negative and significant ( p≤0.001p\leq 0.001 ), confirming the link between low credit entropy and elevated hallucination rate across model families.
Figure 3: Entropy-stratified HR reduction across all seven model configurations. SCR provides the greatest benefit for low-entropy inputs ( H<3.5H<3.5 , avg. 9.8 pp reduction) where credit is most collapsed, and diminishing gains for already-distributed credit ( H>4.5H>4.5 , avg. 2.4 pp), consistent with Design Principle 1.
Figure 4: Ablation studies. (a) Neighbor topology comparison. (b) Semantic structure vs. entropy control. (c) Head-selectivity and computational cost trade-off. See ablation (a) in text for the Uniform-Smooth attention-guidance comparison (Tables 1 - 2 ).
More in this topic

VLM Hallucination Mitigation

Vision-language model hallucination mitigation keeps generated claims grounded in visual evidence.

See all 228 papers

Explore similar work

CardsList
May 28, 2026

Mitigating Hallucination in Vision-Language Models through Barrier-Regulated Adaptive Closed-form Steering

Large vision-language models (LVLMs) often hallucinate objects that are not present in the input image, largely because visual grounding weakens as decoding progresses. Existing inference-time mitigation methods modify logits or hidden states throughout generation, but they suffer from three key limitations: they lack an explicit grounding objective, intervene even when the model is already well-grounded, and use fixed correction strengths that do not adapt to the severity of grounding failure. We propose BRACS (Barrier-Regulated Adaptive Closed-form Steering), a training-free steering framework that addresses these issues through barrier-regulated adaptive closed-form steering. BRACS monitors the model's own attention to measure visual grounding and applies corrections to the hidden states only when grounding deteriorates. The corrective update is computed analytically in closed form, requiring no training of auxiliary networks or model retraining. Experiments on LLaVA-1.5-7B and Qwen-VL-Chat show that BRACS consistently outperforms prior methods on hallucination benchmarks, reducing CHAIRs_s by 9.4 points and improving POPE F1 by 2.7 points, while matching or improving performance on four general multimodal benchmarks. BRACS also remains efficient, operating at 80% of greedy decoding throughput and achieving 1.3 times higher speed on average than the baselines.

Soumyadeep Jana, Pulkit Mittal, Sanasam Ranbir Singh

Indian Institute of Technology Guwahati, India

Hallucination in Language ModelsLarge Vision-Language ModelsObject Hallucination in VLMsVision-Language GroundingActivation SteeringVLM Hallucination Mitigation+1
AskPDF
May 20, 2026

Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy

Large Vision-Language Models (LVLMs) have shown remarkable performance on a wide range of vision-language tasks. Despite this progress, they are still prone to hallucination, generating responses that are inconsistent with visual content. In this work, we find that LVLMs tend to hallucinate when they pay insufficient attention to the correct visual evidence and gradually forget it during the generation process. We empirically find that although LVLMs overall attend insufficiently to visual evidence, they exhibit sensitivity to the correct visual evidence in specific layers, with notable inter-layer discrepancy. Motivated by this observation, we propose a novel hallucination mitigation method that enhances visual evidence based on Inter-Layer Visual Attention Discrepancy (ILVAD). Specifically, we obtain the attention weights from early generated tokens to visual tokens across layers and identify the tokens that are repeatedly activated as visual evidence, forming a saliency map. We then enhance attention to visual evidence during generation through the saliency map to reduce visual forgetting. In addition, we leverage the saliency map to obtain attention scores of generated text to visual evidence, in order to select and emphasize text tokens that are strongly grounded in visual evidence. Our method is training-free and plug-and-play. Multiple benchmark evaluations conducted on five recently released models show that our method can consistently mitigate hallucinations in different LVLMs over various architectures. Code is available at https://github.com/ytx-ML/ILVAD.

Yutong Xie, Zhenglin Hua, Ran Wang +3

School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Artificial Intelligence, Shenzhen University, Shenzhen, China · National Engineering Laboratory for Big Data Systems Computing Technology, Shenzhen University, Shenzhen, China +3

Vision-Language ModelsHallucination in Language ModelsVLM HallucinationLarge Vision-Language ModelsVLM Hallucination Mitigation
AskPDF
Apr 4, 2026

Focus Matters: Phase-Aware Suppression for Hallucination in Vision-Language Models

Large Vision-Language Models (LVLMs) have achieved impressive progress in multimodal reasoning, yet they remain prone to object hallucinations, generating descriptions of objects that are not present in the input image. Recent approaches attempt to mitigate hallucinations by suppressing unreliable visual signals in the vision encoder, but many rely on iterative optimization for each input, resulting in substantial inference latency. In this work, we investigate the internal attention dynamics of vision encoders in LVLMs and identify a consistent three-phase structure of visual information processing: diffusion, focus, and rediffusion. Our analysis reveals that hallucination behavior is particularly sensitive to tokens receiving low attention during the focus phase. Motivated by this observation, we propose a lightweight inference-time intervention that selectively suppresses such tokens during the focus phase. The method operates in a training-free manner using statistics from a single forward pass and employs a Determinantal Point Process (DPP) to preserve diverse visual cues while filtering redundant tokens. Extensive experiments across multiple LVLM backbones and decoding strategies demonstrate that the proposed approach consistently reduces hallucination metrics while maintaining competitive caption quality. Moreover, compared to adversarial uncertainty estimation methods, our approach achieves comparable hallucination mitigation with negligible additional inference latency.

Sohyeon Kim, Sang Yeon Yoon, Kyeongbo Kong

Pusan National University · Pukyong National University

Object Hallucination in VLMsVLM Hallucination Mitigation
AskPDF