Beyond Dominant Patches: Spatial Credit Redistribution For Grounded Vision-Language Models

Niamul Hassan Samin; Md Arifur Rahman; Abdullah Ibne Hanif Arean; Juena Ahmed Noshin; Md Ashikur Rahman

cs.CVFeb 25, 2026

Beyond Dominant Patches: Spatial Credit Redistribution For Grounded Vision-Language Models

Authors: Niamul Hassan Samin, Md Arifur Rahman, Abdullah Ibne Hanif Arean, Juena Ahmed Noshin, Md Ashikur Rahman

Organizations: The KOW Company · University of Dhaka · American International University Bangladesh (AIUB)

Abstract

Vision-Language Models (VLMs) often hallucinate objects that are not present in the input image. We identify a contributing cause of this behavior, which we term spatial credit collapse: in early transformer layers, hidden-state activation concentrates on a small number of visual patches, suppressing surrounding contextual evidence and increasing reliance on language priors. Across seven models we observe a strong correlation between visual attention entropy and hallucination rate (r = -0.65, p < 0.001), suggesting that reduced spatial credit diversity contributes to hallucination. To address this issue we propose Spatial Credit Redistribution (SCR), a training-free inference-time method. SCR uses a lightweight two-pass procedure. A diagnostic pass identifies the top-K high-attention source patches and their spatial neighbors. A redistribution pass then scales each source by 1/lambda (~0.91) and injects a (lambda - 1) weighted copy of its hidden state into neighboring patches, restoring suppressed visual context without modifying model weights. Because the diagnostic pass is performed once per image and reused across the output sequence, the added latency is negligible (<0.5 ms per token for 100-token responses). We evaluate SCR across seven model configurations from four VLM families (Chameleon, LLaVA-1.5, Qwen-VL/Qwen2-VL, and InternVL2) on five benchmarks: POPE, CHAIR, MME, HallusionBench, and AMBER. SCR reduces POPE-Adversarial hallucination by 4.6-6.0 percentage points and CHAIR-s by 41-51 percent while preserving caption quality (CIDEr drop <=0.8). Compared with prior inference-time methods including OPERA, VCD, OA-VCD, DoLa, VLI, SID, and CRoPS, SCR achieves a better trade-off between hallucination reduction, generation quality, and latency.

Figures & tables

Panel of Figure 1 — Figure 1: SCR methodology overview. (a) Full two-pass inference pipeline. (b) Spatial credit redistribution via 8-connected neighborhood structure.

Model	Method	HR $\downarrow$	Acc $\uparrow$	CIDEr $\uparrow$	$\Delta$ HR $\downarrow$
Cham-7B	Vanilla	19.5 $\pm$ .8	80.2 $\pm$ .7	102.3 $\pm$ 1.4	-
	OPERA	16.8 $\pm$ 1.0	82.8 $\pm$ .7	100.3 $\pm$ 1.3	$-$ 2.7
	DoLa	17.2 $\pm$ 1.0	82.5 $\pm$ .8	101.8 $\pm$ 1.4	$-$ 2.3
	VCD	15.1 $\pm$ 1.3	84.6 $\pm$ 1.0	97.4 $\pm$ 2.0	$-$ 4.4
	OA-VCD	14.7 $\pm$ 1.1	85.0 $\pm$ .9	99.2 $\pm$ 1.5	$-$ 4.8
	VLI †	15.8 $\pm$ .9	83.8 $\pm$ .8	100.6 $\pm$ 1.1	$-$ 3.7

Table 1: POPE-Adversarial HR (%

\downarrow

), Accuracy (%

\uparrow

), CIDEr (

\uparrow

) for 7B models. Mean

\pm

std over 3 runs. Green = best HR; Blue = second best.

\Delta

HR relative to Vanilla. Uniform-Smooth : same SCR operation with random (not attention-guided) source selection. † Concurrent preprint, author-reimplemented (no official code; see § 4 ).

Model	Method	HR $\downarrow$	Acc $\uparrow$	CIDEr $\uparrow$	$\Delta$ HR $\downarrow$
Cham-30B	Vanilla	18.2 $\pm$ .7	81.5 $\pm$ .6	108.2 $\pm$ 1.2	-
	OPERA	15.4 $\pm$ 1.0	84.2 $\pm$ .6	106.8 $\pm$ 1.1	$-$ 2.8
	DoLa	16.0 $\pm$ .9	83.6 $\pm$ .7	107.6 $\pm$ 1.1	$-$ 2.2
	VCD	14.1 $\pm$ 1.1	85.6 $\pm$ .8	103.8 $\pm$ 1.7	$-$ 4.1
	OA-VCD	13.7 $\pm$ 1.0	86.1 $\pm$ .7	105.2 $\pm$ 1.1	$-$ 4.5
	VLI †	14.9 $\pm$ .8	84.8 $\pm$ .6	106.2 $\pm$ 1.0	$-$ 3.3

Table 2: POPE-Adversarial HR (%

\downarrow

), Accuracy (%

\uparrow

), CIDEr (

\uparrow

) for large-scale models. Mean

\pm

std over 3 runs. Color coding as Tab. 1 . On LLaVA-13B, our CRoPS † reproduction reaches the lowest HR but degrades CIDEr by 4.0 pp vs. SCR’s 0.7 pp; SCR Pareto-dominates on all three axes per our reproduction.

Figure 2: POPE-Adversarial HR for all seven VLM configurations. SCR reduces HR by

{\approx}

4.6-6.0 pp across all model families.

Model	Random	Popular	Adversarial
Chameleon-7B	12.5 $\to$ 8.5	15.8 $\to$ 10.9	19.5 $\to$ 13.5
Chameleon-30B	11.2 $\to$ 7.8	14.5 $\to$ 9.4	18.2 $\to$ 12.6
LLaVA-1.5-7B	11.8 $\to$ 8.6	15.0 $\to$ 10.4	18.8 $\to$ 13.1
LLaVA-1.5-13B	10.5 $\to$ 7.4	13.8 $\to$ 10.0	17.5 $\to$ 12.8
Qwen-VL	11.5 $\to$ 8.2	14.8 $\to$ 10.1	18.5 $\to$ 12.9
Qwen2-VL-7B	10.2 $\to$ 6.8	13.5 $\to$ 9.2	17.0 $\to$ 11.4

Table 3: POPE split analysis (HR%) across model architectures (Vanilla

\to

SCR). Adversarial split matches Tabs. 1 and 2 ; consistent gains across all splits and architectures.

Model	Method	CHAIR-s $\downarrow$	CHAIR-i $\downarrow$	CIDEr $\uparrow$
Cham-7B	Vanilla	12.3%	8.5%	102.3
	VCD	9.8%	6.8%	97.4
	OA-VCD	9.5%	6.3%	99.2
	CRoPS †	9.1%	5.8%	98.2
	SCR	7.1%	4.8%	101.7
Cham-30B	Vanilla	10.8%	7.2%	108.2

Table 4: CHAIR on COCO captions (3,000 images, val2014). Lower CHAIR is better; higher CIDEr is better. Key baselines shown; full eight-baseline per-model comparison in Appendix B. Green = best per group; Blue = second best. SCR achieves the best CHAIR-s/i in five of seven configurations; on LLaVA-1.5-13B and InternVL2-7B, our CRoPS † reproduction achieves marginally lower CHAIR-s/i but degrades CIDEr by

{\approx}4

pp, versus SCR’s

{\leq}0.8

pp. SCR is the only method that simultaneously achieves competitive hallucination reduction and near-lossless generation quality across all configurations.

Model	MME-P $\uparrow$	HBench $\uparrow$	AMBER-O $\downarrow$	AMBER-A $\downarrow$	AMBER-R $\downarrow$
Model	Base $\to$ SCR	Base $\to$ SCR	Base $\to$ SCR	Base $\to$ SCR	Base $\to$ SCR
Cham-7B	1285 $\to$ 1358	28.5 $\to$ 31.2	18.5 $\to$ 12.8	22.1 $\to$ 19.8	25.3 $\to$ 24.2
Cham-30B	1342 $\to$ 1418	30.1 $\to$ 33.0	17.0 $\to$ 11.6	20.8 $\to$ 18.5	24.0 $\to$ 23.0
LLaVA-7B	1510 $\to$ 1576	31.2 $\to$ 34.1	15.8 $\to$ 10.8	19.5 $\to$ 17.6	23.0 $\to$ 22.1
LLaVA-13B	1588 $\to$ 1642	34.8 $\to$ 37.4	14.5 $\to$ 10.2	17.8 $\to$ 15.9	21.5 $\to$ 20.8
Qwen-VL	1548 $\to$ 1612	33.5 $\to$ 36.0	15.5 $\to$ 10.8	18.6 $\to$ 16.5	22.8 $\to$ 21.4

Table 5: Extended benchmark evaluation (Vanilla

\,\to\,

SCR). MME-P = MME Perception (

\uparrow

); HBench = HallusionBench aAcc (%

\uparrow

); AMBER-O/A/R = AMBER [ 34 ] object/attribute/relation hallucination rate (%

\downarrow

). SCR yields strong object-hallucination reductions (AMBER-O:

-

4.1-5.7 pp), moderate attribute gains (AMBER-A:

-

1.6-2.3 pp), and minimal relational change (AMBER-R:

-

0.7-1.4 pp), consistent with its spatial-grounding scope. InternVL2-7B follows the same pattern, confirming generalization to the latest model family.

Model	Pearson $r$	$p$ -value
Chameleon-7B	$-0.68$	$2.4{\times}10^{-5}$
Chameleon-30B	$-0.63$	$4.1{\times}10^{-4}$
LLaVA-1.5-7B	$-0.72$	$8.7{\times}10^{-6}$
LLaVA-1.5-13B	$-0.69$	$1.8{\times}10^{-5}$
Qwen-VL	$-0.61$	$6.3{\times}10^{-4}$
Qwen2-VL-7B	$-0.58$	$1.2{\times}10^{-3}$

Table 6: Entropy-hallucination correlation (Pearson

r

) across all seven models. All correlations are negative and significant (

p\leq 0.001

), confirming the link between low credit entropy and elevated hallucination rate across model families.

Figure 3: Entropy-stratified HR reduction across all seven model configurations. SCR provides the greatest benefit for low-entropy inputs (

H<3.5

, avg. 9.8 pp reduction) where credit is most collapsed, and diminishing gains for already-distributed credit (

H>4.5

, avg. 2.4 pp), consistent with Design Principle 1.

Panel of Figure 4 — Figure 4: Ablation studies. (a) Neighbor topology comparison. (b) Semantic structure vs. entropy control. (c) Head-selectivity and computational cost trade-off. See ablation (a) in text for the Uniform-Smooth attention-guidance comparison (Tables 1 - 2 ).

VLM Hallucination Mitigation

Vision-language model hallucination mitigation keeps generated claims grounded in visual evidence.

See all 228 papers

Mitigating Hallucination in Vision-Language Models through Barrier-Regulated Adaptive Closed-form Steering
May 28, 2026Soumyadeep Jana, Pulkit Mittal, Sanasam Ranbir SinghHallucination in Language Models Large Vision-Language Models
Indian Institute of Technology Guwahati, India
Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy
May 20, 2026Yutong Xie, Zhenglin Hua, Ran Wang +3Vision-Language Models Hallucination in Language Models
School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Artificial Intelligence, Shenzhen University, Shenzhen, China · National Engineering Laboratory for Big Data Systems Computing Technology, Shenzhen University, Shenzhen, China +3
Focus Matters: Phase-Aware Suppression for Hallucination in Vision-Language Models
Apr 4, 2026Sohyeon Kim, Sang Yeon Yoon, Kyeongbo KongObject Hallucination in VLMs VLM Hallucination Mitigation
Pusan National University · Pukyong National University

May 28, 2026J/K move · Enter open · S save

Mitigating Hallucination in Vision-Language Models through Barrier-Regulated Adaptive Closed-form Steering

Soumyadeep Jana, Pulkit Mittal, Sanasam Ranbir Singh

Indian Institute of Technology Guwahati, India

Hallucination in Language Models Large Vision-Language Models Object Hallucination in VLMs Vision-Language Grounding Activation Steering VLM Hallucination Mitigation+1

Open paper AskPDF arXiv

Large vision-language models (LVLMs) often hallucinate objects that are not present in the input image, largely because visual grounding weakens as decoding progresses. Existing inference-time mitigation methods modify logits or hidden states throughout generation, but they suffer from three key limitations: they lack an explicit grounding objective, intervene even when the model is already well-grounded, and use fixed correction strengths that do not adapt to the severity of grounding failure. We propose BRACS (Barrier-Regulated Adaptive Closed-form Steering), a training-free steering framework that addresses these issues through barrier-regulated adaptive closed-form steering. BRACS monitors the model's own attention to measure visual grounding and applies corrections to the hidden states only when grounding deteriorates. The corrective update is computed analytically in closed form, requiring no training of auxiliary networks or model retraining. Experiments on LLaVA-1.5-7B and Qwen-VL-Chat show that BRACS consistently outperforms prior methods on hallucination benchmarks, reducing CHAIR

_s

by 9.4 points and improving POPE F1 by 2.7 points, while matching or improving performance on four general multimodal benchmarks. BRACS also remains efficient, operating at 80% of greedy decoding throughput and achieving 1.3 times higher speed on average than the baselines.

Abstract

VLM Hallucination Mitigation

Explore similar work

Mitigating Hallucination in Vision-Language Models through Barrier-Regulated Adaptive Closed-form Steering

Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy

Focus Matters: Phase-Aware Suppression for Hallucination in Vision-Language Models