PatchGate: Narrowing the Verbalization Gap with Intrinsic Object Inventories in Frozen Vision-Language Models

Jihyung Ko; Eunji Jung; Hyeongsub Kim; Ziseok Lee; Jae Won Cho; Sanghyun Jo; Kyungsu Kim

cs.CVAug 22, 2026

PatchGate: Narrowing the Verbalization Gap with Intrinsic Object Inventories in Frozen Vision-Language Models

Authors: Jihyung Ko, Eunji Jung, Hyeongsub Kim, Ziseok Lee, Jae Won Cho, Sanghyun Jo, Kyungsu Kim

Organizations: Seoul National University, Korea · LG CNS, Korea · Konkuk University, Korea · OGQ, Korea

Abstract

Reliable image captioning in Vision-Language Models (VLMs) requires captions to be both precise and complete, avoiding unsupported object mentions while covering visible objects. Existing training-free methods primarily address the former requirement, suppressing unsupported object words by intervening on model-predicted mentions during generation. Because they operate only on objects the model is already likely to mention, visible objects omitted from the output remain difficult to recover. We propose PatchGate, a training-free framework that extracts prompt-free object evidence intrinsic to a frozen VLM before generation and uses it to narrow the gap between an intrinsic object set and final object mentions. In the first stage, Visual Evidence eXtraction (VEX) reads patch-level lexical evidence from the latter half of LM decoder layers and constructs an image-conditioned object set without any task prompt. In the second stage, Visual-Evidence Inclusion-Exclusion Decoding (VIED) uses this object evidence to calibrate decoding logits, promoting evidence-supported but under-verbalized objects and suppressing weakly supported but over-verbalized objects. On AMBER, PatchGate improves both sides of object-level reliability, increasing visible-object coverage from 49.4 to 56.0 (+13.4%) and reducing object hallucination by lowering CHAIR from 7.5 to 6.6 (-12.0%), without external detectors or fine-tuning and with one extra forward pass.

Figures & tables

Figure 1: Two complementary captioning failures and PatchGate’s simultaneous mitigation. Reliable captions should be precise and complete, yet generated captions can mention unsupported objects (red: people , truck ) while omitting visible objects (blue: sky , flag ). PatchGate uses prompt-free internal object evidence to calibrate decoding, suppressing unsupported mentions and recovering omitted visible objects (yellow: sky , flag ).

Properties	[CVPR’24] OPERA	[CVPR’24] VCD	[ICLR’25] ProjectAway	[ICML’25] MARINE	[ICLR’26] SHIELD	[CVPR’26] PND	[ICML’26] ILVAD	PatchGate (Ours)
(a) Reduces object hallucinations for precision	✓	✓	✓	✓	✓	✓	✓	✓
(b) Reduces object omissions for completeness	✗	✗	✗	✗	✗	✓	✗	✓
(c) Reads internal patch-level object evidence	✗	✗	✓	✗	✗	✗	✗	✓
(d) Builds a prompt-free object set before generation	✗	✗	✗	✓	✗	✗	✗	✓
(e) Corrects visual evidence-mention mismatch	✗	✗	✗	✗	✗	✗	✗	✓
(f) Requires no external visual module ( e.g. , tagger)	✓	✓	✓	✗	✓	✗	✓	✓

Table 1: Conceptual comparison of PatchGate (Ours) with representative training-free hallucination mitigation approaches: OPERA [ 8 ] , VCD [ 17 ] , ProjectAway [ 12 ] , MARINE [ 36 ] , SHIELD [ 10 ] , PND [ 13 ] and ILVAD [ 30 ] .

Figure 2: Overview of PatchGate. (a) VEX reads patch-level lexical evidence from the later LM decoder layers, scores each object candidate by its strongest normalized patch evidence, and selects a prompt-free object set $\mathcal{T}(I)$ with an adaptive largest-gap cutoff. (b) VIED calibrates decoding logits with $\mathcal{T}(I)$ : inclusion promotes evidence-supported but under-verbalized objects, while exclusion suppresses over-committed object words with weak visual evidence.

Method	Generative Task (Captioning)				Discriminative Task (QA)
Method	Cover $\uparrow$	CHAIR $\downarrow$	Hal $\downarrow$	Cog $\downarrow$	Acc. $\uparrow$	P. $\uparrow$	R. $\uparrow$	F1 $\uparrow$
[CVPR’24] LLaVA-v1.5-7B	49.4	7.5	31.4	3.6	72.0	92.5	62.9	74.9
+ [CVPR’24] OPERA	48.8 (-0.6)	6.6 (-0.9)	28.3	2.9	76.4	92.7 (+0.2)	69.9 (+7.0)	79.7
+ [CVPR’24] VCD	51.7 (+2.3)	8.9 (+1.4)	39.7	4.4	71.1	91.4 (-1.1)	62.2 (-0.7)	74.0
+ [CVPR’25] Devils-in-Mid.	50.3 (+0.9)	3.3 (-4.2)	20.0	1.2	50.5	94.5 (+2.0)	26.8 (-36.1)	41.8
+ [ICML’25] MARINE	49.9 (+0.5)	6.0 (-1.5)	27.3	2.9	72.4	94.8 (+2.3)	61.7 (-1.2)	74.7

Table 2: Performance comparison on AMBER generative and discriminative tasks [ 28 ] . For Cover, CHAIR, Precision (P.), and Recall (R.), colored values show absolute changes from the LLaVA-v1.5-7B baseline [ 20 ] (green: improvement, red: degradation). The best results are in bold .

Figure 3: Qualitative results on AMBER [ 28 ] and POPE [ 18 ] . PatchGate recovers omitted visible objects and enables correct predictions (green), while suppressing unsupported object mentions and incorrect answers (red).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A: End-to-end PatchGate inference for open-ended captioning. VEX (Sec. 3.1 ) selects the largest-gap cutoff at $\theta_{\mathrm{TTD}}=0.57$ , producing an internal object inventory that retains supported objects ( bed , lamp , chair , and table ) while excluding unsupported objects ( cup and couch ). During VIED (Sec. 3.2 ), EDE reduces the cup logit from $6.8$ to $4.3$ , whereas ESI increases the omitted lamp logit from $16.1$ to $17.2$ , causing it to be generated.

Figure B: End-to-end PatchGate inference for binary QA. PatchGate constructs the object inventory $\mathcal{T}(I)$ , in which ship is absent due to its negligible visual support ( $\mathrm{PCS}\approx 0.00$ ). While the unconditioned LLaVA-v1.5-7B logits yield a hallucinated “Yes” answer, inventory-conditioned guidance raises the “No” logit above “Yes”, yielding the correct answer “No”.

Figure C: Limitations of Hal and Cog. Although the sun is visibly present, AMBER annotates it as absent and includes it in the image-specific hallucinatory target set. PatchGate correctly recovers sun , but this mention is penalized by both Hal and Cog.

Method	Generative Task				Discriminative Task
Method	Cover $\uparrow$	CHAIR $\downarrow$	Hal $\downarrow$	Cog $\downarrow$	Acc. $\uparrow$	P. $\uparrow$	R. $\uparrow$	F1 $\uparrow$
[CVPR’24] LLaVA-v1.5-7B
[CVPR’24] LLaVA-v1.5-7B	49.4	7.5	31.4	3.6	72.0	92.5	62.9	74.9
+ [CVPR’24] OPERA	48.8 (-0.6)	6.6 (-0.9)	28.3	2.9	76.4	92.7 (+0.2)	69.9 (+7.0)	79.7
+ [CVPR’24] VCD	51.7 (+2.3)	8.9 (+1.4)	39.7	4.4	71.1	91.4 (-1.1)	62.2 (-0.7)	74.0
+ [CVPR’25] Devils-in-Mid.	50.3 (+0.9)	3.3 (-4.2)	20.0	1.2	50.5	94.5 (+2.0)	26.8 (-36.1)	41.8

Appendix

Table C: Full AMBER results [ 28 ] across different VLM backbones. We include all training-free baselines evaluated on each backbone. For Cover, CHAIR, P., and R., colored values show absolute changes from the corresponding backbone baseline (green: improvement, red: degradation). The best results within each backbone are in bold .

Method	Random				Popular				Adversarial
Method	Acc. $\uparrow$	P. $\uparrow$	R. $\uparrow$	F1 $\uparrow$	Acc. $\uparrow$	P. $\uparrow$	R. $\uparrow$	F1 $\uparrow$	Acc. $\uparrow$	P. $\uparrow$	R. $\uparrow$	F1 $\uparrow$
[CVPR’24] LLaVA-v1.5-7B	83.8	92.4	73.7	82.0	82.6	89.7	73.7	80.9	79.7	83.8	73.7	78.4
+ [CVPR’24] OPERA	89.2	93.4	85.2	89.1	86.8	88.0	85.2	86.6	81.2	78.8	85.2	81.9
+ [CVPR’24] VCD	85.4	92.8	76.8	84.0	83.3	88.3	76.8	82.2	80.4	82.8	76.8	79.7
+ [CVPR’25] Devils-in-Mid.	56.0	54.6	70.3	61.5	56.2	54.8	70.9	61.8	54.5	53.5	69.1	60.3
+ [ICML’25] MARINE	86.5	91.0	79.0	84.6	84.5	89.5	78.0	83.4	82.2	87.7	77.0	82.0

Appendix

Table D: Performance comparison on the POPE benchmark [ 18 ] . We report results across the random, popular, and adversarial splits. The best results are in bold , and the second-best results are underlined .

Figure D: Additional qualitative results on AMBER generative task. PatchGate suppresses unsupported object mentions (red) while recovering omitted visible objects(green), generating precise and complete captions.

Figure E: Additional qualitative results on AMBER and POPE QA. Correct and incorrect answers are shown in green and red, respectively. PatchGate corrects both hallucinated “Yes” answers and omitted “No” answers across diverse question types.

Figure F: Evidence aggregation and adaptive object selection. Mean aggregation retains only a few high-confidence objects, whereas max-softmax aggregation over the selected late-layer range more clearly separates supported objects from the low-evidence tail.

Method	Tag-level				Caption-level				Computational Cost
Method	P. $\uparrow$	R. $\uparrow$	F1 $\uparrow$	$\mathbf{\|T\|}$	Cover $\uparrow$	CHAIR $\downarrow$	Hal $\downarrow$	Cog $\downarrow$	Param.	Peak VRAM	Time
LLaVA-v1.5-7B	–	–	–	–	49.4	7.5	31.4	3.6	7.0 B	14.6 GB	2.92 s
+ GroundingDINO	91.1	50.4	64.9	5.2	51.3	7.3	33.3	3.6	7.2 B	16.2 GB	3.26 s
+ RAM++ (default)	97.0	59.6	71.1	11.6	50.2	7.8	36.8	4.2	7.3 B	16.9 GB	3.17 s
+ RAM++ (top- $10$ )	95.8	53.0	68.3	10.0	50.1	8.2	37.2	4.3	7.3 B	16.9 GB	3.17 s
+ VEX (Ours; Sec. 3.1 )	84.8	49.2	62.3	12.9	56.0	6.6	34.3	3.0	7.0 B	14.9 GB	3.30 s

Appendix

Table K: Comparison of internal (VEX; Sec. 3.1 ) and external object inventories.

Explore similar work

Hallucinations Leave a Grounding Signature:Verifier-Guided Decoding for Selective Object Correction
Jul 30, 2026Lei Yang, Xinze Liu, Dayan Wu +7Hallucination Detection in VLMs LLM Hallucination Mitigation
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · Department of Applied Mathematics and Statistics, Johns Hopkins University, Baltimore, MD, USA
Query-Dependent Use of Generated Descriptions for Reliable Visual Question Answering
May 3, 2026Zeshang Li, Shuoyang ZhangVisual Question Answering Hallucination in Language Models
University of International Relations
From Token Correction to Claim Admission: Training-Free Hallucination Mitigation for Large Vision-Language Models
Oct 13, 2026Hyeonggeun Han, Taehyun Cho, Sangwoo HongVLM Hallucination Mitigation Object Hallucination in VLMs
Independent Researcher · Vector Institute · Fields Institute +2

Jul 30, 2026J/K move · Enter open · S save

Hallucinations Leave a Grounding Signature:Verifier-Guided Decoding for Selective Object Correction

Lei Yang, Xinze Liu, Dayan Wu, Ding Wang+6

Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · Department of Applied Mathematics and Statistics, Johns Hopkins University, Baltimore, MD, USA

Hallucination Detection in VLMs LLM Hallucination Mitigation Object Hallucination in VLMs Vision-Language Grounding VLM Hallucination Mitigation

Open paper AskPDF arXiv

Large vision-language models (LVLMs) often hallucinate objects that are absent from an image. Despite recent progress, existing mitigation methods still lack reliable object-level grounding diagnostics and therefore tend to apply coarse-grained interventions, which can impair visual understanding, shorten responses, and reduce coverage of genuinely grounded objects. The key challenge is thus to detect, during generation, whether each emerging object mention is supported by reliable visual evidence, so that hallucination can be mitigated selectively. Yet output confidence reflects next-token plausibility rather than visual support, allowing language priors to make absent objects appear certain. We show that the missing diagnostic evidence is encoded in an Intrinsic Grounding Signature (IGS), a distributed signed attention pattern that remains informative for such confident hallucinations. Based on IGS, we propose Verifier-Guided Decoding (VGD), a decoding framework in which a lightweight verifier examines each emerging object mention, rolls back the KV cache when the mention is identified as high risk, suppresses the object and its synonyms, and regenerates the affected continuation. Because VGD intervenes only on object mentions identified as high risk, it reduces object hallucination while preserving the model's original visual understanding and grounded object coverage. Experiments on CHAIR and AMBER-G show that VGD achieves state-of-the-art object hallucination reduction: at @rec90, it cuts AMBER-G CHAIR by 43.6% while retaining 99.6% of grounded-object coverage, and reduces CHAIR-MSCOCO CHAIR

_i

/CHAIR

_s

by 37.0%/30.4% without shortening captions.

Abstract

Explore similar work

Hallucinations Leave a Grounding Signature:Verifier-Guided Decoding for Selective Object Correction

Query-Dependent Use of Generated Descriptions for Reliable Visual Question Answering

From Token Correction to Claim Admission: Training-Free Hallucination Mitigation for Large Vision-Language Models