From Token Correction to Claim Admission: Training-Free Hallucination Mitigation for Large Vision-Language Models

Hyeonggeun Han; Taehyun Cho; Sangwoo Hong

cs.CVOct 13, 2026

From Token Correction to Claim Admission: Training-Free Hallucination Mitigation for Large Vision-Language Models

Authors: Hyeonggeun Han, Taehyun Cho, Sangwoo Hong

Organizations: Independent Researcher · Vector Institute · Fields Institute · Seoul National University · Konkuk University

Abstract

Large vision-language models often exhibit object hallucination, generating claims that are not supported by the input image. To mitigate this issue, we propose Evidence-Guided Visual Claim Admission (EGVCA), a training-free framework that admits a semantic claim only when it receives sufficient image-dependent support. EGVCA measures this support by comparing the claim likelihood under the observed image and a visual-null input. For captioning, it controls semantic expansion without altering token generation. For binary visual question answering, it combines atomic visual support with the image-induced change in the model's native Yes--No margin. Experiments on CHAIR, POPE, and MME across three LVLM backbones show that EGVCA substantially reduces object hallucination while preserving caption quality. It achieves the highest POPE F1 in eight of nine backbone--sampling settings and improves the native MME total score on all three backbones. These results demonstrate the effectiveness of claim-level admission as a simple alternative to token-level distribution correction.

Figures & tables

Figure 1: Comparison of greedy decoding, VCD, and EGVCA. Unlike VCD, which modifies next-token predictions, EGVCA uses original-versus-visual-null evidence to reject unsupported claims without altering candidate generation.

	LLaVA-1.5		MiniGPT-4		mPLUG-Owl2
Method	$\mathrm{CHAIR}_{S}\downarrow$	$\mathrm{CHAIR}_{I}\downarrow$	$\mathrm{CHAIR}_{S}\downarrow$	$\mathrm{CHAIR}_{I}\downarrow$	$\mathrm{CHAIR}_{S}\downarrow$	$\mathrm{CHAIR}_{I}\downarrow$
Greedy	$21.93\pm 1.79$	$7.07\pm 0.59$	$20.33\pm 0.52$	$7.26\pm 0.39$	$21.47\pm 1.00$	$7.12\pm 0.39$
Beam	$18.73\pm 2.49$	$6.23\pm 0.37$	$20.00\pm 1.88$	$7.20\pm 0.48$	$19.27\pm 1.59$	$6.61\pm 0.39$
VCD	$23.87\pm 0.25$	$7.91\pm 0.27$	$23.73\pm 0.90$	$9.02\pm 0.07$	$25.40\pm 2.32$	$8.77\pm 0.73$
OPERA	$18.00\pm 1.02$	$6.02\pm 0.19$	$17.80\pm 1.57$	$7.08\pm 0.35$	$17.73\pm 0.74$	$6.21\pm 0.16$
DoLa	$17.27\pm 1.51$	$8.72\pm 2.08$	$15.07\pm 1.89$	$7.08\pm 0.56$	$18.87\pm 1.23$	$6.44\pm 0.23$

Table 1: CHAIR-500 comparison across three LVLM backbones.

	LLaVA-1.5						MiniGPT-4						mPLUG-Owl2
	Random		Popular		Adversarial		Random		Popular		Adversarial		Random		Popular		Adversarial
Method	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1
Native	87.20	85.69	85.83	84.40	83.47	82.25	74.83	72.56	71.40	69.94	69.50	68.70	88.67	88.59	84.77	85.24	80.90	82.12
VCD	84.23	82.50	82.37	80.62	79.33	77.97	58.70	58.74	57.00	57.26	57.60	58.94	83.97	83.39	81.17	81.30	79.00	79.48
OPERA	86.23	84.42	85.17	83.41	82.87	81.27	74.50	68.71	72.03	66.69	69.67	65.00	86.83	85.80	84.40	83.60	80.80	80.51
DoLa	88.43	87.44	86.37	85.52	83.57	83.03	-	-	-	-	-	-	86.67	85.55	84.47	83.56	81.07	80.61

Table 2: POPE results under Random, Popular, and Adversarial sampling across three LVLM backbones. 2 2 2 DoLa and HALC yielded unstable binary outputs with MiniGPT-4 on both POPE and MME under the matched protocol, indicating sensitivity to the backbone–decoding interaction.

Figure 2: General capability profiles across three LVLM backbones. BLEU-4, METEOR, ROUGE-L, and SPICE are evaluated on COCO captions, whereas MME-P and MME-C are evaluated on the separate MME benchmark. Each metric is normalized by the best matched result for the corresponding backbone and metric. Greedy and Beam are omitted for clarity.

Setting	$\mathrm{CHAIR}_{S}\downarrow$	$\mathrm{CHAIR}_{I}\downarrow$	SPICE $\uparrow$
Greedy	$21.93{\pm}1.79$	$7.07{\pm}0.59$	$21.44{\pm}0.30$
$\tau=0$	$20.47{\pm}1.48$	$6.96{\pm}0.56$	$21.67{\pm}0.25$
$\tau=0.125$	$14.73{\pm}1.47$	$6.47{\pm}0.57$	$\mathbf{22.23{\pm}0.14}$
$\tau=0.25$ (Ours)	$10.87{\pm}1.43$	$5.77{\pm}0.61$	$22.19{\pm}0.24$
$\tau=0.5$	$7.47{\pm}1.37$	$4.56{\pm}0.80$	$21.73{\pm}0.25$
$\tau=1.0$	$\mathbf{6.47{\pm}1.11}$	$\mathbf{4.24{\pm}0.61}$	$21.27{\pm}0.25$

Table 3: Sensitivity of caption admission to

\tau

on LLaVA-1.5. Results are the mean and standard deviation over three CHAIR-500 runs.

Variant	POPE F1 $\uparrow$	MME-P $\uparrow$	MME-C $\uparrow$	MME Total $\uparrow$
Native LVLM	84.11	1360.14	321.79	1681.92
w/o $M(q)$	81.93	1371.71	361.79	1733.50
w/o $V(q)$	85.33	1360.32	321.79	1682.11
w/o $m_{I}(q)$	84.95	498.33	204.29	702.62
Full EGVCA	86.48	1373.34	366.79	1740.13

Table 4: Binary component ablation on full POPE and MME using LLaVA-1.5.

Method	Tokens/sec. $\uparrow$	Peak VRAM (GB) $\downarrow$
Greedy	66.61	15.27
Beam	42.78	22.35
VCD	33.56	16.37
OPERA	5.43	25.45
DoLa	19.69	26.31
DeCo	59.10	15.75

Table 5: Inference efficiency on LLaVA-1.5 over CHAIR images.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Object hallucination across claim positions in LLaVA-1.5 Greedy captions. Error bars indicate 95% confidence intervals obtained using image-grouped bootstrap resampling. Positions four and later are pooled.

Null reference	$\mathrm{CHAIR}_{S}\downarrow$	$\mathrm{CHAIR}_{I}\downarrow$	SPICE $\uparrow$
Zero tensor (EGVCA)	$\underline{10.87\pm 1.43}$	$\underline{5.77\pm 0.61}$	$\bm{22.19\pm 0.24}$
Normalized black	$\bm{10.53\pm 1.20}$	$\bm{5.59\pm 0.45}$	$\underline{22.15\pm 0.24}$
VCD diffusion noise	$11.13\pm 2.07$	$5.98\pm 0.87$	$22.04\pm 0.24$
Shuffled image	$13.00\pm 0.85$	$6.63\pm 0.38$	$22.11\pm 0.27$

Appendix

Table 6: Visual-null ablation on LLaVA-1.5 CHAIR-500. Results are mean

\pm

standard deviation over three seeds.

$\tau\backslash\lambda$	1.0	2.0	4.0
0.5	86.20	86.42	85.22
1.0	86.14	86.48	86.18
1.5	85.89	85.65	86.07

Appendix

Table 7: Parameter sensitivity on POPE using LLaVA-1.5. Rows vary

\tau

and columns vary

\lambda

. Values are Avg. F1.

$\tau\backslash\lambda$	0.010	0.025	0.050
$-2.0$	1685.71	1737.87	1706.80
$-1.5$	1684.83	1740.13	1724.06
$-1.0$	1678.08	1728.12	1737.34

Appendix

Table 8: Parameter sensitivity on MME using LLaVA-1.5. Rows vary

\tau

and columns vary

\lambda

. Values are MME Total.

Statistic	Value
Number of captions	1,500
Candidate claims per caption	3.326
Admitted claims per caption	1.462
Captions stopped at a rejected claim	92.27%
Mean first-rejection position	2.320
Evidence of admitted post-first claims	$0.484\pm 0.235$

Appendix

Table 9: Claim-admission statistics on LLaVA-1.5 CHAIR-500, pooled over three seeds.

Method	Sec./image $\downarrow$	Tokens/sec. $\uparrow$	Output tokens	Peak VRAM (GB) $\downarrow$	Relative overhead
Greedy	0.953	66.61	63.48	15.27	1.00 $\times$
Beam	1.493	42.78	63.88	22.35	1.57 $\times$
VCD	1.905	33.56	63.91	16.37	2.00 $\times$
OPERA	11.760	5.43	63.84	25.45	12.34 $\times$
DoLa	3.244	19.69	63.85	26.31	3.40 $\times$
DeCo	1.080	59.10	63.85	15.75	1.13 $\times$

Appendix

Table 10: Inference efficiency on LLaVA-1.5 over 200 CHAIR images after three warm-up images. Each method is measured in one synchronized pass.

Component	Sec./image $\downarrow$
Candidate generation	0.953060
Claim segmentation	0.000018
Original/null claim scoring	0.470771
Admission decision	0.000006
End-to-end	1.423854

Appendix

Table 11: End-to-end latency breakdown of EGVCA on LLaVA-1.5.

	LLaVA-1.5			MiniGPT-4			mPLUG-Owl2
Method	Perception	Cognition	Total	Perception	Cognition	Total	Perception	Cognition	Total
Native	1360.14	321.79	1681.92	693.94	196.79	890.73	1398.51	329.64	1728.15
VCD	1381.62	350.71	1732.33	599.77	170.71	770.48	1310.22	331.43	1641.65
OPERA	1456.40	299.29	1755.68	771.00	182.86	953.86	1344.54	375.00	1719.54
DoLa	1434.32	300.71	1735.04	-	-	-	1354.54	382.14	1736.68
DeCo	1453.74	322.50	1776.24	501.13	198.57	699.70	1343.98	325.71	1669.69

Appendix

Table 12: General visual understanding on the full MME benchmark. Perception and cognition scores are reported alongside their total.

	LLaVA-1.5				MiniGPT-4				mPLUG-Owl2
Method	B-4 $\uparrow$	MET. $\uparrow$	R-L $\uparrow$	SPICE $\uparrow$	B-4 $\uparrow$	MET. $\uparrow$	R-L $\uparrow$	SPICE $\uparrow$	B-4 $\uparrow$	MET. $\uparrow$	R-L $\uparrow$	SPICE $\uparrow$
Greedy	7.89	23.01	27.01	21.44	7.94	22.52	26.37	19.07	8.05	23.26	27.25	21.72
Beam	8.04	23.11	27.55	21.26	8.30	23.09	26.89	20.02	8.48	23.82	28.00	21.90
VCD	6.36	21.92	25.88	19.53	7.17	22.02	26.03	18.61	6.65	22.31	26.37	20.21
OPERA	8.04	23.19	27.63	21.35	9.14	23.54	29.40	19.74	8.09	23.17	27.59	21.06
DoLa	4.34	18.35	21.29	16.48	5.08	17.93	21.62	14.99	8.51	23.82	28.03	21.94

Appendix

Table 13: Reference-based caption quality across three LVLM backbones. Results are averaged over three seeds.

Explore similar work

Jul 5, 2026

SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineering

Large Vision-Language Models (LVLMs) have achieved remarkable progress in visual understanding tasks such as image captioning and visual question answering. However, they remain susceptible to hallucinations, generating content that is inconsistent with the actual visual input. Existing methods primarily intervene at the decoding stage, while overlooking a critical source of hallucinations: irrelevant or noisy visual tokens that mislead the decoding process. To address this issue, we propose SeeMe, a training-free framework that introduces the concept of feature engineering from traditional machine learning into LVLMs. SeeMe restructures visual tokens through a three-stage token engineering process to suppress hallucination sources while preserving informative visual evidence. Experiments on MME, POPE, and AMBER benchmarks across four LVLMs demonstrate that SeeMe consistently reduces hallucinations and improves output consistency, providing a novel perspective for mitigating hallucinations in LVLMs.

Kai Tang, Jinhao You, Bohua Zhang +6

State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Nanyang Technological University · University of Pennsylvania +4

Large Vision-Language Models Object Hallucination in VLMs VLM Hallucination Mitigation

Ask PDF

Aug 11, 2026

Test-Time Hallucination Control in Large Vision-Language Models

Object Hallucination in large vision-language models (LVLMs), where models generate non-factual content about input images, remains a critical barrier to their reliability in real-world applications. Existing mitigation strategies can be categorized into training-based and training-free methods. Training-based methods often achieve strong performance but are costly, requiring extensive computational resources, large-scale data, and time-consuming fine-tuning. Training-free approaches are particularly appealing due to their efficiency. However, existing training-free methods either require multiple decoding rounds, which adds computational overhead, or modify internal states in a model-specific way that risks degrading pretrained knowledge. We propose Test-Time Hallucination Mitigation (TTH) method, a novel training-free method that addresses both limitations. TTH introduces a token-validator module, implemented as a zero-shot Multi-Modal Classifier (MMC), to generate auxiliary logits grounded in the input image. These logits are fused with the original LVLM outputs at the token level for object tokens selected from a candidate pool. An entropy-based weighting scheme is then applied to enable robust and accurate predictions. Extensive experiments across multiple LVLM families and diverse benchmarks demonstrate that TTH consistently improves accuracy and robustness, underscoring its generalizability and practical effectiveness. Code is released at https://github.com/Mehran-TAM/TTH

Mehran Tamjidi, Hamidreza Dastmalchi, Ali Cheraghian +3

University of Technology Sydney · York University, Canada · Australian National University +2

Test-Time Adaptation of VLMs Large Vision-Language Models Object Hallucination in VLMs VLM Hallucination Mitigation

Ask PDF

May 20, 2026

Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy

Large Vision-Language Models (LVLMs) have shown remarkable performance on a wide range of vision-language tasks. Despite this progress, they are still prone to hallucination, generating responses that are inconsistent with visual content. In this work, we find that LVLMs tend to hallucinate when they pay insufficient attention to the correct visual evidence and gradually forget it during the generation process. We empirically find that although LVLMs overall attend insufficiently to visual evidence, they exhibit sensitivity to the correct visual evidence in specific layers, with notable inter-layer discrepancy. Motivated by this observation, we propose a novel hallucination mitigation method that enhances visual evidence based on Inter-Layer Visual Attention Discrepancy (ILVAD). Specifically, we obtain the attention weights from early generated tokens to visual tokens across layers and identify the tokens that are repeatedly activated as visual evidence, forming a saliency map. We then enhance attention to visual evidence during generation through the saliency map to reduce visual forgetting. In addition, we leverage the saliency map to obtain attention scores of generated text to visual evidence, in order to select and emphasize text tokens that are strongly grounded in visual evidence. Our method is training-free and plug-and-play. Multiple benchmark evaluations conducted on five recently released models show that our method can consistently mitigate hallucinations in different LVLMs over various architectures. Code is available at https://github.com/ytx-ML/ILVAD.

Yutong Xie, Zhenglin Hua, Ran Wang +3

School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Artificial Intelligence, Shenzhen University, Shenzhen, China · National Engineering Laboratory for Big Data Systems Computing Technology, Shenzhen University, Shenzhen, China +3

Vision-Language Models Hallucination in Language Models VLM Hallucination Large Vision-Language Models VLM Hallucination Mitigation

Ask PDF