From Token Correction to Claim Admission: Training-Free Hallucination Mitigation for Large Vision-Language Models

Hyeonggeun Han; Taehyun Cho; Sangwoo Hong

cs.CVOct 13, 2026

From Token Correction to Claim Admission: Training-Free Hallucination Mitigation for Large Vision-Language Models

Authors: Hyeonggeun Han, Taehyun Cho, Sangwoo Hong

Organizations: Independent Researcher · Vector Institute · Fields Institute · Seoul National University · Konkuk University

Abstract

Large vision-language models often exhibit object hallucination, generating claims that are not supported by the input image. To mitigate this issue, we propose Evidence-Guided Visual Claim Admission (EGVCA), a training-free framework that admits a semantic claim only when it receives sufficient image-dependent support. EGVCA measures this support by comparing the claim likelihood under the observed image and a visual-null input. For captioning, it controls semantic expansion without altering token generation. For binary visual question answering, it combines atomic visual support with the image-induced change in the model's native Yes--No margin. Experiments on CHAIR, POPE, and MME across three LVLM backbones show that EGVCA substantially reduces object hallucination while preserving caption quality. It achieves the highest POPE F1 in eight of nine backbone--sampling settings and improves the native MME total score on all three backbones. These results demonstrate the effectiveness of claim-level admission as a simple alternative to token-level distribution correction.

Figures & tables

Figure 1: Comparison of greedy decoding, VCD, and EGVCA. Unlike VCD, which modifies next-token predictions, EGVCA uses original-versus-visual-null evidence to reject unsupported claims without altering candidate generation.

	LLaVA-1.5		MiniGPT-4		mPLUG-Owl2
Method	$\mathrm{CHAIR}_{S}\downarrow$	$\mathrm{CHAIR}_{I}\downarrow$	$\mathrm{CHAIR}_{S}\downarrow$	$\mathrm{CHAIR}_{I}\downarrow$	$\mathrm{CHAIR}_{S}\downarrow$	$\mathrm{CHAIR}_{I}\downarrow$
Greedy	$21.93\pm 1.79$	$7.07\pm 0.59$	$20.33\pm 0.52$	$7.26\pm 0.39$	$21.47\pm 1.00$	$7.12\pm 0.39$
Beam	$18.73\pm 2.49$	$6.23\pm 0.37$	$20.00\pm 1.88$	$7.20\pm 0.48$	$19.27\pm 1.59$	$6.61\pm 0.39$
VCD	$23.87\pm 0.25$	$7.91\pm 0.27$	$23.73\pm 0.90$	$9.02\pm 0.07$	$25.40\pm 2.32$	$8.77\pm 0.73$
OPERA	$18.00\pm 1.02$	$6.02\pm 0.19$	$17.80\pm 1.57$	$7.08\pm 0.35$	$17.73\pm 0.74$	$6.21\pm 0.16$
DoLa	$17.27\pm 1.51$	$8.72\pm 2.08$	$15.07\pm 1.89$	$7.08\pm 0.56$	$18.87\pm 1.23$	$6.44\pm 0.23$

Table 1: CHAIR-500 comparison across three LVLM backbones.

	LLaVA-1.5						MiniGPT-4						mPLUG-Owl2
	Random		Popular		Adversarial		Random		Popular		Adversarial		Random		Popular		Adversarial
Method	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1
Native	87.20	85.69	85.83	84.40	83.47	82.25	74.83	72.56	71.40	69.94	69.50	68.70	88.67	88.59	84.77	85.24	80.90	82.12
VCD	84.23	82.50	82.37	80.62	79.33	77.97	58.70	58.74	57.00	57.26	57.60	58.94	83.97	83.39	81.17	81.30	79.00	79.48
OPERA	86.23	84.42	85.17	83.41	82.87	81.27	74.50	68.71	72.03	66.69	69.67	65.00	86.83	85.80	84.40	83.60	80.80	80.51
DoLa	88.43	87.44	86.37	85.52	83.57	83.03	-	-	-	-	-	-	86.67	85.55	84.47	83.56	81.07	80.61

Table 2: POPE results under Random, Popular, and Adversarial sampling across three LVLM backbones. 2 2 2 DoLa and HALC yielded unstable binary outputs with MiniGPT-4 on both POPE and MME under the matched protocol, indicating sensitivity to the backbone–decoding interaction.

Figure 2: General capability profiles across three LVLM backbones. BLEU-4, METEOR, ROUGE-L, and SPICE are evaluated on COCO captions, whereas MME-P and MME-C are evaluated on the separate MME benchmark. Each metric is normalized by the best matched result for the corresponding backbone and metric. Greedy and Beam are omitted for clarity.

Setting	$\mathrm{CHAIR}_{S}\downarrow$	$\mathrm{CHAIR}_{I}\downarrow$	SPICE $\uparrow$
Greedy	$21.93{\pm}1.79$	$7.07{\pm}0.59$	$21.44{\pm}0.30$
$\tau=0$	$20.47{\pm}1.48$	$6.96{\pm}0.56$	$21.67{\pm}0.25$
$\tau=0.125$	$14.73{\pm}1.47$	$6.47{\pm}0.57$	$\mathbf{22.23{\pm}0.14}$
$\tau=0.25$ (Ours)	$10.87{\pm}1.43$	$5.77{\pm}0.61$	$22.19{\pm}0.24$
$\tau=0.5$	$7.47{\pm}1.37$	$4.56{\pm}0.80$	$21.73{\pm}0.25$
$\tau=1.0$	$\mathbf{6.47{\pm}1.11}$	$\mathbf{4.24{\pm}0.61}$	$21.27{\pm}0.25$

Table 3: Sensitivity of caption admission to

\tau

on LLaVA-1.5. Results are the mean and standard deviation over three CHAIR-500 runs.

Variant	POPE F1 $\uparrow$	MME-P $\uparrow$	MME-C $\uparrow$	MME Total $\uparrow$
Native LVLM	84.11	1360.14	321.79	1681.92
w/o $M(q)$	81.93	1371.71	361.79	1733.50
w/o $V(q)$	85.33	1360.32	321.79	1682.11
w/o $m_{I}(q)$	84.95	498.33	204.29	702.62
Full EGVCA	86.48	1373.34	366.79	1740.13

Table 4: Binary component ablation on full POPE and MME using LLaVA-1.5.

Method	Tokens/sec. $\uparrow$	Peak VRAM (GB) $\downarrow$
Greedy	66.61	15.27
Beam	42.78	22.35
VCD	33.56	16.37
OPERA	5.43	25.45
DoLa	19.69	26.31
DeCo	59.10	15.75

Table 5: Inference efficiency on LLaVA-1.5 over CHAIR images.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Object hallucination across claim positions in LLaVA-1.5 Greedy captions. Error bars indicate 95% confidence intervals obtained using image-grouped bootstrap resampling. Positions four and later are pooled.

Null reference	$\mathrm{CHAIR}_{S}\downarrow$	$\mathrm{CHAIR}_{I}\downarrow$	SPICE $\uparrow$
Zero tensor (EGVCA)	$\underline{10.87\pm 1.43}$	$\underline{5.77\pm 0.61}$	$\bm{22.19\pm 0.24}$
Normalized black	$\bm{10.53\pm 1.20}$	$\bm{5.59\pm 0.45}$	$\underline{22.15\pm 0.24}$
VCD diffusion noise	$11.13\pm 2.07$	$5.98\pm 0.87$	$22.04\pm 0.24$
Shuffled image	$13.00\pm 0.85$	$6.63\pm 0.38$	$22.11\pm 0.27$

Appendix

Table 6: Visual-null ablation on LLaVA-1.5 CHAIR-500. Results are mean

\pm

standard deviation over three seeds.

$\tau\backslash\lambda$	1.0	2.0	4.0
0.5	86.20	86.42	85.22
1.0	86.14	86.48	86.18
1.5	85.89	85.65	86.07

Appendix

Table 7: Parameter sensitivity on POPE using LLaVA-1.5. Rows vary

\tau

and columns vary

\lambda

. Values are Avg. F1.

$\tau\backslash\lambda$	0.010	0.025	0.050
$-2.0$	1685.71	1737.87	1706.80
$-1.5$	1684.83	1740.13	1724.06
$-1.0$	1678.08	1728.12	1737.34

Appendix

Table 8: Parameter sensitivity on MME using LLaVA-1.5. Rows vary

\tau

and columns vary

\lambda

. Values are MME Total.

Statistic	Value
Number of captions	1,500
Candidate claims per caption	3.326
Admitted claims per caption	1.462
Captions stopped at a rejected claim	92.27%
Mean first-rejection position	2.320
Evidence of admitted post-first claims	$0.484\pm 0.235$

Appendix

Table 9: Claim-admission statistics on LLaVA-1.5 CHAIR-500, pooled over three seeds.

Method	Sec./image $\downarrow$	Tokens/sec. $\uparrow$	Output tokens	Peak VRAM (GB) $\downarrow$	Relative overhead
Greedy	0.953	66.61	63.48	15.27	1.00 $\times$
Beam	1.493	42.78	63.88	22.35	1.57 $\times$
VCD	1.905	33.56	63.91	16.37	2.00 $\times$
OPERA	11.760	5.43	63.84	25.45	12.34 $\times$
DoLa	3.244	19.69	63.85	26.31	3.40 $\times$
DeCo	1.080	59.10	63.85	15.75	1.13 $\times$

Appendix

Table 10: Inference efficiency on LLaVA-1.5 over 200 CHAIR images after three warm-up images. Each method is measured in one synchronized pass.

Component	Sec./image $\downarrow$
Candidate generation	0.953060
Claim segmentation	0.000018
Original/null claim scoring	0.470771
Admission decision	0.000006
End-to-end	1.423854

Appendix

Table 11: End-to-end latency breakdown of EGVCA on LLaVA-1.5.

	LLaVA-1.5			MiniGPT-4			mPLUG-Owl2
Method	Perception	Cognition	Total	Perception	Cognition	Total	Perception	Cognition	Total
Native	1360.14	321.79	1681.92	693.94	196.79	890.73	1398.51	329.64	1728.15
VCD	1381.62	350.71	1732.33	599.77	170.71	770.48	1310.22	331.43	1641.65
OPERA	1456.40	299.29	1755.68	771.00	182.86	953.86	1344.54	375.00	1719.54
DoLa	1434.32	300.71	1735.04	-	-	-	1354.54	382.14	1736.68
DeCo	1453.74	322.50	1776.24	501.13	198.57	699.70	1343.98	325.71	1669.69

Appendix

Table 12: General visual understanding on the full MME benchmark. Perception and cognition scores are reported alongside their total.

	LLaVA-1.5				MiniGPT-4				mPLUG-Owl2
Method	B-4 $\uparrow$	MET. $\uparrow$	R-L $\uparrow$	SPICE $\uparrow$	B-4 $\uparrow$	MET. $\uparrow$	R-L $\uparrow$	SPICE $\uparrow$	B-4 $\uparrow$	MET. $\uparrow$	R-L $\uparrow$	SPICE $\uparrow$
Greedy	7.89	23.01	27.01	21.44	7.94	22.52	26.37	19.07	8.05	23.26	27.25	21.72
Beam	8.04	23.11	27.55	21.26	8.30	23.09	26.89	20.02	8.48	23.82	28.00	21.90
VCD	6.36	21.92	25.88	19.53	7.17	22.02	26.03	18.61	6.65	22.31	26.37	20.21
OPERA	8.04	23.19	27.63	21.35	9.14	23.54	29.40	19.74	8.09	23.17	27.59	21.06
DoLa	4.34	18.35	21.29	16.48	5.08	17.93	21.62	14.99	8.51	23.82	28.03	21.94

Appendix

Table 13: Reference-based caption quality across three LVLM backbones. Results are averaged over three seeds.

Explore similar work

SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineering
Jul 5, 2026Kai Tang, Jinhao You, Bohua Zhang +6Large Vision-Language Models Object Hallucination in VLMs
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Nanyang Technological University · University of Pennsylvania +4
Test-Time Hallucination Control in Large Vision-Language Models
Aug 11, 2026Mehran Tamjidi, Hamidreza Dastmalchi, Ali Cheraghian +3Test-Time Adaptation of VLMs Large Vision-Language Models
University of Technology Sydney · York University, Canada · Australian National University +2
Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy
May 20, 2026Yutong Xie, Zhenglin Hua, Ran Wang +3Vision-Language Models Hallucination in Language Models
School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Artificial Intelligence, Shenzhen University, Shenzhen, China · National Engineering Laboratory for Big Data Systems Computing Technology, Shenzhen University, Shenzhen, China +3

Jul 5, 2026J/K move · Enter open · S save

SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineering

Kai Tang, Jinhao You, Bohua Zhang, Yichen Guo+5

State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Nanyang Technological University · University of Pennsylvania+4

Large Vision-Language Models Object Hallucination in VLMs VLM Hallucination Mitigation

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) have achieved remarkable progress in visual understanding tasks such as image captioning and visual question answering. However, they remain susceptible to hallucinations, generating content that is inconsistent with the actual visual input. Existing methods primarily intervene at the decoding stage, while overlooking a critical source of hallucinations: irrelevant or noisy visual tokens that mislead the decoding process. To address this issue, we propose SeeMe, a training-free framework that introduces the concept of feature engineering from traditional machine learning into LVLMs. SeeMe restructures visual tokens through a three-stage token engineering process to suppress hallucination sources while preserving informative visual evidence. Experiments on MME, POPE, and AMBER benchmarks across four LVLMs demonstrate that SeeMe consistently reduces hallucinations and improves output consistency, providing a novel perspective for mitigating hallucinations in LVLMs.

Abstract

Explore similar work

SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineering

Test-Time Hallucination Control in Large Vision-Language Models

Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy