First Logit Boosting: Visual Grounding Method to Mitigate Object Hallucination in Large Vision-Language Models

Jiwoo Ha; Jongwoo Baek; Jinhyun So

cs.CVApr 1, 2026

First Logit Boosting: Visual Grounding Method to Mitigate Object Hallucination in Large Vision-Language Models

Authors: Jiwoo Ha, Jongwoo Baek, Jinhyun So

Organizations: DGIST EECS

Abstract

Recent Large Vision-Language Models (LVLMs) have demonstrated remarkable performance across various multimodal tasks that require understanding both visual and linguistic inputs. However, object hallucination -- the generation of nonexistent objects in answers -- remains a persistent challenge. Although several approaches such as retraining and external grounding methods have been proposed to mitigate this issue, they still suffer from high data costs or structural complexity. Training-free methods such as Contrastive Decoding (CD) are more cost-effective, avoiding additional training or external models, but still suffer from long-term decay, where visual grounding weakens and language priors dominate as the generation progresses. In this paper, we propose First Logit Boosting (FLB), a simple yet effective training-free technique designed to alleviate long-term decay in LVLMs. FLB stores the logit of the first generated token and adds it to subsequent token predictions, effectively mitigating long-term decay of visual information. We observe that FLB (1) sustains the visual information embedded in the first token throughout generation, and (2) suppresses hallucinated words through the stabilizing effect of the ``The'' token. Experimental results show that FLB significantly reduces object hallucination across various tasks, benchmarks, and backbone models. Notably, it causes negligible inference overhead, making it highly applicable to real-time multimodal systems. Code is available at https://github.com/jiwooha20/FLB

Figures & tables

Figure 1: Overview of First Logit Boosting (FLB). FLB stores the logit of the first generated token and reuses it during decoding, which leverages two complementary effects. (1) Direct visual grounding : the first token logit inherently carries stronger visual evidence ( man ) than hallucinated token ( women ), serving as an anchor that preserves visual cues weakened by positional drift. (2) Implicit visual referencing : by boosting the probability of starting a sentence with “The”, FLB increases the likelihood of selecting nouns established before long-term decay occurs, thus maintaining referential coherence and mitigating hallucination.

Figure 2: Comparison of the probability between all ground truth (left) and hallucination (right) words across token steps for each mitigation method. As sentence length increases, hallucinated word logits become more dominant; while VCD, ICD, and M3ID fail to suppress this trend, FLB (ours) effectively mitigates hallucinated predictions.

Figure 3: This figure shows the logits of the ground truth words (left) and hallucination words (middle) for the first token during caption generation for a case image (right). The logit of ground truth words are generally higher than that of hallucination words.

Figure 4: Top 20 tokens by logit value for the first token prediction. The list includes common sentence-starting words such as “The”, “In”, and “A”.

AMBER	LLaVA1.5				InstructBLIP
Method	CHAIR $\downarrow$	Cover $\uparrow$	Hal $\downarrow$	Cog $\downarrow$	CHAIR $\downarrow$	Cover $\uparrow$	Hal $\downarrow$	Cog $\downarrow$
Baseline	$11.5\,(\pm\,0.29)$	$50.1\,(\pm\,0.51)$	$48.9\,(\pm\,0.78)$	$4.6\,(\pm\,0.16)$	$11.6\,(\pm\,0.22)$	$53.4\,(\pm\,0.12)$	$51.7\,(\pm\,1.73)$	$5.3\,(\pm\,0.29)$
VCD	$9.9\,(\pm\,0.12)$	$51.2\,(\pm\,0.24)$	$43.4\,(\pm\,0.37)$	$4.6\,(\pm\,0.08)$	$10.2\,(\pm\,0.05)$	$53.5\,(\pm\,0.33)$	$46.9\,(\pm\,1.09)$	$4.8\,(\pm\,0.21)$
ICD	$9.1\,(\pm\,0.29)$	$51.2\,(\pm\,0.05)$	$40.6\,(\pm\,0.98)$	$4.3\,(\pm\,0.12)$	$12.1\,(\pm\,0.61)$	$52.6\,(\pm\,0.17)$	$51.4\,(\pm\,0.86)$	$5.2\,(\pm\,0.21)$
M3ID	$9.8\,(\pm\,0.22)$	$\textbf{55.6}\,(\pm\,0.25)$	$48.4\,(\pm\,1.11)$	$3.6\,(\pm\,0.09)$	$11.5\,(\pm\,0.61)$	$52.5\,(\pm\,0.17)$	$51.4\,(\pm\,1.28)$	$\textbf{4.6}\,(\pm\,0.24)$
FLB (Ours)	$\textbf{6.1}\,(\pm\,0.37)$	$50.4\,(\pm\,0.22)$	$\textbf{31.6}\,(\pm\,0.99)$	$\textbf{2.7}\,(\pm\,0.24)$	$\textbf{ 9.0}\,(\pm\,0.19)$	$\textbf{53.6}\,(\pm\,0.26)$	$\textbf{43.8}\,(\pm\,1.13)$	$4.7\,(\pm\,0.14)$

Table 1: Performance comparison of LLaVA1.5 and InstructBLIP on AMBER generative tasks. The highest scores are marked in bold .

CHAIR	LLaVA1.5			InstructBLIP
Method	CHAIR s $\downarrow$	CHAIR i $\downarrow$	Recall $\uparrow$	CHAIR s $\downarrow$	CHAIR i $\downarrow$	Recall $\uparrow$
Baseline	$57.5\,(\pm\,2.23)$	$17.3\,(\pm\,0.74)$	$73.3\,(\pm\,0.90)$	$59.0\,(\pm\,1.84)$	$18.5\,(\pm\,0.37)$	$69.4\,(\pm\,0.90)$
VCD	$57.0\,(\pm\,1.02)$	$16.3\,(\pm\,0.85)$	$\textbf{76.7}\,(\pm\,0.54)$	$58.7\,(\pm\,0.19)$	$18.5\,(\pm\,0.47)$	$69.3\,(\pm\,0.57)$
ICD	$53.0\,(\pm\,1.93)$	$14.6\,(\pm\,0.85)$	$\textbf{76.7}\,(\pm\,0.46)$	$65.7\,(\pm\,0.77)$	$20.1\,(\pm\,0.34)$	$\textbf{74.9}\,(\pm\,1.61)$
M3ID	$54.5\,(\pm\,0.68)$	$15.9\,(\pm\,0.36)$	$73.5\,(\pm\,0.39)$	$69.8\,(\pm\,2.30)$	$21.4\,(\pm\,1.10)$	$70.8\,(\pm\,0.38)$
FLB (Ours)	$\textbf{43.5}\,(\pm\,1.24)$	$\textbf{12.0}\,(\pm\,0.50)$	$73.6\,(\pm\,0.48)$	$\textbf{52.5}\,(\pm\,0.47)$	$\textbf{15.8}\,(\pm\,0.86)$	$71.3\,(\pm\,0.49)$

Table 2: Performance comparison of LLaVA1.5 and InstructBLIP on CHAIR generative tasks.

	Average words	Average tokens
Baseline	79.58	104.67
FLB	78.62	101.40

Table 3: Average count of words and tokens in generated sequences tested on LLaVA1.5.

	Accuracy	Detailedness
Baseline	5.01	5.47
FLB	7.28	6.51

Table 4: GPT-4V-aided Evaluation Results.

Figure 5: Inference speed comparison across decoding strategies. This speed is measured by the time it takes to generate each token. VCD/ICD/M3ID run about twice as slow as the baseline, while FLB (ours) maintains similar speed.

	CHAIR $\downarrow$	Cover $\uparrow$	Hal $\downarrow$	Cog $\downarrow$
Baseline	11.9	49.6	48.8	4.4
Direct visual grounding only	9.2	50.3	41.1	4.7
“The” effect only	6.5	50.6	29.9	2.4
FLB (full)	5.7	50.3	30.7	2.4

Table 5: Ablation results isolating the two core effects of FLB.

	After The/the	After A/a
Ground truth	0.317 (2,424)	0.359 (853)
Hallucination	0.020 (150)	0.105 (248)

Table 6: Frequency statistics of token occurrences after articles. The table reports the proportion (and raw counts) of ground truth and hallucination words appearing after the articles The/the and A/a.

	After The/the	After A/a
Ground truth	0.279	0.225
Hallucination	0.012	0.029

Table 7: Average prediction probabilities of token occurrences after articles. The table presents the average model prediction probabilities for ground truth and hallucination tokens appearing after The/the and A/a.

Figure 6: Sentence-level impact of “The” on long-term hallucination. We compare probability of ground truth (left) and hallucination (right) words across generated token positions for two groups: sentences beginning with “The” and all other sentences. Sentences beginning with “The” show comparable ground truth probability to the overall distribution but exhibit substantially lower hallucination growth, particularly at later token positions where long-term decay intensifies. The widening gap between two groups demonstrate that “The” token provides a stabilizing initialization that suppresses cumulative hallucination.

		Entropy
Total	All tokens (total steps)	$1.815\,(105,803)$
	All nouns	$2.305\,(16,397)$
AMBER	Ground truth	1.948 (2,949)
	Hallucination	3.265 (589)
After articles	After The/the	2.001 (4,858)
	After other than The/the	2.433 (11,539)

Table 8: Logit entropy statistics under different conditions. The table compares average logit entropy across total tokens, AMBER ground truth versus hallucination nouns, and nouns following different articles. Numbers in parentheses denote sample counts.

	Baseline	“The” effect only	FLB (full)
Entropy (sample count)	2.305 (16,397)	2.210 (15,215)	2.181 (15,648)

Table 9: Comparison of total noun entropy before and after applying FLB. Numbers in parentheses denote sample counts.

	Baseline	VCD	FLB (Ours)
Average Score (variance)	1.944 ( $\pm 0.12$ )	2.098 ( $\pm 0.05$ )	2.230 ( $\pm 0.18$ )

Table 10: Performances of LLaVA1.5 on MMHalbench.

Win rate	Baseline	VCD	FLB (Ours)
1st turn	0.132	0.154	0.159
2nd turn	0.173	0.173	0.178
3rd turn	0.103	0.111	0.108

Table 11: Performances of LLaVA1.5 on Convbench.

Figure 7: MMHalBench results across token positions for each method (left) and the distribution of generated token lengths (right). Based on the token length distribution, we focus on sentences with 0–50 tokens and compare the average MMHalBench score. Unlike other methods, FLB maintains robust performance as sentence length increases.

Figure 8: FLB weight function candidates. Comparison of three

w_{t}

formulations for weight optimization across token positions.

$w_{t}$	(1) inc.	(2) dec.	(3) const.
object score	72.3	72.2	71.75

Table 12: Best object score in Eq. 8 at each weight function setting.

$\lambda$ / $\gamma$	0.1	0.3	0.5
0.005	71.15	71.2	71.4
0.02	71.65	71.8	72.1
0.05	71.8	72.3	71.7

Table 13: Object score in Eq. 8 at each hyperparameter setting on LLaVA1.5.

$\lambda$ / $\gamma$	0.1	0.3	0.5
0.005	72.4	72.3	72.0
0.02	72.3	72.4	72.3
0.05	72.2	72.6	72.1

Table 14: Object score in Eq. 8 at each weight setting on InstructBLIP.

$\beta$	CHAIR $\downarrow$	Cover $\uparrow$	Hal $\downarrow$	Cog $\downarrow$	Object score $\uparrow$
0	$7.8\,(\pm\,0.17)$	$50.2\,(\pm\,0.41)$	$39.7\,(\pm\,1.58)$	$2.9\,(\pm\,0.08)$	$71.2\,(\pm\,0.12)$
0.01	$7.5\,(\pm\,0.25)$	$50.3\,(\pm\,0.5)$	$37.5\,(\pm\,0.64)$	$3.3\,(\pm\,0.17)$	$71.4\,(\pm\,0.13)$
0.05	$6.8\,(\pm\,0.25)$	$50.4\,(\pm\,0.19)$	$33.3\,(\pm\,0.70)$	$2.8\,(\pm\,0.17)$	$71.8\,(\pm\,0.21)$
0.1	$6.1\,(\pm\,0.37)$	$50.4\,(\pm\,0.22)$	$31.4\,(\pm\,1.20)$	$2.7\,(\pm\,0.24)$	72.1 ${(\pm\,0.12)}$
0.2	$6.6\,(\pm\,0.21)$	$50.2\,(\pm\,0.21)$	$31.9\,(\pm\,1.07)$	$2.9\,(\pm\,0.29)$	$71.8\,(\pm\,0.18)$
0.4	$6.5\,(\pm\,0.08)$	$50.5\,(\pm\,0.17)$	$30.4\,(\pm\,0.52)$	$3.3\,(\pm\,0.22)$	$72.0\,(\pm\,0.10)$

Table 15: Hyperparameter optimization on

\beta

on AMBER benchmark.

Method	CHAIR $\downarrow$	Cover $\uparrow$	Hal $\downarrow$	Cog $\downarrow$
Baseline	7.1	50.5	32.4	3.8
VCD	8.2	52.2	38.0	4.0
ICD	6.4	51.0	30.6	3.2
M3ID	7.0	55.8	37.5	2.8
FLB (Ours)	4.9	48.8	25.2	2.3

Table 16: Performance comparison of LLaVA1.5 and greedy sampling on AMBER generative tasks. The highest scores are marked in bold .

Method	Proportion of “The” among sentence-initial tokens	CHAIR $\downarrow$	Cover $\uparrow$	Accuracy $\uparrow$	Detailedness $\uparrow$	Expression Diversity $\uparrow$
Baseline	67.4%	11.9	49.6	4.83	4.65	5.36
FLB ( $\gamma=0.1$ )	83.1%	7.5	51.1	6.21	5.80	6.23
FLB ( $\gamma=0.3$ )	89.4%	5.7	50.3	6.47	5.91	6.34
FLB ( $\gamma=0.5$ )	91.3%	5.8	49.2	6.26	5.77	6.10
FLB ( $\gamma=0.7$ )	92.1%	5.3	48.7	6.29	5.85	6.22

Table 17: Comparison of the frequency of “The” appearing as the sentence-initial token with and without applying FLB.

Method	CHAIR $\downarrow$	Cover $\uparrow$	Hal $\downarrow$	Cog $\downarrow$
Baseline	12.5	51.2	50.8	5.2
VCD	11.3	53.1	46.4	5.5
FLB	7.1	51.6	33.0	2.9

Table 18: Performance comparison of mPLUGOwl2 on AMBER. The highest scores are marked in bold .

Benchmark	POPE						MME
Dataset	Random		Popular		Adversarial
Metric	Acc.	F1	Acc.	F1	Acc.	F1	MME score
Baseline	0.829	0.808	0.811	0.792	0.786	0.771	114.20
$\beta$ -only	0.846	0.826	0.827	0.809	0.801	0.786	115.88
FLB (Ours)	0.846	0.826	0.827	0.809	0.801	0.786	115.88

Table 19: Performance comparison of LLaVA1.5 on discriminative tasks.

Figure 9: Example outputs from LLaVA-1.5 with FLB applied using $\beta=0$ and $\beta=0.1$ , illustrating that removing the adaptive plausible constraint ( $\beta=0$ ) leads to unnatural outputs.

VLM Hallucination Mitigation

Vision-language model hallucination mitigation keeps generated claims grounded in visual evidence.

See all 228 papers

NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors
Feb 25, 2026Lingfeng Ren, Weihao Yu, Runpeng Yu +1Object Hallucination in VLMs VLM Hallucination Mitigation
National University of Singapore, Singapore · Peking University Shenzhen Graduate School, China
Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination
Aug 7, 2026Zichuan Wang, Songlin Yang, Bo Peng +4Hallucination Detection in VLMs Vision-Language Models
School of Artificial Intelligence, University of Chinese Academy of Sciences · 3Hong Kong University of Science and Technology · 2New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Science
Test-Time Hallucination Control in Large Vision-Language Models
Aug 11, 2026Mehran Tamjidi, Hamidreza Dastmalchi, Ali Cheraghian +3Test-Time Adaptation of VLMs Large Vision-Language Models
University of Technology Sydney · York University, Canada · Australian National University +2

Feb 25, 2026J/K move · Enter open · S save

NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors

Lingfeng Ren, Weihao Yu, Runpeng Yu, Xinchao Wang

National University of Singapore, Singapore · Peking University Shenzhen Graduate School, China

Object Hallucination in VLMs VLM Hallucination Mitigation Large Vision-Language Models

Open paper AskPDF arXiv

Object hallucination is a critical issue in Large Vision-Language Models (LVLMs), where outputs include objects that do not appear in the input image. A natural question arises from this phenomenon: Which component of the LVLM pipeline primarily contributes to object hallucinations? The vision encoder to perceive visual information, or the language decoder to generate text responses? In this work, we strive to answer this question through designing a systematic experiment to analyze the roles of the vision encoder and the language decoder in hallucination generation. Our observations reveal that object hallucinations are predominantly associated with the strong priors from the language decoder. Based on this finding, we propose a simple and training-free framework, No-Language-Hallucination Decoding, NoLan, which refines the output distribution by dynamically suppressing language priors, modulated based on the output distribution difference between multimodal and text-only inputs. Experimental results demonstrate that NoLan effectively reduces object hallucinations across various LVLMs on different tasks. For instance, NoLan achieves substantial improvements on POPE, enhancing the accuracy of LLaVA-1.5 7B and Qwen-VL 7B by up to 6.45 and 7.21, respectively. The code is publicly available at: https://github.com/lingfengren/NoLan.

Abstract

VLM Hallucination Mitigation

Explore similar work

NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors

Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination

Test-Time Hallucination Control in Large Vision-Language Models