IKOD: Image-Attention-Guided Key-Value Merging against Object Hallucination in the Decoding of Large Vision-Language Models
Authors: Jiabing Yang, Chenhang Cui, Yixiang Chen, Tao Yu, Yan Huang, Liang Wang
Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences · NLPR, Institute of Automation, Chinese Academy of Sciences · National University of Singapore
Large vision-language models (LVLMs) exhibit strong visual understanding and language generation capabilities yet remain prone to object hallucination, producing plausible descriptions unsupported by the input image. Analyzing generation dynamics across eight diverse LVLMs, we observe that cross-modal visual attention steadily decays as generation unfolds because appended text tokens continuously expand the softmax denominator; consequently, hallucinated objects consistently appear later and under lower image attention than grounded mentions. Unlike existing training-free remedies that rely on input perturbations or external models, we propose IKOD (Image-attention-guided Key-value merging against Object hallucination in Decoding), a collaborative decoder that directly compresses the competing text context within the native key-value (KV) cache. At each step, IKOD keeps visual tokens intact and merges low-attention text entries into a compact auxiliary cache, fusing the resulting image-focused distribution with the base branch to correct ungrounded predictions while preserving linguistic fluency. An adaptive variant (IKOD-A) further schedules collaboration strength online from the measured attention decay under a single universal configuration. Extensive experiments across hallucination and general multimodal benchmarks on eight diverse LVLMs demonstrate that IKOD and IKOD-A consistently mitigate object hallucination while preserving object recall and general multimodal capabilities at minimal inference cost. Code is available at https://github.com/jiabingyang01/IKOD.
Figures & tables
Fig. 1: Visual attention trajectories. Normalized visual attention share against relative response position across eight LVLMs, averaged over 500 MSCOCO validation images. Each curve is scaled to its first bin, with the first-to-last-bin change shown in the legend.
Fig. 2: Layer- and head-wise visual attention. Visual attention share for the first 20% (left) and last 20% (middle) of generated tokens across three representative backbones, alongside their difference Slasti,h−Sfirsti,h (right, where red marks attention decay).
Fig. 3: Joint position–attention density of generated and hallucinated tokens. Kernel density over relative token position and visual attention share for all generated tokens (top) and hallucinated object mentions (bottom) across three representative backbones on 500 MSCOCO captions.
Fig. 4: Overview of the proposed IKOD and IKOD-A framework. (a) At each decoding step t , the base branch evaluates over the full KV cache Ct while the merged branch selects the lowest- smi post-visual text tokens as anchors and averages each contiguous bucket into a centroid (k~bi,v~bi) for additive collaborative decoding. (b) Bucket centroid merging provably increases visual attention share ( S′>S , Theorem 1 ). (c) IKOD-A schedules the collaboration weight αt online from real-time visual attention decay Dt .
Model
Split
Greedy
VCD
M3ID
OPERA ∗
ICD
SID
AGLA
ReVisiT
IKOD
IKOD-A
(Baseline)
(CVPR’24)
(CVPR’24)
(CVPR’24)
(ACL’24)
(ICLR’25)
(CVPR’25)
(ACL’26)
(Ours)
(Ours)
LLaVA-1.5 7B
Random
86.95
89.06
85.10
87.21
85.53
85.83
86.99
85.50
88.90
87.70
Popular
85.96
86.86
84.21
86.08
84.55
84.52
85.92
84.78
87.26
86.54
Adversarial
84.02
83.41
82.21
84.03
83.01
82.54
84.08
82.79
84.32
84.35
Average
85.64
86.45
83.84
85.77
84.36
84.30
85.67
84.36
86.82
86.20
LLaVA-NeXT 7B
Random
85.57
89.01
86.25
86.28
83.19
87.17
86.40
85.60
89.50
87.68
TABLE I: POPE F1 scores (%) on the MSCOCO validation set across eight LVLMs. Average denotes the three-split mean F1. Bold and underline mark the best and second-best results in each row. ∗ : five-beam search.
Model
Metric
Greedy
VCD
M3ID
OPERA ∗
ICD
SID
AGLA
ReVisiT
IKOD
IKOD-A
(Baseline)
(CVPR’24)
(CVPR’24)
(CVPR’24)
(ACL’24)
(ICLR’25)
(CVPR’25)
(ACL’26)
(Ours)
(Ours)
LLaVA-1.5 7B
CS↓
51.4
56.4
49.4
37.4
54.0
51.6
50.2
53.6
40.2
47.6
CI↓
15.9
17.0
15.5
12.3
16.8
15.0
15.6
16.1
12.5
14.3
Recall ↑
77.8
77.2
77.9
74.3
76.6
75.7
77.1
76.6
76.1
79.3
F1 ↑
80.8
80.0
81.1
80.4
79.8
80.1
80.6
80.1
81.4
82.4
Words
84.4
86.0
79.1
70.8
81.0
80.6
84.2
90.7
88.9
85.8
TABLE II: CHAIR results on 500 MSCOCO validation images across eight LVLMs. CS and CI denote sentence- and instance-level hallucination rates (%), Recall and F1 denote object recall and F1 (%), and Words is mean caption length. Bold and underline mark the best and second-best results per row. ∗ : five-beam search.
Method
LLaVA-1.5
LLaVA-NeXT
LLaVA-OV
Qwen2-VL
Qwen2.5-VL
Qwen3-VL
InternVL3
InstructBLIP
Average
(a) Fine-Grained Discriminative Probing: Attribute F1 (Attr ↑ ) / Relation F1 (Rel ↑ )
Attr ↑
Rel ↑
Attr ↑
Rel ↑
Attr ↑
Rel ↑
Attr ↑
Rel ↑
Attr ↑
Rel ↑
Attr ↑
Rel ↑
Attr ↑
Rel ↑
Attr ↑
Rel ↑
Attr ↑
Rel ↑
Greedy
77.5
41.2
81.4
52.0
88.0
77.4
86.2
74.4
83.0
72.3
87.4
87.1
86.4
80.2
79.7
76.6
83.7
70.1
VCD
76.7
49.7
80.3
59.6
87.6
78.1
85.1
72.3
83.1
72.0
86.8
84.9
85.8
81.6
79.6
76.3
83.1
71.8
M3ID
75.8
47.3
79.8
63.7
87.9
80.9
85.9
76.5
83.5
79.9
87.1
90.3
86.4
82.6
78.3
59.4
83.1
72.6
OPERA ∗
80.0
67.5
82.2
70.3
88.0
77.0
86.3
76.5
84.7
76.5
87.6
85.9
86.4
81.3
76.8
68.7
84.0
75.5
TABLE III: Fine-grained discriminative and open-ended generative results on AMBER [ 41 ] across eight LVLMs (%). (a) Attr ↑ (state, number, action) and Rel ↑ report binary F1 on fine-grained object attributes and inter-object relations. (b) Hal ↓ and Cog ↓ denote hallucinated response rate and human-cognition-aligned hallucination rate on 1,004 images. Bold and underline mark the best and second-best results per column within each block. ∗ : five-beam search.
TABLE IV: General multimodal understanding and reasoning results across all eight LVLMs. MME P denotes the MME perception score; MMS, MMB, and SQA denote MMStar, MMBench-EN (dev), and ScienceQA accuracy (%).
Setting
LLaVA-1.5
LLaVA-NeXT
LLaVA-OV
Qwen2-VL
Qwen2.5-VL
Qwen3-VL
InternVL3
InstructBLIP
Average
CI↓
F1 ↑
CI↓
F1 ↑
CI↓
F1 ↑
CI↓
F1 ↑
CI↓
F1 ↑
CI↓
F1 ↑
CI↓
F1 ↑
CI↓
F1 ↑
CI↓
F1 ↑
Greedy (base)
15.9
80.8
9.2
73.0
8.2
76.6
7.4
77.7
6.8
71.1
10.9
81.1
9.1
79.3
13.3
78.2
10.1
77.2
(a) Anchor selection criterion
High-attention
10.4
81.0
7.5
72.0
5.4
73.7
8.2
75.7
7.8
71.5
9.4
81.7
8.0
79.8
10.3
79.8
8.4
76.9
Uniform stride
12.9
81.4
6.6
74.5
5.4
75.2
5.5
77.2
7.3
72.4
9.2
82.2
8.8
79.3
11.4
79.0
8.4
77.7
Low-attention (IKOD)
12.5
81.4
6.1
73.9
6.4
72.9
4.0
78.4
6.4
70.8
9.2
82.1
7.7
79.7
11.3
79.3
8.0
77.3
TABLE V: Component ablations on attention-guided post-visual KV merging on CHAIR ( CI↓ / F1 ↑ , %). Each variant in (a)–(c) alters a single design factor while holding all other settings fixed to full IKOD. † At t=1 , ∣Pgen∣=0 (degenerating to Greedy at 86.95% POPE F1 vs. 87.97% for IKOD).
ℓtbase
ℓtmerged
Sign
POPE ↑
CS↓
CI↓
Recall ↑
F1 ↑
✓
×
–
86.95
36.9
10.1
68.1
77.2
×
✓
–
86.46
31.0
8.5
66.1
76.4
✓
✓
−α
79.30
41.8
15.7
60.5
70.0
✓
✓
+α
87.97
32.4
8.0
67.1
77.3
TABLE VI: Ablation on dual-branch collaborative decoding and fusion sign. Eight-LVLM mean POPE F1 and CHAIR metrics (%) at λ=0.4 . Bold and underline mark the best and second-best results.
Fig. 5: Joint sensitivity to the retained anchor ratio λ and collaboration weight α on POPE. Each 3D surface and floor projection maps three-split mean F1 (%) over λ∈{0.1,0.2,0.4,0.6,0.8} and α∈{0.1,0.5,1,2,3,4} across the LLaVA family (a–c), InternVL3 (d), the Qwen-VL family (e–g), and InstructBLIP (h). Dashed floor contours mark Greedy, crimson curves trace orthogonal cross-sections along λ and α , and gold stars mark the selected IKOD settings (λ=0.4,α) .
Model
Method
CS↓
CI↓
Recall ↑
F1 ↑
LLaVA-1.5-13B
Greedy
50.4
15.3
78.4
81.4
IKOD
38.8
10.0
77.0
83.0
IKOD-A
44.4
12.9
78.8
82.7
InstructBLIP-13B
Greedy
48.4
14.4
68.5
76.1
IKOD
42.2
11.9
67.7
76.6
IKOD-A
42.2
11.8
68.4
77.0
TABLE VII: Zero-shot cross-scale transfer on CHAIR across four 13B–32B backbones. IKOD and IKOD-A directly apply the 7B/8B settings on 500 MSCOCO validation images. Bold and underline mark the best and second-best results per block.
Sampling Strategy
Method
Acc. ↑
Prec. ↑
Recall ↑
F1 ↑
Top- P ( p=0.9,T=1.0 )
Base
86.46
91.90
80.43
85.59
IKOD
87.40
89.97
84.90
87.15
IKOD-A
86.77
90.53
82.86
86.28
Top- K ( k=50,T=1.0 )
Base
85.48
90.61
79.62
84.56
IKOD
87.21
89.85
84.59
86.94
IKOD-A
86.54
90.40
82.51
86.03
TABLE VIII: Robustness across stochastic sampling strategies and temperatures on POPE. Eight-LVLM three-split mean metrics (%). Bold and underline mark the best and second-best results per block.
Prompt / Setting
Method
CS↓
CI↓
Recall ↑
F1 ↑
Words
Describe this image.
Greedy
38.3
9.7
69.6
78.4
191.3
IKOD
32.5
8.0
67.2
77.4
204.9
IKOD-A
34.7
8.3
68.8
78.4
197.0
Please provide a detailed description…
Greedy
37.0
10.2
68.2
77.3
154.0
IKOD
31.1
7.8
66.2
76.7
176.1
IKOD-A
34.9
9.2
67.5
77.2
162.5
TABLE IX: Protocol robustness on CHAIR across eight LVLMs under alternative prompt phrasings (512-token cap), a strictly length-matched 128-token budget, and Top- P ( p=0.9,T=1.0 ) sampling across three random seeds (42, 1234, 2026). Bold and underline mark the best and second-best results per block.
Fig. 6: Dataset-wide closed-loop mechanistic evaluation across all eight LVLMs on CHAIR ( λ=0.4 ). (a) Horizontal dumbbell trajectories of relative visual-attention gain from hallucination-onset steps to late generation ( t≥T/2 ), paired with the empirical satisfaction rate Pr(St′>St) of Theorem 1 on the right. (b) Diverging step-level comparison between hallucination suppression on the left (hallucinated-noun probability drop, merged-branch top-1 reversal, and single-step collaborative reversal) and grounded-object preservation on the right (IKOD and IKOD-A). (c) Internal bucket anatomy showing anchor lexical composition and late-span bucket size (top) alongside the trade-off between grounded value cosine fidelity, hallucination-onset visual gain ΔSthal , and CHAIR CI across merging and pruning rules (bottom).
Fig. 7: Mechanistic case study on LLaVA-1.5 (COCO val2014). (a) Input outdoor vehicle scene containing zero people or handbags, paired with late-stage ( t>T/2 ) spatial image-attention overlays comparing the full base cache against the merged view ( λ=0.4 ). (b) Generated descriptions from Greedy, IKOD, and tuning-free IKOD-A, alongside live post-visual token merging where contiguous intervals centered at the lowest- sm anchors collapse into bucket centroids k~b . (c) Per-step visual attention share verifying Theorem 1 ( St′>St ) with exact logit-reversal points, stacked above IKOD-A’s online decay gate αt and compressed post-visual KV length.
Method
ms/tok ↓
Time ×↓
Mem ×↓
POPE ↑
CI↓
F1 ↑
Greedy
24.3
1.00
1.00
86.95
10.1
77.2
VCD
47.7
1.97
1.03
87.60
11.9
75.9
M3ID
46.8
1.92
1.00
86.96
10.9
77.5
OPERA ∗
253.6
10.67
1.17
87.14
9.6
75.3
ICD
48.0
1.98
1.04
86.03
11.9
75.5
SID
49.6
2.04
1.03
87.03
11.5
76.7
TABLE X: Inference efficiency and effectiveness trade-off across eight LVLMs. Eight-LVLM mean 512-token captioning cost alongside mean POPE F1, CHAIR CI , and object F1 (%). ∗ : five-beam search.
TABLE XI: Hyperparameter settings of all baselines and our methods. ∗ : five-beam search.
Fig. 8: Per-layer and per-head image attention for the earliest 20% (left) and latest 20% (middle) of generated tokens across all eight backbones, alongside their late-minus-early difference (right; red indicates attention decay).
Fig. 9: Joint position–attention density of generated and hallucinated tokens across all eight backbones. Kernel density of all generated tokens (top row) and hallucinated object mentions (bottom row) over relative token position and visual attention share on 500 MSCOCO captions.
TABLE XII: Dataset-wide closed-loop mechanistic verification across eight LVLMs on CHAIR at λ=0.4 (numerical breakdown for Fig. 6 of the main paper): (a) step-level visual attention restoration, online gate response, and object-level intervention; (b) eight-LVLM mean bucket composition, granularity, and representation fidelity across KV merging rules.
Model
Method
Random
Popular
Adversarial
Average
LLaVA-1.5-13B
Greedy
86.28
85.58
83.97
85.28
IKOD
87.85
86.68
84.87
86.47
IKOD-A
87.18
86.38
84.63
86.06
InstructBLIP-13B
Greedy
84.79
82.92
80.89
82.87
IKOD
85.64
83.26
81.27
83.39
IKOD-A
85.64
83.26
81.27
83.39
TABLE XIII: Zero-shot POPE-COCO F1 ( ↑ , %) across four 13B–32B model families without retuning. Bold and underline mark the best and second-best results within each block.
Fig. 10: Per-backbone sensitivity to the retained-anchor ratio λ on CHAIR (512-token budget), showing CI (left axis) and recall (right axis) at the main-table family α . Dashed line: Greedy CI ; dotted line: Greedy recall minus two points; star and band: universal λ=0.4 .
Fig. 11: Per-backbone sensitivity to the collaboration weight α on CHAIR (512-token budget), showing CI (left axis) and recall (right axis) at the universal anchor ratio λ=0.4 . Dashed line: Greedy CI ; dotted line: Greedy recall minus two points; star and band: family α transferred from POPE.
Strategy
Method
LLaVA-1.5
LLaVA-NeXT
LLaVA-OV
Qwen2-VL
Qwen2.5-VL
Qwen3-VL
InternVL3
InstructBLIP
Average
Top- P ( p=0.9,T=1.0 )
Base
82.44
82.77
87.62
87.01
85.49
88.33
90.09
80.93
85.59
IKOD
86.03
87.21
89.91
87.56
86.36
89.50
90.01
80.64
87.15
IKOD-A
83.71
84.37
88.68
87.49
86.68
88.90
89.68
80.72
86.28
Top- K ( k=50,T=1.0 )
Base
80.09
80.71
86.78
86.20
84.90
88.19
89.51
80.13
84.56
IKOD
85.62
86.81
89.82
87.31
86.26
89.46
89.89
80.33
86.94
IKOD-A
83.33
83.86
88.34
87.29
86.51
88.88
89.66
80.33
86.03
TABLE XIV: Per-backbone three-split mean POPE-COCO F1 ( ↑ , %) across five stochastic sampling strategies and temperatures on all eight LVLMs (eight-LVLM means in Table VIII of the main paper). Bold and underline mark the best and second-best results within each block.
Random Seed
Method
LLaVA-1.5
LLaVA-NeXT
LLaVA-OV
Qwen2-VL
Qwen2.5-VL
Qwen3-VL
InternVL3
InstructBLIP
Average
Seed 42
Base
19.2
11.2
10.0
10.7
9.1
10.8
8.9
17.2
12.1
IKOD
13.6
7.5
6.1
7.3
8.7
9.2
9.3
13.8
9.4
IKOD-A
17.3
7.9
5.8
6.4
8.4
9.9
8.1
12.5
9.5
Seed 1234
Base
19.2
10.6
10.0
10.9
10.1
11.1
8.7
17.1
12.2
IKOD
14.3
7.2
3.8
5.6
6.0
9.4
9.2
12.6
8.5
IKOD-A
15.1
8.4
7.5
7.6
7.5
10.3
8.0
14.0
9.8
TABLE XV: Per-backbone CHAIR-500 instance hallucination CI↓ (%) under Top- P ( p=0.9,T=1.0 ) sampling across three independent random seeds on all eight LVLMs (eight-LVLM means in Table IX of the main paper). Bold and underline mark the best and second-best results within each block.
Greedy
IKOD
IKOD-A
Prompt
Model
C S↓
C I↓
Recall ↑
F1 ↑
Words
C S↓
C I↓
Recall ↑
F1 ↑
Words
C S↓
C I↓
Recall ↑
F1 ↑
Words
Describe this image.
LLaVA-1.5
49.2
15.0
78.3
81.5
82.6
40.2
11.6
76.1
81.8
87.9
51.0
15.5
78.3
81.3
84.4
LLaVA-NeXT
27.0
8.3
62.1
74.1
150.0
26.2
6.2
62.0
74.7
156.1
26.8
7.5
62.8
74.8
148.7
LLaVA-OV
44.0
8.6
72.5
80.9
309.1
26.0
4.5
61.2
74.6
390.1
34.0
4.7
68.7
79.8
356.3
Qwen2-VL
31.8
8.8
68.1
78.0
159.0
27.2
11.1
66.5
76.1
165.3
28.0
7.4
69.8
79.6
149.9
Qwen2.5-VL
21.8
7.9
57.3
70.6
87.7
18.6
5.3
54.9
69.5
97.0
17.6
6.1
55.5
69.8
89.9
TABLE XVI: Per-backbone CHAIR-500 metrics (%) and mean caption length (Words) across three alternative prompt phrasings under unified λ=0.4 (eight-LVLM means in Table IX of the main paper). Bold and underline mark the best and second-best eight-backbone averages per prompt.
Greedy
IKOD
IKOD-A
Model
C S↓
C I↓
Recall ↑
F1 ↑
Words
C S↓
C I↓
Recall ↑
F1 ↑
Words
C S↓
C I↓
Recall ↑
F1 ↑
Words
LLaVA-1.5
50.8
15.8
77.3
80.6
82.7
40.2
12.7
75.6
81.0
84.9
47.2
14.1
79.0
82.3
83.7
LLaVA-NeXT
22.4
8.6
54.5
68.3
90.2
20.0
6.1
56.8
70.8
89.8
22.2
7.2
55.2
69.2
89.4
LLaVA-OV
23.8
7.7
55.9
69.6
106.4
20.2
6.1
52.5
67.3
106.6
22.6
6.2
55.6
69.8
106.7
Qwen2-VL
24.2
7.4
66.2
77.2
75.1
21.0
5.0
66.1
78.0
77.3
23.0
6.8
68.1
78.7
75.4
Qwen2.5-VL
23.6
8.7
54.4
68.2
94.4
22.4
6.7
54.3
68.7
95.5
23.4
8.0
53.5
67.8
93.8
TABLE XVII: Per-backbone CHAIR-500 metrics (%) and mean caption length (Words) under the 128-token generation budget at unified λ=0.4 (eight-LVLM means in Table IX of the main paper). Bold and underline mark the best and second-best eight-backbone averages.
Model
Greedy
VCD
M3ID
OPERA ∗
ICD
SID
AGLA
ReVisiT
IKOD
IKOD-A
LLaVA-1.5
21.6
42.4
39.9
321.4
43.5
43.1
44.8
22.1
25.0
25.0
LLaVA-NeXT
22.3
44.1
42.9
331.0
45.4
46.2
47.1
24.3
25.4
25.7
LLaVA-OV
23.5
46.7
45.1
274.4
46.1
48.6
47.0
26.2
26.6
27.0
Qwen2-VL
24.1
46.8
47.1
161.1
47.1
49.8
49.2
24.8
26.5
26.5
Qwen2.5-VL
23.8
45.7
46.9
246.4
45.9
49.0
47.5
24.0
26.2
26.4
Qwen3-VL
31.0
60.8
61.7
218.1
60.9
63.5
62.2
31.3
35.6
35.6
TABLE XVIII: Per-backbone 512-token decoding latency (ms/tok ↓ ) and eight-backbone relative slowdown ( Time×↓ ) across all ten decoding methods under unified λ=0.4 on NVIDIA A100 80GB GPUs. ∗ : five-beam search.
Fig. 12: Additional mechanistic case study on LLaVA-1.5 (COCO val2014, twin parked airplanes under unified λ=0.4 ). (a) Input airport scene containing zero people or passengers and late-stage ( t>T/2 ) spatial attention overlays. (b) Generated captions from Greedy, IKOD, and IKOD-A alongside lowest- sm bucket merging ( Db→k~b ). (c) Per-step visual attention recovery ( St′>St ) with exact logit-reversal steps and IKOD-A’s online gate trajectory αt .
Fig. 13: Qualitative comparison on LLaVA-1.5 against Greedy and the seven baselines of Table II. Hallucinated objects identified by CHAIR are highlighted in red.
Object hallucination is a critical issue in Large Vision-Language Models (LVLMs), where outputs include objects that do not appear in the input image. A natural question arises from this phenomenon: Which component of the LVLM pipeline primarily contributes to object hallucinations? The vision encoder to perceive visual information, or the language decoder to generate text responses? In this work, we strive to answer this question through designing a systematic experiment to analyze the roles of the vision encoder and the language decoder in hallucination generation. Our observations reveal that object hallucinations are predominantly associated with the strong priors from the language decoder. Based on this finding, we propose a simple and training-free framework, No-Language-Hallucination Decoding, NoLan, which refines the output distribution by dynamically suppressing language priors, modulated based on the output distribution difference between multimodal and text-only inputs. Experimental results demonstrate that NoLan effectively reduces object hallucinations across various LVLMs on different tasks. For instance, NoLan achieves substantial improvements on POPE, enhancing the accuracy of LLaVA-1.5 7B and Qwen-VL 7B by up to 6.45 and 7.21, respectively. The code is publicly available at: https://github.com/lingfengren/NoLan.
Lingfeng Ren, Weihao Yu, Runpeng Yu +1
National University of Singapore, Singapore · Peking University Shenzhen Graduate School, China
Large Vision-Language Models (LVLMs) often suffer from object hallucination, generating objects that are absent from the image. Prior work largely attributes this to insufficient visual attention. However, we find that both real and hallucinated objects receive equally strong visual attention in the model's mid-to-late layers, suggesting that the key issue may not be how much the model attends, but what it attends to and why. To this end, we decode the visual features of high-attention regions using Logit Lens, and observe that regions corresponding to real objects can be correctly decoded to the target object tokens, whereas those for hallucinated objects cannot. Building on this, we identify two hallucination mechanisms: (i) visual uncertainty, triggered by semantically similar or confusable regions; masking these regions eliminates the hallucination. (ii) contextual prior, triggered by strong co-occurrence priors; even when the initially attended region is masked, the hallucination persists and attention drifts to other regions. Based on these findings, we propose a simple yet effective training-free Detect-Mitigate framework comprising a Logit-Lens Consistency Check to detect hallucination and targeted remedies: High-Attention Regions Masking (HARM) for visual uncertainty hallucination, and Visual Evidence Enhanced Decoding (VEED) for contextual prior hallucination. Our approach achieves state-of-the-art results on multiple hallucination benchmarks. Code will be available.
Zichuan Wang, Songlin Yang, Bo Peng +4
School of Artificial Intelligence, University of Chinese Academy of Sciences · 3Hong Kong University of Science and Technology · 2New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Science
Recent Large Vision-Language Models (LVLMs) have demonstrated remarkable performance across various multimodal tasks that require understanding both visual and linguistic inputs. However, object hallucination -- the generation of nonexistent objects in answers -- remains a persistent challenge. Although several approaches such as retraining and external grounding methods have been proposed to mitigate this issue, they still suffer from high data costs or structural complexity. Training-free methods such as Contrastive Decoding (CD) are more cost-effective, avoiding additional training or external models, but still suffer from long-term decay, where visual grounding weakens and language priors dominate as the generation progresses. In this paper, we propose First Logit Boosting (FLB), a simple yet effective training-free technique designed to alleviate long-term decay in LVLMs. FLB stores the logit of the first generated token and adds it to subsequent token predictions, effectively mitigating long-term decay of visual information. We observe that FLB (1) sustains the visual information embedded in the first token throughout generation, and (2) suppresses hallucinated words through the stabilizing effect of the ``The'' token. Experimental results show that FLB significantly reduces object hallucination across various tasks, benchmarks, and backbone models. Notably, it causes negligible inference overhead, making it highly applicable to real-time multimodal systems. Code is available at https://github.com/jiwooha20/FLB