IKOD: Image-Attention-Guided Key-Value Merging against Object Hallucination in the Decoding of Large Vision-Language Models
Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences · NLPR, Institute of Automation, Chinese Academy of Sciences · National University of Singapore
Abstract
Large vision-language models (LVLMs) exhibit strong visual understanding and language generation capabilities yet remain prone to object hallucination, producing plausible descriptions unsupported by the input image. Analyzing generation dynamics across eight diverse LVLMs, we observe that cross-modal visual attention steadily decays as generation unfolds because appended text tokens continuously expand the softmax denominator; consequently, hallucinated objects consistently appear later and under lower image attention than grounded mentions. Unlike existing training-free remedies that rely on input perturbations or external models, we propose IKOD (Image-attention-guided Key-value merging against Object hallucination in Decoding), a collaborative decoder that directly compresses the competing text context within the native key-value (KV) cache. At each step, IKOD keeps visual tokens intact and merges low-attention text entries into a compact auxiliary cache, fusing the resulting image-focused distribution with the base branch to correct ungrounded predictions while preserving linguistic fluency. An adaptive variant (IKOD-A) further schedules collaboration strength online from the measured attention decay under a single universal configuration. Extensive experiments across hallucination and general multimodal benchmarks on eight diverse LVLMs demonstrate that IKOD and IKOD-A consistently mitigate object hallucination while preserving object recall and general multimodal capabilities at minimal inference cost. Code is available at https://github.com/jiabingyang01/IKOD.
Figures & tables
| Model | Split | Greedy | VCD | M3ID | OPERA ∗ | ICD | SID | AGLA | ReVisiT | IKOD | IKOD-A |
|---|---|---|---|---|---|---|---|---|---|---|---|
| (Baseline) | (CVPR’24) | (CVPR’24) | (CVPR’24) | (ACL’24) | (ICLR’25) | (CVPR’25) | (ACL’26) | (Ours) | (Ours) | ||
| LLaVA-1.5 7B | Random | 86.95 | 89.06 | 85.10 | 87.21 | 85.53 | 85.83 | 86.99 | 85.50 | 88.90 | 87.70 |
| Popular | 85.96 | 86.86 | 84.21 | 86.08 | 84.55 | 84.52 | 85.92 | 84.78 | 87.26 | 86.54 | |
| Adversarial | 84.02 | 83.41 | 82.21 | 84.03 | 83.01 | 82.54 | 84.08 | 82.79 | 84.32 | 84.35 | |
| Average | 85.64 | 86.45 | 83.84 | 85.77 | 84.36 | 84.30 | 85.67 | 84.36 | 86.82 | 86.20 | |
| LLaVA-NeXT 7B | Random | 85.57 | 89.01 | 86.25 | 86.28 | 83.19 | 87.17 | 86.40 | 85.60 | 89.50 | 87.68 |
| Model | Metric | Greedy | VCD | M3ID | OPERA ∗ | ICD | SID | AGLA | ReVisiT | IKOD | IKOD-A |
|---|---|---|---|---|---|---|---|---|---|---|---|
| (Baseline) | (CVPR’24) | (CVPR’24) | (CVPR’24) | (ACL’24) | (ICLR’25) | (CVPR’25) | (ACL’26) | (Ours) | (Ours) | ||
| LLaVA-1.5 7B | 51.4 | 56.4 | 49.4 | 37.4 | 54.0 | 51.6 | 50.2 | 53.6 | 40.2 | 47.6 | |
| 15.9 | 17.0 | 15.5 | 12.3 | 16.8 | 15.0 | 15.6 | 16.1 | 12.5 | 14.3 | ||
| Recall | 77.8 | 77.2 | 77.9 | 74.3 | 76.6 | 75.7 | 77.1 | 76.6 | 76.1 | 79.3 | |
| F1 | 80.8 | 80.0 | 81.1 | 80.4 | 79.8 | 80.1 | 80.6 | 80.1 | 81.4 | 82.4 | |
| Words | 84.4 | 86.0 | 79.1 | 70.8 | 81.0 | 80.6 | 84.2 | 90.7 | 88.9 | 85.8 |
| Method | LLaVA-1.5 | LLaVA-NeXT | LLaVA-OV | Qwen2-VL | Qwen2.5-VL | Qwen3-VL | InternVL3 | InstructBLIP | Average | |||||||||
| (a) Fine-Grained Discriminative Probing: Attribute F1 (Attr ) / Relation F1 (Rel ) | ||||||||||||||||||
| Attr | Rel | Attr | Rel | Attr | Rel | Attr | Rel | Attr | Rel | Attr | Rel | Attr | Rel | Attr | Rel | Attr | Rel | |
| Greedy | 77.5 | 41.2 | 81.4 | 52.0 | 88.0 | 77.4 | 86.2 | 74.4 | 83.0 | 72.3 | 87.4 | 87.1 | 86.4 | 80.2 | 79.7 | 76.6 | 83.7 | 70.1 |
| VCD | 76.7 | 49.7 | 80.3 | 59.6 | 87.6 | 78.1 | 85.1 | 72.3 | 83.1 | 72.0 | 86.8 | 84.9 | 85.8 | 81.6 | 79.6 | 76.3 | 83.1 | 71.8 |
| M3ID | 75.8 | 47.3 | 79.8 | 63.7 | 87.9 | 80.9 | 85.9 | 76.5 | 83.5 | 79.9 | 87.1 | 90.3 | 86.4 | 82.6 | 78.3 | 59.4 | 83.1 | 72.6 |
| OPERA ∗ | 80.0 | 67.5 | 82.2 | 70.3 | 88.0 | 77.0 | 86.3 | 76.5 | 84.7 | 76.5 | 87.6 | 85.9 | 86.4 | 81.3 | 76.8 | 68.7 | 84.0 | 75.5 |
| Method | LLaVA-1.5 | LLaVA-NeXT | LLaVA-OV | Qwen2-VL | Qwen2.5-VL | Qwen3-VL | InternVL3 | InstructBLIP | Average | |||||||||
| (a) Perception & Vision-Indispensable Reasoning: MME Perception (MME ) / MMStar (MMS ) | ||||||||||||||||||
| MME P | MMS | MME P | MMS | MME P | MMS | MME P | MMS | MME P | MMS | MME P | MMS | MME P | MMS | MME P | MMS | MME P | MMS | |
| Greedy | 1455 | 33.3 | 1525 | 37.5 | 1559 | 58.9 | 1681 | 57.5 | 1639 | 58.2 | 1712 | 64.2 | 1701 | 66.2 | 1247 | 3.7 | 1565 | 47.4 |
| IKOD | 1364 | 33.8 | 1434 | 36.8 | 1354 | 54.8 | 1678 | 51.6 | 1655 | 57.5 | 1727 | 60.2 | 1696 | 65.8 | 1193 | 0.4 | 1513 | 45.1 |
| IKOD-A | 1438 | 33.6 | 1486 | 36.6 | 1506 | 57.9 | 1672 | 54.2 | 1655 | 58.3 | 1722 | 63.8 | 1688 | 65.8 | 1193 | 0.4 | 1545 | 46.3 |
| (b) General Multimodal Comprehension & Science Reasoning: MMBench-EN (MMB ) / ScienceQA-IMG (SQA ) | ||||||||||||||||||
| Setting | LLaVA-1.5 | LLaVA-NeXT | LLaVA-OV | Qwen2-VL | Qwen2.5-VL | Qwen3-VL | InternVL3 | InstructBLIP | Average | |||||||||
| F1 | F1 | F1 | F1 | F1 | F1 | F1 | F1 | F1 | ||||||||||
| Greedy (base) | 15.9 | 80.8 | 9.2 | 73.0 | 8.2 | 76.6 | 7.4 | 77.7 | 6.8 | 71.1 | 10.9 | 81.1 | 9.1 | 79.3 | 13.3 | 78.2 | 10.1 | 77.2 |
| (a) Anchor selection criterion | ||||||||||||||||||
| High-attention | 10.4 | 81.0 | 7.5 | 72.0 | 5.4 | 73.7 | 8.2 | 75.7 | 7.8 | 71.5 | 9.4 | 81.7 | 8.0 | 79.8 | 10.3 | 79.8 | 8.4 | 76.9 |
| Uniform stride | 12.9 | 81.4 | 6.6 | 74.5 | 5.4 | 75.2 | 5.5 | 77.2 | 7.3 | 72.4 | 9.2 | 82.2 | 8.8 | 79.3 | 11.4 | 79.0 | 8.4 | 77.7 |
| Low-attention (IKOD) | 12.5 | 81.4 | 6.1 | 73.9 | 6.4 | 72.9 | 4.0 | 78.4 | 6.4 | 70.8 | 9.2 | 82.1 | 7.7 | 79.7 | 11.3 | 79.3 | 8.0 | 77.3 |
| Sign | POPE | Recall | F1 | ||||
|---|---|---|---|---|---|---|---|
| ✓ | – | 86.95 | 36.9 | 10.1 | 68.1 | 77.2 | |
| ✓ | – | 86.46 | 31.0 | 8.5 | 66.1 | 76.4 | |
| ✓ | ✓ | 79.30 | 41.8 | 15.7 | 60.5 | 70.0 | |
| ✓ | ✓ | 87.97 | 32.4 | 8.0 | 67.1 | 77.3 |
| Model | Method | Recall | F1 | ||
|---|---|---|---|---|---|
| LLaVA-1.5-13B | Greedy | 50.4 | 15.3 | 78.4 | 81.4 |
| IKOD | 38.8 | 10.0 | 77.0 | 83.0 | |
| IKOD-A | 44.4 | 12.9 | 78.8 | 82.7 | |
| InstructBLIP-13B | Greedy | 48.4 | 14.4 | 68.5 | 76.1 |
| IKOD | 42.2 | 11.9 | 67.7 | 76.6 | |
| IKOD-A | 42.2 | 11.8 | 68.4 | 77.0 |
| Sampling Strategy | Method | Acc. | Prec. | Recall | F1 |
|---|---|---|---|---|---|
| Top- ( ) | Base | 86.46 | 91.90 | 80.43 | 85.59 |
| IKOD | 87.40 | 89.97 | 84.90 | 87.15 | |
| IKOD-A | 86.77 | 90.53 | 82.86 | 86.28 | |
| Top- ( ) | Base | 85.48 | 90.61 | 79.62 | 84.56 |
| IKOD | 87.21 | 89.85 | 84.59 | 86.94 | |
| IKOD-A | 86.54 | 90.40 | 82.51 | 86.03 |
| Prompt / Setting | Method | Recall | F1 | Words | ||
|---|---|---|---|---|---|---|
| Describe this image. | Greedy | 38.3 | 9.7 | 69.6 | 78.4 | 191.3 |
| IKOD | 32.5 | 8.0 | 67.2 | 77.4 | 204.9 | |
| IKOD-A | 34.7 | 8.3 | 68.8 | 78.4 | 197.0 | |
| Please provide a detailed description… | Greedy | 37.0 | 10.2 | 68.2 | 77.3 | 154.0 |
| IKOD | 31.1 | 7.8 | 66.2 | 76.7 | 176.1 | |
| IKOD-A | 34.9 | 9.2 | 67.5 | 77.2 | 162.5 |
| Method | ms/tok | Time | Mem | POPE | F1 | |
|---|---|---|---|---|---|---|
| Greedy | 24.3 | 1.00 | 1.00 | 86.95 | 10.1 | 77.2 |
| VCD | 47.7 | 1.97 | 1.03 | 87.60 | 11.9 | 75.9 |
| M3ID | 46.8 | 1.92 | 1.00 | 86.96 | 10.9 | 77.5 |
| OPERA ∗ | 253.6 | 10.67 | 1.17 | 87.14 | 9.6 | 75.3 |
| ICD | 48.0 | 1.98 | 1.04 | 86.03 | 11.9 | 75.5 |
| SID | 49.6 | 2.04 | 1.03 | 87.03 | 11.5 | 76.7 |
| Method | Hyperparameter Settings |
|---|---|
| VCD [ 6 ] | , ; diffusion steps (POPE), (CHAIR) |
| M3ID [ 16 ] | Decay , confidence gate , (POPE) / (CHAIR) |
| OPERA ∗ [ 7 ] | , scale , threshold , , penalty |
| ICD [ 25 ] | , ; negative role disturbance prefix |
| SID [ 26 ] | , , pruning layer , visual token retention |
| AGLA [ 8 ] | , ; BLIP-ITM Grad-CAM block |
| (a) Step-level visual attention restoration ( , %), online gate response ( ), and object intervention | ||||||||||
| Model | Early | Late | GT | Hal | Hal@Late | Early Late | Hal Rev. | GT Keep | ||
| LLaVA-1.5 | 11.1 19.2 | 6.5 11.4 | 13.5 15.3 | 11.6 13.4 | 99.2/72.3 | 75.6% | 0.63 0.83 | 69.2% | 71.9/54.6 | 82.1/91.4 |
| LLaVA-NeXT | 9.1 14.8 | 5.9 13.8 | 11.0 13.9 | 9.5 13.4 | 100/90.5 | 42.1% | 0.67 0.83 | 66.1% | 68.4/40.3 | 85.8/93.5 |
| LLaVA-OV | 12.7 20.7 | 7.6 15.4 | 15.6 21.0 | 12.4 19.6 | 96.7/92.3 | 43.9% | 0.71 0.82 | 66.9% | 69.9/36.7 | 84.4/92.7 |
| Qwen2-VL | 13.2 17.4 | 8.3 13.1 | 14.7 19.2 | 12.5 17.1 | 100/100 | 61.7% | 0.58 0.83 | 66.8% | 71.8/27.7 | 87.7/94.1 |
| Qwen2.5-VL | 11.4 14.9 | 7.6 11.5 | 13.5 17.5 | 11.7 16.0 | 98.3/99.5 | 50.9% | 0.53 0.66 | 65.1% | 68.4/29.3 | 90.4/94.5 |
| Model | Method | Random | Popular | Adversarial | Average |
|---|---|---|---|---|---|
| LLaVA-1.5-13B | Greedy | 86.28 | 85.58 | 83.97 | 85.28 |
| IKOD | 87.85 | 86.68 | 84.87 | 86.47 | |
| IKOD-A | 87.18 | 86.38 | 84.63 | 86.06 | |
| InstructBLIP-13B | Greedy | 84.79 | 82.92 | 80.89 | 82.87 |
| IKOD | 85.64 | 83.26 | 81.27 | 83.39 | |
| IKOD-A | 85.64 | 83.26 | 81.27 | 83.39 |
| Strategy | Method | LLaVA-1.5 | LLaVA-NeXT | LLaVA-OV | Qwen2-VL | Qwen2.5-VL | Qwen3-VL | InternVL3 | InstructBLIP | Average |
|---|---|---|---|---|---|---|---|---|---|---|
| Top- ( ) | Base | 82.44 | 82.77 | 87.62 | 87.01 | 85.49 | 88.33 | 90.09 | 80.93 | 85.59 |
| IKOD | 86.03 | 87.21 | 89.91 | 87.56 | 86.36 | 89.50 | 90.01 | 80.64 | 87.15 | |
| IKOD-A | 83.71 | 84.37 | 88.68 | 87.49 | 86.68 | 88.90 | 89.68 | 80.72 | 86.28 | |
| Top- ( ) | Base | 80.09 | 80.71 | 86.78 | 86.20 | 84.90 | 88.19 | 89.51 | 80.13 | 84.56 |
| IKOD | 85.62 | 86.81 | 89.82 | 87.31 | 86.26 | 89.46 | 89.89 | 80.33 | 86.94 | |
| IKOD-A | 83.33 | 83.86 | 88.34 | 87.29 | 86.51 | 88.88 | 89.66 | 80.33 | 86.03 |
| Random Seed | Method | LLaVA-1.5 | LLaVA-NeXT | LLaVA-OV | Qwen2-VL | Qwen2.5-VL | Qwen3-VL | InternVL3 | InstructBLIP | Average |
|---|---|---|---|---|---|---|---|---|---|---|
| Seed 42 | Base | 19.2 | 11.2 | 10.0 | 10.7 | 9.1 | 10.8 | 8.9 | 17.2 | 12.1 |
| IKOD | 13.6 | 7.5 | 6.1 | 7.3 | 8.7 | 9.2 | 9.3 | 13.8 | 9.4 | |
| IKOD-A | 17.3 | 7.9 | 5.8 | 6.4 | 8.4 | 9.9 | 8.1 | 12.5 | 9.5 | |
| Seed 1234 | Base | 19.2 | 10.6 | 10.0 | 10.9 | 10.1 | 11.1 | 8.7 | 17.1 | 12.2 |
| IKOD | 14.3 | 7.2 | 3.8 | 5.6 | 6.0 | 9.4 | 9.2 | 12.6 | 8.5 | |
| IKOD-A | 15.1 | 8.4 | 7.5 | 7.6 | 7.5 | 10.3 | 8.0 | 14.0 | 9.8 |
| Greedy | IKOD | IKOD-A | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Prompt | Model | C | C | Recall | F1 | Words | C | C | Recall | F1 | Words | C | C | Recall | F1 | Words |
| Describe this image. | LLaVA-1.5 | 49.2 | 15.0 | 78.3 | 81.5 | 82.6 | 40.2 | 11.6 | 76.1 | 81.8 | 87.9 | 51.0 | 15.5 | 78.3 | 81.3 | 84.4 |
| LLaVA-NeXT | 27.0 | 8.3 | 62.1 | 74.1 | 150.0 | 26.2 | 6.2 | 62.0 | 74.7 | 156.1 | 26.8 | 7.5 | 62.8 | 74.8 | 148.7 | |
| LLaVA-OV | 44.0 | 8.6 | 72.5 | 80.9 | 309.1 | 26.0 | 4.5 | 61.2 | 74.6 | 390.1 | 34.0 | 4.7 | 68.7 | 79.8 | 356.3 | |
| Qwen2-VL | 31.8 | 8.8 | 68.1 | 78.0 | 159.0 | 27.2 | 11.1 | 66.5 | 76.1 | 165.3 | 28.0 | 7.4 | 69.8 | 79.6 | 149.9 | |
| Qwen2.5-VL | 21.8 | 7.9 | 57.3 | 70.6 | 87.7 | 18.6 | 5.3 | 54.9 | 69.5 | 97.0 | 17.6 | 6.1 | 55.5 | 69.8 | 89.9 | |
| Greedy | IKOD | IKOD-A | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | C | C | Recall | F1 | Words | C | C | Recall | F1 | Words | C | C | Recall | F1 | Words |
| LLaVA-1.5 | 50.8 | 15.8 | 77.3 | 80.6 | 82.7 | 40.2 | 12.7 | 75.6 | 81.0 | 84.9 | 47.2 | 14.1 | 79.0 | 82.3 | 83.7 |
| LLaVA-NeXT | 22.4 | 8.6 | 54.5 | 68.3 | 90.2 | 20.0 | 6.1 | 56.8 | 70.8 | 89.8 | 22.2 | 7.2 | 55.2 | 69.2 | 89.4 |
| LLaVA-OV | 23.8 | 7.7 | 55.9 | 69.6 | 106.4 | 20.2 | 6.1 | 52.5 | 67.3 | 106.6 | 22.6 | 6.2 | 55.6 | 69.8 | 106.7 |
| Qwen2-VL | 24.2 | 7.4 | 66.2 | 77.2 | 75.1 | 21.0 | 5.0 | 66.1 | 78.0 | 77.3 | 23.0 | 6.8 | 68.1 | 78.7 | 75.4 |
| Qwen2.5-VL | 23.6 | 8.7 | 54.4 | 68.2 | 94.4 | 22.4 | 6.7 | 54.3 | 68.7 | 95.5 | 23.4 | 8.0 | 53.5 | 67.8 | 93.8 |
| Model | Greedy | VCD | M3ID | OPERA ∗ | ICD | SID | AGLA | ReVisiT | IKOD | IKOD-A |
|---|---|---|---|---|---|---|---|---|---|---|
| LLaVA-1.5 | 21.6 | 42.4 | 39.9 | 321.4 | 43.5 | 43.1 | 44.8 | 22.1 | 25.0 | 25.0 |
| LLaVA-NeXT | 22.3 | 44.1 | 42.9 | 331.0 | 45.4 | 46.2 | 47.1 | 24.3 | 25.4 | 25.7 |
| LLaVA-OV | 23.5 | 46.7 | 45.1 | 274.4 | 46.1 | 48.6 | 47.0 | 26.2 | 26.6 | 27.0 |
| Qwen2-VL | 24.1 | 46.8 | 47.1 | 161.1 | 47.1 | 49.8 | 49.2 | 24.8 | 26.5 | 26.5 |
| Qwen2.5-VL | 23.8 | 45.7 | 46.9 | 246.4 | 45.9 | 49.0 | 47.5 | 24.0 | 26.2 | 26.4 |
| Qwen3-VL | 31.0 | 60.8 | 61.7 | 218.1 | 60.9 | 63.5 | 62.2 | 31.3 | 35.6 | 35.6 |
Object Hallucination in VLMs
Object hallucination in vision-language models is the production of object claims unsupported by visual evidence.
See all 135 papers