Say Less, Hallucinate Less? Diagnosing and Restoring Object Coverage in Preference-Optimized Vision-Language Models
Organizations: Department of Computer Science, Seoul Women’s University, Seoul, Republic of Korea
Abstract
Preference optimization (PO) is the standard way to reduce object hallucination in large vision-language models. We find that four public PO checkpoints of LLaVA-1.5-7B also lower object coverage. They name fewer object categories, and their lower hallucination rates bring no significant gain in set-level object F1 on CHAIR. We therefore ask how much of the PO gain the base model can reproduce by stopping earlier. A positive end-of-sequence (EOS) logit bias lets the base model reach the recall of each PO model, and this early stopping alone reproduces 58 to 86 percent of the CHAIRi gain of three checkpoints on two disjoint COCO sets. Writing longer does not bring the lost categories back. Under greedy decoding, a negative EOS bias recovers only 2 to 9 percent of the lost CHAIR recall of these checkpoints, and 94 to 96 percent of the added object mentions repeat categories already named. On CHAIR, the base caption of the same image still names about half of the missed object categories. We therefore propose base-guided object-category restoration, which appends high-confidence base-only categories to the PO caption without training. With thresholds fixed on a development set, it brings three checkpoints within one point of the base recall on a held-out COCO set with a lower distinct-object hallucination rate than the base model. Against the recall-matched base stopping curve, its gain holds for RLAIF-V near the base recall and for RLAIF-V and CHAIR-DPO at an intermediate recall. We will release the code of the recall-controlled evaluation.
Figures & tables
| CHAIR-500 | AMBER | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | CHAIRi | Recall | F1 | Words | Obj. | CHAIR | Cover | F1 proxy | Obj. | ||
| LLaVA-1.5-7B | 14.39 | 26.4 | 80.5 | 76.9 | 89.7 | 3.2 | 7.50 | 12.0 | 51.8 | 65.2 | 4.0 |
| RLAIF-V | 3.39 | 7.4 | 68.4 | 78.7 | 93.4 | 2.2 | 3.04 | 5.1 | 50.0 | 65.5 | 3.5 |
| CHAIR-DPO | 3.90 | 7.3 | 65.1 | 76.5 | 77.6 | 2.1 | 3.12 | 5.1 | 46.6 | 62.5 | 3.3 |
| CSR | 5.49 | 11.4 | 66.0 | 75.7 | 75.1 | 2.2 | 4.41 | 6.7 | 45.1 | 60.8 | 3.3 |
| HA-DPO | 12.07 | 21.5 | 76.2 | 77.3 | 83.5 | 2.8 | 6.61 | 10.5 | 49.8 | 64.0 | 3.8 |
| CHAIR-500 (CHAIRi) | AMBER (CHAIR) | |||||||
| (%) | (%) | |||||||
| Model | [95% CI] | recall | length | [95% CI] | recall | length | ||
| RLAIF-V | 11.00 | 4.26 [2.62, 6.08] | 61 [48, 74] | 0 ∗ | 4.46 | 3.58 [2.57, 4.57] | 20 [6, 37] | 0 ∗ |
| CHAIR-DPO | 10.49 | 2.39 [0.97, 4.13] | 77 [63, 90] | 14 | 4.39 | 1.56 [0.92, 2.30] | 64 [50, 77] | 12 |
| CSR | 8.90 | 1.28 [ 0.03, 2.75] | 86 [72, 100] | 22 | 3.09 | 0.10 [ 0.70, 0.55] | 103 [84, 127] | 19 |
| HA-DPO † | 2.32 | 1.49 [ 4.40, 1.30] | 164 [ 432, 897] | 24 | 0.89 | 0.14 [ 1.04, 0.68] | 115 [45, 469] | 11 |
| Recall | Recovered | Categories | New | |
|---|---|---|---|---|
| Model | drop | (%) | per caption | (%) |
| RLAIF-V | 12.3 | 2 / 17 | 2.16 2.17 | 6 [2, 11] |
| CHAIR-DPO | 15.2 | 9 / 24 | 2.06 2.13 | 4 [3, 5] |
| CSR | 14.7 | 4 / 8 | 2.17 2.21 | 4 [2, 5] |
| HA-DPO | 4.5 | 58 / 76 | 2.83 3.04 | 8 [6, 11] |
| Held-out COCO 500, fixed thresholds | Development | |||||||
|---|---|---|---|---|---|---|---|---|
| Model | Recall | CHAIRs | vs. base | Mid vs. curve | Vs. random | CHAIR | AMBER | |
| LLaVA-1.5-7B | 80.1 | 26.9 | 55.2 | |||||
| RLAIF-V | 66.4 79.2 | 8.4 21.6 | 16.6 46.4 | 5.24 [3.92, 6.60] | 5.42 [2.74, 8.05] | 3.77 [2.66, 4.92] | 5.87 [2.93, 8.83] | 5.66 [4.48, 6.88] |
| CHAIR-DPO | 64.6 79.5 | 7.6 25.0 | 15.2 52.4 | 1.85 [0.98, 2.77] | 4.36 [2.12, 6.87] | 1.59 [0.89, 2.37] | 1.97 [0.54, 4.35] | 2.95 [2.05, 3.88] |
| CSR | 64.7 79.7 | 12.2 25.8 | 23.0 54.6 | 1.08 [0.15, 2.05] | 1.91 [ 0.25, 4.51] | 1.59 [0.87, 2.31] | 1.42 [ 0.29, 3.75] | 1.44 [0.60, 2.35] |
| HA-DPO | 76.3 79.5 | 20.9 23.9 | 40.2 48.6 | 2.94 [1.46, 4.47] | 1.54 [ 1.42, 4.25] | 3.09 [2.24, 3.93] | 2.26 [ 0.53, 3.96] | 0.58 [ 0.41, 1.80] |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Repository | Revision |
|---|---|---|
| RLAIF-V | openbmb/RLAIF-V-7B | 90e8e73 |
| CHAIR-DPO | aimagelab/CHAIR-DPO_LLaVA-1.5-7B | bd9ab77 |
| CSR | charlesdj/CSR_LLaVA_1.5_7b_2Iteration | 7ebdd0b |
| HA-DPO | juliozhao/hadpo-llava-1.5 | 72b1bcf |
| COCO id | Position | Diverging tokens | Original | HF |
|---|---|---|---|---|
| 7276 | 105 | en / giving | 0.016 | 0.016 |
| 8204 | 14 | building / tall | 0.031 | 0.024 |
| 8953 | 84 | interaction / view | 0.055 | 0.001 |
| 11374 | 32 | with / appearing | 0.031 | 0.006 |
| 13004 | 64 | presentation / plate | 0.016 | 0.014 |
| 13783 | 22 | seems / appears | 0.094 | 0.014 |
| RLAIF-V on CHAIR-500 | HF, batch 32 | Original, batch 1 |
|---|---|---|
| CHAIRi | 3.39 | 3.42 |
| Recall | 68.42 | 67.95 |
| 7.41 | 7.55 | |
| Set-level F1 | 78.69 | 78.33 |
| Words | 93.37 | 93.49 |
| [95% CI] | 4.25 [2.62, 6.07] | 4.15 [2.45, 5.76] |
| CHAIR-500 | AMBER | |||||
|---|---|---|---|---|---|---|
| Policy | RLAIF-V | C-DPO | CSR | RLAIF-V | C-DPO | CSR |
| EOS bias (main) | 61 | 77 | 86 | 20 | 64 | 103 |
| EOS mixture | 61 | 77 | 86 | 20 | 65 | 103 |
| Short-answer prompt | 27 | 28 | 33 | 31 | 31 | 44 |
| Oracle words | 6 | 19 | 28 | 7 | 20 | 31 |
| Oracle objects | 52 | 52 | 57 | 28 | 37 | 52 |
| Model | Recall | CHAIRi | CHAIRs | Objects | Words |
|---|---|---|---|---|---|
| RLAIF-V | 68.4 / 67.9 | 3.4 / 7.7 | 14.8 / 22.4 | 5.5 / 4.6 | 93.4 / 51.8 |
| CHAIR-DPO | 65.1 / 65.0 | 3.9 / 6.3 | 14.0 / 18.6 | 5.9 / 4.2 | 77.6 / 47.6 |
| CSR | 66.0 / 66.2 | 5.5 / 6.8 | 21.0 / 20.0 | 6.2 / 4.3 | 75.1 / 49.2 |
| HA-DPO | 76.2 / 76.4 | 12.1 / 10.6 | 38.6 / 34.6 | 7.2 / 5.8 | 83.5 / 66.5 |
| Model | Sel. true [95% CI] | Sel. swap | Dependence [95% CI] | ||
|---|---|---|---|---|---|
| RLAIF-V | 4.21 [3.64, 4.89] | 0.85 | 3.36 [2.71, 3.98] | 2.70 | 6.92 |
| CHAIR-DPO | 0.52 [0.46, 0.58] | 0.52 | 0.00 [ 0.08, 0.07] | 0.03 | 0.55 |
| CSR | 0.32 [0.27, 0.37] | 0.51 | 0.20 [ 0.28, 0.12] | 0.05 | 0.27 |
| HA-DPO | 0.08 [0.06, 0.11] | 0.15 | 0.06 [ 0.10, 0.03] | 0.03 | 0.11 |
| POVID | 0.42 [0.33, 0.53] | 1.04 | 0.62 [ 1.13, 0.14] | 0.12 | 0.30 |
| POPE accuracy | Yes ratio (%) | PO vs. yes-matched base | MMStar | |||||
|---|---|---|---|---|---|---|---|---|
| Model | Split | Base | PO | Base | PO | Acc [95% CI] | Agree (%) | Acc [95% CI] |
| RLAIF-V | r | 88.4 | 82.8 | 40.6 | 33.5 | 0.12 [ 0.53, 0.24] | 96.9 | 0.73 [ 1.20, 2.73] |
| p | 87.2 | 82.3 | 41.8 | 34.0 | 0.20 [ 0.59, 0.25] | 96.8 | ||
| a | 85.1 | 81.7 | 43.9 | 34.6 | 0.17 [ 0.65, 0.42] | 96.5 | ||
| CHAIR-DPO | r | 88.4 | 87.7 | 40.6 | 39.4 | 0.11 [ 0.21, 0.53] | 98.8 | 0.20 [ 1.07, 0.67] |
| p | 87.2 | 86.8 | 41.8 | 40.3 | 0.34 [0.01, 0.71] | 98.6 | ||
| CHAIR-500 | AMBER | |||
|---|---|---|---|---|
| Pair | ||||
| RLAIF-V CHAIR-DPO | 0.51 [ 0.58, 1.60] | 1.87 [0.27, 3.36] | 0.08 [ 0.48, 0.61] | 2.02 [0.99, 2.92] |
| RLAIF-V CSR | 2.09 [0.93, 3.28] | 2.98 [1.37, 4.61] | 1.37 [0.72, 2.06] | 3.68 [2.60, 4.70] |
| CHAIR-DPO CSR | 1.59 [0.68, 2.52] | 1.11 [0.02, 2.34] | 1.29 [0.75, 1.82] | 1.66 [1.07, 2.32] |
| RLAIF-V HA-DPO | 8.68 [6.13, 12.11] | 5.75 [2.86, 8.97] | 3.57 [2.82, 4.32] | 3.72 [2.70, 4.82] |
| CHAIR-DPO HA-DPO | 8.17 [5.64, 11.47] | 3.88 [1.07, 7.26] | 3.50 [2.89, 4.11] | 1.70 [0.87, 2.66] |
| Model | Recall drop | Largest recall ( ) | Recovered CHAIR / AMBER | Categories per caption | New categories | Words |
|---|---|---|---|---|---|---|
| RLAIF-V | 12.3 | 68.6 (10) | 2% / 17% | 2.16 2.17 | 6 of 93 (6% [2, 11]) | 93 98 |
| CHAIR-DPO | 15.2 | 66.7 (7.65) | 9% / 24% | 2.06 2.13 | 38 of 989 (4% [3, 5]) | 78 117 |
| CSR | 14.7 | 66.4 (10) | 4% / 8% | 2.17 2.21 | 19 of 529 (4% [2, 5]) | 75 90 |
| HA-DPO | 4.5 | 78.8 (7.0) | 58% / 76% | 2.83 3.04 | 101 of 1,224 (8% [6, 11]) | 84 123 |
| CHAIR-500 recall | AMBER Cover | |||||||
|---|---|---|---|---|---|---|---|---|
| Model | Unreach. | Pooled gap [95% CI] | Fold range | Limit (%) | Unreach. | Pooled gap [95% CI] | Fold range | Limit (%) |
| RLAIF-V | 10 | 11.99 [ 13.95, 9.88] | 13.56 to 10.35 | 0 | 7 | 1.34 [ 2.37, 0.42] | 2.11 to 0.57 | 0 |
| CHAIR-DPO | 10 | 11.30 [ 13.12, 9.31] | 12.70 to 9.80 | 85 | 9 | 1.84 [ 2.67, 1.09] | 2.85 to 0.82 | 83 |
| CSR | 10 | 14.18 [ 16.09, 12.21] | 15.94 to 12.40 | 0 | 10 | 5.98 [ 6.80, 5.16] | 6.77 to 5.20 | 0 |
| HA-DPO | 0 | 0.37 [ 1.20, 0.29] | 2.82 to 1.72 | 61 | 0 | 0.04 [ 0.12, 0.20] | 1.22 to 1.08 | 17 |
| Base model | Batch 32 | Batch 16 |
|---|---|---|
| CHAIR CHAIRi / Recall | 14.39 / 80.5 | 14.42 / 80.6 |
| CHAIR F1 / CHAIRs | 76.9 / 47.2 | 76.9 / 48.4 |
| AMBER CHAIR / Cover | 7.50 / 51.8 | 7.53 / 51.7 |
| RLAIF-V | 4.26 [2.62, 6.08] | 4.39 [2.66, 6.04] |
| CHAIR-DPO | 2.39 [0.97, 4.13] | 2.54 [1.22, 4.19] |
| CSR | 1.28 [ 0.03, 2.75] | 1.42 [0.16, 2.82] |
| RLAIF-V | CHAIR-DPO | CSR | HA-DPO | |
| Restoration recall / | 79.16 / 21.64 | 79.55 / 25.03 | 79.74 / 25.79 | 79.49 / 23.94 |
| Random recall / | 79.43 / 25.41 | 79.83 / 26.62 | 80.10 / 27.38 | 80.24 / 27.03 |
| Restoration minus random, recall | 0.26 [ 1.12, 0.58] | 0.28 [ 0.90, 0.29] | 0.35 [ 0.99, 0.22] | 0.76 [ 1.44, 0.08] |
| Random minus restoration, | 3.77 [2.66, 4.92] | 1.59 [0.89, 2.37] | 1.59 [0.87, 2.31] | 3.09 [2.24, 3.93] |
| Per-seed range | 3.13 to 4.09 | 1.23 to 1.91 | 0.92 to 2.09 | 2.55 to 3.81 |
| Count-matched random recall | 77.27 | 78.84 | 78.78 | 78.69 |
| Model | Threshold | Restored recall | Curve recall | Restored | Curve | Gap | Conditional CI | Recalibrated CI |
|---|---|---|---|---|---|---|---|---|
| RLAIF-V | 79.16 | 79.20 | 21.64 | 24.75 | 3.10 | [1.73, 4.56] | [0.31, 5.21] | |
| Midpoint | 72.28 | 72.31 | 12.12 | 17.54 | 5.42 | [3.38, 7.53] | [2.74, 8.05] | |
| CHAIR-DPO | 79.55 | 79.50 | 25.03 | 25.21 | 0.18 | [ 0.81, 1.24] | [ 1.68, 1.58] | |
| Midpoint | 70.55 | 70.42 | 12.03 | 16.39 | 4.36 | [2.44, 6.28] | [2.12, 6.87] | |
| CSR | 79.74 | 79.58 | 25.79 | 25.51 | 0.28 | [ 1.23, 0.75] | [ 2.38, 1.10] | |
| Midpoint | 71.32 | 71.15 | 14.89 | 16.80 | 1.91 | [0.12, 3.67] | [ 0.25, 4.51] |
| Model | Recall | CHAIRi | F1 | Appended | Precision (%) | vs. base | Gap to curve (recalibrated) | ||
|---|---|---|---|---|---|---|---|---|---|
| RLAIF-V | 0.50 | 81.48 | 26.34 | 14.00 | 77.4 | 594 | 39.6 | 0.54 [ 0.30, 1.44] | recall above base |
| 0.25 | 80.71 | 24.26 | 12.72 | 78.1 | 531 | 42.0 | 2.62 [1.61, 3.63] | recall above base | |
| 0.10 | 79.94 | 22.65 | 11.74 | 78.6 | 480 | 44.0 | 4.23 [3.06, 5.43] | 2.93 [0.68, 4.77] | |
| 0.00 | 79.16 | 21.64 | 11.12 | 78.8 | 444 | 44.8 | 5.24 [3.92, 6.60] | 3.10 [0.31, 5.21] | |
| +0.10 | 78.65 | 20.79 | 10.62 | 78.9 | 417 | 45.8 | 6.09 [4.79, 7.54] | 2.84 [0.30, 5.65] | |
| +0.25 | 77.30 | 18.95 | 9.55 | 79.1 | 356 | 47.8 | 7.93 [6.50, 9.57] | 2.51 [0.22, 6.05] |
| Model | CHAIRi | Recall | [95% CI] | |||
|---|---|---|---|---|---|---|
| LLaVA-1.5-7B | 15.91 | 80.13 | 26.88 | |||
| RLAIF-V | 4.05 | 66.37 | 8.43 | 11.86 | 4.99 [3.53, 6.43] | 58% |
| CHAIR-DPO | 4.07 | 64.57 | 7.64 | 11.84 | 4.61 [3.10, 6.11] | 61% |
| CSR | 6.29 | 64.69 | 12.22 | 9.62 | 2.46 [1.10, 3.58] | 74% |
| HA-DPO | 11.74 | 76.27 | 20.93 | 4.18 | 1.09 [ 0.39, 3.06] | 74% |
Object Hallucination in VLMs
Object hallucination in vision-language models is the production of object claims unsupported by visual evidence.
See all 136 papers