Organizations: Department of Information Technology Sri Krishna College of Technology Coimbatore 641042, India · Department of Computer Science and Engineering Sri Eshwar College of Engineering Coimbatore 641202, India
We set out to reproduce a recent hallucination-mitigation result on the POPE benchmark. The LLaVA-1.5-7B baseline we measured, 82.21 F1 on the adversarial split, sat 1.21 to 8.67 points above the adversarial baselines we planned to compare against (81.00, 79.30, and 73.54). We looked at the evaluation code and found a readout choice that moves F1 on its own. A two-token logit readout decides yes versus no by comparing two vocabulary IDs, 3582 ("yes") and 1217 ("no"). Across 9,000 greedy-decode questions spanning all three splits, neither ID is generated even once. The model emits " Yes" (token 3869) and " No" (token 1939), because SentencePiece prepends a space byte after the prompt suffix "ASSISTANT:". Read with an eight-token lookup that covers the space-prefixed surface forms, LLaVA-1.5-7B scores F1 0.8221, 0.8498, and 0.8713 on the adversarial, popular, and random splits. That 6.13-point adversarial shift exceeds the adversarial gain reported by five of the six papers we surveyed. None of them states its readout, and our reference ran under 4-bit NF4 quantization, so we cannot say how much of any reported gain a matched evaluation would keep. That comparison stays unresolved until readout and precision are matched. On the fixed baseline we run nine inference-time corrections of our own (none beat it) and dissect why the adversarial split resists logit-level intervention. In a 3,000-question probe, wrong predictions draw more image attention than correct ones (Cohen's d=−0.46, p=1.4×10−24), the opposite of what grounding-based corrections assume. A four-readout audit across four further VLMs from three tokenizer families confirms the confound is protocol-level, not model-specific: porting the LLaVA-1.5 IDs verbatim drops F1 to 0.00 on the two disjoint-vocabulary models, whereas a tokenizer-derived readout holds between 0.80 and 0.92.
Figures & tables
Figure 1: Decision path at the POPE answer position for LLaVA-1.5-7B. The model always emits a space-prefixed surface form ( ‘ Yes’ or ‘ No’ ). The two-token check inspects two IDs the model never produces in this context, so every comparison is between two candidates the decoder excluded. On the adversarial split that comparison reads 924 of the model’s 1,644 ‘ No’ answers (56%) as yes, pushing the yes-rate to 0.760 on a 50% positive split.
Split
‘ Yes’ (3869)
‘ No’ (1939)
yes (3582)
no (1217)
Adversarial
1,356
1,644
0
0
Popular
1,303
1,697
0
0
Random
1,234
1,766
0
0
Total
3,893
5,107
0
0
Table 1: First generated token counts across all POPE splits. LLaVA-1.5-7B produces only space-prefixed forms. The two-token protocol reads the two rightmost columns, zero in every split.
Split
Protocol
F1
Precision
Recall
Yes-rate
Δ
Adversarial
Two-token
0.7608
0.6307
0.9587
0.760
+0.0613
Eight-token
0.8221
0.8658
0.7827
0.452
Popular
Two-token
0.7961
0.6797
0.9607
0.707
+0.0537
Eight-token
0.8498
0.9140
0.7940
0.434
Random
Two-token
0.8397
0.7459
0.9607
0.644
+0.0316
Eight-token
0.8713
0.9652
0.7940
0.411
Table 2: POPE results for LLaVA-1.5-7B under both protocols. Two-token evaluation buys recall on the adversarial split through a 0.760 yes-bias, not accurate detection; the precision row shows how much of that coverage is wrong.
Method
Rep. base.
Rep. res.
Rep. gain
Readout
Tang et al. ( Tang et al., 2025 ) (adv.)
73.54
84.35
+10.81
not stated
Tang et al. ( Tang et al., 2025 ) (pop.)
76.53
87.09
+10.56
not stated
Tang et al. ( Tang et al., 2025 ) (rand.)
81.54
89.55
+8.01
not stated
Seo et al. ( Seo et al., 2025 ) (adv.)
79.30
80.00
+0.70
not stated
Chen et al. ( Chen et al., 2025 ) (adv.)
81.00
81.80
+0.80
not stated
Huo et al. ( Huo et al., 2024 ) (adv.)
80.90
82.50
+1.60
not stated
Table 3: Baselines, results, and gains as each paper reports them for LLaVA-1.5-7B on POPE (F1 ×100 ). The “Readout” column records what each paper states. Our 82.21 reference ran under 4-bit NF4 quantization, so a comparison with published baselines measured at higher precision would mix readout effects with precision differences.
Model (tokenizer)
Split
legacy_2tok
legacy_8tok
dynamic_single
string_parse
LLaVA-1.5-7B (LLaMA-2)
adversarial
0.7608
0.8221
0.8221
0.8221
popular
0.7961
0.8498
0.8498
0.8498
random
0.8397
0.8713
0.8713
0.8713
LLaVA-1.6-Mistral-7B (Mistral)
adversarial
0.6667
0.0000
0.8521
0.8521
popular
0.6667
0.0000
0.8953
0.8953
random
0.6667
0.0000
0.9163
0.9163
Table 4: POPE F1 under the four readouts for four VLMs beyond the LLaVA-1.5-7B reference (top block, from Table 2 ). legacy_2tok and legacy_8tok hard-code the LLaVA-1.5 LLaMA-2 IDs; dynamic_single re-derives the single-token yes/no IDs from each model’s own tokenizer; string_parse greedily decodes and parses the first yes / no substring. Bold marks the F1 a careful practitioner would report.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Answer
Token ID
Decoded surface form
yes
3582
yes
yes
8241
Yes
yes
4874
‘ yes’
yes
3869 ⋆
‘ Yes’
no
1217
no
no
3782
No
Appendix
Table 5: Yes/no token IDs for LLaMA-family models. ⋆ : the only IDs LLaVA-1.5-7B generated after the ASSISTANT: prompt in our runs.
Figure 2: Confusion matrices for POPE adversarial (3,000 questions) under each protocol. Under two-token evaluation, 660 of the 1,318 true-No predictions migrate to the FP cell, leaving precision at 0.6307. Under eight-token evaluation only 182 stay there, and precision is 0.8658. The model’s behaviour is identical across both matrices; only the readout changes.
Figure 3: Running yes-rate over the 3,000-question POPE adversarial stream. The eight-token protocol (blue) stabilises near the ground-truth yes-rate (dashed). The two-token protocol (red) stabilises around 0.760 because it reads 924 of the model’s 1,644 ‘ No’ emissions (56%) as yes.
Figure 4: Running yes-rate across all three POPE splits under both protocols. The two-token protocol (left) holds a yes-bias on every split, settling at 0.760, 0.707 and 0.644 on adversarial, popular and random; the eight-token protocol (right) tracks the underlying ground-truth mix.
Figure 5: First-token frequency at the answer position for all eight yes/no surface forms across the three POPE splits (log scale on y ). The four right-side forms, including yes (3582) and no (1217), receive zero first-token mass in any split. The two-token protocol reads only 3582 and 1217.
Figure 6: Eight POPE adversarial samples from our run (question IDs 1, 7, 26, 72 on top; 14, 20, 35, 39 on the bottom), two per confusion-matrix outcome under eight-token evaluation. Blue borders mark predictions that match the label; red borders mark errors. The split queries objects that co-occur with present ones, which is what makes the FP and FN cases hard to fix with a generic visual-uncertainty signal.
Symbol
Meaning
P,N
ground-truth positive / negative question counts in the split
T
true-yes greedy generations (token 3869 produced when label = yes)
G
true-negative count: token 1939 produced when label = no (the TN cell under the eight-token rule)
Gtot
total token-1939 emissions across the split, Gtot=G+(P−T) , because the model emits ‘ No’ on true-negative and false-negative questions alike
ρN
fraction of true-negative ‘ No’ emissions the two-token rule flips to yes, counted per split
φ
fraction of false-negative ‘ No’ emissions (label = yes) the two-token rule reads as yes, counted per split
Appendix
Table 6: Notation used in the derivation.
Split
Label
8y → 2y
8y → 2n
8n → 2y
8n → 2n
Rate
Value
Adversarial
yes
1,174
0
264
62
φ
264/326=0.810
no
182
0
660
658
ρN
660/1,318=0.501
Popular
yes
1,191
0
250
59
φ
250/309=0.809
no
112
0
567
821
ρN
567/1,388=0.409
Random
yes
1,191
0
250
59
φ
250/309=0.809
no
43
0
448
1,009
ρN
448/1,457=0.307
Appendix
Table 7: Paired readout transitions for LLaVA-1.5-7B on each POPE split (3,000 questions each, P=N=1,500 ). Each column gives the eight-token decision followed by the two-token decision. The 8y → 2n column is zero everywhere, so the two-token rule only changes ‘ No’ emissions. Both rates are read from the 8n → 2y column.
Method
F1
Precision
Recall
Eight-token baseline
0.8221
0.8658
0.7827
CLIP-B/32 certainty ( β=1.0 )
0.8171
0.9072
0.7433
CLIP-B/32 certainty ( β=1.5 )
0.8104
0.9278
0.7193
CLIP-L/14-336 certainty
0.8159
0.9031
0.7453
Entropy gating
0.8164
0.9023
0.7453
Magnitude gating
0.8148
0.8981
0.7467
Appendix
Table 8: Nine inference-time corrections on POPE adversarial (3,000 questions, eight-token rule). The closest, CLIP-B/32 β=1.0 , sits 0.005 F1 below baseline.
Figure 7: Nine inference-time corrections on POPE adversarial, two views. (a) F1 against the corrected baseline at 0.8221. (b) The same methods on the precision-recall plane with F1 iso-curves. Every yes-suppression variant moves north and west of the baseline; no path crosses the F1=0.8221 iso-curve.
Figure 8: Distribution of mean attention from the final text token to image positions, layers 14–20, on the 3,000-question adversarial run. Blue bars are correct ( n=2490 ); red bars are wrong ( n=510 ). Higher image attention associates with wrong predictions, so suppressing yes-logits when attention is low removes signal from the wrong side of the distribution.
Figure 9: Absolute logit gap ∣ℓyes−ℓno∣ at the answer position on the 3,000-question adversarial run. Left: per-category mean with one-standard-deviation error bars (TP n=1173 , TN n=1317 , FP n=183 , FN n=327 ; means 2.07, 1.48, 0.70, 0.96). Right: kernel-density estimates. The order TP > TN > FN > FP at the mean separates confidence from correctness, which is why uncertainty-keyed corrections trade TPs for FNs at about the same rate as FPs for TNs.
Signal
Group
n
Mean
SD
Attention
correct
2,490
0.0738
0.0142
Attention
wrong
510
0.0801
0.0128
Logit gap
TP
1,173
2.066
0.959
Logit gap
TN
1,317
1.482
0.670
Logit gap
FP
183
0.700
0.621
Logit gap
FN
327
0.962
0.681
Appendix
Table 9: Per-group means and standard deviations underlying the effect sizes. SD is the sample standard deviation.
Figure 10: Yes/no surface forms map to different token IDs under each model. Blue rows are the forms the model actually generates first after ASSISTANT: ; grey rows are valid single-token IDs that never appear; orange multi-token rows are space-prefixed forms the Mistral tokenizer splits into two tokens, so they cannot enter a first-token comparison.
Form
Tokenization
In dynamic set?
yes
[5081]
yes (single token)
Yes
[5592]
yes (single token)
‘ yes’
[28705, 5081]
no (multi-token)
‘ Yes’
[28705, 5592]
no (multi-token)
no
[708]
yes (single token)
No
[1770]
yes (single token)
Appendix
Table 10: Tokenizer-derived single-token IDs for yes/no surface forms under llava-hf/llava-v1.6-mistral-7b-hf . The four space-prefixed forms tokenize to two tokens each ( [28705, ID] ) and are excluded from the dynamic single-token readout.
Prompt template
Readout
F1
Precision
Recall
Yes-rate
Paper template
legacy_2tok
0.6667
0.5000
1.0000
1.000
legacy_8tok
0.0000
0.0000
0.0000
0.000
dynamic_single
0.8719
0.8352
0.9120
0.546
string_parse
0.8719
0.8352
0.9120
0.546
Native chat template
legacy_2tok
0.6925
0.5297
1.0000
0.944
legacy_8tok
0.0000
0.0000
0.0000
0.000
Appendix
Table 11: Four readouts on the first 500 POPE adversarial questions under LLaVA-1.6-Mistral-7B. dynamic_single compares max-logits over the tokenizer-derived IDs {5081,5592} vs {708,1770} ; string_parse decodes up to 6 tokens and parses the first yes / no substring.
Paper
Rep. base.
Rep. res.
Rep. gain
Readout
Tang et al. [ Tang et al., 2025 ] (adv.)
73.54
84.35
+10.81
not stated
Tang et al. [ Tang et al., 2025 ] (pop.)
76.53
87.09
+10.56
not stated
Tang et al. [ Tang et al., 2025 ] (rand.)
81.54
89.55
+8.01
not stated
Seo et al. [ Seo et al., 2025 ] (adv.)
79.30
80.00
+0.70
not stated
Chen et al. [ Chen et al., 2025 ] (adv.)
81.00
81.80
+0.80
not stated
Huo et al. [ Huo et al., 2024 ] (adv.)
80.90
82.50
+1.60
not stated
Appendix
Table 12: Papers reporting LLaVA-1.5-7B POPE adversarial baselines, verified entries as of submission. Values are as reported by each paper (F1 ×100 ).
Split
TP
TN
FP
FN
Adversarial
1,174
1,318
182
326
Popular
1,191
1,388
112
309
Random
1,191
1,457
43
309
Appendix
Table 13: Confusion matrices under eight-token evaluation, full 3,000-question splits. Counts are exact, taken from the per-question prediction JSON files; row sums equal 3,000 per split.
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
LLM-as-a-Judge systems supply the reward signal in modern RLHF and RLVR pipelines, but their binary verdict reduces to a single linear readout F_gap on one hidden state. We show this readout is shallow enough that short, low-perplexity tokens flip the verdict from "No" to "Yes". These tokens are sampled from the judge's own next-token distribution at the response position, with no manual seed set and no gradient-based optimization. Our procedure, AdvJudge-Zero, reaches >90% ensemble false-positive rate on 22 of 24 (model, dataset) cells across six Qwen, Llama, and Gemma judges, versus 54-72% for the prior curated 10-token benchmark, and the discovered surface transfers cross-format to a 70B scalar reward model. The same discovered pool enables a defense: a LoRA fine-tune stratified by a 9-class mechanism taxonomy hardens cross-family generalization where naive sampling on the same pool fails, with mechanism breadth rather than pool size carrying the gain. Under GRPO training, the hardened judge eliminates the reward-collapse failures (false-positive spikes and length collapse) we observe in the unhardened baseline on both MATH and GSM8K at ten seeds per condition. The discovered pool, the mechanism taxonomy, and per-prompt flip records will be released under responsible disclosure.
Recent inference-time hallucination mitigation methods for large vision-language models (LVLMs) report strong gains on hallucination benchmarks. However, it remains unclear whether lower hallucination scores reflect improved multimodal grounding or more conservative generation. We evaluate six mitigation methods across three LVLMs and four benchmarks, including hallucination-focused evaluation and the diverse capability benchmark MMStar. Our analysis reveals two consistent patterns. First, hallucination reduction is often coupled with reduced informativeness: methods that lower hallucination rates also reduce object recall, visual coverage, or response detailedness. Second, improvements on hallucination benchmarks do not reliably transfer to broader multimodal capabilities, with methods showing inconsistent or degraded performance on fine-grained perception and reasoning tasks. Our findings suggest that current evaluation protocols may overestimate progress by rewarding conservative generation. We argue that hallucination mitigation should be evaluated as a faithfulness--informativeness--capability trade-off rather than through hallucination scores alone.
Mehrdad Fazli, Sina Mansouri, Mohit Marvania +1
Department of Computer Science, George Mason University Fairfax, VA 22030, USA
A language model can give the wrong answer even when the correct answer is decodable from its intermediate states. To study this gap between decodability and selection, we distinguish read from write at the first answer token. Read asks whether the gold token can be decoded from intermediate residual states under same-relation decoy controls. Write asks whether the final readout ranks that token first among content tokens. Under three different readers, with a randomized-label control, a substantial fraction of failures remain readable while another content token is selected. We explain this through the selection margin at the final readout, the difference between the answer logit and the logit of its strongest alternative, which is answer support minus alternative support, and can also be split into a context-averaged baseline linked to token frequency and an item-specific term. Setting the answer support to the level typical of successful generations is sufficient to recover first-token selection for the majority of failures in most of the models we study; the original alternative remains ahead in most remaining failures under this edit, and this outcome follows directly from the readout geometry. Removing the frequency direction alone shifts selection but rarely recovers the answer. Prompt variants of the same fact that succeed supply support that transfers to failing variants through the residual stream and through late MLP outputs, with less consistent effects through late attention. First-token recovery leaves most full answers wrong, which limits the recovery achieved by these edits and separates three things that are easily conflated, decodability, recoverability, and generation.