Token-Set Choice Confounds POPE: A Systematic Audit of Yes/No Extraction in VLM Hallucination Evaluation

Kesav Kumar Jayakumar; Karthigeyan Thilak

cs.CVOct 19, 2026

Token-Set Choice Confounds POPE: A Systematic Audit of Yes/No Extraction in VLM Hallucination Evaluation

Authors: Kesav Kumar Jayakumar, Karthigeyan Thilak

Organizations: Department of Information Technology Sri Krishna College of Technology Coimbatore 641042, India · Department of Computer Science and Engineering Sri Eshwar College of Engineering Coimbatore 641202, India

Abstract

We set out to reproduce a recent hallucination-mitigation result on the POPE benchmark. The LLaVA-1.5-7B baseline we measured, 82.21 F1 on the adversarial split, sat 1.21 to 8.67 points above the adversarial baselines we planned to compare against (81.00, 79.30, and 73.54). We looked at the evaluation code and found a readout choice that moves F1 on its own. A two-token logit readout decides yes versus no by comparing two vocabulary IDs, 3582 ("yes") and 1217 ("no"). Across 9,000 greedy-decode questions spanning all three splits, neither ID is generated even once. The model emits " Yes" (token 3869) and " No" (token 1939), because SentencePiece prepends a space byte after the prompt suffix "ASSISTANT:". Read with an eight-token lookup that covers the space-prefixed surface forms, LLaVA-1.5-7B scores F1 0.8221, 0.8498, and 0.8713 on the adversarial, popular, and random splits. That 6.13-point adversarial shift exceeds the adversarial gain reported by five of the six papers we surveyed. None of them states its readout, and our reference ran under 4-bit NF4 quantization, so we cannot say how much of any reported gain a matched evaluation would keep. That comparison stays unresolved until readout and precision are matched. On the fixed baseline we run nine inference-time corrections of our own (none beat it) and dissect why the adversarial split resists logit-level intervention. In a 3,000-question probe, wrong predictions draw more image attention than correct ones (Cohen's $d=-0.46$ , $p=1.4\times10^{-24}$ ), the opposite of what grounding-based corrections assume. A four-readout audit across four further VLMs from three tokenizer families confirms the confound is protocol-level, not model-specific: porting the LLaVA-1.5 IDs verbatim drops F1 to 0.00 on the two disjoint-vocabulary models, whereas a tokenizer-derived readout holds between 0.80 and 0.92.

Figures & tables

Panel of Figure 1 — Figure 1: Decision path at the POPE answer position for LLaVA-1.5-7B. The model always emits a space-prefixed surface form ( ‘ Yes’ or ‘ No’ ). The two-token check inspects two IDs the model never produces in this context, so every comparison is between two candidates the decoder excluded. On the adversarial split that comparison reads 924 of the model’s 1,644 ‘ No’ answers (56%) as yes, pushing the yes-rate to 0.760 on a 50% positive split.

Split	‘ Yes’ (3869)	‘ No’ (1939)
Adversarial	1,356	1,644
Popular	1,303	1,697
Random	1,234	1,766
Total	3,893	5,107

Table 1: First generated token counts across all POPE splits. LLaVA-1.5-7B produces only space-prefixed forms. The two-token protocol reads the two rightmost columns, zero in every split.

Split	Protocol	F1	Precision	Recall	Yes-rate	$\Delta$
Adversarial	Two-token	0.7608	0.6307	0.9587	0.760	+0.0613
Adversarial	Eight-token	0.8221	0.8658	0.7827	0.452	+0.0613
Popular	Two-token	0.7961	0.6797	0.9607	0.707	+0.0537
Popular	Eight-token	0.8498	0.9140	0.7940	0.434	+0.0537
Random	Two-token	0.8397	0.7459	0.9607	0.644	+0.0316
Random	Eight-token	0.8713	0.9652	0.7940	0.411	+0.0316

Table 2: POPE results for LLaVA-1.5-7B under both protocols. Two-token evaluation buys recall on the adversarial split through a 0.760 yes-bias, not accurate detection; the precision row shows how much of that coverage is wrong.

Method	Rep. base.	Rep. res.	Rep. gain	Readout
Tang et al. ( Tang et al., 2025 ) (adv.)	73.54	84.35	$+10.81$	not stated
Tang et al. ( Tang et al., 2025 ) (pop.)	76.53	87.09	$+10.56$	not stated
Tang et al. ( Tang et al., 2025 ) (rand.)	81.54	89.55	$+8.01$	not stated
Seo et al. ( Seo et al., 2025 ) (adv.)	79.30	80.00	$+0.70$	not stated
Chen et al. ( Chen et al., 2025 ) (adv.)	81.00	81.80	$+0.80$	not stated
Huo et al. ( Huo et al., 2024 ) (adv.)	80.90	82.50	$+1.60$	not stated

Table 3: Baselines, results, and gains as each paper reports them for LLaVA-1.5-7B on POPE (F1

\times 100

). The “Readout” column records what each paper states. Our 82.21 reference ran under 4-bit NF4 quantization, so a comparison with published baselines measured at higher precision would mix readout effects with precision differences.

Model (tokenizer)	Split	legacy_2tok	legacy_8tok	dynamic_single	string_parse
LLaVA-1.5-7B (LLaMA-2)	adversarial	0.7608	0.8221	0.8221	0.8221
	popular	0.7961	0.8498	0.8498	0.8498
	random	0.8397	0.8713	0.8713	0.8713
LLaVA-1.6-Mistral-7B (Mistral)	adversarial	0.6667	0.0000	0.8521	0.8521
	popular	0.6667	0.0000	0.8953	0.8953
	random	0.6667	0.0000	0.9163	0.9163

Table 4: POPE F1 under the four readouts for four VLMs beyond the LLaVA-1.5-7B reference (top block, from Table 2 ). legacy_2tok and legacy_8tok hard-code the LLaVA-1.5 LLaMA-2 IDs; dynamic_single re-derives the single-token yes/no IDs from each model’s own tokenizer; string_parse greedily decodes and parses the first yes / no substring. Bold marks the F1 a careful practitioner would report.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Answer	Token ID	Decoded surface form
yes	3582	yes
yes	8241	Yes
yes	4874	‘ yes’
yes	3869 $\star$	‘ Yes’
no	1217	no
no	3782	No

Appendix

Table 5: Yes/no token IDs for LLaMA-family models.

\star

: the only IDs LLaVA-1.5-7B generated after the ASSISTANT: prompt in our runs.

Figure 2: Confusion matrices for POPE adversarial (3,000 questions) under each protocol. Under two-token evaluation, 660 of the 1,318 true-No predictions migrate to the FP cell, leaving precision at 0.6307. Under eight-token evaluation only 182 stay there, and precision is 0.8658. The model’s behaviour is identical across both matrices; only the readout changes.

Figure 3: Running yes-rate over the 3,000-question POPE adversarial stream. The eight-token protocol (blue) stabilises near the ground-truth yes-rate (dashed). The two-token protocol (red) stabilises around 0.760 because it reads 924 of the model’s 1,644 ‘ No’ emissions (56%) as yes.

Figure 4: Running yes-rate across all three POPE splits under both protocols. The two-token protocol (left) holds a yes-bias on every split, settling at 0.760, 0.707 and 0.644 on adversarial, popular and random; the eight-token protocol (right) tracks the underlying ground-truth mix.

Figure 5: First-token frequency at the answer position for all eight yes/no surface forms across the three POPE splits (log scale on

y

). The four right-side forms, including yes (3582) and no (1217), receive zero first-token mass in any split. The two-token protocol reads only 3582 and 1217.

Panel of Figure 6 — Figure 6: Eight POPE adversarial samples from our run (question IDs 1, 7, 26, 72 on top; 14, 20, 35, 39 on the bottom), two per confusion-matrix outcome under eight-token evaluation. Blue borders mark predictions that match the label; red borders mark errors. The split queries objects that co-occur with present ones, which is what makes the FP and FN cases hard to fix with a generic visual-uncertainty signal.

Symbol	Meaning
$P,\,N$	ground-truth positive / negative question counts in the split
$T$	true-yes greedy generations (token 3869 produced when label $=$ yes)
$G$	true-negative count: token 1939 produced when label $=$ no (the $\mathrm{TN}$ cell under the eight-token rule)
$G_{\text{tot}}$	total token-1939 emissions across the split, $G_{\text{tot}}=G+(P-T)$ , because the model emits ‘ No’ on true-negative and false-negative questions alike
$\rho_{N}$	fraction of true-negative ‘ No’ emissions the two-token rule flips to yes, counted per split
$\varphi$	fraction of false-negative ‘ No’ emissions (label $=$ yes) the two-token rule reads as yes, counted per split

Appendix

Table 6: Notation used in the derivation.

Split	Label	8y $\to$ 2y	8n $\to$ 2y	8n $\to$ 2n	Rate	Value
Adversarial	yes	1,174	264	62	$\varphi$	$264/326=0.810$
Adversarial	no	182	660	658	$\rho_{N}$	$660/1{,}318=0.501$
Popular	yes	1,191	250	59	$\varphi$	$250/309=0.809$
Popular	no	112	567	821	$\rho_{N}$	$567/1{,}388=0.409$
Random	yes	1,191	250	59	$\varphi$	$250/309=0.809$
Random	no	43	448	1,009	$\rho_{N}$	$448/1{,}457=0.307$

Appendix

Table 7: Paired readout transitions for LLaVA-1.5-7B on each POPE split (3,000 questions each,

P{=}N{=}1{,}500

). Each column gives the eight-token decision followed by the two-token decision. The 8y

\to

2n column is zero everywhere, so the two-token rule only changes ‘ No’ emissions. Both rates are read from the 8n

\to

2y column.

Method	F1	Precision	Recall
Eight-token baseline	0.8221	0.8658	0.7827
CLIP-B/32 certainty ( $\beta=1.0$ )	0.8171	0.9072	0.7433
CLIP-B/32 certainty ( $\beta=1.5$ )	0.8104	0.9278	0.7193
CLIP-L/14-336 certainty	0.8159	0.9031	0.7453
Entropy gating	0.8164	0.9023	0.7453
Magnitude gating	0.8148	0.8981	0.7467

Appendix

Table 8: Nine inference-time corrections on POPE adversarial (3,000 questions, eight-token rule). The closest, CLIP-B/32

\beta{=}1.0

, sits 0.005 F1 below baseline.

Panel of Figure 7 — Figure 7: Nine inference-time corrections on POPE adversarial, two views. (a) F1 against the corrected baseline at 0.8221. (b) The same methods on the precision-recall plane with F1 iso-curves. Every yes-suppression variant moves north and west of the baseline; no path crosses the $\mathrm{F1}{=}0.8221$ iso-curve.

Figure 8: Distribution of mean attention from the final text token to image positions, layers 14–20, on the 3,000-question adversarial run. Blue bars are correct (

n{=}2490

); red bars are wrong (

n{=}510

). Higher image attention associates with wrong predictions, so suppressing yes-logits when attention is low removes signal from the wrong side of the distribution.

Figure 9: Absolute logit gap

|\ell_{\text{yes}}-\ell_{\text{no}}|

at the answer position on the 3,000-question adversarial run. Left: per-category mean with one-standard-deviation error bars (TP

n{=}1173

, TN

n{=}1317

, FP

n{=}183

, FN

n{=}327

; means 2.07, 1.48, 0.70, 0.96). Right: kernel-density estimates. The order TP

>

>

>

FP at the mean separates confidence from correctness, which is why uncertainty-keyed corrections trade TPs for FNs at about the same rate as FPs for TNs.

Signal	Group	$n$	Mean	SD
Attention	correct	2,490	0.0738	0.0142
Attention	wrong	510	0.0801	0.0128
Logit gap	TP	1,173	2.066	0.959
Logit gap	TN	1,317	1.482	0.670
Logit gap	FP	183	0.700	0.621
Logit gap	FN	327	0.962	0.681

Appendix

Table 9: Per-group means and standard deviations underlying the effect sizes. SD is the sample standard deviation.

Figure 10: Yes/no surface forms map to different token IDs under each model. Blue rows are the forms the model actually generates first after ASSISTANT: ; grey rows are valid single-token IDs that never appear; orange multi-token rows are space-prefixed forms the Mistral tokenizer splits into two tokens, so they cannot enter a first-token comparison.

Form	Tokenization	In dynamic set?
yes	[5081]	yes (single token)
Yes	[5592]	yes (single token)
‘ yes’	[28705, 5081]	no (multi-token)
‘ Yes’	[28705, 5592]	no (multi-token)
no	[708]	yes (single token)
No	[1770]	yes (single token)

Appendix

Table 10: Tokenizer-derived single-token IDs for yes/no surface forms under llava-hf/llava-v1.6-mistral-7b-hf . The four space-prefixed forms tokenize to two tokens each ( [28705, ID] ) and are excluded from the dynamic single-token readout.

Prompt template	Readout	F1	Precision	Recall	Yes-rate
Paper template	legacy_2tok	0.6667	0.5000	1.0000	1.000
	legacy_8tok	0.0000	0.0000	0.0000	0.000
	dynamic_single	0.8719	0.8352	0.9120	0.546
	string_parse	0.8719	0.8352	0.9120	0.546
Native chat template	legacy_2tok	0.6925	0.5297	1.0000	0.944
	legacy_8tok	0.0000	0.0000	0.0000	0.000

Appendix

Table 11: Four readouts on the first 500 POPE adversarial questions under LLaVA-1.6-Mistral-7B. dynamic_single compares max-logits over the tokenizer-derived IDs

\{5081,5592\}

\{708,1770\}

; string_parse decodes up to 6 tokens and parses the first yes / no substring.

Paper	Rep. base.	Rep. res.	Rep. gain	Readout
Tang et al. [ Tang et al., 2025 ] (adv.)	73.54	84.35	$+10.81$	not stated
Tang et al. [ Tang et al., 2025 ] (pop.)	76.53	87.09	$+10.56$	not stated
Tang et al. [ Tang et al., 2025 ] (rand.)	81.54	89.55	$+8.01$	not stated
Seo et al. [ Seo et al., 2025 ] (adv.)	79.30	80.00	$+0.70$	not stated
Chen et al. [ Chen et al., 2025 ] (adv.)	81.00	81.80	$+0.80$	not stated
Huo et al. [ Huo et al., 2024 ] (adv.)	80.90	82.50	$+1.60$	not stated

Appendix

Table 12: Papers reporting LLaVA-1.5-7B POPE adversarial baselines, verified entries as of submission. Values are as reported by each paper (F1

\times 100

Split	TP	TN	FP	FN
Adversarial	1,174	1,318	182	326
Popular	1,191	1,388	112	309
Random	1,191	1,457	43	309

Appendix

Table 13: Confusion matrices under eight-token evaluation, full 3,000-question splits. Counts are exact, taken from the per-question prediction JSON files; row sums equal 3,000 per split.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 989 papers

AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokens
Dec 19, 2025Tung-Ling Li, Yuhao Wu, Hongliang LiuLLM-as-a-Judge Adversarial Attacks on LLMs
Palo Alto Networks
Does Playing it Safe Count as Faithfulness? Reassessing LVLM Hallucination Mitigation Methods
Sep 1, 2026Mehrdad Fazli, Sina Mansouri, Mohit Marvania +1VLM Evaluation Faithfulness in VLMs
Department of Computer Science, George Mason University Fairfax, VA 22030, USA
On the Tip of the Tongue: Why LLMs Hallucinate Answers They Can Decode
Mar 14, 2026Valeria Ruscio, Keiran ThompsonHallucination in Language Models LLM Interpretability
Intuition Machines Inc.

Dec 19, 2025J/K move · Enter open · S save

AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokens

Tung-Ling Li, Yuhao Wu, Hongliang Liu

Palo Alto Networks

LLM-as-a-Judge Adversarial Attacks on LLMs Adversarial Robustness of Language Models Reward Modeling

Open paper AskPDF arXiv

LLM-as-a-Judge systems supply the reward signal in modern RLHF and RLVR pipelines, but their binary verdict reduces to a single linear readout F_gap on one hidden state. We show this readout is shallow enough that short, low-perplexity tokens flip the verdict from "No" to "Yes". These tokens are sampled from the judge's own next-token distribution at the response position, with no manual seed set and no gradient-based optimization. Our procedure, AdvJudge-Zero, reaches

>

90% ensemble false-positive rate on 22 of 24 (model, dataset) cells across six Qwen, Llama, and Gemma judges, versus 54-72% for the prior curated 10-token benchmark, and the discovered surface transfers cross-format to a 70B scalar reward model. The same discovered pool enables a defense: a LoRA fine-tune stratified by a 9-class mechanism taxonomy hardens cross-family generalization where naive sampling on the same pool fails, with mechanism breadth rather than pool size carrying the gain. Under GRPO training, the hardened judge eliminates the reward-collapse failures (false-positive spikes and length collapse) we observe in the unhardened baseline on both MATH and GSM8K at ten seeds per condition. The discovered pool, the mechanism taxonomy, and per-prompt flip records will be released under responsible disclosure.

Abstract

VLM Evaluation

Explore similar work

AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokens

Does Playing it Safe Count as Faithfulness? Reassessing LVLM Hallucination Mitigation Methods

On the Tip of the Tongue: Why LLMs Hallucinate Answers They Can Decode