Not Every Mention Is a Claim: Rethinking Object Hallucination Detection in Vision-Language Models

Hsin-Ling Hsu; Neeraja Kirtane; Kuan-Hao Huang

cs.CVOct 16, 2026

Not Every Mention Is a Claim: Rethinking Object Hallucination Detection in Vision-Language Models

Authors: Hsin-Ling Hsu, Neeraja Kirtane, Kuan-Hao Huang

Organizations: National Chengchi University · Texas A&M University

Abstract

Object hallucination in vision-language models (VLMs) is usually measured by checking each object a caption names against the image. Most benchmarks and detectors alike treat every object mention as a claim that the object is present. This assumption fails when a caption names an object without asserting it, as in "a bench for people to sit on", which mentions people without claiming that any are in the image. We call such mentions non-assertive. Treating them as hallucinations inflates the error rates attributed to models that use such language. To separate them from real hallucinations, we introduce NAME-Bench (Non-Assertive Mention Evaluation Benchmark), 25,497 object mentions from captions that five VLMs generate for COCO images. Each mention is labelled concrete (asserted and present), hallucinated (asserted and absent), or non-assertive, and non-assertive mentions are further divided into four subtypes: depictive, purpose, negation and hedged. All the hallucination detectors that we evaluate flag non-assertive mentions as hallucinations, at rates between 14% and 80%. Adding non-assertive mentions to detector training as negatives largely corrects this on LLaVA-1.5. On the four newer VLMs, a comparable correction takes five to ten times more data, and the detectors then catch fewer real hallucinations. These findings show that current detectors conflate mentioning an object with asserting it, and that this conflation is harder to undo on the four newer VLMs than on LLaVA-1.5. Hallucination evaluation should therefore separate what a caption asserts from what it merely mentions.

Figures & tables

Panel of Figure 1 — Figure 1: Hallucination detectors flag non-assertive mentions. Six illustrative held-out examples from LLaVA-1.5: concrete, hallucinated, and four non-assertive subtypes (hedged, depictive, purpose, and negation). Bars show Lookback Lens scores before non-assertive training; circles show scores after adding 20 non-assertive examples. The dashed line marks the threshold calibrated at 10% concrete FPR. Training reduces the overall non-assertive FPR from 40% to 17%.

Model	Non-assertive share	Halluc. rate
LLaVA-1.5-7B	1.3 [1.0, 1.6]	28.2 [26.9, 29.3]
Qwen2.5-VL-7B	5.0 [4.4, 5.8]	6.6 [5.7, 7.5]
Qwen2.5-VL-3B	4.8 [4.1, 5.6]	8.9 [8.0, 9.9]
InternVL3-8B	4.2 [3.6, 4.9]	7.0 [6.2, 7.8]
LLaVA-OneVision-2-8B	4.4 [3.7, 5.1]	5.7 [4.9, 6.5]

Table 1: Natural rates on captions of random images (2,000 per model), with 95% image-bootstrap confidence intervals. Non-assertive share : fraction of object mentions that are non-assertive. Halluc. rate : fraction of existential mentions that are hallucinated.

Model	Hedged	Depictive	Purpose	Negation	Total	Halluc.	Concrete
	Non-assertive
LLaVA-1.5-7B	564 (35.4)	265 (16.6)	630 (39.5)	135 (8.5)	1,594	1,974	1,975
Qwen2.5-VL-7B	958 (51.3)	642 (34.4)	182 (9.8)	84 (4.5)	1,866	1,983	1,878
Qwen2.5-VL-3B	802 (46.0)	516 (29.6)	211 (12.1)	216 (12.4)	1,745	1,979	1,891
InternVL3-8B	198 (29.2)	307 (45.3)	43 (6.3)	130 (19.2)	678	1,321	1,947
LLaVA-OneVision-2-8B	561 (42.7)	450 (34.2)	172 (13.1)	131 (10.0)	1,314	1,433	1,919

Table 2: Mentions per model in NAME-Bench. Each non-assertive count is followed by its share of that model’s non-assertive total (%).

	Halluc. Recall $\uparrow$		Non-assertive (All) $\downarrow$		Hedged $\downarrow$		Depictive $\downarrow$		Purpose $\downarrow$		Negation $\downarrow$
Model	C	T	C	T	C	T	C	T	C	T	C	T
LLaVA-1.5-7B	92	100	46	55	59	59	36	59	37	62	53	1
Qwen2.5-VL-7B	68	97	49	53	52	55	51	59	34	43	30	14
Qwen2.5-VL-3B	74	98	51	49	54	57	51	60	38	39	53	2
InternVL3-8B	69	98	54	52	51	62	52	66	36	56	68	0
LLaVA-OneVision-2-8B	64	98	46	48	49	56	46	58	36	29	49	1

Table 3: Columns: hallucination recall (

\uparrow

), then non-assertive mentions overall and per subtype (

\downarrow

). C: CHAIR; T: THRONE (majority vote).

	LLaVA-1.5-7B					Qwen2.5-VL-7B					Qwen2.5-VL-3B					InternVL3-8B					LLaVA-OneVision-2
Detector	H	D	P	N	All	H	D	P	N	All	H	D	P	N	All	H	D	P	N	All	H	D	P	N	All
Lookback Lens	64	27	24	45	40	71	75	39	29	67	81	57	59	37	66	80	68	67	11	62	88	77	76	58	80
Hidden state (mid)	64	43	46	48	52	63	66	38	28	60	64	51	42	14	51	70	72	52	22	61	72	72	40	36	63
Hidden state (last)	49	31	27	3	34	69	68	48	32	65	70	57	43	14	56	69	61	56	1	53	76	76	53	24	68
DHCP	28	14	10	14	18	67	53	45	49	59	72	41	62	71	62	75	57	56	42	60	84	55	68	66	71
HALP	18	38	15	33	22	15	13	12	7	13	12	12	6	10	11	24	17	20	9	18	17	18	10	15	16

Table 4: Non-assertive FPR (%,

\downarrow

) per category for detectors built without non-assertive training data. Top: trained probes; bottom: training-free detectors. H: hedged, D: depictive, P: purpose, N: negation; All: all non-assertive mentions of the model (weighted by category size). HALP scores the image before the caption is generated and so gives every mention in a caption the same score.

Non-assertive FPR (%) $\downarrow$
	LLaVA-1.5-7B					Qwen2.5-VL-7B					Qwen2.5-VL-3B					InternVL3-8B					LLaVA-OneVision-2
Detector	0	nat.	10	30	max	0	nat.	10	30	max	0	nat.	10	30	max	0	nat.	10	30	max	0	nat.	10	30	max
Lookback Lens	40	17	4	2	1	67	32	31	15	7	66	31	31	15	5	62	37	38	15	9	80	38	39	17	7
Hidden state (mid)	52	23	6	3	2	60	23	24	14	7	51	23	22	13	7	61	19	19	13	10	63	28	29	18	9
Hidden state (last)	34	17	5	3	2	65	24	24	14	7	56	23	23	15	7	53	20	21	12	9	68	30	31	17	9
DHCP	18	11	5	2	1	59	34	34	15	7	62	28	27	13	6	60	23	22	10	6	71	46	46	22	8

Table 5: Non-assertive FPR (top) and hallucination TPR (bottom) of the trained detectors as non-assertive mentions are added to the training data as negatives. Columns give the number of non-assertive training examples relative to the training hallucinations (%). nat. : natural share (20 examples for LLaVA-1.5, 76–108 for the other models). max : all available non-assertive examples, for models with fewer non-assertive than hallucination examples.

Model	Hedged	Depictive	Purpose	Negation
LLaVA-1.5-7B	3 $\rightarrow$ 19	4 $\rightarrow$ 18	1 $\rightarrow$ 4	0 $\rightarrow$ 2
Qwen2.5-VL-7B	18 $\rightarrow$ 44	10 $\rightarrow$ 35	8 $\rightarrow$ 10	4 $\rightarrow$ 7
Qwen2.5-VL-3B	22 $\rightarrow$ 47	8 $\rightarrow$ 26	11 $\rightarrow$ 13	1 $\rightarrow$ 5
InternVL3-8B	19 $\rightarrow$ 56	9 $\rightarrow$ 39	32 $\rightarrow$ 41	0 $\rightarrow$ 3
LLaVA-OneVision-2-8B	24 $\rightarrow$ 58	15 $\rightarrow$ 39	15 $\rightarrow$ 19	2 $\rightarrow$ 8

Table 6: Non-assertive FPR (%,

\downarrow

) on each subtype when it is included in (seen) → excluded from (unseen) training. Both conditions use a 30% budget; in the unseen condition, the same total number of non-assertive examples is drawn from the other three subtypes. Mean over five trained probes.

Probe	Model	Abs.	Pres.	AUC	Abs.	Pres.	AUC
		Hedged			All non-assertive
Hidden state (mid)	LLaVA-1.5	64	62	0.52	55	43	0.58
	Qwen-7B	74	45	0.71	68	43	0.68
	Qwen-3B	73	43	0.70	59	33	0.66
	InternVL3	80	44	0.78	64	54	0.60
	LLaVA-OV2	83	47	0.77	74	42	0.73

Table 7: Non-assertive mentions flagged (%,

\downarrow

) when the object is absent (Abs.) or present (Pres.) in the image. Results for the mid-layer hidden-state and text-only LM probes, trained without non-assertive examples. AUC measures absent-vs.-present score separation.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Detector	Source	Signal	Score / classifier
Training-free
SVAR	Jiang et al. (2025b)	total attention to image tokens at the object’s generation step, head-mean, summed over the middle layers (5–18 of 32, scaled to depth)	$-$ attention (low image attention = hallucination)
VADE (VisAttnAgg)	Prabhakaran et al. (2025)	mean attention to image tokens over layers, heads and the W4 window	$-$ attention
Internal Confidence	Jiang et al. (2025a)	logit-lens probability of the object word at image tokens, max over layers and image tokens (top-50 softmax)	$-$ confidence
GLSim	Park and Li (2026)	$0.6\times$ global + $0.4\times$ local similarity between the object’s representation and the image (official defaults: text layer $L{-}1$ , image layer $L$ , $k{=}32$ )	$-$ similarity
PAS	Hoang et al. (2026)	attention of the object token to previously generated tokens, layer 0 (paper default)	score as defined

Appendix

Table 8: Detectors.

L

: number of layers of the captioning model; W4: the four caption tokens ending at the object token. All MLPs use Adam (learning rate

10^{-3}

) and binary cross-entropy; DHCP inputs are standardised with fit-set statistics.

Step-2 / step-3 output	Category
Any subtype except depictive, with a negation cue	Negation
Hedging	Hedged
Depictive (with or without a negation cue)	Depictive
Purpose, intensional	Purpose
Generic without a negation cue, other	excluded
Compound (step 3)	excluded

Appendix

Table 9: Mapping from the step-2 subtype and the step-3 compound check to the four non-assertive categories.

Non-assertive FPR (%) $\downarrow$
	LLaVA-1.5-7B					Qwen2.5-VL-7B					Qwen2.5-VL-3B					InternVL3-8B					LLaVA-OV2-8B
Detector	0	nat.	10	30	max	0	nat.	10	30	max	0	nat.	10	30	max	0	nat.	10	30	max	0	nat.	10	30	max
Lookback Lens	40	17	4	2	1	44	9	3	2	1	34	10	2	1	1	31	10	3	2	1	38	10	3	2	1
Hidden state (mid)	52	23	6	3	2	47	23	7	4	2	50	23	9	4	3	55	29	11	4	3	44	16	6	4	3
Hidden state (last)	34	17	5	3	2	34	19	7	4	2	32	20	6	3	2	39	21	9	5	3	39	20	8	4	2
DHCP	18	11	5	2	1	39	13	3	2	1	38	10	3	2	1	29	10	4	2	2	27	12	3	1	1

Appendix

Table 10: Non-assertive FPR (top) and hallucination TPR (bottom) of the trained detectors when every model reads LLaVA-1.5’s captions instead of its own (the LLaVA-1.5 columns repeat Table 5 ). Columns as in Table 5 ; nat. is LLaVA-1.5’s natural share (about 20 examples) and max is 77% for every model, since all read the same captions.

	Filter recall on non-assertive (%)					Halluc.	Training-free FPR		Trained probes FPR
Model	Hedg.	Dep.	Purp.	Neg.	All	removed (%)	no filter	filter	mixing	+ filter
LLaVA-1.5-7B	67	17	79	80	64	0.2	14–57	2–11	11–23	4–13
Qwen2.5-VL-7B	53	70	65	88	62	2.5	21–53	7–19	17–34	14–18
Qwen2.5-VL-3B	52	67	65	99	64	3.1	21–53	6–16	22–31	14–17
InternVL3-8B	58	72	8	99	69	2.1	16–46	4–13	16–37	11–16
LLaVA-OneVision-2-8B	50	61	76	98	62	3.8	16–57	4–18	26–46	19–22

Appendix

Table 11: Text filter placed in front of each detector. Left: share of non-assertive mentions it catches and share of hallucinations it wrongly removes. Right: range of non-assertive FPR (%) over the seven training-free detectors and over the five trained probes mixed at the natural share, without and with the filter.

			Disqualified		Non-assertive counted as halluc.					L1.5
Judge	Vote	Recall	Non-assertive	Conc.	All	Hedg.	Dep.	Purp.	Neg.	All
Flan-T5	majority	98	0	0	51	57	60	38	3	55
	unanimous	78	61	6	46	55	59	39	0	60
Llama	majority	93	0	0	41	50	47	24	2	30
	unanimous	86	41	2	39	50	48	28	2	28

Appendix

Table 12: THRONE with its Flan-T5 judges (nine votes) and with Llama-3.1-8B-Instruct (three votes), under majority and unanimous voting, on the evaluation-set mentions of the four newer models (pooled; L1.5 : LLaVA-1.5). Recall: hallucinations counted as hallucinated, disqualified ones counted as missed. Disqualified: share of non-assertive and concrete mentions with split votes. Non-assertive columns: share counted as hallucinated among the mentions that receive a verdict. All values in %.

	LLaVA-1.5-7B		Qwen2.5-VL-7B		Qwen2.5-VL-3B		InternVL3-8B		LLaVA-OV2-8B
Detector	All	Pos.	All	Pos.	All	Pos.	All	Pos.	All	Pos.
Position only	0.81	0.51	0.67	0.50	0.72	0.50	0.69	0.50	0.68	0.50
PAS	0.82	0.53	0.67	0.51	0.72	0.51	0.68	0.51	0.68	0.51
SVAR	0.77	0.51	0.70	0.62	0.73	0.61	0.67	0.55	0.75	0.68
VADE	0.76	0.48	0.59	0.57	0.67	0.60	0.54	0.54	0.60	0.59
Internal Conf.	0.74	0.69	0.55	0.54	0.63	0.60	0.58	0.57	0.62	0.63

Appendix

Table 13: Concrete-vs-hallucination AUC on all held-out mentions (All) and on a subset matched for token position (Pos.), for detectors built without non-assertive training data. Top: training-free detectors and position alone; bottom: trained probes. Detectors whose AUC falls to about 0.5 once position is matched mostly measure position.

Model	Category	$n$	0%	nat.	10%	30%	max
LLaVA-1.5-7B	Hedged	197	64 / 64	30 / 25	5 / 6	3 / 3	3 / 2
	Depictive	99	27 / 43	22 / 37	8 / 14	3 / 6	2 / 2
	Purpose	211	24 / 46	5 / 18	2 / 3	1 / 2	1 / 1
	Negation	46	45 / 48	4 / 8	1 / 1	1 / 0	0 / 0
Qwen2.5-VL-7B	Hedged	312	71 / 63	39 / 29	39 / 29	21 / 17	10 / 10
	Depictive	212	75 / 66	30 / 19	30 / 20	11 / 10	4 / 5

Appendix

Table 14: Non-assertive FPR (%) per category as non-assertive mentions are added to the training data. Cells: Lookback Lens / mid-layer hidden-state probe. Columns as in Table 5 ( max is 100% for Qwen2.5-VL-7B and LLaVA-OneVision-2).

n

: held-out mentions of the category per seed; cells with

n<30

are noisy.

No non-assertive training data (0%)
	LLaVA-1.5-7B				Qwen2.5-VL-7B				Qwen2.5-VL-3B				InternVL3-8B				LLaVA-OV2-8B
	FPR $\downarrow$		TPR $\uparrow$		FPR $\downarrow$		TPR $\uparrow$		FPR $\downarrow$		TPR $\uparrow$		FPR $\downarrow$		TPR $\uparrow$		FPR $\downarrow$		TPR $\uparrow$
Detector	Ours	Claim	Ours	Claim	Ours	Claim	Ours	Claim	Ours	Claim	Ours	Claim	Ours	Claim	Ours	Claim	Ours	Claim	Ours	Claim
Lookback Lens	27	37	54	50	63	50	64	58	53	42	64	57	54	48	55	54	74	58	71	67
Hidden state (mid)	46	58	49	49	57	51	69	62	40	29	64	55	58	56	63	63	57	51	75	73
Hidden state (last)	25	41	46	44	60	50	72	64	44	32	64	55	46	43	62	60	61	50	77	72

Appendix

Table 15: The failure does not depend on how hedges are labelled. Ours : hedges are non-assertive (our convention); Claim : every hedge is re-labelled as a claim (object absent

\rightarrow

hallucination, present

\rightarrow

concrete) and the probes are retrained. FPR (%): held-out depictive, purpose and negation mentions flagged as hallucinated (the same mentions under both conventions); TPR (%): held-out hallucinations flagged. Threshold flags 10% of concrete mentions; each model on its own captions; mean of 3 seeds.

Model	4 cat.	Other	Comp.	4 cat.	Other	Comp.	per seed
	Trained Probes (%) $\downarrow$			SVAR (%) $\downarrow$			Other $n$
LLaVA-1.5-7B	18–52	14–32	11–39	57	67	19	282
Qwen2.5-VL-7B	56–67	36–50	51–74	53	70	35	158
Qwen2.5-VL-3B	48–66	40–56	47–62	53	65	30	166
InternVL3-8B	53–62	56–75	52–60	46	61	22	46
LLaVA-OV2-8B	60–80	53–69	61–77	57	63	42	95

Appendix

Table 16: Detector flag rates (%,

\downarrow

) without non-assertive training. 4 cat. : retained non-assertive subtypes; Other : excluded valid non-assertive mentions; Comp. : invalid compound matches. Trained probe results show ranges across five detectors. Same splits and thresholds as Table 4 ; mean over three seeds.

n

denotes held-out Other mentions per seed.

Model	Category	$n$	Concepts	Five most frequent concepts (% of the category)
LLaVA-1.5-7B	Hedged	564	58	person 31, bird 7, bed 6, sports ball 5, couch 5
	Depictive	265	50	person 28, dog 11, train 6, bird 6, cat 6
	Purpose	630	26	person 48, sports ball 26, train 8, bus 4, tv 2
	Negation	135	6	person 69, car 27, boat 2, tv 1, train 1
Qwen2.5-VL-7B	Hedged	958	69	person 10, bed 7, cell phone 7, train 6, couch 5
	Depictive	642	57	person 23, bird 16, car 8, bicycle 7, dog 6

Appendix

Table 17: COCO concepts of the non-assertive mentions in NAME-Bench, per model and category: number of mentions (

n

, as in Table 2 ), number of distinct concepts, and the five most frequent concepts with their share of the category (%). The concept “orange” is excluded.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 986 papers

Overthinking Causes Hallucination: Tracing Confounder Propagation in Vision Language Models
Mar 8, 2026Abin Shoby, Ta Duc Huy, Tuan Dung Nguyen +6Hallucination Detection in VLMs Object Hallucination in VLMs
Australian Institute for Machine Learning, University of Adelaide · Hanoi University of Science and Technology · Mohamed bin Zayed University of Artificial Intelligence
VisER: Visual Evidence and Reliance for Object Hallucination Detection in LVLMs
Aug 31, 2026Afsaneh Hasanebrahimi, Hanxun Huang, Christopher Leckie +1Hallucination Detection in VLMs VLM Hallucination
School of Computing and Information Systems The University of Melbourne Melbourne, Australia
HalDec-Bench: Benchmarking Hallucination Detector in Image Captioning
Mar 16, 2026Kuniaki Saito, Risa Shinoda, Shohei Tanaka +3Image Captioning Evaluation Hallucination Detection in VLMs
OMRON SINICX · The University of Tokyo · The University of Osaka

Mar 8, 2026J/K move · Enter open · S save

Overthinking Causes Hallucination: Tracing Confounder Propagation in Vision Language Models

Abin Shoby, Ta Duc Huy, Tuan Dung Nguyen, Minh Khoi Ho+5

Australian Institute for Machine Learning, University of Adelaide · Hanoi University of Science and Technology · Mohamed bin Zayed University of Artificial Intelligence

Hallucination Detection in VLMs Object Hallucination in VLMs VLM Interpretability

Open paper AskPDF arXiv

Vision Language models (VLMs) often hallucinate non-existent objects. Detecting hallucination is analogous to detecting deception: a single final statement is insufficient, one must examine the underlying reasoning process. Yet existing detectors rely mostly on final-layer signals. Attention-based methods assume hallucinated tokens exhibit low attention, while entropy-based ones use final-step uncertainty. Our analysis reveals the opposite: hallucinated objects can exhibit peaked attention due to contextual priors; and models often express high confidence because intermediate layers have already converged to an incorrect hypothesis. We show that the key to hallucination detection lies within the model's thought process, not its final output. By probing decoder layers, we uncover a previously overlooked behavior, overthinking: models repeatedly revise object hypotheses across layers before committing to an incorrect answer. Once the model latches onto a confounded hypothesis, it can propagate through subsequent layers, ultimately causing hallucination. To capture this behavior, we introduce the Overthinking Score, a metric to measure how many competing hypotheses the model entertains and how unstable these hypotheses are across layers. This score significantly improves hallucination detection: 78.9% F1 on MSCOCO and 71.58% on AMBER.

Abstract

VLM Evaluation

Explore similar work

Overthinking Causes Hallucination: Tracing Confounder Propagation in Vision Language Models

VisER: Visual Evidence and Reliance for Object Hallucination Detection in LVLMs

HalDec-Bench: Benchmarking Hallucination Detector in Image Captioning