Object hallucination in vision-language models (VLMs) is usually measured by checking each object a caption names against the image. Most benchmarks and detectors alike treat every object mention as a claim that the object is present. This assumption fails when a caption names an object without asserting it, as in "a bench for people to sit on", which mentions people without claiming that any are in the image. We call such mentions non-assertive. Treating them as hallucinations inflates the error rates attributed to models that use such language. To separate them from real hallucinations, we introduce NAME-Bench (Non-Assertive Mention Evaluation Benchmark), 25,497 object mentions from captions that five VLMs generate for COCO images. Each mention is labelled concrete (asserted and present), hallucinated (asserted and absent), or non-assertive, and non-assertive mentions are further divided into four subtypes: depictive, purpose, negation and hedged. All the hallucination detectors that we evaluate flag non-assertive mentions as hallucinations, at rates between 14% and 80%. Adding non-assertive mentions to detector training as negatives largely corrects this on LLaVA-1.5. On the four newer VLMs, a comparable correction takes five to ten times more data, and the detectors then catch fewer real hallucinations. These findings show that current detectors conflate mentioning an object with asserting it, and that this conflation is harder to undo on the four newer VLMs than on LLaVA-1.5. Hallucination evaluation should therefore separate what a caption asserts from what it merely mentions.
Figures & tables
Figure 1: Hallucination detectors flag non-assertive mentions. Six illustrative held-out examples from LLaVA-1.5: concrete, hallucinated, and four non-assertive subtypes (hedged, depictive, purpose, and negation). Bars show Lookback Lens scores before non-assertive training; circles show scores after adding 20 non-assertive examples. The dashed line marks the threshold calibrated at 10% concrete FPR. Training reduces the overall non-assertive FPR from 40% to 17%.
Model
Non-assertive share
Halluc. rate
LLaVA-1.5-7B
1.3 [1.0, 1.6]
28.2 [26.9, 29.3]
Qwen2.5-VL-7B
5.0 [4.4, 5.8]
6.6 [5.7, 7.5]
Qwen2.5-VL-3B
4.8 [4.1, 5.6]
8.9 [8.0, 9.9]
InternVL3-8B
4.2 [3.6, 4.9]
7.0 [6.2, 7.8]
LLaVA-OneVision-2-8B
4.4 [3.7, 5.1]
5.7 [4.9, 6.5]
Table 1: Natural rates on captions of random images (2,000 per model), with 95% image-bootstrap confidence intervals. Non-assertive share : fraction of object mentions that are non-assertive. Halluc. rate : fraction of existential mentions that are hallucinated.
Non-assertive
Model
Hedged
Depictive
Purpose
Negation
Total
Halluc.
Concrete
LLaVA-1.5-7B
564 (35.4)
265 (16.6)
630 (39.5)
135 (8.5)
1,594
1,974
1,975
Qwen2.5-VL-7B
958 (51.3)
642 (34.4)
182 (9.8)
84 (4.5)
1,866
1,983
1,878
Qwen2.5-VL-3B
802 (46.0)
516 (29.6)
211 (12.1)
216 (12.4)
1,745
1,979
1,891
InternVL3-8B
198 (29.2)
307 (45.3)
43 (6.3)
130 (19.2)
678
1,321
1,947
LLaVA-OneVision-2-8B
561 (42.7)
450 (34.2)
172 (13.1)
131 (10.0)
1,314
1,433
1,919
Table 2: Mentions per model in NAME-Bench. Each non-assertive count is followed by its share of that model’s non-assertive total (%).
Halluc. Recall ↑
Non-assertive (All) ↓
Hedged ↓
Depictive ↓
Purpose ↓
Negation ↓
Model
C
T
C
T
C
T
C
T
C
T
C
T
LLaVA-1.5-7B
92
100
46
55
59
59
36
59
37
62
53
1
Qwen2.5-VL-7B
68
97
49
53
52
55
51
59
34
43
30
14
Qwen2.5-VL-3B
74
98
51
49
54
57
51
60
38
39
53
2
InternVL3-8B
69
98
54
52
51
62
52
66
36
56
68
0
LLaVA-OneVision-2-8B
64
98
46
48
49
56
46
58
36
29
49
1
Table 3: Columns: hallucination recall ( ↑ ), then non-assertive mentions overall and per subtype ( ↓ ). C: CHAIR; T: THRONE (majority vote).
LLaVA-1.5-7B
Qwen2.5-VL-7B
Qwen2.5-VL-3B
InternVL3-8B
LLaVA-OneVision-2
Detector
H
D
P
N
All
H
D
P
N
All
H
D
P
N
All
H
D
P
N
All
H
D
P
N
All
Lookback Lens
64
27
24
45
40
71
75
39
29
67
81
57
59
37
66
80
68
67
11
62
88
77
76
58
80
Hidden state (mid)
64
43
46
48
52
63
66
38
28
60
64
51
42
14
51
70
72
52
22
61
72
72
40
36
63
Hidden state (last)
49
31
27
3
34
69
68
48
32
65
70
57
43
14
56
69
61
56
1
53
76
76
53
24
68
DHCP
28
14
10
14
18
67
53
45
49
59
72
41
62
71
62
75
57
56
42
60
84
55
68
66
71
HALP
18
38
15
33
22
15
13
12
7
13
12
12
6
10
11
24
17
20
9
18
17
18
10
15
16
Table 4: Non-assertive FPR (%, ↓ ) per category for detectors built without non-assertive training data. Top: trained probes; bottom: training-free detectors. H: hedged, D: depictive, P: purpose, N: negation; All: all non-assertive mentions of the model (weighted by category size). HALP scores the image before the caption is generated and so gives every mention in a caption the same score.
LLaVA-1.5-7B
Qwen2.5-VL-7B
Qwen2.5-VL-3B
InternVL3-8B
LLaVA-OneVision-2
Detector
0
nat.
10
30
max
0
nat.
10
30
max
0
nat.
10
30
max
0
nat.
10
30
max
0
nat.
10
30
max
Non-assertive FPR (%) ↓
Lookback Lens
40
17
4
2
1
67
32
31
15
7
66
31
31
15
5
62
37
38
15
9
80
38
39
17
7
Hidden state (mid)
52
23
6
3
2
60
23
24
14
7
51
23
22
13
7
61
19
19
13
10
63
28
29
18
9
Hidden state (last)
34
17
5
3
2
65
24
24
14
7
56
23
23
15
7
53
20
21
12
9
68
30
31
17
9
DHCP
18
11
5
2
1
59
34
34
15
7
62
28
27
13
6
60
23
22
10
6
71
46
46
22
8
Table 5: Non-assertive FPR (top) and hallucination TPR (bottom) of the trained detectors as non-assertive mentions are added to the training data as negatives. Columns give the number of non-assertive training examples relative to the training hallucinations (%). nat. : natural share (20 examples for LLaVA-1.5, 76–108 for the other models). max : all available non-assertive examples, for models with fewer non-assertive than hallucination examples.
Model
Hedged
Depictive
Purpose
Negation
LLaVA-1.5-7B
3 → 19
4 → 18
1 → 4
0 → 2
Qwen2.5-VL-7B
18 → 44
10 → 35
8 → 10
4 → 7
Qwen2.5-VL-3B
22 → 47
8 → 26
11 → 13
1 → 5
InternVL3-8B
19 → 56
9 → 39
32 → 41
0 → 3
LLaVA-OneVision-2-8B
24 → 58
15 → 39
15 → 19
2 → 8
Table 6: Non-assertive FPR (%, ↓ ) on each subtype when it is included in (seen) → excluded from (unseen) training. Both conditions use a 30% budget; in the unseen condition, the same total number of non-assertive examples is drawn from the other three subtypes. Mean over five trained probes.
Hedged
All non-assertive
Probe
Model
Abs.
Pres.
AUC
Abs.
Pres.
AUC
Hidden state (mid)
LLaVA-1.5
64
62
0.52
55
43
0.58
Qwen-7B
74
45
0.71
68
43
0.68
Qwen-3B
73
43
0.70
59
33
0.66
InternVL3
80
44
0.78
64
54
0.60
LLaVA-OV2
83
47
0.77
74
42
0.73
Table 7: Non-assertive mentions flagged (%, ↓ ) when the object is absent (Abs.) or present (Pres.) in the image. Results for the mid-layer hidden-state and text-only LM probes, trained without non-assertive examples. AUC measures absent-vs.-present score separation.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Detector
Source
Signal
Score / classifier
Training-free
SVAR
Jiang et al. (2025b)
total attention to image tokens at the object’s generation step, head-mean, summed over the middle layers (5–18 of 32, scaled to depth)
− attention (low image attention = hallucination)
VADE (VisAttnAgg)
Prabhakaran et al. (2025)
mean attention to image tokens over layers, heads and the W4 window
− attention
Internal Confidence
Jiang et al. (2025a)
logit-lens probability of the object word at image tokens, max over layers and image tokens (top-50 softmax)
− confidence
GLSim
Park and Li (2026)
0.6× global + 0.4× local similarity between the object’s representation and the image (official defaults: text layer L−1 , image layer L , k=32 )
− similarity
PAS
Hoang et al. (2026)
attention of the object token to previously generated tokens, layer 0 (paper default)
score as defined
Appendix
Table 8: Detectors. L : number of layers of the captioning model; W4: the four caption tokens ending at the object token. All MLPs use Adam (learning rate 10−3 ) and binary cross-entropy; DHCP inputs are standardised with fit-set statistics.
Step-2 / step-3 output
Category
Any subtype except depictive, with a negation cue
Negation
Hedging
Hedged
Depictive (with or without a negation cue)
Depictive
Purpose, intensional
Purpose
Generic without a negation cue, other
excluded
Compound (step 3)
excluded
Appendix
Table 9: Mapping from the step-2 subtype and the step-3 compound check to the four non-assertive categories.
LLaVA-1.5-7B
Qwen2.5-VL-7B
Qwen2.5-VL-3B
InternVL3-8B
LLaVA-OV2-8B
Detector
0
nat.
10
30
max
0
nat.
10
30
max
0
nat.
10
30
max
0
nat.
10
30
max
0
nat.
10
30
max
Non-assertive FPR (%) ↓
Lookback Lens
40
17
4
2
1
44
9
3
2
1
34
10
2
1
1
31
10
3
2
1
38
10
3
2
1
Hidden state (mid)
52
23
6
3
2
47
23
7
4
2
50
23
9
4
3
55
29
11
4
3
44
16
6
4
3
Hidden state (last)
34
17
5
3
2
34
19
7
4
2
32
20
6
3
2
39
21
9
5
3
39
20
8
4
2
DHCP
18
11
5
2
1
39
13
3
2
1
38
10
3
2
1
29
10
4
2
2
27
12
3
1
1
Appendix
Table 10: Non-assertive FPR (top) and hallucination TPR (bottom) of the trained detectors when every model reads LLaVA-1.5’s captions instead of its own (the LLaVA-1.5 columns repeat Table 5 ). Columns as in Table 5 ; nat. is LLaVA-1.5’s natural share (about 20 examples) and max is 77% for every model, since all read the same captions.
Filter recall on non-assertive (%)
Halluc.
Training-free FPR
Trained probes FPR
Model
Hedg.
Dep.
Purp.
Neg.
All
removed (%)
no filter
filter
mixing
+ filter
LLaVA-1.5-7B
67
17
79
80
64
0.2
14–57
2–11
11–23
4–13
Qwen2.5-VL-7B
53
70
65
88
62
2.5
21–53
7–19
17–34
14–18
Qwen2.5-VL-3B
52
67
65
99
64
3.1
21–53
6–16
22–31
14–17
InternVL3-8B
58
72
8
99
69
2.1
16–46
4–13
16–37
11–16
LLaVA-OneVision-2-8B
50
61
76
98
62
3.8
16–57
4–18
26–46
19–22
Appendix
Table 11: Text filter placed in front of each detector. Left: share of non-assertive mentions it catches and share of hallucinations it wrongly removes. Right: range of non-assertive FPR (%) over the seven training-free detectors and over the five trained probes mixed at the natural share, without and with the filter.
Disqualified
Non-assertive counted as halluc.
L1.5
Judge
Vote
Recall
Non-assertive
Conc.
All
Hedg.
Dep.
Purp.
Neg.
All
Flan-T5
majority
98
0
0
51
57
60
38
3
55
unanimous
78
61
6
46
55
59
39
0
60
Llama
majority
93
0
0
41
50
47
24
2
30
unanimous
86
41
2
39
50
48
28
2
28
Appendix
Table 12: THRONE with its Flan-T5 judges (nine votes) and with Llama-3.1-8B-Instruct (three votes), under majority and unanimous voting, on the evaluation-set mentions of the four newer models (pooled; L1.5 : LLaVA-1.5). Recall: hallucinations counted as hallucinated, disqualified ones counted as missed. Disqualified: share of non-assertive and concrete mentions with split votes. Non-assertive columns: share counted as hallucinated among the mentions that receive a verdict. All values in %.
LLaVA-1.5-7B
Qwen2.5-VL-7B
Qwen2.5-VL-3B
InternVL3-8B
LLaVA-OV2-8B
Detector
All
Pos.
All
Pos.
All
Pos.
All
Pos.
All
Pos.
Position only
0.81
0.51
0.67
0.50
0.72
0.50
0.69
0.50
0.68
0.50
PAS
0.82
0.53
0.67
0.51
0.72
0.51
0.68
0.51
0.68
0.51
SVAR
0.77
0.51
0.70
0.62
0.73
0.61
0.67
0.55
0.75
0.68
VADE
0.76
0.48
0.59
0.57
0.67
0.60
0.54
0.54
0.60
0.59
Internal Conf.
0.74
0.69
0.55
0.54
0.63
0.60
0.58
0.57
0.62
0.63
Appendix
Table 13: Concrete-vs-hallucination AUC on all held-out mentions (All) and on a subset matched for token position (Pos.), for detectors built without non-assertive training data. Top: training-free detectors and position alone; bottom: trained probes. Detectors whose AUC falls to about 0.5 once position is matched mostly measure position.
Model
Category
n
0%
nat.
10%
30%
max
LLaVA-1.5-7B
Hedged
197
64 / 64
30 / 25
5 / 6
3 / 3
3 / 2
Depictive
99
27 / 43
22 / 37
8 / 14
3 / 6
2 / 2
Purpose
211
24 / 46
5 / 18
2 / 3
1 / 2
1 / 1
Negation
46
45 / 48
4 / 8
1 / 1
1 / 0
0 / 0
Qwen2.5-VL-7B
Hedged
312
71 / 63
39 / 29
39 / 29
21 / 17
10 / 10
Depictive
212
75 / 66
30 / 19
30 / 20
11 / 10
4 / 5
Appendix
Table 14: Non-assertive FPR (%) per category as non-assertive mentions are added to the training data. Cells: Lookback Lens / mid-layer hidden-state probe. Columns as in Table 5 ( max is 100% for Qwen2.5-VL-7B and LLaVA-OneVision-2). n : held-out mentions of the category per seed; cells with n<30 are noisy.
LLaVA-1.5-7B
Qwen2.5-VL-7B
Qwen2.5-VL-3B
InternVL3-8B
LLaVA-OV2-8B
FPR ↓
TPR ↑
FPR ↓
TPR ↑
FPR ↓
TPR ↑
FPR ↓
TPR ↑
FPR ↓
TPR ↑
Detector
Ours
Claim
Ours
Claim
Ours
Claim
Ours
Claim
Ours
Claim
Ours
Claim
Ours
Claim
Ours
Claim
Ours
Claim
Ours
Claim
No non-assertive training data (0%)
Lookback Lens
27
37
54
50
63
50
64
58
53
42
64
57
54
48
55
54
74
58
71
67
Hidden state (mid)
46
58
49
49
57
51
69
62
40
29
64
55
58
56
63
63
57
51
75
73
Hidden state (last)
25
41
46
44
60
50
72
64
44
32
64
55
46
43
62
60
61
50
77
72
Appendix
Table 15: The failure does not depend on how hedges are labelled. Ours : hedges are non-assertive (our convention); Claim : every hedge is re-labelled as a claim (object absent → hallucination, present → concrete) and the probes are retrained. FPR (%): held-out depictive, purpose and negation mentions flagged as hallucinated (the same mentions under both conventions); TPR (%): held-out hallucinations flagged. Threshold flags 10% of concrete mentions; each model on its own captions; mean of 3 seeds.
Trained Probes (%) ↓
SVAR (%) ↓
Other n
Model
4 cat.
Other
Comp.
4 cat.
Other
Comp.
per seed
LLaVA-1.5-7B
18–52
14–32
11–39
57
67
19
282
Qwen2.5-VL-7B
56–67
36–50
51–74
53
70
35
158
Qwen2.5-VL-3B
48–66
40–56
47–62
53
65
30
166
InternVL3-8B
53–62
56–75
52–60
46
61
22
46
LLaVA-OV2-8B
60–80
53–69
61–77
57
63
42
95
Appendix
Table 16: Detector flag rates (%, ↓ ) without non-assertive training. 4 cat. : retained non-assertive subtypes; Other : excluded valid non-assertive mentions; Comp. : invalid compound matches. Trained probe results show ranges across five detectors. Same splits and thresholds as Table 4 ; mean over three seeds. n denotes held-out Other mentions per seed.
Model
Category
n
Concepts
Five most frequent concepts (% of the category)
LLaVA-1.5-7B
Hedged
564
58
person 31, bird 7, bed 6, sports ball 5, couch 5
Depictive
265
50
person 28, dog 11, train 6, bird 6, cat 6
Purpose
630
26
person 48, sports ball 26, train 8, bus 4, tv 2
Negation
135
6
person 69, car 27, boat 2, tv 1, train 1
Qwen2.5-VL-7B
Hedged
958
69
person 10, bed 7, cell phone 7, train 6, couch 5
Depictive
642
57
person 23, bird 16, car 8, bicycle 7, dog 6
Appendix
Table 17: COCO concepts of the non-assertive mentions in NAME-Bench, per model and category: number of mentions ( n , as in Table 2 ), number of distinct concepts, and the five most frequent concepts with their share of the category (%). The concept “orange” is excluded.
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
Vision Language models (VLMs) often hallucinate non-existent objects. Detecting hallucination is analogous to detecting deception: a single final statement is insufficient, one must examine the underlying reasoning process. Yet existing detectors rely mostly on final-layer signals. Attention-based methods assume hallucinated tokens exhibit low attention, while entropy-based ones use final-step uncertainty. Our analysis reveals the opposite: hallucinated objects can exhibit peaked attention due to contextual priors; and models often express high confidence because intermediate layers have already converged to an incorrect hypothesis. We show that the key to hallucination detection lies within the model's thought process, not its final output. By probing decoder layers, we uncover a previously overlooked behavior, overthinking: models repeatedly revise object hypotheses across layers before committing to an incorrect answer. Once the model latches onto a confounded hypothesis, it can propagate through subsequent layers, ultimately causing hallucination. To capture this behavior, we introduce the Overthinking Score, a metric to measure how many competing hypotheses the model entertains and how unstable these hypotheses are across layers. This score significantly improves hallucination detection: 78.9% F1 on MSCOCO and 71.58% on AMBER.
Abin Shoby, Ta Duc Huy, Tuan Dung Nguyen +6
Australian Institute for Machine Learning, University of Adelaide · Hanoi University of Science and Technology · Mohamed bin Zayed University of Artificial Intelligence
Object hallucination remains a persistent reliability issue in large vision-language models, where generated object mentions may sound plausible but lack visual grounding. Recent training-free detectors use internal signals such as token likelihood, attention, visual confidence, or image-text similarity to identify hallucinated objects. These signals are useful, but they are often source-confounded. They measure how strongly an object is supported inside the model without distinguishing whether that support comes from object-specific visual evidence or the generated text prefix. In difficult cases, a hallucinated object can still receive high internal support because it fits the scene, is associated with nearby visual cues, or follows naturally from the generated text prefix. We propose VisER, a training-free two-sided metric for object-level hallucination detection. VisER evaluates each generated object mention from two complementary views. Visual Evidence measures whether object-context compatibility is backed by object-specific evidence from image tokens. Visual Reliance measures whether the object is supported more by the image than by the generated prefix. Combining these views gives a more source-aware grounding score, while avoiding additional object-level verification generations. Across multiple LVLMs and benchmarks, VisER improves AUROC and AUPR over a range of baselines.
Afsaneh Hasanebrahimi, Hanxun Huang, Christopher Leckie +1
School of Computing and Information Systems The University of Melbourne Melbourne, Australia
Hallucination detection in captions (HalDec) assesses a vision-language model's ability to correctly align image content with text by identifying errors in captions that misrepresent the image. Beyond evaluation, effective hallucination detection is also essential for curating high-quality image-caption pairs used to train VLMs. However, the generalizability of VLMs as hallucination detectors across different captioning models and hallucination types remains unclear due to the lack of a comprehensive benchmark. In this work, we introduce HalDec-Bench, a benchmark designed to evaluate hallucination detectors in a principled and interpretable manner. HalDec-Bench contains captions generated by diverse VLMs together with human annotations indicating the presence of hallucinations, detailed hallucination-type categories, and segment-level labels. The benchmark provides tasks with a wide range of difficulty levels and reveals performance differences across models that are not visible in existing multimodal reasoning or alignment benchmarks. Our analysis further uncovers two key findings. First, detectors tend to recognize sentences appearing at the beginning of a response as correct, regardless of their actual correctness. Second, our experiments suggest that dataset noise can be substantially reduced by using strong VLMs as filters while employing recent VLMs as caption generators. Our project page is available at https://dahlian00.github.io/HalDec-Bench-Page/.
Kuniaki Saito, Risa Shinoda, Shohei Tanaka +3
OMRON SINICX · The University of Tokyo · The University of Osaka