Beyond the Global Scores: Fine-Grained Token Grounding as a Robust Detector of LVLM Hallucinations
Authors: Tuan Dung Nguyen, Minh Khoi Ho, Qi Chen, Yutong Xie, Nguyen Cam-Tu, Minh Khoi Nguyen, Dang Huy Pham Nguyen, Anton van den Hengel, Johan W. Verjans, Phi Le Nguyen, Vu Minh Hieu Phan
Organizations: Hanoi University of Science and Technology · Australian Institute for Machine Learning, University of Adelaide · Mohamed bin Zayed University of Artificial Intelligence · Nanjing University · Hanoi-Amsterdam High School for the Gifted
Large vision-language models (LVLMs) achieve strong performance on visual reasoning tasks but remain highly susceptible to hallucination. Existing detection methods predominantly rely on coarse, whole-image measures of how an object token relates to the input image. This global strategy is limited: hallucinated tokens may exhibit weak but widely scattered correlations across many local regions, which aggregate into deceptively high overall relevance, thus evading the current global hallucination detectors. We begin with a simple yet critical observation: a faithful object token must be strongly grounded in a specific image region. Building on this insight, we introduce a patch-level hallucination detection framework that examines fine-grained token-level interactions across model layers. Our analysis uncovers two characteristic signatures of hallucinated tokens: (i) they yield diffuse, non-localized attention patterns, in contrast to the compact, well-focused attention seen in faithful tokens; and (ii) they fail to exhibit meaningful semantic alignment with any visual region. Guided by these findings, we develop a lightweight and interpretable detection method that leverages patch-level statistical features, combined with hidden-layer representations. Our approach achieves up to 90% accuracy in token-level hallucination detection, demonstrating the superiority of fine-grained structural analysis for detecting hallucinations.
Figures & tables
Figure 1: Left: SVAR [ 12 ] captures image-level statistics by globally summing the attention across the image, ignoring local attention structures. This triggers a false alarm when the local noisy attention appears. Right: Our proposed “attention dispersion score” encoded token-level attention distribution , quantifying how much the model allocates attention to various local regions simultaneously. Spreading the focus across multiple regions, i.e . high entropy, indicates hallucination. Specifically, our pipeline first suppresses noisy regions and detects focused objects by grouping, and then quantifies the attention entropy of the token-level distributions. Low entropy means the model “sharply” focuses on particular objects, thus being likely to predict true objects. In other words, high entropy means the model is unfocused when predicting, indicating hallucination.
Figure 2: Overview of our token-level hallucination detection framework . We reveal two key indicators of hallucination: (1) Attention Dispersion Score (ADS) , hallucinated tokens exhibit highly diffused, non-localized attention across local image patches, while faithful tokens show concentrated focus; and (2) Cross-modal Grounding Consistency (CGC) , hallucinated text tokens exhibit low alignment with any object regions, as shown in the right figure. Our proposed metrics encode local structures of the VLMs’ behaviors, enabling lightweight and explainable hallucination detection robust to attention sink scenarios.
Figure 3: Illustration of the Attention Dispersion Score (ADS) computation. After predicting an object token, we extract the text-to-patch cross-modal attention map. The top k% activations are kept to isolate highly focused regions. Then, we form the attended object regions and suppress attention sinks by applying N -connected component. Finally, the proposed ADS score is computed by quantifying the entropy of the resulting token-level attention distribution. The peaky distribution indicates sharp object focus, indicating real object token predictions. In contrast, the uniform patch-wise distribution implies that the model scatters its attention when predicting, indicating hallucinated predictions.
Figure 4: The visual attention maps for or a true token (“camera”, top) vs. a hallucinated token (“wristwatch”, bottom) across layers (10, 17, 22). True tokens reveal a focused attention pattern aligned with the object’s location, while hallucination tokens have scattered attention across the image.
Figure 5: Layer-wise attention entropy of true vs. hallucinated tokens across LVLMs (lower is better; more focused). Reported p -values indicate strong separation in early/mid layers.
Figure 6: Attention dispersion distribution for true vs. hallucinated tokens for LLaVA-1.5-7B.
Model
Prec.
Rec.
F1
LLaVA-1.5-7B [ 19 ]
0.81
0.74
0.77
InternVL-2.5-8B [ 6 ]
0.79
0.69
0.73
Qwen 2.5-VL-7B [ 3 ]
0.75
0.71
0.73
Figure 7: Performance of ADS classifier across LVLMs.
Figure 8: Layer-wise patch-level similarity scores for true vs. hallucinated tokens across LVLMs. Early/mid layers are most discriminative; deeper layers converge due to stronger language priors.
Figure 9: Cross-modal grounding consistency heatmap for a true token (“television”, top) vs. a hallucinated token (“table”, bottom) across layers (10, 17, 22). True tokens display similarity clusters aligned with the object’s local structure, while hallucination tokens have low alignment with all patches across the image.
Figure 10: Top– 5% similarities for true vs. hallucinated tokens for LLaVA-1.5-7B.
Model
Prec.
Rec.
F1
LLaVA-1.5-7B [ 19 ]
0.82
0.76
0.79
InternVL-2.5-8B [ 6 ]
0.80
0.73
0.76
Qwen 2.5-VL-7B [ 3 ]
0.83
0.79
0.81
Table 1: Performance of the CGC classifier across LVLMs.
Method
LLaVA-1.5-7B
Qwen2.5-VL-7B
InternVL2.5-8B
PR
RC
F1
AUC
PR
RC
F1
AUC
PR
RC
F1
AUC
HalLoc [ 21 ]
0.70
0.73
0.71
0.76
0.71
0.73
0.73
0.80
0.64
0.68
0.67
0.87
MetaToken (LR) [ 8 ]
0.61
0.43
0.51
0.57
0.46
0.65
0.54
0.69
0.68
0.62
0.65
0.60
MetaToken (GB) [ 8 ]
0.65
0.72
0.68
0.72
0.70
0.44
0.54
0.77
0.66
0.96
0.78
0.72
SVAR [ 12 ]
0.66
0.89
0.76
0.75
0.45
0.78
0.57
0.68
0.74
0.89
0.80
0.87
DHCP [ 29 ]
0.75
0.71
0.73
0.80
0.75
0.77
0.76
0.86
0.80
0.83
0.81
0.87
Table 2: Token-level hallucination detection across LVLMs for image captioning task on MS COCO dataset. Bold indicates best, underline indicates second best.
Method
F1
AUC
MetaToken [ 8 ]
0.22
0.73
HalLoc [ 21 ]
0.29
0.71
SVAR [ 12 ]
0.25
0.37
DHCP [ 29 ]
0.27
0.69
ProjectAway [ 11 ]
0.38
0.70
Ours
0.41
0.75
Table 3: Hallucination detection performance on POPE with LLaVA-1.5-7B. Due to class imbalance, we report F1 and AUC, which provide more stable and threshold-robust evaluation.
Model
Features
AUC
F1
LLaVA-1.5
ADS
0.86
0.80
CGC
0.84
0.79
CGC+ADS
0.92
0.84
Qwen 2.5-VL-7B
ADS
0.93
0.76
CGC
0.93
0.80
CGC+ADS
0.95
0.85
Table 4: Effect of combining CGC and ADS features on hallucination detection across three VLMs.
Model
Hyper - parameter
Values
XGB
depth
{4,6,8}
learning rates
{0.1,0.05}
no. estimators
{100,200,500}
RF
depth
{None,10,20}
trees
{200,400,600}
MLP
hidden size
{64,128,256}
Table 5: Hyper - parameter ranges for each classifier.
Figure 11: Mean absolute SHAP values for ADS and CGC across layers.
Layer Set
ADS
CGC
AUC
F1
AUC
F1
Layers 6–12
72.38
67.88
76.51
73.26
Layers 13–18
69.17
62.91
73.34
70.55
Layers 19–24
68.31
63.74
68.79
63.44
Layers 25–33
64.58
62.88
75.44
74.34
All layers
85.84
80.36
84.41
78.90
Table 6: Layer-wise ablation of ADS and CGC features.
Model
Features
AUC
F1
LLaVA-1.5
SVAR
85.12
69.35
+CGC
87.72
73.52
+ADS
89.04
74.03
+CGC+ADS
89.41
74.07
Qwen 2.5-VL-8B
SVAR
87.85
76.38
+CGC
86.79
76.24
Table 7: Effect of adding CGC and ADS features to hallucination detectors across three VLMs.
ADS
CGC
Top- x %
AUC
F1
Top- k %
AUC
F1
5%
81.65
76.54
1%
81.36
75.31
10%
83.86
76.83
3%
82.55
76.54
15%
82.87
77.65
5%
83.86
76.83
20%
81.65
76.07
10%
82.57
77.78
30%
79.83
74.70
20%
81.82
72.96
Table 8: Sensitivity of ADS foreground threshold and CGC top- k % patch selection on LLaVA-1.5-7B. Each side varies one hyperparameter while fixing the other at its default. Bold indicates default.
School of Artificial Intelligence, University of Chinese Academy of Sciences · 3Hong Kong University of Science and Technology · 2New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Science