Large language models frequently exhibit hallucinations: fluent and confident outputs that are factually incorrect or unsupported by the input context. While recent hallucination detection methods have explored various features derived from attention maps, the underlying mechanisms they exploit remain poorly understood. In this work, we propose SinkProbe, a hallucination detection method grounded in the observation that hallucinations are deeply entangled with attention sinks - tokens that accumulate disproportionate attention mass during generation - indicating a transition from distributed, input-grounded attention to compressed, prior-dominated computation. Importantly, although sink scores are computed solely from attention maps, we find that the classifier preferentially relies on sinks whose associated value vectors have large norms. Moreover, we show that previous methods implicitly depend on attention sinks by establishing their mathematical relationship to sink scores. Our findings yield a novel hallucination detection method grounded in theory that produces state-of-the-art results across popular datasets and LLMs.
Figures & tables
Figure 1: Pipeline for hallucination detection based on attention sink scores. For each layer l and head h , we compute sink scores s(l,h) , defined as the average of attention scores directed toward each token position. These scores are then sorted, and the top- k values are selected as features. The selected features from all layers and heads are concatenated to form the final feature vector z , which is passed to a hallucination probe – a binary classifier trained on labeled examples to distinguish hallucinated outputs from factually correct ones at inference time.
Figure 2: Relationship between attention output norm differences and layer importance scores for Llama3.2-3B-Instruct on GSM8K (a) and NQ-Open (b). The blue dashed line shows the average difference in attention output norms between hallucinated and non-hallucinated examples, i.e., Avg(∥Ou(l)(hallu)∥2−∥Ou(l)(non-hallu)∥2) , with error bars indicating standard error. The orange solid line shows layer importance scores from the hallucination probe (obtained from regularized logistic regression model, see Section 4.1 for details). Both datasets exhibit alignment in middle layers, where norm differences and importance scores peak together.
Figure 3: Average frequency that the k -th token–ranked by sink score–appears in the prompt, averaged across all attention heads for Llama 3.2-3B.
Dataset
GSM8K
HaluEvalQA
NQ-Open
SQuADv2
TriviaQA
TruthfulQA
UMWP
LLM
Feature
Llama3.2-3B
AttnScore
0.743 ± 0.049
0.737 ± 0.015
0.661 ± 0.027
0.637 ± 0.027
0.672 ± 0.013
0.667 ± 0.041
0.702 ± 0.017
AttnLogDet
0.819 ± 0.010
0.817 ± 0.012
0.723 ± 0.014
0.739 ± 0.009
0.789 ± 0.009
0.771 ± 0.048
0.809 ± 0.017
AttnEigval
0.802 ± 0.025
0.817 ± 0.014
0.732 ± 0.024
0.738 ± 0.027
0.798 ± 0.017
0.765 ± 0.030
0.704 ± 0.025
LapEigval
0.825 ± 0.009
0.837 ± 0.011
0.736 ± 0.012
0.753 ± 0.014
0.824 ± 0.017
0.748 ± 0.047
0.720 ± 0.010
LookbackLens
0.835 ± 0.017
0.836 ± 0.009
0.743 ± 0.020
0.765 ± 0.008
0.811 ± 0.015
0.741 ± 0.072
0.749 ± 0.020
Table 1: Results of hallucination detection experiments, the values represent mean and standard deviation of ROC-AUC scores over 5-fold cross-validation, Δ represents relative improvement of SinkProbe compared with second best or best performing method.
Total Features
Total Important
LLM
Dataset
Llama3.2-3B
GSM8K
6720
38 (1%)
NQ-Open
6720
204 (3%)
TruthfulQA
6720
58 (1%)
UMWP
6720
228 (3%)
Mistral-Nemo
GSM8K
12800
82 (1%)
Table 2: Number of non-zero coefficients in the Logistic Regression model trained with ℓ1 regularization, i.e., number of sink score features selected for hallucination detection.
Figure 4: Importance scores of attention sinks derived from an ℓ1 -regularized probe, aggregated across heads and sink indices per layer: Il=∑h∑i∣βi(l,h)∣ . Scores are plotted against relative layer depth to facilitate cross-model comparison.
Figure 5: Influence of the hyperparameter k . Varying the number of retained sink scores per head, we find that SinkProbe attains best or near-best performance even for small values of k . This suggests that hallucination-related signals are concentrated in a few dominant sinks and that sink-score features capture this signal more directly and robustly than existing attention- or spectral-based representations. The presented results are for Llama3.2-3B, for other models and datasets see Appendix C .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Split / Subset
# Examples
Source
NQ-Open ( Kwiatkowski et al., 2019 )
Validation
3,610
huggingface
TriviaQA † ( Joshi et al., 2017 )
Validation
7,983
huggingface
SQuADv2 † ( Rajpurkar et al., 2018 )
Dev ( rc.nocontext )
9,960
official website
HaluEvalQA ( Li et al., 2023 )
QA
10,000
official repository
TruthfulQA ( Lin et al., 2022 )
Generation
817
huggingface
GSM8k ( Cobbe et al., 2021 )
Test
1,319
huggingface
Appendix
Table 3: Detailed references for the datasets used in the experiments. ( † Preprocessed following Chen et al. (2024) ).
LLM
HuggingFace Repository
Llama3.2-3B ( AI@Meta, 2024 )
meta-llama/Llama-3.2-3B-Instruct
Llama3.1-8B ( AI@Meta, 2024 )
meta-llama/Llama-3.1-8B-Instruct
Mistral-Nemo ( Mistral AI Team and NVIDIA, 2024 )
mistralai/Mistral-Nemo-Instruct-2407
Phi3.5 ( Abdin and others, 2024 )
microsoft/Phi-3.5-mini-instruct
Appendix
Table 4: LLM details.
Dataset
LLM
#Invalid
#Valid Total
Valid Ratio
#Non-Hallucinated
#Hallucinated
Hallucination Ratio
TriviaQA
Llama3.1-8B
368
9,592
0.96
6,423
3,169
0.33
TriviaQA
Llama3.2-3B
343
9,617
0.97
5,092
4,525
0.47
TriviaQA
Mistral-Nemo
31
9,929
1.00
6,599
3,330
0.34
TriviaQA
Phi3.5
56
9,904
0.99
5,294
4,610
0.47
NQ-Open
Llama3.1-8B
731
2,879
0.80
1,056
1,823
0.63
NQ-Open
Llama3.2-3B
888
2,722
0.75
1,173
1,549
0.57
Appendix
Table 5: Statistics of the obtained hallucination datasets. #Valid - number of verifiable, valid answers, #Invalid - number of invalid answers (answers which exceeded allowed number of tokens or did not contain answer (judge model marked them unverifiable)), #Non-Hallucinated - number of answers among Valid which were not hallucinated, #Hallucinated - number of answers among Valid which were hallucinated, Hallucination Ratio - ratio of hallucinated answers to the total number of valid answers.
Dataset
GSM8K
HaluEvalQA
NQ-Open
SQuADv2
TriviaQA
TruthfulQA
UMWP
LLM
Feature
Llama3.2-3B
AttnEigval
100
100
100
100
100
50
100
LapEigval
100
100
100
100
100
100
50
SinkProbe
100
50
100
25
100
50
100
Phi3.5
AttnEigval
100
50
100
100
100
100
100
LapEigval
10
50
100
100
100
100
100
Appendix
Table 6: Values of top- k providing best quality
Figure 6: Hallucination detection performance (ROC-AUC) as a function of top- k across all models and datasets. Each subplot shows the ROC-AUC performance on 5-fold cross-validation for different values of k∈{1,2,3,4,5,10,25,50,100} and models. Columns represent different LLMs, rows represent different datasets.
Figure 7: Token-semantic characterization. Each subplot shows the 3 most frequent tokens in the prompt and response segments of the top-ranked attention sinks, grouped by hallucination label: non-hallucinated and hallucinated . Counts reflect the total number of tokens’ occurrences.
We propose a lightweight and single-pass uncertainty quantification method for detecting hallucinations in Large Language Models. The method uses attention matrices to estimate uncertainty without requiring repeated sampling or external models. Specifically, we measure the Kullback-Leibler divergence between each attention head's distribution and a uniform reference distribution, and use these features in a logistic regression probe. Across multiple datasets, task types, and model families, attention divergence is highly predictive of answer correctness and performs competitively with existing uncertainty estimation methods. We find that this signal is concentrated in middle layers and on factual tokens such as named entities and numbers, suggesting that attention dynamics provides an efficient and interpretable white-box signal of model uncertainty.
Large Language Models (LLMs) are powerful and widely adopted, but their practical impact is limited by the well-known hallucination phenomenon. While recent hallucination detection methods have made notable progress, we find most of them overlook the dynamic nature and underlying mechanisms of it. To address this gap, we propose HalluSAE, a phase transition-inspired framework that models hallucination as a critical shift in the model's latent dynamics. By modeling the generation process as a trajectory through a potential energy landscape, HalluSAE identifies critical transition zones and attributes factual errors to specific high-energy sparse features. Our approach consists of three stages: (1) Potential Energy Empowered Phase Zone Localization via sparse autoencoders and a geometric potential energy metric; (2) Hallucination-related Sparse Feature Attribution using contrastive logit attribution; and (3) Probing-based Causal Hallucination Detection through linear probes on disentangled features. Extensive experiments on Gemma-2-9B demonstrate that HalluSAE achieves state-of-the-art hallucination detection performance.
Boshui Chen, Zhaoxin Fan, Ke Wang +5
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, School of Artificial Intelligence, Beihang University, Beijing, China. · State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China. · Beijing Academy of Blockchain and Edge Computing, Beijing, China. +1
Large Language Models (LLMs) frequently exhibit hallucinations, presenting a major barrier to reliability in complex reasoning tasks. While traditional detection methods rely on output-based confidence metrics, these logits are often miscalibrated by modern alignment techniques. In this paper, we investigate the temporal volatility of internal attention mechanisms as an alternative diagnostic signal for hallucination that does not depend on output calibration. By introducing an unsupervised metric for attention dispersion, we show that epistemic uncertainty leaves a measurable trace within intermediate layers, where spikes in attention entropy are associated with reasoning breakdowns. We evaluate our approach on mathematical reasoning benchmarks (GSM8K and MATH-500) using the Qwen2.5 model family (1.5B and 3B parameters), finding statistically significant AUC improvements of up to +0.076 over output-based baselines across all tested conditions. These findings suggest that attention dispersion is a promising complement to traditional hallucination detection methods, requiring further investigation across broader model families and task domains.
Shardul P. More, Tanuja S. Pawar
Rajarambapu Institute of Technology Ishwarpur, Maharashtra, India