Attention Sinks as Internal Signals for Hallucination Detection in Large Language Models

Jakub Binkowski; Kamil Adamczewski; Tomasz Kajdanowicz

cs.CLApr 12, 2026

Attention Sinks as Internal Signals for Hallucination Detection in Large Language Models

Authors: Jakub Binkowski, Kamil Adamczewski, Tomasz Kajdanowicz

Organizations: Department of Artificial Intelligence, Wroclaw University of Science and Technology

Abstract

Large language models frequently exhibit hallucinations: fluent and confident outputs that are factually incorrect or unsupported by the input context. While recent hallucination detection methods have explored various features derived from attention maps, the underlying mechanisms they exploit remain poorly understood. In this work, we propose SinkProbe, a hallucination detection method grounded in the observation that hallucinations are deeply entangled with attention sinks - tokens that accumulate disproportionate attention mass during generation - indicating a transition from distributed, input-grounded attention to compressed, prior-dominated computation. Importantly, although sink scores are computed solely from attention maps, we find that the classifier preferentially relies on sinks whose associated value vectors have large norms. Moreover, we show that previous methods implicitly depend on attention sinks by establishing their mathematical relationship to sink scores. Our findings yield a novel hallucination detection method grounded in theory that produces state-of-the-art results across popular datasets and LLMs.

Figures & tables

Figure 1: Pipeline for hallucination detection based on attention sink scores. For each layer

l

and head

h

, we compute sink scores

\mathbf{s}^{(l,h)}

, defined as the average of attention scores directed toward each token position. These scores are then sorted, and the top-

k

values are selected as features. The selected features from all layers and heads are concatenated to form the final feature vector

\mathbf{z}

, which is passed to a hallucination probe – a binary classifier trained on labeled examples to distinguish hallucinated outputs from factually correct ones at inference time.

Panel of Figure 2 — Figure 2: Relationship between attention output norm differences and layer importance scores for Llama3.2-3B-Instruct on GSM8K (a) and NQ-Open (b). The blue dashed line shows the average difference in attention output norms between hallucinated and non-hallucinated examples, i.e., $\operatorname{Avg}\left(\|O_{u}^{(l)}(\text{hallu})\|_{2}-\|O_{u}^{(l)}(\text{non-hallu})\|_{2}\right)$ , with error bars indicating standard error. The orange solid line shows layer importance scores from the hallucination probe (obtained from regularized logistic regression model, see Section 4.1 for details). Both datasets exhibit alignment in middle layers, where norm differences and importance scores peak together.

Figure 3: Average frequency that the

k

-th token–ranked by sink score–appears in the prompt, averaged across all attention heads for Llama 3.2-3B.

	Dataset	GSM8K	HaluEvalQA	NQ-Open	SQuADv2	TriviaQA	TruthfulQA	UMWP
LLM	Feature
Llama3.2-3B	AttnScore	0.743 $\pm$ 0.049	0.737 $\pm$ 0.015	0.661 $\pm$ 0.027	0.637 $\pm$ 0.027	0.672 $\pm$ 0.013	0.667 $\pm$ 0.041	0.702 $\pm$ 0.017
	AttnLogDet	0.819 $\pm$ 0.010	0.817 $\pm$ 0.012	0.723 $\pm$ 0.014	0.739 $\pm$ 0.009	0.789 $\pm$ 0.009	0.771 $\pm$ 0.048	0.809 $\pm$ 0.017
	AttnEigval	0.802 $\pm$ 0.025	0.817 $\pm$ 0.014	0.732 $\pm$ 0.024	0.738 $\pm$ 0.027	0.798 $\pm$ 0.017	0.765 $\pm$ 0.030	0.704 $\pm$ 0.025
	LapEigval	0.825 $\pm$ 0.009	0.837 $\pm$ 0.011	0.736 $\pm$ 0.012	0.753 $\pm$ 0.014	0.824 $\pm$ 0.017	0.748 $\pm$ 0.047	0.720 $\pm$ 0.010
	LookbackLens	0.835 $\pm$ 0.017	0.836 $\pm$ 0.009	0.743 $\pm$ 0.020	0.765 $\pm$ 0.008	0.811 $\pm$ 0.015	0.741 $\pm$ 0.072	0.749 $\pm$ 0.020

Table 1: Results of hallucination detection experiments, the values represent mean and standard deviation of ROC-AUC scores over 5-fold cross-validation,

\Delta

represents relative improvement of SinkProbe compared with second best or best performing method.

		Total Features	Total Important
LLM	Dataset
Llama3.2-3B	GSM8K	6720	38 (1%)
	NQ-Open	6720	204 (3%)
	TruthfulQA	6720	58 (1%)
	UMWP	6720	228 (3%)
Mistral-Nemo	GSM8K	12800	82 (1%)

Table 2: Number of non-zero coefficients in the Logistic Regression model trained with

\ell_{1}

regularization, i.e., number of sink score features selected for hallucination detection.

Figure 4: Importance scores of attention sinks derived from an $\ell_{1}$ -regularized probe, aggregated across heads and sink indices per layer: $I_{l}=\sum_{h}\sum_{i}|\beta^{(l,h)}_{i}|$ . Scores are plotted against relative layer depth to facilitate cross-model comparison.

Panel of Figure 5 — Figure 5: Influence of the hyperparameter $k$ . Varying the number of retained sink scores per head, we find that SinkProbe attains best or near-best performance even for small values of $k$ . This suggests that hallucination-related signals are concentrated in a few dominant sinks and that sink-score features capture this signal more directly and robustly than existing attention- or spectral-based representations. The presented results are for Llama3.2-3B, for other models and datasets see Appendix C .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Split / Subset	# Examples	Source
NQ-Open ( Kwiatkowski et al., 2019 )	Validation	3,610	huggingface
TriviaQA † ( Joshi et al., 2017 )	Validation	7,983	huggingface
SQuADv2 † ( Rajpurkar et al., 2018 )	Dev ( rc.nocontext )	9,960	official website
HaluEvalQA ( Li et al., 2023 )	QA	10,000	official repository
TruthfulQA ( Lin et al., 2022 )	Generation	817	huggingface
GSM8k ( Cobbe et al., 2021 )	Test	1,319	huggingface

Appendix

Table 3: Detailed references for the datasets used in the experiments. ( † Preprocessed following Chen et al. (2024) ).

LLM	HuggingFace Repository
Llama3.2-3B ( AI@Meta, 2024 )	meta-llama/Llama-3.2-3B-Instruct
Llama3.1-8B ( AI@Meta, 2024 )	meta-llama/Llama-3.1-8B-Instruct
Mistral-Nemo ( Mistral AI Team and NVIDIA, 2024 )	mistralai/Mistral-Nemo-Instruct-2407
Phi3.5 ( Abdin and others, 2024 )	microsoft/Phi-3.5-mini-instruct

Appendix

Table 4: LLM details.

Dataset	LLM	#Invalid	#Valid Total	Valid Ratio	#Non-Hallucinated	#Hallucinated	Hallucination Ratio
TriviaQA	Llama3.1-8B	368	9,592	0.96	6,423	3,169	0.33
TriviaQA	Llama3.2-3B	343	9,617	0.97	5,092	4,525	0.47
TriviaQA	Mistral-Nemo	31	9,929	1.00	6,599	3,330	0.34
TriviaQA	Phi3.5	56	9,904	0.99	5,294	4,610	0.47
NQ-Open	Llama3.1-8B	731	2,879	0.80	1,056	1,823	0.63
NQ-Open	Llama3.2-3B	888	2,722	0.75	1,173	1,549	0.57

Appendix

Table 5: Statistics of the obtained hallucination datasets. #Valid - number of verifiable, valid answers, #Invalid - number of invalid answers (answers which exceeded allowed number of tokens or did not contain answer (judge model marked them unverifiable)), #Non-Hallucinated - number of answers among Valid which were not hallucinated, #Hallucinated - number of answers among Valid which were hallucinated, Hallucination Ratio - ratio of hallucinated answers to the total number of valid answers.

	Dataset	GSM8K	HaluEvalQA	NQ-Open	SQuADv2	TriviaQA	TruthfulQA	UMWP
LLM	Feature
Llama3.2-3B	AttnEigval	100	100	100	100	100	50	100
	LapEigval	100	100	100	100	100	100	50
	SinkProbe	100	50	100	25	100	50	100
Phi3.5	AttnEigval	100	50	100	100	100	100	100
	LapEigval	10	50	100	100	100	100	100

Appendix

Table 6: Values of top-

k

providing best quality

Figure 6: Hallucination detection performance (ROC-AUC) as a function of top-

k

across all models and datasets. Each subplot shows the ROC-AUC performance on 5-fold cross-validation for different values of

k\in\{1,2,3,4,5,10,25,50,100\}

and models. Columns represent different LLMs, rows represent different datasets.

Panel of Figure 7 — Figure 7: Token-semantic characterization. Each subplot shows the 3 most frequent tokens in the prompt and response segments of the top-ranked attention sinks, grouped by hallucination label: non-hallucinated and hallucinated . Counts reflect the total number of tokens’ occurrences.

LLM Hallucination Detection

Large language model hallucination detection identifies unsupported, incorrect, or fabricated claims in model outputs and traces.

See all 198 papers

Detecting Hallucinations in Large Language Models via Internal Attention Divergence Signals
May 6, 2026Gijs van DijkLanguage Model Error Detection Confidence Estimation in Language Models
Utrecht University
HalluSAE: Detecting Hallucinations in Large Language Models via Sparse Auto-Encoders
Apr 6, 2026Boshui Chen, Zhaoxin Fan, Ke Wang +5LLM Hallucination Detection
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, School of Artificial Intelligence, Beihang University, Beijing, China. · State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China. · Beijing Academy of Blockchain and Edge Computing, Beijing, China. +1
Attention Dispersion as a Diagnostic Signal for Hallucination in Large Language Models
Sep 16, 2026Shardul P. More, Tanuja S. PawarHallucination Detection LLM Hallucination Detection
Rajarambapu Institute of Technology Ishwarpur, Maharashtra, India

May 6, 2026J/K move · Enter open · S save

Detecting Hallucinations in Large Language Models via Internal Attention Divergence Signals

Gijs van Dijk

Utrecht University

Language Model Error Detection Confidence Estimation in Language Models LLM Interpretability Hallucination in Language Models LLM Reliability LLM Hallucination Detection+1

Open paper AskPDF arXiv

We propose a lightweight and single-pass uncertainty quantification method for detecting hallucinations in Large Language Models. The method uses attention matrices to estimate uncertainty without requiring repeated sampling or external models. Specifically, we measure the Kullback-Leibler divergence between each attention head's distribution and a uniform reference distribution, and use these features in a logistic regression probe. Across multiple datasets, task types, and model families, attention divergence is highly predictive of answer correctness and performs competitively with existing uncertainty estimation methods. We find that this signal is concentrated in middle layers and on factual tokens such as named entities and numbers, suggesting that attention dynamics provides an efficient and interpretable white-box signal of model uncertainty.

Abstract

LLM Hallucination Detection

Explore similar work

Detecting Hallucinations in Large Language Models via Internal Attention Divergence Signals

HalluSAE: Detecting Hallucinations in Large Language Models via Sparse Auto-Encoders

Attention Dispersion as a Diagnostic Signal for Hallucination in Large Language Models