Embedding Graph Kernel Entropy (EGKE): Fine-grained Hallucination Detection for LLMs from Embeddings
Authors: Dilrangi Sankalpana, Pragatheeswaran Vipulanandan, Uthayasanker Thayasivam, Kamal Premaratne
Organizations: Department of Computer Science and Engineering, University of Moratuwa, Sri Lanka · Department of Electrical and Computer Engineering, University of Miami, Coral Gables, Florida, USA
Hallucination detection is critical for ensuring the reliability of Large Language Models (LLMs) in real-world deployments. While recent works, such as Semantic Entropy (SE) and Kernel Language Entropy (KLE), identify hallucinations by measuring semantic disagreement across multiple generated responses using Natural Language Inference (NLI) models, these methods depend on separately loaded NLI classifiers to estimate pairwise semantic similarity, introducing significant computational overhead. We propose Embedding Graph Kernel Entropy (EGKE), a variant of Kernel Language Entropy (KLE) that replaces its NLI-derived semantic graph with a Radial Basis Function (RBF) kernel applied to dense sentence embeddings generated by a BGE embedding model, while retaining KLE's graph Laplacian, heat kernel, and von Neumann entropy formulation unchanged. By computing similarities directly in a continuous embedding space, the proposed method provides a symmetric and computationally efficient alternative to NLI-based semantic comparison. We evaluate our approach across 30 experimental setups (covering multiple question-answer datasets and open-weight LLM families) using both AUROC and AUARC, and demonstrate that this RBF-based method achieves competitive hallucination detection performance against state-of-the-art methods.
Figures & tables
Fig. 1: Overview of the proposed Embedding Graph Kerenel Entropy pipeline. Responses generated by the LLM are embedded using BGE model, transformed into an RBF-weighted graph, and converted into a normalized heat kernel whose Von Neumann entropy is used as the semantic uncertainty estimate.
Fig. 2: Pairwise AUROC win-rate heatmap across 30 model–dataset settings. Each cell shows the fraction of settings in which the row method outperforms the column method.
Fig. 3: Pairwise AUARC win-rate heatmap across 30 model–dataset settings. Each cell shows the fraction of settings in which the row method outperforms the column method; ties are counted as 0.5.
σ
falcon-7b
falcon-7b-instruct
llama-2-7b
llama-2-7b-chat
mistral-7b-v0.1
mistral-7b-v0.1-instruct
0.1
0.770
0.738
0.814
0.792
0.819
0.833
0.2
0.732
0.714
0.788
0.779
0.795
0.808
0.3
0.716
0.704
0.776
0.775
0.782
0.795
0.4
0.710
0.700
0.772
0.773
0.776
0.790
0.5
0.707
0.698
0.770
0.772
0.773
0.788
0.6
0.706
0.697
0.769
0.772
0.771
0.787
TABLE I: Summary of selected hyperparameters after grid search.
Method
TriviaQA
SQuAD
NQ
SVAMP
BioASQ
AUROC
AUARC
AUROC
AUARC
AUROC
AUARC
AUROC
AUARC
AUROC
AUARC
SE
0.779 ± 0.02
0.691 ± 0.03
0.711 ± 0.03
0.344 ± 0.03
0.731 ± 0.03
0.467 ± 0.03
0.760 ± 0.03
0.467 ± 0.04
0.874 ± 0.02
0.655 ± 0.03
DSE
0.770 ± 0.02
0.682 ± 0.03
0.698 ± 0.03
0.328 ± 0.03
0.738 ± 0.03
0.454 ± 0.03
0.738 ± 0.03
0.439 ± 0.04
0.872 ± 0.02
0.636 ± 0.03
PE
0.738 ± 0.03
0.673 ± 0.03
0.725 ± 0.03
0.351 ± 0.03
0.751 ± 0.03
0.469 ± 0.03
0.740 ± 0.03
0.456 ± 0.04
0.804 ± 0.02
0.617 ± 0.03
P_true
0.576 ± 0.03
0.577 ± 0.03
0.546 ± 0.03
0.248 ± 0.03
0.566 ± 0.03
0.352 ± 0.03
0.560 ± 0.03
0.338 ± 0.03
0.714 ± 0.03
0.562 ± 0.03
ER
0.676 ± 0.03
0.645 ± 0.03
0.637 ± 0.03
0.305 ± 0.03
0.586 ± 0.03
0.371 ± 0.03
0.764 ± 0.03
0.449 ± 0.04
0.803 ± 0.02
0.607 ± 0.03
TABLE II: AUROC / AUARC on Falcon-7B across datasets. Best results per metric in bold .
Method
TriviaQA
SQuAD
NQ
SVAMP
BioASQ
AUROC
AUARC
AUROC
AUARC
AUROC
AUARC
AUROC
AUARC
AUROC
AUARC
SE
0.791 ± 0.02
0.549 ± 0.03
0.665 ± 0.04
0.251 ± 0.03
0.716 ± 0.03
0.369 ± 0.03
0.698 ± 0.04
0.279 ± 0.04
0.831 ± 0.02
0.499 ± 0.03
DSE
0.779 ± 0.02
0.532 ± 0.03
0.655 ± 0.04
0.241 ± 0.03
0.714 ± 0.03
0.351 ± 0.03
0.711 ± 0.04
0.267 ± 0.03
0.828 ± 0.02
0.479 ± 0.03
PE
0.750 ± 0.03
0.522 ± 0.03
0.665 ± 0.04
0.257 ± 0.03
0.671 ± 0.03
0.348 ± 0.03
0.732 ± 0.04
0.290 ± 0.03
0.655 ± 0.03
0.406 ± 0.03
P_true
0.544 ± 0.03
0.390 ± 0.03
0.404 ± 0.03
0.133 ± 0.02
0.554 ± 0.03
0.280 ± 0.03
0.592 ± 0.04
0.226 ± 0.03
0.598 ± 0.03
0.358 ± 0.03
ER
0.719 ± 0.03
0.490 ± 0.03
0.628 ± 0.04
0.232 ± 0.03
0.658 ± 0.03
0.325 ± 0.03
0.683 ± 0.04
0.258 ± 0.03
0.829 ± 0.02
0.507 ± 0.03
TABLE III: AUROC and AUARC on Falcon-7B-Instruct across datasets. Best results per metric in bold .
Method
TriviaQA
SQuAD
NQ
SVAMP
BioASQ
AUROC
AUARC
AUROC
AUARC
AUROC
AUARC
AUROC
AUARC
AUROC
AUARC
SE
0.784 ± 0.03
0.745 ± 0.02
0.785 ± 0.03
0.405 ± 0.03
0.775 ± 0.02
0.541 ± 0.03
0.878 ± 0.02
0.649 ± 0.03
0.852 ± 0.02
0.642 ± 0.03
DSE
0.778 ± 0.02
0.742 ± 0.02
0.798 ± 0.03
0.398 ± 0.03
0.776 ± 0.02
0.527 ± 0.03
0.870 ± 0.02
0.636 ± 0.03
0.848 ± 0.02
0.618 ± 0.03
PE
0.751 ± 0.03
0.730 ± 0.02
0.757 ± 0.03
0.408 ± 0.03
0.750 ± 0.03
0.533 ± 0.03
0.841 ± 0.02
0.632 ± 0.03
0.819 ± 0.02
0.620 ± 0.03
P_true
0.607 ± 0.03
0.668 ± 0.03
0.628 ± 0.03
0.331 ± 0.03
0.678 ± 0.03
0.480 ± 0.03
0.751 ± 0.03
0.555 ± 0.04
0.672 ± 0.03
0.521 ± 0.03
ER
0.619 ± 0.03
0.664 ± 0.03
0.573 ± 0.03
0.288 ± 0.03
0.569 ± 0.03
0.403 ± 0.03
0.758 ± 0.03
0.574 ± 0.03
0.748 ± 0.02
0.583 ± 0.03
TABLE IV: AUROC and AUARC on LLaMA-2-7B across datasets. Best results per column in bold .
Method
TriviaQA
SQuAD
NQ
SVAMP
BioASQ
AUROC
AUARC
AUROC
AUARC
AUROC
AUARC
AUROC
AUARC
AUROC
AUARC
SE
0.771 ± 0.03
0.698 ± 0.02
0.750 ± 0.03
0.360 ± 0.03
0.724 ± 0.03
0.514 ± 0.03
0.841 ± 0.02
0.590 ± 0.03
0.842 ± 0.02
0.640 ± 0.03
DSE
0.787 ± 0.02
0.704 ± 0.02
0.755 ± 0.03
0.356 ± 0.03
0.726 ± 0.03
0.510 ± 0.03
0.845 ± 0.02
0.588 ± 0.03
0.840 ± 0.02
0.632 ± 0.03
PE
0.737 ± 0.03
0.683 ± 0.03
0.719 ± 0.03
0.359 ± 0.03
0.715 ± 0.03
0.510 ± 0.03
0.835 ± 0.02
0.589 ± 0.03
0.678 ± 0.03
0.554 ± 0.03
P_true
0.694 ± 0.03
0.666 ± 0.03
0.598 ± 0.04
0.297 ± 0.03
0.647 ± 0.03
0.465 ± 0.03
0.573 ± 0.03
0.422 ± 0.04
0.790 ± 0.02
0.633 ± 0.03
ER
0.643 ± 0.03
0.635 ± 0.03
0.562 ± 0.03
0.266 ± 0.03
0.570 ± 0.03
0.412 ± 0.03
0.803 ± 0.03
0.575 ± 0.03
0.729 ± 0.03
0.569 ± 0.03
TABLE V: AUROC and AUARC on LLaMA-2-7B-Chat across datasets. Best results per column in bold .
Method
TriviaQA
SQuAD
NQ
SVAMP
BioASQ
AUROC
AUARC
AUROC
AUARC
AUROC
AUARC
AUROC
AUARC
AUROC
AUARC
SE
0.771 ± 0.03
0.761 ± 0.02
0.750 ± 0.03
0.444 ± 0.03
0.792 ± 0.02
0.607 ± 0.03
0.884 ± 0.02
0.776 ± 0.02
0.870 ± 0.02
0.721 ± 0.02
DSE
0.774 ± 0.02
0.760 ± 0.02
0.758 ± 0.03
0.426 ± 0.03
0.773 ± 0.02
0.598 ± 0.03
0.867 ± 0.02
0.761 ± 0.02
0.861 ± 0.02
0.707 ± 0.02
PE
0.766 ± 0.02
0.766 ± 0.02
0.749 ± 0.03
0.452 ± 0.03
0.740 ± 0.02
0.580 ± 0.03
0.824 ± 0.02
0.756 ± 0.02
0.742 ± 0.02
0.653 ± 0.03
P_true
0.714 ± 0.03
0.731 ± 0.03
0.715 ± 0.03
0.424 ± 0.03
0.715 ± 0.02
0.563 ± 0.03
0.745 ± 0.03
0.701 ± 0.03
0.734 ± 0.03
0.658 ± 0.03
ER
0.642 ± 0.03
0.698 ± 0.03
0.640 ± 0.03
0.371 ± 0.03
0.581 ± 0.03
0.466 ± 0.03
0.853 ± 0.02
0.769 ± 0.02
0.773 ± 0.02
0.674 ± 0.03
TABLE VI: AUROC and AUARC on Mistral-7B-v0.1 across datasets. Best results per column in bold .
Method
TriviaQA
SQuAD
NQ
SVAMP
BioASQ
AUROC
AUARC
AUROC
AUARC
AUROC
AUARC
AUROC
AUARC
AUROC
AUARC
SE
0.803 ± 0.02
0.676 ± 0.03
0.756 ± 0.03
0.367 ± 0.03
0.774 ± 0.03
0.457 ± 0.03
0.898 ± 0.02
0.730 ± 0.03
0.901 ± 0.02
0.666 ± 0.03
DSE
0.811 ± 0.02
0.676 ± 0.03
0.760 ± 0.03
0.361 ± 0.03
0.781 ± 0.02
0.441 ± 0.03
0.895 ± 0.02
0.721 ± 0.03
0.896 ± 0.02
0.645 ± 0.02
PE
0.784 ± 0.02
0.662 ± 0.03
0.748 ± 0.03
0.374 ± 0.03
0.740 ± 0.03
0.434 ± 0.03
0.854 ± 0.02
0.713 ± 0.03
0.740 ± 0.03
0.570 ± 0.03
P_true
0.742 ± 0.02
0.632 ± 0.03
0.608 ± 0.03
0.292 ± 0.03
0.706 ± 0.03
0.419 ± 0.03
0.760 ± 0.03
0.658 ± 0.03
0.736 ± 0.03
0.578 ± 0.03
ER
0.677 ± 0.03
0.605 ± 0.03
0.620 ± 0.03
0.301 ± 0.03
0.598 ± 0.03
0.333 ± 0.03
0.860 ± 0.02
0.710 ± 0.03
0.805 ± 0.02
0.605 ± 0.03
TABLE VII: AUROC and AUARC on Mistral-7B-v0.1-instruct across datasets. Best results per column in bold .
Hallucination in large language models (LLMs), defined as the generation of factually incorrect or unsupported content, remains a critical barrier to reliable deployment. We present BEACON (Behavioral Entropy Aggregation for Cross-model hallucination detectiON), a black-box hallucination detection framework that operates purely on model outputs without requiring access to internal representations or external knowledge bases. BEACON extracts a 31-dimensional feature vector from structured multi-pass generation, integrating NLI-based semantic entropy, embedding geometry, chain-of-thought consistency, and paraphrase stability signals. A gradient-boosted classifier trained on 7,617 labeled examples across seven benchmarks achieves 0.8123 +/- 0.0102 AUROC (95% CI: 0.7632-0.8251), outperforming standalone semantic entropy (+0.2298) and SelfCheckGPT-style consistency baselines (+0.2457). Feature importance analysis shows that hallucination is inherently multi-dimensional, requiring combined uncertainty signals. An efficient 5-call variant achieves 0.7795 AUROC, enabling practical deployment across black-box LLM APIs.
Naveen Bera, Pulijala Sai Nikhila, Kondaguduru Abhiram +6
Large Language Models (LLMs) are powerful and widely adopted, but their practical impact is limited by the well-known hallucination phenomenon. While recent hallucination detection methods have made notable progress, we find most of them overlook the dynamic nature and underlying mechanisms of it. To address this gap, we propose HalluSAE, a phase transition-inspired framework that models hallucination as a critical shift in the model's latent dynamics. By modeling the generation process as a trajectory through a potential energy landscape, HalluSAE identifies critical transition zones and attributes factual errors to specific high-energy sparse features. Our approach consists of three stages: (1) Potential Energy Empowered Phase Zone Localization via sparse autoencoders and a geometric potential energy metric; (2) Hallucination-related Sparse Feature Attribution using contrastive logit attribution; and (3) Probing-based Causal Hallucination Detection through linear probes on disentangled features. Extensive experiments on Gemma-2-9B demonstrate that HalluSAE achieves state-of-the-art hallucination detection performance.
Boshui Chen, Zhaoxin Fan, Ke Wang +5
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, School of Artificial Intelligence, Beihang University, Beijing, China. · State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China. · Beijing Academy of Blockchain and Edge Computing, Beijing, China. +1
Knowledge graph (KG) reasoning infers new knowledge from existing facts and is widely applied in question answering, recommendation, and decision support. With the rapid development of large language models (LLMs), LLM-based KG reasoning frameworks have become increasingly popular by leveraging retrieved KG information. However, hallucinations in LLMs remain a critical issue. Even when relevant KG knowledge is incorporated, models may still generate incorrect outputs, leading to misinformation and unreliable decisions. Existing hallucination detection methods either focus on LLM internal states or verify consistency with retrieved contexts, but both overlook the structural information in KGs, resulting in suboptimal performance. To address this gap, we propose LUCID, the first halLUcination deteCtIon method for LLM-based knowleDge graph reasoning frameworks. LUCID jointly leverages LLM attention scores, KG semantics, and structural information. Specifically, it extracts node and edge features from attention scores and semantic similarities, and integrates them with KG structure using a graph neural network. We also construct manually annotated benchmark datasets for evaluation. Experiments on nine datasets show that LUCID achieves state of the art performance compared to 15 baselines.
Xinyan Zhu, Yaoqi Liu, Yue Gao +3
Beijing University of Posts and Telecommunications · Tsinghua University