Claim-level Uncertainty Quantification (UQ) aims to mitigate the lack of reliability of Large Language Models (LLMs) by evaluating the factuality of each claim in their outputs. We introduce Kernel Token Contradiction (KTC), a lightweight approach to compute claim-level UQ under realistic white-box conditions. KTC represents the candidate tokens involved in LLM generation as a positive semi-definite kernel that integrates both the LLM's conditional distribution and a token contradiction score. We then use the Von Neumann entropy to quantify the uncertainty of this kernel. To estimate token contradiction, we develop a new approach based on frequency statistics from the Wikipedia corpus. Although CPU-only, our approach achieves over an 8.2x speedup compared to state-of-the-art GPU-accelerated methods based on cross-encoders, and over a 65x speedup compared to CPU-only methods with comparable performance. Our evaluation spans two benchmarks across four European languages and 16 different models. KTC not only matches the average performance of existing methods but also outperforms them in high-precision regimes. This combination of computational efficiency and accuracy makes real-time monitoring of LLM outputs practical in production.
Figures & tables
Figure 1: Example from the MUCH dataset [ 7 ] : only one of the nine claims in the LLM generation is incorrect. KTC correctly identifies the wrong price ($1,699), yielding a high uncertainty score for the incorrect claim, while CCP [ 9 ] produces several false positives and thus lower precision. Our method computes token-level scores that are aggregated at the claim level (see dashed lines on the plot).
Figure 2: The KTC score is computed for each LLM-generated token. We construct a non-contradiction graph over candidate tokens (step [A]) and represent it as a PSD semantic kernel (step [B]). This kernel is then combined with the predictive kernel containing candidate probabilities (step [C]). Finally, we compute the Von Neumann entropy (VNE) of the combined kernel (step [D]).
Figure 3: Illustration of KTC computation for three tokens from the example in Figure 1 . (top) candidate tokens and their likelihoods, (centre) the non-contradiction graph, (bottom) the entropy at different stages of the computation. (left) epistemic uncertainty, where the model hesitates between contradicting prices, resulting in a sparse non-contradiction graph and a high KTC score. (centre) aleatoric uncertainty, where the model hesitates between non-contradicting formulations, yielding high conditional entropy but low semantic entropy and thus a low KTC score. (right) low uncertainty, with low conditional entropy; despite high semantic entropy, scaling leads to a low KTC score.
Figure 4: Accuracy–efficiency trade-off for white-box claim-level UQ methods. Each point corresponds to a method evaluated under a different set of hyperparameters. The x -axis reports the UQ computation time as a fraction of the LLM generation cost. KTC achieves competitive accuracy at a fraction of the cost of NLI-based methods (CCP, SAR), while requiring no GPU.
Figure 5: Precision-Recall curves for claim-level hallucination detection across four languages, using product as token-to-claim aggregation method. Results are averaged over 16 LLMs across two benchmarks. KTC notably outperforms competing methods in the high-precision regime (low recall), while remaining competitive overall.
Figure 6: Precision–recall curves for KTC under four token-to-claim aggregation strategies (Mean, Maximum, Geometric mean, Product), averaged across four languages and 16 LLMs. Each method is evaluated over its full grid search space, and we report its best-performing configuration.
ν
SCALE
COMB
α=0.
α=0.25
α=0.5
α=0.75
α=1
3
0.471
0.402
0.398
0.410
0.432
0.452
4
0.476
0.404
0.400
0.412
0.433
0.452
5
0.472
0.395
0.395
0.409
0.431
0.452
6
0.467
0.391
0.392
0.407
0.431
0.452
8
0.457
0.379
0.383
0.401
0.428
0.452
Table 1: Precision–recall AUC of aggregation methods for varying neighbourhood size ν . SCALE consistently outperforms COMB across all configurations. Within COMB, performance peaks around α∈[0.5,0.75] , while both extremes ( α=0 and α=1 ) yield lower scores. This trend is stable across all ν , which supports the importance of coupling both semantic and predictive kernels.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: ROC curves for claim-level hallucination detection across four languages, using product for token-to-claim aggregation. Results are averaged over 16 LLMs across two benchmarks. Each method is evaluated over its full grid search space, and we report its best-performing configuration.
Figure 8: ROC curves for KTC under four token-to-claim aggregation strategies (Mean, Maximum, Geometric mean, Product), averaged over all four languages and 16 LLMs. Each method is evaluated over its full grid search space, and we report its best-performing configuration.
Pair type / category
Non-contradiction
Synonyms
0.860±0.152
Hypernyms
0.837±0.132
Semantically close, non-hypernym
0.574±0.234
European countries
0.475±0.146
Social media platforms
0.525±0.146
Months
0.238±0.153
Appendix
Table 2: Non-contradiction scores produced by the Wikipedia heuristic. Synonyms and hypernyms receive substantially higher scores than semantically close but non-hypernym concepts, indicating that the heuristic does not simply measure generic semantic similarity. Values are mean ± standard deviation over 100 pairs per category.
Method
Hyperparameter
Values
KTC
ν
{3,4,5,6,8}
τ
{0.1,0.2,0.3,0.4,0.5,0.7}
δ
24 (fixed)
SAR
σ
{3,5,8}
NLI model
Small, Medium, Large
CCP
σ
{3,5,8}
Appendix
Table 3: Hyperparameter configurations evaluated for each method.
Figure 9: Hyperparameter generalization. Each point corresponds to a method evaluated under a different set of hyperparameters. The x -axis reports the UQ computation time as a fraction of the LLM generation cost. KTC configurations are selected on the manually annotated data and evaluated without retuning on the 4,673 automatically annotated samples in MUCH, showing that the accuracy-efficiency trade-off generalizes to unseen data.
Method
Configuration
GPU time
CPU time
ROC-AUC
PR-AUC
KTC
ν=3,τ=0.2
–
0.0566
0.6795
0.7483
KTC
ν=3,τ=0.3
–
0.0566
0.6786
0.7590
KTC
ν=4,τ=0.2
–
0.0566
0.6787
0.7540
KTC
ν=4,τ=0.3
–
0.0566
0.6779
0.7808
KTC
ν=4,τ=0.4
–
0.0566
0.6725
0.7704
KTC
ν=5,τ=0.2
–
0.0566
0.6767
0.7515
Appendix
Table 4: Performance of the evaluated hyperparameter configurations on the base evaluation set (698 manually-annotated samples from MUCH [ 7 ] and Mu-SHROOM [ 32 ] ). Time is relative to LLM generation time.
Method
Configuration
GPU time
CPU time
ROC-AUC
PR-AUC
KTC
ν=3,τ=0.2
–
0.0566
0.7480
0.9354
KTC
ν=3,τ=0.3
–
0.0566
0.7503
0.9311
KTC
ν=4,τ=0.2
–
0.0566
0.7482
0.9349
KTC
ν=4,τ=0.3
–
0.0566
0.7524
0.9352
KTC
ν=4,τ=0.4
–
0.0566
0.7481
0.9162
KTC
ν=5,τ=0.2
–
0.0566
0.7476
0.9311
Appendix
Table 5: Performance on the test set consisting of the 4,673 automatically annotated samples from MUCH [ 7 ] . The KTC configurations are evaluated without retuning.
College of Computer Science and Software Engineering, Shenzhen University · Tsinghua University · Behavioral and Spatial AI Lab, Peking University & Tongji University