Claim-level Uncertainty Quantification (UQ) aims to mitigate the lack of reliability of Large Language Models (LLMs) by evaluating the factuality of each claim in their outputs. We introduce Kernel Token Contradiction (KTC), a lightweight approach to compute claim-level UQ under realistic white-box conditions. KTC represents the candidate tokens involved in LLM generation as a positive semi-definite kernel that integrates both the LLM's conditional distribution and a token contradiction score. We then use the Von Neumann entropy to quantify the uncertainty of this kernel. To estimate token contradiction, we develop a new approach based on frequency statistics from the Wikipedia corpus. Although CPU-only, our approach achieves over an 8.2x speedup compared to state-of-the-art GPU-accelerated methods based on cross-encoders, and over a 65x speedup compared to CPU-only methods with comparable performance. Our evaluation spans two benchmarks across four European languages and 16 different models. KTC not only matches the average performance of existing methods but also outperforms them in high-precision regimes. This combination of computational efficiency and accuracy makes real-time monitoring of LLM outputs practical in production.
Figures & tables
Figure 1: Example from the MUCH dataset [ 7 ] : only one of the nine claims in the LLM generation is incorrect. KTC correctly identifies the wrong price ($1,699), yielding a high uncertainty score for the incorrect claim, while CCP [ 9 ] produces several false positives and thus lower precision. Our method computes token-level scores that are aggregated at the claim level (see dashed lines on the plot).
Figure 2: The KTC score is computed for each LLM-generated token. We construct a non-contradiction graph over candidate tokens (step [A]) and represent it as a PSD semantic kernel (step [B]). This kernel is then combined with the predictive kernel containing candidate probabilities (step [C]). Finally, we compute the Von Neumann entropy (VNE) of the combined kernel (step [D]).
Figure 3: Illustration of KTC computation for three tokens from the example in Figure 1 . (top) candidate tokens and their likelihoods, (centre) the non-contradiction graph, (bottom) the entropy at different stages of the computation. (left) epistemic uncertainty, where the model hesitates between contradicting prices, resulting in a sparse non-contradiction graph and a high KTC score. (centre) aleatoric uncertainty, where the model hesitates between non-contradicting formulations, yielding high conditional entropy but low semantic entropy and thus a low KTC score. (right) low uncertainty, with low conditional entropy; despite high semantic entropy, scaling leads to a low KTC score.
Figure 4: Accuracy–efficiency trade-off for white-box claim-level UQ methods. Each point corresponds to a method evaluated under a different set of hyperparameters. The x -axis reports the UQ computation time as a fraction of the LLM generation cost. KTC achieves competitive accuracy at a fraction of the cost of NLI-based methods (CCP, SAR), while requiring no GPU.
Figure 5: Precision-Recall curves for claim-level hallucination detection across four languages, using product as token-to-claim aggregation method. Results are averaged over 16 LLMs across two benchmarks. KTC notably outperforms competing methods in the high-precision regime (low recall), while remaining competitive overall.
Figure 6: Precision–recall curves for KTC under four token-to-claim aggregation strategies (Mean, Maximum, Geometric mean, Product), averaged across four languages and 16 LLMs. Each method is evaluated over its full grid search space, and we report its best-performing configuration.
ν
SCALE
COMB
α=0.
α=0.25
α=0.5
α=0.75
α=1
3
0.471
0.402
0.398
0.410
0.432
0.452
4
0.476
0.404
0.400
0.412
0.433
0.452
5
0.472
0.395
0.395
0.409
0.431
0.452
6
0.467
0.391
0.392
0.407
0.431
0.452
8
0.457
0.379
0.383
0.401
0.428
0.452
Table 1: Precision–recall AUC of aggregation methods for varying neighbourhood size ν . SCALE consistently outperforms COMB across all configurations. Within COMB, performance peaks around α∈[0.5,0.75] , while both extremes ( α=0 and α=1 ) yield lower scores. This trend is stable across all ν , which supports the importance of coupling both semantic and predictive kernels.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: ROC curves for claim-level hallucination detection across four languages, using product for token-to-claim aggregation. Results are averaged over 16 LLMs across two benchmarks. Each method is evaluated over its full grid search space, and we report its best-performing configuration.
Figure 8: ROC curves for KTC under four token-to-claim aggregation strategies (Mean, Maximum, Geometric mean, Product), averaged over all four languages and 16 LLMs. Each method is evaluated over its full grid search space, and we report its best-performing configuration.
Pair type / category
Non-contradiction
Synonyms
0.860±0.152
Hypernyms
0.837±0.132
Semantically close, non-hypernym
0.574±0.234
European countries
0.475±0.146
Social media platforms
0.525±0.146
Months
0.238±0.153
Appendix
Table 2: Non-contradiction scores produced by the Wikipedia heuristic. Synonyms and hypernyms receive substantially higher scores than semantically close but non-hypernym concepts, indicating that the heuristic does not simply measure generic semantic similarity. Values are mean ± standard deviation over 100 pairs per category.
Method
Hyperparameter
Values
KTC
ν
{3,4,5,6,8}
τ
{0.1,0.2,0.3,0.4,0.5,0.7}
δ
24 (fixed)
SAR
σ
{3,5,8}
NLI model
Small, Medium, Large
CCP
σ
{3,5,8}
Appendix
Table 3: Hyperparameter configurations evaluated for each method.
Figure 9: Hyperparameter generalization. Each point corresponds to a method evaluated under a different set of hyperparameters. The x -axis reports the UQ computation time as a fraction of the LLM generation cost. KTC configurations are selected on the manually annotated data and evaluated without retuning on the 4,673 automatically annotated samples in MUCH, showing that the accuracy-efficiency trade-off generalizes to unseen data.
Method
Configuration
GPU time
CPU time
ROC-AUC
PR-AUC
KTC
ν=3,τ=0.2
–
0.0566
0.6795
0.7483
KTC
ν=3,τ=0.3
–
0.0566
0.6786
0.7590
KTC
ν=4,τ=0.2
–
0.0566
0.6787
0.7540
KTC
ν=4,τ=0.3
–
0.0566
0.6779
0.7808
KTC
ν=4,τ=0.4
–
0.0566
0.6725
0.7704
KTC
ν=5,τ=0.2
–
0.0566
0.6767
0.7515
Appendix
Table 4: Performance of the evaluated hyperparameter configurations on the base evaluation set (698 manually-annotated samples from MUCH [ 7 ] and Mu-SHROOM [ 32 ] ). Time is relative to LLM generation time.
Method
Configuration
GPU time
CPU time
ROC-AUC
PR-AUC
KTC
ν=3,τ=0.2
–
0.0566
0.7480
0.9354
KTC
ν=3,τ=0.3
–
0.0566
0.7503
0.9311
KTC
ν=4,τ=0.2
–
0.0566
0.7482
0.9349
KTC
ν=4,τ=0.3
–
0.0566
0.7524
0.9352
KTC
ν=4,τ=0.4
–
0.0566
0.7481
0.9162
KTC
ν=5,τ=0.2
–
0.0566
0.7476
0.9311
Appendix
Table 5: Performance on the test set consisting of the 4,673 automatically annotated samples from MUCH [ 7 ] . The KTC configurations are evaluated without retuning.
Large language models (LLMs) have demonstrated remarkable capabilities across diverse tasks. However, the truthfulness of their outputs is not guaranteed, and their tendency toward overconfidence further limits reliability. Uncertainty quantification offers a promising way to identify potentially unreliable outputs, but most existing methods rely on repeated sampling or auxiliary models, introducing substantial computational overhead. To address these limitations, we propose Semantic Token Clustering (STC), an efficient uncertainty quantification method that leverages the semantic information inherently encoded in LLMs. Specifically, we group tokens into semantically consistent clusters using embedding clustering and prefix matching, and quantify uncertainty based on the probability mass aggregated over the corresponding semantic cluster. Our approach requires only a single generation and does not depend on auxiliary models. Experimental results show that STC achieves performance comparable to state-of-the-art baselines while substantially reducing computational overhead.
Uncertainty Quantification (UQ) is widely regarded as the primary safeguard for deploying Large Language Models (LLMs) in high-stakes domains. However, we argue that the field suffers from a category error: mainstream UQ methods for LLMs are just unsupervised clustering algorithms. We demonstrate that most current approaches inherently quantify the internal consistency of the model's generations rather than their external correctness. Consequently, current methods are fundamentally blind to factual reality and fail to detect ``confident hallucinations,'' where models exhibit high confidence in stable but incorrect answers. Therefore, the current UQ methods may create a deceptive sense of safety when deploying the models with uncertainty. In detail, we identify three critical pathologies resulting from this dependence on internal state: a hyperparameter sensitivity crisis that renders deployment unsafe, an internal evaluation cycle that conflates stability with truth, and a fundamental lack of ground truth that forces reliance on unstable proxy metrics to evaluate uncertainty. To resolve this impasse, we advocate for a paradigm shift to UQ and outline a roadmap for the research community to adopt better evaluation metrics and settings, implement mechanism changes for native uncertainty, and anchor verification in objective truth, ensuring that model confidence serves as a reliable proxy for reality.
Tiejin Chen, Longchao Da, Xiaoou Liu +1
School of Computing and Augmented Intelligence, Arizona State University
As the chain-of-thought reasoning capabilities of large language models improve, evaluating and calibrating their reasoning confidence is becoming increasingly important for quantifying the uncertainty of their answers. Current methods for estimating the confidence of large language models are generally based on probabilities of selected key tokens, but the underlying mechanism remains unclear. Our pilot study finds that replacing selected token probabilities with coarse substitutes can also improve calibration, motivating us to further explore effective signals of model confidence. We introduce Divergent Token Confidence (DTC), a framework that estimates confidence by counting tokens at which two models strongly disagree during decoding. DTC identifies these divergent tokens using the Jensen-Shannon divergence between next-token distributions evaluated along the same reasoning trajectory. We find that their count is almost negatively associated with answer accuracy, thereby serving as a simple yet effective signal for uncertainty quantification. DTC supports both white-box and black-box evaluation using auxiliary models, without explicit training and affecting the generation process. Experiments across multiple model families and six mathematical benchmarks demonstrate improved calibration over probability-based and verbalized baselines. Under white-box evaluation, the count-only estimator achieves an average expected calibration error of 13.0%, compared with 32.7%-42.4% for standard full-sequence confidence methods. In black-box settings, it also improves calibration over the original verbalized scores. For example, mean expected calibration error falls from 32.1%-40.2% to 13.7%-16.3% on DeepSeek-V3.2. These findings provide new insights for improving reasoning uncertainty quantification in large language models. The code is released at https://github.com/szu-tera/DTC.git.
Feiyang Li, Shengjing Liu, Qi Zhan +7
College of Computer Science and Software Engineering, Shenzhen University · Tsinghua University · Behavioral and Spatial AI Lab, Peking University & Tongji University