Embedding Semantic Entropy: NLI-Free Semantic Uncertainty Estimation with Geometry-Aware Clustering

Akshiya Raveendran; Praveenasarma Baskarakurukkal; Pragatheeswaran Vipulanandan; Kamal Premaratne; Uthayasanker Thayasivam

cs.LGNov 2, 2026

Embedding Semantic Entropy: NLI-Free Semantic Uncertainty Estimation with Geometry-Aware Clustering

Authors: Akshiya Raveendran, Praveenasarma Baskarakurukkal, Pragatheeswaran Vipulanandan, Kamal Premaratne, Uthayasanker Thayasivam

Organizations: University of Moratuwa, Sri Lanka · University of Miami, USA

Abstract

Large Language Models (LLMs) often generate fluent yet incorrect responses, making un- certainty estimation important for reliable deployment. Semantic Entropy (SE) estimates uncertainty by clustering responses based on semantic equivalence, but it relies on hard entailment-based clustering. We propose Embedding Semantic Entropy, which replaces entailment-based clustering with K-Medoids in semantic embedding space and uses dynamic cluster selection to account for intra-cluster compactness and inter-cluster separation. We evaluate the method on TriviaQA, SQuAD, NQ-Open, SVAMP, and BioASQ using Falcon- 7B-Instruct, LLaMA-2-7B, LLaMA-2-7B-Chat, and Gemma-7B. Results show competitive AUROC and AUARC performance, suggesting that NLI-based entailment clustering can be replaced with embedding-space clustering with limited degradation while improving the inspectability of semantic grouping. This matters because semantic uncertainty scores depend on underlying semantic groups; by making these groups explicit, compact, and separable in embedding space, Emb-SE turns uncertainty estimation from a black-box en- tailment decision into an inspectable clustering problem.

Figures & tables

Panel of Figure 1 — Figure 1: Flowchart of the proposed embedding-based dynamic clustering algorithm.

Panel of Figure 2 — Figure 2: Pairwise AUROC win-rate heatmap across 20 model–dataset settings. Each cell shows the fraction of settings in which the row method outperforms the column method.

Panel of Figure 3 — Figure 3: Pairwise AUARC win-rate heatmap across 20 model–dataset settings. Each cell shows the fraction of settings in which the row method outperforms the column method; ties are counted as 0.5.

Panel of Figure 4 — Figure 4: Pairwise average $\Delta$ AUROC across 20 model–dataset settings. Each cell shows the average AUROC difference between the row method and the column method, computed as $\mathrm{AUROC}_{\text{row}}-\mathrm{AUROC}_{\text{column}}$ . Positive values indicate that the row method performs better on average. Colors reuse the same red–white–blue scale as the win-rate heatmap, centered at zero average difference.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Group	Avg. selected value	Selected range
$\gamma$	NQ	0.39	0.23–0.50
$\gamma$	SQuAD	0.34	0.23–0.39
$\gamma$	SVAMP	0.28	0.05–0.46
$\gamma$	TriviaQA	0.25	0.14–0.44
$\gamma$	BioASQ	0.21	0.01–0.42
$\tau$	Gemma-7B	0.16	0.15–0.20

Appendix

Table 1: Summary of selected hyperparameters after 5-fold cross-validation. The selected range reports the minimum and maximum values chosen within each group, not the full grid-search range.

Method	TriviaQA		SQuAD		NQ		SVAMP		BioASQ
Method	AUROC	AUARC	AUROC	AUARC	AUROC	AUARC	AUROC	AUARC	AUROC	AUARC
SE	0.791 $\pm$ 0.02	0.549 $\pm$ 0.03	0.665 $\pm$ 0.04	0.251 $\pm$ 0.03	0.716 $\pm$ 0.03	0.369 $\pm$ 0.03	0.698 $\pm$ 0.04	0.279 $\pm$ 0.04	0.831 $\pm$ 0.02	0.499 $\pm$ 0.03
DSE	0.779 $\pm$ 0.02	0.532 $\pm$ 0.03	0.655 $\pm$ 0.04	0.241 $\pm$ 0.03	0.714 $\pm$ 0.03	0.351 $\pm$ 0.03	0.711 $\pm$ 0.04	0.267 $\pm$ 0.03	0.828 $\pm$ 0.02	0.479 $\pm$ 0.03
PE	0.750 $\pm$ 0.03	0.522 $\pm$ 0.03	0.665 $\pm$ 0.04	0.257 $\pm$ 0.03	0.671 $\pm$ 0.03	0.348 $\pm$ 0.03	0.732 $\pm$ 0.04	0.290 $\pm$ 0.03	0.655 $\pm$ 0.03	0.406 $\pm$ 0.03
$P_{\mathrm{true}}$	0.544 $\pm$ 0.03	0.390 $\pm$ 0.03	0.404 $\pm$ 0.03	0.133 $\pm$ 0.02	0.554 $\pm$ 0.03	0.280 $\pm$ 0.03	0.592 $\pm$ 0.04	0.226 $\pm$ 0.03	0.598 $\pm$ 0.03	0.358 $\pm$ 0.03
ER	0.719 $\pm$ 0.03	0.490 $\pm$ 0.03	0.628 $\pm$ 0.04	0.232 $\pm$ 0.03	0.658 $\pm$ 0.03	0.325 $\pm$ 0.03	0.683 $\pm$ 0.04	0.258 $\pm$ 0.03	0.829 $\pm$ 0.02	0.507 $\pm$ 0.03

Appendix

Table 2: Falcon-7B-Instruct AUROC and AUARC scores across datasets. Best results per column are shown in bold.

Method	TriviaQA		SQuAD		NQ		SVAMP		BioASQ
Method	AUROC	AUARC	AUROC	AUARC	AUROC	AUARC	AUROC	AUARC	AUROC	AUARC
SE	0.784 $\pm$ 0.03	0.745 $\pm$ 0.02	0.785 $\pm$ 0.03	0.405 $\pm$ 0.03	0.775 $\pm$ 0.02	0.541 $\pm$ 0.03	0.878 $\pm$ 0.02	0.649 $\pm$ 0.03	0.852 $\pm$ 0.02	0.642 $\pm$ 0.03
DSE	0.778 $\pm$ 0.02	0.742 $\pm$ 0.02	0.798 $\pm$ 0.03	0.398 $\pm$ 0.03	0.776 $\pm$ 0.02	0.527 $\pm$ 0.03	0.870 $\pm$ 0.02	0.636 $\pm$ 0.03	0.848 $\pm$ 0.02	0.618 $\pm$ 0.03
PE	0.751 $\pm$ 0.03	0.730 $\pm$ 0.02	0.757 $\pm$ 0.03	0.408 $\pm$ 0.03	0.750 $\pm$ 0.03	0.533 $\pm$ 0.03	0.841 $\pm$ 0.02	0.632 $\pm$ 0.03	0.819 $\pm$ 0.02	0.620 $\pm$ 0.03
$P_{\mathrm{true}}$	0.607 $\pm$ 0.03	0.668 $\pm$ 0.03	0.628 $\pm$ 0.03	0.331 $\pm$ 0.03	0.678 $\pm$ 0.03	0.480 $\pm$ 0.03	0.751 $\pm$ 0.03	0.555 $\pm$ 0.04	0.672 $\pm$ 0.03	0.521 $\pm$ 0.03
ER	0.619 $\pm$ 0.03	0.664 $\pm$ 0.03	0.573 $\pm$ 0.03	0.288 $\pm$ 0.03	0.569 $\pm$ 0.03	0.403 $\pm$ 0.03	0.758 $\pm$ 0.03	0.574 $\pm$ 0.03	0.748 $\pm$ 0.02	0.583 $\pm$ 0.03

Appendix

Table 3: Llama-2-7B AUROC and AUARC scores across datasets. Best results per column are shown in bold.

Method	TriviaQA		SQuAD		NQ		SVAMP		BioASQ
Method	AUROC	AUARC	AUROC	AUARC	AUROC	AUARC	AUROC	AUARC	AUROC	AUARC
SE	0.787 $\pm$ 0.02	0.731 $\pm$ 0.02	0.750 $\pm$ 0.03	0.360 $\pm$ 0.03	0.724 $\pm$ 0.03	0.514 $\pm$ 0.03	0.841 $\pm$ 0.02	0.590 $\pm$ 0.03	0.842 $\pm$ 0.02	0.640 $\pm$ 0.03
DSE	0.788 $\pm$ 0.02	0.730 $\pm$ 0.02	0.755 $\pm$ 0.03	0.356 $\pm$ 0.03	0.726 $\pm$ 0.03	0.510 $\pm$ 0.03	0.845 $\pm$ 0.02	0.588 $\pm$ 0.03	0.840 $\pm$ 0.02	0.632 $\pm$ 0.03
PE	0.758 $\pm$ 0.02	0.722 $\pm$ 0.02	0.719 $\pm$ 0.03	0.359 $\pm$ 0.03	0.715 $\pm$ 0.03	0.510 $\pm$ 0.03	0.835 $\pm$ 0.02	0.589 $\pm$ 0.03	0.678 $\pm$ 0.03	0.554 $\pm$ 0.03
$P_{\mathrm{true}}$	0.670 $\pm$ 0.03	0.662 $\pm$ 0.03	0.598 $\pm$ 0.04	0.297 $\pm$ 0.03	0.647 $\pm$ 0.03	0.465 $\pm$ 0.03	0.573 $\pm$ 0.03	0.422 $\pm$ 0.04	0.790 $\pm$ 0.02	0.633 $\pm$ 0.03
ER	0.599 $\pm$ 0.02	0.636 $\pm$ 0.03	0.562 $\pm$ 0.03	0.266 $\pm$ 0.03	0.570 $\pm$ 0.03	0.412 $\pm$ 0.03	0.803 $\pm$ 0.03	0.575 $\pm$ 0.03	0.729 $\pm$ 0.03	0.569 $\pm$ 0.03

Appendix

Table 4: Llama-2-7B-Chat AUROC and AUARC scores across datasets. Best results per column are shown in bold.

Method	TriviaQA		SQuAD		NQ		SVAMP		BioASQ
Method	AUROC	AUARC	AUROC	AUARC	AUROC	AUARC	AUROC	AUARC	AUROC	AUARC
SE	0.767 $\pm$ 0.02	0.705 $\pm$ 0.03	0.744 $\pm$ 0.03	0.364 $\pm$ 0.03	0.735 $\pm$ 0.03	0.528 $\pm$ 0.03	0.890 $\pm$ 0.02	0.763 $\pm$ 0.02	0.869 $\pm$ 0.02	0.702 $\pm$ 0.02
DSE	0.764 $\pm$ 0.02	0.695 $\pm$ 0.02	0.749 $\pm$ 0.03	0.350 $\pm$ 0.03	0.750 $\pm$ 0.03	0.522 $\pm$ 0.03	0.878 $\pm$ 0.02	0.749 $\pm$ 0.03	0.865 $\pm$ 0.02	0.683 $\pm$ 0.02
PE	0.744 $\pm$ 0.02	0.698 $\pm$ 0.02	0.737 $\pm$ 0.03	0.361 $\pm$ 0.03	0.725 $\pm$ 0.03	0.513 $\pm$ 0.03	0.858 $\pm$ 0.02	0.753 $\pm$ 0.03	0.825 $\pm$ 0.02	0.686 $\pm$ 0.02
$P_{\mathrm{true}}$	0.739 $\pm$ 0.03	0.690 $\pm$ 0.03	0.644 $\pm$ 0.03	0.328 $\pm$ 0.03	0.684 $\pm$ 0.03	0.501 $\pm$ 0.03	0.710 $\pm$ 0.03	0.649 $\pm$ 0.03	0.808 $\pm$ 0.02	0.681 $\pm$ 0.02
ER	0.647 $\pm$ 0.02	0.625 $\pm$ 0.03	0.575 $\pm$ 0.03	0.298 $\pm$ 0.03	0.598 $\pm$ 0.03	0.416 $\pm$ 0.03	0.627 $\pm$ 0.02	0.604 $\pm$ 0.03	0.784 $\pm$ 0.02	0.547 $\pm$ 0.02

Appendix

Table 5: Gemma-7B AUROC and AUARC scores across datasets. Best results per column are shown in bold.

LLM Uncertainty Estimation

Large language model uncertainty estimation measures when outputs may be wrong, ambiguous, or unreliable to guide safer decisions.

See all 408 papers

LLMs Uncertainty Quantification via Adaptive Conformal Semantic Entropy
May 5, 2026Hamed Karimi, Vaishali Meyappan, Reza SamaviSelective Prediction Semantic Entropy
Toronto Metropolitan University, Toronto, Ontario, Canada · Vector Institute, Toronto, Ontario, Canada
Semantic Token Clustering for Efficient Uncertainty Quantification in Large Language Models
Mar 20, 2026Qi Cao, Andrew Gambardella, Takeshi Kojima +2LLM Uncertainty Estimation LLM Reliability
The University of Tokyo, Japan
Semantic Uncertainty Quantification Needs Factual Equivalence
Sep 28, 2026Joseph Hoche, Quentin Guimard, Gianni FranchiConfidence Estimation in Language Models Factual Consistency Evaluation
AMIAD, Pˆole Recherche Palaiseau, France

May 5, 2026J/K move · Enter open · S save

LLMs Uncertainty Quantification via Adaptive Conformal Semantic Entropy

Hamed Karimi, Vaishali Meyappan, Reza Samavi

Toronto Metropolitan University, Toronto, Ontario, Canada · Vector Institute, Toronto, Ontario, Canada

Selective Prediction Semantic Entropy LLM Uncertainty Estimation Uncertainty Calibration Semantic Uncertainty Estimation

Open paper AskPDF arXiv

LLMs' overconfidence, particularly when hallucinating, poses a significant challenge for the deployment of the models in safety-critical settings and makes a reliable estimation of uncertainty necessary. Existing approaches for uncertainty quantification typically prioritize lexical or probabilistic measures; however, these techniques often ignore the semantic variance of different responses with similar meaning. In this paper, we propose Adaptive Conformal Semantic Entropy (ACSE), a method for estimating prompt-level uncertainty by adaptively measuring semantic dispersion in LLMs outputs. Our uncertainty scoring function is based on clustering semantic entropy of multiple diverse responses to the same prompt. The function adaptively adjusts the uncertainty score based on semantic features of each cluster. To ensure statistical reliability of our score, we use conformal calibration to apply a decision rule to accept/abstain the prompts, providing a finite-sample, distribution-free guarantee such that the error rate among the accepted responses remains bounded by a user-specified tolerance. Our extensive experimental evaluations using different LLMs and datasets, demonstrate that our approach consistently outperforms state-of-the-art uncertainty quantification baselines using discriminative performance, conformal guarantees, and probabilistic calibration indicators. As a highlight, for TriviaQA dataset, AUROC of our approach is 0.88 compared to 0.65 produced by the token entropy approach.

Abstract

LLM Uncertainty Estimation

Explore similar work

LLMs Uncertainty Quantification via Adaptive Conformal Semantic Entropy

Semantic Token Clustering for Efficient Uncertainty Quantification in Large Language Models

Semantic Uncertainty Quantification Needs Factual Equivalence