Semantic Uncertainty Quantification Needs Factual Equivalence

Joseph Hoche; Quentin Guimard; Gianni Franchi

cs.CLSep 28, 2026

Semantic Uncertainty Quantification Needs Factual Equivalence

Authors: Joseph Hoche, Quentin Guimard, Gianni Franchi

Organizations: AMIAD, Pˆole Recherche Palaiseau, France

Abstract

Semantic uncertainty quantification for large language models rests on a common template: sample several answers, measure how much they agree, and treat disagreement as uncertainty. We first formalize this template as two separate roles: an operator that compares two answers, and an aggregator that combines all pairwise comparisons into a scalar. Existing methods differ almost entirely in how they aggregate, while taking the operator off the shelf, typically an NLI model or a generic sentence encoder. We show that this reliance on off-the-shelf operators is the primary bottleneck of semantic UQ: they do not accurately measure factual equivalence of multiple answers to the same question. We resolve this with a deliberately simple recipe: a single encoder trained contrastively to isolate the targeted fact, utilizing synthetic data generated by an LLM and dataset both disjoint from all evaluation settings. Integrating the resulting operator into existing methods improves performance on 120 of 126 evaluation settings (95%) spanning 18 model dataset combinations across language and vision-language models. The best variant reaches 0.76 mean AUROC against 0.68 for the strongest baseline, while replacing the quadratic cross-encoder comparisons of entailment-based operators with one encoder pass per answer. The uniformity of the improvement supports the view that the operator, not the aggregator, is the limiting factor. The same operator also improves single generation token-level estimators: the norm it assigns to each token measures how much that token bears on the answer, and reweighting token log-likelihoods accordingly sharpens the estimate.

Figures & tables

Figure 1: The Operator Bottleneck in Semantic UQ. Semantic uncertainty methods decompose into a pairwise comparison operator ( $s$ ) and a structural aggregator ( $F$ ). While prior work focuses on designing sophisticated aggregators, they inherit off-the-shelf operators that suffer from misaligned training objectives or severe scaling bottlenecks. We identify the operator as the true bottleneck and introduce a contrastively trained, question-conditioned bi-encoder. Serving as a universal drop-in replacement, our learned operator universally improves the performance of existing aggregators at a fraction of the computational cost of NLI baselines.

Figure 2: Learning Semantic Geometry from Predictive Variability. (A) Mining Predictive Variability (Section 4.2 ): We extract positive and negative semantic triplets by evaluating repeated LLM generations for factual correctness, naturally capturing the model’s native syntax distribution. (B) Contrastive Factual Equivalence (Sections 4.1 and 4.3 ): A shared bi-encoder contextualizes the answers against the query. The contrastive margin separates hard negatives (incorrect answers to the same question) from valid paraphrases. (C) Zero-Shot Inference & UQ (Section 5 ): The frozen encoder generates symmetric $O(N)$ comparison matrices that drop seamlessly into existing aggregators, while its geometric norm enables zero-cost token reweighting.

	ADVQA		OKVQA		VizWiz		HotpotQA		TriviaQA		WebQuestions			post-hoc
Method	AUROC	ECE	AUROC	ECE	AUROC	ECE	AUROC	ECE	AUROC	ECE	AUROC	ECE	$\Delta$	compute
CAE	.637	.116	.643	.229	.647	.199	.673	.235	.583	.212	.490	.176		1.11 s
$\mathrm{CAE}_{\mathrm{Enc}}$	.663	.081	.733	.194	.674	.159	.722	.215	.840	.135	.716	.055	+.113	0.03 s
SE	.637	.114	.643	.228	.644	.198	.674	.235	.577	.212	.488	.175		1.18 s
$\mathrm{SE}_{\mathrm{Enc}}$	.666	.080	.733	.193	.675	.159	.725	.214	.840	.133	.719	.058	+.116	0.03 s
KLE-Heat	.675	.080	.678	.187	.690	.161	.719	.208	.747	.127	.555	.112		1.21 s

Table 1: Main results. AUROC (

\uparrow

) and ECE (

\downarrow

) for each estimator in its published form ( base ) and with our operator substituted ( shaded ), averaged over the 18 model–dataset combinations and reported by dataset (top) and by model (bottom).

\Delta

is the change in mean AUROC over all columns.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Correctness	Answer 1	Answer 2	Similarity
Both correct	The capital of France is Paris.	Paris is the capital of France.	.975
Both incorrect	The capital of France is Madrid.	Madrid is the capital of France.	.980

Appendix

Table B.1: Similarity assigned by our encoder to pairs of semantically equivalent answers, for correct and incorrect content. Question: “What is the capital of France?”

Backbone	AUROC
sentence-t5-large	.759
all-roberta-large-v1	.759
all-mpnet-base-v2	.756
multi-qa-mpnet-base-dot-v1	.755
bge-base-en-v1.5	.752
gtr-t5-large	.750

Appendix

Table B.2: The framework is insensitive to the backbone. Mean AUROC of

\mathrm{COS}_{\mathrm{Enc}}

over the 18 model–dataset combinations, for six sentence encoders used as the starting point of the training procedure of Sec. 4 , with all other settings held fixed.

	ADVQA	OKVQA	VizWiz	HotpotQA	TriviaQA	WebQuestions	Mean
hard negatives (ours)	.694	.758	.741	.765	.852	.741	.759
random negatives	.666	.690	.697	.738	.823	.618	.705
untrained encoder	.651	.673	.705	.707	.785	.554	.679
no negatives	.585	.616	.610	.687	.760	.640	.650
	idefics2-8B	Qwen2.5-VL-7B	llava-1.5-7B	Qwen2.5-7B	Phi-3.5-mini	Mistral-7B	Mean
hard negatives (ours)	.730	.740	.723	.790	.793	.775	.759

Appendix

Table B.3: Hard negatives matter, not just negatives. AUROC of

\mathrm{COS}_{\mathrm{Enc}}

for the untrained encoder and under three training regimes: no negatives (cosine-similarity loss on positive pairs only), random negatives (drawn from a different question), and hard negatives (drawn from the same question, used throughout this paper), averaged over models (top) and over datasets (bottom).

Margin	Learning rate	AUROC
0.10	$2\times 10^{-5}$	.765
0.15	$2\times 10^{-5}$	.758
0.20	$2\times 10^{-5}$	.759
0.30	$2\times 10^{-5}$	.757
0.50	$2\times 10^{-5}$	.749
0.20	$1\times 10^{-5}$	.759

Appendix

Table B.4: Small variations across hyperparameters. AUROC of

\mathrm{COS}_{\mathrm{Enc}}

, averaged over the 18 model–dataset combinations, when the margin and learning rate are varied around the configuration used throughout (shaded). All other settings are held fixed.

	AUROC	ECE
$\tau=0.3$	.656	.139
$\tau=0.4$	.693	.138
$\tau=0.5$	.724	.140
$\tau=0.6$	.743	.137
$\tau=0.7$	.751	.141

Appendix

Table B.5: Sensitivity to the clustering threshold. AUROC and ECE of

\mathrm{CAE}_{\mathrm{Enc}}

over the 18 model–dataset combinations, as the threshold

\tau

applied to

{\bm{S}}^{\mathrm{E}}

is varied. The value used throughout the paper is shaded.

HotpotQA
	Qwen2.5-7B		Phi-3.5-mini		Mistral-7B
Method	AUROC	ECE	AUROC	ECE	AUROC	ECE
CAE	.683 ±.070	.234 ±.050	.651 ±.021	.259 ±.014	.685 ±.016	.212 ±.012
$\mathrm{CAE}_{\mathrm{Enc}}$	.724 ±.067	.216 ±.051	.722 ±.019	.235 ±.014	.719 ±.015	.193 ±.012
SE	.684 ±.067	.234 ±.046	.653 ±.020	.258 ±.014	.686 ±.015	.212 ±.013
$\mathrm{SE}_{\mathrm{Enc}}$	.726 ±.064	.215 ±.050	.726 ±.021	.235 ±.014	.722 ±.016	.193 ±.012

Appendix

Table G.1: Per-cell results on the textual benchmarks. AUROC (

\uparrow

) and ECE (

\downarrow

) with bootstrapped 95% confidence intervals, for each estimator in its published form ( base ) and with our operator substituted ( shaded ), reported for each of the three text-only models.

	ADVQA	OKVQA	VizWiz	HotpotQA	TriviaQA	WebQuestions	Mean
with question	.694	.758	.741	.765	.852	.741	.759
answer only	.668	.735	.730	.759	.842	.719	.742
	idefics2-8B	Qwen2.5-VL-7B	llava-1.5-7B	Qwen2.5-7B	Phi-3.5-mini	Mistral-7B	Mean
with question	.730	.740	.723	.790	.793	.775	.759
answer only	.711	.725	.697	.776	.782	.761	.742

Appendix

Table G.2: Conditioning on the question helps in every setting. AUROC of

\mathrm{COS}_{\mathrm{Enc}}

with and without the question, by dataset (top) and by model (bottom). In the answer-only variant, the question is removed from both training and evaluation.

	Greedy		Majority vote
Method	AUROC	$\Delta$	AUROC	$\Delta$	Order preserved
CAE	.612		.629
$\mathrm{CAE}_{\mathrm{Enc}}$	.724	+.112	.759	+.130	✓
SE	.610		.626
$\mathrm{SE}_{\mathrm{Enc}}$	.726	+.116	.761	+.135	✓
KLE-Heat	.677		.706

Appendix

Table G.3: AUROC under greedy and majority-vote correctness labels. Scores are averaged over the 18 model–dataset combinations.

\Delta

is the AUROC gain of our variant over its base counterpart (over COS-ext for the COS block). The last column indicates whether the ranking within the block is the same under both labelling strategies. Best value per block in bold.

ADVQA
	idefics2-8B		Qwen2.5-VL-7B		llava-1.5-7B
Method	AUROC	ECE	AUROC	ECE	AUROC	ECE
CAE	.644 ±.021	.115 ±.018	.631 ±.024	.127 ±.020	.635 ±.022	.107 ±.019
$\mathrm{CAE}_{\mathrm{Enc}}$	.649 ±.022	.085 ±.019	.698 ±.021	.059 ±.019	.642 ±.022	.099 ±.020
SE	.646 ±.022	.111 ±.019	.629 ±.025	.129 ±.020	.636 ±.022	.103 ±.018
$\mathrm{SE}_{\mathrm{Enc}}$	.652 ±.023	.083 ±.020	.702 ±.022	.060 ±.019	.645 ±.023	.098 ±.020

Appendix

Table G.4: Per-cell results on the visual benchmarks. AUROC (

\uparrow

) and ECE (

\downarrow

) with bootstrapped 95% confidence intervals, for each estimator in its published form ( base ) and with our operator substituted ( shaded ), reported for each of the three vision-language models.

Explore similar work

Apr 18, 2026cs.AI

Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification

Large language models (LLMs) often produce confident yet incorrect responses, and uncertainty quantification is one potential solution to more robust usage. Recent works routinely rely on self-consistency to estimate aleatoric uncertainty (AU), yet this proxy collapses when models are overconfident and produce the same incorrect answer across samples. We analyze this regime and show that cross-model semantic disagreement is higher on incorrect answers precisely when AU is low. Motivated by this, we introduce an epistemic uncertainty (EU) term that operates in the black-box access setting: EU uses only generated text from a small, scale-matched ensemble and is computed as the gap between inter-model and intra-model sequence-semantic similarity. We then define total uncertainty (TU) as the sum of AU and EU. In a comprehensive study across five 7-9B instruction-tuned models and ten long-form tasks, TU improves ranking calibration and selective abstention relative to AU, and EU reliably flags confident failures where AU is low. We further characterize when EU is most useful via agreement and complementarity diagnostics.

Kimia Hamidieh, Veronika Thost, Walter Gerych +2

1MIT · MIT-IBM Watson AI Lab · 3WPI +1

Large Language Model Uncertainty Uncertainty Quantification Uncertainty Self-Consistency Disagreement

Ask PDF

Jul 18, 2026cs.AI

Beyond Semantic Equivalence: Logical Graphs for LLM Uncertainty Quantification

Large Language Models often produce confidently stated yet unreliable outputs, posing critical challenges for deployment in safety-sensitive applications. Existing uncertainty metrics such as semantic entropy capture agreement at the level of semantic equivalence, but largely ignore the logical relationships between distinct answers. As a result, they tend to overestimate uncertainty and falsely flag hallucinations in settings where generated responses are diverse in form yet logically compatible (e.g., differing only in granularity or specificity). We propose Logical Graph Uncertainty (LGU), a framework that explicitly models implication and incompatibility among answers. LGU aggregates probability mass along entailment chains onto the most specific hypotheses the answers support, measures the entropy of the resulting distribution, and penalizes mutual incompatibility among those hypotheses. Across multiple question-answering benchmarks and model families, LGU ranks first on average among existing uncertainty measures, with its largest gains---up to +7.1% AUROC and +3.5% AUARC over semantic entropy---on questions whose sampled answers are logically structured.

Yanni Dong, Minghua Liu, Meilin Zhu +2

Institute of Intelligent Software, Guangzhou, China · Institute of Software, Chinese Academy of Sciences, Beijing, China · University of Liverpool, Liverpool, UK

Large Language Model Uncertainty Semantic Entropy LLM Reasoning Strategies Entailment Uncertainty

Ask PDF

Feb 19, 2026cs.CL

Fine-Grained Uncertainty Quantification for Long-Form Language Model Outputs: A Comparative Study

Uncertainty quantification has emerged as an effective approach to closed-book hallucination detection for LLMs, but existing methods are largely designed for short-form outputs and do not generalize well to long-form generation. We introduce a taxonomy for fine-grained uncertainty quantification in long-form LLM outputs that distinguishes methods by design choices at three stages: response decomposition, unit-level scoring, and response-level aggregation. We formalize several families of consistency-based black-box scorers, providing generalizations and extensions of existing methods. We also introduce FactScore-STEM-Geo, a new 400-question long-form QA dataset spanning four categories across STEM and Geography. In our experiments across multiple LLMs and datasets, we find 1) claim-response entailment consistently performs better or on par with more complex claim-level scorers, 2) claim-level scoring generally yields better results than sentence-level scoring, and 3) uncertainty-aware decoding is highly effective for improving the factuality of long-form outputs. Our framework clarifies relationships between prior methods, enables apples-to-apples comparisons, and provides practical guidance for selecting components for fine-grained UQ.

Dylan Bouchard, Mohit Singh Chauhan, Viren Bajaj +1

CVS Health, Wellesley, MA

Large Language Model Uncertainty Citation Hallucination Detection Uncertainty Quantification Uncertainty Responses

Ask PDF