Improving Semantic Uncertainty Quantification in Language Model Question-Answering via Token-Level Temperature Scaling

Tom A. Lamb; Desi R. Ivanova; Philip H. S. Torr; Tim G. J. Rudner

cs.LGApr 8, 2026

Improving Semantic Uncertainty Quantification in Language Model Question-Answering via Token-Level Temperature Scaling

Authors: Tom A. Lamb, Desi R. Ivanova, Philip H. S. Torr, Tim G. J. Rudner

Organizations: University of Oxford · University of Toronto & Vijil

Abstract

Calibration is central to reliable semantic uncertainty quantification, yet prior work has largely focused on discrimination, neglecting calibration. As calibration and discrimination capture distinct aspects of uncertainty, focusing on discrimination alone yields an incomplete picture. We address this gap by systematically evaluating both aspects across a broad set of confidence measures. We show that current approaches, particularly fixed-temperature heuristics, produce systematically miscalibrated and poorly discriminative semantic confidence distributions. We demonstrate that optimising a single scalar temperature, which, we argue, provides a suitable inductive bias, is a surprisingly simple yet effective solution. Our exhaustive evaluation confirms that temperature scaling consistently improves semantic calibration, discrimination, and downstream entropy, outperforming both heuristic baselines and more expressive token-level recalibration methods on question-answering tasks.

Figures & tables

Figure 1: Temperature Scaling Improves Semantic Uncertainty Quantification . Same model using different temperatures generates 10 responses for the same input, which are clustered into semantic groups. We compute the L-SC semantic confidence measure of Kuhn et al. (2023) defined in Equation 3 . Panel (a) uses the standard temperature of $0.5$ ; panel (b) uses a temperature optimised on a calibration set. We find that optimised temperature scaling improves both semantic calibration and discrimination.

Figure 2: Semantic Confidence Measures. Figure showing how existing ( Farquhar et al., 2024 , E-SC;) and ( Kuhn et al., 2023 , L-SC;) ) as well as a broad set of novel baseline semantic measures (ML-SC, B-SC, T-SC, IC-SC, and G-SC) are computed. For an input $\bm{x}$ , we sample multiple responses from the model $p(\cdot\mid\bm{x})$ , and use an NLI model to assess bidirectional entailment, determining whether responses $\bm{y}^{i}$ and $\bm{y}^{j}$ are semantically equivalent ( $\bm{y}^{i}\sim\bm{y}^{j}\mid\bm{x}$ ). Here, $s(C\mid\bm{x})$ denotes the sum, and $\bar{s}(C\mid\bm{x})$ the average, $\mathcal{H}(p_{C_{i}})$ the entropy, and $\mathcal{E}(C_{i}\mid\bm{x})$ the entropy of the length-normalised log-likelihoods of generations within cluster $C$ . See Section 3 for details.

Figure 3: Uncertainty Metrics of SC Measures Across Methods. Mean and standard error of

\widehat{\text{ACE}}

(

\downarrow

) and AUROC (

\uparrow

) scores for SC measures across baseline, calibration methods, and datasets. Closer to top left of plots indicates better discrimination and calibration, and hence better overall semantic uncertainty quantification.

Table 1: Discrimination Comparison of Entropy for Qwen. Mean and standard error of AUROC (

\uparrow

) values. (a) reports

\text{SE}_{\text{conf}}

, where correctness is determined by the most confident semantic cluster under a given SC measure. (b) reports

\text{SE}_{\text{vanilla}}

from Kuhn et al. (2023) , where correctness is determined via greedy decoding. Bold entries denote the best result within each SC measure per dataset, and underlined entries indicate the best overall per dataset.

Figure 4: Selective Accuracy for Qwen Across Rejection Rates and Datasets. Mean selective accuracy (

\uparrow

) curves, comparing E-SC, L-SC, B-SC and G-SC measures across baseline and calibration methods. Error bars show one standard error across runs.

Figure 5: SC Measures over Varying Numbers of Samples for Llama. Mean

\widehat{\text{ACE }}

(

\downarrow

) and AUROC (

\uparrow

) on the NQ dataset over varying number of sample generations per example. Error bars show one standard error across runs.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Split	TriviaQA	Natural Questions	SQuAD
Calibration	Training	59374	61600	59577
Calibration	Validation	2000	2000	2000
Final Evaluation	Test	2000	2000	2000

Appendix

Table 2: Dataset Split Sizes. For each dataset, we report the number of examples used at different stages of the calibration pipeline: calibration training, calibration validation, and final test evaluation. Split sizes are matched across datasets to ensure fair comparison.

Figure 6: Training and Evaluation Prompts. Prompts applied for both calibration and evaluation across all methods, shown separately for (a) closed-book datasets (TriviaQA and Natural Questions) and (b) open-book datasets (SQuAD).

Model	TriviaQA	Natural Questions	SQuAD
Llama	71.9	42.5	95.5
Qwen	53.0	33.2	94.4
Mistral	64.8	33.2	94.2

Appendix

Table 3: Base Model Accuracies (%). Test-set accuracies (

\uparrow

) of the base models through beam search.

Panel of Figure 7 — Figure 7: Comparison of Uncertainty Metrics of Semantic Measures Across Calibration Methods Using SS and NLL Losses. Mean and standard deviation across four inference runs of $\widehat{\text{ACE}}$ ( $\downarrow$ ) and AUROC ( $\uparrow$ ) scores for semantic confidence measures for each model across baselines and calibration methods. We compare using a (a) NLL and (b) SS calibration loss.

		$\widehat{\textbf{ACE}}$							AUROC
		E-SC	L-SC	ML-SC	B-SC	IC-SC	T-SC	G-SC	E-SC	L-SC	ML-SC	B-SC	IC-SC	T-SC	G-SC
TriviaQA	Base	$0.184_{{\pm 0.002}}$	$0.174_{{\pm 0.002}}$	$0.180_{{\pm 0.001}}$	$0.130_{{\pm 0.001}}$	$0.191_{{\pm 0.002}}$	$0.130_{{\pm 0.002}}$	$0.172_{{\pm 0.002}}$	$0.757_{{\pm 0.001}}$	$0.755_{{\pm 0.001}}$	$0.757_{{\pm 0.001}}$	$0.759_{{\pm 0.001}}$	$0.752_{{\pm 0.001}}$	$0.760_{{\pm 0.001}}$	$0.758_{{\pm 0.002}}$
	SE	$0.070_{{\pm 0.003}}$	$0.056_{{\pm 0.001}}$	$0.064_{{\pm 0.003}}$	$0.082_{{\pm 0.002}}$	$0.092_{{\pm 0.002}}$	$0.082_{{\pm 0.002}}$	$0.066_{{\pm 0.003}}$	$0.832_{{\pm 0.003}}$	$0.832_{{\pm 0.002}}$	$0.832_{{\pm 0.003}}$	$0.831_{{\pm 0.002}}$	$0.829_{{\pm 0.003}}$	$0.830_{{\pm 0.002}}$	$0.833_{{\pm 0.003}}$
	TS	$\mathbf{0.049_{{\pm 0.001}}}$	$\mathbf{0.055_{{\pm 0.001}}}$	$\mathbf{0.061_{{\pm 0.001}}}$	$0.104_{{\pm 0.004}}$	$\mathbf{0.033_{{\pm 0.003}}}$	$0.106_{{\pm 0.002}}$	$\mathbf{0.056_{{\pm 0.004}}}$	$\mathbf{0.852_{{\pm 0.003}}}$	$\mathbf{0.849_{{\pm 0.004}}}$	$\mathbf{0.853_{{\pm 0.005}}}$	$\mathbf{0.846_{{\pm 0.003}}}$	$\mathbf{0.858_{{\pm 0.005}}}$	$\mathbf{0.847_{{\pm 0.004}}}$	$\mathbf{0.853_{{\pm 0.003}}}$
	Platt	$0.078_{{\pm 0.001}}$	$0.060_{{\pm 0.002}}$	$0.071_{{\pm 0.002}}$	$0.078_{{\pm 0.002}}$	$0.097_{{\pm 0.002}}$	$0.077_{{\pm 0.001}}$	$0.066_{{\pm 0.003}}$	$0.840_{{\pm 0.002}}$	$0.833_{{\pm 0.002}}$	$0.842_{{\pm 0.001}}$	$0.841_{{\pm 0.002}}$	$0.837_{{\pm 0.002}}$	$0.842_{{\pm 0.002}}$	$0.840_{{\pm 0.002}}$
	ATS	$0.084_{{\pm 0.001}}$	$0.061_{{\pm 0.002}}$	$0.069_{{\pm 0.002}}$	$\mathbf{0.070_{{\pm 0.004}}}$	$0.093_{{\pm 0.003}}$	$\mathbf{0.069_{{\pm 0.005}}}$	$0.070_{{\pm 0.002}}$	$0.843_{{\pm 0.001}}$	$0.832_{{\pm 0.002}}$	$0.838_{{\pm 0.002}}$	$0.835_{{\pm 0.001}}$	$0.831_{{\pm 0.003}}$	$0.836_{{\pm 0.001}}$	$0.844_{{\pm 0.001}}$

Appendix

Table 4: Uncertainty Metrics for SC Measures Across Methods for Llama. Mean and standard error of

\widehat{\text{ACE}}

(

\downarrow

) and AUROC (

\uparrow

) scores for SC measures across measures baseline and recalibration methods for Llama.

		$\widehat{\textbf{ACE}}$							AUROC
		E-SC	L-SC	ML-SC	B-SC	IC-SC	T-SC	G-SC	E-SC	L-SC	ML-SC	B-SC	IC-SC	T-SC	G-SC
TriviaQA	Base	$0.308_{{\pm 0.001}}$	$0.299_{{\pm 0.001}}$	$0.304_{{\pm 0.000}}$	$0.240_{{\pm 0.000}}$	$0.320_{{\pm 0.001}}$	$0.240_{{\pm 0.000}}$	$0.293_{{\pm 0.001}}$	$0.763_{{\pm 0.002}}$	$0.759_{{\pm 0.002}}$	$0.762_{{\pm 0.002}}$	$0.765_{{\pm 0.001}}$	$0.761_{{\pm 0.002}}$	$0.766_{{\pm 0.001}}$	$0.764_{{\pm 0.002}}$
	SE	$0.165_{{\pm 0.002}}$	$0.159_{{\pm 0.001}}$	$0.158_{{\pm 0.001}}$	$0.097_{{\pm 0.002}}$	$0.194_{{\pm 0.001}}$	$0.099_{{\pm 0.002}}$	$0.145_{{\pm 0.002}}$	$0.832_{{\pm 0.002}}$	$0.831_{{\pm 0.001}}$	$0.830_{{\pm 0.002}}$	$0.836_{{\pm 0.002}}$	$0.825_{{\pm 0.002}}$	$0.837_{{\pm 0.002}}$	$0.833_{{\pm 0.002}}$
	TS	$\mathbf{0.080_{{\pm 0.001}}}$	$\mathbf{0.066_{{\pm 0.005}}}$	$\mathbf{0.069_{{\pm 0.002}}}$	$\mathbf{0.082_{{\pm 0.003}}}$	$\mathbf{0.085_{{\pm 0.004}}}$	$\mathbf{0.081_{{\pm 0.003}}}$	$\mathbf{0.067_{{\pm 0.002}}}$	$\mathbf{0.855_{{\pm 0.002}}}$	$\mathbf{0.870_{{\pm 0.002}}}$	$\mathbf{0.853_{{\pm 0.002}}}$	$\mathbf{0.856_{{\pm 0.000}}}$	$\mathbf{0.864_{{\pm 0.004}}}$	$\mathbf{0.855_{{\pm 0.001}}}$	$\mathbf{0.855_{{\pm 0.002}}}$
	Platt	$0.170_{{\pm 0.001}}$	$0.168_{{\pm 0.002}}$	$0.163_{{\pm 0.002}}$	$0.100_{{\pm 0.002}}$	$0.203_{{\pm 0.001}}$	$0.101_{{\pm 0.003}}$	$0.150_{{\pm 0.001}}$	$0.837_{{\pm 0.002}}$	$0.839_{{\pm 0.001}}$	$0.834_{{\pm 0.001}}$	$0.838_{{\pm 0.001}}$	$0.833_{{\pm 0.001}}$	$0.839_{{\pm 0.001}}$	$0.838_{{\pm 0.002}}$
	ATS	$0.204_{{\pm 0.003}}$	$0.168_{{\pm 0.002}}$	$0.178_{{\pm 0.002}}$	$0.132_{{\pm 0.003}}$	$0.219_{{\pm 0.001}}$	$0.133_{{\pm 0.002}}$	$0.183_{{\pm 0.003}}$	$0.841_{{\pm 0.001}}$	$0.839_{{\pm 0.001}}$	$0.840_{{\pm 0.002}}$	$0.842_{{\pm 0.001}}$	$0.833_{{\pm 0.002}}$	$0.843_{{\pm 0.001}}$	$0.842_{{\pm 0.001}}$

Appendix

Table 5: Uncertainty Metrics for SC Measures Across Methods for Qwen. Mean and standard error of

\widehat{\text{ACE}}

(

\downarrow

) and AUROC (

\uparrow

) scores for SC measures across measures baseline and recalibration methods for Qwen.

		$\widehat{\textbf{ACE}}$							AUROC
		E-SC	L-SC	ML-SC	B-SC	IC-SC	T-SC	G-SC	E-SC	L-SC	ML-SC	B-SC	IC-SC	T-SC	G-SC
TriviaQA	Base	$0.175_{{\pm 0.001}}$	$0.177_{{\pm 0.001}}$	$0.174_{{\pm 0.001}}$	$0.108_{{\pm 0.001}}$	$0.200_{{\pm 0.001}}$	$0.108_{{\pm 0.001}}$	$0.159_{{\pm 0.001}}$	$0.792_{{\pm 0.002}}$	$0.787_{{\pm 0.001}}$	$0.792_{{\pm 0.002}}$	$0.791_{{\pm 0.001}}$	$0.785_{{\pm 0.002}}$	$0.791_{{\pm 0.001}}$	$0.793_{{\pm 0.002}}$
	SE	$\mathbf{0.062_{{\pm 0.001}}}$	$0.051_{{\pm 0.006}}$	$0.059_{{\pm 0.004}}$	$0.100_{{\pm 0.007}}$	$0.080_{{\pm 0.002}}$	$0.099_{{\pm 0.006}}$	$0.056_{{\pm 0.003}}$	$0.842_{{\pm 0.002}}$	$0.849_{{\pm 0.004}}$	$0.843_{{\pm 0.003}}$	$0.842_{{\pm 0.002}}$	$0.842_{{\pm 0.001}}$	$0.842_{{\pm 0.003}}$	$0.843_{{\pm 0.002}}$
	TS	$0.064_{{\pm 0.002}}$	$0.061_{{\pm 0.007}}$	$0.058_{{\pm 0.001}}$	$0.100_{{\pm 0.004}}$	$\mathbf{0.053_{{\pm 0.004}}}$	$0.100_{{\pm 0.004}}$	$\mathbf{0.055_{{\pm 0.002}}}$	$\mathbf{0.843_{{\pm 0.003}}}$	$\mathbf{0.865_{{\pm 0.003}}}$	$\mathbf{0.844_{{\pm 0.004}}}$	$\mathbf{0.843_{{\pm 0.004}}}$	$\mathbf{0.852_{{\pm 0.004}}}$	$0.843_{{\pm 0.005}}$	$\mathbf{0.845_{{\pm 0.006}}}$
	Platt	$0.109_{{\pm 0.002}}$	$0.044_{{\pm 0.003}}$	$\mathbf{0.057_{{\pm 0.004}}}$	$0.085_{{\pm 0.005}}$	$0.082_{{\pm 0.004}}$	$0.075_{{\pm 0.005}}$	$0.088_{{\pm 0.003}}$	$0.834_{{\pm 0.001}}$	$0.847_{{\pm 0.004}}$	$0.839_{{\pm 0.001}}$	$0.842_{{\pm 0.002}}$	$0.842_{{\pm 0.002}}$	$\mathbf{0.845_{{\pm 0.002}}}$	$0.835_{{\pm 0.001}}$
	ATS	$0.152_{{\pm 0.001}}$	$\mathbf{0.044_{{\pm 0.002}}}$	$0.074_{{\pm 0.003}}$	$\mathbf{0.065_{{\pm 0.004}}}$	$0.108_{{\pm 0.003}}$	$\mathbf{0.062_{{\pm 0.002}}}$	$0.126_{{\pm 0.001}}$	$0.832_{{\pm 0.001}}$	$0.846_{{\pm 0.003}}$	$0.840_{{\pm 0.002}}$	$0.837_{{\pm 0.004}}$	$0.841_{{\pm 0.002}}$	$0.840_{{\pm 0.002}}$	$0.834_{{\pm 0.001}}$

Appendix

Table 6: Uncertainty Metrics for SC Measures Across Methods for Mistral. Mean and standard error of

\widehat{\text{ACE}}

(

\downarrow

) and AUROC (

\uparrow

) scores for SC measures across measures baseline and recalibration methods for Mistral.

		E-SC	L-SC	ML-SC	B-SC	IC-SC	T-SC	G-SC
TriviaQA	Base	$0.766_{{\pm 0.002}}$	$0.761_{{\pm 0.002}}$	$0.764_{{\pm 0.002}}$	$0.765_{{\pm 0.002}}$	$0.765_{{\pm 0.002}}$	$0.766_{{\pm 0.002}}$	$0.767_{{\pm 0.002}}$
	SE	$0.834_{{\pm 0.002}}$	$0.833_{{\pm 0.001}}$	$0.832_{{\pm 0.002}}$	$0.835_{{\pm 0.002}}$	$0.830_{{\pm 0.002}}$	$0.836_{{\pm 0.002}}$	$0.834_{{\pm 0.002}}$
	TS	$\mathbf{0.853_{{\pm 0.002}}}$	$\mathbf{\underline{0.865}_{{\pm 0.002}}}$	$\mathbf{0.849_{{\pm 0.002}}}$	$\mathbf{0.854_{{\pm 0.000}}}$	$\mathbf{\underline{0.864}_{{\pm 0.004}}}$	$\mathbf{0.853_{{\pm 0.001}}}$	$\mathbf{0.851_{{\pm 0.002}}}$
	Platt	$0.839_{{\pm 0.002}}$	$0.840_{{\pm 0.001}}$	$0.836_{{\pm 0.001}}$	$0.837_{{\pm 0.001}}$	$0.836_{{\pm 0.001}}$	$0.838_{{\pm 0.001}}$	$0.839_{{\pm 0.002}}$
	ATS	$0.843_{{\pm 0.001}}$	$0.840_{{\pm 0.001}}$	$0.840_{{\pm 0.002}}$	$0.843_{{\pm 0.001}}$	$0.838_{{\pm 0.002}}$	$0.843_{{\pm 0.001}}$	$0.843_{{\pm 0.001}}$
NQ	Base	$0.693_{{\pm 0.003}}$	$0.691_{{\pm 0.002}}$	$0.695_{{\pm 0.003}}$	$0.694_{{\pm 0.002}}$	$0.692_{{\pm 0.003}}$	$0.694_{{\pm 0.002}}$	$0.693_{{\pm 0.003}}$

Appendix

Table 7: AUROC for

\text{SE}_{\text{conf}}

Across Baseline and Calibration Methods for Qwen. Mean and standard error of AUROC (

\uparrow

) scores computed using

\text{SE}_{\text{conf}}

, where correctness is defined relative to the most confident semantic class. Results are reported for TriviaQA, NQ, and SQuAD across baseline and calibration methods. Bold values indicate the best result within each SC measure for a dataset, while underlined bold values denote the overall best result for that dataset.

		E-SC	L-SC	ML-SC	B-SC	IC-SC	T-SC	G-SC
TriviaQA	Base	$0.755_{{\pm 0.002}}$	$0.755_{{\pm 0.002}}$	$0.755_{{\pm 0.002}}$	$0.756_{{\pm 0.002}}$	$0.754_{{\pm 0.002}}$	$0.756_{{\pm 0.002}}$	$0.755_{{\pm 0.002}}$
	SE	$0.833_{{\pm 0.001}}$	$0.833_{{\pm 0.001}}$	$0.833_{{\pm 0.001}}$	$0.833_{{\pm 0.001}}$	$0.833_{{\pm 0.001}}$	$0.833_{{\pm 0.001}}$	$0.833_{{\pm 0.001}}$
	TS	$\mathbf{0.849_{{\pm 0.001}}}$	$\mathbf{0.844_{{\pm 0.001}}}$	$\mathbf{0.849_{{\pm 0.002}}}$	$\mathbf{0.848_{{\pm 0.002}}}$	$\mathbf{\underline{0.857_{{\pm 0.002}}}}$	$\mathbf{0.848_{{\pm 0.002}}}$	$\mathbf{0.848_{{\pm 0.002}}}$
	Platt	$0.832_{{\pm 0.002}}$	$0.832_{{\pm 0.002}}$	$0.832_{{\pm 0.002}}$	$0.833_{{\pm 0.002}}$	$0.832_{{\pm 0.002}}$	$0.833_{{\pm 0.002}}$	$0.832_{{\pm 0.002}}$
	ATS	$0.834_{{\pm 0.001}}$	$0.832_{{\pm 0.002}}$	$0.835_{{\pm 0.001}}$	$0.836_{{\pm 0.001}}$	$0.832_{{\pm 0.002}}$	$0.836_{{\pm 0.001}}$	$0.834_{{\pm 0.001}}$
NQ	Base	$0.690_{{\pm 0.002}}$	$0.689_{{\pm 0.002}}$	$0.691_{{\pm 0.002}}$	$0.692_{{\pm 0.002}}$	$0.689_{{\pm 0.002}}$	$0.692_{{\pm 0.002}}$	$0.691_{{\pm 0.002}}$

Appendix

Table 8: AUROC for

\text{SE}_{\text{vanilla}}

Across Baseline and Calibration Methods for Qwen. Mean and standard error of AUROC (

\uparrow

) scores computed using

\text{SE}_{\text{vanilla}}

, where correctness is defined via an external decoding procedure ( Kuhn et al., 2023 ) . Results are reported for TriviaQA, NQ, and SQuAD across baseline and calibration methods. Bold values indicate the best result within each SC measure for a dataset, while underlined bold values denote the overall best result for that dataset.

Dataset	Model	E-SC	L-SC	ML-SC	B-SC	IC-SC	T-SC	G-SC
TriviaQA	Llama	1.26	1.63	1.26	1.21	1.21	1.21	1.21
	Qwen	1.80	2.07	1.32	1.56	1.32	1.56	1.56
	Mistral	1.18	1.18	1.00	1.00	1.00	1.00	1.00
NQ	Llama	2.00	2.00	1.47	1.47	1.47	1.71	1.71
	Qwen	2.41	2.41	2.41	2.03	2.41	2.41	2.41
	Mistral	1.70	1.70	1.28	1.28	1.28	1.28	1.28

Appendix

Table 9: Optimised Temperature Settings. Best temperature settings for the temperature scaling (TS) method across models and datasets. Temperatures represent the final optimised after calibration training and hyperparameter selection and are those that are used to give the results shown in Figure 3 .

Figure 8: Selective Accuracy for Qwen Across Varying Rejection Rates. Mean selective accuracy (

\uparrow

) curves of the Qwen model across datasets, comparing SC measures across baseline and recalibration. Error bars show one standard error across runs.

		E-SC	L-SC	ML-SC	B-SC	IC-SC	T-SC	G-SC
TriviaQA	Base	$0.197_{{\pm 0.001}}$	$0.189_{{\pm 0.001}}$	$0.194_{{\pm 0.001}}$	$0.174_{{\pm 0.001}}$	$0.200_{{\pm 0.001}}$	$0.174_{{\pm 0.001}}$	$0.190_{{\pm 0.001}}$
	SE	$0.147_{{\pm 0.001}}$	$0.140_{{\pm 0.001}}$	$0.142_{{\pm 0.001}}$	$0.147_{{\pm 0.001}}$	$0.147_{{\pm 0.001}}$	$0.147_{{\pm 0.001}}$	$0.145_{{\pm 0.001}}$
	TS	$\mathbf{0.140_{{\pm 0.002}}}$	$\mathbf{0.135_{{\pm 0.002}}}$	$\mathbf{0.136_{{\pm 0.003}}}$	$0.150_{{\pm 0.001}}$	$\mathbf{0.130_{{\pm 0.003}}}$	$0.151_{{\pm 0.002}}$	$\mathbf{0.142_{{\pm 0.002}}}$
	Platt	$0.147_{{\pm 0.001}}$	$0.140_{{\pm 0.001}}$	$0.140_{{\pm 0.001}}$	$0.143_{{\pm 0.001}}$	$0.145_{{\pm 0.000}}$	$\mathbf{0.142_{{\pm 0.001}}}$	$0.144_{{\pm 0.001}}$
	ATS	$0.150_{{\pm 0.001}}$	$0.140_{{\pm 0.001}}$	$0.142_{{\pm 0.001}}$	$\mathbf{0.143_{{\pm 0.001}}}$	$0.146_{{\pm 0.002}}$	$0.143_{{\pm 0.001}}$	$0.147_{{\pm 0.001}}$
NQ	Base	$0.358_{{\pm 0.001}}$	$0.350_{{\pm 0.002}}$	$0.353_{{\pm 0.001}}$	$0.309_{{\pm 0.000}}$	$0.374_{{\pm 0.001}}$	$0.310_{{\pm 0.000}}$	$0.345_{{\pm 0.001}}$

Appendix

Table 10: Brier Scores for SC Measures Across Baseline and Recalibration Methods for Llama. Mean and standard error of Brier scores (

\downarrow

) are reported for each SC measure, comparing baseline and recalibration methods for the Llama model.

		$\widehat{\textbf{ACE}}$							CORP-MCB
		E-SC	L-SC	ML-SC	B-SC	IC-SC	T-SC	G-SC	E-SC	L-SC	ML-SC	B-SC	IC-SC	T-SC	G-SC
TriviaQA	Base	$0.308_{{\pm 0.001}}$	$0.299_{{\pm 0.001}}$	$0.304_{{\pm 0.000}}$	$0.240_{{\pm 0.000}}$	$0.320_{{\pm 0.001}}$	$0.240_{{\pm 0.000}}$	$0.293_{{\pm 0.001}}$	$0.105_{{\pm 0.001}}$	$0.096_{{\pm 0.001}}$	$0.101_{{\pm 0.001}}$	$0.062_{{\pm 0.000}}$	$0.114_{{\pm 0.001}}$	$0.062_{{\pm 0.000}}$	$0.093_{{\pm 0.000}}$
	SE	$0.165_{{\pm 0.002}}$	$0.159_{{\pm 0.001}}$	$0.158_{{\pm 0.001}}$	$0.097_{{\pm 0.002}}$	$0.194_{{\pm 0.001}}$	$0.099_{{\pm 0.002}}$	$0.145_{{\pm 0.002}}$	$0.035_{{\pm 0.001}}$	$0.028_{{\pm 0.001}}$	$0.031_{{\pm 0.000}}$	$0.015_{{\pm 0.000}}$	$0.045_{{\pm 0.001}}$	$0.015_{{\pm 0.000}}$	$0.027_{{\pm 0.001}}$
	TS	$\mathbf{0.080_{{\pm 0.001}}}$	$\mathbf{0.066_{{\pm 0.005}}}$	$\mathbf{0.069_{{\pm 0.002}}}$	$\mathbf{0.082_{{\pm 0.003}}}$	$\mathbf{0.085_{{\pm 0.004}}}$	$\mathbf{0.081_{{\pm 0.003}}}$	$\mathbf{0.067_{{\pm 0.002}}}$	$\mathbf{0.010_{{\pm 0.000}}}$	$\mathbf{0.006_{{\pm 0.001}}}$	$\mathbf{0.009_{{\pm 0.000}}}$	$\mathbf{0.011_{{\pm 0.001}}}$	$\mathbf{0.011_{{\pm 0.001}}}$	$\mathbf{0.010_{{\pm 0.001}}}$	$\mathbf{0.008_{{\pm 0.000}}}$
	Platt	$0.170_{{\pm 0.001}}$	$0.168_{{\pm 0.002}}$	$0.163_{{\pm 0.002}}$	$0.100_{{\pm 0.002}}$	$0.203_{{\pm 0.001}}$	$0.101_{{\pm 0.003}}$	$0.150_{{\pm 0.001}}$	$0.035_{{\pm 0.000}}$	$0.032_{{\pm 0.001}}$	$0.032_{{\pm 0.001}}$	$0.015_{{\pm 0.000}}$	$0.050_{{\pm 0.001}}$	$0.015_{{\pm 0.000}}$	$0.028_{{\pm 0.000}}$
	ATS	$0.204_{{\pm 0.003}}$	$0.168_{{\pm 0.002}}$	$0.178_{{\pm 0.002}}$	$0.132_{{\pm 0.003}}$	$0.219_{{\pm 0.001}}$	$0.133_{{\pm 0.002}}$	$0.183_{{\pm 0.003}}$	$0.050_{{\pm 0.001}}$	$0.032_{{\pm 0.001}}$	$0.037_{{\pm 0.001}}$	$0.020_{{\pm 0.001}}$	$0.057_{{\pm 0.001}}$	$0.021_{{\pm 0.001}}$	$0.040_{{\pm 0.001}}$

Appendix

Table 11: Calibration Metric Comparison of SC Measures Across Methods for Qwen. Mean and standard error of calibration error for SC measures under

\widehat{\text{ACE }}

(bin-based) (

\downarrow

) and MCB-CORP (bin-free) (

\downarrow

) across baseline and recalibration methods on TriviaQA, NQ, and SQuAD. Bold entries indicate the best-performing method for each SC measure within a dataset, allowing direct comparison between bin-based and bin-free calibration metrics.

	Method	ACE							ECE
		E-SC	L-SC	ML-SC	B-SC	IC-SC	T-SC	G-SC	E-SC	L-SC	ML-SC	B-SC	IC-SC	T-SC	G-SC
TriviaQA	Base	$0.308_{{\pm 0.001}}$	$0.299_{{\pm 0.001}}$	$0.304_{{\pm 0.000}}$	$0.240_{{\pm 0.000}}$	$0.320_{{\pm 0.001}}$	$0.240_{{\pm 0.000}}$	$0.293_{{\pm 0.001}}$	$0.308_{{\pm 0.001}}$	$0.299_{{\pm 0.001}}$	$0.304_{{\pm 0.000}}$	$0.240_{{\pm 0.000}}$	$0.321_{{\pm 0.001}}$	$0.240_{{\pm 0.000}}$	$0.293_{{\pm 0.001}}$
	SE	$0.165_{{\pm 0.002}}$	$0.159_{{\pm 0.001}}$	$0.158_{{\pm 0.001}}$	$0.097_{{\pm 0.002}}$	$0.194_{{\pm 0.001}}$	$0.099_{{\pm 0.002}}$	$0.145_{{\pm 0.002}}$	$0.165_{{\pm 0.002}}$	$0.158_{{\pm 0.001}}$	$0.158_{{\pm 0.001}}$	$0.100_{{\pm 0.002}}$	$0.194_{{\pm 0.001}}$	$0.101_{{\pm 0.002}}$	$0.145_{{\pm 0.002}}$
	TS	$\mathbf{0.080_{{\pm 0.001}}}$	$\mathbf{0.066_{{\pm 0.005}}}$	$\mathbf{0.069_{{\pm 0.002}}}$	$\mathbf{0.082_{{\pm 0.003}}}$	$\mathbf{0.085_{{\pm 0.004}}}$	$\mathbf{0.081_{{\pm 0.003}}}$	$\mathbf{0.067_{{\pm 0.002}}}$	$\mathbf{0.080_{{\pm 0.001}}}$	$\mathbf{0.066_{{\pm 0.004}}}$	$\mathbf{0.069_{{\pm 0.002}}}$	$\mathbf{0.086_{{\pm 0.004}}}$	$\mathbf{0.087_{{\pm 0.004}}}$	$\mathbf{0.085_{{\pm 0.005}}}$	$\mathbf{0.069_{{\pm 0.002}}}$
	Platt	$0.170_{{\pm 0.001}}$	$0.168_{{\pm 0.002}}$	$0.163_{{\pm 0.002}}$	$0.100_{{\pm 0.002}}$	$0.203_{{\pm 0.001}}$	$0.101_{{\pm 0.003}}$	$0.150_{{\pm 0.001}}$	$0.170_{{\pm 0.001}}$	$0.168_{{\pm 0.002}}$	$0.163_{{\pm 0.002}}$	$0.101_{{\pm 0.002}}$	$0.203_{{\pm 0.001}}$	$0.102_{{\pm 0.002}}$	$0.150_{{\pm 0.001}}$
	ATS	$0.204_{{\pm 0.003}}$	$0.168_{{\pm 0.002}}$	$0.178_{{\pm 0.002}}$	$0.132_{{\pm 0.003}}$	$0.219_{{\pm 0.001}}$	$0.133_{{\pm 0.002}}$	$0.183_{{\pm 0.003}}$	$0.204_{{\pm 0.003}}$	$0.168_{{\pm 0.002}}$	$0.178_{{\pm 0.002}}$	$0.132_{{\pm 0.003}}$	$0.219_{{\pm 0.001}}$	$0.133_{{\pm 0.002}}$	$0.183_{{\pm 0.003}}$

Appendix

Table 12:

\widehat{\text{ACE }}

\widehat{\text{ECE }}

Comparison of SC Measures Across Methods for Qwen. Mean and standard error of calibration error for SC measures under

\widehat{\text{ACE }}

(bin-based) (

\downarrow

) and

\widehat{\text{ECE }}

(

\downarrow

) across baseline and recalibration methods on TriviaQA, NQ, and SQuAD. Bold entries indicate the best-performing method for each SC measure within a dataset.

	TriviaQA	NQ	SQuAD
Base	0.00	0.00	1.43
SE	0.29	0.14	1.29
TS	0.00	0.00	0.71
Platt	0.29	0.00	1.29
ATS	0.00	0.14	2.14
Mean	0.11	0.06	1.37

Appendix

Table 13: Ranking Volatility of Calibration Metrics Across Methods for Qwen. Mean absolute rank change (volatility) of methods on TriviaQA, NQ, and SQuAD when switching from ACE (bin-based) to MCB-CORP (bin-free). Higher values indicate greater sensitivity of method rankings to the choice of calibration metric. Bold entries denote the lowest mean volatility per dataset, corresponding to the most stable ranking of methods.

Figure 9: SC Distribution Plots. Distributions of semantic confidence measures for the Base, SE, and TS methods on the NQ dataset with the Llama model.

Figure 10: Correlation matrices of semantic confidence measures. Pairwise Pearson correlations between SC measures for the Base, SE, and TS methods on the NQ dataset with the Llama model. While correlations are very high under Base and SE settings, indicating redundancy between measures, optimised Temperature Scaling (TS) substantially lowers correlations, revealing greater diversity in how measures capture semantic uncertainty.

Language Model Calibration

Language model calibration studies whether models’ confidence estimates match their actual accuracy across tasks, prompts, and deployment settings.

See all 366 papers

Eigenvalue Calibration for Semantic Embeddings of Large Language Models
Jul 9, 2026Sebastian G. Gruber, Nassim Walha, Francis Bach +1Confidence Estimation in Language Models Language Model Calibration
1ESAT-PSI, KU Leuven, Belgium · 2German Cancer Research Center (DKFZ), Heidelberg, Germany · 3German Cancer Consortium (DKTK), Germany +3
SAGE: Answer-Conditioned Uncertainty Targets for Verbal Uncertainty Alignment
Jun 9, 2026Kaiwen Shi, Zheyuan Zhang, Yanfang YeConfidence Estimation in Language Models Preference Optimization
University of Notre Dame
Semantic Uncertainty Quantification Needs Factual Equivalence
Sep 28, 2026Joseph Hoche, Quentin Guimard, Gianni FranchiConfidence Estimation in Language Models Factual Consistency Evaluation
AMIAD, Pˆole Recherche Palaiseau, France

Jul 9, 2026J/K move · Enter open · S save

Eigenvalue Calibration for Semantic Embeddings of Large Language Models

Sebastian G. Gruber, Nassim Walha, Francis Bach, Florian Buettner

1ESAT-PSI, KU Leuven, Belgium · 2German Cancer Research Center (DKFZ), Heidelberg, Germany · 3German Cancer Consortium (DKTK), Germany+3

Confidence Estimation in Language Models Language Model Calibration Model Calibration LLM Uncertainty Estimation

Open paper AskPDF arXiv

Uncertainty quantification is central to the reliable deployment of large language models (LLMs), and eigenvalues of semantic embeddings have recently emerged as a key tool in state-of-the-art methods. However, conventional calibration results developed for classification probabilities cannot be directly transferred to eigenvalues. We address this gap by proposing a novel framework for calibrating the eigenvalues of semantic embeddings. We interpret LLMs combined with semantic embeddings of their generated answers as density matrix predictors, and we propose a novel approach to calibrate density matrix predictors by applying temperature scaling to their eigenvalues. We establish entropy-risk equivalence under calibration, derive a central calibration inequality specific to eigenvalues, and prove that temperature-scaled eigenvalues optimize calibration when minimizing proper score risks. Experiments on a variety of real-world settings show that current LLMs are systematically overconfident, and validate our theoretical findings. Together, these results advance the foundations and practice of uncertainty quantification for semantic embeddings.

Abstract

Language Model Calibration

Explore similar work

Eigenvalue Calibration for Semantic Embeddings of Large Language Models

SAGE: Answer-Conditioned Uncertainty Targets for Verbal Uncertainty Alignment

Semantic Uncertainty Quantification Needs Factual Equivalence