cs.LGApr 8, 2026

Improving Semantic Uncertainty Quantification in Language Model Question-Answering via Token-Level Temperature Scaling

Authors: Tom A. Lamb, Desi R. Ivanova, Philip H. S. Torr, Tim G. J. Rudner

Organizations: University of Oxford · University of Toronto & Vijil

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Calibration is central to reliable semantic uncertainty quantification, yet prior work has largely focused on discrimination, neglecting calibration. As calibration and discrimination capture distinct aspects of uncertainty, focusing on discrimination alone yields an incomplete picture. We address this gap by systematically evaluating both aspects across a broad set of confidence measures. We show that current approaches, particularly fixed-temperature heuristics, produce systematically miscalibrated and poorly discriminative semantic confidence distributions. We demonstrate that optimising a single scalar temperature, which, we argue, provides a suitable inductive bias, is a surprisingly simple yet effective solution. Our exhaustive evaluation confirms that temperature scaling consistently improves semantic calibration, discrimination, and downstream entropy, outperforming both heuristic baselines and more expressive token-level recalibration methods on question-answering tasks.

Figures & tables

Figure 1: Temperature Scaling Improves Semantic Uncertainty Quantification . Same model using different temperatures generates 10 responses for the same input, which are clustered into semantic groups. We compute the L-SC semantic confidence measure of Kuhn et al. (2023) defined in Equation 3 . Panel (a) uses the standard temperature of 0.50.5 ; panel (b) uses a temperature optimised on a calibration set. We find that optimised temperature scaling improves both semantic calibration and discrimination.
Figure 2: Semantic Confidence Measures. Figure showing how existing ( Farquhar et al., 2024 , E-SC;) and ( Kuhn et al., 2023 , L-SC;) ) as well as a broad set of novel baseline semantic measures (ML-SC, B-SC, T-SC, IC-SC, and G-SC) are computed. For an input x\bm{x} , we sample multiple responses from the model p(⋅∣x)p(\cdot\mid\bm{x}) , and use an NLI model to assess bidirectional entailment, determining whether responses yi\bm{y}^{i} and yj\bm{y}^{j} are semantically equivalent ( yi∼yj∣x\bm{y}^{i}\sim\bm{y}^{j}\mid\bm{x} ). Here, s(C∣x)s(C\mid\bm{x}) denotes the sum, and sˉ(C∣x)\bar{s}(C\mid\bm{x}) the average, H(pCi)\mathcal{H}(p_{C_{i}}) the entropy, and E(Ci∣x)\mathcal{E}(C_{i}\mid\bm{x}) the entropy of the length-normalised log-likelihoods of generations within cluster CC . See Section 3 for details.
Figure 3: Uncertainty Metrics of SC Measures Across Methods. Mean and standard error of ACE^\widehat{\text{ACE}} ( ↓\downarrow ) and AUROC ( ↑\uparrow ) scores for SC measures across baseline, calibration methods, and datasets. Closer to top left of plots indicates better discrimination and calibration, and hence better overall semantic uncertainty quantification.
Table 1: Discrimination Comparison of Entropy for Qwen. Mean and standard error of AUROC ( ↑\uparrow ) values. (a) reports SEconf\text{SE}_{\text{conf}} , where correctness is determined by the most confident semantic cluster under a given SC measure. (b) reports SEvanilla\text{SE}_{\text{vanilla}} from Kuhn et al. (2023) , where correctness is determined via greedy decoding. Bold entries denote the best result within each SC measure per dataset, and underlined entries indicate the best overall per dataset.
Figure 4: Selective Accuracy for Qwen Across Rejection Rates and Datasets. Mean selective accuracy ( ↑\uparrow ) curves, comparing E-SC, L-SC, B-SC and G-SC measures across baseline and calibration methods. Error bars show one standard error across runs.
Figure 5: SC Measures over Varying Numbers of Samples for Llama. Mean ACE ^\widehat{\text{ACE }} ( ↓\downarrow ) and AUROC ( ↑\uparrow ) on the NQ dataset over varying number of sample generations per example. Error bars show one standard error across runs.
Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

StageSplitTriviaQANatural QuestionsSQuAD
CalibrationTraining593746160059577
Validation200020002000
Final EvaluationTest200020002000
Appendix
Table 2: Dataset Split Sizes. For each dataset, we report the number of examples used at different stages of the calibration pipeline: calibration training, calibration validation, and final test evaluation. Split sizes are matched across datasets to ensure fair comparison.
Figure 6: Training and Evaluation Prompts. Prompts applied for both calibration and evaluation across all methods, shown separately for (a) closed-book datasets (TriviaQA and Natural Questions) and (b) open-book datasets (SQuAD).
ModelTriviaQANatural QuestionsSQuAD
Llama71.942.595.5
Qwen53.033.294.4
Mistral64.833.294.2
Appendix
Table 3: Base Model Accuracies (%). Test-set accuracies ( ↑\uparrow ) of the base models through beam search.
Figure 7: Comparison of Uncertainty Metrics of Semantic Measures Across Calibration Methods Using SS and NLL Losses. Mean and standard deviation across four inference runs of ACE^\widehat{\text{ACE}} ( ↓\downarrow ) and AUROC ( ↑\uparrow ) scores for semantic confidence measures for each model across baselines and calibration methods. We compare using a (a) NLL and (b) SS calibration loss.
  ACE^\widehat{\textbf{ACE}}AUROC
  E-SCL-SCML-SCB-SCIC-SCT-SCG-SCE-SCL-SCML-SCB-SCIC-SCT-SCG-SC
TriviaQABase0.184±0.0020.184_{{\pm 0.002}}0.174±0.0020.174_{{\pm 0.002}}0.180±0.0010.180_{{\pm 0.001}}0.130±0.0010.130_{{\pm 0.001}}0.191±0.0020.191_{{\pm 0.002}}0.130±0.0020.130_{{\pm 0.002}}0.172±0.0020.172_{{\pm 0.002}}0.757±0.0010.757_{{\pm 0.001}}0.755±0.0010.755_{{\pm 0.001}}0.757±0.0010.757_{{\pm 0.001}}0.759±0.0010.759_{{\pm 0.001}}0.752±0.0010.752_{{\pm 0.001}}0.760±0.0010.760_{{\pm 0.001}}0.758±0.0020.758_{{\pm 0.002}}
SE0.070±0.0030.070_{{\pm 0.003}}0.056±0.0010.056_{{\pm 0.001}}0.064±0.0030.064_{{\pm 0.003}}0.082±0.0020.082_{{\pm 0.002}}0.092±0.0020.092_{{\pm 0.002}}0.082±0.0020.082_{{\pm 0.002}}0.066±0.0030.066_{{\pm 0.003}}0.832±0.0030.832_{{\pm 0.003}}0.832±0.0020.832_{{\pm 0.002}}0.832±0.0030.832_{{\pm 0.003}}0.831±0.0020.831_{{\pm 0.002}}0.829±0.0030.829_{{\pm 0.003}}0.830±0.0020.830_{{\pm 0.002}}0.833±0.0030.833_{{\pm 0.003}}
TS0.049±0.001\mathbf{0.049_{{\pm 0.001}}}0.055±0.001\mathbf{0.055_{{\pm 0.001}}}0.061±0.001\mathbf{0.061_{{\pm 0.001}}}0.104±0.0040.104_{{\pm 0.004}}0.033±0.003\mathbf{0.033_{{\pm 0.003}}}0.106±0.0020.106_{{\pm 0.002}}0.056±0.004\mathbf{0.056_{{\pm 0.004}}}0.852±0.003\mathbf{0.852_{{\pm 0.003}}}0.849±0.004\mathbf{0.849_{{\pm 0.004}}}0.853±0.005\mathbf{0.853_{{\pm 0.005}}}0.846±0.003\mathbf{0.846_{{\pm 0.003}}}0.858±0.005\mathbf{0.858_{{\pm 0.005}}}0.847±0.004\mathbf{0.847_{{\pm 0.004}}}0.853±0.003\mathbf{0.853_{{\pm 0.003}}}
Platt0.078±0.0010.078_{{\pm 0.001}}0.060±0.0020.060_{{\pm 0.002}}0.071±0.0020.071_{{\pm 0.002}}0.078±0.0020.078_{{\pm 0.002}}0.097±0.0020.097_{{\pm 0.002}}0.077±0.0010.077_{{\pm 0.001}}0.066±0.0030.066_{{\pm 0.003}}0.840±0.0020.840_{{\pm 0.002}}0.833±0.0020.833_{{\pm 0.002}}0.842±0.0010.842_{{\pm 0.001}}0.841±0.0020.841_{{\pm 0.002}}0.837±0.0020.837_{{\pm 0.002}}0.842±0.0020.842_{{\pm 0.002}}0.840±0.0020.840_{{\pm 0.002}}
ATS0.084±0.0010.084_{{\pm 0.001}}0.061±0.0020.061_{{\pm 0.002}}0.069±0.0020.069_{{\pm 0.002}}0.070±0.004\mathbf{0.070_{{\pm 0.004}}}0.093±0.0030.093_{{\pm 0.003}}0.069±0.005\mathbf{0.069_{{\pm 0.005}}}0.070±0.0020.070_{{\pm 0.002}}0.843±0.0010.843_{{\pm 0.001}}0.832±0.0020.832_{{\pm 0.002}}0.838±0.0020.838_{{\pm 0.002}}0.835±0.0010.835_{{\pm 0.001}}0.831±0.0030.831_{{\pm 0.003}}0.836±0.0010.836_{{\pm 0.001}}0.844±0.0010.844_{{\pm 0.001}}
Appendix
Table 4: Uncertainty Metrics for SC Measures Across Methods for Llama. Mean and standard error of ACE^\widehat{\text{ACE}} ( ↓\downarrow ) and AUROC ( ↑\uparrow ) scores for SC measures across measures baseline and recalibration methods for Llama.
  ACE^\widehat{\textbf{ACE}}AUROC
  E-SCL-SCML-SCB-SCIC-SCT-SCG-SCE-SCL-SCML-SCB-SCIC-SCT-SCG-SC
TriviaQABase0.308±0.0010.308_{{\pm 0.001}}0.299±0.0010.299_{{\pm 0.001}}0.304±0.0000.304_{{\pm 0.000}}0.240±0.0000.240_{{\pm 0.000}}0.320±0.0010.320_{{\pm 0.001}}0.240±0.0000.240_{{\pm 0.000}}0.293±0.0010.293_{{\pm 0.001}}0.763±0.0020.763_{{\pm 0.002}}0.759±0.0020.759_{{\pm 0.002}}0.762±0.0020.762_{{\pm 0.002}}0.765±0.0010.765_{{\pm 0.001}}0.761±0.0020.761_{{\pm 0.002}}0.766±0.0010.766_{{\pm 0.001}}0.764±0.0020.764_{{\pm 0.002}}
SE0.165±0.0020.165_{{\pm 0.002}}0.159±0.0010.159_{{\pm 0.001}}0.158±0.0010.158_{{\pm 0.001}}0.097±0.0020.097_{{\pm 0.002}}0.194±0.0010.194_{{\pm 0.001}}0.099±0.0020.099_{{\pm 0.002}}0.145±0.0020.145_{{\pm 0.002}}0.832±0.0020.832_{{\pm 0.002}}0.831±0.0010.831_{{\pm 0.001}}0.830±0.0020.830_{{\pm 0.002}}0.836±0.0020.836_{{\pm 0.002}}0.825±0.0020.825_{{\pm 0.002}}0.837±0.0020.837_{{\pm 0.002}}0.833±0.0020.833_{{\pm 0.002}}
TS0.080±0.001\mathbf{0.080_{{\pm 0.001}}}0.066±0.005\mathbf{0.066_{{\pm 0.005}}}0.069±0.002\mathbf{0.069_{{\pm 0.002}}}0.082±0.003\mathbf{0.082_{{\pm 0.003}}}0.085±0.004\mathbf{0.085_{{\pm 0.004}}}0.081±0.003\mathbf{0.081_{{\pm 0.003}}}0.067±0.002\mathbf{0.067_{{\pm 0.002}}}0.855±0.002\mathbf{0.855_{{\pm 0.002}}}0.870±0.002\mathbf{0.870_{{\pm 0.002}}}0.853±0.002\mathbf{0.853_{{\pm 0.002}}}0.856±0.000\mathbf{0.856_{{\pm 0.000}}}0.864±0.004\mathbf{0.864_{{\pm 0.004}}}0.855±0.001\mathbf{0.855_{{\pm 0.001}}}0.855±0.002\mathbf{0.855_{{\pm 0.002}}}
Platt0.170±0.0010.170_{{\pm 0.001}}0.168±0.0020.168_{{\pm 0.002}}0.163±0.0020.163_{{\pm 0.002}}0.100±0.0020.100_{{\pm 0.002}}0.203±0.0010.203_{{\pm 0.001}}0.101±0.0030.101_{{\pm 0.003}}0.150±0.0010.150_{{\pm 0.001}}0.837±0.0020.837_{{\pm 0.002}}0.839±0.0010.839_{{\pm 0.001}}0.834±0.0010.834_{{\pm 0.001}}0.838±0.0010.838_{{\pm 0.001}}0.833±0.0010.833_{{\pm 0.001}}0.839±0.0010.839_{{\pm 0.001}}0.838±0.0020.838_{{\pm 0.002}}
ATS0.204±0.0030.204_{{\pm 0.003}}0.168±0.0020.168_{{\pm 0.002}}0.178±0.0020.178_{{\pm 0.002}}0.132±0.0030.132_{{\pm 0.003}}0.219±0.0010.219_{{\pm 0.001}}0.133±0.0020.133_{{\pm 0.002}}0.183±0.0030.183_{{\pm 0.003}}0.841±0.0010.841_{{\pm 0.001}}0.839±0.0010.839_{{\pm 0.001}}0.840±0.0020.840_{{\pm 0.002}}0.842±0.0010.842_{{\pm 0.001}}0.833±0.0020.833_{{\pm 0.002}}0.843±0.0010.843_{{\pm 0.001}}0.842±0.0010.842_{{\pm 0.001}}
Appendix
Table 5: Uncertainty Metrics for SC Measures Across Methods for Qwen. Mean and standard error of ACE^\widehat{\text{ACE}} ( ↓\downarrow ) and AUROC ( ↑\uparrow ) scores for SC measures across measures baseline and recalibration methods for Qwen.
  ACE^\widehat{\textbf{ACE}}AUROC
  E-SCL-SCML-SCB-SCIC-SCT-SCG-SCE-SCL-SCML-SCB-SCIC-SCT-SCG-SC
TriviaQABase0.175±0.0010.175_{{\pm 0.001}}0.177±0.0010.177_{{\pm 0.001}}0.174±0.0010.174_{{\pm 0.001}}0.108±0.0010.108_{{\pm 0.001}}0.200±0.0010.200_{{\pm 0.001}}0.108±0.0010.108_{{\pm 0.001}}0.159±0.0010.159_{{\pm 0.001}}0.792±0.0020.792_{{\pm 0.002}}0.787±0.0010.787_{{\pm 0.001}}0.792±0.0020.792_{{\pm 0.002}}0.791±0.0010.791_{{\pm 0.001}}0.785±0.0020.785_{{\pm 0.002}}0.791±0.0010.791_{{\pm 0.001}}0.793±0.0020.793_{{\pm 0.002}}
SE0.062±0.001\mathbf{0.062_{{\pm 0.001}}}0.051±0.0060.051_{{\pm 0.006}}0.059±0.0040.059_{{\pm 0.004}}0.100±0.0070.100_{{\pm 0.007}}0.080±0.0020.080_{{\pm 0.002}}0.099±0.0060.099_{{\pm 0.006}}0.056±0.0030.056_{{\pm 0.003}}0.842±0.0020.842_{{\pm 0.002}}0.849±0.0040.849_{{\pm 0.004}}0.843±0.0030.843_{{\pm 0.003}}0.842±0.0020.842_{{\pm 0.002}}0.842±0.0010.842_{{\pm 0.001}}0.842±0.0030.842_{{\pm 0.003}}0.843±0.0020.843_{{\pm 0.002}}
TS0.064±0.0020.064_{{\pm 0.002}}0.061±0.0070.061_{{\pm 0.007}}0.058±0.0010.058_{{\pm 0.001}}0.100±0.0040.100_{{\pm 0.004}}0.053±0.004\mathbf{0.053_{{\pm 0.004}}}0.100±0.0040.100_{{\pm 0.004}}0.055±0.002\mathbf{0.055_{{\pm 0.002}}}0.843±0.003\mathbf{0.843_{{\pm 0.003}}}0.865±0.003\mathbf{0.865_{{\pm 0.003}}}0.844±0.004\mathbf{0.844_{{\pm 0.004}}}0.843±0.004\mathbf{0.843_{{\pm 0.004}}}0.852±0.004\mathbf{0.852_{{\pm 0.004}}}0.843±0.0050.843_{{\pm 0.005}}0.845±0.006\mathbf{0.845_{{\pm 0.006}}}
Platt0.109±0.0020.109_{{\pm 0.002}}0.044±0.0030.044_{{\pm 0.003}}0.057±0.004\mathbf{0.057_{{\pm 0.004}}}0.085±0.0050.085_{{\pm 0.005}}0.082±0.0040.082_{{\pm 0.004}}0.075±0.0050.075_{{\pm 0.005}}0.088±0.0030.088_{{\pm 0.003}}0.834±0.0010.834_{{\pm 0.001}}0.847±0.0040.847_{{\pm 0.004}}0.839±0.0010.839_{{\pm 0.001}}0.842±0.0020.842_{{\pm 0.002}}0.842±0.0020.842_{{\pm 0.002}}0.845±0.002\mathbf{0.845_{{\pm 0.002}}}0.835±0.0010.835_{{\pm 0.001}}
ATS0.152±0.0010.152_{{\pm 0.001}}0.044±0.002\mathbf{0.044_{{\pm 0.002}}}0.074±0.0030.074_{{\pm 0.003}}0.065±0.004\mathbf{0.065_{{\pm 0.004}}}0.108±0.0030.108_{{\pm 0.003}}0.062±0.002\mathbf{0.062_{{\pm 0.002}}}0.126±0.0010.126_{{\pm 0.001}}0.832±0.0010.832_{{\pm 0.001}}0.846±0.0030.846_{{\pm 0.003}}0.840±0.0020.840_{{\pm 0.002}}0.837±0.0040.837_{{\pm 0.004}}0.841±0.0020.841_{{\pm 0.002}}0.840±0.0020.840_{{\pm 0.002}}0.834±0.0010.834_{{\pm 0.001}}
Appendix
Table 6: Uncertainty Metrics for SC Measures Across Methods for Mistral. Mean and standard error of ACE^\widehat{\text{ACE}} ( ↓\downarrow ) and AUROC ( ↑\uparrow ) scores for SC measures across measures baseline and recalibration methods for Mistral.
  E-SCL-SCML-SCB-SCIC-SCT-SCG-SC
TriviaQABase0.766±0.0020.766_{{\pm 0.002}}0.761±0.0020.761_{{\pm 0.002}}0.764±0.0020.764_{{\pm 0.002}}0.765±0.0020.765_{{\pm 0.002}}0.765±0.0020.765_{{\pm 0.002}}0.766±0.0020.766_{{\pm 0.002}}0.767±0.0020.767_{{\pm 0.002}}
SE0.834±0.0020.834_{{\pm 0.002}}0.833±0.0010.833_{{\pm 0.001}}0.832±0.0020.832_{{\pm 0.002}}0.835±0.0020.835_{{\pm 0.002}}0.830±0.0020.830_{{\pm 0.002}}0.836±0.0020.836_{{\pm 0.002}}0.834±0.0020.834_{{\pm 0.002}}
TS0.853±0.002\mathbf{0.853_{{\pm 0.002}}}0.865‾±0.002\mathbf{\underline{0.865}_{{\pm 0.002}}}0.849±0.002\mathbf{0.849_{{\pm 0.002}}}0.854±0.000\mathbf{0.854_{{\pm 0.000}}}0.864‾±0.004\mathbf{\underline{0.864}_{{\pm 0.004}}}0.853±0.001\mathbf{0.853_{{\pm 0.001}}}0.851±0.002\mathbf{0.851_{{\pm 0.002}}}
Platt0.839±0.0020.839_{{\pm 0.002}}0.840±0.0010.840_{{\pm 0.001}}0.836±0.0010.836_{{\pm 0.001}}0.837±0.0010.837_{{\pm 0.001}}0.836±0.0010.836_{{\pm 0.001}}0.838±0.0010.838_{{\pm 0.001}}0.839±0.0020.839_{{\pm 0.002}}
ATS0.843±0.0010.843_{{\pm 0.001}}0.840±0.0010.840_{{\pm 0.001}}0.840±0.0020.840_{{\pm 0.002}}0.843±0.0010.843_{{\pm 0.001}}0.838±0.0020.838_{{\pm 0.002}}0.843±0.0010.843_{{\pm 0.001}}0.843±0.0010.843_{{\pm 0.001}}
NQBase0.693±0.0030.693_{{\pm 0.003}}0.691±0.0020.691_{{\pm 0.002}}0.695±0.0030.695_{{\pm 0.003}}0.694±0.0020.694_{{\pm 0.002}}0.692±0.0030.692_{{\pm 0.003}}0.694±0.0020.694_{{\pm 0.002}}0.693±0.0030.693_{{\pm 0.003}}
Appendix
Table 7: AUROC for SEconf\text{SE}_{\text{conf}} Across Baseline and Calibration Methods for Qwen. Mean and standard error of AUROC ( ↑\uparrow ) scores computed using SEconf\text{SE}_{\text{conf}} , where correctness is defined relative to the most confident semantic class. Results are reported for TriviaQA, NQ, and SQuAD across baseline and calibration methods. Bold values indicate the best result within each SC measure for a dataset, while underlined bold values denote the overall best result for that dataset.
  E-SCL-SCML-SCB-SCIC-SCT-SCG-SC
TriviaQABase0.755±0.0020.755_{{\pm 0.002}}0.755±0.0020.755_{{\pm 0.002}}0.755±0.0020.755_{{\pm 0.002}}0.756±0.0020.756_{{\pm 0.002}}0.754±0.0020.754_{{\pm 0.002}}0.756±0.0020.756_{{\pm 0.002}}0.755±0.0020.755_{{\pm 0.002}}
SE0.833±0.0010.833_{{\pm 0.001}}0.833±0.0010.833_{{\pm 0.001}}0.833±0.0010.833_{{\pm 0.001}}0.833±0.0010.833_{{\pm 0.001}}0.833±0.0010.833_{{\pm 0.001}}0.833±0.0010.833_{{\pm 0.001}}0.833±0.0010.833_{{\pm 0.001}}
TS0.849±0.001\mathbf{0.849_{{\pm 0.001}}}0.844±0.001\mathbf{0.844_{{\pm 0.001}}}0.849±0.002\mathbf{0.849_{{\pm 0.002}}}0.848±0.002\mathbf{0.848_{{\pm 0.002}}}0.857±0.002‾\mathbf{\underline{0.857_{{\pm 0.002}}}}0.848±0.002\mathbf{0.848_{{\pm 0.002}}}0.848±0.002\mathbf{0.848_{{\pm 0.002}}}
Platt0.832±0.0020.832_{{\pm 0.002}}0.832±0.0020.832_{{\pm 0.002}}0.832±0.0020.832_{{\pm 0.002}}0.833±0.0020.833_{{\pm 0.002}}0.832±0.0020.832_{{\pm 0.002}}0.833±0.0020.833_{{\pm 0.002}}0.832±0.0020.832_{{\pm 0.002}}
ATS0.834±0.0010.834_{{\pm 0.001}}0.832±0.0020.832_{{\pm 0.002}}0.835±0.0010.835_{{\pm 0.001}}0.836±0.0010.836_{{\pm 0.001}}0.832±0.0020.832_{{\pm 0.002}}0.836±0.0010.836_{{\pm 0.001}}0.834±0.0010.834_{{\pm 0.001}}
NQBase0.690±0.0020.690_{{\pm 0.002}}0.689±0.0020.689_{{\pm 0.002}}0.691±0.0020.691_{{\pm 0.002}}0.692±0.0020.692_{{\pm 0.002}}0.689±0.0020.689_{{\pm 0.002}}0.692±0.0020.692_{{\pm 0.002}}0.691±0.0020.691_{{\pm 0.002}}
Appendix
Table 8: AUROC for SEvanilla\text{SE}_{\text{vanilla}} Across Baseline and Calibration Methods for Qwen. Mean and standard error of AUROC ( ↑\uparrow ) scores computed using SEvanilla\text{SE}_{\text{vanilla}} , where correctness is defined via an external decoding procedure ( Kuhn et al., 2023 ) . Results are reported for TriviaQA, NQ, and SQuAD across baseline and calibration methods. Bold values indicate the best result within each SC measure for a dataset, while underlined bold values denote the overall best result for that dataset.
DatasetModelE-SCL-SCML-SCB-SCIC-SCT-SCG-SC
TriviaQALlama1.261.631.261.211.211.211.21
Qwen1.802.071.321.561.321.561.56
Mistral1.181.181.001.001.001.001.00
NQLlama2.002.001.471.471.471.711.71
Qwen2.412.412.412.032.412.412.41
Mistral1.701.701.281.281.281.281.28
Appendix
Table 9: Optimised Temperature Settings. Best temperature settings for the temperature scaling (TS) method across models and datasets. Temperatures represent the final optimised after calibration training and hyperparameter selection and are those that are used to give the results shown in Figure 3 .
Figure 8: Selective Accuracy for Qwen Across Varying Rejection Rates. Mean selective accuracy ( ↑\uparrow ) curves of the Qwen model across datasets, comparing SC measures across baseline and recalibration. Error bars show one standard error across runs.
  E-SCL-SCML-SCB-SCIC-SCT-SCG-SC
TriviaQABase0.197±0.0010.197_{{\pm 0.001}}0.189±0.0010.189_{{\pm 0.001}}0.194±0.0010.194_{{\pm 0.001}}0.174±0.0010.174_{{\pm 0.001}}0.200±0.0010.200_{{\pm 0.001}}0.174±0.0010.174_{{\pm 0.001}}0.190±0.0010.190_{{\pm 0.001}}
SE0.147±0.0010.147_{{\pm 0.001}}0.140±0.0010.140_{{\pm 0.001}}0.142±0.0010.142_{{\pm 0.001}}0.147±0.0010.147_{{\pm 0.001}}0.147±0.0010.147_{{\pm 0.001}}0.147±0.0010.147_{{\pm 0.001}}0.145±0.0010.145_{{\pm 0.001}}
TS0.140±0.002\mathbf{0.140_{{\pm 0.002}}}0.135±0.002\mathbf{0.135_{{\pm 0.002}}}0.136±0.003\mathbf{0.136_{{\pm 0.003}}}0.150±0.0010.150_{{\pm 0.001}}0.130±0.003\mathbf{0.130_{{\pm 0.003}}}0.151±0.0020.151_{{\pm 0.002}}0.142±0.002\mathbf{0.142_{{\pm 0.002}}}
Platt0.147±0.0010.147_{{\pm 0.001}}0.140±0.0010.140_{{\pm 0.001}}0.140±0.0010.140_{{\pm 0.001}}0.143±0.0010.143_{{\pm 0.001}}0.145±0.0000.145_{{\pm 0.000}}0.142±0.001\mathbf{0.142_{{\pm 0.001}}}0.144±0.0010.144_{{\pm 0.001}}
ATS0.150±0.0010.150_{{\pm 0.001}}0.140±0.0010.140_{{\pm 0.001}}0.142±0.0010.142_{{\pm 0.001}}0.143±0.001\mathbf{0.143_{{\pm 0.001}}}0.146±0.0020.146_{{\pm 0.002}}0.143±0.0010.143_{{\pm 0.001}}0.147±0.0010.147_{{\pm 0.001}}
NQBase0.358±0.0010.358_{{\pm 0.001}}0.350±0.0020.350_{{\pm 0.002}}0.353±0.0010.353_{{\pm 0.001}}0.309±0.0000.309_{{\pm 0.000}}0.374±0.0010.374_{{\pm 0.001}}0.310±0.0000.310_{{\pm 0.000}}0.345±0.0010.345_{{\pm 0.001}}
Appendix
Table 10: Brier Scores for SC Measures Across Baseline and Recalibration Methods for Llama. Mean and standard error of Brier scores ( ↓\downarrow ) are reported for each SC measure, comparing baseline and recalibration methods for the Llama model.
  ACE^\widehat{\textbf{ACE}}CORP-MCB
  E-SCL-SCML-SCB-SCIC-SCT-SCG-SCE-SCL-SCML-SCB-SCIC-SCT-SCG-SC
TriviaQABase0.308±0.0010.308_{{\pm 0.001}}0.299±0.0010.299_{{\pm 0.001}}0.304±0.0000.304_{{\pm 0.000}}0.240±0.0000.240_{{\pm 0.000}}0.320±0.0010.320_{{\pm 0.001}}0.240±0.0000.240_{{\pm 0.000}}0.293±0.0010.293_{{\pm 0.001}}0.105±0.0010.105_{{\pm 0.001}}0.096±0.0010.096_{{\pm 0.001}}0.101±0.0010.101_{{\pm 0.001}}0.062±0.0000.062_{{\pm 0.000}}0.114±0.0010.114_{{\pm 0.001}}0.062±0.0000.062_{{\pm 0.000}}0.093±0.0000.093_{{\pm 0.000}}
SE0.165±0.0020.165_{{\pm 0.002}}0.159±0.0010.159_{{\pm 0.001}}0.158±0.0010.158_{{\pm 0.001}}0.097±0.0020.097_{{\pm 0.002}}0.194±0.0010.194_{{\pm 0.001}}0.099±0.0020.099_{{\pm 0.002}}0.145±0.0020.145_{{\pm 0.002}}0.035±0.0010.035_{{\pm 0.001}}0.028±0.0010.028_{{\pm 0.001}}0.031±0.0000.031_{{\pm 0.000}}0.015±0.0000.015_{{\pm 0.000}}0.045±0.0010.045_{{\pm 0.001}}0.015±0.0000.015_{{\pm 0.000}}0.027±0.0010.027_{{\pm 0.001}}
TS0.080±0.001\mathbf{0.080_{{\pm 0.001}}}0.066±0.005\mathbf{0.066_{{\pm 0.005}}}0.069±0.002\mathbf{0.069_{{\pm 0.002}}}0.082±0.003\mathbf{0.082_{{\pm 0.003}}}0.085±0.004\mathbf{0.085_{{\pm 0.004}}}0.081±0.003\mathbf{0.081_{{\pm 0.003}}}0.067±0.002\mathbf{0.067_{{\pm 0.002}}}0.010±0.000\mathbf{0.010_{{\pm 0.000}}}0.006±0.001\mathbf{0.006_{{\pm 0.001}}}0.009±0.000\mathbf{0.009_{{\pm 0.000}}}0.011±0.001\mathbf{0.011_{{\pm 0.001}}}0.011±0.001\mathbf{0.011_{{\pm 0.001}}}0.010±0.001\mathbf{0.010_{{\pm 0.001}}}0.008±0.000\mathbf{0.008_{{\pm 0.000}}}
Platt0.170±0.0010.170_{{\pm 0.001}}0.168±0.0020.168_{{\pm 0.002}}0.163±0.0020.163_{{\pm 0.002}}0.100±0.0020.100_{{\pm 0.002}}0.203±0.0010.203_{{\pm 0.001}}0.101±0.0030.101_{{\pm 0.003}}0.150±0.0010.150_{{\pm 0.001}}0.035±0.0000.035_{{\pm 0.000}}0.032±0.0010.032_{{\pm 0.001}}0.032±0.0010.032_{{\pm 0.001}}0.015±0.0000.015_{{\pm 0.000}}0.050±0.0010.050_{{\pm 0.001}}0.015±0.0000.015_{{\pm 0.000}}0.028±0.0000.028_{{\pm 0.000}}
ATS0.204±0.0030.204_{{\pm 0.003}}0.168±0.0020.168_{{\pm 0.002}}0.178±0.0020.178_{{\pm 0.002}}0.132±0.0030.132_{{\pm 0.003}}0.219±0.0010.219_{{\pm 0.001}}0.133±0.0020.133_{{\pm 0.002}}0.183±0.0030.183_{{\pm 0.003}}0.050±0.0010.050_{{\pm 0.001}}0.032±0.0010.032_{{\pm 0.001}}0.037±0.0010.037_{{\pm 0.001}}0.020±0.0010.020_{{\pm 0.001}}0.057±0.0010.057_{{\pm 0.001}}0.021±0.0010.021_{{\pm 0.001}}0.040±0.0010.040_{{\pm 0.001}}
Appendix
Table 11: Calibration Metric Comparison of SC Measures Across Methods for Qwen. Mean and standard error of calibration error for SC measures under ACE ^\widehat{\text{ACE }} (bin-based) ( ↓\downarrow ) and MCB-CORP (bin-free) ( ↓\downarrow ) across baseline and recalibration methods on TriviaQA, NQ, and SQuAD. Bold entries indicate the best-performing method for each SC measure within a dataset, allowing direct comparison between bin-based and bin-free calibration metrics.
 MethodACEECE
  E-SCL-SCML-SCB-SCIC-SCT-SCG-SCE-SCL-SCML-SCB-SCIC-SCT-SCG-SC
TriviaQABase0.308±0.0010.308_{{\pm 0.001}}0.299±0.0010.299_{{\pm 0.001}}0.304±0.0000.304_{{\pm 0.000}}0.240±0.0000.240_{{\pm 0.000}}0.320±0.0010.320_{{\pm 0.001}}0.240±0.0000.240_{{\pm 0.000}}0.293±0.0010.293_{{\pm 0.001}}0.308±0.0010.308_{{\pm 0.001}}0.299±0.0010.299_{{\pm 0.001}}0.304±0.0000.304_{{\pm 0.000}}0.240±0.0000.240_{{\pm 0.000}}0.321±0.0010.321_{{\pm 0.001}}0.240±0.0000.240_{{\pm 0.000}}0.293±0.0010.293_{{\pm 0.001}}
SE0.165±0.0020.165_{{\pm 0.002}}0.159±0.0010.159_{{\pm 0.001}}0.158±0.0010.158_{{\pm 0.001}}0.097±0.0020.097_{{\pm 0.002}}0.194±0.0010.194_{{\pm 0.001}}0.099±0.0020.099_{{\pm 0.002}}0.145±0.0020.145_{{\pm 0.002}}0.165±0.0020.165_{{\pm 0.002}}0.158±0.0010.158_{{\pm 0.001}}0.158±0.0010.158_{{\pm 0.001}}0.100±0.0020.100_{{\pm 0.002}}0.194±0.0010.194_{{\pm 0.001}}0.101±0.0020.101_{{\pm 0.002}}0.145±0.0020.145_{{\pm 0.002}}
TS0.080±0.001\mathbf{0.080_{{\pm 0.001}}}0.066±0.005\mathbf{0.066_{{\pm 0.005}}}0.069±0.002\mathbf{0.069_{{\pm 0.002}}}0.082±0.003\mathbf{0.082_{{\pm 0.003}}}0.085±0.004\mathbf{0.085_{{\pm 0.004}}}0.081±0.003\mathbf{0.081_{{\pm 0.003}}}0.067±0.002\mathbf{0.067_{{\pm 0.002}}}0.080±0.001\mathbf{0.080_{{\pm 0.001}}}0.066±0.004\mathbf{0.066_{{\pm 0.004}}}0.069±0.002\mathbf{0.069_{{\pm 0.002}}}0.086±0.004\mathbf{0.086_{{\pm 0.004}}}0.087±0.004\mathbf{0.087_{{\pm 0.004}}}0.085±0.005\mathbf{0.085_{{\pm 0.005}}}0.069±0.002\mathbf{0.069_{{\pm 0.002}}}
Platt0.170±0.0010.170_{{\pm 0.001}}0.168±0.0020.168_{{\pm 0.002}}0.163±0.0020.163_{{\pm 0.002}}0.100±0.0020.100_{{\pm 0.002}}0.203±0.0010.203_{{\pm 0.001}}0.101±0.0030.101_{{\pm 0.003}}0.150±0.0010.150_{{\pm 0.001}}0.170±0.0010.170_{{\pm 0.001}}0.168±0.0020.168_{{\pm 0.002}}0.163±0.0020.163_{{\pm 0.002}}0.101±0.0020.101_{{\pm 0.002}}0.203±0.0010.203_{{\pm 0.001}}0.102±0.0020.102_{{\pm 0.002}}0.150±0.0010.150_{{\pm 0.001}}
ATS0.204±0.0030.204_{{\pm 0.003}}0.168±0.0020.168_{{\pm 0.002}}0.178±0.0020.178_{{\pm 0.002}}0.132±0.0030.132_{{\pm 0.003}}0.219±0.0010.219_{{\pm 0.001}}0.133±0.0020.133_{{\pm 0.002}}0.183±0.0030.183_{{\pm 0.003}}0.204±0.0030.204_{{\pm 0.003}}0.168±0.0020.168_{{\pm 0.002}}0.178±0.0020.178_{{\pm 0.002}}0.132±0.0030.132_{{\pm 0.003}}0.219±0.0010.219_{{\pm 0.001}}0.133±0.0020.133_{{\pm 0.002}}0.183±0.0030.183_{{\pm 0.003}}
Appendix
Table 12: ACE ^\widehat{\text{ACE }} vs ECE ^\widehat{\text{ECE }} Comparison of SC Measures Across Methods for Qwen. Mean and standard error of calibration error for SC measures under ACE ^\widehat{\text{ACE }} (bin-based) ( ↓\downarrow ) and ECE ^\widehat{\text{ECE }} ( ↓\downarrow ) across baseline and recalibration methods on TriviaQA, NQ, and SQuAD. Bold entries indicate the best-performing method for each SC measure within a dataset.
 TriviaQANQSQuAD
Base0.000.001.43
SE0.290.141.29
TS0.000.000.71
Platt0.290.001.29
ATS0.000.142.14
Mean0.110.061.37
Appendix
Table 13: Ranking Volatility of Calibration Metrics Across Methods for Qwen. Mean absolute rank change (volatility) of methods on TriviaQA, NQ, and SQuAD when switching from ACE (bin-based) to MCB-CORP (bin-free). Higher values indicate greater sensitivity of method rankings to the choice of calibration metric. Bold entries denote the lowest mean volatility per dataset, corresponding to the most stable ranking of methods.
Figure 9: SC Distribution Plots. Distributions of semantic confidence measures for the Base, SE, and TS methods on the NQ dataset with the Llama model.
Figure 10: Correlation matrices of semantic confidence measures. Pairwise Pearson correlations between SC measures for the Base, SE, and TS methods on the NQ dataset with the Llama model. While correlations are very high under Base and SE settings, indicating redundancy between measures, optimised Temperature Scaling (TS) substantially lowers correlations, revealing greater diversity in how measures capture semantic uncertainty.
More in this topic

Language Model Calibration

Language model calibration studies whether models’ confidence estimates match their actual accuracy across tasks, prompts, and deployment settings.

See all 366 papers

Explore similar work

CardsList
Jul 9, 2026

Eigenvalue Calibration for Semantic Embeddings of Large Language Models

Uncertainty quantification is central to the reliable deployment of large language models (LLMs), and eigenvalues of semantic embeddings have recently emerged as a key tool in state-of-the-art methods. However, conventional calibration results developed for classification probabilities cannot be directly transferred to eigenvalues. We address this gap by proposing a novel framework for calibrating the eigenvalues of semantic embeddings. We interpret LLMs combined with semantic embeddings of their generated answers as density matrix predictors, and we propose a novel approach to calibrate density matrix predictors by applying temperature scaling to their eigenvalues. We establish entropy-risk equivalence under calibration, derive a central calibration inequality specific to eigenvalues, and prove that temperature-scaled eigenvalues optimize calibration when minimizing proper score risks. Experiments on a variety of real-world settings show that current LLMs are systematically overconfident, and validate our theoretical findings. Together, these results advance the foundations and practice of uncertainty quantification for semantic embeddings.

Sebastian G. Gruber, Nassim Walha, Francis Bach +1

1ESAT-PSI, KU Leuven, Belgium · 2German Cancer Research Center (DKFZ), Heidelberg, Germany · 3German Cancer Consortium (DKTK), Germany +3

Confidence Estimation in Language ModelsLanguage Model CalibrationModel CalibrationLLM Uncertainty Estimation
AskPDF
Jun 9, 2026

SAGE: Answer-Conditioned Uncertainty Targets for Verbal Uncertainty Alignment

Large language models increasingly express uncertainty through natural-language statements, yet these expressions often fail to reflect the model's sampled behavior. We study verbal uncertainty alignment as a distributional calibration problem: the appropriate uncertainty target for a prompt should be estimated from repeated model outputs rather than from an isolated response. However, group rollouts alone are insufficient, since the resulting target must provide a useful training signal. Existing targets only partially satisfy this requirement. We propose SAGE, Semantic-Answer Guided Entropy, a group-level uncertainty target that constructs an answer-conditioned uncertainty geometry over sampled responses. SAGE preserves categorical, numeric, and symbolic answer distinctions while maintaining a smooth and scale-preserving calibration signal. We further apply this target through Group-Uncertainty Preference Optimization, or GUPO, an uncertainty-channel training framework that supervises verbal uncertainty expressions rather than the full response. Experiments across factual, mathematical, and multiple-choice reasoning tasks show improved uncertainty ranking, lower calibration error, and reduced overconfidence.

Kaiwen Shi, Zheyuan Zhang, Yanfang Ye

University of Notre Dame

Confidence Estimation in Language ModelsPreference OptimizationLLM Uncertainty EstimationUncertainty Calibration
AskPDF
Sep 28, 2026

Semantic Uncertainty Quantification Needs Factual Equivalence

Semantic uncertainty quantification for large language models rests on a common template: sample several answers, measure how much they agree, and treat disagreement as uncertainty. We first formalize this template as two separate roles: an operator that compares two answers, and an aggregator that combines all pairwise comparisons into a scalar. Existing methods differ almost entirely in how they aggregate, while taking the operator off the shelf, typically an NLI model or a generic sentence encoder. We show that this reliance on off-the-shelf operators is the primary bottleneck of semantic UQ: they do not accurately measure factual equivalence of multiple answers to the same question. We resolve this with a deliberately simple recipe: a single encoder trained contrastively to isolate the targeted fact, utilizing synthetic data generated by an LLM and dataset both disjoint from all evaluation settings. Integrating the resulting operator into existing methods improves performance on 120 of 126 evaluation settings (95%) spanning 18 model dataset combinations across language and vision-language models. The best variant reaches 0.76 mean AUROC against 0.68 for the strongest baseline, while replacing the quadratic cross-encoder comparisons of entailment-based operators with one encoder pass per answer. The uniformity of the improvement supports the view that the operator, not the aggregator, is the limiting factor. The same operator also improves single generation token-level estimators: the norm it assigns to each token measures how much that token bears on the answer, and reweighting token log-likelihoods accordingly sharpens the estimate.

Joseph Hoche, Quentin Guimard, Gianni Franchi

AMIAD, Pˆole Recherche Palaiseau, France

Confidence Estimation in Language ModelsFactual Consistency EvaluationLLM ReliabilityLLM Uncertainty EstimationSemantic Uncertainty Estimation
AskPDF