Large language models (LLMs) are evaluated for calibration using metrics such as Expected Calibration Error that conflate two distinct components: the model's ability to discriminate correct from incorrect answers (sensitivity) and its tendency toward confident or cautious responding (bias). Signal Detection Theory (SDT) decomposes these components. While SDT-derived metrics such as AUROC are increasingly used, the full parametric framework - unequal-variance model fitting, criterion estimation, z-ROC analysis - has not been applied to LLMs as signal detectors. In this pre-registered study, we treat three LLMs as observers performing factual discrimination across 168,000 trials and test whether temperature functions as a criterion shift analogous to payoff manipulations in human psychophysics. Critically, this analogy may break down because temperature changes the generated answer itself, not only the confidence assigned to it. Our results confirm the breakdown with temperature simultaneously increasing sensitivity (AUC) and shifting criterion. All models exhibited unequal-variance evidence distributions (z-ROC slopes 0.52-0.84), with instruct models showing more extreme asymmetry (0.52-0.63) than the base model (0.77-0.87) or human recognition memory (~0.80). The SDT decomposition revealed that models occupying distinct positions in sensitivity-bias space could not be distinguished by calibration metrics alone, demonstrating that the full parametric framework provides diagnostic information unavailable from existing metrics.
Figures & tables
Model
T
AUC [95% CI]
da [95% CI]
c [95% CI]
Slope
Llama-3-Inst
0.1
.738 [.726, .750]
1.43 [1.32, 1.56]
− 2.85 [ − 2.88, − 2.80]
0.52
0.5
.748 [.735, .760]
1.37 [1.27, 1.48]
− 2.64 [ − 2.73, − 2.56]
0.56
1.0
.773 [.761, .785]
1.39 [1.31, 1.48]
− 2.40 [ − 2.49, − 2.32]
0.63
1.5
.795 [.783, .806]
1.45 [1.37, 1.53]
− 2.19 [ − 2.28, − 2.11]
0.62
2.0
.809 [.797, .820]
1.50 [1.42, 1.58]
− 2.07 [ − 2.16, − 1.99]
0.65
Mistral-7B-Inst
0.1
.620 [.610, .629]
1.86 [1.51, 4.55] †
− 2.88 [ − 2.88, − 2.88]
0.56
Table 1: SDT parameters at selected temperatures (TriviaQA). Bootstrap 95% CIs from 10,000 resamples.
Figure 1: ROC curves for three models on TriviaQA across seven temperatures (T = 0.1–2.0). Warmer colours indicate higher temperatures. Curves shift outward with increasing temperature, indicating increasing AUC.
Figure 2: Temperature effects on SDT parameters (TriviaQA). Left: AUC increases monotonically ( ρ≥ 0.964). Centre: da trend. Right: Criterion c shifts monotonically from liberal toward less liberal. Shaded bands: bootstrap 95% CIs.
Figure 3: SDT operating points at T = 1.0. Position: ( da , c); fill: ECE. Edge colour: model identity. Circles: TriviaQA; squares: NQ. Models occupy distinct positions invisible to ECE alone.
Figure 4: Bland-Altman plot: da (Paradigm A, T = 1.0) versus d4AFC′ (Paradigm B) across domains. Systematic negative bias reflects near-ceiling 4AFC inflating d4AFC′ .
Figure 5: z-ROC plots (TriviaQA, all temperatures overlaid). Black line: T = 1.0 regression. Slopes < 1.0 confirm unequal variance. Instruct models: 0.52–0.63; base model: 0.78.
Model
Dataset
Slope
s
E5 ρ
Llama-3-Inst
TQA
0.63
0.88
0.93 ∗
NQ
0.58
0.75
0.89 ∗
Mistral-7B-Inst
TQA
0.57
0.57
0.32
NQ
0.54
0.68
− 0.57
Llama-3-Base
TQA
0.78
1.17
0.89 ∗
NQ
0.84
1.38
− 0.71
Table 2: z-ROC slopes at T = 1.0 and slope × temperature correlations (E5). ∗p<.01 .
Figure 6: Domain-specific da at T = 1.0 (TriviaQA). Science & Technology is consistently hardest. Mistral shows widest range (1.43–2.34).
Figure 7: z-ROC slope as a function of temperature. Slope increases significantly with T for Llama-3 models ( ρ≥ 0.89, p < .01) but not Mistral, indicating model-specific changes in evidence distribution shape.
Figure 8: NQ replication: AUC and da versus temperature. All primary patterns replicate on Natural Questions.
Standard evaluation of LLM confidence relies on calibration metrics (ECE, Brier score) that conflate two capacities: how much a model knows (Type-1 accuracy) and how well its confidence signal tracks that knowledge (Type-2 metacognitive sensitivity). We apply Signal Detection Theory to decompose them, treating token-level normalised log-probability as a graded confidence variable and answer correctness as the state to be discriminated. We characterise the Type-2 ROC of this signal, including its unequal-variance structure via z-ROC analysis, and -- because the meta-d' efficiency ratio is not well defined for open-ended QA, which lacks a two-alternative Type-1 decision -- quantify efficiency with a model-free information measure, normalised metacognitive information (meta-I_2r). Across four LLMs and 224,000 factual QA trials we find: (1) metacognitive information varies by a factor of 1.98 across models and is not predicted by accuracy, the rank correlation being -0.80 on TriviaQA and +0.00 on Natural Questions; (2) the confidence signal has model-specific unequal-variance structure (z-ROC slopes 0.78 to 1.18) invisible to calibration metrics, the slope ordering replicating on NQ; (3) efficiency is domain-specific, weakest in Science & Technology for every model; (4) temperature dissociates accuracy from metacognitive information, which stays near-flat for three of four models while accuracy falls; and (5) metacognitive information tracks the accuracy gain from confidence-based abstention exactly (rho = +1.00) while accuracy does not. All estimates carry permutation nulls and bootstrap confidence intervals. This version (v3) corrects a differential length bias in the automated correctness scorer, validated against 1,830 human adjudications; the inverse accuracy-efficiency coupling reported in v1 and v2 does not survive relabelling. See the version note on page 1.
Standard calibration evaluation and recalibration methods assume access to predicted probabilities. We study one-bit threshold access, a setting in which probabilities are hidden and the evaluator observes only whether the predicted probability exceeds a single chosen threshold per sample. Under this restriction, we show that classifiers can nevertheless be evaluated and recalibrated without reconstructing their hidden scores. We derive a consistent estimator of True Calibration Error, unbiased estimators of strictly proper scoring rules, and a one-bit objective for Temperature Scaling whose expectation matches the standard log-loss objective. Across these settings, we quantify the statistical cost incurred by replacing standard probability-access procedures by our one-bit estimators. Finally, we instantiate the framework for black-box LLMs, where the logit_bias API parameter allows us to realize the required threshold queries in zero-shot binary question answering.
Roman Plaud, Antoine Saillenfest, Matthieu Labeau +2
Institut Polytechnique de Paris · Onepoint, France · Ghent University
Large language models (LLM) can state their confidence in an answer (verbalized confidence), or the confidence can be read from the probability of the answer's tokens (token confidence). Comparisons of the two ask which is better calibrated, but the token probability of an answer depends on the text that precedes it. Across five open LLMs and four question-answering datasets, we fix each answer, its verbalized confidence, and its correctness label, and vary how each score is measured. The scores themselves change with the protocol, and so does the comparison between them. Changing the context in which the token probability is read shifts the gap between the two scores by about the size of the gap itself (.287 vs. .304 in expected calibration error, ECE; .189 vs. .145 in AUROC). The 95% interval of the shift excludes zero in 19 and 17 of the 20 settings, and under AUROC the shift has the same sign in all of them, so the context changes how answers are ranked. The better score flips in 8 settings under each metric, counting only flips that the intervals support, and the shift also appears in four larger models of up to 72B parameters. Most of the shift appears with changes to the instruction, not to the position of the answer. Other measurement choices also matter, and stated confidence depends on how a supplied answer is written. We turn these results into a reporting checklist for LLM confidence and its comparisons.