Whether a language model "knows what it knows" matters for trustworthy deployment and for deciding when it should abstain. Prior work usually measures this through one channel: verbalized confidence, a linear probe on hidden states, or agreement among sampled answers (semantic entropy), and often treats these as interchangeable. We compare all three in a pre-registered audit of five Qwen2.5 models (0.5B-7B, including a base/instruct pair at 7B) on factual retrieval (PopQA, SimpleQA) and multi-step reasoning (GSM8K, MATH). First, probes trained on prefill states to predict whether sampled answers will disagree become informative earlier for retrieval (about 7% of layer depth on average) than for reasoning (about 28%). The pre-registered five-layer test supported this only marginally; a finer 21-layer follow-up, designed after those results, supports it more clearly. In most retrieval cells, however, a prompt-only TF-IDF classifier matches the probe, so the early signal may reflect surface features of the question. Second, verbalized confidence is unstable: slightly different prompt templates yield scores that barely correlate. Third, Expected Calibration Error is misleading: verbal confidence looks well calibrated because it clusters near the base rate while discriminating poorly, whereas sampling agreement has worse ECE but about twice the resolution. Of five pre-registered hypotheses, two were falsified, one was partly supported, one was not supported, and one could not be tested because the base model's verbal confidence was degenerate.
Figures & tables
Figure 1: Complete 30-cell evaluation matrix. Pilot accuracy per (model × tier) cell. Dotted outlines indicate cells outside the 25% to 80% accuracy band.
Figure 2: AUROC across 21 layer depth percentiles. Calibration-split AUROC of linear probes predicting the median-split semantic-entropy label from prefill hidden states, faceted by model size. Retrieval tiers (cool lines) mostly cross 0.65 by 5% depth; reasoning tiers (warm lines) cross later on average. In 9 of 11 retrieval cells the onset-layer probe does not beat the prompt-only TF-IDF baseline (Table 1 ).
Model
Tier
Onset (%)
Onset layer
Probe AUROC (onset)
Probe AUROC (peak)
TF-IDF AUROC
Probe beats TF-IDF at onset
Panel A: Factual retrieval tiers ( N=11 cells; mean onset =6.82% , median =5.0% )
Qwen2.5-1.5B-Instruct
R1 (PopQA-high)
5.0
Layer 1 / 28
0.864
0.941
0.882
No
Qwen2.5-1.5B-Instruct
R2 (PopQA-low)
5.0
Layer 1 / 28
0.877
0.935
0.889
No
Qwen2.5-1.5B-Instruct
R3 (SimpleQA)
5.0
Layer 1 / 28
0.775
0.817
0.728
Yes
Qwen2.5-3B-Instruct
R2 (PopQA-low)
5.0
Layer 2 / 36
0.867
0.905
0.868
No
Qwen2.5-3B-Instruct
R3 (SimpleQA)
15.0
Layer 5 / 36
0.654
0.784
0.670
No
Table 1: Onset depth for all 24 probe-valid cells (21-tap follow-up sweep).
Comparison Pair
Spearman ρ
Pre-Registered Target
Status
Format A vs. Format B
+0.126
≥0.60
FALSIFIED
Format A vs. Format C
−0.069
≥0.60
FALSIFIED
Format B vs. Format C
+0.247
≥0.60
FALSIFIED
Table 2: Verbal format correlation matrix (Gate 2 compliance).
Figure 3: Signal calibration and Murphy decomposition on the test split. Left: Calibration curves close to the diagonal; points with N<10 pruned to remove small-sample noise. Right: Murphy decomposition of the Brier score. All signals show small reliability error (solid bars), while behavioral consensus achieves more than double the resolution (light bars) of verbal confidence.
Figure 4: Epistemic alignment quadrants and forced-answer abstention behavior. Left: Calibrated verbal vs. behavioral/internal confidence. Right: The abstention split on Format C passes (justified hedge vs. missed knowledge).
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Cross-signal rank correlations. Per-cell Spearman correlations between the raw verbal, behavioral and internal signals, computed where both signals in a pair are present. Pooled over the test split, raw verbal and internal signals correlate at ρ=0.086 . After per-cell calibration the pooled correlation rises to 0.83, which mostly reflects differences in difficulty between cells.
Large language models can report a numerical confidence together with generated content, but it is unclear whether this report is more than calibrated rhetoric. We analyze three training-free signals: confidence verbalized with the answer, post-hoc P(True), and agreement with three additional generations on the same 100 TriviaQA questions for two model families. Direct verbalization is a surprisingly strong baseline: after auditing benchmark errors, it reaches AUROC 0.956 and 0.937 for correctness prediction. Three-sample agreement is substantially weaker (0.765 and 0.790), and a fixed interpolation with verbalized confidence has no statistically reliable benefit. Four of nine errors from one model and two of eight from the other receive unanimous sample support, showing that self-consistency can amplify shared misconceptions. Re-eliciting confidence for the same fixed answers with equivalent prompts changes scores by 0.043 to 0.084 on average and flips 4% to 9% of decisions at a 0.8 threshold. An exploratory audit of 100 confidence-tagged biography claims further finds only a modest confidence gap between supported and contradicted claims. These results argue that useful self-reports remain sensitive to elicitation, correlated errors, and benchmark noise.
Large language models (LLMs) tend to verbalize confidence scores that are largely detached from their actual accuracy, yet the geometric relationship governing this behavior remain poorly understood. In this work, we present a mechanistic interpretability analysis of verbalized confidence, using linear probes and contrastive activation addition (CAA) steering to show that calibration and verbalized confidence signals are encoded linearly but are orthogonal to one another -- a finding consistent across three open-weight models and four datasets. Interestingly, when models are prompted to simultaneously reason through a problem and verbalize a confidence score, the reasoning process disrupts the verbalized confidence direction, exacerbating miscalibration. We term this the "Reasoning Contamination Effect." Leveraging this insight, we introduce a two-stage adaptive steering pipeline that reads the model's internal accuracy estimate and steers verbalized output to match it, substantially improving calibration alignment across all evaluated models.
Miranda Muqing Miao, Lyle Ungar
University of Pennsylvania · Department of Computer and Information Science
Knowledge probing quantifies how much relational knowledge a language model (LM) has acquired during pre-training. Existing knowledge probes evaluate model capabilities through metrics like prediction accuracy and precision. Such evaluations fail to account for the model's reliability, reflected in the calibration of its confidence scores. In this paper, we establish an empirical framework to evaluate the calibration of relational knowledge in a closed answer-set setting. We cover three modalities of model confidence: (1) intrinsic confidence, (2) paraphrase consistency and (3) linguistic confidence. Our extensive analysis of 21 causal and 8 masked language models reveals that most models, but especially those pre-trained with the masking objective, are overconfident. The best-calibrated scores come from confidence estimates that account for inconsistencies due to statement rephrasing. Moreover, calibration improves when increasing the number of answer options.
Christopher Kissling, Elena Merdjanovska, Alan Akbik
Humboldt-Universität zu Berlin · Science of Intelligence