Closing the Confidence-Faithfulness Gap in Large Language Models

Miranda Muqing Miao; Lyle Ungar

cs.CLMar 26, 2026

Closing the Confidence-Faithfulness Gap in Large Language Models

Authors: Miranda Muqing Miao, Lyle Ungar

Organizations: University of Pennsylvania · Department of Computer and Information Science

Abstract

Large language models (LLMs) tend to verbalize confidence scores that are largely detached from their actual accuracy, yet the geometric relationship governing this behavior remain poorly understood. In this work, we present a mechanistic interpretability analysis of verbalized confidence, using linear probes and contrastive activation addition (CAA) steering to show that calibration and verbalized confidence signals are encoded linearly but are orthogonal to one another -- a finding consistent across three open-weight models and four datasets. Interestingly, when models are prompted to simultaneously reason through a problem and verbalize a confidence score, the reasoning process disrupts the verbalized confidence direction, exacerbating miscalibration. We term this the "Reasoning Contamination Effect." Leveraging this insight, we introduce a two-stage adaptive steering pipeline that reads the model's internal accuracy estimate and steers verbalized output to match it, substantially improving calibration alignment across all evaluated models.

Figures & tables

Figure 1: Ridge probe projection at layer 21 (Qwen-2.5-7B-Base). Left: Distribution of activations projected onto the probe weight vector, separated by correct (blue) and incorrect (pink) answers (Cohen’s $d=1.88$ ). Right: The same scalar projection plotted against binned empirical accuracy ( $r=0.80$ ). Takeaway: The model encodes well-calibrated accuracy information in a single linear direction, even when never asked about confidence.

Figure 2: High and low verbalized confidence occupy distinct regions of activation space (25th vs. 75th percentile split). First principal component of activations from the pure confidence prompt, colored by whether the model verbalized high or low confidence. Takeaway: Verbalized confidence is linearly separable in later layers, confirming that the model constructs a dedicated confidence representation during processing.

Figure 3: Probe fit and directional analysis across layers (Qwen-2.5-7B-Base). (a, b) Train and test

R^{2}

of ridge probes predicting empirical accuracy (gold calibration, blue) and verbalized confidence (pure verbal, orange). (c) Cosine similarity between the two probe weight vectors (pure verbal vs. gold calibration). (d) Cosine similarity between contrastive confidence and accuracy directions, computed separately under the pure confidence prompt (blue) and the joint solve-and-rate prompt (red). Shaded region indicates the gap between the two conditions, the reasoning contamination effect . Takeaway: Accuracy and confidence are encoded in nearly orthogonal directions (cosine similarity

<0.04

), and joint prompting inverts their relationship (from

+0.26

-0.63

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

$k$	Target Conf.	Appended Note
1	10–20%	“You are extremely uncertain about your mathematical abilities. You should assume you are likely wrong unless the problem is trivially simple. Assign confidence below 20 for most problems, and only above 50 if you are absolutely certain of every step.”
2	25–35%	“You are making important decisions, thus you should avoid giving a wrong answer with high confidence. You should be very cautious. For most problems, your confidence should be below 30 unless you are very sure of every step.”
3	35–50%	“You are making important decisions, thus you should avoid giving a wrong answer with high confidence. You should be cautious and critical of your own reasoning. Assign confidence below 50 unless you have verified every step carefully.”
4	45–60%	“You are making important decisions, thus you should avoid giving a wrong answer with high confidence. Be realistic about the difficulty of the problem. Most math problems have subtle traps, so moderate confidence (40–70) is appropriate for most answers.”
5	20–40%	“(1) You are making important decisions, thus you should avoid giving a wrong answer with high confidence. (2) You should be very cautious, and tend to give low confidence on almost all of the answers.”
6	30–50%	“(1) You are making important decisions, thus you should avoid giving a wrong answer with high confidence. When in doubt, err on the side of lower confidence.”

Appendix

Table 5: Prompt framings used to elicit diverse verbalized confidence levels (

K{=}11

). Each framing appends a “Note” to the base prompt (see below). The Vanilla framing appends no note. Target ranges are approximate and were calibrated on a pilot study with Qwen-2.5-7B-Base.

Figure 5: Subspace orthogonality analysis between gold calibration and verbalized confidence representations across transformer layers. (a) Mean principal angle between 10-dimensional predictive subspaces extracted via iterative ridge regression with deflation; the gray band shows the

\pm 2\sigma

range for random subspace pairs of equal dimensionality. (b) Top two canonical correlations from CCA applied to the 5-dimensional projections of each concept’s subspace. (c)

R^{2}

retention ratio after projecting out the other concept’s top-10 subspace (cross-concept removal) versus projecting out one’s own subspace (self-removal control). (d) Variance decomposition showing unique and shared

R^{2}

for each concept, where shared

R^{2}

is measured by predicting one target using only the other concept’s subspace directions. Across all four analyses and all layers, the two representations occupy nearly orthogonal subspaces with negligible shared structure.

Confidence Estimation in Language Models

Confidence estimation in language models measures whether a model’s uncertainty signals predict answer correctness and support reliable decisions.

See all 225 papers

ORCE: Order-Aware Alignment of Verbalized Confidence in Large Language Models
May 12, 2026Chen Li, Xiaoling Hu, Songzhu Zheng +2Confidence Estimation in Language Models Language Model Calibration
Stony Brook University, NY, USA · Massachusetts General Hospital and Harvard Medical School, MA, USA · Morgan Stanley, NY, USA
Wired for Overconfidence: A Mechanistic Perspective on Inflated Verbalized Confidence in LLMs
Apr 1, 2026Tianyi Zhao, Yinhan He, Wendy Zheng +2Confidence Estimation in Language Models Language Model Calibration
University of Virginia
Quantifying Faithful Confidence Expression in Large Reasoning Models
Jun 2, 2026Areeb Gani, Asal Meskin, Gabrielle Kaili-May Liu +1LLM Evaluation Confidence Estimation in Language Models
1Yale University

May 12, 2026J/K move · Enter open · S save

ORCE: Order-Aware Alignment of Verbalized Confidence in Large Language Models

Chen Li, Xiaoling Hu, Songzhu Zheng, Jiawei Zhou+1

Stony Brook University, NY, USA · Massachusetts General Hospital and Harvard Medical School, MA, USA · Morgan Stanley, NY, USA

Confidence Estimation in Language Models Language Model Calibration LLM Reliability LLM Uncertainty Estimation

Open paper AskPDF arXiv

Large language models (LLMs) often produce answers with high certainty even when they are incorrect, making reliable confidence estimation essential for deployment in real-world scenarios. Verbalized confidence, where models explicitly state their confidence in natural language, provides a flexible and user-facing uncertainty signal that can be applied even when token logits are unavailable. However, existing verbalized-confidence methods often optimize answer generation and confidence generation jointly, which can cause confidence-alignment objectives to interfere with answer accuracy. In this work, we propose a decoupled and order-aware framework for verbalized confidence calibration. Our method first generates an answer and then estimates confidence conditioned on the fixed question--answer pair, allowing confidence optimization without directly perturbing the answer-generation process. To align confidence with correctness likelihood, we construct a sampling-based surrogate from multiple model completions and optimize rank-based reinforcement learning objectives that encourage responses with higher estimated correctness likelihood to receive higher verbalized confidence. Experiments on reasoning and knowledge-intensive benchmarks show that our method improves calibration and failure prediction performance while largely preserving answer accuracy. These results demonstrate that verbalized confidence can be more reliably aligned by decoupling confidence estimation from answer generation and optimizing the relative ordering of confidence across responses.

Abstract

Confidence Estimation in Language Models

Explore similar work

ORCE: Order-Aware Alignment of Verbalized Confidence in Large Language Models

Wired for Overconfidence: A Mechanistic Perspective on Inflated Verbalized Confidence in LLMs

Quantifying Faithful Confidence Expression in Large Reasoning Models