Large language models (LLMs) often produce confident yet incorrect answers, which can lead to risky failures in real-world applications. We study whether post-training can make a model's self-assessment explicit: when the model is uncertain, can it be trained to signal so within its own response? A central design question is where in the response this signal should be exposed -- during reasoning, while the answer is still being formed, or at the end, once the answer has been produced. We study both. For end-of-reasoning self-assessment, we train the model to verbalize a confidence score for its response, with the aim of high confidence on correct answers and low confidence on incorrect ones. For during-reasoning self-assessment, we train the model to emit the marker <uncertain> whenever its current reasoning state appears unreliable. Across factual reasoning tasks, both forms sharply reduce overconfident errors while improving answer quality, and both can be used as triggers for retrieval augmented generation (RAG) to improve the final response. We further analyze their internal mechanisms: end-of-reasoning verbalized confidence sharpens a confidence-related structure already present in the pretrained model, whereas during-reasoning <uncertain> emission teaches the model to mark high-risk reasoning steps, with parameter changes concentrated in the model's late layers.
Figures & tables
Figure 1: We train LLMs to express uncertainty at two points in the response: during reasoning by emitting <uncertain> at risky steps, and after reasoning by verbalizing a confidence score for the final answer. Both signals can trigger retrieval or abstention, while our analysis suggests that confidence training sharpens an existing confidence-related structure and <uncertain> training teaches the model to mark high-risk reasoning states through late-layer changes.
Figure 2: Logit-lens analysis of the confidence-token hidden state. Calibration sharpens late-layer confidence routing and yields a cleaner final-layer confidence structure.
Figure 3: PCA analysis of the confidence-token hidden state, colored by verbalized confidence.
Error type
Base
Cal.
Epistemic
92.4%
34.9%
Aleatoric
7.6%
65.1%
Strict epi.
88.6%
3.9%
Table 4
Figure 5
Figure 5: Token-level KL by position type. Both objectives concentrate distributional change at their signal positions, but the <uncertain> marker has a broader local footprint.
Figure 6: Layer-wise CKA between base and calibrated models. Verbalized confidence preserves representation geometry, whereas the <uncertain> marker induces increasing late-layer divergence.
A. Verbalized confidence
B. <uncertain> marker
Method
EM
F1
Brier
ECE
OConf
Method
Emit
Prec.
Recall
Acc ¬t
Wrong/Pos.
Base
24.5
37.3
-0.108
0.357
88.5
Emit heur.
0.336
0.959
0.444
0.392
0.726
P(True)
24.4
37.2
-0.096
0.340
39.7
Hidden probe
0.699
0.889
0.856
0.653
0.726
Global TS
24.5
37.3
+0.116
0.185
69.4
Output clf.
0.925
0.754
0.961
0.622
0.726
ATS
24.5
37.3
+0.123
0.166
53.4
Self-RAG
0.444
0.861
0.478
0.250
0.799
SFT-Conf
21.1
33.5
+0.083
0.226
7.3
FLARE
0.586
0.738
0.598
0.300
0.722
Table 3: Panel A evaluates verbalized confidence; OConf is the percentage of wrong answers with confidence >0.5 . Panel B evaluates <uncertain> marker triggers; Prec. = P(wrong ∣ trigger), Recall = P(trigger ∣ wrong), and Acc ¬t is accuracy on untriggered examples.
HotpotQA
MuSiQue
2WikiMultiHop
NQ
TriviaQA
Overall
Method
EM
F1
T
EM
F1
T
EM
F1
T
EM
F1
T
EM
F1
T
EM
F1
T
No-Ret
23.4
31.9
–
5.6
10.3
–
16.4
20.1
–
29.2
42.0
–
54.8
61.2
–
25.9
33.1
–
SR-7B
4.2
16.6
57.8
0.6
5.2
56.0
4.4
14.8
45.6
17.8
22.0
18.0
5.6
24.9
53.6
6.5
16.7
46.2
SR-13B
2.6
16.0
42.4
0.8
6.1
39.8
3.4
16.6
30.0
30.8
36.9
5.4
6.4
36.5
29.8
8.8
22.4
29.5
Ret-All
34.0
44.9
100
9.8
17.4
100
30.0
34.7
100
25.4
35.8
100
43.0
49.1
100
28.4
36.4
100
ADARAGUE
27.8
37.2
57.0
9.6
15.9
53.8
20.6
25.8
57.4
28.6
40.0
21.8
52.2
58.1
29.0
27.8
35.4
43.8
Table 4: Adaptive RAG evaluation results. EM, F1 (both %), and trigger rate T (%) are shown per dataset when available. Darker shaded cells indicate stronger EM/F1 performance within each metric column; yellow marks the highest selective trigger rate, excluding Ret-All .
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: KL mass fractions by token type. These plots complement the main-text boxplots by showing total KL allocation rather than per-token enrichment. Because reasoning spans are much longer than uncertainty spans, raw mass fractions should be interpreted together with the per-token statistics in the main text.
Figure 8: Hidden-state patching at the signal position versus reasoning positions. In both methods, patching the reasoning trace is at least as disruptive as patching the signal token itself, which is consistent with uncertainty being assembled across the trajectory and only exposed at the designated output position.
Figure 9: Relative Frobenius weight drift across layers and module types. Both calibrated models show similar update structure in parameter space, with the largest changes in v_proj , o_proj , and MLP projection layers. This makes the difference in representation geometry especially noteworthy: similar magnitudes of weight drift yield very different geometric consequences.
Figure 10: Mechanism-to-behavior linkage for verbalized confidence. Localization-related features predict per-example confidence shifts with cross-validated R2=0.51 , indicating that the strength of the learned confidence mechanism varies meaningfully across examples rather than appearing only as a population-level average.
Figure 11: Mechanism-to-utility linkage for the <uncertain> marker. Under the current proxy utility target, the <uncertain> -marker model shows weak and unstable within-subset linkage. This likely reflects a mismatch between the current utility proxy and the model’s operative mechanism, which is more naturally framed as a binary emission decision than as graded variation within already-emitting examples.
Figure 12: Detailed binned routing view for verbalized confidence calibration. In the base model, correct answers and many wrong answers both terminate with dominant mass in the High confidence bin. After calibration, low-confidence errors are redirected away from High and into Low , while correct answers remain more conservative. This makes the main mechanism visually explicit: calibration sharpens the late-stage mapping from hidden states to confidence outputs rather than uniformly lowering confidence everywhere.
Figure 13: Final-layer PCA of the confidence-token hidden state, grouped by outcome type. In the base model, correct and wrong low-confidence examples remain substantially mixed. After calibration, the representation aligns more cleanly with outcome structure: wrong low-confidence cases concentrate on the low-confidence side of the structure, while correct examples occupy the higher-confidence region more consistently.
Metric
Llama-3-8B (base)
Llama-3-8B (calibrated)
Accuracy ( ↑ )
0.345
0.358
Avg. verbalized confidence
0.869
0.403
Overconfidence gap (conf − acc) ( ↓ )
+0.523
+0.045
ECE ( ↓ )
0.383
0.049
Brier score ( ↓ )
0.504
0.166
NLL (confidence) ( ↓ )
4.987
0.498
Appendix
Table 5: Calibration metrics for Llama-3-8B before and after calibration training.
Figure 14: Training dynamics and calibration quality of the calibrated Llama-3-8B model.
Accuracy
ECE
AUSC
Overconf.
Conf. on wrong
Dataset
n
Base
Cal.
Base
Cal.
Base
Cal.
Base
Cal.
Base
Cal.
2WikiMultihopQA
500
22.4
28.0
0.407
0.164
0.267
0.458
84.8
13.3
0.823
0.343
HotpotQA
500
35.7
35.6
0.459
0.085
0.477
0.536
92.2
4.0
0.854
0.293
MuSiQue
500
12.1
14.4
0.543
0.089
0.126
0.224
84.6
0.0
0.809
0.207
NQ
500
46.5
48.2
0.367
0.033
0.560
0.625
96.3
0.4
0.888
0.426
TriviaQA
500
63.9
68.4
0.242
0.211
0.749
0.838
94.4
0.6
0.866
0.329
Appendix
Table 6: Dataset-level summary for verbalized-confidence calibration. “Overconf.” denotes the fraction of wrong answers with confidence >0.5 .
Table 7: Supplementary diagnostics for verbalized-confidence calibration.
Accuracy
Answer Line
Emit Rate
Wrong / Correct + Emit
Dataset
Base
Calibrated
Δ
Base
Calibrated
Base
Calibrated
W+E
C+E
2WikiMultihopQA
12.4
25.6
+13.2
51.4
100.0
43.2
59.8
43.2 / 55.4
0.0 / 4.4
HotpotQA
22.0
27.8
+5.8
62.8
99.6
36.6
70.8
35.8 / 59.0
0.8 / 11.8
MuSiQue
4.2
6.6
+2.4
50.8
100.0
49.8
94.6
49.8 / 89.0
0.0 / 5.6
NQ
21.6
40.8
+19.2
77.0
100.0
23.6
56.6
23.0 / 39.6
0.6 / 17.0
TriviaQA
36.2
56.0
+19.8
72.0
100.0
31.8
44.2
30.2 / 31.6
1.6 / 12.6
Appendix
Table 8: Base model vs. calibrated model on six factual reasoning datasets. “Answer Line” is the fraction of responses containing an explicit final answer line. “Emit Rate” is the overall <uncertain> emission rate. Each entry in the last two columns is Base / Calibrated : the fraction of wrong answers co-occurring with emission (W+E) and the fraction of correct answers co-occurring with emission (C+E). All values are percentages.
Table 9: Probe diagnostics for the <uncertain> marker.
Figure 15: Layer-wise probe performance for retrieval triggering.
A. Verbalized confidence reward ablation
Setting
Reward / format
Acc.
Brier
ECE
Overconf.
Gap
Base prompt
none, decimal
24.5
-0.337
0.647
89.4
+0.111
Final design
Brier, decimal
27.4
+0.110
0.133
3.0
+0.330
Format variant
Brier, integer-style
24.6
+0.043
0.234
25.5
+0.611
B. Local marker reward ablation
Setting
Reward order {CEˉ,CE,WE,WEˉ}
Acc.
Emit
Rec.-err.
Sep.
Len.
Appendix
Table 10: Reward ablations for the two methods. Verbalized-confidence rows use a 500-example held-out set; <uncertain> -marker rows use a 1,100-example held-out set. Values are percentages except Brier, confidence gap, separation gap in percentage points, and response length. In Panel B, C / W denote correct/wrong and E / Eˉ denote marker emission/no emission.
A. Verbalized confidence
Family
Recipe
Acc.
Brier
ECE
Overconf.
Gap
Conf. support
Llama-3-8B
Brier GRPO
27.4
+0.110
0.133
3.0
+0.330
7 values
Qwen2.5-7B
Brier GRPO
21.1
+0.039
0.039
7.2
+0.636
8 values
B. <uncertain> marker
Family
Recipe
Acc.
Emit
Rec.-err.
Sep.
Mean len.
CNE / CE / WE / WNE
Llama-3.1-8B
marker GRPO
47.1
57.7
72.9
+32.7
478
27.9 / 19.2 / 38.5 / 14.4
Appendix
Table 11: Cross-family robustness results. Panel A uses the verbalized-confidence held-out set ( n=500 ); Panel B uses the <uncertain> -marker held-out set ( n=1100 ). Values are percentages except Brier, confidence gap, separation gap in percentage points, and mean response length. CNE, CE, WE, and WNE denote correct/no-emission, correct/emission, wrong/emission, and wrong/no-emission.
Figure 16: Qualitative examples of epistemic and aleatoric errors before and after GRPO calibration training. Epistemic errors (Examples 1–2) are wrong answers delivered with high stated confidence and no uncertainty signal in the response text; they represent knowledge gaps the model fails to recognise. Aleatoric errors (Example 3) are wrong answers accompanied by explicit hedging in the reasoning, indicating the model correctly identifies the limits of its knowledge. Error type is determined by the response content independently of the confidence number, motivating the use of an LLM judge (see § F.1 ) in addition to the verbalized confidence threshold.
Figure 17: Qualitative examples for the <uncertain> marker, which signals uncertainty via an explicit reasoning-time marker rather than a verbalized confidence score. Epistemic errors (Examples 1–2) occur when the model produces a wrong answer without emitting <uncertain> in the reasoning chain; the baseline (Example 1) never uses the token, while the trained model (Example 2) still fails to trigger it for short, over-confident responses. Aleatoric errors (Example 3) occur when <uncertain> is correctly inserted at the knowledge boundary, signalling that the error is not a hallucination but an honest absence of information. Unlike the verbalized confidence setting (Fig. 16 ), the uncertainty signal here is binary and localised : the marker either appears or it does not, and its position in the chain-of-thought identifies the specific reasoning step that fails.
Reliable Large Language Models (LLMs) should abstain when confidence is insufficient. However, prior studies often treat refusal as a generic "I don't know'', failing to distinguish input-level ambiguity (data uncertainty) from capability limitations (model uncertainty). This lack of distinction limits downstream action decisions like requesting clarification or invoking external tools. In this work, we introduce UA-Bench, a benchmark of over 3,500 questions drawn from six datasets spanning knowledge-intensive and reasoning-intensive tasks, designed to evaluate explicit uncertainty attribution. An evaluation of 18 frontier LLMs shows that even state-of-the-art models struggle to reliably discriminate between data uncertainty and model uncertainty, and that high answer accuracy does not necessarily imply strong uncertainty attribution ability. To narrow this gap, we propose a lightweight data synthesis and reinforcement learning strategy. Experiments on both Qwen3-4B-Instruct-2507 and Qwen3-8B in thinking mode show that the proposed method improves uncertainty attribution while preserving answer accuracy. Our code and data are publicly available now.
Jingyi Ren, Ante Wang, Yunghwei Lai +5
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China · Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China
Reliable deployment of large language models (LLMs) requires accurate uncertainty estimation. Existing methods are predominantly answer-first, producing confidence only after generating an answer, which measure the correctness of a specific response and limits practical usability. We study a confidence-first paradigm, where the model outputs its confidence before answering, interpreting this score as the model's probability of answering the question correctly under its current policy. We propose CoCA(Co-optimized Confidence and Answers), a GRPO reinforcement learning framework that jointly optimizes confidence calibration and answer accuracy via segmented credit assignment. By assigning separate rewards and group-relative advantages to confidence and answer segments, CoCA enables stable joint optimization and avoids reward hacking. Experiments across math, code, and factual QA benchmarks show improved calibration and uncertainty discrimination while preserving answer quality, thereby enabling a broader range of downstream applications.
Changcheng Li, Jiancan Wu, Hengheng Zhang +5
University of Science and Technology of China · Huawei Inc.
Large language models (LLMs) demonstrate remarkable performance across diverse tasks, but they often generate responses that appear plausible while being factually incorrect. This problem is compounded by the lack of explicit uncertainty estimates, which makes it difficult for users to judge the reliability of model outputs. Existing uncertainty quantification methods typically rely on indirect signals, such as entropy across sampled generations. These signals can be difficult to interpret and do not fully leverage the model's ability to assess its own uncertainty. We propose a simple yet effective self-assessment method for uncertainty quantification in LLMs. Our approach groups sampled generations into semantically distinct clusters, converts them into answer options in a structured multiple-choice question, and uses the probability assigned by the LLM to each option as a confidence estimate. Experiments across multiple models and datasets show that our method consistently outperforms baseline approaches. Notably, it achieves competitive performance with as few as two additional samples, demonstrating both its effectiveness and efficiency.