Claim-Level Confidence Calibration for Reliable Decision Making with Large Language Models
Authors: Toghrul Abbasli, Kentaroh Toyoda, Yuan Wang, Li Chen
Organizations: Tsinghua University, Haidian District, Beijing, 100084, P. R. China · Vulcan Research, AIFT, Singapore · Keio Global Research Institute (KGRI), Japan · China Mobile Research Institute, China · Zhongguancun Laboratory, China
Large Language Models (LLMs) increasingly support decision-making in high-stakes domains, but they often hallucinate and express confidence that is misaligned with factual correctness. Response-level confidence is a coarse signal: a single generation can mix correct and incorrect statements, so a single number is not actionable for users that must accept, reject, or verify individual pieces of information. We study claim-level confidence calibration as a decision-relevant uncertainty signal: each response is decomposed into atomic, verifiable claims, and each claim is assigned a calibrated confidence using inference-time signals from consistency across samples and self-verification. Our framework operates in closed-box settings (no logits, no fine-tuning) and applies post-hoc calibration directly at the claim level, enabling selective intervention such as evidence retrieval or human review for low-confidence claims. Across TriviaQA and TruthfulQA we evaluate seven baselines on six recent models (Llama-3.1, Mistral, Qwen2.5, DeepSeek-R1, GPT-4, GPT-4o), and show that claim-level decomposition combined with post-hoc calibration reduces expected calibration error on factual questions while exposing failure modes on adversarial false-premise questions where decision-makers most need reliable uncertainty estimates.
Figures & tables
Figure 1. Proposed pipeline for claim-level confidence calibration. Given a question x , we (i) sample N candidate responses via greedy + stochastic decoding, (ii) decompose the primary response into atomic claims {c1,…,ck} , and (iii) score each claim with two complementary signals—verification ver(ci)∈[0,1] from self-evaluation and consistency freq(ci)=N1∑j1[y(j)⊨ci] from cross-sample support. The two signals are combined via a convex mixture p^(ci)=αfreq(ci)+(1−α)ver(ci) with α∗ tuned on held-out validation, and thresholded at τ to retain or flag each claim. A horizontal pipeline diagram with six color-coded stages flowing left to right: (1) a blue Question node showing an example query; (2) a steel-blue In-context reasoning node producing one greedy and N-1 stochastic samples, with three example sampled responses shown to the right; (3) a green Claim extraction node decomposing the primary response into atomic claims, with three example claims shown to the right; (4) two parallel branches—an orange Verification (ver) branch producing per-claim probabilities, and a mauve Consistency (freq) branch counting verifier-confirmed support across the N samples; (5) a purple Calibrated mixing node combining the two signals as a convex combination with weight alpha; (6) a crimson Per-claim confidence output node applying a retention threshold tau. A bottom legend explains each color's role and notes that verification is strong when the model self-assesses well, while consistency is strong on adversarial or false-premise inputs.
Models & Baselines
TriviaQA
TruthfulQA
↓ECE
↓smECE
↓BS
↑AUROC
↓AURC
BLEU
α
↓ECE
↓smECE
↓BS
↑AUROC
↓AURC
BLEU
α
Llama-3.1-8B-Instruct
Few-shot
0.046
0.040
0.286
0.532
0.523
0.041
–
0.386
0.333
0.165
0.794
0.001
0.190
–
Few-shot CoT
0.087
0.065
0.273
0.539
0.436
0.033
–
0.346
0.309
0.136
0.569
0.006
0.191
–
Verbal. CoT
0.113
0.064
0.293
0.564
0.413
0.033
–
0.499
0.348
0.349
0.161
0.005
0.191
–
PS
0.013
0.013
0.249
0.495
0.523
0.041
–
0.363
0.319
0.134
0.794
0.001
0.190
–
Table 1. Evaluation results for Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, Qwen2.5-7B-Instruct, DeepSeek-R1, GPT-4, and GPT-4o on TriviaQA and TruthfulQA. “Ours” rows report the test-set values at the per-model best α tuned on a held-out split (or α=0 when no held-out tuning was available). A comprehensive results table showing calibration metrics (ECE, smECE, BS, AUROC, AURC, BLEU) for six language models across two datasets (TriviaQA and TruthfulQA). Rows are grouped by model, with baselines including Few-shot, Few-shot CoT, Verbalized CoT, PS, TS, LMvsLM, HalluMeasure, and the proposed method. Bold values indicate best performance per model and dataset. The proposed method is highlighted in yellow.
Figure 2. Reliability diagrams (10 bins) for GPT-4o on TriviaQA. Bar colors and percentages indicate the proportion of points per bin. Additional models, datasets, and baselines are in Appendix D .
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Open-box?
Claim-level?
Consistency?
Calibration?
Fine-tuning ( 19 )
Yes
No
No
Yes
TS/PS ( 12 )
Logits
No
No
Yes
Verbalized ( 35 )
No
No
No
Limited
Self-consistency ( 40 )
No
No
Yes
No
SelfCheckGPT ( 30 )
No
No
Yes
No
LMvsLM ( 5 )
No
Yes
No
No
Appendix
Table 2. Comparison of uncertainty estimation methods for LLMs.
Expression
Score
Very low
0.0
Low
0.3
Somewhat low
0.45
Medium
0.5
Somewhat high
0.65
High
0.7
Appendix
Table 3. Mapping for qualitative confidence expressions and equivalent probability score for LLM.
Method
API calls/query
Relative cost
Few-shot / CoT
1
1×
TS / PS
1 + calibration
∼1×
Verbalized CoT
1
1×
HalluMeasure
1 + NLI ×nc
∼5×
LMvsLM
5 rounds × 2
∼10×
Ours
1 + 1 extract + nc verify
∼6×
Appendix
Table 4. Approximate inference cost per query in number of LLM API calls.
Dataset
Avg. claims/resp.
Range
TriviaQA
3–5
1–8
TruthfulQA
4–6
2–10
Appendix
Table 5. Average claims extracted per response.
Model / Dataset
Best α
Held-out ECE
GPT-4o / TriviaQA
1.0
0.022
DeepSeek-R1 / TruthfulQA
1.0
0.096
Llama-3.1 / TriviaQA
0.8
0.139
Qwen2.5 / TriviaQA
0.7
0.022
Appendix
Table 6. α sensitivity (held-out ECE) and best α per configuration.
Figure 3. Reliability diagrams for calibration methods using GPT-4o on TriviaQA. The number of diagram bins is 10. The color as well as the percentage number within each bar indicate the proportion of total points contained in each bin.
Figure 4. Reliability diagrams for calibration methods with 10 bins using DeepSeek-R1 on TriviaQA and TruthfulQA. The number of diagram bins is 10. The color and the percentage number on each bar indicate the proportion of total points contained in each bin.
Figure 5. Reliability diagrams for calibration methods with 10 bins using Llama 3.1 8B on TriviaQA. The color and the percentage number on each bar indicate the proportion of total points contained in each bin. A grid of seven reliability diagrams showing confidence calibration for Llama-3.1-8B-Instruct on TriviaQA. Each subplot displays predicted confidence versus actual accuracy with a diagonal reference line for perfect calibration. Bars are colored and labeled with percentages indicating the proportion of data points in each of 10 bins. Methods shown include sequence likelihood, CoT sequence likelihood, verbalized CoT, Platt scaling, temperature scaling, LMvsLM, and HalluMeasure.
Figure 6. Reliability diagrams for calibration methods with 10 bins using Mistral v0.3 7B on TriviaQA. The color and the percentage number within each bar indicate the proportion of total points contained in each bin.
Figure 7. Reliability diagrams for calibration methods using Qwen2.5 7B on TriviaQA. The number of diagram bins is 10. The color and the percentage number within each bar indicate the proportion of total points contained in each bin.
Figure 8. Reliability diagrams for different calibration methods with 10 bins using Llama 3.1 8B on TruthfulQA. The number of diagram bins is 10. The color and the percentage number on each bar indicate the proportion of total points contained in each bin.
Figure 9. Reliability diagrams for different calibration methods with 10 bins using Mistral v0.3 7B on TruthfulQA. The number of diagram bins is 10. The color and the percentage number on each bar indicate the proportion of total points contained in each bin.
Figure 10. Reliability diagrams for calibration methods using Qwen2.5 7B on TruthfulQA. The number of diagram bins is 10. The color and the percentage number within each bar indicate the proportion of total points contained in each bin.
Figure 11. Reliability diagrams for calibration methods using GPT-4o on TruthfulQA. The number of diagram bins is 10. The color as well as the percentage number within each bar indicate the proportion of total points contained in each bin.
Accurate confidence estimation is critical for reliable use of large language models (LLMs). Prior work on LLM calibration largely focuses on response-level confidence, which estimates the correctness of a single generated output. However, this formulation is misaligned with many practical settings where the central question is how likely a model is to solve a query overall. We show that this mismatch results from the stochastic nature of modern LLM decoding, under which single-response correctness fails to reflect underlying model capability. To address this issue, we introduce capability calibration, a new evaluation framework for measuring how well query-level confidence aligns with a model's expected accuracy on individual queries. We formally distinguish capability calibration (CC) from response calibration (RC) and show that the two differ both theoretically and empirically. We further show that CC is better suited than RC to applications like pass@k prediction and inference budget allocation. Finally, we evaluate common confidence estimation methods to understand the practical feasibility of CC.
As large language models (LLMs) are increasingly deployed in critical decision-making systems, the lack of reliable methods to measure their uncertainty presents a fundamental trustworthiness risk. We introduce a normalized confidence score based on output anchor token probabilities: classification labels for structured tasks and self-evaluation responses (Yes/No) for open-ended generation. This enables direct detection of errors and hallucinations with minimal overhead and without external validation. We make three key contributions. First, we propose a normalized confidence score and self-evaluation framework that exposes reliable confidence estimates for error detection across seven diverse benchmark tasks and five LLMs of varying architectures and sizes. Second, our theoretical analysis reveals that supervised fine-tuning (SFT) yields well-calibrated confidence through maximum-likelihood estimation, whereas reinforcement learning methods (PPO, GRPO) and DPO induce overconfidence via reward exploitation. Third, we propose post-RL SFT with self-distillation to restore confidence reliability in RL-trained models. Empirical results demonstrated that SFT improved average confidence-correctness AUROC from 0.806 to 0.879 and reduced calibration error from 0.163 to 0.034 on Qwen3-4B, while GRPO and DPO degraded confidence reliability. We demonstrated practical value through adaptive retrieval-augmented generation (RAG) that selectively retrieves context when the model lacks confidence, using only 58% of retrieval operations to recover 95% of the maximum achievable accuracy gain on TriviaQA
Large language models (LLMs) tend to verbalize confidence scores that are largely detached from their actual accuracy, yet the geometric relationship governing this behavior remain poorly understood. In this work, we present a mechanistic interpretability analysis of verbalized confidence, using linear probes and contrastive activation addition (CAA) steering to show that calibration and verbalized confidence signals are encoded linearly but are orthogonal to one another -- a finding consistent across three open-weight models and four datasets. Interestingly, when models are prompted to simultaneously reason through a problem and verbalize a confidence score, the reasoning process disrupts the verbalized confidence direction, exacerbating miscalibration. We term this the "Reasoning Contamination Effect." Leveraging this insight, we introduce a two-stage adaptive steering pipeline that reads the model's internal accuracy estimate and steers verbalized output to match it, substantially improving calibration alignment across all evaluated models.
Miranda Muqing Miao, Lyle Ungar
University of Pennsylvania · Department of Computer and Information Science