Calibrated Confidence Estimation for Tabular Question Answering

Lukas Voss

cs.CLApr 14, 2026

Calibrated Confidence Estimation for Tabular Question Answering

Authors: Lukas Voss

Organizations: Independent Researcher

Abstract

Large language models (LLMs) are increasingly deployed for tabular question answering, yet calibration on structured data is largely unstudied. This paper presents the first systematic comparison of five confidence estimation methods across five frontier LLMs and two tabular QA benchmarks. All models are severely overconfident (smooth ECE 0.35-0.64 versus 0.10-0.15 reported for textual QA). A consistent self-evaluation versus perturbation dichotomy replicates across both benchmarks and all four fully-covered models: self-evaluation methods (verbalized, P(True)) achieve AUROC 0.42-0.76, while perturbation methods (semantic entropy, self-consistency, and our Multi-Format Agreement) achieve AUROC 0.78-0.86. Per-model paired bootstrap tests reject the null at p<0.001 after Holm-Bonferroni correction, and a 3-seed check on GPT-4o-mini gives a per-seed standard deviation of only 0.006. The paper proposes Multi-Format Agreement (MFA), which exploits the lossless and deterministic serialization variation unique to structured data (Markdown, HTML, JSON, CSV) to estimate confidence at 20% lower API cost than sampling baselines. MFA reduces ECE by 44-63%, generalizes across all four models on TableBench (mean AUROC 0.80), and combines complementarily with sampling: an MFA + self-consistency ensemble lifts AUROC from 0.74 to 0.82. A secondary contribution, structure-aware recalibration, improves AUROC by +10 percentage points over standard post-hoc methods.

Figures & tables

Model	N	Acc.	Conf.	Gap	smECE	ECE 10	AUROC
GPT-4o	2000	.762	.995	23.4pp	.345	.234	.522
DeepSeek-V3	2000	.693	.985	29.2pp	.404	.292	.543
Llama-3.3-70B	2000	.690	.985	29.6pp	.401	.302	.523
GPT-4o-mini	2000	.656	.955	29.9pp	.638	.300	.664
Gemini 2.5 Flash	2000	.655	.889	23.5pp	.400	.235	.812

Table 1: Verbalized confidence calibration on WikiTableQuestions. All models are severely overconfident, with smECE between 0.35 and 0.64, substantially worse than the

\sim

0.10–0.15 reported for textual QA ( Xiong et al., 2024 ) . The model with the highest accuracy (i.e., GPT-4o) shows the worst discrimination (AUROC 0.52). All AUROC point estimates have 95% bootstrap CI half-widths below

\pm 0.02

Figure 1: Smooth reliability diagrams ( Błasiok and Nakkiran, 2024 ) with 90% bootstrap confidence bands for verbalized confidence. All models show curves well below the diagonal, confirming systematic overconfidence. GPT-4o clusters nearly all predictions above 95%.

Model	Method	Acc.	Conf.	ECE 10	AUROC
GPT-4o ( $n{=}500$ )	Verbalized	.762	.995	.234	.522
GPT-4o ( $n{=}500$ )	MFA	.792	.897	.105	.782
Gemini 2.5 Flash	Verbalized	.655	.889	.235	.812
Gemini 2.5 Flash	MFA	.741	.729	.088	.776
DeepSeek-V3	Verbalized	.693	.985	.292	.543
DeepSeek-V3	MFA	.723	.847	.124	.753

Table 2: Multi-Format Agreement (MFA) versus verbalized confidence across five models. MFA reduces ECE by 45–63%, improves accuracy by 1–9 percentage points, and improves AUROC substantially for models where verbalized confidence has near-random discrimination. GPT-4o is evaluated on the first 500 WTQ validation examples due to API budget constraints; all other models use

n{=}2000

Figure 2: Five confidence elicitation methods on Llama-3.3-70B. Self-evaluation methods (red) cannot exceed AUROC 0.6. Perturbation methods (blue) all achieve AUROC

\sim

0.74–0.75. MFA (hatched) matches output-perturbation methods at lower cost.

Model	Method	Acc.	ECE 10	AUROC [95% CI]	Calls/q
Llama-3.3-70B	Verbalized	.330	.586	.697 [.671, .722]	1
	P(True)	.330	.502	.624 [.587, .660]	2
	MFA (ours)	.343	.295	.776 [.744, .806]	4
	Self-Consistency	.425	.223	.854 [.828, .878]	5
	Semantic Entropy	.423	.145	.858 [.834, .882]	5
Gemini 2.5 Flash	Verbalized	.371	.316	.756 [.725, .787]	1

Table 3: Cross-model dichotomy on TableBench ( Wu et al., 2025 ) (

n

=836, all non-Visualization examples). All four fully-tested models replicate the self-evaluation versus perturbation dichotomy observed on WikiTableQuestions: self-eval methods (verbalized, P(True)) cluster at AUROC

0.42

–

0.76

(mean

0.62

), while perturbation methods (MFA, SC, SE) cluster at AUROC

0.78

–

0.86

(mean

0.81

). Statistical robustness: unbracketed AUROC values include 95% percentile bootstrap CIs (10K resamples on questions); the GPT-4o-mini SC and SE cells are reported as mean

{\pm}

std across 3 random seeds (42, 123, 456) to verify that stochastic-sampling variance is small, and we observe a per-seed std of only

{\pm}0.006

AUROC — substantially smaller than the per-cell question-level CI (

\pm 0.025

) and 30

\times

smaller than the dichotomy gap. P(True) is strictly CI-separated from every perturbation method on every model , including from MFA on the smallest-gap pair (gpt-4o-mini, P(True) upper bound

0.734

vs. MFA lower bound

0.742

). Verbalized confidence is strictly CI-separated on Llama and DeepSeek; on Gemini, verbalized is anomalously strong (§ 6 ) and its CI overlaps with all three perturbation methods, and on GPT-4o-mini verbalized overlaps with MFA only (SC and SE remain strictly separated). MFA reduces ECE by

41

–

57

% over verbalized and matches sampling baselines while using

20

% fewer API calls. Gemini’s P(True) falls below random (AUROC

0.423

, CI

[0.407,0.438]

), revealing systematic anti-correlation between self-assessed confidence and correctness. GPT-4o is shown for verbalized and P(True) only (MFA, SC, SE deferred to extended version due to OpenAI batch budget constraints); both GPT-4o cells are self-eval methods, and both fall within the self-eval cluster (

0.72

and

0.77

). DeepSeek-V3 SC/SE were not run due to Together-AI credit exhaustion (§ A ).

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Community	Calibration?	Tabular QA?
LLM calibration	✓Extensively	$\times$ Never
Text-to-SQL calibration	✓Emerging	✓SQL only
Tabular QA systems (TabLaP, STaR)	✓One method each	✓Extensively
Tabular QA benchmarks	$\times$ Never	✓Extensively
This work	✓Five methods compared	✓Two benchmarks

Appendix

Table 4: Research landscape at the intersection of calibration and tabular QA. The contribution of this paper is the first systematic comparison of multiple general-purpose confidence estimation methods on standard tabular QA benchmarks.

Model (Provider)	Verb	P(True)	MFA	SC	SE
Llama-3.3-70B (Together)	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
Gemini 2.5 Flash (Google)	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
GPT-4o-mini (OpenAI)	$\checkmark$	$\checkmark$	$\checkmark$	T †	T †
DeepSeek-V3 (Together)	$\checkmark$	T	$\checkmark$	–	–
GPT-4o (OpenAI)	$\checkmark^{\ddagger}$	T	–	–	–

Appendix

Table 5: Method-by-model coverage matrix for the experimental matrix in this paper. Each cell indicates whether a (model, method) pair is evaluated on WikiTableQuestions (

n{=}2000

), TableBench (

n{=}836

), both, or neither. Cells with

\checkmark

are evaluated on both benchmarks at the standard sample size. T indicates TableBench only; W indicates WTQ only; – indicates not run. Caveats:

\ddagger

marks GPT-4o MFA on WTQ at

n{=}500

(subset due to API budget).

\dagger

marks GPT-4o-mini SC and SE on TableBench, which are reported as mean

\pm

std across 3 random seeds (42, 123, 456); all other cells are single-seed point estimates with 95% bootstrap CIs. The two empty DeepSeek-V3 SC and SE cells correspond to runs that could not be completed due to Together AI credit exhaustion on the DeepSeek-V3 endpoint.

Method	Type	Acc.	ECE 10	AUROC [95% CI]	Calls/q
Self-evaluation methods
Verbalized [ Xiong et al., 2024 ]	self-eval	.690	.302	.522 [.514, .533]	1
P(True) [ Kadavath et al., 2022 ]	self-eval	.716	.281	.568 [.550, .587]	2
Input/output perturbation methods
MFA (ours)	input-perturb	.696	.169	.737 [.717, .763]	4
Semantic Entropy [ Kuhn et al., 2023 ]	output-perturb	.804	.087	.753 [.724, .776]	5

Appendix

Table 6: All five confidence elicitation methods on Llama-3.3-70B (WikiTableQuestions,

n

=2000). A consistent pattern emerges: self-evaluation methods (verbalized, P(True)) achieve near-random AUROC, while perturbation methods (MFA, SE, SC) achieve AUROC

\sim

0.74–0.75 and reduce ECE by 50–70%. MFA matches the best baselines at 20% lower API cost. AUROC values include 95% percentile bootstrap CIs (10K resamples on questions); the self-eval and perturbation clusters are strictly CI-separated (P(True) upper bound .587 vs. MFA lower bound .717).

Model	Method	Acc.	Conf.	ECE 10	AUROC
Gemini 2.5 Flash	Verbalized	.655	.889	.235	.812
Gemini 2.5 Flash	P(True)	.745	.511	.240	.507
GPT-4o-mini	Verbalized	.656	.955	.300	.664
GPT-4o-mini	P(True)	.668	.582	.282	.608
Llama-3.3-70B	Verbalized	.690	.985	.302	.523
Llama-3.3-70B	P(True)	.716	.614	.281	.568

Appendix

Table 7: P(True) self-evaluation [ Kadavath et al., 2022 ] fails consistently across models on WikiTableQuestions. Despite a two-pass approach (i.e., generate answer, then ask the model to evaluate it), AUROC remains near random (i.e., 0.51–0.61) and is even worse than verbalized confidence for Gemini. Self-evaluation cannot recover the discriminative signal that input-perturbation methods provide.

Figure 3: P(True) self-evaluation fails consistently across three models. Even after the model is explicitly asked “Is your answer correct?” in a separate turn, AUROC remains near random (0.51–0.61). MFA (right bar in each group) provides 0.74+ AUROC across all three models.

Figure 4: Confidence CDFs for correct versus wrong answers (verbalized confidence). GPT-4o and Llama show near-complete overlap, which means confidence is uninformative. Gemini shows meaningful separation, which explains the higher AUROC.

Model	Pert. method	Self-eval	$\Delta$ AUROC	95% CI	$p_{\text{Holm}}$	Sig.
Best-vs-best (cluster maxima)
Llama-3.3-70B	SE (.858)	verb (.697)	$+0.162$	$[+0.128,+0.196]$	$<0.001$
Gemini 2.5 Flash	SE (.814)	verb (.756)	$+0.058$	$[+0.027,+0.089]$	$<0.001$
DeepSeek-V3	MFA (.852)	verb (.675)	$+0.177$	$[+0.142,+0.212]$	$<0.001$
GPT-4o-mini	SE (.835)	verb (.747)	$+0.088$	$[+0.046,+0.131]$	$<0.001$
Worst-vs-best (cluster minimum perturbation vs cluster maximum self-eval)

Appendix

Table 8: Per-model paired bootstrap significance tests on TableBench. The paper reports two complementary tests for the dichotomy. The best-versus-best test (top block) compares the strongest perturbation method against the strongest self-evaluation method on each model. The worst-versus-best test (bottom block) is a stronger form: the weakest perturbation cell on each model is compared against the strongest self-evaluation cell, which removes any selection bias toward the most favourable perturbation. Both tests use a paired bootstrap on questions (10K resamples), with each method evaluated against its own predicted answer; pairing is by question index, which preserves per-question correlation. Holm-Bonferroni correction is applied within each test family across the four model-level comparisons. CIs are for the difference

\Delta\text{AUROC}=\text{perturbation}-\text{self-eval}

Figure 5: MFA improvements generalize from WikiTableQuestions to TableBench, a substantially harder tabular QA benchmark (33% versus 69% accuracy). ECE is reduced by 50% on both datasets; AUROC improvements persist (+21pp on WTQ, +8pp on TableBench).

Method	ECE 10	Brier	AUROC
Raw verbalized	.315	.313	.518
Temperature scaling	.148	.238	.518
Platt scaling	.029	.216	.518
Isotonic regression	.032	.216	.518
Structure-aware (ours)	.031	.210	.619

Appendix

Table 9: Post-hoc recalibration on Llama-3.3-70B verbalized confidence (WTQ,

n

=1000 test). Standard methods (i.e., Platt, isotonic) reduce ECE to below 0.05 but cannot improve AUROC: discrimination is invariant to monotone recalibration. The structure-aware recalibration in this paper uniquely improves AUROC by incorporation of table features as additional covariates, which enables better identification of likely-incorrect answers.

Recalibration Features	ECE 10	AUROC
Confidence only (Platt scaling)	.032	.518
+ Table dimensions ( $\log$ rows, $\log$ cols)	.031	.588
+ Column type distribution (4 features)	.032	.519
+ Query complexity (word count, op keywords)	.034	.596
Full structure-aware (all 8 features)	.031	.619

Appendix

Table 10: Feature-group ablation for structure-aware recalibration on Llama-3.3-70B (50/50 train/test split,

n

=1000 each). Each structural feature group provides an incremental AUROC improvement; combination of all features gives the best result. Standard methods (i.e., confidence-only Platt) achieve identical AUROC to raw verbalized confidence (0.518), which confirms that monotone recalibration cannot improve discrimination.

Figure 6: Feature importance for structure-aware recalibration on Llama-3.3-70B (scaled logistic regression coefficients). Operation keywords and

\log

rows are the most influential features, both with negative coefficients: models are systematically less likely to be correct on complex questions over large tables.

Figure 7: ECE and Brier score before and after recalibration. Post-hoc methods reduce ECE to below 0.10 but cannot improve AUROC.

Comparison	$\Delta$ AUROC	95% CI	$p$	Sig.
MFA vs. Verbalized	$+0.217$	$[+0.193,+0.240]$	$<0.001$
MFA vs. P(True)	$+0.172$	$[+0.145,+0.198]$	$<0.001$
MFA vs. Self-Consistency	$-0.012$	$[-0.042,+0.019]$	$0.78$	ns
MFA vs. Semantic Entropy	$-0.011$	$[-0.042,+0.021]$	$0.75$	ns

Appendix

Table 11: Paired bootstrap significance tests on Llama-3.3-70B (10K resamples).

p

-values are reported with Holm-Bonferroni correction for

m{=}4

pairwise comparisons. MFA significantly outperforms the two self-evaluation methods (i.e., verbalized, P(True)) at

p<0.001

even after correction, and is statistically equivalent to the two sampling-based methods (i.e., SC, SE), at lower API cost. CIs are for the difference

\Delta\text{AUROC}=\text{MFA}-\text{other}

Model	$K$	Acc.	ECE 10	smECE	AUROC	Calls/q
Gemini 2.5 Flash	2	.720	.080	.110	.726	2
	3	.735	.077	.123	.765	3
	4	.741	.088	.154	.776	4
GPT-4o-mini	2	.655	.214	.292	.674	2
	3	.662	.194	.313	.721	3
	4	.666	.176	.304	.744	4

Appendix

Table 12: MFA

K

-ablation: results averaged over all

\binom{4}{K}

format subsets. The numbers show that

K{=}3

captures most of the benefit at 25% lower cost.

K{=}2

incurs a 0.05–0.07 AUROC drop relative to

K{=}4

, while

K{=}3

closes most of the gap. Practitioners can trade off cost versus marginal discrimination.

Model	Format Subset	AUROC	ECE 10
Gemini 2.5 Flash	Best $K{=}2$ (json+csv)	.720	—
	Mean over 6 $K{=}2$ pairs	.712	—
	$K{=}3$ Markdown + CSV + JSON	.765	.081
	$K{=}3$ Markdown + HTML + CSV	.762	.078
	$K{=}3$ HTML + JSON + CSV (most div.)	.770	.075
	$K{=}4$ All four formats	.776	.088

Appendix

Table 13: Format-diversity ablation: comparing

K{=}2

K{=}3

, and

K{=}4

MFA configurations. Inter-format agreement rates (averaged across models): Markdown&CSV (0.725), Markdown&JSON (0.706), HTML&JSON (0.721), JSON&CSV (0.682) — the more structurally diverse pairs (markup vs. delimited vs. key-value) tend to disagree more, providing a richer uncertainty signal. The best

K{=}2

pair is consistently json+csv (3 of 4 models) and the worst is consistently markdown+csv (3 of 4 models). At

K{=}3

, the “most diverse” subset (HTML+JSON+CSV, omitting Markdown) gives the highest AUROC for three of four models on the point estimates, consistent with the ensemble-diversity principle of Lakshminarayanan et al. [2017] , Jain et al. [2020] . Paired bootstrap testing (10K resamples) finds none of the inter-

K{=}3

-subset differences statistically significant after Holm-Bonferroni correction over 8 comparisons (best Holm-adjusted

p{=}0.09

K{=}2

shows a clear

0.05

–

0.08

AUROC drop relative to

K{=}4

, and we recommend

K{\geq}3

as the minimum useful budget;

K{=}4

is best when API spend permits since it strictly dominates every smaller subset.

Model	Method	ECE 10	AUROC	$w_{\text{MFA}}$
Gemini 2.5 Flash	MFA only	.095	.785	—
	P(True) only	.240	.507	—
	CISC (MFA + P(True))	.088	.792	0.7
GPT-4o-mini	MFA only	.165	.735	—
	P(True) only	.282	.608	—
	CISC (MFA + P(True))	.217	.746	0.5

Appendix

Table 14: CISC-style combination of MFA + P(True) confidence scores. The optimal weight

w

is fit on a held-out half (

n

=1000) and evaluated on the other half. Combining the two signals consistently improves AUROC over MFA alone (+0.007 to +0.013), confirming that P(True) contains useful complementary signal even though it fails as a standalone confidence score, as predicted by Taubenfeld et al. [2025] . Stability: across 5 different random 50/50 splits, the optimal

w

is exactly stable for Llama (

w{=}0.90{\pm}0.00

) and GPT-4o-mini (

w{=}0.30{\pm}0.00

); for Gemini the loss surface is flat near the optimum and

w

varies more across splits (

w{=}0.76{\pm}0.21

), but every recovered

w

still gives AUROC within

0.787{\pm}0.008

of the reported value, so the qualitative finding (P(True) adds complementary signal to MFA) is robust to split choice for all three models.

Question Type	$N$	Acc(V)	Acc(M)	AUROC(V)	AUROC(M)	$\Delta$ AUROC
Temporal	128	.812	.844	.516	.762	$+0.245$
Superlative	249	.735	.799	.528	.758	$+0.230$
Other	248	.746	.806	.511	.738	$+0.228$
Count/Sum	883	.581	.599	.523	.726	$+0.203$
Lookup	443	.747	.833	.537	.733	$+0.196$
Comparison	34	.706	.735	.550	.553	$+0.003$

Appendix

Table 15: MFA improvement by question type on Llama-3.3-70B (WikiTableQuestions). MFA improves AUROC across all major question types, with the largest gains on temporal reasoning (

+0.245

) and superlative questions (

+0.230

). The smallest gain is on comparison questions, which are also the smallest subset.

Method	AUROC	ECE 10	$\Delta$ AUROC	Weights
MFA alone	.728	.163	—	—
SC alone	.742	.192	—	—
SE alone	.729	.168	—	—
MFA + SC	.807	.173	$+0.065$	$w_{\text{MFA}}{=}0.55$
MFA + SE	.796	.159	$+0.054$	$w_{\text{MFA}}{=}0.65$
MFA + SC + SE	.819	.172	$+0.077$	$(0.5,0.4,0.1)$

Appendix

Table 16: Ensemble combinations of MFA with sampling-based methods on Llama-3.3-70B (50/50 train/test split,

n

=1000 each). Weights are optimized on the train half via grid search; metrics reported on the test half. MFA combined with sampling methods substantially outperforms either alone , supporting the hypothesis that input-perturbation (MFA) and output-perturbation (SC, SE) capture orthogonal uncertainty signals. The three-way ensemble achieves AUROC 0.819, an improvement of

+0.077

over the best single method (SC at 0.742). Stability: across 5 random 50/50 splits, the optimal weights are stable: MFA+SC

w_{\text{MFA}}{=}0.46{\pm}0.10

, MFA+SE

w_{\text{MFA}}{=}0.54{\pm}0.14

, three-way

(0.44,0.38,0.18){\pm}(0.05,0.04,0.07)

. Test-set AUROCs across the 5 splits are

0.807{\pm}0.010

0.795{\pm}0.007

, and

0.817{\pm}0.008

respectively, confirming the ensemble gains are not split-dependent.

Figure 8: Risk-coverage curves for the five elicitation methods on Llama-3.3-70B. Self-evaluation methods (dotted lines) produce nearly flat curves. Perturbation methods achieve substantially higher accuracy at low coverage. SC and SE reach 90% accuracy at

\sim

73% coverage; MFA reaches

\sim

80% accuracy at

\sim

66% coverage.

Method	Accuracy	ECE 10	AUROC
Verbalized (strict)	.660	.331	.522
Verbalized (fuzzy)	.690	.302	.522
MFA (strict)	.669	.196	.745
MFA (fuzzy)	.720	.148	.740

Appendix

Table 17: Sensitivity of calibration metrics to answer evaluation methodology on Llama-3.3-70B (WTQ). Strict matching uses only type-aware exact comparison; the fuzzy pipeline additionally extracts numbers from natural language and applies word-boundary containment. Accuracy drops 3–5 percentage points under strict matching (i.e., more answers marked wrong due to surface-level formatting), yet AUROC is essentially unchanged: the discrimination findings here are robust to evaluation methodology.

Model	Exact	Numeric	Fuzzy	Contain.	None
GPT-4o	189	173	8	9	116
GPT-4o-mini	688	513	13	73	688
Gemini	519	457	148	174	690
Llama-3.3-70B	729	566	10	48	621
DeepSeek-V3	716	562	24	58	614

Appendix

Table 18: Answer match type distribution for verbalized confidence on WTQ (v2 corrected evaluation).

Model	ECE 10	ECE 15	ECE 20	Brier	AUROC
GPT-4o (n=2000)	.234	.234	.234	.234	.522
GPT-4o-mini	.300	.300	.300	.304	.664
Gemini 2.5 Flash	.235	.235	.235	.182	.812
Llama-3.3-70B	.302	.302	.302	.300	.522
DeepSeek-V3	.292	.292	.292	.291	.543

Appendix

Table 19: Full calibration metrics for verbalized confidence on WikiTableQuestions.

Error Type	GPT-4o-mini	Gemini	Llama	DeepSeek
Aggregation	434	312	298	305
Filtering	98	87	92	89
Temporal	82	76	71	74
Format mismatch	—	174	48	58
Refusal	—	30	—	—
Unknown	150	95	120	102

Appendix

Table 20: Error type distribution for incorrect predictions (verbalized, WTQ).

Figure 9: MFA agreement distribution for each model on WTQ (blue) versus TableBench (red). WTQ saturates the all-agree bin; TableBench disperses across all bins, restoring the dynamic range of MFA.

Confidence Estimation in Language Models

Confidence estimation in language models measures whether a model’s uncertainty signals predict answer correctness and support reliable decisions.

See all 246 papers

A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering
May 8, 2026Zhanliang Wang, Jiancong Xiao, Ruochen Jin +3LLM Evaluation Confidence Estimation in Language Models
University of Pennsylvania, Philadelphia, PA · Dartmouth College, Hanover, NH
From token probabilities to calibrated confidence: An empirical study of mathematical question answering
Aug 8, 2026Avery Ma, Lorne Schell, Vin Bhaskara +1LLM Answer Verification Confidence Estimation in Language Models
RBC Borealis
On Calibration of Large Language Models: From Response To Capability
Feb 14, 2026Sin-Han Yang, Cheng-Kuang Wu, Chieh-Yen Lin +3LLM Evaluation Confidence Estimation in Language Models
Appier AI Research · National Taiwan University

May 8, 2026J/K move · Enter open · S save

A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering

Zhanliang Wang, Jiancong Xiao, Ruochen Jin, Shu Yang+2

University of Pennsylvania, Philadelphia, PA · Dartmouth College, Hanover, NH

LLM Evaluation Confidence Estimation in Language Models Language Model Calibration Model Calibration LLM Reliability LLM Uncertainty Estimation

Open paper AskPDF arXiv

Calibration measures whether a model's predicted confidence aligns with its empirical accuracy, and is central to the reliable deployment of large language models (LLMs) in high-stakes domains such as medicine and law. While much recent work focuses on improving LLM calibration, the equally important question of how to evaluate it in realistic settings remains underdeveloped. Open-ended question answering (QA), the most common deployment setting for modern LLMs, is where existing evaluation methods fall short: logit-based metrics need restricted output formats and internal probabilities; verbalized confidence is self-reported and often overconfident; and sampling-based methods rely on task-specific extraction rules without a clear finite-sample target. We introduce Sem-ECE (Semantic-Sampling Expected Calibration Error), a calibration evaluation framework for open-ended QA that samples answers from the model, groups them into semantic classes, and uses the resulting frequencies as confidence. We study two estimators within this framework: Sem

_1

-ECE, the same-sample self-consistency score, and Sem

_2

-ECE, a held-out variant that separates answer selection from confidence evaluation. We prove both are asymptotically unbiased, and further show that they agree on easy questions but diverge on hard ones with Sem

_2

achieving strictly smaller calibration error, so their gap also serves as a diagnostic for question difficulty. Experiments on three open-ended QA benchmarks across five leading commercial LLMs match our theoretical predictions and show that Sem-ECE outperforms verbalized confidence and existing sampling-based methods, while complementing logit-based evaluation when internal probabilities are unavailable.

Abstract

Confidence Estimation in Language Models

Explore similar work

A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering

From token probabilities to calibrated confidence: An empirical study of mathematical question answering

On Calibration of Large Language Models: From Response To Capability