Organizations: Department of Computer Science, UNC-Chapel Hill · Department of Mathematics, The University of Hong Kong · Department of Data Science, City University of Hong Kong
Humans express uncertainty verbally via markers (e.g., "possible," "likely"), yet most LLM uncertainty quantification (UQ) relies on costing likelihood- or consistency-based signals. From a cognitive perspective, accurate verbal uncertainty reflects metacognitive monitoring, representing knowledge boundaries ("knowing that you don't know") to support regulation and information seeking. In this paper, we investigate how LLMs diverge from humans in verbal uncertainty quantification and whether verbal markers can reliably quantify LLM uncertainty. We curate a corpus of human uncertainty markers from psychology and decision-science literature and benchmark LLMs against it. We observe that LLMs encode verbal uncertainty with numerical levels that differ substantially from those of humans. We then introduce METHODNAME, a novel optimization-based algorithm that learns an optimal uncertainty profile over uncertainty markers directly from LLM outputs. By fitting a marker-uncertainty mapping to best explain empirical correctness, METHODNAME discovers how much probability mass each verbal marker should convey, rather than estimating uncertainty via repeated sampling. METHODNAME enables a direct, marker-level comparison of confidence semantics between humans and LLMs, disentangling mismatch and revealing systematic confidence disparities in verbal expressions.
Figures & tables
Figure 1: Comparison of traditional uncertainty quantification (UQ) methods and our method VOCAL . Traditional UQ methods (sampling-based and logits-based) exhibit a gap with human uncertainty expressions. In VOCAL , UM-lookup tables derived from human data alone cannot fully capture model uncertainty, so they are optimized with the model’s confidence distribution to better align with its internal uncertainty expressions.
Figure 2: The results of verbal uncertainty quantification QVU-H with UM-Lookup table collected from human. QVU-H achieves non-trivial UQ performance in many cases, indicating that LLMs share similar uncertainty expression as humans to a certain degree.
Figure 3: The distributions of uncertainty markers expressed by LLMs. We show that advanced LLMs, such as GPT-4o, express uncertainty in a more diverse manner compared to small LLMs (e.g., Llama-3.1-8B-Instruct and Llama-3.2-3B-Instruct). This also reveals that small LLMs tend to be over confident.
Figure 4: The evaluation results of VOCAL when comparing with human-sourced UM-Lookup , i.e., QVU-H . It demonstrates that VOCAL produces LLM-tailored UM-Lookup table.
Figure 5: The evaluation results of VOCAL and multi-sample based UQ methods. It is worth noting that sampling-based methods rely on semantic consistency calculations, which are expensive and introduce latency in real-world deployment. It is shown that VOCAL achieves comparable performance to sampling-based UQ methods.
Dataset
Model
G-NLL
PPL
VOCAL
Trivia QA
GPT-4o
0.538
0.575
0.573
Qwen2.5-72B-Ins.
0.627
0.619
0.645
SciQ
GPT-4o
0.663
0.648
0.700
Qwen2.5-72B-Ins.
0.568
0.555
0.717
GSM-Hard
DeepSeek-V3.1
0.520
0.567
0.715
Qwen2.5-72B-Ins
0.507
0.580
0.679
Table 1: The comparison results between VOCAL and single-sample UQ baselines. It is shown that VOCAL is significantly better than these methods.
Figure 6: Cross-LLM transferability. LLMs share a substantial common structure in verbal uncertainty expression.
Phrase
GPT-4o Prob.
Human Prob.
absolutely certain
1.000
0.920
confident
0.839
0.900
positive
0.839
0.900
sure
0.839
0.830
i think
0.710
0.630
almost certain
0.677
0.920
Table 2: Mean probabilities of verbal uncertainty markers for GPT-4o and humans, sorted by the GPT-4o score. Row colors indicate the relationship between probabilities: uncolored for aligned values (within a 0.05 tolerance), and light gray where the GPT-4o probability is higher or the human probability is higher.
Method
Vanilla
w/ TS
w/ IR
GSM-Hard
GSM8K
MedQA
Avg
GSM-Hard
GSM8K
MedQA
Avg
GSM-Hard
GSM8K
MedQA
Avg
VOCAL
0.2212
0.0462
0.1269
0.1314
0.0545
0.0197
0.0339
0.0360
0.0538
0.0051
0.0255
0.0281
G-NLL
0.0731
0.1476
0.1516
0.1241
0.0314
0.0623
0.0906
0.0614
0.0346
0.0298
0.0542
0.0395
PPL
0.0686
0.2553
0.1351
0.1530
0.0952
0.1316
0.0691
0.0986
0.0591
0.0286
0.0312
0.0396
SE
0.7100
0.5490
0.7568
0.6719
0.3964
0.4589
0.4468
0.4340
0.0591
0.0390
0.0512
0.0498
PE
0.2006
0.0598
0.1311
0.1305
0.0454
0.0290
0.0448
0.0397
0.0691
0.0364
0.0649
0.0568
Table 3: Expected Calibration Error (ECE) of GPT-4o on GSM-Hard, GSM8K, and MedQA. Lower ECE indicates better calibration. For vanilla results, uncertainty scores are min-max normalized before computing ECE. Temperature scaling (TS) and isotonic regression (IR) are applied as post-hoc calibration methods.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Uncertainty Expression
Uncertainty Probability
Frequency (N)
Definite
0.990
447.0
Certain
0.962
905.0
Virtually certain
0.950
447.0
Almost certain
0.920
782.0
Absolutely certain
0.920
96.0
Very high chance
0.915
27.0
Appendix
Table 4: Full Lookup Table for Verbalized Uncertainty Expressions (VUE) with their associated probabilities and frequencies.
Figure 7: Verbal uncertainty prompting maintains general performance.
Figure 8: Verbal uncertainty marker distributions of LLMs.
Figure 9: Ablation study on train samples and γ measured by AUROC.
Phrase
Probability
absolutely certain
1.000
i’m sure
1.000
pretty sure
1.000
quite certain
1.000
confident
0.839
positive
0.839
Appendix
Table 5: Verbal uncertainty markers and their mean probabilities for GPT-4o on the SciQ dataset, sorted by probability.
Phrase
Probability
best course of action
1.000
choice
1.000
given
1.000
highly probable
1.000
increased risk
1.000
most likely
1.000
Appendix
Table 6: Verbal uncertainty markers and their mean probabilities for GPT-3.5-Turbo on the MedQA dataset, sorted by probability.
Phrase
Probability
i know for a fact
1.000
pretty sure
1.000
but i think
0.960
inclined to say
0.960
not certain
0.960
quite certain
0.920
Appendix
Table 7: Verbal uncertainty markers and their mean probabilities for GPT-3.5-Turbo on the SciQ dataset, sorted by probability.
LLMs' linguistically expressed confidence should faithfully reflect their intrinsic uncertainty. While recent work shows LLMs struggle to use epistemic markers (e.g., "it is likely...") in a human-aligned fashion, it remains unclear whether models can apply their own linguistic confidence framework to associate markers with specific confidence levels in a stable and generalizable way, and how contextual features impact this ability. We conduct the first systematic study of this question, formalizing marker internal confidence (MIC) as the estimated intrinsic confidence a model associates with a specific epistemic marker in a given task domain. We present 7 metrics to evaluate the stability of MICs within and across distributions. Applying our analysis framework to diverse models and tasks, we find that LLMs remain faithfully miscalibrated even under model-centric interpretation of marker meanings, struggling to differentiate markers by internal confidence across distributions despite preserving a somewhat consistent ranking order across tasks. This supplies critical, complementary evidence to existing work toward a holistic understanding of faithful calibration in LLMs, emphasizing the need for more aligned and stable marker use to improve trustworthiness and reliability.
Large language models increasingly produce and interpret verbal probability expressions, yet whether these expressions carry consistent meaning across models (or match human perceptions of uncertainty) remains unknown. We present a systematic cross-model evaluation using a word-to-number mapping task grounded in established human benchmarks. Eleven uncertainty expressions were presented to 19 models under two conditions, forced single-number response and explanation elicitation, alongside a novel bidirectional roundtrip test of internal consistency. LLMs track the human benchmark with surprising fidelity: word ordering is preserved, three anchor points are recovered, and possible'' shows the highest variance and cross-model disagreement of any expression tested, consistent with its documented bimodal interpretation in humans. However, models show a systematic upward bias for negative expressions such as unlikely'' and ``improbable.'' Explanation elicitation reduces within-model variance while increasing between-model divergence, stabilizing individual models at the cost of inter-model consensus, and the roundtrip experiment reveals clear stratification, with frontier models maintaining coherent bidirectional representations. LLMs thus reproduce the structure of human verbal probability cognition, including its biases, while diverging systematically at the negative end---with implications for any setting where humans and models exchange probabilistic language.
Department of Computer and Information Sciences Temple University Philadelphia, PA, USA · Department of Informatics and Networked Systems University of Pittsburgh Pittsburgh, PA, USA
Verbalized confidence, in which LLMs report a numerical certainty score, is widely used to estimate uncertainty in black-box settings, yet the confidence scale itself (typically 0--100) is rarely examined. We show that this design choice is not neutral. Across six LLMs and three datasets, verbalized confidence is heavily discretized, with more than 78% of responses concentrating on just three round-number values. To investigate this phenomenon, we systematically manipulate confidence scales along three dimensions: granularity, boundary placement, and range regularity, and evaluate metacognitive sensitivity using meta-d′. We find that a 0--20 scale consistently improves metacognitive efficiency over the standard 0--100 format, while boundary compression degrades performance and round-number preferences persist even under irregular ranges. These results demonstrate that confidence scale design directly affects the quality of verbalized uncertainty and should be treated as a first-class experimental variable in LLM evaluation.