Users often ask large language models (LLMs) to report how confident they are, but it is unclear whether such linguistic confidence tracks the model's internal confidence. We study this question across 8 classification tasks, 2 generation tasks and 30 models from three families. For classification, we compare linguistic confidence with logits-based confidence along three axes: association, magnitude agreement and calibration. For generation, we test whether linguistic confidence tracks semantic-entropy-based uncertainty. The axes frequently diverge. Instance-level association is weak on average, although it improves on easier items and for stronger base models. Instruction-tuned models often report higher confidence and sometimes show higher association, but they also have larger confidence gaps and worse calibration. Prompt design mostly changes the distribution of reported confidence. Attitude cues inflate confidence without improving alignment, while score exemplars can preserve rank-order signal when they avoid collapsed confidence values. Regression analyses show that distributional properties of confidence scores explain much of the observed alignment pattern, with model metadata playing a smaller role after controls. These results support a lossy-channel view of linguistic confidence. A more dispersed verbal confidence distribution can carry useful rank information, but it does not make the scores calibrated. Linguistic confidence should therefore be evaluated with multi-axis diagnostics before being used in downstream reliability pipelines.
Figures & tables
Figure 1: Mean logits-based confidence and mean linguistic confidence across task, model and prompt settings. Each point is one setting. Instruction-tuned models tend to have higher values on both axes. The overall association is statistically significant, but the split by model type shows that base models have weak positive association while instruction-tuned models have almost no aggregate association.
Figure 2: Task difficulty and confidence association. Tasks are ordered by decreasing normalized accuracy in the top panel. The bottom panel shows the Pearson correlation between linguistic confidence and logits-based confidence. Markers show per-task means and error bars show ±1 standard deviation across models. Association tends to weaken as task difficulty increases.
Figure 3: Model size and association between linguistic and logits-based confidence. Association tends to rise with model size for base models, but it plateaus or decreases for instruction-tuned models.
Figure 4: Representative distributions of logits-based and linguistic confidence on CoLA for Mistral-v0.1 models. The top row shows base models under Base Prompt 1, and the bottom row shows instruction-tuned models.
Metric
mean instruct
mean base
t
p
r
0.172
0.118
1.95
5.4e-02
Distance
0.271
0.241
2.03
4.4e-02
ECE(linguistic)
0.287
0.191
6.02
2.5e-08
Table 1: Paired t-test results comparing instruction-tuned and base models. Underlined values indicate significant differences with p<0.05 .
Outcome
n
Predictors
R2
Adj. R2
Dist
363
28
0.570
0.536
ECE
363
28
0.644
0.615
Correlation
363
28
0.498
0.459
Table 2: OLS fit statistics for distance, confidence-to-confidence ECE and correlation between linguistic and logits-based confidence.
Method
Accuracy
Entropy
Majority voting
0.7311
N/A
Level confidence
0.7531
0.4362
Score confidence
0.7830
1.6896
Table 3: Confidence-weighted aggregation on eight classification tasks using gpt-4.1-nano . Finer-grained confidence gives a more dispersed weighting signal and improves accuracy in this setting.
Regime
Diagnostic
Recommended use
Collapsed verbal scores
Scores cluster near 9–10, or have low standard deviation
Avoid using verbal confidence for fine-grained decisions
High verbal confidence with weak internal confidence
Linguistic confidence is much higher than logits-based confidence
Flag, abstain, rerank, or sample additional answers
Dispersed verbal scores with validated association
Held-out data show positive association with correctness or internal confidence
Use as a ranking or weighting signal, not as a calibrated probability
Prompt-induced confidence shifts
Tone or framing raises confidence without improving association
Treat as confidence inflation, not improved reliability
Table 4: Practical interpretation of linguistic confidence under different diagnostic regimes.
Appendix figures & tables33 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Base Prompt 1 example.
Figure 6: Base Prompt 2 example.
Figure 7: Instruct prompt with example confidence score.
Figure 8: Instruct prompt without example confidence score.
Figure 9: Example prompt for CoQA.
Figure 10: Example prompt for TriviaQA.
Figure 11: Base Prompt without asking for linguistic confidence.
Figure 12: Instruct prompt without asking for linguistic confidence.
Figure 13: Attitude cues for prompt perturbation.
Task / pool
Pool n
Median ∣Δ∣
Within ±0.10
Full in 95% interval
Conclusion-level metrics
CoLA
1,043
0.024
95.3%
117/117
45/45
QNLI
5,463
0.027
93.9%
116/117
45/45
QQP
5,000
0.027
94.1%
115/117
45/45
MMLU
1,000
0.026
94.4%
116/117
45/45
Temporal Sequences
800
0.024
95.3%
117/117
45/45
Appendix
Table 5: Stability of estimates from 500 random 200-example subsamples.
Base Model
Instruction-tuned / Chat
Llama-2-7b-hf
Llama-2-7b-chat-hf
Llama-2-13b-hf
Llama-2-13b-chat-hf
Llama-3.1-8B
Meta-Llama-3.1-8B-Instruct
Llama-3.2-3B
Llama-3.2-3B-Instruct
Meta-Llama-3-8B
Meta-Llama-3-8B-Instruct
Mistral-7B-v0.1
Mistral-7B-Instruct-v0.1
Appendix
Table 6: List of models used in our experiments, categorized by version and type.
Figure 14: Effect of model size on the correlation between linguistic and logits-based confidence, with error bars indicating the standard deviation of correlation values within each model group. Correlation tends to rise with model size for base models but plateaus for instruction-tuned ones.
Figure 15: Base models: model size (x-axis, in B) vs. (a) Distance between two confidence measures and (b) ECE of linguistic confidence.
Figure 16: Instruction-tuned models: model size (x-axis, in B) vs. (a) Distance between two confidence measures and (b) ECE of linguistic confidence.
Figure 17: Distribution of Logits-based and Linguistic Confidence on CoLA (Base Prompt 1) for Qwen2.5 Base Models.
Figure 18: Distribution of Logits-based and Linguistic Confidence on CoLA (Base Prompt 1) for Qwen3 Base Models.
Figure 19: Distribution of Logits-based and Linguistic Confidence on CoLA (Base Prompt 1) for Llama2 Base Models.
Figure 20: Distribution of Logits-based and Linguistic Confidence on CoLA for Qwen2.5 Instruct-tuned Models.
Figure 21: Distribution of Logits-based and Linguistic Confidence on CoLA for Llama2 Instruction-tuned Models.
Metric
LLaMA
Mistral
Qwen
pL→M
pM→Q
pL→Q
r
0.081
0.137
0.180
0.033
0.102
0.0004
Distance
0.274
0.259
0.218
0.287
0.0009
0.00002
ECE(linguistic)
0.261
0.219
0.186
0.024
0.070
0.0003
Appendix
Table 7: Paired t -tests across model families. Mean value of the three families and p between them are reported. Underlined values indicate significant differences ( p<0.05 ).
Metric
mean_a
mean_b
p
LLaMA3 → LLaMA3.1
r
0.057
0.126
0.028
Distance
0.272
0.281
0.527
ECE (linguistic)
0.197
0.234
0.058
Mistral-v0.1 → v0.2
r
0.125
0.116
0.741
Appendix
Table 8: Paired t -tests across model versions within each family. Underlined values indicate significant differences ( p<0.05 ). Metrics include overall correlation ( r ), distance between linguistic and logits-based confidence, and ECE based on linguistic confidence.
Figure 22: Comparison of average Pearson correlation and accuracy across different confidence score pairs (i,j) on two base prompt settings. Each heatmap shows how the model’s linguistic confidence and task performance vary with score combinations.
Figure 23: Average correlation (top) and task performance (bottom) across different confidence scoring schemes. The x -axis indicates the index of linguistic confidence scores (1–10), a random choice, or no confidence used.
Metric
mean a
mean b
t
p
Base Prompt 2 → Base Prompt 1
Normalized Accuracy
0.413
0.405
0.55
5.9e-01
Average Logits
0.596
0.569
8.22
2.1e-13
Logits Variance
0.112
0.108
3.08
2.5e-03
ECE(logits)
0.129
0.140
-1.51
1.3e-01
Base Prompt 1 → Base without Confidence
Appendix
Table 9: Paired t-test results comparing logits-based statistics under different prompting conditions. Mean values ( a,b ), t -statistics, and p -values are reported. Underlined values indicate significant differences ( p<0.05 ).
Metric
mean a
mean b
t
p
Origin → Criticism
Avg(Conf(li))
8.428
8.612
-2.34
0.028
r
0.266
0.146
3.07
0.0056
Origin → Approval
Avg(Conf(li))
8.428
8.807
-3.75
0.0011
r
0.266
0.130
2.43
0.0235
Appendix
Table 10: Paired t-test results comparing the effect of different attitude and calibration variants on average linguistic confidence Avg(Conf(li)) and its Pearson correlation with logits-based confidence r . Underlined values indicate significant differences ( p<0.05 ).
Llama Family
Mistral / Mixtral Family
Model
CoQA ( r )
TriviaQA ( r )
Model
CoQA ( r )
TriviaQA ( r )
Base
Instruct
Base
Instruct
Base
Instruct
Base
Instruct
Llama-2-7B
0.056
0.063
0.211
0.433
Mistral-7B-v0.1
− 0.040
− 0.017
0.288
0.392
Llama-2-13B
− 0.148
0.064
0.224
0.469
Mistral-7B-v0.2
− 0.008
0.213
0.251
0.338
Llama-3.1-8B
− 0.015
0.161
0.004
0.318
Mistral-7B-v0.3
− 0.048
0.012
0.033
0.549
Llama-3.2-3B
0.000
0.245
0.261
0.591
Mistral-Nemo-2407
− 0.050
− 0.015
0.355
0.472
Appendix
Table 11: Pearson correlation ( r ) between semantic-entropy-based and linguistic confidence on CoQA and TriviaQA. Higher correlations within each base/instruction-tuned pair are highlighted in bold; underlined values indicate statistical significance ( p<0.05 ).
Task
Base Prompt 1
Base Prompt 2
Logits Corr.
Entropy Corr.
Logits Corr.
Entropy Corr.
Conceptual Combinations
0.4441
-0.3111
0.8189
-0.1970
Ruin Names
0.4479
-0.1096
0.4142
-0.3364
Temporal Sequences
0.1246
0.0103
0.0431
-0.0401
CoLA
0.4469
0.2283
0.5672
0.1077
Cause and Effect
0.1413
0.0133
-0.0853
-0.3545
Appendix
Table 12: Correlation between linguistic confidence and internal metrics (logits-based confidence vs. negated semantic entropy) on Qwen2.5-14B under two base prompt templates.
Predictor
Coef
Std. β
SE
t
p
CI 2.5
CI 97.5
Linguistic_std
0.080817
0.780064
0.005486
14.733
3.02e-38
0.070026
0.091607
Num_choices
0.029610
0.286977
0.005590
5.297
2.14e-07
0.018613
0.040606
Logits_mean
-0.281884
-0.558972
0.053266
-5.292
2.19e-07
-0.386661
-0.177108
Linguistic_mean
0.019635
0.227861
0.005047
3.890
1.21e-04
0.009707
0.029563
Task_QNLI
0.041021
0.131019
0.015235
2.693
0.0074
0.011053
0.070989
Model_version_3
0.031787
0.127700
0.015214
2.089
0.0374
0.001860
0.061713
Appendix
Table 13: Full OLS coefficients for Dist_linguistic-logits under specified predictors.
Predictor
Coef
Std. β
SE
t
p
CI 2.5
CI 97.5
Logits_mean
-0.457089
-0.911912
0.045934
-9.951
1.29e-20
-0.547444
-0.366734
Linguistic_mean
0.033871
0.395460
0.004352
7.782
8.89e-14
0.025309
0.042432
Linguistic_std
0.032748
0.318015
0.004731
6.923
2.26e-11
0.023443
0.042053
Logits_std
-0.365030
-0.213445
0.073691
-4.954
1.16e-06
-0.509985
-0.220076
Num_choices
0.021411
0.208775
0.004821
4.441
1.22e-05
0.011928
0.030894
ECE_linguistic
0.159993
0.229478
0.038584
4.147
4.28e-05
0.084096
0.235890
Appendix
Table 14: Full OLS coefficients for ECE_linguistic-logits under specified predictors.
Predictor
Coef
Std. β
SE
t
p
CI 2.5
CI 97.5
Task_QNLI
-0.155245
-0.251365
0.038621
-4.020
7.27e-05
-0.231227
-0.079263
Model_size
0.011347
0.194214
0.003532
3.213
0.0014
0.004398
0.018296
Linguistic_std
0.041036
0.194311
0.014786
2.775
0.0058
0.011945
0.070126
Task_cause_and_effect
-0.107774
-0.171142
0.039022
-2.762
0.0061
-0.184544
-0.031003
Task_QQP
-0.102290
-0.167177
0.038298
-2.671
0.0080
-0.177638
-0.026943
Linguistic_mean
0.028679
0.164087
0.013086
2.192
0.0291
0.002934
0.054423
Appendix
Table 15: Full OLS coefficients for Correlation under specified predictors.
Predictor
Std.( β )
Model_size
-0.012746
Num_choices
0.143658
Linguistic_mean
0.201203
Linguistic_std
0.741303
Logits_mean
-0.496499
Logits_std
0.009902
Appendix
Table 16: Full ridge (CV) standardized coefficients for Dist_linguistic-logits .
Predictor
Std.( β )
Model_size
0.071043
Num_choices
0.080202
Linguistic_mean
0.381558
Linguistic_std
0.302399
Logits_mean
-0.843727
Logits_std
-0.215614
Appendix
Table 17: Full ridge (CV) standardized coefficients for ECE_linguistic-logits .
Predictor
Std.( β )
Model_size
0.113674
Num_choices
0.109605
Linguistic_mean
0.092039
Linguistic_std
0.087174
Logits_mean
-0.007872
Logits_std
0.036264
Appendix
Table 18: Full ridge (CV) standardized coefficients for Correlation .
Large language models can report a numerical confidence together with generated content, but it is unclear whether this report is more than calibrated rhetoric. We analyze three training-free signals: confidence verbalized with the answer, post-hoc P(True), and agreement with three additional generations on the same 100 TriviaQA questions for two model families. Direct verbalization is a surprisingly strong baseline: after auditing benchmark errors, it reaches AUROC 0.956 and 0.937 for correctness prediction. Three-sample agreement is substantially weaker (0.765 and 0.790), and a fixed interpolation with verbalized confidence has no statistically reliable benefit. Four of nine errors from one model and two of eight from the other receive unanimous sample support, showing that self-consistency can amplify shared misconceptions. Re-eliciting confidence for the same fixed answers with equivalent prompts changes scores by 0.043 to 0.084 on average and flips 4% to 9% of decisions at a 0.8 threshold. An exploratory audit of 100 confidence-tagged biography claims further finds only a modest confidence gap between supported and contradicted claims. These results argue that useful self-reports remain sensitive to elicitation, correlated errors, and benchmark noise.
LLMs' linguistically expressed confidence should faithfully reflect their intrinsic uncertainty. While recent work shows LLMs struggle to use epistemic markers (e.g., "it is likely...") in a human-aligned fashion, it remains unclear whether models can apply their own linguistic confidence framework to associate markers with specific confidence levels in a stable and generalizable way, and how contextual features impact this ability. We conduct the first systematic study of this question, formalizing marker internal confidence (MIC) as the estimated intrinsic confidence a model associates with a specific epistemic marker in a given task domain. We present 7 metrics to evaluate the stability of MICs within and across distributions. Applying our analysis framework to diverse models and tasks, we find that LLMs remain faithfully miscalibrated even under model-centric interpretation of marker meanings, struggling to differentiate markers by internal confidence across distributions despite preserving a somewhat consistent ranking order across tasks. This supplies critical, complementary evidence to existing work toward a holistic understanding of faithful calibration in LLMs, emphasizing the need for more aligned and stable marker use to improve trustworthiness and reliability.
Prior work has shown that instruction-tuned large language models (LLMs) are less well calibrated than their base pre-trained counterparts. However, little is known about the frequently used chat template's effect on the calibration of conversational LLMs. In this work, we investigate the mechanisms driving this miscalibration by decoupling the effects of the post-training algorithm and the chat format. We find that, while instruction tuning fundamentally harms calibration, the chat template aggravates the issue through an "ownership bias" -- models are significantly more confident in their own answers than in identical answers provided by a user. Extensive experiments across six recent open-weight LLMs, three benchmarks, and three confidence elicitation methods show that models assign up to 26% higher confidence to their own responses. Leveraging this insight, we propose a simple inference-time strategy: framing the model's answer as user input during confidence elicitation. This approach significantly reduces overconfidence and improves calibration by up to 26% without the need for retraining, narrowing the gap between base and instruction-tuned models.
Mario Sanz-Guerrero, Manuel Mager, Katharina von der Wense
∇Johannes Gutenberg University Mainz, Germany · ♠University of Colorado Boulder, USA