When Linguistic and Internal Confidence Diverge in Large Language Models

Hefan Zhang; Bingquan Zhang; Ming Cheng; Saeed Hassanpour; Weicheng Ma; Soroush Vosoughi

cs.CLAug 28, 2026

When Linguistic and Internal Confidence Diverge in Large Language Models

Authors: Hefan Zhang, Bingquan Zhang, Ming Cheng, Saeed Hassanpour, Weicheng Ma, Soroush Vosoughi

Organizations: Dartmouth College · Oakland University

Abstract

Users often ask large language models (LLMs) to report how confident they are, but it is unclear whether such linguistic confidence tracks the model's internal confidence. We study this question across 8 classification tasks, 2 generation tasks and 30 models from three families. For classification, we compare linguistic confidence with logits-based confidence along three axes: association, magnitude agreement and calibration. For generation, we test whether linguistic confidence tracks semantic-entropy-based uncertainty. The axes frequently diverge. Instance-level association is weak on average, although it improves on easier items and for stronger base models. Instruction-tuned models often report higher confidence and sometimes show higher association, but they also have larger confidence gaps and worse calibration. Prompt design mostly changes the distribution of reported confidence. Attitude cues inflate confidence without improving alignment, while score exemplars can preserve rank-order signal when they avoid collapsed confidence values. Regression analyses show that distributional properties of confidence scores explain much of the observed alignment pattern, with model metadata playing a smaller role after controls. These results support a lossy-channel view of linguistic confidence. A more dispersed verbal confidence distribution can carry useful rank information, but it does not make the scores calibrated. Linguistic confidence should therefore be evaluated with multi-axis diagnostics before being used in downstream reliability pipelines.

Figures & tables

Figure 1: Mean logits-based confidence and mean linguistic confidence across task, model and prompt settings. Each point is one setting. Instruction-tuned models tend to have higher values on both axes. The overall association is statistically significant, but the split by model type shows that base models have weak positive association while instruction-tuned models have almost no aggregate association.

Figure 2: Task difficulty and confidence association. Tasks are ordered by decreasing normalized accuracy in the top panel. The bottom panel shows the Pearson correlation between linguistic confidence and logits-based confidence. Markers show per-task means and error bars show

\pm 1

standard deviation across models. Association tends to weaken as task difficulty increases.

Panel of Figure 3 — Figure 3: Model size and association between linguistic and logits-based confidence. Association tends to rise with model size for base models, but it plateaus or decreases for instruction-tuned models.

Panel of Figure 4 — Figure 4: Representative distributions of logits-based and linguistic confidence on CoLA for Mistral-v0.1 models. The top row shows base models under Base Prompt 1, and the bottom row shows instruction-tuned models.

Metric	mean ${}_{\text{instruct}}$	mean ${}_{\text{base}}$	$t$	$p$
$r$	0.172	0.118	1.95	5.4e-02
Distance	0.271	0.241	2.03	4.4e-02
ECE(linguistic)	0.287	0.191	6.02	2.5e-08

Table 1: Paired t-test results comparing instruction-tuned and base models. Underlined values indicate significant differences with

p<0.05

Outcome	$n$	Predictors	$R^{2}$	Adj. $R^{2}$
Dist	363	28	0.570	0.536
ECE	363	28	0.644	0.615
Correlation	363	28	0.498	0.459

Table 2: OLS fit statistics for distance, confidence-to-confidence ECE and correlation between linguistic and logits-based confidence.

Method	Accuracy	Entropy
Majority voting	0.7311	N/A
Level confidence	0.7531	0.4362
Score confidence	0.7830	1.6896

Table 3: Confidence-weighted aggregation on eight classification tasks using gpt-4.1-nano . Finer-grained confidence gives a more dispersed weighting signal and improves accuracy in this setting.

Regime	Diagnostic	Recommended use
Collapsed verbal scores	Scores cluster near 9–10, or have low standard deviation	Avoid using verbal confidence for fine-grained decisions
High verbal confidence with weak internal confidence	Linguistic confidence is much higher than logits-based confidence	Flag, abstain, rerank, or sample additional answers
Dispersed verbal scores with validated association	Held-out data show positive association with correctness or internal confidence	Use as a ranking or weighting signal, not as a calibrated probability
Prompt-induced confidence shifts	Tone or framing raises confidence without improving association	Treat as confidence inflation, not improved reliability

Table 4: Practical interpretation of linguistic confidence under different diagnostic regimes.

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Base Prompt 1 example.

Figure 6: Base Prompt 2 example.

Figure 7: Instruct prompt with example confidence score.

Figure 8: Instruct prompt without example confidence score.

Figure 9: Example prompt for CoQA.

Figure 10: Example prompt for TriviaQA.

Figure 11: Base Prompt without asking for linguistic confidence.

Figure 12: Instruct prompt without asking for linguistic confidence.

Figure 13: Attitude cues for prompt perturbation.

Task / pool	Pool $n$	Median $\|\Delta\|$	Within $\pm 0.10$	Full in 95% interval	Conclusion-level metrics
CoLA	1,043	0.024	95.3%	117/117	45/45
QNLI	5,463	0.027	93.9%	116/117	45/45
QQP	5,000	0.027	94.1%	115/117	45/45
MMLU	1,000	0.026	94.4%	116/117	45/45
Temporal Sequences	800	0.024	95.3%	117/117	45/45

Appendix

Table 5: Stability of estimates from 500 random 200-example subsamples.

Base Model	Instruction-tuned / Chat
Llama-2-7b-hf	Llama-2-7b-chat-hf
Llama-2-13b-hf	Llama-2-13b-chat-hf
Llama-3.1-8B	Meta-Llama-3.1-8B-Instruct
Llama-3.2-3B	Llama-3.2-3B-Instruct
Meta-Llama-3-8B	Meta-Llama-3-8B-Instruct
Mistral-7B-v0.1	Mistral-7B-Instruct-v0.1

Appendix

Table 6: List of models used in our experiments, categorized by version and type.

Panel of Figure 14 — Figure 14: Effect of model size on the correlation between linguistic and logits-based confidence, with error bars indicating the standard deviation of correlation values within each model group. Correlation tends to rise with model size for base models but plateaus for instruction-tuned ones.

Panel of Figure 15 — Figure 15: Base models: model size (x-axis, in B) vs. (a) Distance between two confidence measures and (b) ECE of linguistic confidence.

Panel of Figure 16 — Figure 16: Instruction-tuned models: model size (x-axis, in B) vs. (a) Distance between two confidence measures and (b) ECE of linguistic confidence.

Panel of Figure 17 — Figure 17: Distribution of Logits-based and Linguistic Confidence on CoLA (Base Prompt 1) for Qwen2.5 Base Models.

Panel of Figure 18 — Figure 18: Distribution of Logits-based and Linguistic Confidence on CoLA (Base Prompt 1) for Qwen3 Base Models.

Panel of Figure 19 — Figure 19: Distribution of Logits-based and Linguistic Confidence on CoLA (Base Prompt 1) for Llama2 Base Models.

Panel of Figure 20 — Figure 20: Distribution of Logits-based and Linguistic Confidence on CoLA for Qwen2.5 Instruct-tuned Models.

Panel of Figure 21 — Figure 21: Distribution of Logits-based and Linguistic Confidence on CoLA for Llama2 Instruction-tuned Models.

Metric	LLaMA	Mistral	Qwen	$p_{L\rightarrow M}$	$p_{M\rightarrow Q}$	$p_{L\rightarrow Q}$
$r$	0.081	0.137	0.180	0.033	0.102	0.0004
Distance	0.274	0.259	0.218	0.287	0.0009	0.00002
ECE(linguistic)	0.261	0.219	0.186	0.024	0.070	0.0003

Appendix

Table 7: Paired

t

-tests across model families. Mean value of the three families and

p

between them are reported. Underlined values indicate significant differences (

p<0.05

LLaMA3 $\rightarrow$ LLaMA3.1
Metric	mean_a	mean_b	$p$
$r$	0.057	0.126	0.028
Distance	0.272	0.281	0.527
ECE (linguistic)	0.197	0.234	0.058
Mistral-v0.1 $\rightarrow$ v0.2
$r$	0.125	0.116	0.741

Appendix

Table 8: Paired

t

-tests across model versions within each family. Underlined values indicate significant differences (

p<0.05

). Metrics include overall correlation (

r

), distance between linguistic and logits-based confidence, and ECE based on linguistic confidence.

Panel of Figure 22 — Figure 22: Comparison of average Pearson correlation and accuracy across different confidence score pairs $(i,j)$ on two base prompt settings. Each heatmap shows how the model’s linguistic confidence and task performance vary with score combinations.

Figure 23: Average correlation (top) and task performance (bottom) across different confidence scoring schemes. The

x

-axis indicates the index of linguistic confidence scores (1–10), a random choice, or no confidence used.

Base Prompt 2 $\rightarrow$ Base Prompt 1
Metric	mean a	mean b	$t$	$p$
Normalized Accuracy	0.413	0.405	0.55	5.9e-01
Average Logits	0.596	0.569	8.22	2.1e-13
Logits Variance	0.112	0.108	3.08	2.5e-03
ECE(logits)	0.129	0.140	-1.51	1.3e-01
Base Prompt 1 $\rightarrow$ Base without Confidence

Appendix

Table 9: Paired t-test results comparing logits-based statistics under different prompting conditions. Mean values (

a,b

t

-statistics, and

p

-values are reported. Underlined values indicate significant differences (

p<0.05

Origin $\rightarrow$ Criticism
Metric	mean a	mean b	$t$	$p$
$\mathrm{Avg}(\mathrm{Conf}(li))$	8.428	8.612	-2.34	0.028
$r$	0.266	0.146	3.07	0.0056
Origin $\rightarrow$ Approval
$\mathrm{Avg}(\mathrm{Conf}(li))$	8.428	8.807	-3.75	0.0011
$r$	0.266	0.130	2.43	0.0235

Appendix

Table 10: Paired t-test results comparing the effect of different attitude and calibration variants on average linguistic confidence

\mathrm{Avg}(\mathrm{Conf}(li))

and its Pearson correlation with logits-based confidence

r

. Underlined values indicate significant differences (

p<0.05

Llama Family					Mistral / Mixtral Family
Model	CoQA ( $r$ )		TriviaQA ( $r$ )		Model	CoQA ( $r$ )		TriviaQA ( $r$ )
	Base	Instruct	Base	Instruct		Base	Instruct	Base	Instruct
Llama-2-7B	0.056	0.063	0.211	0.433	Mistral-7B-v0.1	$-$ 0.040	$-$ 0.017	0.288	0.392
Llama-2-13B	$-$ 0.148	0.064	0.224	0.469	Mistral-7B-v0.2	$-$ 0.008	0.213	0.251	0.338
Llama-3.1-8B	$-$ 0.015	0.161	0.004	0.318	Mistral-7B-v0.3	$-$ 0.048	0.012	0.033	0.549
Llama-3.2-3B	0.000	0.245	0.261	0.591	Mistral-Nemo-2407	$-$ 0.050	$-$ 0.015	0.355	0.472

Appendix

Table 11: Pearson correlation (

r

) between semantic-entropy-based and linguistic confidence on CoQA and TriviaQA. Higher correlations within each base/instruction-tuned pair are highlighted in bold; underlined values indicate statistical significance (

p<0.05

Task	Base Prompt 1		Base Prompt 2
	Logits Corr.	Entropy Corr.	Logits Corr.	Entropy Corr.
Conceptual Combinations	0.4441	-0.3111	0.8189	-0.1970
Ruin Names	0.4479	-0.1096	0.4142	-0.3364
Temporal Sequences	0.1246	0.0103	0.0431	-0.0401
CoLA	0.4469	0.2283	0.5672	0.1077
Cause and Effect	0.1413	0.0133	-0.0853	-0.3545

Appendix

Table 12: Correlation between linguistic confidence and internal metrics (logits-based confidence vs. negated semantic entropy) on Qwen2.5-14B under two base prompt templates.

Predictor	Coef	Std. $\beta$	SE	$t$	$p$	CI 2.5	CI 97.5
Linguistic_std	0.080817	0.780064	0.005486	14.733	3.02e-38	0.070026	0.091607
Num_choices	0.029610	0.286977	0.005590	5.297	2.14e-07	0.018613	0.040606
Logits_mean	-0.281884	-0.558972	0.053266	-5.292	2.19e-07	-0.386661	-0.177108
Linguistic_mean	0.019635	0.227861	0.005047	3.890	1.21e-04	0.009707	0.029563
Task_QNLI	0.041021	0.131019	0.015235	2.693	0.0074	0.011053	0.070989
Model_version_3	0.031787	0.127700	0.015214	2.089	0.0374	0.001860	0.061713

Appendix

Table 13: Full OLS coefficients for Dist_linguistic-logits under specified predictors.

Predictor	Coef	Std. $\beta$	SE	$t$	$p$	CI 2.5	CI 97.5
Logits_mean	-0.457089	-0.911912	0.045934	-9.951	1.29e-20	-0.547444	-0.366734
Linguistic_mean	0.033871	0.395460	0.004352	7.782	8.89e-14	0.025309	0.042432
Linguistic_std	0.032748	0.318015	0.004731	6.923	2.26e-11	0.023443	0.042053
Logits_std	-0.365030	-0.213445	0.073691	-4.954	1.16e-06	-0.509985	-0.220076
Num_choices	0.021411	0.208775	0.004821	4.441	1.22e-05	0.011928	0.030894
ECE_linguistic	0.159993	0.229478	0.038584	4.147	4.28e-05	0.084096	0.235890

Appendix

Table 14: Full OLS coefficients for ECE_linguistic-logits under specified predictors.

Predictor	Coef	Std. $\beta$	SE	$t$	$p$	CI 2.5	CI 97.5
Task_QNLI	-0.155245	-0.251365	0.038621	-4.020	7.27e-05	-0.231227	-0.079263
Model_size	0.011347	0.194214	0.003532	3.213	0.0014	0.004398	0.018296
Linguistic_std	0.041036	0.194311	0.014786	2.775	0.0058	0.011945	0.070126
Task_cause_and_effect	-0.107774	-0.171142	0.039022	-2.762	0.0061	-0.184544	-0.031003
Task_QQP	-0.102290	-0.167177	0.038298	-2.671	0.0080	-0.177638	-0.026943
Linguistic_mean	0.028679	0.164087	0.013086	2.192	0.0291	0.002934	0.054423

Appendix

Table 15: Full OLS coefficients for Correlation under specified predictors.

Predictor	Std.( $\beta$ )
Model_size	-0.012746
Num_choices	0.143658
Linguistic_mean	0.201203
Linguistic_std	0.741303
Logits_mean	-0.496499
Logits_std	0.009902

Appendix

Table 16: Full ridge (CV) standardized coefficients for Dist_linguistic-logits .

Predictor	Std.( $\beta$ )
Model_size	0.071043
Num_choices	0.080202
Linguistic_mean	0.381558
Linguistic_std	0.302399
Logits_mean	-0.843727
Logits_std	-0.215614

Appendix

Table 17: Full ridge (CV) standardized coefficients for ECE_linguistic-logits .

Predictor	Std.( $\beta$ )
Model_size	0.113674
Num_choices	0.109605
Linguistic_mean	0.092039
Linguistic_std	0.087174
Logits_mean	-0.007872
Logits_std	0.036264

Appendix

Table 18: Full ridge (CV) standardized coefficients for Correlation .

Explore similar work

An Analysis of Training-Free Self-Reported Confidence in Language Models
Sep 17, 2026Lukas Meyer, Sofia Rossi, Wei Chen +2Confidence Estimation in Language Models Language Model Self-Assessment
DreamAI
Can LLMs Use Linguistic Uncertainty Markers to Reliably Reflect Intrinsic Confidence?
May 27, 2026Gabrielle Kaili-May Liu, Arman CohanConfidence Estimation in Language Models Language Model Calibration
Yale University
Large Language Models Are Overconfident in Their Own Responses
Jun 2, 2026Mario Sanz-Guerrero, Manuel Mager, Katharina von der WenseConfidence Estimation in Language Models LLM Prompting
∇Johannes Gutenberg University Mainz, Germany · ♠University of Colorado Boulder, USA

Sep 17, 2026, cs.CLJ/K move · Enter open · S save

An Analysis of Training-Free Self-Reported Confidence in Language Models

Lukas Meyer, Sofia Rossi, Wei Chen, Thomas Laurent+1

DreamAI

Confidence Estimation in Language Models Language Model Self-Assessment LLM Reliability LLM Uncertainty Estimation Self-Consistency in Language Models

Open paper AskPDF arXiv

Large language models can report a numerical confidence together with generated content, but it is unclear whether this report is more than calibrated rhetoric. We analyze three training-free signals: confidence verbalized with the answer, post-hoc

P(\mathrm{True})

, and agreement with three additional generations on the same 100 TriviaQA questions for two model families. Direct verbalization is a surprisingly strong baseline: after auditing benchmark errors, it reaches AUROC 0.956 and 0.937 for correctness prediction. Three-sample agreement is substantially weaker (0.765 and 0.790), and a fixed interpolation with verbalized confidence has no statistically reliable benefit. Four of nine errors from one model and two of eight from the other receive unanimous sample support, showing that self-consistency can amplify shared misconceptions. Re-eliciting confidence for the same fixed answers with equivalent prompts changes scores by 0.043 to 0.084 on average and flips 4% to 9% of decisions at a 0.8 threshold. An exploratory audit of 100 confidence-tagged biography claims further finds only a modest confidence gap between supported and contradicted claims. These results argue that useful self-reports remain sensitive to elicitation, correlated errors, and benchmark noise.

Abstract

Explore similar work

An Analysis of Training-Free Self-Reported Confidence in Language Models

Can LLMs Use Linguistic Uncertainty Markers to Reliably Reflect Intrinsic Confidence?

Large Language Models Are Overconfident in Their Own Responses