Confident in a Confidence Score: Investigating the Sensitivity of Confidence Scores to Supervised Fine-Tuning

Lorenzo Jaime Yu Flores; Cesare Spinoso di-Piano; Jackie Chi Kit Cheung

cs.CLApr 10, 2026

Confident in a Confidence Score: Investigating the Sensitivity of Confidence Scores to Supervised Fine-Tuning

Authors: Lorenzo Jaime Yu Flores, Cesare Spinoso di-Piano, Jackie Chi Kit Cheung

Organizations: Mila - Quebec AI Institute · McGill University · Canada CIFAR AI Chair, Mila

Abstract

Uncertainty quantification techniques measure confidence in language model outputs to support critical applications like hallucination detection and selective prediction. While prior work has developed various confidence metrics and demonstrated their calibration for classification tasks or using verbalized confidence, the robustness of probability-based and self-consistency-based UQ metrics for natural language generation remains underexplored particularly under model adaptation. Since practitioners routinely apply supervised fine-tuning to adapt models to new tasks, a key question arises: do confidence metrics maintain their calibration when models are fine-tuned? We investigate this question across NLG tasks including translation, question answering, and mathematical reasoning. We find that calibration shifts substantially after SFT: across 216 configurations, it degrades in 112 cases and improves in 104, with confidence scores shifting due to factors beyond output quality, such as proximity to the training distribution. Degradation is therefore neither universal nor rare, and its direction cannot be anticipated from the pre-SFT model. Through a downstream task evaluation, we show that this miscalibration substantially reduces the practical utility of confidence scores for identifying correct answers. Our findings reveal that existing confidence metrics for NLG cannot be reliably deployed off-the-shelf after fine-tuning, highlighting the need for calibration-robust UQ methods under model adaptation.

Figures & tables

Figure 1: Before SFT, a model had a relatively low confidence score in a correct answer, whereas post-SFT, it had a much higher confidence in an incorrect answer, demonstrating a case of relative overconfidence

	BART-Base			Flan-T5-Base			Llama 3.1-8B
	Eng-Afr	SQUAD	GSM8K	Eng-Afr	SQUAD	GSM8K	Eng-Afr	SQUAD	GSM8K
Avg Tok Prob	0.209 (0.01)	0.626 (0.006)	-0.467 (0.013)	0.335 (0.034)	0.128 (0.015)	-0.473 (0.008)	0.092 (0.107)	0.081 (0.072)	-0.257 (0.122)
Avg Tok Ent	-0.208 (0.011)	-0.638 (0.01)	0.467 (0.011)	-0.329 (0.034)	-0.135 (0.018)	0.473 (0.007)	-0.09 (0.099)	-0.077 (0.063)	0.259 (0.123)
DO Ent	-0.173 (0.015)	-0.521 (0.042)	0.475 (0.01)	-0.403 (0.04)	-0.133 (0.011)	0.471 (0.011)	-0.081 (0.103)	-0.138 (0.049)	0.345 (0.097)
BS Imp Wt	0.233 (0.007)	0.472 (0.014)	-0.469 (0.006)	0.352 (0.043)	0.091 (0.004)	-0.483 (0.003)	0.089 (0.082)	0.067 (0.053)	-0.266 (0.117)
BS Ratios	0.194 (0.014)	0.616 (0.013)	-0.468 (0.007)	0.282 (0.031)	0.003 (0.004)	-0.474 (0.011)	0.076 (0.101)	0.26 (0.076)	-0.289 (0.098)

Table 1: PRR gets significantly worse post-SFT in 43/216 cases, and significantly better in 47/216 cases (one-way ANOVA, with Holm correction,

\alpha=0.05

); Mean delta in PRR pre/post SFT (3 seeds, st. dev in parentheses)

Figure 2: The best confidence metric changes by epoch; PRR values of selected metrics (QWen-14B, Eng-Afr)

Figure 3: The calibration of confidence metrics differs significantly depending on the number of fine-tuning samples used, both before (left) and after (right) controlling for the number of fine-tuning steps used; Plots shown using average token entropy as the confidence metric, used on SQUAD for BART-Base

Figure 4: We classify SFT’s effect on the calibration of samples using four quadrants

Panel of Figure 5 — Figure 5: Plots reveal that average log probabilities are more prone to relative overconfidence, while dropout BLEU variance is more prone to relative underconfidence after SFT; plotting change in F1 between epoch one and ten for SQUAD vs confidence score

	Dataset	Cncd	R. Over	R. Under
BART	Eng-Afr	0.94	0.06	0.00
	SQUAD	0.85	0.11	0.03
	GSM8K	0.98	0.02	0.00
Flan-T5	Eng-Afr	0.17	0.02	0.80
	SQUAD	0.82	0.02	0.16
	GSM8K	0.99	0.00	0.01

Table 2: Proportion of changes in test set samples that are concordant, relatively over/underconfident after SFT, using average token log probs as the measure of confidence; Reporting average proportions over 3 seeds

Confidence increase?		No	Yes
Quality improve?			No	Yes
BART	Eng-Afr	32.0	26.3	41.7
	SQUAD	45.0	45.5	9.5
	GSM8K	37.1	61.4	1.6
Flan-T5	Eng-Afr	45.1	27.5	27.4
	SQUAD	49.0	47.2	3.9

Table 3: Percentage of changes in test set samples between epochs

t

and

t+1

from

1\leq t\leq 9

classified based on directionality of changes in confidence and quality; Reporting average percentages over 3 seeds

Rel Qual.		Same		Flips
Rel Conf.		Same	Flips	Flips	Same
BART	Eng-Afr	0.70	0.12	0.11	0.06
	SQUAD	0.66	0.14	0.13	0.06
	GSM8K	0.79	0.16	0.03	0.01
Flan-T5	Eng-Afr	0.65	0.11	0.17	0.07
	SQUAD	0.74	0.18	0.06	0.02

Table 4: Case 1 miscalibration happens more frequently than Case 2 in 10/18 configurations; Table shows proportion of datapoint pairs classified by change in relative quality and confidence between ep.

t

and

t+1

; Results averaged across 10 epochs and 3 seeds, using average log probs as confidence scores

	BART		Flan-T5		Llama-3.1 8B
	1 Ep	Full	Pre	Full	Pre	Full
Tok Prob	0.563	0.639 $\uparrow$	0.545	0.610 $\uparrow$	0.532	0.465 $\downarrow$
Tok Ent	0.454	0.330 $\downarrow$	0.492	0.368 $\downarrow$	0.458	0.512 $\uparrow$
DO Ent	0.378	0.463 $\uparrow$	0.068	0.359 $\uparrow$	0.355	0.453 $\uparrow$
BS ImpWt	0.314	0.437 $\uparrow$	0.511	0.349 $\downarrow$	0.506	0.492 $\downarrow$
BS Rat	0.542	0.560 $\uparrow$	0.609	0.713 $\uparrow$	0.593	0.409 $\downarrow$

Table 5: AUROC in detecting correct answers using various confidence metrics using the pre-trained and post-SFT model

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

	BART-Base			Flan-T5-Base			Llama 3.1-8B
	Eng-Afr	SQUAD	GSM8K	Eng-Afr	SQUAD	GSM8K	Eng-Afr	SQUAD	GSM8K
Avg Tok Prob	0.131 (0.019)	0.01 (0.011)	0.016 (0.014)	0.161 (0.021)	0.131 (0.02)	0.019 (0.006)	0.111 (0.033)	-0.115 (0.107)	0.101 (0.053)
Avg Tok Ent	-0.132 (0.017)	-0.011 (0.005)	-0.014 (0.013)	-0.159 (0.018)	-0.139 (0.021)	-0.018 (0.004)	-0.101 (0.033)	0.116 (0.097)	-0.099 (0.056)
DO Ent	-0.114 (0.026)	0.002 (0.014)	-0.006 (0.007)	-0.194 (0.065)	-0.083 (0.023)	-0.02 (0.011)	-0.058 (0.027)	0.17 (0.112)	-0.026 (0.027)
BS Imp Wt	0.124 (0.021)	-0.05 (0.033)	0.007 (0.011)	0.189 (0.031)	0.095 (0.012)	0.013 (0.002)	0.11 (0.026)	-0.099 (0.083)	0.087 (0.053)
BS Ratios	0.11 (0.027)	0.046 (0.013)	0.016 (0.009)	0.227 (0.054)	0.011 (0.008)	0.018 (0.012)	0.03 (0.034)	-0.334 (0.102)	0.103 (0.025)

Appendix

Table 6: Difference in PRR after one epoch of SFT and after performing SFT with early stopping; Averaged across 3 seeds with st. dev in parentheses

	BART-Base			Flan-T5-Base			Llama 3.1-8B
	Eng-Afr	SQUAD	GSM8K	Eng-Afr	SQUAD	GSM8K	Eng-Afr	SQUAD	GSM8K
Avg Tok Prob	-0.080 (0.054)	-0.036 (0.013)	-0.022 (0.009)	-0.106 (0.030)	-0.029 (0.015)	-0.020 (0.007)	-0.178 (0.028)	-0.116 (0.029)	-0.171 (0.073)
Avg Tok Ent	-0.079 (0.047)	-0.030 (0.007)	-0.021 (0.002)	-0.105 (0.030)	-0.074 (0.037)	-0.021 (0.009)	-0.120 (0.035)	-0.068 (0.064)	-0.161 (0.077)
DO Ent	-0.071 (0.051)	-0.053 (0.014)	-0.016 (0.008)	-0.100 (0.020)	-0.041 (0.006)	-0.021 (0.008)	-0.136 (0.018)	-0.079 (0.053)	-0.145 (0.054)
BS Imp Wt	-0.070 (0.041)	-0.049 (0.010)	-0.022 (0.006)	-0.115 (0.035)	-0.026 (0.004)	-0.014 (0.009)	-0.167 (0.033)	-0.121 (0.038)	-0.173 (0.076)
BS Ratios	-0.070 (0.034)	-0.032 (0.012)	-0.021 (0.004)	-0.108 (0.064)	-0.162 (0.010)	-0.018 (0.007)	-0.104 (0.023)	-0.221 (0.051)	-0.197 (0.047)

Appendix

Table 7: From epoch-to-epoch, there can be a considerable decline in calibration, reaching differences of up to 0.290 PRR points; Results averaged over 3 seeds with st. dev in parentheses

	Eng-Afr			SQUAD			GSM8K
	No SFT	1 Ep	Full	No SFT	1 Ep	Full	No SFT	1 Ep	Full
Avg Tok Prob	0.092 (0.014)	0.347 (0.102)	0.37 (0.047)	0.064 (0.0)	0.364 (0.045)	0.345 (0.015)	0.0 (0.0)	-0.016 (0.041)	0.024 (0.037)
Avg Tok Ent	0.109 (0.015)	0.321 (0.112)	0.378 (0.051)	0.048 (0.0)	0.384 (0.05)	0.379 (0.017)	0.0 (0.0)	-0.007 (0.042)	0.023 (0.029)
DO Ent	0.047 (0.099)	0.125 (0.067)	0.11 (0.094)	-0.24 (0.023)	-0.004 (0.074)	-0.059 (0.103)	0.0 (0.0)	-0.022 (0.04)	-0.021 (0.036)
BS Imp Wt	-0.235 (0.0)	-0.435 (0.091)	-0.448 (0.069)	-0.144 (0.0)	-0.127 (0.06)	0.083 (0.03)	0.0 (0.0)	-0.047 (0.012)	-0.001 (0.033)
BS Ratios	-0.218 (0.0)	0.03 (0.068)	-0.068 (0.046)	0.239 (0.0)	0.276 (0.017)	0.369 (0.048)	0.0 (0.0)	-0.019 (0.035)	0.005 (0.046)

Appendix

Table 8: PRR of confidence metrics before SFT, after one epoch, and after full SFT with early stopping for BART, Flan-T5, Llama 3.1-8B, Gemma 2-2B, Gemma 2-9B, and QWen 2.5-14B

	Eng-Afr			SQUAD			GSM8K
	No SFT	1 Ep	Full	No SFT	1 Ep	Full	No SFT	1 Ep	Full
Avg Tok Prob	0.0 (0.043)	-0.184 (0.253)	0.327 (0.089)	0.268 (0.048)	-0.018 (0.08)	0.019 (0.18)	0.0 (0.0)	0.163 (0.077)	0.155 (0.075)
Avg Tok Ent	-0.043 (0.047)	-0.169 (0.242)	0.314 (0.081)	0.26 (0.04)	-0.006 (0.1)	0.03 (0.187)	0.0 (0.0)	0.174 (0.07)	0.135 (0.062)
DO Ent	-0.143 (0.02)	0.004 (0.032)	0.049 (0.116)	0.177 (0.056)	0.006 (0.067)	-0.094 (0.081)	0.0 (0.0)	0.022 (0.072)	-0.017 (0.083)
BS Imp Wt	-0.002 (0.043)	0.173 (0.264)	-0.395 (0.079)	-0.263 (0.041)	-0.173 (0.111)	0.116 (0.142)	0.0 (0.0)	-0.218 (0.141)	-0.238 (0.081)
BS Ratios	0.098 (0.04)	0.089 (0.216)	0.018 (0.029)	0.093 (0.107)	0.339 (0.105)	0.324 (0.075)	0.0 (0.0)	0.169 (0.042)	0.255 (0.1)

Appendix

Table 9: PRR of confidence metrics before SFT, after one epoch, and after full SFT with early stopping for BART, Flan-T5, Llama 3.1-8B, Gemma 2-2B, Gemma 2-9B, and QWen 2.5-14B

Panel of Figure 6 — Figure 6: PRR of confidence metrics differs significantly depending on the number of fine-tuning samples. Plots shown for BART-Base fine-tuned on SQUAD for (top) 10 epochs and (bottom) 2500 steps using 3 seeds.

Panel of Figure 7 — Figure 7: The average log probs across all test set samples generally increase for BART and Llama 3.1 8B (top), which may explain the rel. overconfidence in probability-based metrics, and fluctuations in calibration (bottom)

	Dataset	Cncd	R. Over	R. Under
BART	Eng-Afr	0.41	0.06	0.54
	SQUAD	0.80	0.08	0.12
	GSM8K	0.97	0.01	0.02
Flan-T5	Eng-Afr	0.34	0.05	0.61
	SQUAD	0.88	0.04	0.08
	GSM8K	0.97	0.00	0.02

Appendix

Table 10: Proportion of changes in test set samples that are concordant, relatively over/underconfident after SFT, using dropout BLEU variance as the measure of confidence; Reporting average proportions over 3 seeds

Rel Qual.		Same		Flips
Rel Conf.		Same	Flips	Flips	Same
BART	Eng-Afr	0.70	0.13	0.12	0.06
	SQUAD	0.76	0.20	0.03	0.01
	GSM8K	0.63	0.18	0.14	0.05
Flan-T5	Eng-Afr	0.65	0.12	0.15	0.08
	SQUAD	0.79	0.18	0.02	0.01

Appendix

Table 11: Case 1 miscalibration happens more frequently than Case 2 in 6/18 configurations; Table shows proportion of datapoint pairs classified by change in relative quality and confidence between ep.

t

and

t+1

; Results avg across 10 epochs and 3 seeds, using dropout BLEU variance

	Eng-Afr				SQUAD				GSM8K
Ep	BART	T5	Llma	Gmma	BART	T5	Llma	Gmma	BART	T5	Llma	Gmma
2	0.429	0.182	0.236	0.013	0.804	0.875	0.161	0.394	0.992	0.989	0.924	0.924
3	0.447	0.277	0.167	0.262	0.876	0.908	0.008	0.430	0.999	0.990	0.923	0.941
4	0.472	0.426	0.264	0.135	0.879	0.935	0.103	0.586	0.994	0.989	0.929	0.912
5	0.470	0.245	0.139	0.226	0.893	0.926	0.142	0.501	0.995	0.994	0.905	0.931
6	0.569	0.429	0.095	0.148	0.882	0.961	0.027	0.673	0.996	0.993	0.955	0.927

Appendix

Table 12: Proportion of Case 1 miscalibrated pairs where either the worse sample’s output did not change, but the confidence score increased, or the better sample’s output did not change, but the confidence score decreased, using average log probs (top) and variance in BLEU scores (bottom) as confidence scores

	Eng-Afr		SQUAD		GSM8K
Epoch	BART	Flan-T5	BART	Flan-T5	BART	Flan-T5
1	0.005	-0.028	*0.208	0.043	*0.121	-0.086
2	0.057	-0.127	*0.165	*0.154	*0.206	-0.082
3	-0.004	-0.061	*0.179	*0.211	*0.227	-0.038
4	0.071	-0.183	*0.185	*0.138	*0.259	-0.006
5	0.062	-0.179	*0.165	0.079	*0.177	-0.053

Appendix

Table 13: PRR between confidence metric and the similarity of the sample to the training data, measured using the maximum cosine similarity between the last encoder hidden state embedding of the test set and a sample in the training set

Confidence Estimation in Language Models

Confidence estimation in language models measures whether a model’s uncertainty signals predict answer correctness and support reliable decisions.

See all 246 papers

Calibration as a First-Class Criterion in LLM Evaluation
Sep 22, 2026Mario Sanz-Guerrero, Katharina von der WenseLLM Evaluation Confidence Estimation in Language Models
Johannes Gutenberg University Mainz, Germany · University of Colorado Boulder, USA
Know When You're Wrong: Aligning Confidence with Correctness for LLM Error Detection
Feb 18, 2026Xie Xiaohu, Liu Xiaohu, Yao BenjaminLanguage Model Error Detection Confidence Estimation in Language Models
Alexa AI, Amazon, Seattle, WA, USA
Binary Agreement for Efficient Unsupervised Confidence Calibration of Reasoning LLMs
Apr 21, 2026Thomas Zollo, Jimmy Wang, Richard ZemelConfidence Estimation in Language Models Language Model Calibration
Columbia University

Sep 22, 2026J/K move · Enter open · S save

Calibration as a First-Class Criterion in LLM Evaluation

Mario Sanz-Guerrero, Katharina von der Wense

Johannes Gutenberg University Mainz, Germany · University of Colorado Boulder, USA

LLM Evaluation Confidence Estimation in Language Models Language Model Calibration LLM Reliability

Open paper AskPDF arXiv

Calibration of language models -- the alignment between expressed or implicit confidence and empirical correctness -- is a well-studied subfield within NLP. Methods to measure it already exist. The problem is adoption: outside this subfield, NLP research regularly introduces new models, datasets, and benchmarks without checking whether the model's confidence scores are meaningful. We argue that this adoption gap is a major obstacle to trustworthy LLM evaluation. Miscalibration causes problems in two distinct areas: at deployment, where overconfident mistakes cause real harm, and inside the research pipeline, where methods like LLM-as-a-judge, synthetic data generation, and active learning rely on calibrated confidence without verifying it. Standard calibration metrics only require two inputs per example: a confidence score and a correctness judgment. Most benchmarks in use today already provide both, meaning calibration can be reported immediately. For open-ended generation, however, defining these two inputs is still an open challenge. We argue that each NLP subfield should pair its main performance metric with a calibration score and call for treating calibration as an essential property of every model rather than a niche topic.

Abstract

Confidence Estimation in Language Models

Explore similar work

Calibration as a First-Class Criterion in LLM Evaluation

Know When You're Wrong: Aligning Confidence with Correctness for LLM Error Detection

Binary Agreement for Efficient Unsupervised Confidence Calibration of Reasoning LLMs