Fine-grained Approaches for Confidence Calibration of LLMs in Automated Code Revision

Hong Yi Lin; Chunhua Liu; Haoyu Gao; Patanamon Thongtanunam; Christoph Treude

cs.SEApr 8, 2026

Fine-grained Approaches for Confidence Calibration of LLMs in Automated Code Revision

Authors: Hong Yi Lin, Chunhua Liu, Haoyu Gao, Patanamon Thongtanunam, Christoph Treude

Abstract

In today's AI-assisted software engineering landscape, developers increasingly depend on LLMs that are highly capable, yet inherently imperfect. The tendency of these models to produce incorrect outputs can reduce developer productivity. To this end, a canonical mitigation method is to provide calibrated confidence scores that faithfully reflect their likelihood of correctness at the instance-level. Such information allows users to make immediate decisions regarding output acceptance, abstain error-prone outputs, and better align their expectations with the model's capabilities. Since post-trained LLMs do not inherently produce well-calibrated confidence scores, researchers have developed post-hoc calibration methods, with global Platt-scaling of sequence-level confidence scores proving effective in many generative software engineering tasks but remaining unreliable or unexplored for automated code revision (ACR) tasks such as program repair, vulnerability repair, and code refinement. We hypothesise that the coarse-grained nature of this conventional method makes it ill-suited for ACR tasks, where correctness is often determined by local edit decisions and miscalibration can be sample-dependent, thereby motivating fine-grained confidence calibration. To address this, our study proposes local Platt-scaling applied separately to three different fine-grained confidence scores. Through experiments across 3 separate tasks and correctness metrics, as well as 14 different models of various sizes, we find that fine-grained confidence scores consistently achieve lower calibration error across a broader range of probability intervals, and this effect is further amplified when global Platt-scaling is applied. Our proposed approaches offer a practical solution to eliciting well-calibrated confidence scores, enabling more trustworthy and streamlined usage of imperfect models in ACR tasks.

Figures & tables

Study	Task	Single-Inference Confidence Score	Post-hoc Scaler
Spiess et al. [ 21 ]	Function Synthesis Line-level Completion Automated Program Repair	Average Token Probability Sequence Likelihood	Global Platt-Scaling
Virk et al. [ 22 ]	Code Summarisation	Average Token Probability Normalised Sequence Likelihood	Global Platt-Scaling
(Ours)	Automated Program Repair Vulnerability Repair Automated Code Refinement	Average Token Probability Normalised Sequence Likelihood Minimum Token Probability Lowest- $K$ Token Probability Attention-Weighted Uncertainty	Global Platt-Scaling Local Platt-Scaling
Bold: Contribution from this Study

TABLE I: Comparison with Past Work on Confidence Calibration for Code Generation Tasks in Software Engineering

Fig. 1: Miscalibrated (Left) and Well-Calibrated (Right) Confidence Scores for Code Revisions Generated by CodeLlama-70B in Automated Code Refinement

Panel of Figure 3 — Fig. 2: Zero-Shot Prompt Templates for ACR Tasks

		DCF-Bug		DCF-Vul			CR-Trans
		n=777		n=1,041			n=900
Model	Size	EM	EP	EM	EP	CP	EM	EP
	8B	32.9	59.3	7.7	29.2	59.6	36.3	59.3
Llama-3.1	70B	34.5	58.0	9.7	31.2	72.9	45.9	67.1
	7B	32.3	55.6	8.8	26.1	40.2	40.1	60.4
CodeLlama	70B	40.5	66.8	12.0	29.0	44.9	51.8	72.4

TABLE II: Correctness (%) Achieved by LLMs in ACR Tasks

Fig. 3: Median Skewness of Token-Level Softmax Probabilities for LLM-Generated Code Revisions ( $\tilde{\gamma}_{1}$ )

Fig. 4: DCF-Bug (Wasserstein Distance $W_{1}$ )

Fig. 5: DCF-Bug (Kendall’s $\tau_{b}$ Coefficient)

Fig. 6: DCF-Vul (Wasserstein Distance $W_{1}$ )

Fig. 7: DCF-Vul (Kendall’s $\tau_{b}$ Coefficient)

Fig. 8: CR-Trans (Wasserstein Distance $W_{1}$ )

Fig. 9: CR-Trans (Kendall’s $\tau_{b}$ Coefficient)

		EM															EP
		sl_norm			avg			min			low-K			attn- $\omega$			sl_norm			avg			min			low-K			attn- $\omega$
Model	Size	ECE	$\boldsymbol{\mathcal{B}}$	BC	ECE	$\boldsymbol{\mathcal{B}}$	BC	ECE	$\boldsymbol{\mathcal{B}}$	BC	ECE	$\boldsymbol{\mathcal{B}}$	BC	ECE	$\boldsymbol{\mathcal{B}}$	BC	ECE	$\boldsymbol{\mathcal{B}}$	BC	ECE	$\boldsymbol{\mathcal{B}}$	BC	ECE	$\boldsymbol{\mathcal{B}}$	BC	ECE	$\boldsymbol{\mathcal{B}}$	BC	ECE	$\boldsymbol{\mathcal{B}}$	BC
	8B	0.11	0.21	3	0.03	0.21	2	0.05	0.16	9	0.06	0.18	7	0.04	0.18	7	0.07	0.23	3	0.06	0.23	2	0.08	0.21	8	0.08	0.21	7	0.08	0.22	7
Llama-3.1	70B	0.11	0.21	3	0.12	0.22	2	0.04	0.18	8	0.04	0.18	7	0.04	0.18	7	0.12	0.22	2	0.10	0.23	2	0.04	0.20	7	0.04	0.21	6	0.04	0.21	6
	7B	-	-	1	-	-	1	0.04	0.19	8	0.06	0.21	5	0.04	0.20	6	0.05	0.24	2	-	-	1	0.04	0.22	7	0.03	0.23	6	0.03	0.23	6
CodeLlama	70B	0.03	0.24	2	-	-	1	0.04	0.20	8	0.08	0.21	7	0.08	0.21	7	-	-	1	-	-	1	0.05	0.18	6	0.06	0.19	5	0.06	0.19	5

TABLE III: Global Platt-Scaling Results for DCF-Bug in Terms of Exact Match and Edit Progress

		EP															CP
		sl_norm			avg			min			low-K			attn- $\omega$			sl_norm			avg			min			low-K			attn- $\omega$
Model	Size	ECE	$\boldsymbol{\mathcal{B}}$	BC	ECE	$\boldsymbol{\mathcal{B}}$	BC	ECE	$\boldsymbol{\mathcal{B}}$	BC	ECE	$\boldsymbol{\mathcal{B}}$	BC	ECE	$\boldsymbol{\mathcal{B}}$	BC	ECE	$\boldsymbol{\mathcal{B}}$	BC	ECE	$\boldsymbol{\mathcal{B}}$	BC	ECE	$\boldsymbol{\mathcal{B}}$	BC	ECE	$\boldsymbol{\mathcal{B}}$	BC	ECE	$\boldsymbol{\mathcal{B}}$	BC
	8B	0.04	0.20	2	0.02	0.20	2	0.02	0.19	7	0.02	0.20	4	0.02	0.20	4	0.04	0.24	2	-	-	1	0.06	0.23	5	0.04	0.24	4	-	-	1
Llama-3.1	70B	0.03	0.21	2	0.03	0.21	2	0.05	0.19	6	0.05	0.20	5	0.05	0.20	5	-	-	1	-	-	1	0.05	0.19	3	-	-	1	-	-	1
	7B	0.02	0.19	2	-	-	1	0.03	0.19	3	0.02	0.19	3	0.01	0.19	3	0.03	0.24	2	0.03	0.24	2	0.04	0.24	3	0.02	0.24	4	0.03	0.24	2
CodeLlama	70B	0.03	0.21	2	-	-	1	0.03	0.20	2	0.03	0.20	3	0.03	0.20	3	-	-	1	-	-	1	-	-	1	0.08	0.25	2	0.08	0.25	2

TABLE IV: Global Platt-Scaling Results for DCF-Vul in Terms of Edit Progress and Checks Passed

		EM															EP
		sl_norm			avg			min			low-K			attn- $\omega$			sl_norm			avg			min			low-K			attn- $\omega$
Model	Size	ECE	$\boldsymbol{\mathcal{B}}$	BC	ECE	$\boldsymbol{\mathcal{B}}$	BC	ECE	$\boldsymbol{\mathcal{B}}$	BC	ECE	$\boldsymbol{\mathcal{B}}$	BC	ECE	$\boldsymbol{\mathcal{B}}$	BC	ECE	$\boldsymbol{\mathcal{B}}$	BC	ECE	$\boldsymbol{\mathcal{B}}$	BC	ECE	$\boldsymbol{\mathcal{B}}$	BC	ECE	$\boldsymbol{\mathcal{B}}$	BC	ECE	$\boldsymbol{\mathcal{B}}$	BC
	8B	0.22	0.27	2	0.23	0.27	2	0.16	0.20	6	0.17	0.22	4	0.17	0.23	4	0.23	0.27	3	0.24	0.28	2	0.17	0.23	7	0.18	0.24	6	0.18	0.24	5
Llama-3.1	70B	0.25	0.28	2	0.25	0.29	2	0.17	0.22	7	0.19	0.25	5	0.19	0.25	5	0.21	0.24	3	0.21	0.25	3	0.14	0.21	7	0.16	0.23	5	0.16	0.23	5
	7B	-	-	1	-	-	1	0.19	0.23	6	0.21	0.25	4	0.21	0.25	4	0.29	0.31	2	0.29	0.31	2	0.25	0.27	5	0.26	0.29	4	0.26	0.29	4
CodeLlama	70B	0.31	0.33	2	0.31	0.34	2	0.21	0.25	7	0.25	0.29	5	0.25	0.29	5	0.31	0.29	2	0.32	0.30	2	0.26	0.25	5	0.28	0.27	4	0.28	0.27	4

TABLE V: Global Platt-Scaling Results for CR-Trans in Terms of Exact Match and Edit Progress

min

low-K

attn-

\omega

min

low-K

attn-

\omega

Model

Size

ECE

\boldsymbol{\mathcal{B}}

ECE

\boldsymbol{\mathcal{B}}

ECE

\boldsymbol{\mathcal{B}}

ECE

\boldsymbol{\mathcal{B}}

ECE

\boldsymbol{\mathcal{B}}

ECE

\boldsymbol{\mathcal{B}}

0.03 (-0.02)

0.15 (-0.01)

9 (=0)

0.03 (-0.03)

0.18 (=0)

7 (=0)

0.03 (-0.01)

0.18 (=0)

7 (=0)

0.06 (-0.02)

0.20 (-0.01)

9 (+1)

0.05 (-0.03)

0.21 (=0)

8 (+1)

0.06 (-0.02)

0.22 (=0)

9 (+2)

Llama-3.1

70B

0.03 (-0.01)

0.17 (-0.01)

8 (=0)

0.04 (=0)

0.18 (=0)

7 (=0)

0.04 (=0)

0.18 (=0)

7 (=0)

0.03 (-0.01)

0.20 (=0)

8 (+1)

0.04 (=0)

0.21 (=0)

6 (=0)

0.04 (=0)

0.21 (=0)

6 (=0)

0.02 (-0.02)

0.19 (=0)

8 (=0)

0.03 (-0.03)

0.21 (=0)

5 (=0)

0.02 (-0.02)

0.20 (=0)

6 (=0)

0.03 (-0.01)

0.22 (=0)

7 (=0)

0.03 (=0)

0.23 (=0)

6 (=0)

0.03 (=0)

0.23 (=0)

6 (=0)

CodeLlama

70B

0.04 (=0)

0.20 (=0)

8 (=0)

0.07 (-0.01)

0.20 (-0.01)

7 (=0)

0.07 (-0.01)

0.20 (-0.01)

7 (=0)

0.04 (-0.01)

0.18 (=0)

7 (+1)

0.06 (=0)

0.19 (=0)

5 (=0)

0.06 (=0)

0.19 (=0)

5 (=0)

TABLE VI: Local Platt-Scaling Results for DCF-Bug in Terms of Exact Match and Edit Progress

min

low-K

attn-

\omega

min

low-K

attn-

\omega

Model

Size

ECE

\boldsymbol{\mathcal{B}}

ECE

\boldsymbol{\mathcal{B}}

ECE

\boldsymbol{\mathcal{B}}

ECE

\boldsymbol{\mathcal{B}}

ECE

\boldsymbol{\mathcal{B}}

ECE

\boldsymbol{\mathcal{B}}

0.02 (=0)

0.19 (=0)

7 (=0)

0.02 (=0)

0.20 (=0)

4 (=0)

0.02 (=0)

0.20 (=0)

5 (+1)

0.03 (-0.03)

0.23 (=0)

5 (=0)

0.03 (-0.01)

0.24 (=0)

4 (=0)

0.02

0.24

3 (+2)

Llama-3.1

70B

0.05 (=0)

0.19 (=0)

6 (=0)

0.05 (=0)

0.20 (=0)

5 (=0)

0.05 (=0)

0.20 (=0)

5 (=0)

0.04 (-0.01)

0.19 (=0)

3 (=0)

0.05

0.20

3 (+2)

0.05

0.20

3 (+2)

0.03 (=0)

0.19 (=0)

3 (=0)

0.03 (=0)

0.19 (=0)

3 (=0)

0.01 (=0)

0.19 (=0)

3 (=0)

0.03 (-0.01)

0.24 (=0)

3 (=0)

0.02 (=0)

0.24 (=0)

4 (=0)

0.03 (=0)

0.24 (=0)

2 (=0)

CodeLlama

70B

0.02 (-0.01)

0.20 (=0)

3 (+1)

0.03 (=0)

0.20 (=0)

4 (+1)

0.03 (=0)

0.20 (=0)

4 (+1)

0.05

0.23

6 (+5)

0.05 (-0.03)

0.24 (-0.01)

2 (=0)

0.05 (-0.03)

0.24 (-0.01)

2 (=0)

TABLE VII: Local Platt-Scaling Results for DCF-Vul in Terms of Edit Progress and Checks Passed

min

low-K

attn-

\omega

min

low-K

attn-

\omega

Model

Size

ECE

\boldsymbol{\mathcal{B}}

ECE

\boldsymbol{\mathcal{B}}

ECE

\boldsymbol{\mathcal{B}}

ECE

\boldsymbol{\mathcal{B}}

ECE

\boldsymbol{\mathcal{B}}

ECE

\boldsymbol{\mathcal{B}}

0.13 (-0.03)

0.20 (=0)

9 (+3)

0.18 (-0.01)

0.22 (=0)

4 (=0)

0.17 (=0)

0.23 (=0)

4 (=0)

0.07 (-0.10)

0.22 (-0.01)

9 (+2)

0.11 (-0.07)

0.23 (-0.01)

8 (+2)

0.18 (=0)

0.24 (=0)

5 (=0)

Llama-3.1

70B

0.08 (-0.09)

0.22 (=0)

9 (+2)

0.19 (=0)

0.25 (=0)

5 (=0)

0.19 (=0)

0.25 (=0)

5 (=0)

0.09 (-0.05)

0.20 (-0.01)

8 (+1)

0.08 (-0.08)

0.21 (-0.02)

7 (+2)

0.16 (=0)

0.23 (=0)

5 (=0)

0.12 (-0.07)

0.22 (-0.01)

8 (+2)

0.21 (=0)

0.25 (=0)

4 (=0)

0.21 (=0)

0.25 (=0)

4 (=0)

0.10 (-0.15)

0.22 (-0.05)

7 (+2)

0.18 (-0.08)

0.25 (-0.04)

7 (+3)

0.26 (=0)

0.29 (=0)

4 (=0)

CodeLlama

70B

0.10 (-0.11)

0.22 (-0.03)

8 (+1)

0.23 (+0.02)

0.28 (-0.01)

8 (+3)

0.25 (=0)

0.29 (=0)

5 (=0)

0.18 (-0.08)

0.22 (-0.03)

7 (+2)

0.12 (-0.16)

0.21 (-0.06)

8 (+4)

0.28 (=0)

0.27 (=0)

4 (=0)

TABLE VIII: Local Platt-Scaling Results for CR-Trans in Terms of Exact Match and Edit Progress

		EM			EP
Model	Scaler	ECE	$\mathcal{B}$	BC	ECE	$\mathcal{B}$	BC
	Platt-scaling	0.04 (0.04)	0.20 (0.20)	8 (8)	0.05 (0.04)	0.18 (0.18)	6 (7)
	Temperature-scaling	0.18 (0.18)	0.24 (0.24)	4 (4)	0.02 (0.02)	0.18 (0.18)	4 (6)
	Histogram Binning	0.04 (0.05)	0.20 (0.20)	5 (7)	0.05 (0.05)	0.18 (0.18)	6 (6)
CodeLlama-70b	Isotonic Regression	0.05 (0.04)	0.20 (0.20)	8 (6)	0.05 (0.05)	0.18 (0.19)	5 (6)
	Platt-scaling	0.04 (0.04)	0.18 (0.17)	8 (8)	0.05 (0.03)	0.18 (0.18)	7 (8)

TABLE IX: Comparing Scalers for Fine-Grained Confidence Calibration using Minimum Token Probability on DCF-Bug

		EP			CP
Model	Scaler	ECE	$\mathcal{B}$	BC	ECE	$\mathcal{B}$	BC
	Platt-scaling	0.05 (0.05)	0.19 (0.19)	6 (6)	0.05 (0.04)	0.19 (0.19)	3 (3)
	Temperature-scaling	0.18 (0.17)	0.22 (0.21)	7 (10)	0.23 (0.23)	0.25 (0.25)	2 (2)
	Histogram Binning	0.04 (0.04)	0.19 (0.19)	5 (5)	0.05 (0.04)	0.19 (0.19)	2 (3)
Llama-3.1-70B	Isotonic Regression	0.04 (0.04)	0.19 (0.19)	5 (5)	0.05 (0.04)	0.19 (0.19)	2 (3)
	Platt-scaling	0.06 (0.05)	0.19 (0.19)	6 (6)	0.03 (0.03)	0.23 (0.22)	3 (5)

TABLE X: Comparing Scalers for Fine-Grained Confidence Calibration using Minimum Token Probability on DCF-Vul

		EM			EP
Model	Scaler	ECE	$\mathcal{B}$	BC	ECE	$\mathcal{B}$	BC
	Platt-scaling	0.21 (0.10)	0.25 (0.22)	7 (8)	0.26 (0.18)	0.25 (0.22)	5 (7)
	Temperature-scaling	0.09 (0.08)	0.21 (0.21)	4 (5)	0.21 (0.13)	0.23 (0.20)	3 (5)
	Histogram Binning	0.20 (0.08)	0.24 (0.21)	6 (7)	0.23 (0.16)	0.24 (0.21)	5 (6)
CodeLlama-70b	Isotonic Regression	0.20 (0.07)	0.24 (0.21)	7 (7)	0.23 (0.16)	0.24 (0.21)	5 (7)
	Platt-scaling	0.19 (0.12)	0.21 (0.21)	6 (10)	0.20 (0.08)	0.20 (0.17)	7 (8)

TABLE XI: Comparing Scalers for Fine-Grained Confidence Calibration using Minimum Token Probability on CR-Trans

Fig. 10: UMAP Visualisations of Qwen3-Embedding-8B Representations of Inputs w/ CodeLlama-70B Outputs

Fig. 11: Optimal Hyperparameters for Local Platt-Scaling w/ Minimum Token Probability

Fig. 12: Hyperparameter Search for Local Platt-Scaling w/ Minimum Token Probability using CR-Trans Test Samples

Fig. 13: Precision at Abstention Threshold for CodeLlama-70B in DCF-Bug

Fig. 14: Precision at Abstention Threshold for Llama-3.1-70B in DCF-Vul

Fig. 15: Precision at Abstention Threshold for CodeLlama-70B in CR-Trans

Confidence Estimation in Language Models

Confidence estimation in language models measures whether a model’s uncertainty signals predict answer correctness and support reliable decisions.

See all 247 papers

When Can We Trust LLM Graders? Calibrating Confidence for Automated Assessment
Mar 31, 2026Robinson Ferrer, Damla Turgut, Zhongzhou Chen +1Confidence Estimation in Language Models Language Model Calibration
University of Central Florida
An Empirical Study of Security Calibration in Large Language Models for Code
Jun 30, 2026Mohammed Latif Siddiq, Md. Nafiu Rahman, Joanna C. S. SantosLLM Security Language Model Calibration
University of Notre Dame Notre Dame, IN, USA · Brac University Dhaka, Bangladesh
Are You Sure Your LLM Is Sure? Repairing and Auditing Confidence
Oct 15, 2026Arijit Dey, Bhartendu Kumar, Ramya Hebbalaguppe +1Confidence Estimation in Language Models Language Model Calibration
MBZUAI · TCS Research Labs, India · IIT Delhi

Mar 31, 2026J/K move · Enter open · S save

When Can We Trust LLM Graders? Calibrating Confidence for Automated Assessment

Robinson Ferrer, Damla Turgut, Zhongzhou Chen, Shashank Sonkar

University of Central Florida

Confidence Estimation in Language Models Language Model Calibration Automated Grading Selective Prediction LLM-as-a-Judge

Open paper AskPDF arXiv

Large Language Models (LLMs) show promise for automated grading, but their outputs can be unreliable. Rather than improving grading accuracy directly, we address a complementary problem: predicting when an LLM grader is likely to be correct. This enables selective automation where high-confidence predictions are processed automatically while uncertain cases are flagged for human review. We compare three confidence estimation methods (self-reported confidence, self-consistency voting, and token probability) across seven LLMs of varying scale (4B to 120B parameters) on three educational datasets: RiceChem (long-answer chemistry), SciEntsBank, and Beetle (short-answer science). Our experiments reveal that self-reported confidence consistently achieves the best calibration across all conditions (avg ECE 0.166 vs 0.229 for self-consistency). Surprisingly, self-consistency remains 38% worse despite requiring 5

\times

the inference cost. Larger models exhibit substantially better calibration though gains vary by dataset and method (e.g., a 28% ECE reduction for self-reported), with GPT-OSS-120B achieving the best calibration (avg ECE 0.100) and strong discrimination (avg AUC 0.668). We also observe that confidence is strongly top-skewed across methods, creating a ``confidence floor'' that practitioners must account for when setting thresholds. These findings suggest that simply asking LLMs to report their confidence provides a practical approach for identifying reliable grading predictions. Code is available here.

Abstract

Confidence Estimation in Language Models

Explore similar work

When Can We Trust LLM Graders? Calibrating Confidence for Automated Assessment

An Empirical Study of Security Calibration in Large Language Models for Code

Are You Sure Your LLM Is Sure? Repairing and Auditing Confidence