Self-Calibrating Language Models via Test-Time Discriminative Distillation

Mohamed Rissal Hedna; Jan Strich; Martin Semmann; Chris Biemann

cs.CLMar 18, 2026

Self-Calibrating Language Models via Test-Time Discriminative Distillation

Authors: Mohamed Rissal Hedna, Jan Strich, Martin Semmann, Chris Biemann

Organizations: Hub of Computing and Data Science (HCDS), University of Hamburg, Germany

Abstract

Large language models (LLMs) are systematically overconfident: they routinely express high certainty on questions they often answer incorrectly. Existing calibration methods either require labeled validation data, degrade under distribution shifts, or incur substantial inference costs. Recent work has shown that LLMs already contain a better-calibrated signal than the one they verbalize: the token probability of "True" when the model is asked "Is this answer correct?" ( $P(\text{True})$ ) consistently outperforms their stated confidence, a gap that is theoretically grounded as generative error is lower-bounded by roughly twice the corresponding discriminative error. We introduce $\textbf{SECL}$ ( $\textbf{SE}$ lf- $\textbf{C}$ alibrating $\textbf{L}$ anguage Models), a test-time training (TTT) pipeline that exploits this gap as label-free self-supervision, requiring no labeled data or human supervision. SECL adapts only when the input distribution shifts, training on just 6--26% of the question stream at lower cost than the baseline it distills from. Across four small language models from three model families and four diverse domains, SECL reduces Expected Calibration Error (ECE) by 56--78%, outperforming its own supervision signal and matching or outperforming recent inference-time methods. SECL is the first method to apply TTT to calibration; seven ablations covering signal quality, gating strategy, weight accumulation, loss design, domain ordering, hyperparameter sensitivity, and layer selection confirm that each component is crucial and robust across configurations. Code: https://anonymous.4open.science/r/secl-emnlp26-submission-C890

Figures & tables

Figure 1: Calibration error vs. inference cost for Llama 3.2-3B (lower-left is better). Each point is a calibration method; the

x

-axis measures cost in forward-pass equivalents per question, and the

y

-axis measures Expected Calibration Error (ECE). SECL achieves the lowest calibration error at a fraction of the cost of existing methods.

Figure 2: Overview of SECL. (a) Test-Time Inference. An entropy-based change detector (Section 3.1 ) monitors the input stream. If no shift is detected, the adapted model $\theta^{\prime}_{t}$ is used directly; otherwise, a calibration burst updates it to $\theta^{\prime}_{t+1}$ . (b) Calibration Burst. For each of $B{=}50$ questions: the frozen model generates an answer with confidence $c_{t}$ and distractors, computes $\text{NormP}_{\text{True}}$ (Section 3.2 ), and applies a LoRA update when the two signals disagree by more than one bin (Section 3.3 ). Weights accumulate across questions without resetting.

		Llama 3.2-3B			Llama 3.1-8B			Gemma 2-2B			Phi 3.5-Mini
Method	Cost $\downarrow$	ECE $\downarrow$	Brier $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$	AUROC $\uparrow$
Verbalized	1	.170	.292	.510	.225	.258	.684	.256	.314	.558	.251	.275	.600
P(True) Norm	6	.065	.223	.694	.120	.211	.718	.141	.259	.650	.154	.227	.675
DINCO †	$\sim$ 10	.101	.207	.762	.117	.210	.756	.408	.410	.566	.110	.212	.749
SECL (Ours)	1.8–4.6	.050	.241	.587	.083	.222	.643	.056	.254	.548	.110	.251	.521

Table 1: Overall results across the full 2,000-question stream. Best values per column in bold (lowest for ECE/Brier/Cost, highest for AUROC). Cost is amortized forward-pass equivalents per question; SECL’s range reflects model-dependent gating rates (Appendix M ). Accuracy is preserved within 1 percentage point for all models (Appendix D ). † Wang and Stengel-Eskin (2025) .

Model	Verbalized	SECL $\mathcal{O}$	SECL $\mathcal{R}$
Llama 3.2-3B	.170	.050 ( $-$ 71%)	.068 ( $-$ 60%)
Llama 3.1-8B	.225	.083 ( $-$ 63%)	.189 ( $-$ 16%)
Gemma 2-2B	.256	.056 ( $-$ 78%)	.149 ( $-$ 42%)
Phi 3.5-Mini	.251	.110 ( $-$ 56%)	.173 ( $-$ 31%)

Table 2: Domain order robustness (ECE).

\mathcal{O}

: GSM8K

\to

MMLU

\to

ARC

\to

TruthfulQA (default).

\mathcal{R}

: reversed. SECL improves over the verbalized baseline under both orderings for all models; per-domain breakdowns in Appendix L .

Strategy	ECE $\downarrow$	Brier $\downarrow$	AUROC $\uparrow$	Trained
Always-on MSE	0.047	0.240	0.593	100%
Bin-gate ( $\leq\!1$ )	0.052	0.242	0.585	55.8%
Bin-gate ( $\leq\!2$ )	0.044	0.242	0.578	32.6%
Entropy-gated ( $B\!=\!50$ )	0.050	0.241	0.587	25.6%

Table 3: Gating strategy ablation (Llama 3.2-3B). Entropy-gated bursts achieve comparable ECE to always-on training while requiring only 25.6% of questions.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Total params	LoRA layers	LoRA params	% of total
Llama 3.2-3B	3.21B	4	327,680	0.010%
Llama 3.1-8B	8.03B	8	1,048,576	0.013%
Gemma 2-2B	2.61B	8	491,520	0.019%
Phi 3.5-Mini	3.82B	8	786,432	0.021%

Appendix

Table 4: LoRA trainable parameters per model (rank

r{=}8

). Llama 3.2-3B: last 4 layers (24–27 of 28); Llama 3.1-8B: last 8 layers; Gemma and Phi: last 8 layers (Gemma: 18–25 of 26; Phi: 24–31 of 32).

Component	Parameter	Value
LoRA	Rank $r$	8
	Scaling factor $\alpha$	16
	Target layers (Llama 3.2-3B)	Last 4
	Target layers (Llama 3.1-8B, Gemma, Phi)	Last 8
	Target modules (Llama, Gemma)	q_proj , v_proj
	Target modules (Phi)	qkv_proj

Appendix

Table 5: Full hyperparameter settings used across all experiments. Model-specific values are noted where applicable.

Model	Method	ECE $\downarrow$	AdaECE $\downarrow$	Brier $\downarrow$	AUROC $\uparrow$	Acc
Llama 3.2-3B	Verbalized	.170	.167	.292	.510	.576
	Self-Consistency †	.093	.096	.211	.728	.624
	+ Temp Scaling ∗	.047	.075	.250	.504	.576
	P(True) Norm	.065	.067	.223	.694	.576
	+ Temp Scaling ∗	.029	.030	.218	.692	.576
	SECL (Ours)	.050	.060	.241	.587	.577

Appendix

Table 6: Overall results across all models and methods (2,000 questions each). Bold : best overall per model for ECE, AdaECE, Brier, and AUROC. Underline : best among label-free methods.

Metric	Seed 42	Seed 43	Seed 44	Mean $\pm$ Std
Verbalized ECE	.1695	.1757	.1757	.1736 $\pm$ .0029
SECL ECE	.0496	.0502	.0392	.0464 $\pm$ .0051
Verbalized AUROC	.5095	.5119	.5119	.5111 $\pm$ .0011
SECL AUROC	.5870	.6005	.5920	.5932 $\pm$ .0056

Appendix

Table 7: Llama 3.2-3B multi-seed robustness (seeds 42, 43, 44). SECL remains strongly better calibrated than the verbalized baseline across seeds and also improves AUROC on this model.

Model	Method	ECE $\downarrow$	AdaECE $\downarrow$	Brier $\downarrow$	AUROC $\uparrow$	Conf. range
Llama 3.2-3B	Verbalized + Temp ( $T{=}17.6$ )	.047	.075	.250	.504	[.44, .56]
	Verbalized + Platt	.021	.057	.244	.481	[.56, .61]
	P(True) Norm + Temp ( $T{=}1.8$ )	.029	.030	.218	.692	[.14, .86]
	SECL (no labels)	.050	.060	.241	.587	[.05, .85]
Gemma 2-2B	Verbalized + Temp ( $T{=}11.2$ )	.047	.037	.249	.550	[.41, .59]
	Verbalized + Platt	.035	.050	.250	.550	[.42, .55]

Appendix

Table 8: Post-hoc calibration baselines (5-fold CV). Temperature scaling compresses the confidence range substantially, achieving low ECE by predicting near-constant values for models with weak verbalized discrimination (fitted

T

shown).

Model	Method	ECE $\downarrow$	AdaECE $\downarrow$	Brier $\downarrow$	AUROC $\uparrow$
Llama 3.2-3B	P(True) Norm + Temp ∗	.029	.030	.218	.692
	SECL	.050	.060	.241	.587
	SECL + Temp ∗	.049	.053	.241	.584
Gemma 2-2B	P(True) Norm + Temp ∗	.037	.033	.233	.647
	SECL	.056	.060	.254	.548
	SECL + Temp ∗	.011	.037	.249	.542

Appendix

Table 9: SECL combined with temperature scaling (5-fold CV). SECL+Temp achieves the best or near-best ECE overall while preserving SECL’s discrimination improvements.

Method	ECE $\downarrow$	Brier $\downarrow$	AUROC $\uparrow$	Acc
Verbalized	.170	.292	.510	.576
Raw P(True)	.161	.248	.673	.576
P(True) Norm $\tau{=}0.3$	.118	.238	.685	.576
P(True) Norm $\tau{=}0.7$	.065	.223	.694	.576
P(True) Norm $\tau{=}1.5$	.169	.254	.658	.579

Appendix

Table 10: Effect of P(True) normalization (Llama 3.2-3B,

N{=}2{,}000

). Raw P(True) improves over the verbalized baseline. Normalization with temperature

\tau

yields further gains. The optimal

\tau

varies by model (see Table 11 ).

$\boldsymbol{\tau}$	Llama 3.2-3B	Gemma 2-2B	Phi 3.5-Mini	Llama 3.1-8B
0.3	.118	.255	.172	.270
0.7	.065	.215	.168	.234
1.0	.083	.180	.161	.195
1.5	.169	.141	.154	.153
3.0	.247	.143	.245	.120

Appendix

Table 11: P(True) Norm ECE as a function of normalization temperature

\tau

. Bold: selected value per model.

Parameter	Value	ECE $\downarrow$	AdaECE $\downarrow$
$\alpha_{\text{step}}$ †	0.2	.067	.083
	0.3	.066	.069
	0.5	.052	.073
$\delta$ †	0.15	.052	.073
$\delta$ †	0.20	.064	.067
Loss †	Plain MSE (no directional)	.085	.088

Appendix

Table 12: Hyperparameter sensitivity (Llama 3.2-3B). Bold indicates the selected configuration. † Without entropy gating. ‡ With entropy gating.

Model	Method	Domain	ECE $\downarrow$	AdaECE $\downarrow$	Brier $\downarrow$	AUROC $\uparrow$	Acc
Llama 3.2-3B	Verbalized	GSM8K	.218	.215	.294	.565	.472
		MMLU	.106	.109	.253	.601	.562
		ARC	.095	.112	.207	.568	.726
		TruthfulQA	.372	.371	.414	.301	.544
	P(True) Norm	GSM8K	.133	.138	.264	.594	.476
		MMLU	.091	.090	.239	.652	.566

Appendix

Table 13: Per-domain results for Llama 3.2-3B, Gemma 2-2B, and Phi 3.5-Mini. Best ECE and AdaECE per domain are in bold .

Setting	ECE $\downarrow$	Brier $\downarrow$	AUROC $\uparrow$	Acc
SECL (accumulate)	.050	.241	.587	.577
Reset per question	.237	.327	.484	.576

Appendix

Table 14: Weight accumulation ablation (Llama 3.2-3B).

Setting	ECE $\downarrow$	Brier $\downarrow$	AUROC $\uparrow$	Trained %
Always-train	.047	.240	.593	100%
Entropy-gated (SECL)	.050	.241	.587	25.6%

Appendix

Table 15: Entropy gating vs. always-train (Llama 3.2-3B). Entropy gating matches always-train ECE at a fraction of the cost.

Model	$\boldsymbol{\beta}$	ECE $\downarrow$	Brier $\downarrow$	AUROC $\uparrow$	Acc
Llama 3.2-3B	0 (default)	.050	.241	.587	.577
	0.01	.044	.242	.591	.573
	0.1	.149	.279	.527	.569
Gemma 2-2B	0 (default)	.056	.254	.548	.515
	0.01	.127	.271	.486	.514
	0.1	.238	.305	.551	.520

Appendix

Table 16: KL regularization ablation across models (SECL + entropy gating).

Model	Layers	Range	ECE $\downarrow$	Brier $\downarrow$	AUROC $\uparrow$	Acc
Llama 3.2-3B	Early	0–3	.046	.235	.619	.591
	Mid	12–15	.039	.233	.628	.573
	Late (default)	24–27	.039	.241	.592	.570
	Last 8	20–27	.044	.240	.609	.558
	All	0–27	.058	.232	.640	.588
Gemma 2-2B	Last 4	22–25	.116	.277	.464	.513

Appendix

Table 17: LoRA layer ablation. Top: layer position on Llama 3.2-3B (4 layers each except "All"). Bottom: layer count on Gemma 2-2B and Phi 3.5-Mini (3.8B).

Figure 5: Raw

P(\text{True})

( left ) vs. distractor-normalized

\text{NormP}_{\text{True}}

( right ) on Llama 3.2-3B. Normalization suppresses suggestibility bias, producing a better-calibrated supervision signal (ECE: 0.161

\to

0.065).

Figure 6: Raw

P(\text{True})

( left ) vs.

\text{NormP}_{\text{True}}

( right ) for the three non-8B models. Distractor normalization consistently reduces calibration error across model families.

Figure 7: Reliability diagrams: Verbalized baseline ( left ) vs. SECL ( right ) for Llama 3.2-3B (ECE: 0.170

\to

0.050), Gemma 2-2B (ECE: 0.256

\to

0.056), and Phi 3.5-Mini (3.8B) (ECE: 0.251

\to

0.110). SECL consistently shifts predictions toward the diagonal, eliminating the systematic overconfidence visible in the verbalized baselines.

Model	Order	ECE $\downarrow$	AdaECE $\downarrow$	Brier $\downarrow$	AUROC $\uparrow$	Acc
Llama 3.2-3B	Verbalized	.170	.167	.292	.510	.576
	SECL $\mathcal{O}$	.050	.060	.241	.587	.577
	SECL $\mathcal{R}$	.068	.082	.248	.578	.575
Gemma 2-2B	Verbalized	.256	.252	.314	.558	.516
	SECL $\mathcal{O}$	.056	.060	.254	.548	.515
	SECL $\mathcal{R}$	.149	.133	.258	.625	.520

Appendix

Table 18: Domain order sensitivity (overall metrics).

\mathcal{O}

: GSM8K

\to

MMLU

\to

ARC

\to

TruthfulQA.

\mathcal{R}

: TruthfulQA

\to

ARC

\to

MMLU

\to

GSM8K.

Model	Order	GSM8K	MMLU	ARC	TruthfulQA
Llama 3.2-3B	$\mathcal{O}$	.070	.067	.112	.068
Llama 3.2-3B	$\mathcal{R}$	.117	.032	.081	.134
Gemma 2-2B	$\mathcal{O}$	.356	.054	.144	.210
Gemma 2-2B	$\mathcal{R}$	.396	.096	.046	.189
Phi 3.5-Mini (3.8B)	$\mathcal{O}$	.283	.054	.129	.229
Phi 3.5-Mini (3.8B)	$\mathcal{R}$	.068	.213	.099	.343

Appendix

Table 19: Per-domain ECE for original (

\mathcal{O}

) vs. reversed (

\mathcal{R}

) domain order.

Model	Trained	Skipped	SECL	P(True)
Llama 3.2-3B	512 (25.6%)	1,488	9,168	12,000
Llama 3.1-8B	251 (12.6%)	1,749	5,514	12,000
Gemma 2-2B	119 (5.9%)	1,881	3,666	12,000
Phi 3.5-Mini (3.8B)	160 (8.0%)	1,840	4,240	12,000

Appendix

Table 20: Computational cost comparison (in FWD-equivalents). Trained = questions receiving TTT; Skipped = entropy-gated questions (generation only). Per trained question:

{\approx}15

FWD-eq (1 gen

+

5 P(True)

+

3 epochs

\times

3 FWD-eq/step). Per skipped question:

1

FWD-eq. P(True) Norm baseline: 6 FWD-eq

\times

2,000

=12{,}000

Method	ECE $\downarrow$
Verbalized	.247
P(True) Norm $\tau{=}0.3$	.290
P(True) Norm $\tau{=}0.7$	.263
P(True) Norm $\tau{=}1.0$	.257
P(True) Norm $\tau{=}1.5$	.265
P(True) Norm $\tau{=}2.0$	.267

Appendix

Table 21: Qwen 2.5-3B: no generation–discrimination gap. All P(True) Norm temperatures yield worse ECE than the verbalized baseline.

Model	Total	Trained (%)	Skipped (%)
Llama 3.2-3B	2,000	512 (25.6%)	1,488 (74.4%)
Gemma 2-2B	2,000	119 (5.9%)	1,881 (94.1%)
Phi 3.5-Mini (3.8B)	2,000	160 (8.0%)	1,840 (92.0%)
Llama 3.1-8B	2,000	251 (12.6%)	1,749 (87.4%)

Appendix

Table 22: Entropy gating trigger statistics per model. Trained% is the fraction of questions receiving TTT adaptation.

Method	ECE $\downarrow$	AdaECE $\downarrow$	Brier $\downarrow$	AUROC $\uparrow$	Acc
Verbalized	.225	.225	.258	.684	.644
+ Temp Scaling ∗	.080	.089	.213	.681	.644
P(True) Norm	.120	.117	.211	.718	.646
+ Temp Scaling ∗	.108	.103	.208	.716	.646
SECL (Ours)	.083	.080	.222	.643	.646

Appendix

Table 23: Llama 3.1-8B results. SECL reduces ECE by 63% relative to the verbalized baseline while adapting only 12.6% of questions. ∗ Requires ground-truth labels (5-fold CV).

Method	Domain	ECE $\downarrow$	Brier $\downarrow$	AUROC $\uparrow$	Acc
Verbalized	GSM8K	.298	.305	.714	.602
	MMLU	.208	.251	.644	.670
	ARC	.108	.164	.633	.798
	TruthfulQA	.287	.310	.669	.506
SECL	GSM8K	.066	.238	.602	.572
	MMLU	.120	.223	.591	.686

Appendix

Table 24: Per-domain results for Llama 3.1-8B.

Training Target	ECE $\downarrow$	Brier $\downarrow$	AUROC $\uparrow$	Acc
None (Verbalized baseline)	.170	.292	.510	.576
Self-Consistency ( $N{=}10$ )	.432	.470	.443	.564
P(True) Norm (ours)	.050	.241	.587	.577

Appendix

Table 25: Effect of training target on Llama 3.2-3B. All rows use the same SECL pipeline; only the pseudo-label differs. Self-Consistency as a target degrades calibration below the untrained baseline.

Figure 8: Reliability diagrams comparing candidate training targets on Llama 3.2-3B. Self-Consistency ( left , ECE

=

0.093) produces a reasonable calibration curve but with systematic bias. Raw P(True) ( center , ECE

=

0.161) exhibits upward bias from suggestibility. P(True) Norm ( right , ECE

=

0.065) tracks the diagonal most closely, confirming it as the best-calibrated supervision signal for SECL training.

Figure 9: Confidence score distributions for correct (green) and incorrect (red) predictions on Llama 3.2-3B. Self-Consistency and Raw P(True) both cluster near 1.0 with poor class separation; P(True) Norm spreads scores across

[0,1]

with better separation between correct and incorrect predictions.

Model	Method	ECE $\downarrow$	Brier $\downarrow$	AUROC $\uparrow$	Acc
Llama 3.2-3B	Verbalized (4,000q)	.190	.313	.508	.577
Llama 3.2-3B	SECL (4,000q)	.051	.239	.607	.581
Gemma 2-2B	Verbalized (4,000q)	.276	.328	.548	.508
Gemma 2-2B	SECL (4,000q)	.086	.263	.500	.512
Phi 3.5-Mini (3.8B)	Verbalized (4,000q)	.192	.245	.593	.687
Phi 3.5-Mini (3.8B)	SECL (4,000q)	.259	.260	.712	.696

Appendix

Table 26: 4,000-question comparison against same-stream verbalized baselines. SECL substantially improves calibration on Llama and Gemma. For Phi, stronger long-stream calibration requires sufficient distillation; with the 4,000q run, calibration remains worse than verbalized while discrimination and accuracy improve.

Model	Method	FWD-eq	ECE $\downarrow$	Brier $\downarrow$	AUROC $\uparrow$	Acc
Llama 3.2-3B	Verbalized	1	.170	.292	.510	.576
	DINCO	$\sim$ 10	.101	.207	.762	.466
	SECL (ours)	4.6	.050	.241	.587	.577
Phi 3.5-Mini (3.8B)	Verbalized	1	.251	.275	.600	.667
	DINCO	$\sim$ 10	.110	.212	.749	.560
	SECL (ours)	2.1	.110	.251	.521	.665

Appendix

Table 27: SECL vs. DINCO across all models. SECL achieves better calibration (ECE) at lower amortized cost, while DINCO provides superior discrimination (AUROC) at higher cost.

Method	Order	Stream	ECE $\downarrow$	Brier $\downarrow$	AUROC $\uparrow$	Acc
Verbalized	Forward	2k	.195	.287	.586	.506
P(True) Norm	Forward	2k	.173	.273	.625	.506
SECL (Ours)	Forward	2k	.047	.239	.618	.500
SECL (Ours)	Reversed	2k	.047	.241	.625	.497
Verbalized	Forward	4k	.179	.278	.598	.525
SECL (Ours)	Forward	4k	.061	.237	.625	.522

Appendix

Table 28: TruthfulQA generation-domain evaluation (Llama 3.2-3B). SECL reduces ECE by 76% on the 2k forward stream (0.195

\to

0.047), remains stable under reversed ordering, and at 4k still improves over the same-stream verbalized baseline by 66% (0.179

\to

0.061).

Inference-Time Adaptation of Language Models

See all 25 papers

Jun 4, 2026

CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction

Existing calibration methods for Large Language Models (LLMs) often overlook a critical dimension of trustworthiness: a model's {\em behavioral robustness} to irrelevant or misleading information. In this paper, we argue that a model's true confidence should reflect its stability under cognitive pressure. We introduce CaliDist, a novel post-hoc calibration approach that directly measures and penalizes a model's susceptibility to distraction. CaliDist quantifies how an LLM's predictions and uncertainty change when its input prompt is perturbed with semantic distractors. This stability (or lack thereof) signal is then used to adaptively scale the model's initial confidence score. Our extensive experiments on seven Natural Language Understanding classification benchmarks using six distinct LLMs show that CaliDist consistently achieves lower Expected Calibration Error (ECE) and Brier Score compared with strong baselines. Remarkably, our method reduces the ECE from 23% to 7% on average--a relative improvement of 70%--demonstrating that behavioral stability is a powerful signal for calibration. We make our code and datasets available at github.com/m-anas-j/CaliDist.

Mohammad Anas Jawad, Cornelia Caragea

Computer Science, University of Illinois Chicago, USA.

Confidence Estimation in Language Models Language Model Calibration LLM Reliability LLM Uncertainty Estimation

Ask PDF

Oct 2, 2026

Test-time Calibration Learning for Large Language Model Reasoning

Reliable large language models (LLMs) must not only produce accurate answers but also express confidence that faithfully reflects their probability of being correct. Such calibration is essential for identifying uncertain predictions and supporting reliable decision-making in real-world deployment. Recent studies incorporate calibration learning into reinforcement learning (RL), jointly optimizing answer correctness and verbalized confidence using ground-truth correctness supervision. However, their reliance on labeled data limits their applicability in practical test-time settings, where ground-truth labels are unavailable and calibration may need to adapt to newly encountered target tasks. To address this challenge, we propose Test-Time Calibration Learning (TTCL), a label-free framework that jointly adapts reasoning accuracy and verbalized confidence directly on unlabeled target-task data. Specifically, TTCL derives self-supervision signals for both correctness and calibration from multiple model-generated responses, enabling calibration learning at test time without ground-truth labels. Theoretical analysis further establishes TTCL as a bounded surrogate for the ideal calibration objective. Extensive experiments on mathematical reasoning and factual question answering demonstrate that TTCL consistently improves both accuracy and calibration across diverse models and tasks. On base models, TTCL achieves an average relative accuracy improvement of +40.13% and an ECE reduction of +70.80% across eight benchmarks. Moreover, TTCL can further improve both accuracy and calibration for already calibrated models under domain shift, particularly when source-domain calibration transfers poorly to target tasks. In the math-to-factQA setting, TTCL achieves an average relative accuracy gain of +20.35% and reduces ECE by +53.83%. The source code is released at https://github.com/tmlr-group/TTCL.

Zizhuo Zhang, Xiong Peng, Jingwei Sun +4

TMLR Group, Department of Computer Science, Hong Kong Baptist University · Huawei Noah’s Ark Lab.

Inference-Time Adaptation of Language Models Language Model Calibration Confidence Estimation in Language Models

Ask PDF

Mar 26, 2026

Closing the Confidence-Faithfulness Gap in Large Language Models

Large language models (LLMs) tend to verbalize confidence scores that are largely detached from their actual accuracy, yet the geometric relationship governing this behavior remain poorly understood. In this work, we present a mechanistic interpretability analysis of verbalized confidence, using linear probes and contrastive activation addition (CAA) steering to show that calibration and verbalized confidence signals are encoded linearly but are orthogonal to one another -- a finding consistent across three open-weight models and four datasets. Interestingly, when models are prompted to simultaneously reason through a problem and verbalize a confidence score, the reasoning process disrupts the verbalized confidence direction, exacerbating miscalibration. We term this the "Reasoning Contamination Effect." Leveraging this insight, we introduce a two-stage adaptive steering pipeline that reads the model's internal accuracy estimate and steers verbalized output to match it, substantially improving calibration alignment across all evaluated models.

Miranda Muqing Miao, Lyle Ungar

University of Pennsylvania · Department of Computer and Information Science

Confidence Estimation in Language Models Verbalized Confidence Estimation Language Model Calibration LLM Interpretability Activation Steering

Ask PDF

Abstract

Inference-Time Adaptation of Language Models

Explore similar work

CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction

Test-time Calibration Learning for Large Language Model Reasoning

Closing the Confidence-Faithfulness Gap in Large Language Models