Residual Calibration via Local Feature-Space Refinement

Xihao Wang; Chengyao Yu; Ruixing Ming

cs.CVOct 22, 2026

Residual Calibration via Local Feature-Space Refinement

Authors: Xihao Wang, Chengyao Yu, Ruixing Ming

Organizations: School of Statistics and Data Science, Zhejiang Gongshang University · Department of Statistics and Data Science, Southern University of Science and Technology

Abstract

Deep neural networks often struggle to provide reliable confidence estimates, making post-hoc calibration essential for reliable decision-making. Existing post-hoc methods largely rely on score-based transformations, which improve marginal calibration but often fail to capture spatially varying miscalibration in local regions of the learned feature space. To address this gap, we propose Local Adaptive Probability Estimation (LAPE), a post-hoc residual calibration framework. Specifically, LAPE treats the global calibrator output as a stable anchor and refines the individual prediction by leveraging non-parametric local evidence from neighboring calibration samples. As a plug-and-play refinement module, LAPE effectively utilizes the information about feature-space geometry and can reuse the calibration data that are used to construct the global calibrator. Furthermore, we extend LAPE to the setting with multi-view inputs through Cascade Feature Augmentation Fusion (CFAF). Extensive experiments demonstrate that the proposed methods can improve the calibration quality of various existing baselines and remain efficient and robust across different settings.

Figures & tables

Panel of Figure 1 — Figure 1: Representation separation can not guarantee uniform calibration support. (a) Raw features of training and calibration samples. (b) Learned features separate the training samples clearly, but calibration samples can remain mixed near induced boundary regions. Different colors represent different classes. The highlighted neighborhood illustrates the local evidence used by LAPE.

L(m,\alpha)\leftarrow L(m,\alpha)-w_{i}^{(m)}(x)\log\hat{p}^{(i)}(y_{i};x_{i},m,\alpha).

Algorithm 1 Local Adaptive Probability Estimation

Figure 2: Paired comparison of LAPE-side refinement across calibration anchors. Each anchor and its LAPE refinement are evaluated under the same dataset and backbone. Rows report NLL, ECE, and accuracy. Bar annotations show relative changes, computed as

(v_{\rm anchor}-v_{\rm LAPE})/v_{\rm anchor}\times 100\%

for NLL/ECE and

(v_{\rm LAPE}-v_{\rm anchor})/v_{\rm anchor}\times 100\%

for accuracy.

Setting	Method	NLL $\downarrow$	ECE $\downarrow$	Acc. $\uparrow$	Full results
Synthetic XOR views	Late fusion	0.507	0.099	0.790	Table 17
	CFAF-LAPE	0.413	0.057	0.848
Multi-view Digits	Late fusion	0.112	0.029	0.960	Table
	CFAF-LAPE	0.092	0.025	0.964
HAM10000 melanoma vs nevus	Late fusion	0.255	0.023	0.896	Table 18
	CFAF-LAPE	0.224	0.014	0.907

Table 1: Results of CFAF-LAPE and late fusion on multi-view tasks.

Figure 3: Sensitivity to calibration data size (calibration ratio). NLL, ECE, and accuracy are reported, with shaded areas indicating standard deviations across five calibration split seeds.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Method family	Examples	Main idea	Main limitation / difference from LAPE
Global score/logit maps	TS, VS, MS, Beta, Dirichlet, ETS [ Guo et al., 2017 , Kull et al., 2017 , Kull et al., 2019 , Zhang et al., 2020 ]	Fit a global transformation on logits, confidence scores, or probability vectors.	Stable but mainly score-level; similar score patterns receive similar corrections even when feature-space neighborhoods have different empirical reliability.
Adaptive parametric or group-wise maps	PTS, AdaTS, LTS, SAG [ Tomani et al., 2022 , Joy et al., 2023 , Ding et al., 2021b , Yang et al., 2023a ]	Learn sample-specific temperatures, local temperature fields, or semantic group-wise calibrators.	More adaptive, but introduces extra fitted components such as temperature predictors or grouping heads; expressive calibrators may be fragile with limited calibration data [ Balanya et al., 2024 ] .
Score-level non-parametric calibration	Histogram binning, isotonic regression, BBQ, IROvA, MIR/IRM, scaling-binning [ Zadrozny and Elkan, 2001 , Zadrozny and Elkan, 2002 , Naeini et al., 2015 , Zhang et al., 2020 , Kumar et al., 2019 ]	Estimate flexible calibration maps from score bins, monotone score transformations, or probability-vector entries.	Non-parametric but still mainly score/probability-level; does not directly exploit learned feature geometry and can be data-inefficient in finite calibration samples.
LAPE	Pure LAPE; TS+LAPE; VS+LAPE; Dir+LAPE; FC+LAPE; SAG+LAPE	Preserve a global calibrated anchor and refine it with local reference-label evidence in feature space.	Uses non-parametric local evidence only as a residual correction, rather than learning a richer calibration map or replacing the classifier output.

Appendix

Table 2: Main families of confidence calibration methods and the position of LAPE. The comparison focuses on how each method family obtains calibration adaptivity and whether it directly uses feature-space reference-label evidence.

Table	Dataset / setting	Backbone or framework
Table 6	GMM local heterogeneity	Special oracle-based probability-distortion design; details are given in the corresponding subsection.
Table 7	Sample-ratio sensitivity	Fixed MLP $(64,32)$ and fixed test set; only the calibration/reference ratio is varied.
Table 8	Covariate shift	Source-trained MLP $(64,32)$ with target-domain calibration/reference splits.
Table 9	CIFAR-10-C Gaussian-noise shift	Frozen ImageNet-pretrained ResNet18 with a multinomial linear probe trained on clean CIFAR-10; target-domain labeled reference samples are used for calibration.
Table 10	High-dimensional stress test	MLP $(64,32)$ on synthetic high-dimensional features.
Table 11	Structural-noise two moons	MLP $(64,32)$ on two-moons data with controlled structural noise.

Appendix

Table 3: Summary of experimental frameworks for supplementary experiments. Each row reports the dataset or diagnostic setting, together with the backbone or calibration framework used for the corresponding result table.

$N_{\mathrm{ref}}$	Seeds	Naive ms/sample	Accelerated ms/sample	Speedup
250	5	$1.437\pm 0.264$	$0.146\pm 0.019$	$9.81\times\pm 1.23$
500	5	$2.415\pm 0.104$	$0.293\pm 0.047$	$8.38\times\pm 1.21$
1000	5	$2.115\pm 0.019$	$0.326\pm 0.029$	$6.53\times\pm 0.63$
2000	5	$3.033\pm 0.707$	$0.632\pm 0.150$	$5.00\times\pm 1.56$

Appendix

Table 4: Multi-seed inference-time comparison between the naive and accelerated implementations. Latency is reported in milliseconds per sample as mean

\pm

standard deviation over five seeds. Lower latency is better, and higher speedup is better.

Group	$\mathbf{V_{k}(x)}$	Local bias	Base MAE	LAPE MAE	Base KL	LAPE KL
Low oracle variation	0.0073	0.0024	0.0151	0.0145	0.0124	0.0110
High oracle variation	0.2056	0.1542	0.1118	0.0943	0.0727	0.0472

Appendix

Table 5: Oracle neighborhood-variation diagnostic in a fixed feature space. Test points are ranked by oracle neighborhood variation

V_{k}(x)

with fixed diagnostic size

k=80

. The table reports the bottom

30\%

and top

30\%

groups, averaged over five seeds. Lower local bias, MAE, and KL are better;

V_{k}(x)

is a diagnostic variation score. The oracle neighborhood-variation score is defined below; local bias, MAE, and KL are assessed relative to the known oracle conditional probabilities.

Setting	Method	NLL	Acc. (%)	ECE	ACE
GMM local heterogeneity	Base	1.0010	60.13	0.1084	0.1103
	TS	0.8480	60.13	0.0394	0.0822
	ETS	0.8429	60.13	0.0301	0.0789
	VS	0.8493	60.40	0.0444	0.0852
	Dir	0.8488	59.87	0.0390	0.0863
	LTS	0.8171	60.13	0.0422	0.0605

Appendix

Table 6: Full local-heterogeneity stress-test results on the GMM manifold. All methods are evaluated under the controlled probability-distortion setting.

Ratio	Base	VS	Pure LAPE	VS+LAPE	Gain over VS
10%	0.1205 / 0.0128 / 97.75	0.1116 / 0.0227 / 97.37	0.1284 / 0.0642 / 96.89	0.0869 / 0.0220 / 97.39	22.18%
20%	0.1205 / 0.0128 / 97.75	0.0974 / 0.0165 / 97.55	0.1003 / 0.0429 / 97.33	0.0817 / 0.0193 / 97.65	16.14%
40%	0.1205 / 0.0128 / 97.75	0.0959 / 0.0172 / 97.47	0.0886 / 0.0326 / 97.48	0.0801 / 0.0212 / 97.61	16.42%
60%	0.1205 / 0.0128 / 97.75	0.0951 / 0.0179 / 97.59	0.0796 / 0.0267 / 97.61	0.0756 / 0.0216 / 97.74	20.45%
80%	0.1205 / 0.0128 / 97.75	0.0954 / 0.0189 / 97.68	0.0759 / 0.0240 / 97.69	0.0739 / 0.0215 / 97.78	22.53%
100%	0.1205 / 0.0128 / 97.75	0.0951 / 0.0185 / 97.68	0.0744 / 0.0229 / 97.77	0.0727 / 0.0202 / 97.77	23.52%

Appendix

Table 7: Sample-ratio sensitivity in the local-residual diagnostic. Each method cell reports NLL / ECE / accuracy (%) averaged over five calibration split seeds.

Shift	Method	$\Delta$ NLL (%)	$\Delta$ Acc. (%)	$\Delta$ ECE (%)	$\Delta$ ACE (%)
Shift 0.4	Pure LAPE	+3.83	+0.12	+15.60	+18.67
	TS+LAPE	+1.83	+0.18	+9.40	-10.42
	VS+LAPE	+1.32	+0.18	-3.05	-11.29
	Dir+LAPE	+1.40	+0.18	-8.96	+1.10
	LTS+LAPE	+2.00	+0.18	+14.51	+1.65
	ETS+LAPE	+1.85	+0.18	+8.10	-8.81

Appendix

Table 8: Target-calibrated covariate-shift LAPE-side results. Entries are relative percentage changes versus the corresponding anchor, with positive values indicating improvement.

Severity	Method	$\Delta$ NLL (%)	$\Delta$ Acc. (%)	$\Delta$ ECE (%)	$\Delta$ ACE (%)
1	Pure LAPE	+39.18	+16.80	+87.12	+49.68
	TS+LAPE	+17.44	+20.10	-109.93	+27.95
	VS+LAPE	+2.61	+2.39	-82.98	-40.95
	Dir+LAPE	+1.50	+1.16	-45.51	-12.64
	LTS+LAPE	+16.50	+20.70	-97.52	+27.26
	ETS+LAPE	+17.47	+20.01	-104.91	+28.14

Appendix

Table 9: CIFAR-10-C Gaussian-noise LAPE-side results. Entries are relative percentage changes versus the corresponding anchor, with positive values indicating improvement. Severity levels 1, 3, and 5 represent increasingly strong Gaussian-noise corruption.

Dimension	Method	$\Delta$ NLL (%)	$\Delta$ Acc. (%)	$\Delta$ ECE (%)	$\Delta$ ACE (%)
D=10	ETS+LAPE	+0.74	+0.37	-60.00	-9.39
	LTS+LAPE	+2.78	+0.37	-3.54	-17.19
	Dir+LAPE	+4.67	+0.76	-25.49	-1.92
	TS+LAPE	+3.59	+0.49	-21.49	-12.20
	VS+LAPE	+3.79	+0.62	-22.80	-10.65
	IROvA+LAPE	+7.74	+0.26	+8.33	-5.41

Appendix

Table 10: Full high-dimensional stress-test results. Only LAPE-side refinements are reported for each feature dimension. Entries are relative percentage changes versus the corresponding anchor, with positive values indicating improvement.

Noise	Method	$\Delta$ NLL (%)	$\Delta$ Acc. (%)	$\Delta$ ECE (%)	$\Delta$ ACE (%)
Noise 0.1	TS+LAPE	+6.06	+0.00	+17.02	-93.82
	ETS+LAPE	+6.06	+0.00	+17.02	-93.82
	Dir+LAPE	+1.71	+0.00	-5.96	+0.50
	VS+LAPE	+1.66	+0.00	-5.63	+0.41
	Pure LAPE	+57.43	+0.00	+59.86	+48.74
	LTS+LAPE	+3.66	+0.00	-11.47	-8.32

Appendix

Table 11: Full structural-noise results on the two-moons setting. Only LAPE-side refinements are reported at each noise level. Entries are relative percentage changes versus the corresponding anchor, with positive values indicating improvement.

Dataset / backbone	Method	$\Delta$ NLL (%)	$\Delta$ Acc. (%)	$\Delta$ ECE (%)	$\Delta$ ACE (%)
Obesity / MLP-deep	ETS+LAPE	+10.91	+1.24	-14.09	-6.42
	TS+LAPE	+10.88	+1.24	-14.29	-5.82
	Pure LAPE	+8.55	+1.24	-29.34	-12.48
	Dir+LAPE	+10.81	-0.41	+7.78	-13.06
	LTS+LAPE	+5.57	+1.24	+3.82	+3.21
	MIR+LAPE	+8.81	+1.24	+12.10	+10.52

Appendix

Table 12: Architecture-depth results on Obesity. Each block reports only LAPE-side refinements for the corresponding backbone. Entries are relative percentage changes versus the corresponding anchor, with positive values indicating improvement.

Method	NLL	Acc. (%)	ECE	ACE	Zone A	Zone B
Adaptive LAPE	0.0384	99.00	0.0122	0.0070	0.0303	0.0467
Fixed (A=0.2, K=30)	0.1660	94.15	0.0589	0.0593	0.2262	0.1044
Base Model (Alpha=0)	0.1756	90.90	0.0339	0.0306	0.3016	0.0467
Fixed (A=0.8, K=30)	0.1949	99.25	0.1540	0.1321	0.0766	0.3161
Fixed (A=0.5, K=100)	0.2343	94.20	0.1101	0.1053	0.2494	0.2188

Appendix

Table 13: Ice-and-Fire ablation study. Zone columns report region-wise NLL.

Method	$\Delta$ NLL (%)	$\Delta$ Acc. (%)	$\Delta$ ECE (%)	$\Delta$ ACE (%)
CoOp+LAPE	+25.87	+20.45	+67.73	+35.74
CoOp+DAC+LAPE	+13.45	+21.15	+31.34	+24.91

Appendix

Table 14: CoOp/DAC LAPE-side results on DTD. Entries are relative percentage changes versus the corresponding anchor, with positive values indicating improvement.

Method	$\Delta$ NLL (%)	$\Delta$ Acc. (%)	$\Delta$ ECE (%)	$\Delta$ ACE (%)
CE+LAPE	+3.27	+0.70	-8.95	-3.66
MDCA+LAPE	+3.74	+0.80	-2.78	-6.83
MbLS+LAPE	+22.54	+2.40	+7.50	+10.40

Appendix

Table 15: Training-time calibration LAPE-side results on Obesity. Entries are relative percentage changes versus the corresponding anchor, with positive values indicating improvement.

Strategy	NLL	Acc.	ECE	ACE
Direct reuse	1.3884 $\pm$ 0.1132	0.6485 $\pm$ 0.0291	0.3316 $\pm$ 0.0291	0.3227 $\pm$ 0.0300
50/50 split	0.6198 $\pm$ 0.0304	0.6816 $\pm$ 0.0193	0.0957 $\pm$ 0.0302	0.0935 $\pm$ 0.0303
5-fold cross-fitting	0.5951 $\pm$ 0.0284	0.7021 $\pm$ 0.0228	0.0757 $\pm$ 0.0206	0.0728 $\pm$ 0.0193

Appendix

Table 16: Cross-fitting versus data-splitting results. Results are reported as mean

\pm

standard deviation over 20 independent seeds.

Scenario	Method	NLL	Acc. (%)	ECE	ACE
Interaction (XOR)	CFAF-LAPE	0.4134	84.80	0.0567	0.0639
	Fused (VS+Weight)	0.5067	78.96	0.0985	0.1037
	VS View A	0.5439	68.72	0.0708	0.0753
Local Heterogeneity (Blobs)	CFAF-LAPE	0.3940	79.36	0.0696	0.0696
	Fused (VS+Weight)	0.4643	78.88	0.0473	0.0610
	VS View B	0.4651	78.96	0.0424	0.0600

Appendix

Table 17: Full CFAF synthetic stress-test results.

Dataset	Method	NLL	Acc. (%)	ECE	ACE
UCI Multiple Features	View A only	0.1654	94.00	0.0259	0.0373
	View B only	0.1497	95.20	0.0235	0.0194
	Late fusion	0.1117	96.00	0.0294	0.0147
	CFAF-LAPE	0.0919	96.40	0.0252	0.0156
HAM10000 malignant vs benign	Image only	0.3461	84.98	0.0146	0.0250
	Metadata only	0.4167	81.95	0.0210	0.0259

Appendix

Table 18: Real multi-view and multimodal CFAF results. Each block reports two single-view baselines, a late-fusion baseline, and CFAF-LAPE.

Method	$\Delta$ NLL (%)	$\Delta$ Acc. (%)	$\Delta$ ECE (%)	$\Delta$ ACE (%)
QMF+CFAF-LAPE	+44.67	+2.81	+55.93	+56.95
PDF+CFAF-LAPE	+25.49	+2.13	+22.54	+32.22

Appendix

Table 19: Dynamic-fusion LAPE-side results on the multi-view digits benchmark. Entries are relative percentage changes versus the corresponding anchor, with positive values indicating improvement.

Model Calibration

Model calibration aligns predicted confidence or model outputs with observed outcomes and measurements to support more reliable decisions.

See all 218 papers

Quantile Adaptive Temperature Scaling for Confidence Calibration
Jun 19, 2026Omprakash Chakraborty, Leo Fillioux, Ismail Ben Ayed +1Post-Hoc Calibration Probability Calibration
1 ÉTS Montréal, Canada · Université Paris-Saclay, CentraleSupélec, Gustave Roussy, INSERM, CDSU, IHU PRISM, Gif-sur-Yvette
Confidence Calibration of Deep Learning Systems
Aug 12, 2026Coby PensoUnsupervised Domain Adaptation Conformal Prediction
The Faculty of Engineering, Bar-Ilan University, Ramat Gan, Israel
Enhancing Deep Neural Network Reliability with Refinement and Calibration
May 22, 2026Ramya Hebbalaguppe, Ajay Shastry, Soumya Suvra Ghosal +1Probability Calibration Uncertainty Calibration
SIT, Indian Institute of Technology Delhi, New Delhi, India

Jun 19, 2026J/K move · Enter open · S save

Quantile Adaptive Temperature Scaling for Confidence Calibration

Omprakash Chakraborty, Leo Fillioux, Ismail Ben Ayed, Jose Dolz

1 ÉTS Montréal, Canada · Université Paris-Saclay, CentraleSupélec, Gustave Roussy, INSERM, CDSU, IHU PRISM, Gif-sur-Yvette

Post-Hoc Calibration Probability Calibration Model Calibration Uncertainty Calibration

Open paper AskPDF arXiv

Deep neural networks often produce poorly calibrated confidence estimates, overstating their certainty even when predictions are incorrect. Temperature Scaling remains the most widely used posthoc calibration method due to its simplicity and effectiveness, yet its global, uniform rescaling of logits fails to correct the highly heterogeneous structure of miscalibration observed across the confidence spectrum. In particular, the largest correctness confidence discrepancies arise in different quantile regions depending on the setting, low confidence predictions, where uncertainty matters most, tend to exhibit the largest correctness confidence discrepancies, which standard TS leaves largely unaddressed. We introduce Quantile Adaptive Temperature Scaling (QaTS), a simple and efficient post hoc calibration method that adapts the temperature as a function of a predictions empirical confidence quantile. By mapping confidences into the quantile space, QaTS normalizes the calibration problem, makes the structure of miscalibration explicit and enables a monotone temperature function that adapts across quantiles while leaving well calibrated high confidence predictions largely unchanged. preserving high confidence behavior. This quantile aware formulation aligns naturally with a reparameterized Expected Calibration Error (ECE) objective and yields a sample wise temperature that is robust across a variety of challenging scenarios, such as class imbalance and distributional shifts. Across a broad range of datasets, architectures, evaluation scenarios and diverse tasks, QaTS consistently, and substantially, outperforms state of the art post hoc calibration methods, delivering more reliable and trustworthy confidence estimates without modifying model predictions.

Abstract

Model Calibration

Explore similar work

Quantile Adaptive Temperature Scaling for Confidence Calibration

Confidence Calibration of Deep Learning Systems

Enhancing Deep Neural Network Reliability with Refinement and Calibration