Socrates Loss: Unifying Confidence Calibration and Classification by Leveraging the Unknown

Sandra Gómez-Gálvez; Tobias Olenyi; Gillian Dobbie; Katerina Taškova

cs.LGApr 14, 2026

Socrates Loss: Unifying Confidence Calibration and Classification by Leveraging the Unknown

Authors: Sandra Gómez-Gálvez, Tobias Olenyi, Gillian Dobbie, Katerina Taškova

Organizations: University of Auckland · Technical University of Munich

Abstract

Deep neural networks, despite their high accuracy, often exhibit poor confidence calibration, limiting their reliability in high-stakes applications. Current ad-hoc confidence calibration methods attempt to fix this during training but face a fundamental trade-off: two-phase training methods achieve strong classification performance at the cost of training instability and poorer confidence calibration, while single-loss methods are stable but underperform in classification. This paper addresses and mitigates this stability-performance trade-off. We propose Socrates Loss, a novel, unified loss function that explicitly leverages uncertainty by incorporating an auxiliary unknown class, whose predictions directly influence the loss function and a dynamic uncertainty penalty. This unified objective allows the model to be optimized for both classification and confidence calibration simultaneously, without the instability of complex, scheduled losses. We provide theoretical guarantees that our method regularizes the model to prevent miscalibration and overfitting. Across four benchmark datasets and multiple architectures, our comprehensive experiments demonstrate that Socrates Loss consistently improves training stability while achieving more favorable accuracy-calibration trade-off, often converging faster than existing methods.

Figures & tables

Figure 1: Evolution of sub-components of the Socrates Loss, the dynamic uncertainty penalty (orange curve) and adaptive target (blue curve), alongside the Expected Calibration Error (red curve) and error rate ( $1-accuracy$ , black curve) across epochs on CIFAR-100 with VGG-16. Results are averaged over five runs.

Panel of Figure 2 — Figure 2: Loss trends for CIFAR-10 with VGG-16 (a), SVHN with VGG-16 (b), Food-101 with ResNet-34 (c), CIFAR-100 with VGG-16 (d), ResNet-110 (e), ViT (f), and ViT Transfer Learning (g). The black rectangles indicate the periods of loss change in the two-phase training approaches.

Figure 3: Error rate (1 - accuracy) versus Expected Calibration Error (ECE) across epochs for different datasets-architectures. Dotted lines indicate the threshold for acceptable calibration. Lower and leftward values indicate better performance. SAT is excluded from Food-101 due to premature convergence. Lines are drawn every 25 epochs.

Panel of Figure 4 — Figure 4: Pareto plot across epochs (lines are drawn every 2 epochs) (a) and reliability diagram at the final epoch 50 (b) for the CIFAR-100 validation set using ViT with Transfer Learning (TL). In (b), the SAT curve overlaps with that of CE.

Figure 5: Reliability diagrams for the validation set at the final training epoch (epoch 300).

		Metric	Socrates	SAT	CCL-SC	Focal	FLSD	Brier	MC
CIFAR-10	VGG-16	Acc	$\textbf{88.42 }\pm\textbf{ 0.05}$	$88.29\pm 0.34$	$87.92\pm 0.26$	$\underline{88.31\pm 0.19}$	$88.16\pm 0.17$	$87.67\pm 0.10$	$87.53\pm 0.12$
		ECE	$\textbf{4.39 }\pm\textbf{ 0.25}$	$9.00\pm 0.23$	$9.57\pm 0.36$	$\underline{6.27\pm 0.16}$	$6.29\pm 0.16$	$7.21\pm 0.12$	$9.34\pm 0.17$
		AdaECE	$\textbf{6.03 }\pm\textbf{ 0.18}$	$9.03\pm 0.23$	$9.50\pm 0.36$	$\underline{6.32\pm 0.25}$	$6.38\pm 0.20$	$7.23\pm 0.15$	$9.34\pm 0.17$
		CW-ECE	$\textbf{1.31 }\pm\textbf{ 0.01}$	$\underline{1.34\pm 0.05}$	$1.55\pm 0.02$	$1.41\pm 0.03$	$1.42\pm 0.02$	$1.50\pm 0.03$	$1.49\pm 0.03$
SVHN	VGG-16	Acc	$\underline{97.25\pm 0.08}$	$97.21\pm 0.03$	$19.59\pm 0.00$	$66.08\pm 42.44$	$81.62\pm 34.68$	$\textbf{97.31 }\pm\textbf{ 0.05}$	$35.06\pm 34.59$
		ECE	$2.49\pm 0.03$	$\underline{0.60\pm 0.04}$	$\textbf{0.57 }\pm\textbf{ 0.28}$	$4.77\pm 3.15$	$2.31\pm 0.52$	$1.32\pm 0.12$	$2.70\pm 0.84$

Table 1: Test set performance at epoch 300 for standard training and at epoch 50 for transfer learning (TL). Metrics reported: accuracy (acc), ECE, AdaptiveECE (AdaECE), and Classwise-ECE (CW-ECE). Poor performance using SAT on Food-101, and CCL-SC and MC on SVHN is attributed to premature convergence. Best results are highlighted in bold , and second-best are underlined .

		Metric	Socrates	Socrates+TS	Socrates+MS	Socrates+VS	CE	CE+TS	CE+MS	CE+VS
CIFAR-10	VGG-16	Acc	$88.42\pm 0.05$	$88.40\pm 0.04$	$88.46\pm 0.14$	$88.37\pm 0.06$	$88.46\pm 0.28$	$88.52\pm 0.28$	$88.45\pm 0.29$	$\textbf{88.55}\pm\textbf{0.37}$
		ECE	$4.39\pm 0.25$	$8.32\pm 0.33$	$\textbf{2.60}\pm\textbf{0.48}$	$3.21\pm 0.44$	$9.09\pm 0.24$	$4.67\pm 0.20$	$2.78\pm 0.10$	$2.63\pm 0.31$
		AdaECE	$6.03\pm 0.18$	$8.90\pm 0.14$	$\textbf{3.25}\pm\textbf{0.48}$	$3.82\pm 0.35$	$9.08\pm 0.23$	$5.31\pm 0.23$	$3.58\pm 0.23$	$3.53\pm 0.07$
		CW-ECE	$1.31\pm 0.01$	$2.09\pm 0.02$	$\textbf{1.29}\pm\textbf{0.01}$	$1.30\pm 0.01$	$1.42\pm 0.02$	$1.39\pm 0.04$	$1.41\pm 0.04$	$1.42\pm 0.05$
SVHN	VGG-16	Acc	$97.25\pm 0.08$	$97.24\pm 0.09$	$\textbf{97.30}\pm\textbf{0.05}$	$97.27\pm 0.09$	$97.19\pm 0.04$	$97.20\pm 0.04$	$97.16\pm 0.05$	$97.19\pm 0.05$
		ECE	$2.49\pm 0.03$	$6.91\pm 0.05$	$\textbf{0.56}\pm\textbf{0.03}$	$0.63\pm 0.08$	$1.58\pm 0.05$	$1.57\pm 0.07$	$0.71\pm 0.05$	$0.76\pm 0.05$

Table 2: Test set performance with post-hoc confidence calibration methods at epoch 300 for standard training and at epoch 50 for transfer learning (TL). Metrics reported: accuracy (acc), ECE, AdaptiveECE (AdaECE), and Classwise-ECE (CW-ECE). Best results are highlighted in bold .

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

		CIFAR-10	SVHN	Food-101	CIFAR-100	CIFAR-100	CIFAR-100	CIFAR-100
Method	Hyperp.	VGG-16	VGG-16	ResNet-34	VGG-16	ViT	ViT TL	ResNet-110
	$\gamma$	2	1	2	2	4	1	1
Socrates	$\alpha$	0.8	0.9	0.999	0.999	1	0.999	0.999
	$E_{s}$	0	0	0	0	0	0	0
	MB	128	128	128	128	128	128	128
	$q$	0.999	0.99	0.9	0.99	0.9	0.9	0.9

Appendix

Table 3: Final hyperparameters. Underlined values are from their original research.

Figure 6: Evolution of the Socrates method on CIFAR-100 with VGG-16 for all hyperparameter configurations. The curves show the mean values across epochs of the dynamic uncertainty penalty ( $\beta$ penalty, orange curve), the adaptive target (blue curve), the Expected Calibration Error (ECE, red curve) and error rate ( $1-accuracy$ , black curve).

		$\gamma$ sensitivity
Epoch	Metric	0	1	2	3	4
25	Acc	$\textbf{36.06}\pm\textbf{0.64}$	$34.56\pm 1.55$	$35.21\pm 2.01$	$\underline{35.22\pm 1.03}$	$33.98\pm 0.58$
25	ECE	$\textbf{2.37}\pm\textbf{0.35}$	$\underline{2.63\pm 1.06}$	$3.10\pm 1.15$	$4.45\pm 1.48$	$5.00\pm 1.00$
50	Acc	$\textbf{51.62}\pm\textbf{0.53}$	$\underline{50.95\pm 0.45}$	$50.71\pm 0.33$	$49.48\pm 0.65$	$49.10\pm 0.68$
50	ECE	$\textbf{2.10}\pm\textbf{0.33}$	$\underline{3.17\pm 0.99}$	$5.64\pm 0.99$	$8.32\pm 1.42$	$10.60\pm 0.25$
75	Acc	$\textbf{59.80}\pm\textbf{0.73}$	$58.48\pm 0.46$	$\underline{59.09\pm 0.91}$	$57.65\pm 0.64$	$56.60\pm 0.50$

Appendix

Table 4: Hyperparameter sensitivity analysis -

\gamma

sensitivity. ECE values in a range of

[0,1]

and accuracy (

\%

) scores (Acc) for the validation dataset including mean and standard deviation. Best results are highlighted in bold and second-best results are underlined . The models that perform best in terms of both accuracy and ECE, according to their Pareto plot positions, are shown in italics (epochs 150 and 300).

		$\alpha$ sensitivity
Epoch	Metric	0.8	0.9	0.99	0.999	1
25	Acc	$4.72\pm 1.62$	$17.30\pm 0.76$	$34.87\pm 0.80$	$\textbf{35.21}\pm\textbf{2.01}$	$\underline{34.97\pm 0.72}$
25	ECE	$15.36\pm 2.08$	$12.56\pm 0.47$	$5.48\pm 1.82$	$\underline{3.10\pm 1.15}$	$\textbf{2.39}\pm\textbf{0.82}$
50	Acc	$15.12\pm 2.03$	$27.98\pm 2.17$	$\underline{50.52\pm 0.41}$	$\textbf{50.71}\pm\textbf{0.33}$	$50.39\pm 0.79$
50	ECE	$16.69\pm 1.49$	$15.62\pm 1.96$	$10.20\pm 0.98$	$\underline{5.64\pm 0.99}$	$\textbf{4.78}\pm\textbf{0.94}$
75	Acc	$22.86\pm 1.02$	$38.80\pm 0.92$	$58.20\pm 0.81$	$\textbf{59.09}\pm\textbf{0.91}$	$\underline{58.50\pm 0.51}$

Appendix

Table 5: Hyperparameter sensitivity analysis -

\alpha

sensitivity. ECE values in a range of

[0,1]

and accuracy (

\%

Panel of Figure 7 — Figure 7: Error rate (1 - accuracy) versus Expected Calibration Error (ECE) across epochs for $\gamma$ (left) and $\alpha$ (right). Dotted lines indicate the threshold for acceptable calibration. Lower and leftward values indicate better performance. Lines are drawn every 25 epochs.

Panel of Figure 8 — Figure 8: Curves depicting the average values of the unknown class confidences (left) and Expected Calibration Error (ECE) values (right) across epochs for CIFAR-100 with VGG-16 (a) and ResNet-110 (b), using the SAT method. Gray boxes highlight instability and miscalibration due to loss function change at epoch 200 (a) and 150 (b). It is evident that when confidence in the unknown class increases on average, ECE suddenly changes and increases as well.

		$\beta$ sensitivity
Epoch	Metric	$\beta_{\setminus GT}$	$\beta_{\setminus GT,idk}$	$\beta_{fixed}$	$\setminus\beta$
25	Acc	$\underline{35.21\pm 2.01}$	$\textbf{35.72}\pm\textbf{0.95}$	$34.95\pm 0.81$	$34.46\pm 1.84$
25	ECE	$3.10\pm 1.15$	$3.03\pm 1.42$	$\textbf{2.09}\pm\textbf{0.86}$	$\underline{2.33\pm 1.40}$
50	Acc	$\underline{50.71\pm 0.33}$	$\textbf{51.21}\pm\textbf{0.27}$	$50.39\pm 0.33$	$50.70\pm 1.03$
50	ECE	$5.64\pm 0.99$	$6.32\pm 0.96$	$\underline{5.01\pm 0.45}$	$\textbf{4.90}\pm\textbf{1.23}$
75	Acc	$\textbf{59.09}\pm\textbf{0.91}$	$58.56\pm 0.58$	$58.71\pm 0.68$	$\underline{58.73\pm 0.47}$

Appendix

Table 6: Hyperparameter sensitivity analysis -

\beta

sensitivity. ECE values in a range of

[0,1]

and accuracy (

\%

Figure 9: Error rate (1 - accuracy) versus Expected Calibration Error (ECE) across epochs for $\beta$ . Dotted lines indicate the threshold for acceptable calibration. Lower and leftward values indicate better performance. Lines are drawn every 25 epochs.

		Loss functions
Epoch	Metric	$\mathcal{L}_{\text{Soc}}$	$\mathcal{L}_{\text{Soc}\setminus\beta}$	$\mathcal{L}_{\text{Soc}\setminus FT}$	$\mathcal{L}_{\text{Soc}\setminus FT_{idk}}$	$\mathcal{L}_{\text{Soc}\setminus FT_{gt}}$	$\mathcal{L}_{\text{Soc}\setminus FT,\beta}$	$\mathcal{L}_{\text{Soc}\setminus FT_{idk},\beta}$	$\mathcal{L}_{\text{Soc}\setminus FT_{gt},\beta}$	$\mathcal{L}_{\text{Soc}\setminus t_{a}}$	$\mathcal{L}_{\text{Soc}\setminus t_{a},FT}$
25	Acc	$35.21$	$34.46$	36.06	$35.46$	$35.42$	$35.21$	$34.99$	$\underline{35.98}$	$34.74$	$35.64$
25	ECE	$3.10$	$\underline{2.33}$	$2.37$	$3.09$	$2.51$	$2.67$	$3.19$	2.19	$2.93$	$2.54$
50	Acc	$50.71$	$50.70$	$51.62$	$50.32$	$\underline{51.77}$	$51.04$	$50.89$	52.10	$50.55$	$51.71$
50	ECE	$5.64$	$4.90$	2.10	$5.59$	$2.25$	$2.45$	$6.49$	$2.89$	$\underline{2.13}$	$2.65$
75	Acc	$59.09$	$58.73$	59.80	$58.63$	$59.62$	$59.07$	$58.90$	$60.10$	$59.52$	$\underline{59.60}$

Appendix

Table 7: Ablation Study for CIFAR-100 with VGG-16. ECE values and accuracy scores (Acc) in

\%

for the validation dataset including mean. Standard deviations are omitted due to space constraints and to facilitate clearer comparison, as they remained below 2% for accuracy and 0.2% for ECE. The results in bold represent the best outcomes and those in underlined the second-best. The models that perform best in terms of both accuracy and ECE, according to their Pareto plot positions, are shown in italics (epochs 150 and 300).

		Loss functions
Epoch	Metric	$\mathcal{L}_{\text{Soc}}$	$\mathcal{L}_{\text{Soc}\setminus\beta}$	$\mathcal{L}_{\text{Soc}\setminus FT}$	$\mathcal{L}_{\text{Soc}\setminus FT_{idk}}$	$\mathcal{L}_{\text{Soc}\setminus FT_{gt}}$	$\mathcal{L}_{\text{Soc}\setminus FT,\beta}$	$\mathcal{L}_{\text{Soc}\setminus FT_{idk},\beta}$	$\mathcal{L}_{\text{Soc}\setminus FT_{gt},\beta}$	$\mathcal{L}_{\text{Soc}\setminus t_{a}}$	$\mathcal{L}_{\text{Soc}\setminus t_{a},FT}$
25	Acc	$50.66$	$50.99$	$\underline{52.33}$	$50.89$	$52.31$	53.00	$50.48$	$51.54$	$50.35$	$48.43$
25	ECE	$6.53$	$5.69$	$8.53$	$5.44$	$8.99$	$7.96$	$\underline{4.64}$	$8.76$	2.51	$11.87$
50	Acc	$61.04$	$60.92$	$61.81$	$\underline{61.90}$	61.92	$60.78$	$60.96$	$60.78$	$57.98$	$60.37$
50	ECE	$6.61$	$7.25$	$10.31$	$6.05$	$10.97$	$12.07$	$\underline{3.97}$	$11.24$	3.21	$10.26$
75	Acc	$65.30$	$65.08$	$65.93$	$65.61$	66.58	$66.01$	$65.00$	$65.66$	$64.18$	$\underline{66.06}$

Appendix

Table 8: Ablation Study for CIFAR-100 with ResNet-110. ECE values and accuracy scores (Acc) in

\%

Panel of Figure 10 — Figure 10: Error rate (1 - accuracy) versus Expected Calibration Error (ECE) across epochs for VGG-16 (left) and ResNet-110 (right). Dotted lines indicate the threshold for acceptable calibration. Lower and leftward values indicate better performance. Lines are drawn every 25 epochs.

Figure 11: Curves illustrating the evolution of the Socrates method on CIFAR-100 with VGG-16 across multiple random seeds, evaluated on the validation set. The top plot shows the Expected Calibration Error (ECE, red curve) and error rate ( $1-accuracy$ , black curve) across epochs. The middle plot illustrates the average confidence values across epochs for the unknown class (denoted as idk, dark-blue curve) and the ground-truth class (denoted as gt, light-blue curve), including: 1) the idk (purple curve) and gt (pink curve) confidences when the model’s predictions are correct, and 2) the idk (orange curve) and gt (brown curve) confidences when the model’s predictions are incorrect. The bottom plot shows the frequency with which the idk class is selected as the top-1 prediction across epochs.

		Metric	Socrates	SAT	CCL-SC	Focal	FLSD	Brier	MC
CIFAR-10	VGG-16	Acc	$\textbf{88.80 }\pm\textbf{ 0.21}$	$88.37\pm 0.29$	$88.29\pm 0.35$	$\underline{88.53\pm 0.17}$	$88.51\pm 0.52$	$88.24\pm 0.33$	$88.02\pm 0.22$
		ECE	$\textbf{4.70 }\pm\textbf{ 0.41}$	$8.91\pm 0.33$	$9.22\pm 0.28$	$6.12\pm 0.26$	$\underline{6.03\pm 0.46}$	$6.81\pm 0.31$	$8.98\pm 0.29$
		AdaECE	$\textbf{6.20 }\pm\textbf{ 0.50}$	$8.98\pm 0.31$	$9.17\pm 0.24$	$\underline{6.38\pm 0.32}$	$6.47\pm 0.50$	$7.01\pm 0.39$	$8.97\pm 0.30$
		CW-ECE	$\textbf{1.34 }\pm\textbf{ 0.04}$	$\underline{1.40\pm 0.04}$	$1.56\pm 0.04$	$1.48\pm 0.03$	$1.47\pm 0.06$	$1.55\pm 0.07$	$1.51\pm 0.04$
SVHN	VGG-16	Acc	$\underline{96.51\pm 0.08}$	$96.50\pm 0.08$	$19.95\pm 0.00$	$65.75\pm 41.82$	$81.09\pm 34.18$	$\textbf{96.73 }\pm\textbf{ 0.13}$	$35.25\pm 34.23$
		ECE	$2.31\pm 0.13$	$\underline{0.95\pm 0.08}$	$\textbf{0.93 }\pm\textbf{ 0.28}$	$4.65\pm 2.72$	$2.43\pm 0.67$	$1.37\pm 0.18$	$3.01\pm 0.95$

Appendix

Table 9: Final validation set performance at epoch 300 for standard training and epoch 50 for transfer learning (TL). Metrics reported: accuracy, ECE, AdaptiveECE, and Classwise-ECE. Poor performance in SAT with Food-101, CCL-SC and MC with SVHN is attributed to premature convergence. Best results are highlighted in bold , and second-best are underlined .

Figure 12: Error rate (1 - accuracy) versus Expected Calibration Error (ECE) at the last epoch for different dataset–architectures on the test set. Dotted lines indicate the threshold for acceptable calibration. Lower and leftward values indicate better performance. SAT is excluded from Food-101 due to premature convergence.

Figure 13: Post-hoc results (Socrates and CE baselines). Error rate (1 - accuracy) versus Expected Calibration Error (ECE) at the last epoch for different dataset–architectures on the test set. Dotted lines indicate the threshold for acceptable calibration. Lower and leftward values indicate better performance. In the SVHN plot, the CE result is not visible because it is overlapped by CE+TS result.

Panel of Figure 14 — Figure 14: Pareto plot at the last epoch for the CIFAR-100 test set using ViT with Transfer Learning (TL) comparing ad-hoc methods (a) and post-hoc methods (b). CE is included in the comparison with post-hoc methods.

Uncertainty Calibration

Uncertainty calibration aligns model confidence and predictive intervals with observed correctness and outcome frequencies.

See all 264 papers

Oct 8, 2026

Do Not Train Away Uncertainty: Early Uncertainty Anchored Calibration

Deep neural networks, including large language models, have achieved remarkable performance across various tasks. However, they are prone to overconfidence during training or fine-tuning. In this work, we observe a consistent phenomenon across different models that the early model is better calibrated, while later training or fine-tuning yields marginal accuracy gains but substantially increases calibration errors. Our analysis suggests that the early model retains uncertainty awareness in both its predictions and features, which is gradually lost with continued training. To avoid training away this uncertainty awareness, we propose EUA-Cal, a novel method that exploits the Early model as an Uncertainty Anchor for Calibration. EUA-Cal introduces early prediction regularization to preserve early predictive uncertainty and prototype structure regularization to exploit uncertainty reflected in the early feature space, jointly mitigating overconfidence. Extensive experiments on image classification and multiple-choice question answering across eight diverse models demonstrate that EUA-Cal outperforms state-of-the-art calibration methods.

Yutong Xie, Jiawei Tang, Zhenglin Hua +6

School of Software Engineering, Southeast University, Nanjing, China · School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Computing Information Sciences, Saint Francis University, Hong Kong, China +2

Model Calibration Uncertainty Calibration Language Model Calibration

Ask PDF

May 20, 2026

Expectation Consistency Loss: Rethink Confidence Calibration under Covariate Shift

Confidence calibration for classification models is vital in safety-critical decision-making scenarios and has received extensive attention. General confidence calibration methods assume training and test data are independent and identically distributed, limiting their effectiveness under covariate shifts. Previous calibration methods under covariate shift struggle with class-wise or canonical calibrations and often rely on unstable importance weighting when density ratios are large or unbounded. Given the above limitations, this paper rethinks confidence calibration under covariate shifts. First, we derive a necessary and sufficient condition for confidence calibration under covariate shifts, named Expectation consistency condition, which reveals covariate shifts do not necessarily lead to uncalibrated confidence and provides a weaker condition for confidence calibration than global covariate distribution alignment. Then, utilizing Expectation consistency condition, this paper proposes an unsupervised domain adaptation loss to calibrate confidence of the target domain, named Expectation consistency loss (ECL), which is compatible with canonical calibration, class-wise calibration, and top-label calibration. Third, we prove that computing ECL loss has the same sample complexity as Expected Calibration Error (ECE) and provide a theoretically grounded mini-batch trainable scheme for ECL loss. Finally, we validate the effectiveness of our method on both simulated and real-world covariate shift datasets.

Jinzong Dong, Zhaohui Jiang, Bo Yang

School of Automation, Central South University, Changsha, China.

Probability Calibration Unsupervised Domain Adaptation Covariate Shift Model Calibration Uncertainty Calibration

Ask PDF

Aug 12, 2026

Confidence Calibration of Deep Learning Systems

In high-stakes applications, reliable confidence estimates are as important as the predictions themselves. Confidence calibration ensures that predicted probabilities reflect the likelihood of correctness, making it essential for safe deployment of deep learning models. However, existing methods typically assume access to clean validation data, which is often unrealistic due to label noise and domain shifts. This thesis develops methods for improving calibration under these conditions. First, we address calibration under label noise. Standard methods can produce misleading confidence estimates when labels are unreliable. We propose a framework that uses an estimated noise model to reconstruct noise-free confidence estimates by modeling the relationship between noisy and clean label distributions. We extend this approach to Conformal Prediction (CP), which provides set-valued predictions with guaranteed coverage. Our noise-aware CP method estimates clean conformity scores despite label noise, enabling reliable uncertainty quantification. Next, we study calibration in unsupervised domain adaptation, where a model trained on a labeled source domain is adapted to an unlabeled target domain. Since labeled target data are unavailable, we estimate target-domain accuracy from source performance and domain discrepancies, enabling calibration without target labels. We also consider privacy-preserving settings in which user labels and model outputs must remain protected. We propose a locally differentially private conformal prediction framework that provides valid uncertainty quantification while maintaining privacy guarantees and balancing privacy, computational feasibility, and prediction reliability. Our results bridge calibration theory and practical deployment in safety-critical applications, contributing to reliable, privacy-preserving, and noise-resilient neural network predictions.

Coby Penso

The Faculty of Engineering, Bar-Ilan University, Ramat Gan, Israel

Unsupervised Domain Adaptation Conformal Prediction Model Calibration Uncertainty Calibration

Ask PDF

Abstract

Uncertainty Calibration

Explore similar work

Do Not Train Away Uncertainty: Early Uncertainty Anchored Calibration

Expectation Consistency Loss: Rethink Confidence Calibration under Covariate Shift

Confidence Calibration of Deep Learning Systems