Socrates Loss: Unifying Confidence Calibration and Classification by Leveraging the Unknown

Sandra Gómez-Gálvez; Tobias Olenyi; Gillian Dobbie; Katerina Taškova

cs.LGApr 14, 2026

Socrates Loss: Unifying Confidence Calibration and Classification by Leveraging the Unknown

Authors: Sandra Gómez-Gálvez, Tobias Olenyi, Gillian Dobbie, Katerina Taškova

Organizations: University of Auckland · Technical University of Munich

Abstract

Deep neural networks, despite their high accuracy, often exhibit poor confidence calibration, limiting their reliability in high-stakes applications. Current ad-hoc confidence calibration methods attempt to fix this during training but face a fundamental trade-off: two-phase training methods achieve strong classification performance at the cost of training instability and poorer confidence calibration, while single-loss methods are stable but underperform in classification. This paper addresses and mitigates this stability-performance trade-off. We propose Socrates Loss, a novel, unified loss function that explicitly leverages uncertainty by incorporating an auxiliary unknown class, whose predictions directly influence the loss function and a dynamic uncertainty penalty. This unified objective allows the model to be optimized for both classification and confidence calibration simultaneously, without the instability of complex, scheduled losses. We provide theoretical guarantees that our method regularizes the model to prevent miscalibration and overfitting. Across four benchmark datasets and multiple architectures, our comprehensive experiments demonstrate that Socrates Loss consistently improves training stability while achieving more favorable accuracy-calibration trade-off, often converging faster than existing methods.

Figures & tables

Figure 1: Evolution of sub-components of the Socrates Loss, the dynamic uncertainty penalty (orange curve) and adaptive target (blue curve), alongside the Expected Calibration Error (red curve) and error rate ( $1-accuracy$ , black curve) across epochs on CIFAR-100 with VGG-16. Results are averaged over five runs.

Panel of Figure 2 — Figure 2: Loss trends for CIFAR-10 with VGG-16 (a), SVHN with VGG-16 (b), Food-101 with ResNet-34 (c), CIFAR-100 with VGG-16 (d), ResNet-110 (e), ViT (f), and ViT Transfer Learning (g). The black rectangles indicate the periods of loss change in the two-phase training approaches.

Figure 3: Error rate (1 - accuracy) versus Expected Calibration Error (ECE) across epochs for different datasets-architectures. Dotted lines indicate the threshold for acceptable calibration. Lower and leftward values indicate better performance. SAT is excluded from Food-101 due to premature convergence. Lines are drawn every 25 epochs.

Panel of Figure 4 — Figure 4: Pareto plot across epochs (lines are drawn every 2 epochs) (a) and reliability diagram at the final epoch 50 (b) for the CIFAR-100 validation set using ViT with Transfer Learning (TL). In (b), the SAT curve overlaps with that of CE.

Figure 5: Reliability diagrams for the validation set at the final training epoch (epoch 300).

		Metric	Socrates	SAT	CCL-SC	Focal	FLSD	Brier	MC
CIFAR-10	VGG-16	Acc	$\textbf{88.42 }\pm\textbf{ 0.05}$	$88.29\pm 0.34$	$87.92\pm 0.26$	$\underline{88.31\pm 0.19}$	$88.16\pm 0.17$	$87.67\pm 0.10$	$87.53\pm 0.12$
		ECE	$\textbf{4.39 }\pm\textbf{ 0.25}$	$9.00\pm 0.23$	$9.57\pm 0.36$	$\underline{6.27\pm 0.16}$	$6.29\pm 0.16$	$7.21\pm 0.12$	$9.34\pm 0.17$
		AdaECE	$\textbf{6.03 }\pm\textbf{ 0.18}$	$9.03\pm 0.23$	$9.50\pm 0.36$	$\underline{6.32\pm 0.25}$	$6.38\pm 0.20$	$7.23\pm 0.15$	$9.34\pm 0.17$
		CW-ECE	$\textbf{1.31 }\pm\textbf{ 0.01}$	$\underline{1.34\pm 0.05}$	$1.55\pm 0.02$	$1.41\pm 0.03$	$1.42\pm 0.02$	$1.50\pm 0.03$	$1.49\pm 0.03$
SVHN	VGG-16	Acc	$\underline{97.25\pm 0.08}$	$97.21\pm 0.03$	$19.59\pm 0.00$	$66.08\pm 42.44$	$81.62\pm 34.68$	$\textbf{97.31 }\pm\textbf{ 0.05}$	$35.06\pm 34.59$
		ECE	$2.49\pm 0.03$	$\underline{0.60\pm 0.04}$	$\textbf{0.57 }\pm\textbf{ 0.28}$	$4.77\pm 3.15$	$2.31\pm 0.52$	$1.32\pm 0.12$	$2.70\pm 0.84$

Table 1: Test set performance at epoch 300 for standard training and at epoch 50 for transfer learning (TL). Metrics reported: accuracy (acc), ECE, AdaptiveECE (AdaECE), and Classwise-ECE (CW-ECE). Poor performance using SAT on Food-101, and CCL-SC and MC on SVHN is attributed to premature convergence. Best results are highlighted in bold , and second-best are underlined .

		Metric	Socrates	Socrates+TS	Socrates+MS	Socrates+VS	CE	CE+TS	CE+MS	CE+VS
CIFAR-10	VGG-16	Acc	$88.42\pm 0.05$	$88.40\pm 0.04$	$88.46\pm 0.14$	$88.37\pm 0.06$	$88.46\pm 0.28$	$88.52\pm 0.28$	$88.45\pm 0.29$	$\textbf{88.55}\pm\textbf{0.37}$
		ECE	$4.39\pm 0.25$	$8.32\pm 0.33$	$\textbf{2.60}\pm\textbf{0.48}$	$3.21\pm 0.44$	$9.09\pm 0.24$	$4.67\pm 0.20$	$2.78\pm 0.10$	$2.63\pm 0.31$
		AdaECE	$6.03\pm 0.18$	$8.90\pm 0.14$	$\textbf{3.25}\pm\textbf{0.48}$	$3.82\pm 0.35$	$9.08\pm 0.23$	$5.31\pm 0.23$	$3.58\pm 0.23$	$3.53\pm 0.07$
		CW-ECE	$1.31\pm 0.01$	$2.09\pm 0.02$	$\textbf{1.29}\pm\textbf{0.01}$	$1.30\pm 0.01$	$1.42\pm 0.02$	$1.39\pm 0.04$	$1.41\pm 0.04$	$1.42\pm 0.05$
SVHN	VGG-16	Acc	$97.25\pm 0.08$	$97.24\pm 0.09$	$\textbf{97.30}\pm\textbf{0.05}$	$97.27\pm 0.09$	$97.19\pm 0.04$	$97.20\pm 0.04$	$97.16\pm 0.05$	$97.19\pm 0.05$
		ECE	$2.49\pm 0.03$	$6.91\pm 0.05$	$\textbf{0.56}\pm\textbf{0.03}$	$0.63\pm 0.08$	$1.58\pm 0.05$	$1.57\pm 0.07$	$0.71\pm 0.05$	$0.76\pm 0.05$

Table 2: Test set performance with post-hoc confidence calibration methods at epoch 300 for standard training and at epoch 50 for transfer learning (TL). Metrics reported: accuracy (acc), ECE, AdaptiveECE (AdaECE), and Classwise-ECE (CW-ECE). Best results are highlighted in bold .

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

		CIFAR-10	SVHN	Food-101	CIFAR-100	CIFAR-100	CIFAR-100	CIFAR-100
Method	Hyperp.	VGG-16	VGG-16	ResNet-34	VGG-16	ViT	ViT TL	ResNet-110
	$\gamma$	2	1	2	2	4	1	1
Socrates	$\alpha$	0.8	0.9	0.999	0.999	1	0.999	0.999
	$E_{s}$	0	0	0	0	0	0	0
	MB	128	128	128	128	128	128	128
	$q$	0.999	0.99	0.9	0.99	0.9	0.9	0.9

Appendix

Table 3: Final hyperparameters. Underlined values are from their original research.

Figure 6: Evolution of the Socrates method on CIFAR-100 with VGG-16 for all hyperparameter configurations. The curves show the mean values across epochs of the dynamic uncertainty penalty ( $\beta$ penalty, orange curve), the adaptive target (blue curve), the Expected Calibration Error (ECE, red curve) and error rate ( $1-accuracy$ , black curve).

		$\gamma$ sensitivity
Epoch	Metric	0	1	2	3	4
25	Acc	$\textbf{36.06}\pm\textbf{0.64}$	$34.56\pm 1.55$	$35.21\pm 2.01$	$\underline{35.22\pm 1.03}$	$33.98\pm 0.58$
25	ECE	$\textbf{2.37}\pm\textbf{0.35}$	$\underline{2.63\pm 1.06}$	$3.10\pm 1.15$	$4.45\pm 1.48$	$5.00\pm 1.00$
50	Acc	$\textbf{51.62}\pm\textbf{0.53}$	$\underline{50.95\pm 0.45}$	$50.71\pm 0.33$	$49.48\pm 0.65$	$49.10\pm 0.68$
50	ECE	$\textbf{2.10}\pm\textbf{0.33}$	$\underline{3.17\pm 0.99}$	$5.64\pm 0.99$	$8.32\pm 1.42$	$10.60\pm 0.25$
75	Acc	$\textbf{59.80}\pm\textbf{0.73}$	$58.48\pm 0.46$	$\underline{59.09\pm 0.91}$	$57.65\pm 0.64$	$56.60\pm 0.50$

Appendix

Table 4: Hyperparameter sensitivity analysis -

\gamma

sensitivity. ECE values in a range of

[0,1]

and accuracy (

\%

) scores (Acc) for the validation dataset including mean and standard deviation. Best results are highlighted in bold and second-best results are underlined . The models that perform best in terms of both accuracy and ECE, according to their Pareto plot positions, are shown in italics (epochs 150 and 300).

		$\alpha$ sensitivity
Epoch	Metric	0.8	0.9	0.99	0.999	1
25	Acc	$4.72\pm 1.62$	$17.30\pm 0.76$	$34.87\pm 0.80$	$\textbf{35.21}\pm\textbf{2.01}$	$\underline{34.97\pm 0.72}$
25	ECE	$15.36\pm 2.08$	$12.56\pm 0.47$	$5.48\pm 1.82$	$\underline{3.10\pm 1.15}$	$\textbf{2.39}\pm\textbf{0.82}$
50	Acc	$15.12\pm 2.03$	$27.98\pm 2.17$	$\underline{50.52\pm 0.41}$	$\textbf{50.71}\pm\textbf{0.33}$	$50.39\pm 0.79$
50	ECE	$16.69\pm 1.49$	$15.62\pm 1.96$	$10.20\pm 0.98$	$\underline{5.64\pm 0.99}$	$\textbf{4.78}\pm\textbf{0.94}$
75	Acc	$22.86\pm 1.02$	$38.80\pm 0.92$	$58.20\pm 0.81$	$\textbf{59.09}\pm\textbf{0.91}$	$\underline{58.50\pm 0.51}$

Appendix

Table 5: Hyperparameter sensitivity analysis -

\alpha

sensitivity. ECE values in a range of

[0,1]

and accuracy (

\%

Panel of Figure 7 — Figure 7: Error rate (1 - accuracy) versus Expected Calibration Error (ECE) across epochs for $\gamma$ (left) and $\alpha$ (right). Dotted lines indicate the threshold for acceptable calibration. Lower and leftward values indicate better performance. Lines are drawn every 25 epochs.

Panel of Figure 8 — Figure 8: Curves depicting the average values of the unknown class confidences (left) and Expected Calibration Error (ECE) values (right) across epochs for CIFAR-100 with VGG-16 (a) and ResNet-110 (b), using the SAT method. Gray boxes highlight instability and miscalibration due to loss function change at epoch 200 (a) and 150 (b). It is evident that when confidence in the unknown class increases on average, ECE suddenly changes and increases as well.

		$\beta$ sensitivity
Epoch	Metric	$\beta_{\setminus GT}$	$\beta_{\setminus GT,idk}$	$\beta_{fixed}$	$\setminus\beta$
25	Acc	$\underline{35.21\pm 2.01}$	$\textbf{35.72}\pm\textbf{0.95}$	$34.95\pm 0.81$	$34.46\pm 1.84$
25	ECE	$3.10\pm 1.15$	$3.03\pm 1.42$	$\textbf{2.09}\pm\textbf{0.86}$	$\underline{2.33\pm 1.40}$
50	Acc	$\underline{50.71\pm 0.33}$	$\textbf{51.21}\pm\textbf{0.27}$	$50.39\pm 0.33$	$50.70\pm 1.03$
50	ECE	$5.64\pm 0.99$	$6.32\pm 0.96$	$\underline{5.01\pm 0.45}$	$\textbf{4.90}\pm\textbf{1.23}$
75	Acc	$\textbf{59.09}\pm\textbf{0.91}$	$58.56\pm 0.58$	$58.71\pm 0.68$	$\underline{58.73\pm 0.47}$

Appendix

Table 6: Hyperparameter sensitivity analysis -

\beta

sensitivity. ECE values in a range of

[0,1]

and accuracy (

\%

Figure 9: Error rate (1 - accuracy) versus Expected Calibration Error (ECE) across epochs for $\beta$ . Dotted lines indicate the threshold for acceptable calibration. Lower and leftward values indicate better performance. Lines are drawn every 25 epochs.

		Loss functions
Epoch	Metric	$\mathcal{L}_{\text{Soc}}$	$\mathcal{L}_{\text{Soc}\setminus\beta}$	$\mathcal{L}_{\text{Soc}\setminus FT}$	$\mathcal{L}_{\text{Soc}\setminus FT_{idk}}$	$\mathcal{L}_{\text{Soc}\setminus FT_{gt}}$	$\mathcal{L}_{\text{Soc}\setminus FT,\beta}$	$\mathcal{L}_{\text{Soc}\setminus FT_{idk},\beta}$	$\mathcal{L}_{\text{Soc}\setminus FT_{gt},\beta}$	$\mathcal{L}_{\text{Soc}\setminus t_{a}}$	$\mathcal{L}_{\text{Soc}\setminus t_{a},FT}$
25	Acc	$35.21$	$34.46$	36.06	$35.46$	$35.42$	$35.21$	$34.99$	$\underline{35.98}$	$34.74$	$35.64$
25	ECE	$3.10$	$\underline{2.33}$	$2.37$	$3.09$	$2.51$	$2.67$	$3.19$	2.19	$2.93$	$2.54$
50	Acc	$50.71$	$50.70$	$51.62$	$50.32$	$\underline{51.77}$	$51.04$	$50.89$	52.10	$50.55$	$51.71$
50	ECE	$5.64$	$4.90$	2.10	$5.59$	$2.25$	$2.45$	$6.49$	$2.89$	$\underline{2.13}$	$2.65$
75	Acc	$59.09$	$58.73$	59.80	$58.63$	$59.62$	$59.07$	$58.90$	$60.10$	$59.52$	$\underline{59.60}$

Appendix

Table 7: Ablation Study for CIFAR-100 with VGG-16. ECE values and accuracy scores (Acc) in

\%

for the validation dataset including mean. Standard deviations are omitted due to space constraints and to facilitate clearer comparison, as they remained below 2% for accuracy and 0.2% for ECE. The results in bold represent the best outcomes and those in underlined the second-best. The models that perform best in terms of both accuracy and ECE, according to their Pareto plot positions, are shown in italics (epochs 150 and 300).

		Loss functions
Epoch	Metric	$\mathcal{L}_{\text{Soc}}$	$\mathcal{L}_{\text{Soc}\setminus\beta}$	$\mathcal{L}_{\text{Soc}\setminus FT}$	$\mathcal{L}_{\text{Soc}\setminus FT_{idk}}$	$\mathcal{L}_{\text{Soc}\setminus FT_{gt}}$	$\mathcal{L}_{\text{Soc}\setminus FT,\beta}$	$\mathcal{L}_{\text{Soc}\setminus FT_{idk},\beta}$	$\mathcal{L}_{\text{Soc}\setminus FT_{gt},\beta}$	$\mathcal{L}_{\text{Soc}\setminus t_{a}}$	$\mathcal{L}_{\text{Soc}\setminus t_{a},FT}$
25	Acc	$50.66$	$50.99$	$\underline{52.33}$	$50.89$	$52.31$	53.00	$50.48$	$51.54$	$50.35$	$48.43$
25	ECE	$6.53$	$5.69$	$8.53$	$5.44$	$8.99$	$7.96$	$\underline{4.64}$	$8.76$	2.51	$11.87$
50	Acc	$61.04$	$60.92$	$61.81$	$\underline{61.90}$	61.92	$60.78$	$60.96$	$60.78$	$57.98$	$60.37$
50	ECE	$6.61$	$7.25$	$10.31$	$6.05$	$10.97$	$12.07$	$\underline{3.97}$	$11.24$	3.21	$10.26$
75	Acc	$65.30$	$65.08$	$65.93$	$65.61$	66.58	$66.01$	$65.00$	$65.66$	$64.18$	$\underline{66.06}$

Appendix

Table 8: Ablation Study for CIFAR-100 with ResNet-110. ECE values and accuracy scores (Acc) in

\%

Panel of Figure 10 — Figure 10: Error rate (1 - accuracy) versus Expected Calibration Error (ECE) across epochs for VGG-16 (left) and ResNet-110 (right). Dotted lines indicate the threshold for acceptable calibration. Lower and leftward values indicate better performance. Lines are drawn every 25 epochs.

Figure 11: Curves illustrating the evolution of the Socrates method on CIFAR-100 with VGG-16 across multiple random seeds, evaluated on the validation set. The top plot shows the Expected Calibration Error (ECE, red curve) and error rate ( $1-accuracy$ , black curve) across epochs. The middle plot illustrates the average confidence values across epochs for the unknown class (denoted as idk, dark-blue curve) and the ground-truth class (denoted as gt, light-blue curve), including: 1) the idk (purple curve) and gt (pink curve) confidences when the model’s predictions are correct, and 2) the idk (orange curve) and gt (brown curve) confidences when the model’s predictions are incorrect. The bottom plot shows the frequency with which the idk class is selected as the top-1 prediction across epochs.

		Metric	Socrates	SAT	CCL-SC	Focal	FLSD	Brier	MC
CIFAR-10	VGG-16	Acc	$\textbf{88.80 }\pm\textbf{ 0.21}$	$88.37\pm 0.29$	$88.29\pm 0.35$	$\underline{88.53\pm 0.17}$	$88.51\pm 0.52$	$88.24\pm 0.33$	$88.02\pm 0.22$
		ECE	$\textbf{4.70 }\pm\textbf{ 0.41}$	$8.91\pm 0.33$	$9.22\pm 0.28$	$6.12\pm 0.26$	$\underline{6.03\pm 0.46}$	$6.81\pm 0.31$	$8.98\pm 0.29$
		AdaECE	$\textbf{6.20 }\pm\textbf{ 0.50}$	$8.98\pm 0.31$	$9.17\pm 0.24$	$\underline{6.38\pm 0.32}$	$6.47\pm 0.50$	$7.01\pm 0.39$	$8.97\pm 0.30$
		CW-ECE	$\textbf{1.34 }\pm\textbf{ 0.04}$	$\underline{1.40\pm 0.04}$	$1.56\pm 0.04$	$1.48\pm 0.03$	$1.47\pm 0.06$	$1.55\pm 0.07$	$1.51\pm 0.04$
SVHN	VGG-16	Acc	$\underline{96.51\pm 0.08}$	$96.50\pm 0.08$	$19.95\pm 0.00$	$65.75\pm 41.82$	$81.09\pm 34.18$	$\textbf{96.73 }\pm\textbf{ 0.13}$	$35.25\pm 34.23$
		ECE	$2.31\pm 0.13$	$\underline{0.95\pm 0.08}$	$\textbf{0.93 }\pm\textbf{ 0.28}$	$4.65\pm 2.72$	$2.43\pm 0.67$	$1.37\pm 0.18$	$3.01\pm 0.95$

Appendix

Table 9: Final validation set performance at epoch 300 for standard training and epoch 50 for transfer learning (TL). Metrics reported: accuracy, ECE, AdaptiveECE, and Classwise-ECE. Poor performance in SAT with Food-101, CCL-SC and MC with SVHN is attributed to premature convergence. Best results are highlighted in bold , and second-best are underlined .

Figure 12: Error rate (1 - accuracy) versus Expected Calibration Error (ECE) at the last epoch for different dataset–architectures on the test set. Dotted lines indicate the threshold for acceptable calibration. Lower and leftward values indicate better performance. SAT is excluded from Food-101 due to premature convergence.

Figure 13: Post-hoc results (Socrates and CE baselines). Error rate (1 - accuracy) versus Expected Calibration Error (ECE) at the last epoch for different dataset–architectures on the test set. Dotted lines indicate the threshold for acceptable calibration. Lower and leftward values indicate better performance. In the SVHN plot, the CE result is not visible because it is overlapped by CE+TS result.

Panel of Figure 14 — Figure 14: Pareto plot at the last epoch for the CIFAR-100 test set using ViT with Transfer Learning (TL) comparing ad-hoc methods (a) and post-hoc methods (b). CE is included in the comparison with post-hoc methods.

Uncertainty Calibration

Uncertainty calibration aligns model confidence and predictive intervals with observed correctness and outcome frequencies.

See all 264 papers

Do Not Train Away Uncertainty: Early Uncertainty Anchored Calibration
Oct 8, 2026Yutong Xie, Jiawei Tang, Zhenglin Hua +6Model Calibration Uncertainty Calibration
School of Software Engineering, Southeast University, Nanjing, China · School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Computing Information Sciences, Saint Francis University, Hong Kong, China +2
Expectation Consistency Loss: Rethink Confidence Calibration under Covariate Shift
May 20, 2026Jinzong Dong, Zhaohui Jiang, Bo YangProbability Calibration Unsupervised Domain Adaptation
School of Automation, Central South University, Changsha, China.
Confidence Calibration of Deep Learning Systems
Aug 12, 2026Coby PensoUnsupervised Domain Adaptation Conformal Prediction
The Faculty of Engineering, Bar-Ilan University, Ramat Gan, Israel

Oct 8, 2026J/K move · Enter open · S save

Do Not Train Away Uncertainty: Early Uncertainty Anchored Calibration

Yutong Xie, Jiawei Tang, Zhenglin Hua, Yuxiang Ma+5

School of Software Engineering, Southeast University, Nanjing, China · School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Computing Information Sciences, Saint Francis University, Hong Kong, China+2

Model Calibration Uncertainty Calibration Language Model Calibration

Open paper AskPDF arXiv

Deep neural networks, including large language models, have achieved remarkable performance across various tasks. However, they are prone to overconfidence during training or fine-tuning. In this work, we observe a consistent phenomenon across different models that the early model is better calibrated, while later training or fine-tuning yields marginal accuracy gains but substantially increases calibration errors. Our analysis suggests that the early model retains uncertainty awareness in both its predictions and features, which is gradually lost with continued training. To avoid training away this uncertainty awareness, we propose EUA-Cal, a novel method that exploits the Early model as an Uncertainty Anchor for Calibration. EUA-Cal introduces early prediction regularization to preserve early predictive uncertainty and prototype structure regularization to exploit uncertainty reflected in the early feature space, jointly mitigating overconfidence. Extensive experiments on image classification and multiple-choice question answering across eight diverse models demonstrate that EUA-Cal outperforms state-of-the-art calibration methods.

Abstract

Uncertainty Calibration

Explore similar work

Do Not Train Away Uncertainty: Early Uncertainty Anchored Calibration

Expectation Consistency Loss: Rethink Confidence Calibration under Covariate Shift

Confidence Calibration of Deep Learning Systems