Calibration in One Go: Jointly Learning Accurate and Calibrated Neural Networks

Nektarios Kalampalikis; Pablo M. Olmos; Isabel Valera

cs.LGOct 9, 2026

Calibration in One Go: Jointly Learning Accurate and Calibrated Neural Networks

Authors: Nektarios Kalampalikis, Pablo M. Olmos, Isabel Valera

Organizations: Dept. of Computer Science Saarland University, Germany

Abstract

Neural networks (NNs), while highly flexible and versatile, are often poorly calibrated. Post-hoc calibration methods adjust the outputs of an already trained model using a separate calibration dataset and additional calibration parameters. This process adds overhead that requires separate optimization of the calibrator and subsequent model selection. In this paper, we show that standard (post-hoc) calibrators can be trained end-to-end with the backbone via a bilevel Stackelberg formulation. This formulation eliminates the need for a separate calibration set or a dedicated calibration loss by jointly optimizing accuracy and calibration using cross-entropy, introducing only one additional hyperparameter: the calibrator learning rate. Although our framework supports calibrators ranging from temperature and matrix scaling to NN layers, our results show that using the final classification layer itself as the calibrator yields accurate, well-calibrated models with no additional parameters. Our approach outperforms SOTA calibrators with minimal added complexity, while also avoiding the multiplicative overhead of separate calibration components in settings such as multi-task learning.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Reliability diagrams on CIFAR-100 for the baseline, COG, and temperature scaling, with COG achieving the best calibration. (b) Post-hoc calibration: the model is trained and frozen before fitting a calibrator. (c) Our framework, COG, integrates the calibrator into the training procedure and constrains its learning rate to be slower than that of the model.

		# Extra	End	Calibration	Extra
	Method	Trainable Params.	To End	Data	Burden
Single-Task	Baseline	0	✓	✗	-
	COG-FC	0	✓	✗	1 Learning Rate
	Label Smoothing	0	✓	✗	1 Smoothing Level
	COG+LS	0	✓	✗	(a) 1 Smoothing Level (b) 1 Learning Rate
	Temp. Scaling	1	✗	✓	Calibrator Training

Table 1: Complexity of different calibration methods in Single-Task settings, where

K

denotes the number of classes. Tab. 7 extends this comparison to MTL settings.

Figure 2: Pairwise comparison of different calibrators and their corresponding COG variants. The warning symbol indicates failure to converge.

	SVHN			CIFAR-10			CIFAR-100
Method	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$
Baseline	$95.77_{0.18}$	$0.32_{0.01}$	$0.07_{0.00}$	$\mathbf{90.50_{0.21}}$	$0.64_{0.02}$	$0.17_{0.00}$	$65.04_{1.33}$	$2.98_{0.06}$	$0.60_{0.02}$
COG-FC	$95.84_{0.20}$	$\mathbf{0.16_{0.00}}$	$\mathbf{0.06_{0.00}}$	$89.94_{0.25}$	$0.38_{0.01}$	$0.16_{0.00}$	$70.46_{0.30}$	$\mathbf{1.08_{0.01}}$	$\mathbf{0.41_{0.00}}$
COG+LS	$\mathbf{96.25_{0.15}}$	$0.21_{0.00}$	$0.07_{0.00}$	$89.92_{0.21}$	$0.38_{0.00}$	$0.16_{0.00}$	$\mathbf{72.67_{0.78}}$	$1.31_{0.04}$	$\mathbf{0.41_{0.01}}$
Label Smoothing	$95.97_{0.12}$	$0.23_{0.00}$	$0.07_{0.00}$	$90.05_{0.31}$	$0.43_{0.01}$	$0.16_{0.00}$	$65.76_{0.88}$	$1.58_{0.04}$	$0.48_{0.01}$
Temp. Scaling	$95.77_{0.18}$	$0.19_{0.01}$	$0.07_{0.00}$	$\mathbf{90.50_{0.21}}$	$0.38_{0.01}$	$\mathbf{0.15_{0.00}}$	$65.04_{1.33}$	$1.78_{0.03}$	$0.53_{0.02}$

Table 2: Comparison of calibration metrics across datasets, and methods. The best value in each column among the subset of models is bolded. The Top

\text{Ranked}^{*}

row reports the best-performing model per dataset based on average rank across metrics as determined from the full results in Tab. 12 : COG-MS on SVHN, Matrix Scaling on CIFAR-10, and COG-FC on CIFAR-100.

Figure 3: Reliability diagrams for COG-FC and the top-ranked methods using 15 bins. For CIFAR-100, where COG-FC is the top-ranked method, we compare it with COG+LS.

	SVHN		CIFAR-10		CIFAR-100
Method	AUROC $(\uparrow)$	FPR95 $(\downarrow)$	AUROC $(\uparrow)$	FPR95 $(\downarrow)$	AUROC $(\uparrow)$	FPR95 $(\downarrow)$
Baseline	$94.78_{0.24}$	$34.43_{1.27}$	$83.97_{2.70}$	$76.39_{3.20}$	$71.92_{1.33}$	$87.67_{1.34}$
COG-FC	$\mathbf{96.60_{0.41}}$	$\mathbf{21.60_{2.59}}$	$\mathbf{87.00_{1.32}}$	$73.98_{4.03}$	$68.50_{2.22}$	$85.83_{1.43}$
COG+LS	$93.80_{0.61}$	$22.58_{2.45}$	$81.93_{3.10}$	$\mathbf{73.43_{3.18}}$	$65.48_{7.85}$	$\mathbf{83.71_{3.42}}$
Label Smoothing	$80.58_{1.66}$	$42.84_{2.40}$	$72.17_{6.88}$	$78.88_{3.09}$	$\mathbf{74.35_{2.98}}$	$84.06_{1.86}$
Temp. Scaling	$95.33_{0.25}$	$31.14_{1.85}$	$85.09_{2.43}$	$75.04_{4.22}$	$73.23_{1.53}$	$85.46_{2.00}$

Table 3: Performance on out-of-distribution detection. The best value in each column among the subset of models is bolded. The Top

\text{Ranked}^{*}

row represents the best model per dataset based on average rank across metrics as determined from the full results in Tab. 16 : COG-MS on SVHN, and COG-TST on CIFAR-10 and CIFAR-100.

	SVHN			CIFAR-10			CIFAR-100
Method	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$
Baseline	$\mathbf{47.01_{1.06}}$	$5.29_{0.23}$	$0.94_{0.02}$	$\mathbf{74.28_{0.59}}$	$2.03_{0.11}$	$0.45_{0.01}$	$45.85_{1.36}$	$5.70_{0.21}$	$0.92_{0.02}$
COG-FC	$45.82_{1.32}$	$2.43_{0.12}$	$0.80_{0.02}$	$68.80_{0.68}$	$1.57_{0.09}$	$0.50_{0.01}$	$44.09_{0.42}$	$2.88_{0.04}$	$0.77_{0.01}$
COG+LS	$46.05_{0.57}$	$\mathbf{1.91_{0.02}}$	$\mathbf{0.77_{0.00}}$	$69.31_{0.42}$	$1.11_{0.02}$	$0.46_{0.01}$	$44.92_{0.69}$	$2.90_{0.11}$	$0.81_{0.03}$
Label Smoothing	$46.19_{0.59}$	$2.19_{0.04}$	$0.86_{0.01}$	$74.22_{0.42}$	$1.03_{0.02}$	$0.42_{0.01}$	$\mathbf{47.14_{0.91}}$	$\mathbf{2.54_{0.10}}$	$\mathbf{0.71_{0.03}}$
Temp. Scaling	$\mathbf{47.01_{1.06}}$	$3.05_{0.12}$	$0.85_{0.02}$	$\mathbf{74.28_{0.59}}$	$1.19_{0.05}$	$0.41_{0.01}$	$45.85_{1.36}$	$3.32_{0.10}$	$0.81_{0.02}$

Table 4: Performance under covariate shifts. The best value in each column among the subset of models is bolded. The Top

\text{Ranked}^{*}

row reports the best-performing model per dataset based on average rank across metrics as determined from the full results in Tab. 17 : Matrix Scaling on SVHN and CIFAR-10, Label Smoothing on CIFAR-100.

Method	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$
Baseline	$73.51_{0.57}$	$0.81_{0.02}$	$0.38_{0.01}$
COG-FC	$\mathbf{74.97}_{0.47}$	$\mathbf{0.76}_{0.02}$	$\mathbf{0.36}_{0.01}$
Temp. Scaling	$73.51_{0.57}$	$\mathbf{0.76_{0.01}}$	$0.37_{0.01}$
COG-TS	$73.73_{0.43}$	$0.81_{0.02}$	$0.38_{0.00}$
Matrix Scaling	$74.04_{0.35}$	$0.74_{0.01}$	$\mathbf{0.36}_{0.00}$
COG-MS	$73.14_{0.71}$	$0.81_{0.02}$	$0.38_{0.01}$

Table 5: Performance with a ViT backbone on CIFAR-10. Bold indicates the best value per metric

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

		# Extra	End	Calibration	Extra
	Method	Trainable Params.	To End	Data	Burden
Single-Task	Baseline	0	✓	✗	-
	COG-FC	0	✓	✗	1 Learning Rate
	Label Smoothing	0	✓	✗	1 Smoothing Level
	COG+LS	0	✓	✗	(a) 1 Smoothing Level (b) 1 Learning Rate
	Temp. Scaling	1	✗	✓	Calibrator Training

Appendix

Table 7: Complexity of different calibration methods in Single-Task and Multi-Task learning settings, where

K

denotes the number of classes.

		CIFAR-10		CIFAR-100		SVHN		CelebA
	Method	$\lambda_{f}$	$\lambda_{c}$	$\lambda_{f}$	$\lambda_{c}$	$\lambda_{f}$	$\lambda_{c}$	$\lambda_{f}$	$\lambda_{c}$
ResNet	COG-FC	1e-3	5e-5	1e-3	1e-5	1e-3	1e-5	1e-3	1e-5
	COG-TS	1e-3	1e-4	1e-3	1e-5	1e-3	1e-4	1e-2	5e-5
	COG-MS	1e-3	5e-5	5e-4	1e-5	5e-4	1e-5	-	-
	COG-TST	1e-3	5e-5	5e-4	1e-5	5e-4	1e-4	-	-
WRN	COG-FC	1e-3	1e-5	1e-2	1e-5	1e-3	5e-5	-	-

Appendix

Table 8: Hyperparameter configurations selected based on validation performance for the different COG variants.

\lambda_{f}

and

\lambda_{c}

denote the learning rates used for model training and calibration, respectively. A dash indicates that the corresponding component was not used for that method.

Figure 5: Reliability Diagrams for all COG variants and their corresponding calibration methods using 15 bins.

	SVHN				CIFAR-10				CIFAR-100
Method	ECE-10	ECE-15	ECE-100	ECE-500	ECE-10	ECE-15	ECE-100	ECE-500	ECE-10	ECE-15	ECE-100	ECE-500
Baseline	$3.26_{0.11}$ [8]	$3.25_{0.11}$ [8]	$3.29_{0.10}$ [8]	$3.45_{0.10}$ [8]	$7.38_{0.25}$ [8]	$7.39_{0.23}$ [8]	$7.47_{0.23}$ [8]	$7.89_{0.22}$ [8]	$26.56_{0.79}$ [7]	$26.56_{0.78}$ [7]	$26.60_{0.79}$ [7]	$26.76_{0.83}$ [7]
COG-FC	$1.14_{0.30}$ [2]	$1.16_{0.29}$ [2]	$1.32_{0.17}$ [2]	$1.86_{0.07}$ [2]	$5.32_{0.16}$ [4]	$5.32_{0.14}$ [4]	$5.49_{0.11}$ [4]	$6.22_{0.12}$ [5]	$8.00_{0.35}$ [2]	$8.00_{0.36}$ [2]	$8.13_{0.37}$ [2]	$9.20_{0.28}$ [2]
Temp Scaling	$2.19_{0.09}$ [7]	$2.20_{0.09}$ [7]	$2.26_{0.08}$ [7]	$2.57_{0.05}$ [7]	$5.40_{0.29}$ [5]	$5.41_{0.28}$ [5]	$5.57_{0.26}$ [5]	$6.21_{0.14}$ [4]	$18.88_{0.41}$ [6]	$18.88_{0.41}$ [6]	$18.91_{0.42}$ [6]	$19.26_{0.43}$ [6]
COG-TS	$1.23_{0.14}$ [4]	$1.24_{0.14}$ [4]	$1.41_{0.11}$ [4]	$1.95_{0.11}$ [4]	$5.52_{0.40}$ [6]	$5.52_{0.40}$ [6]	$5.66_{0.40}$ [6]	$6.40_{0.40}$ [6]	$11.14_{2.05}$ [3]	$11.14_{2.05}$ [3]	$11.21_{2.03}$ [3]	$11.96_{1.77}$ [3]
Matrix Scaling	$\mathbf{0.71_{0.16}}$ [1]	$\mathbf{0.73_{0.15}}$ [1]	$\mathbf{1.05_{0.07}}$ [1]	$\mathbf{1.84_{0.09}}$ [1]	$\mathbf{0.73_{0.12}}$ [1]	$\mathbf{0.89_{0.10}}$ [1]	$\mathbf{1.76_{0.07}}$ [1]	$\mathbf{3.74_{0.22}}$ [1]	$50.18_{1.79}$ [8]	$50.18_{1.79}$ [8]	$50.19_{1.79}$ [8]	$50.25_{1.79}$ [8]

Appendix

Table 9: Comparison of ECE across different COG variants and their corresponding post-hoc counterparts for varying numbers of bins. We report

\text{mean}_{\text{std}}\text{ [Rank]}

	SVHN				CIFAR-10				CIFAR-100
Method	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500
Baseline	$5.03_{0.44}$ [8]	$5.57_{0.28}$ [8]	$7.22_{0.22}$ [8]	$8.85_{0.22}$ [8]	$8.98_{0.43}$ [8]	$9.65_{0.41}$ [8]	$11.89_{0.54}$ [8]	$14.58_{0.46}$ [8]	$27.69_{0.86}$ [7]	$28.27_{0.84}$ [7]	$30.69_{0.83}$ [7]	$32.47_{0.84}$ [7]
COG-FC	$1.96_{0.66}$ [2]	$2.16_{0.67}$ [2]	$3.17_{0.33}$ [2]	$5.59_{0.17}$ [2]	$6.98_{0.36}$ [4]	$7.27_{0.29}$ [4]	$9.08_{0.36}$ [4]	$11.87_{0.34}$ [4]	$9.18_{0.46}$ [2]	$9.39_{0.47}$ [2]	$10.27_{0.51}$ [2]	$12.98_{0.46}$ [2]
Temp Scaling	$3.62_{0.27}$ [7]	$3.89_{0.30}$ [7]	$4.93_{0.30}$ [7]	$6.72_{0.29}$ [7]	$7.03_{0.40}$ [6]	$7.58_{0.41}$ [6]	$9.46_{0.46}$ [7]	$12.18_{0.33}$ [6]	$20.57_{0.39}$ [6]	$21.07_{0.44}$ [6]	$22.51_{0.55}$ [6]	$24.14_{0.50}$ [6]
COG-TS	$2.03_{0.35}$ [3]	$2.23_{0.39}$ [4]	$3.33_{0.25}$ [5]	$5.83_{0.35}$ [6]	$6.99_{0.51}$ [5]	$7.49_{0.56}$ [5]	$9.20_{0.65}$ [5]	$11.97_{0.78}$ [5]	$12.34_{2.19}$ [3]	$12.56_{2.28}$ [3]	$13.51_{2.42}$ [3]	$15.85_{2.03}$ [3]
Matrix Scaling	$\mathbf{1.26_{0.25}}$ [1]	$\mathbf{1.40_{0.23}}$ [1]	$\mathbf{2.63_{0.16}}$ [1]	$\mathbf{5.34_{0.14}}$ [1]	$\mathbf{1.46_{0.25}}$ [1]	$\mathbf{1.90_{0.34}}$ [1]	$\mathbf{4.15_{0.28}}$ [1]	$\mathbf{8.52_{0.37}}$ [1]	$50.30_{1.79}$ [8]	$50.38_{1.79}$ [8]	$50.91_{1.73}$ [8]	$51.63_{1.64}$ [8]

Appendix

Table 10: Comparison of RMSCE across different COG variants and their corresponding post-hoc counterparts for varying numbers of bins. We report

\text{mean}_{\text{std}}\text{ [Rank]}

	SVHN				CIFAR-10				CIFAR-100
Method	MCE-10	MCE-15	MCE-100	MCE-500	MCE-10	MCE-15	MCE-100	MCE-500	MCE-10	MCE-15	MCE-100	MCE-500
Baseline	$41.31_{18.97}$ [8]	$44.57_{16.42}$ [8]	$67.18_{6.90}$ [6]	$86.85_{2.54}$ [8]	$36.20_{3.99}$ [8]	$38.15_{2.71}$ [7]	$62.04_{7.32}$ [4]	$89.10_{1.47}$ [8]	$48.99_{2.05}$ [7]	$52.19_{1.21}$ [7]	$68.53_{10.04}$ [7]	$83.80_{4.79}$ [7]
COG-FC	$17.27_{6.00}$ [3]	$19.03_{6.81}$ [3]	$48.96_{19.74}$ [2]	$74.04_{1.04}$ [2]	$24.65_{3.82}$ [7]	$28.05_{9.34}$ [5]	$64.03_{12.07}$ [6]	$74.84_{2.75}$ [3]	$15.22_{1.28}$ [2]	$15.74_{1.50}$ [2]	$40.79_{27.35}$ [3]	$72.55_{19.63}$ [2]
Temp Scaling	$20.24_{5.18}$ [4]	$33.59_{25.20}$ [6]	$66.79_{16.02}$ [5]	$78.23_{3.06}$ [5]	$24.28_{3.67}$ [6]	$25.01_{1.70}$ [4]	$59.11_{13.25}$ [3]	$78.52_{4.12}$ [7]	$36.93_{1.15}$ [6]	$37.08_{1.10}$ [6]	$53.48_{17.14}$ [6]	$82.83_{1.14}$ [4]
COG-TS	$33.36_{28.15}$ [7]	$35.96_{26.44}$ [7]	$80.52_{2.18}$ [8]	$80.67_{1.99}$ [7]	$22.90_{4.49}$ [2]	$\mathbf{23.01_{2.66}}$ [1]	$\mathbf{50.63_{17.00}}$ [1]	$\mathbf{74.07_{4.23}}$ [1]	$19.63_{3.45}$ [3]	$20.76_{3.28}$ [3]	$42.74_{22.05}$ [4]	$83.17_{1.57}$ [6]
Matrix Scaling	$30.13_{29.41}$ [6]	$30.16_{29.38}$ [5]	$68.95_{21.59}$ [7]	$78.93_{2.55}$ [6]	$\mathbf{13.89_{8.78}}$ [1]	$23.63_{30.26}$ [2]	$70.17_{14.95}$ [8]	$77.00_{1.27}$ [5]	$59.68_{2.69}$ [8]	$67.51_{3.68}$ [8]	$77.04_{2.03}$ [8]	$92.62_{2.22}$ [8]

Appendix

Table 11: Comparison of MCE across different COG variants and their corresponding post-hoc counterparts for varying numbers of bins. We report

\text{mean}_{\text{std}}\text{ [Rank]}

. The relative ranking of methods changes across binning choices, while the MCE values increase substantially as the number of bins increases, highlighting the sensitivity of MCE to the binning scheme.

	SVHN				CIFAR-10				CIFAR-100
Method	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	Rank $(\downarrow)$	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	Rank $(\downarrow)$	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	Rank $(\downarrow)$
Baseline	$95.77_{0.18}$	$0.32_{0.01}$	$0.07_{0.00}$	8.67	$90.50_{0.21}$	$0.64_{0.02}$	$0.17_{0.00}$	8.00	$65.04_{1.33}$	$2.98_{0.06}$	$0.60_{0.02}$	8.33
COG-FC	$95.84_{0.20}$	$\mathbf{0.16_{0.00}}$	$\mathbf{0.06_{0.00}}$	2.67	$89.94_{0.25}$	$0.38_{0.01}$	$0.16_{0.00}$	5.67	$70.46_{0.30}$	$\mathbf{1.08_{0.01}}$	$\mathbf{0.41_{0.00}}$	$\color[rgb]{0,0.75,0.16}{\mathbf{1.33}}$
COG+LS	$\mathbf{96.25_{0.15}}$	$0.21_{0.00}$	$0.07_{0.00}$	4.67	$89.92_{0.21}$	$0.38_{0.00}$	$0.16_{0.00}$	6.00	$\mathbf{72.67_{0.78}}$	$1.31_{0.04}$	$\mathbf{0.41_{0.01}}$	2.00
COG-TS	$95.86_{0.05}$	$\mathbf{0.16_{0.00}}$	$\mathbf{0.06_{0.00}}$	2.33	$89.54_{0.15}$	$0.40_{0.02}$	$0.16_{0.00}$	8.00	$67.55_{0.62}$	$1.30_{0.06}$	$0.46_{0.00}$	3.33
COG-MS	$96.02_{0.11}$	$\mathbf{0.16_{0.00}}$	$\mathbf{0.06_{0.00}}$	$\color[rgb]{0,0.75,0.16}{\mathbf{1.33}}$	$89.27_{0.26}$	$0.41_{0.01}$	$0.17_{0.00}$	10.33	$67.74_{0.56}$	$1.93_{0.95}$	$0.46_{0.01}$	4.67

Appendix

Table 12: Comparison of all models and all variants seen in this work. We report

\text{mean}_{\text{std}}

. The best value in each column is shown in bold. Rank denotes the mean rank across ACC, NLL, and Brier for a given dataset (1 = best; ties share a rank).

	SVHN				CIFAR-10				CIFAR-100
Method	ECE-10	ECE-15	ECE-100	ECE-500	ECE-10	ECE-15	ECE-100	ECE-500	ECE-10	ECE-15	ECE-100	ECE-500
Baseline	$3.26_{0.11}$	$3.25_{0.11}$	$3.29_{0.10}$	$3.45_{0.10}$	$7.38_{0.25}$	$7.39_{0.23}$	$7.47_{0.23}$	$7.89_{0.22}$	$26.56_{0.79}$	$26.56_{0.78}$	$26.60_{0.79}$	$26.76_{0.83}$
COG-FC	$1.14_{0.30}$	$1.16_{0.29}$	$1.32_{0.17}$	$1.86_{0.07}$	$5.32_{0.16}$	$5.32_{0.14}$	$5.49_{0.11}$	$6.22_{0.12}$	$8.00_{0.35}$	$8.00_{0.36}$	$8.13_{0.37}$	$9.20_{0.28}$
COG+LS	$7.44_{0.20}$	$7.51_{0.19}$	$7.82_{0.17}$	$8.32_{0.17}$	$4.90_{0.18}$	$5.10_{0.31}$	$5.65_{0.31}$	$7.17_{0.26}$	$6.81_{2.10}$	$6.94_{1.85}$	$7.95_{1.59}$	$9.33_{1.07}$
COG-TS	$1.23_{0.14}$	$1.24_{0.14}$	$1.41_{0.11}$	$1.95_{0.11}$	$5.52_{0.40}$	$5.52_{0.40}$	$5.66_{0.40}$	$6.40_{0.40}$	$11.14_{2.05}$	$11.14_{2.05}$	$11.21_{2.03}$	$11.96_{1.77}$
COG-MS	$1.37_{0.08}$	$1.38_{0.08}$	$1.51_{0.06}$	$1.97_{0.10}$	$5.67_{0.24}$	$5.68_{0.24}$	$5.84_{0.25}$	$6.61_{0.15}$	$11.46_{2.24}$	$11.46_{2.24}$	$11.55_{2.19}$	$12.43_{1.82}$

Appendix

Table 13: Comparison of ECE (

\downarrow

lower is better) for all models and all variants seen in this work for multiple choices of the number of bins. We report

\text{mean}_{\text{std}}

. The best value in each column is shown in bold.

	SVHN				CIFAR-10				CIFAR-100
Method	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500
Baseline	$5.03_{0.44}$	$5.57_{0.28}$	$7.22_{0.22}$	$8.85_{0.22}$	$8.98_{0.43}$	$9.65_{0.41}$	$11.89_{0.54}$	$14.58_{0.46}$	$27.69_{0.86}$	$28.27_{0.84}$	$30.69_{0.83}$	$32.47_{0.84}$
COG-FC	$1.96_{0.66}$	$2.16_{0.67}$	$3.17_{0.33}$	$5.59_{0.17}$	$6.98_{0.36}$	$7.27_{0.29}$	$9.08_{0.36}$	$11.87_{0.34}$	$9.18_{0.46}$	$9.39_{0.47}$	$10.27_{0.51}$	$12.98_{0.46}$
COG+LS	$7.59_{0.17}$	$7.70_{0.14}$	$8.10_{0.13}$	$9.19_{0.14}$	$5.28_{0.28}$	$5.42_{0.42}$	$6.53_{0.37}$	$9.73_{0.25}$	$9.65_{2.03}$	$10.91_{2.71}$	$11.87_{2.54}$	$14.68_{1.98}$
COG-TS	$2.03_{0.35}$	$2.23_{0.39}$	$3.33_{0.25}$	$5.83_{0.35}$	$6.99_{0.51}$	$7.49_{0.56}$	$9.20_{0.65}$	$11.97_{0.78}$	$12.34_{2.19}$	$12.56_{2.28}$	$13.51_{2.42}$	$15.85_{2.03}$
COG-MS	$2.31_{0.14}$	$2.53_{0.21}$	$3.56_{0.13}$	$5.80_{0.19}$	$7.17_{0.49}$	$7.61_{0.60}$	$9.33_{0.53}$	$12.19_{0.41}$	$12.75_{2.35}$	$13.06_{2.48}$	$14.00_{2.52}$	$16.35_{2.28}$

Appendix

Table 14: Comparison of RMSCE (

\downarrow

lower is better) for all models and all variants seen in this work for multiple choices of the number of bins. We report

\text{mean}_{\text{std}}

. The best value in each column is shown in bold.

	SVHN				CIFAR-10				CIFAR-100
Method	MCE-10	MCE-15	MCE-100	MCE-500	MCE-10	MCE-15	MCE-100	MCE-500	MCE-10	MCE-15	MCE-100	MCE-500
Baseline	$41.31_{18.97}$	$44.57_{16.42}$	$67.18_{6.90}$	$86.85_{2.54}$	$36.20_{3.99}$	$38.15_{2.71}$	$62.04_{7.32}$	$89.10_{1.47}$	$48.99_{2.05}$	$52.19_{1.21}$	$68.53_{10.04}$	$83.80_{4.79}$
COG-FC	$17.27_{6.00}$	$19.03_{6.81}$	$48.96_{19.74}$	$74.04_{1.04}$	$24.65_{3.82}$	$28.05_{9.34}$	$64.03_{12.07}$	$74.84_{2.75}$	$15.22_{1.28}$	$15.74_{1.50}$	$40.79_{27.35}$	$72.55_{19.63}$
COG+LS	$26.96_{36.96}$	$31.07_{34.26}$	$64.62_{19.09}$	$82.10_{10.28}$	$\mathbf{13.53_{4.93}}$	$\mathbf{17.21_{8.08}}$	$62.42_{17.53}$	$74.22_{1.98}$	$25.15_{4.64}$	$25.52_{5.26}$	$42.97_{10.78}$	$90.75_{7.47}$
COG-TS	$33.36_{28.15}$	$35.96_{26.44}$	$80.52_{2.18}$	$80.67_{1.99}$	$22.90_{4.49}$	$23.01_{2.66}$	$50.63_{17.00}$	$\mathbf{74.07_{4.23}}$	$19.63_{3.45}$	$20.76_{3.28}$	$42.74_{22.05}$	$83.17_{1.57}$
COG-MS	$20.63_{8.74}$	$26.11_{4.82}$	$43.57_{16.80}$	$\mathbf{73.42_{6.72}}$	$23.96_{3.41}$	$24.99_{3.38}$	$52.36_{16.96}$	$74.15_{4.45}$	$21.13_{4.24}$	$22.67_{5.13}$	$43.17_{22.18}$	$83.00_{1.05}$

Appendix

Table 15: Comparison of MCE (

\downarrow

lower is better) for all models and all variants seen in this work for multiple choices of the number of bins. We report

\text{mean}_{\text{std}}

. The best value in each column is shown in bold.

	SVHN			CIFAR-10			CIFAR-100
Method	AUROC $(\uparrow)$	FPR95 $(\downarrow)$	Rank $(\downarrow)$	AUROC $(\uparrow)$	FPR95 $(\downarrow)$	Rank $(\downarrow)$	AUROC $(\uparrow)$	FPR95 $(\downarrow)$	Rank $(\downarrow)$
Baseline	$94.78_{0.24}$	$34.43_{1.27}$	8.00	$83.97_{2.70}$	$76.39_{3.20}$	7.00	$71.92_{1.33}$	$87.67_{1.34}$	7.00
COG-FC	$96.60_{0.41}$	$21.60_{2.59}$	2.00	$87.00_{1.32}$	$73.98_{4.03}$	3.00	$68.50_{2.22}$	$85.83_{1.43}$	8.00
COG+LS	$93.80_{0.61}$	$22.58_{2.45}$	6.50	$81.93_{3.10}$	$73.43_{3.18}$	6.00	$65.48_{7.85}$	$83.71_{3.42}$	6.50
COG-TS	$96.21_{0.35}$	$24.71_{2.66}$	4.50	$86.04_{0.89}$	$77.01_{2.21}$	6.00	$67.31_{1.22}$	$88.67_{0.81}$	10.50
COG-MS	$\mathbf{96.70_{0.28}}$	$\mathbf{20.36_{2.18}}$	$\color[rgb]{0,0.75,0.16}{\mathbf{1.00}}$	$86.20_{0.56}$	$77.27_{1.74}$	6.00	$69.40_{1.55}$	$86.22_{1.06}$	8.00

Appendix

Table 16: Performance in out-of-distribution detection or all models and all variants seen in this work.

	SVHN				CIFAR-10				CIFAR-100
Method	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	Rank $(\downarrow)$	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	Rank $(\downarrow)$	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	Rank $(\downarrow)$
Baseline	$47.01_{1.06}$	$5.29_{0.23}$	$0.94_{0.02}$	8.67	$74.28_{0.59}$	$2.03_{0.11}$	$0.45_{0.01}$	6.67	$45.85_{1.36}$	$5.70_{0.21}$	$0.92_{0.02}$	7.67
COG-FC	$45.82_{1.32}$	$2.43_{0.12}$	$0.80_{0.02}$	6.00	$68.80_{0.68}$	$1.57_{0.09}$	$0.50_{0.01}$	9.00	$44.09_{0.42}$	$2.88_{0.04}$	$0.77_{0.01}$	4.00
COG+LS	$46.05_{0.57}$	$\mathbf{1.91_{0.02}}$	$0.77_{0.00}$	3.00	$69.31_{0.42}$	$1.11_{0.02}$	$0.46_{0.01}$	7.33	$44.92_{0.69}$	$2.90_{0.11}$	$0.81_{0.03}$	5.67
COG-TS	$45.59_{0.50}$	$3.35_{0.12}$	$0.88_{0.01}$	10.33	$67.98_{0.22}$	$2.36_{0.17}$	$0.55_{0.01}$	11.33	$40.99_{0.22}$	$5.23_{0.63}$	$0.93_{0.02}$	9.33
COG-MS	$46.38_{0.51}$	$2.53_{0.10}$	$0.81_{0.01}$	6.00	$67.67_{0.63}$	$1.68_{0.11}$	$0.52_{0.01}$	10.67	$43.63_{0.80}$	$4.10_{1.23}$	$0.80_{0.02}$	6.67

Appendix

Table 17: Performance under covariate shifts. Shift metrics are averaged across all considered covariate shifts.

	SVHN				CIFAR-10				CIFAR-100
Method	ECE-10	ECE-15	ECE-100	ECE-500	ECE-10	ECE-15	ECE-100	ECE-500	ECE-10	ECE-15	ECE-100	ECE-500
Baseline	$43.81_{1.01}$	$43.81_{1.01}$	$43.81_{1.01}$	$43.82_{1.01}$	$20.68_{0.56}$	$20.68_{0.56}$	$20.68_{0.56}$	$20.66_{0.56}$	$41.90_{0.89}$	$41.89_{0.89}$	$41.89_{0.89}$	$41.86_{0.89}$
COG-FC	$29.55_{2.32}$	$29.55_{2.32}$	$29.55_{2.32}$	$29.56_{2.31}$	$20.46_{0.86}$	$20.45_{0.86}$	$20.45_{0.86}$	$20.43_{0.86}$	$22.93_{0.19}$	$22.93_{0.19}$	$22.93_{0.19}$	$22.93_{0.19}$
COG+LS	$24.90_{0.61}$	$24.90_{0.61}$	$24.91_{0.61}$	$24.91_{0.61}$	$11.96_{1.14}$	$11.96_{1.14}$	$11.99_{1.10}$	$12.00_{1.09}$	$25.93_{3.92}$	$25.93_{3.92}$	$25.93_{3.92}$	$25.93_{3.92}$
COG-TS	$37.76_{0.75}$	$37.76_{0.75}$	$37.76_{0.75}$	$37.77_{0.76}$	$24.80_{0.63}$	$24.80_{0.63}$	$24.80_{0.63}$	$24.78_{0.63}$	$38.37_{2.30}$	$38.37_{2.30}$	$38.36_{2.29}$	$38.35_{2.29}$
COG-MS	$30.83_{1.39}$	$30.83_{1.39}$	$30.83_{1.39}$	$30.83_{1.39}$	$21.30_{0.92}$	$21.29_{0.92}$	$21.29_{0.92}$	$21.27_{0.92}$	$26.76_{2.95}$	$26.76_{2.95}$	$26.76_{2.95}$	$26.75_{2.95}$

Appendix

Table 18: ECE averaged across all considered covariate shifts for all models and all variants seen in this work for multiple choices of the number of bins.

	SVHN				CIFAR-10				CIFAR-100
Method	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500
Baseline	$44.25_{1.00}$	$44.43_{0.99}$	$45.42_{0.97}$	$46.12_{0.97}$	$21.71_{0.49}$	$22.21_{0.47}$	$24.29_{0.40}$	$25.44_{0.39}$	$42.51_{0.86}$	$42.80_{0.84}$	$44.18_{0.77}$	$44.98_{0.75}$
COG-FC	$30.73_{2.20}$	$31.00_{2.21}$	$31.68_{2.37}$	$31.82_{2.39}$	$21.66_{0.79}$	$22.17_{0.78}$	$23.72_{0.86}$	$24.20_{0.92}$	$24.08_{0.18}$	$24.29_{0.18}$	$24.63_{0.17}$	$24.68_{0.17}$
COG+LS	$26.50_{0.73}$	$26.71_{0.64}$	$27.18_{0.66}$	$27.31_{0.66}$	$14.31_{0.88}$	$14.26_{0.71}$	$14.75_{0.80}$	$14.81_{0.81}$	$27.11_{3.86}$	$27.88_{4.18}$	$28.26_{4.31}$	$28.29_{4.32}$
COG-TS	$38.64_{0.79}$	$38.92_{0.80}$	$40.05_{0.89}$	$40.58_{0.92}$	$25.64_{0.56}$	$26.09_{0.53}$	$27.94_{0.53}$	$28.86_{0.58}$	$39.19_{2.20}$	$39.48_{2.19}$	$40.50_{2.26}$	$40.87_{2.38}$
COG-MS	$31.92_{1.35}$	$32.18_{1.37}$	$32.90_{1.42}$	$33.04_{1.43}$	$22.41_{0.86}$	$22.90_{0.84}$	$24.41_{0.88}$	$24.89_{0.92}$	$27.80_{2.91}$	$28.04_{2.97}$	$28.52_{3.15}$	$28.60_{3.18}$

Appendix

Table 19: RMSCE averaged across all considered covariate shifts for all models and all variants seen in this work for multiple choices of the number of bins.

	SVHN				CIFAR-10				CIFAR-100
Method	MCE-10	MCE-15	MCE-100	MCE-500	MCE-10	MCE-15	MCE-100	MCE-500	MCE-10	MCE-15	MCE-100	MCE-500
Baseline	$58.64_{1.18}$	$62.43_{0.81}$	$72.23_{7.88}$	$76.60_{4.84}$	$44.30_{0.51}$	$46.83_{0.37}$	$54.91_{13.78}$	$65.07_{13.82}$	$59.43_{0.36}$	$62.38_{0.52}$	$64.12_{0.60}$	$72.86_{10.41}$
COG-FC	$45.33_{2.35}$	$46.36_{2.83}$	$47.61_{2.88}$	$58.79_{14.22}$	$37.89_{1.23}$	$38.70_{1.44}$	$47.84_{18.37}$	$71.61_{17.00}$	$33.81_{0.21}$	$34.23_{0.21}$	$40.17_{10.67}$	$49.31_{24.20}$
COG+LS	$39.56_{1.40}$	$58.27_{19.86}$	$97.99_{1.03}$	$98.13_{1.02}$	$25.85_{1.53}$	$26.18_{1.67}$	$44.33_{24.12}$	$60.95_{22.11}$	$41.05_{5.79}$	$46.22_{9.70}$	$77.67_{15.92}$	$92.00_{16.61}$
COG-TS	$54.80_{1.62}$	$57.11_{1.40}$	$59.03_{1.37}$	$74.21_{11.88}$	$52.15_{16.14}$	$54.01_{15.10}$	$61.46_{16.53}$	$77.84_{2.28}$	$53.65_{2.26}$	$55.01_{2.77}$	$55.75_{3.00}$	$57.57_{2.84}$
COG-MS	$46.55_{1.60}$	$47.67_{1.92}$	$49.19_{1.52}$	$53.37_{1.47}$	$38.17_{1.23}$	$39.19_{1.23}$	$57.12_{21.10}$	$72.99_{16.32}$	$39.09_{4.42}$	$39.43_{4.28}$	$40.26_{4.15}$	$50.85_{21.98}$

Appendix

Table 20: MCE averaged across all considered covariate shifts for all models and all variants seen in this work for multiple choices of the number of bins.

Method	ECE-10	ECE-15	ECE-100	ECE-500	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500	MCE-10	MCE-15	MCE-100	MCE-500
Baseline	$7.73_{0.66}$ [4]	$7.73_{0.66}$ [4]	$7.90_{0.66}$ [4]	$9.18_{0.77}$ [4]	$8.49_{0.79}$ [3]	$8.77_{0.86}$ [4]	$9.67_{0.89}$ [4]	$12.46_{0.87}$ [3]	$15.84_{3.92}$ [2]	$15.94_{3.67}$ [2]	$41.11_{21.89}$ [2]	$76.03_{1.74}$ [3]
COG-FC	$7.70_{0.70}$ [3]	$7.70_{0.67}$ [3]	$7.87_{0.61}$ [3]	$8.99_{0.61}$ [3]	$8.61_{0.73}$ [5]	$8.88_{0.75}$ [5]	$9.75_{0.72}$ [5]	$12.52_{0.68}$ [4]	$28.76_{28.91}$ [6]	$29.45_{28.51}$ [6]	$59.26_{23.83}$ [5]	$77.84_{1.67}$ [5]
Temp Scaling	$1.65_{0.34}$ [2]	$1.72_{0.32}$ [2]	$3.11_{0.24}$ [2]	$6.19_{0.13}$ [2]	$1.96_{0.30}$ [2]	$2.16_{0.33}$ [2]	$4.28_{0.36}$ [2]	$8.85_{0.11}$ [2]	$21.00_{5.63}$ [4]	$21.00_{5.63}$ [4]	$\mathbf{40.90_{28.62}}$ [1]	$78.78_{2.61}$ [6]
COG-TS	$8.09_{1.46}$ [6]	$8.11_{1.45}$ [6]	$8.28_{1.40}$ [6]	$9.50_{1.06}$ [6]	$8.89_{1.50}$ [6]	$9.08_{1.57}$ [6]	$10.00_{1.56}$ [6]	$12.85_{1.24}$ [6]	$\mathbf{14.22_{2.15}}$ [1]	$\mathbf{15.73_{2.69}}$ [1]	$58.58_{25.29}$ [4]	$75.97_{2.26}$ [2]
Matrix Scaling	$\mathbf{1.00_{0.13}}$ [1]	$\mathbf{1.22_{0.21}}$ [1]	$\mathbf{2.72_{0.38}}$ [1]	$\mathbf{5.87_{0.22}}$ [1]	$\mathbf{1.44_{0.19}}$ [1]	$\mathbf{1.82_{0.32}}$ [1]	$\mathbf{4.02_{0.46}}$ [1]	$\mathbf{8.68_{0.08}}$ [1]	$20.99_{10.35}$ [3]	$20.99_{10.35}$ [3]	$61.38_{28.21}$ [6]	$\mathbf{75.84_{12.01}}$ [1]
COG-MS	$7.74_{0.54}$ [5]	$7.76_{0.53}$ [5]	$7.91_{0.54}$ [5]	$9.25_{0.40}$ [5]	$8.53_{0.54}$ [4]	$8.76_{0.62}$ [3]	$9.63_{0.59}$ [3]	$12.53_{0.49}$ [5]	$27.80_{29.72}$ [5]	$28.35_{29.50}$ [5]	$56.64_{22.11}$ [3]	$77.62_{2.36}$ [4]

Appendix

Table 21: Comparison of calibration errors for ViT in CIFAR-10 for multiple choices of the number of bins.

Figure 6: Per-task metrics on CelebA. Tasks are ordered by their mean Brier. COG-FC achieves higher accuracy than Temperature Scaling and Label Smoothing on all 40 tasks. In terms of Brier, COG-FC outperforms Temperature Scaling on 32 of 40 tasks and Label Smoothing on all 40 tasks.

	CelebA
Method	ECE-10	ECE-15	ECE-100	ECE-500	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500	MCE-10	MCE-15	MCE-100	MCE-500
Baseline	$5.84_{0.08}$	$5.85_{0.08}$	$6.01_{0.08}$	$6.56_{0.08}$	$7.52_{0.16}$	$7.83_{0.14}$	$9.29_{0.11}$	$11.47_{0.10}$	$26.89_{0.74}$	$29.35_{0.69}$	$49.93_{0.46}$	$82.51_{1.17}$
COG-FC	$\mathbf{5.01}_{0.04}$	$\mathbf{5.03}_{0.03}$	$\mathbf{5.19}_{0.04}$	$\mathbf{5.77}_{0.03}$	$\mathbf{6.50}_{0.07}$	$\mathbf{6.82}_{0.06}$	$\mathbf{8.27}_{0.06}$	$\mathbf{10.56}_{0.05}$	$\mathbf{23.38}_{0.86}$	$\mathbf{25.50}_{0.54}$	$\mathbf{48.25}_{0.45}$	$\mathbf{81.91}_{0.65}$
Label Smoothing	$4.88_{0.05}$	$4.90_{0.06}$	$5.09_{0.05}$	$5.76_{0.05}$	$6.59_{0.11}$	$6.92_{0.11}$	$8.26_{0.10}$	$10.45_{0.07}$	$23.80_{0.65}$	$25.46_{0.77}$	$46.98_{1.62}$	$80.59_{0.41}$
COG+LS	$\mathbf{4.09}_{0.04}$	$\mathbf{4.11}_{0.05}$	$\mathbf{4.33}_{0.04}$	$\mathbf{5.03}_{0.04}$	$\mathbf{5.61}_{0.04}$	$\mathbf{5.94}_{0.05}$	$\mathbf{7.27}_{0.03}$	$\mathbf{9.64}_{0.05}$	$\mathbf{19.57}_{0.32}$	$\mathbf{21.82}_{0.42}$	$\mathbf{45.07}_{0.51}$	$\mathbf{79.47}_{0.56}$
Temp. Scaling	$\mathbf{3.27}_{0.09}$	$\mathbf{3.29}_{0.09}$	$\mathbf{3.53}_{0.08}$	$\mathbf{4.40}_{0.06}$	$\mathbf{4.78}_{0.17}$	$\mathbf{4.98}_{0.16}$	$\mathbf{6.06}_{0.13}$	$\mathbf{8.63}_{0.10}$	$\mathbf{17.11}_{0.81}$	$\mathbf{19.02}_{0.93}$	$\mathbf{39.36}_{0.92}$	$\mathbf{76.39}_{1.60}$

Appendix

Table 22: Calibration errors for ResNet101 on CelebA, averaged across tasks. Bold indicates the best value within each group.

Figure 7: Per-task ECE-100 for the best-performing models evaluated in the main text.

	SVHN				CIFAR-10				CIFAR-100
Method	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	Rank $(\downarrow)$	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	Rank $(\downarrow)$	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	Rank $(\downarrow)$
Baseline	$95.35_{0.20}$	$0.35_{0.01}$	$0.08_{0.00}$	9.67	$91.33_{0.60}$	$0.53_{0.02}$	$0.15_{0.01}$	7.67	$69.49_{1.15}$	$2.33_{0.11}$	$0.52_{0.02}$	7.67
COG-FC	$95.60_{0.15}$	$0.17_{0.00}$	$\mathbf{0.07_{0.00}}$	3.00	$88.88_{0.34}$	$0.36_{0.01}$	$0.17_{0.00}$	8.00	$67.89_{0.55}$	$\mathbf{1.19_{0.04}}$	$0.44_{0.01}$	3.00
COG+LS	$\mathbf{95.92_{0.19}}$	$0.22_{0.00}$	$\mathbf{0.07_{0.00}}$	3.67	$89.41_{0.30}$	$0.37_{0.01}$	$0.16_{0.00}$	7.67	$71.98_{1.04}$	$1.32_{0.04}$	$0.40_{0.01}$	2.33
COG-TS	$95.64_{0.15}$	$\mathbf{0.16_{0.00}}$	$\mathbf{0.07_{0.00}}$	2.00	$88.55_{0.11}$	$0.38_{0.01}$	$0.17_{0.00}$	9.00	$65.14_{0.57}$	$1.34_{0.06}$	$0.48_{0.01}$	5.67
COG-MS	$95.76_{0.19}$	$\mathbf{0.16_{0.01}}$	$\mathbf{0.07_{0.00}}$	$\color[rgb]{0,0.75,0.16}{\mathbf{1.33}}$	$87.74_{1.17}$	$0.40_{0.02}$	$0.18_{0.01}$	11.00	$66.31_{0.86}$	$1.35_{0.01}$	$0.47_{0.01}$	5.33

Appendix

Table 23: Comparison of all models and all variants seen in this work. We report

\text{mean}_{\text{std}}

. The best value in each column is shown in bold. Rank denotes the mean rank across ACC, NLL, and Brier for a given dataset (1 = best; ties share a rank).

	SVHN				CIFAR-10				CIFAR-100
Method	ECE-10	ECE-15	ECE-100	ECE-500	ECE-10	ECE-15	ECE-100	ECE-500	ECE-10	ECE-15	ECE-100	ECE-500
Baseline	$3.63_{0.15}$	$3.63_{0.15}$	$3.67_{0.16}$	$3.83_{0.15}$	$6.70_{0.41}$	$6.70_{0.40}$	$6.80_{0.37}$	$7.21_{0.34}$	$22.90_{0.88}$	$22.90_{0.88}$	$22.91_{0.88}$	$23.12_{0.89}$
COG-FC	$1.07_{0.10}$	$1.08_{0.09}$	$1.27_{0.08}$	$1.89_{0.08}$	$4.25_{0.33}$	$4.22_{0.34}$	$4.55_{0.28}$	$5.62_{0.26}$	$7.82_{1.49}$	$7.82_{1.49}$	$8.01_{1.46}$	$9.45_{1.19}$
COG+LS	$7.46_{0.14}$	$7.48_{0.14}$	$7.70_{0.14}$	$8.29_{0.14}$	$4.16_{0.12}$	$4.37_{0.12}$	$5.29_{0.12}$	$6.93_{0.09}$	$3.69_{0.40}$	$4.67_{0.41}$	$5.62_{0.33}$	$7.90_{0.38}$
COG-TS	$1.00_{0.07}$	$1.01_{0.08}$	$1.23_{0.11}$	$1.88_{0.08}$	$4.60_{0.17}$	$4.61_{0.18}$	$4.89_{0.13}$	$5.94_{0.14}$	$9.84_{1.33}$	$9.84_{1.33}$	$9.87_{1.33}$	$10.87_{1.17}$
COG-MS	$1.24_{0.12}$	$1.26_{0.12}$	$1.42_{0.14}$	$1.93_{0.11}$	$5.17_{0.27}$	$5.19_{0.27}$	$5.39_{0.32}$	$6.33_{0.32}$	$11.82_{1.04}$	$11.82_{1.04}$	$11.90_{1.01}$	$12.53_{0.87}$

Appendix

Table 24: Comparison of ECE (

\downarrow

lower is better) for all models and all variants seen in this work for multiple choices of the number of bins. We report

\text{mean}_{\text{std}}

. The best value in each column is shown in bold.

	SVHN				CIFAR-10				CIFAR-100
Method	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500
Baseline	$5.25_{0.27}$	$5.89_{0.27}$	$7.68_{0.18}$	$9.34_{0.12}$	$8.35_{0.61}$	$8.91_{0.55}$	$11.50_{0.42}$	$13.94_{0.43}$	$24.34_{0.91}$	$25.02_{0.98}$	$27.40_{0.90}$	$29.24_{0.84}$
COG-FC	$1.58_{0.18}$	$1.76_{0.20}$	$2.87_{0.22}$	$5.48_{0.34}$	$5.47_{0.50}$	$5.93_{0.56}$	$7.52_{0.44}$	$10.64_{0.48}$	$8.78_{1.67}$	$8.99_{1.78}$	$9.97_{1.71}$	$13.08_{1.42}$
COG+LS	$7.56_{0.14}$	$7.62_{0.14}$	$7.97_{0.17}$	$9.13_{0.18}$	$4.48_{0.08}$	$4.76_{0.11}$	$6.09_{0.21}$	$9.40_{0.25}$	$5.01_{0.30}$	$5.73_{0.45}$	$7.18_{0.53}$	$11.35_{0.37}$
COG-TS	$1.64_{0.10}$	$1.82_{0.13}$	$2.92_{0.28}$	$5.48_{0.29}$	$5.79_{0.28}$	$6.25_{0.31}$	$7.86_{0.17}$	$11.14_{0.34}$	$10.93_{1.47}$	$11.12_{1.48}$	$11.93_{1.50}$	$14.54_{1.38}$
COG-MS	$1.99_{0.14}$	$2.27_{0.19}$	$3.29_{0.24}$	$5.69_{0.27}$	$6.48_{0.50}$	$6.91_{0.34}$	$8.46_{0.58}$	$11.42_{0.58}$	$13.12_{1.15}$	$13.31_{1.22}$	$14.21_{1.23}$	$16.43_{0.99}$

Appendix

Table 25: Comparison of RMSCE (

\downarrow

lower is better) for all models and all variants seen in this work for multiple choices of the number of bins. We report

\text{mean}_{\text{std}}

. The best value in each column is shown in bold.

	SVHN				CIFAR-10				CIFAR-100
Method	MCE-10	MCE-15	MCE-100	MCE-500	MCE-10	MCE-15	MCE-100	MCE-500	MCE-10	MCE-15	MCE-100	MCE-500
Baseline	$40.91_{18.18}$	$35.91_{3.53}$	$64.66_{5.09}$	$86.39_{5.07}$	$34.74_{1.87}$	$37.27_{4.82}$	$62.85_{4.98}$	$88.29_{1.94}$	$46.89_{1.18}$	$50.50_{2.59}$	$63.17_{12.17}$	$81.55_{5.53}$
COG-FC	$29.45_{28.99}$	$31.63_{27.43}$	$50.32_{24.93}$	$76.26_{2.50}$	$15.82_{1.90}$	$19.94_{3.79}$	$64.66_{15.10}$	$73.02_{2.25}$	$13.84_{3.09}$	$14.60_{3.64}$	$23.54_{2.02}$	$85.11_{2.45}$
COG+LS	$28.77_{31.43}$	$30.17_{30.53}$	$54.89_{25.22}$	$81.89_{9.10}$	$13.48_{6.73}$	$17.64_{4.10}$	$69.21_{20.08}$	$77.80_{1.23}$	$11.97_{2.43}$	$12.48_{2.21}$	$40.61_{32.25}$	$74.45_{20.82}$
COG-TS	$46.93_{31.63}$	$56.62_{34.70}$	$74.02_{15.11}$	$80.11_{2.95}$	$20.19_{6.53}$	$32.24_{13.47}$	$61.07_{22.01}$	$74.28_{4.21}$	$17.48_{2.92}$	$18.71_{3.04}$	$26.67_{5.20}$	$71.28_{19.95}$
COG-MS	$19.20_{8.44}$	$26.31_{13.57}$	$60.68_{26.69}$	$78.79_{4.65}$	$16.12_{0.84}$	$36.00_{22.68}$	$67.60_{17.03}$	$74.53_{2.90}$	$20.77_{2.37}$	$21.35_{2.66}$	$43.62_{24.09}$	$83.56_{3.11}$

Appendix

Table 26: Comparison of MCE (

\downarrow

lower is better) for all models and all variants seen in this work for multiple choices of the number of bins. We report

\text{mean}_{\text{std}}

. The best value in each column is shown in bold.

	SVHN			CIFAR-10			CIFAR-100
Method	AUROC $(\uparrow)$	FPR95 $(\downarrow)$	Rank $(\downarrow)$	AUROC $(\uparrow)$	FPR95 $(\downarrow)$	Rank $(\downarrow)$	AUROC $(\uparrow)$	FPR95 $(\downarrow)$	Rank $(\downarrow)$
Baseline	$93.26_{0.75}$	$42.67_{3.11}$	9.50	$87.00_{1.36}$	$71.21_{4.41}$	4.50	$70.66_{2.85}$	$86.80_{1.38}$	8.50
COG-FC	$96.11_{0.34}$	$26.30_{2.79}$	2.50	$85.11_{1.13}$	$77.61_{2.59}$	9.50	$70.20_{3.78}$	$84.34_{2.25}$	8.50
COG+LS	$93.98_{0.27}$	$25.37_{1.76}$	4.00	$82.40_{2.77}$	$74.75_{4.02}$	9.00	$73.32_{2.30}$	$81.67_{1.68}$	4.00
COG-TS	$95.43_{0.48}$	$31.51_{3.42}$	4.50	$85.53_{0.57}$	$76.66_{1.60}$	8.50	$67.50_{2.59}$	$87.76_{1.51}$	10.50
COG-MS	$95.69_{0.31}$	$29.02_{2.13}$	3.50	$84.58_{1.10}$	$78.71_{3.14}$	11.00	$68.82_{4.94}$	$88.05_{1.70}$	10.50

Appendix

Table 27: Performance in out-of-distribution detection or all models and all variants seen in this work.

	SVHN				CIFAR-10				CIFAR-100
Method	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	Rank $(\downarrow)$	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	Rank $(\downarrow)$	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	Rank $(\downarrow)$
Baseline	$44.53_{0.94}$	$5.95_{0.19}$	$1.00_{0.02}$	10.67	$74.24_{0.54}$	$1.99_{0.13}$	$0.45_{0.01}$	8.33	$46.50_{1.29}$	$5.16_{0.34}$	$0.91_{0.02}$	8.33
COG-FC	$45.34_{0.93}$	$2.34_{0.06}$	$0.81_{0.01}$	4.00	$67.42_{0.31}$	$1.28_{0.01}$	$0.50_{0.00}$	9.33	$44.03_{1.53}$	$2.71_{0.11}$	$0.76_{0.02}$	4.33
COG+LS	$\mathbf{45.48_{1.00}}$	$\mathbf{1.87_{0.03}}$	$\mathbf{0.76_{0.01}}$	$\color[rgb]{0,0.75,0.16}{\mathbf{1.00}}$	$67.67_{0.35}$	$1.10_{0.01}$	$0.47_{0.00}$	7.67	$47.04_{1.07}$	$2.59_{0.07}$	$0.71_{0.02}$	2.00
COG-TS	$44.78_{0.68}$	$3.27_{0.09}$	$0.89_{0.02}$	9.00	$66.55_{0.39}$	$1.92_{0.04}$	$0.55_{0.01}$	11.67	$45.39_{2.00}$	$3.73_{0.21}$	$0.83_{0.03}$	8.00
COG-MS	$44.79_{0.35}$	$2.53_{0.09}$	$0.83_{0.01}$	6.00	$67.22_{1.29}$	$1.47_{0.07}$	$0.51_{0.02}$	10.67	$40.61_{1.31}$	$2.87_{0.14}$	$0.79_{0.02}$	7.33

Appendix

Table 28: Performance under covariate shifts. Shift metrics are averaged across all considered covariate shifts.

	SVHN				CIFAR-10				CIFAR-100
Method	ECE-10	ECE-15	ECE-100	ECE-500	ECE-10	ECE-15	ECE-100	ECE-500	ECE-10	ECE-15	ECE-100	ECE-500
Baseline	$47.06_{0.83}$	$47.06_{0.83}$	$47.06_{0.83}$	$47.06_{0.83}$	$20.89_{0.53}$	$20.89_{0.53}$	$20.89_{0.53}$	$20.86_{0.53}$	$40.89_{1.24}$	$40.88_{1.24}$	$40.88_{1.24}$	$40.86_{1.24}$
COG-FC	$30.31_{0.98}$	$30.31_{0.98}$	$30.30_{0.98}$	$30.31_{0.98}$	$17.97_{0.28}$	$17.97_{0.28}$	$17.96_{0.28}$	$17.94_{0.28}$	$20.91_{1.30}$	$20.91_{1.30}$	$20.90_{1.30}$	$20.90_{1.30}$
COG+LS	$\mathbf{22.99_{0.56}}$	$\mathbf{22.99_{0.56}}$	$\mathbf{22.99_{0.55}}$	$\mathbf{23.00_{0.55}}$	$10.02_{0.33}$	$10.02_{0.33}$	$10.02_{0.33}$	$10.02_{0.33}$	$12.78_{2.20}$	$12.79_{2.17}$	$12.81_{2.18}$	$12.82_{2.17}$
COG-TS	$38.11_{0.94}$	$38.11_{0.94}$	$38.11_{0.94}$	$38.12_{0.94}$	$23.93_{0.32}$	$23.93_{0.32}$	$23.93_{0.32}$	$23.91_{0.32}$	$32.18_{1.10}$	$32.17_{1.10}$	$32.16_{1.10}$	$32.16_{1.10}$
COG-MS	$32.44_{1.20}$	$32.44_{1.20}$	$32.44_{1.20}$	$32.45_{1.19}$	$19.02_{0.87}$	$19.02_{0.87}$	$19.01_{0.87}$	$19.00_{0.87}$	$20.49_{1.95}$	$20.49_{1.95}$	$20.49_{1.95}$	$20.49_{1.95}$

Appendix

Table 29: ECE averaged across all considered covariate shifts for all models and all variants seen in this work for multiple choices of the number of bins.

	SVHN				CIFAR-10				CIFAR-100
Method	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500
Baseline	$47.45_{0.83}$	$47.59_{0.83}$	$48.37_{0.83}$	$49.01_{0.86}$	$21.87_{0.62}$	$22.35_{0.65}$	$24.44_{0.81}$	$25.63_{0.90}$	$41.56_{1.20}$	$41.87_{1.18}$	$43.27_{1.11}$	$44.06_{1.09}$
COG-FC	$31.40_{1.02}$	$31.66_{1.01}$	$32.16_{1.07}$	$32.25_{1.06}$	$19.19_{0.36}$	$19.65_{0.37}$	$20.75_{0.43}$	$20.93_{0.43}$	$22.07_{1.34}$	$22.26_{1.36}$	$22.59_{1.45}$	$22.63_{1.46}$
COG+LS	$\mathbf{24.11_{0.56}}$	$\mathbf{24.76_{0.65}}$	$\mathbf{25.07_{0.63}}$	$\mathbf{25.20_{0.64}}$	$11.89_{0.27}$	$11.79_{0.26}$	$12.09_{0.27}$	$12.15_{0.27}$	$14.94_{2.26}$	$15.37_{2.34}$	$15.54_{2.35}$	$15.57_{2.35}$
COG-TS	$38.94_{0.92}$	$39.21_{0.94}$	$40.27_{0.98}$	$40.71_{1.02}$	$24.84_{0.35}$	$25.29_{0.37}$	$27.01_{0.44}$	$27.74_{0.44}$	$33.18_{1.06}$	$33.52_{1.05}$	$34.54_{1.02}$	$34.81_{1.03}$
COG-MS	$33.51_{1.18}$	$33.79_{1.18}$	$34.39_{1.22}$	$34.49_{1.22}$	$20.18_{0.83}$	$20.63_{0.83}$	$21.74_{0.92}$	$22.03_{0.97}$	$21.67_{2.00}$	$21.85_{2.01}$	$22.16_{2.05}$	$22.19_{2.06}$

Appendix

Table 30: RMSCE averaged across all considered covariate shifts for all models and all variants seen in this work for multiple choices of the number of bins.

	SVHN				CIFAR-10				CIFAR-100
Method	MCE-10	MCE-15	MCE-100	MCE-500	MCE-10	MCE-15	MCE-100	MCE-500	MCE-10	MCE-15	MCE-100	MCE-500
Baseline	$60.58_{0.79}$	$64.16_{0.72}$	$68.20_{1.46}$	$74.20_{0.61}$	$44.61_{2.00}$	$47.01_{1.82}$	$60.00_{15.71}$	$76.09_{2.62}$	$59.26_{0.73}$	$61.89_{0.86}$	$68.29_{10.60}$	$69.41_{9.97}$
COG-FC	$45.30_{1.81}$	$45.53_{1.47}$	$47.61_{1.88}$	$\mathbf{51.98_{1.85}}$	$31.98_{0.97}$	$32.05_{0.80}$	$32.98_{0.97}$	$82.21_{1.30}$	$31.84_{2.22}$	$32.16_{2.21}$	$33.02_{2.18}$	$46.77_{26.24}$
COG+LS	$\mathbf{36.84_{1.36}}$	$41.11_{2.38}$	$81.65_{18.07}$	$95.96_{6.08}$	$19.74_{0.50}$	$19.88_{0.61}$	$23.76_{6.11}$	$59.58_{32.80}$	$27.12_{3.57}$	$27.44_{3.97}$	$49.32_{29.19}$	$67.27_{30.02}$
COG-TS	$54.03_{1.25}$	$56.26_{1.50}$	$58.00_{1.73}$	$69.84_{12.18}$	$49.59_{17.78}$	$50.80_{17.10}$	$66.09_{19.91}$	$79.75_{1.81}$	$47.66_{1.03}$	$47.98_{1.28}$	$48.84_{1.04}$	$50.20_{1.24}$
COG-MS	$48.16_{1.48}$	$48.68_{1.56}$	$57.82_{15.40}$	$72.49_{15.32}$	$33.43_{1.39}$	$33.61_{1.66}$	$34.62_{1.69}$	$54.18_{23.89}$	$31.79_{2.98}$	$43.52_{28.00}$	$34.01_{5.52}$	$48.23_{25.65}$

Appendix

Table 31: MCE averaged across all considered covariate shifts for all models and all variants seen in this work for multiple choices of the number of bins.

Model Calibration

Model calibration aligns predicted confidence or model outputs with observed outcomes and measurements to support more reliable decisions.

See all 214 papers

Too Sharp, Too Sure: When Calibration Follows Curvature
Apr 22, 2026Alessandro Morosini, Matea Gjika, Tomaso Poggio +1Loss Landscape Sharpness Margin-Based Learning
Massachusetts Institute of Technology, Cambridge, MA, USA.
Neighborhood Smoothing for Calibration
Oct 6, 2026Idan Horowitz, Avigdor GalModel Calibration
Faculty of Data and Decision Sciences Technion Haifa, 3200003, IL
Do Not Train Away Uncertainty: Early Uncertainty Anchored Calibration
Oct 8, 2026Yutong Xie, Jiawei Tang, Zhenglin Hua +6Model Calibration Uncertainty Calibration
School of Software Engineering, Southeast University, Nanjing, China · School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Computing Information Sciences, Saint Francis University, Hong Kong, China +2

Apr 22, 2026J/K move · Enter open · S save

Too Sharp, Too Sure: When Calibration Follows Curvature

Alessandro Morosini, Matea Gjika, Tomaso Poggio, Pierfrancesco Beneventano

Massachusetts Institute of Technology, Cambridge, MA, USA.

Loss Landscape Sharpness Margin-Based Learning Model Calibration Neural Network Training Dynamics Statistical Learning Theory

Open paper AskPDF arXiv

Modern neural networks can achieve high accuracy while remaining poorly calibrated, producing confidence estimates that do not match empirical correctness. Yet calibration is often treated as a post-hoc attribute. We take a different perspective: we study calibration as a training-time phenomenon on small vision tasks, and ask whether calibrated solutions can be obtained reliably by intervening on the training procedure. We identify a tight coupling between calibration, curvature, and margins during training of deep networks under multiple gradient-based methods. Empirically, Expected Calibration Error (ECE) closely tracks curvature-based sharpness throughout optimization. Mathematically, we show that both ECE and Gauss--Newton curvature are controlled, up to problem-specific constants, by the same margin-dependent exponential tail functional along the trajectory. Guided by this mechanism, we introduce a margin-aware training objective that explicitly targets robust-margin tails and local smoothness, yielding improved out-of-sample calibration across optimizers without sacrificing accuracy.

Abstract

Model Calibration

Explore similar work

Too Sharp, Too Sure: When Calibration Follows Curvature

Neighborhood Smoothing for Calibration

Do Not Train Away Uncertainty: Early Uncertainty Anchored Calibration