Calibration in One Go: Jointly Learning Accurate and Calibrated Neural Networks

Nektarios Kalampalikis; Pablo M. Olmos; Isabel Valera

cs.LGOct 9, 2026

Calibration in One Go: Jointly Learning Accurate and Calibrated Neural Networks

Authors: Nektarios Kalampalikis, Pablo M. Olmos, Isabel Valera

Organizations: Dept. of Computer Science Saarland University, Germany

Abstract

Neural networks (NNs), while highly flexible and versatile, are often poorly calibrated. Post-hoc calibration methods adjust the outputs of an already trained model using a separate calibration dataset and additional calibration parameters. This process adds overhead that requires separate optimization of the calibrator and subsequent model selection. In this paper, we show that standard (post-hoc) calibrators can be trained end-to-end with the backbone via a bilevel Stackelberg formulation. This formulation eliminates the need for a separate calibration set or a dedicated calibration loss by jointly optimizing accuracy and calibration using cross-entropy, introducing only one additional hyperparameter: the calibrator learning rate. Although our framework supports calibrators ranging from temperature and matrix scaling to NN layers, our results show that using the final classification layer itself as the calibrator yields accurate, well-calibrated models with no additional parameters. Our approach outperforms SOTA calibrators with minimal added complexity, while also avoiding the multiplicative overhead of separate calibration components in settings such as multi-task learning.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Reliability diagrams on CIFAR-100 for the baseline, COG, and temperature scaling, with COG achieving the best calibration. (b) Post-hoc calibration: the model is trained and frozen before fitting a calibrator. (c) Our framework, COG, integrates the calibrator into the training procedure and constrains its learning rate to be slower than that of the model.

		# Extra	End	Calibration	Extra
	Method	Trainable Params.	To End	Data	Burden
Single-Task	Baseline	0	✓	✗	-
	COG-FC	0	✓	✗	1 Learning Rate
	Label Smoothing	0	✓	✗	1 Smoothing Level
	COG+LS	0	✓	✗	(a) 1 Smoothing Level (b) 1 Learning Rate
	Temp. Scaling	1	✗	✓	Calibrator Training

Table 1: Complexity of different calibration methods in Single-Task settings, where

K

denotes the number of classes. Tab. 7 extends this comparison to MTL settings.

Figure 2: Pairwise comparison of different calibrators and their corresponding COG variants. The warning symbol indicates failure to converge.

	SVHN			CIFAR-10			CIFAR-100
Method	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$
Baseline	$95.77_{0.18}$	$0.32_{0.01}$	$0.07_{0.00}$	$\mathbf{90.50_{0.21}}$	$0.64_{0.02}$	$0.17_{0.00}$	$65.04_{1.33}$	$2.98_{0.06}$	$0.60_{0.02}$
COG-FC	$95.84_{0.20}$	$\mathbf{0.16_{0.00}}$	$\mathbf{0.06_{0.00}}$	$89.94_{0.25}$	$0.38_{0.01}$	$0.16_{0.00}$	$70.46_{0.30}$	$\mathbf{1.08_{0.01}}$	$\mathbf{0.41_{0.00}}$
COG+LS	$\mathbf{96.25_{0.15}}$	$0.21_{0.00}$	$0.07_{0.00}$	$89.92_{0.21}$	$0.38_{0.00}$	$0.16_{0.00}$	$\mathbf{72.67_{0.78}}$	$1.31_{0.04}$	$\mathbf{0.41_{0.01}}$
Label Smoothing	$95.97_{0.12}$	$0.23_{0.00}$	$0.07_{0.00}$	$90.05_{0.31}$	$0.43_{0.01}$	$0.16_{0.00}$	$65.76_{0.88}$	$1.58_{0.04}$	$0.48_{0.01}$
Temp. Scaling	$95.77_{0.18}$	$0.19_{0.01}$	$0.07_{0.00}$	$\mathbf{90.50_{0.21}}$	$0.38_{0.01}$	$\mathbf{0.15_{0.00}}$	$65.04_{1.33}$	$1.78_{0.03}$	$0.53_{0.02}$

Table 2: Comparison of calibration metrics across datasets, and methods. The best value in each column among the subset of models is bolded. The Top

\text{Ranked}^{*}

row reports the best-performing model per dataset based on average rank across metrics as determined from the full results in Tab. 12 : COG-MS on SVHN, Matrix Scaling on CIFAR-10, and COG-FC on CIFAR-100.

Figure 3: Reliability diagrams for COG-FC and the top-ranked methods using 15 bins. For CIFAR-100, where COG-FC is the top-ranked method, we compare it with COG+LS.

	SVHN		CIFAR-10		CIFAR-100
Method	AUROC $(\uparrow)$	FPR95 $(\downarrow)$	AUROC $(\uparrow)$	FPR95 $(\downarrow)$	AUROC $(\uparrow)$	FPR95 $(\downarrow)$
Baseline	$94.78_{0.24}$	$34.43_{1.27}$	$83.97_{2.70}$	$76.39_{3.20}$	$71.92_{1.33}$	$87.67_{1.34}$
COG-FC	$\mathbf{96.60_{0.41}}$	$\mathbf{21.60_{2.59}}$	$\mathbf{87.00_{1.32}}$	$73.98_{4.03}$	$68.50_{2.22}$	$85.83_{1.43}$
COG+LS	$93.80_{0.61}$	$22.58_{2.45}$	$81.93_{3.10}$	$\mathbf{73.43_{3.18}}$	$65.48_{7.85}$	$\mathbf{83.71_{3.42}}$
Label Smoothing	$80.58_{1.66}$	$42.84_{2.40}$	$72.17_{6.88}$	$78.88_{3.09}$	$\mathbf{74.35_{2.98}}$	$84.06_{1.86}$
Temp. Scaling	$95.33_{0.25}$	$31.14_{1.85}$	$85.09_{2.43}$	$75.04_{4.22}$	$73.23_{1.53}$	$85.46_{2.00}$

Table 3: Performance on out-of-distribution detection. The best value in each column among the subset of models is bolded. The Top

\text{Ranked}^{*}

row represents the best model per dataset based on average rank across metrics as determined from the full results in Tab. 16 : COG-MS on SVHN, and COG-TST on CIFAR-10 and CIFAR-100.

	SVHN			CIFAR-10			CIFAR-100
Method	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$
Baseline	$\mathbf{47.01_{1.06}}$	$5.29_{0.23}$	$0.94_{0.02}$	$\mathbf{74.28_{0.59}}$	$2.03_{0.11}$	$0.45_{0.01}$	$45.85_{1.36}$	$5.70_{0.21}$	$0.92_{0.02}$
COG-FC	$45.82_{1.32}$	$2.43_{0.12}$	$0.80_{0.02}$	$68.80_{0.68}$	$1.57_{0.09}$	$0.50_{0.01}$	$44.09_{0.42}$	$2.88_{0.04}$	$0.77_{0.01}$
COG+LS	$46.05_{0.57}$	$\mathbf{1.91_{0.02}}$	$\mathbf{0.77_{0.00}}$	$69.31_{0.42}$	$1.11_{0.02}$	$0.46_{0.01}$	$44.92_{0.69}$	$2.90_{0.11}$	$0.81_{0.03}$
Label Smoothing	$46.19_{0.59}$	$2.19_{0.04}$	$0.86_{0.01}$	$74.22_{0.42}$	$1.03_{0.02}$	$0.42_{0.01}$	$\mathbf{47.14_{0.91}}$	$\mathbf{2.54_{0.10}}$	$\mathbf{0.71_{0.03}}$
Temp. Scaling	$\mathbf{47.01_{1.06}}$	$3.05_{0.12}$	$0.85_{0.02}$	$\mathbf{74.28_{0.59}}$	$1.19_{0.05}$	$0.41_{0.01}$	$45.85_{1.36}$	$3.32_{0.10}$	$0.81_{0.02}$

Table 4: Performance under covariate shifts. The best value in each column among the subset of models is bolded. The Top

\text{Ranked}^{*}

row reports the best-performing model per dataset based on average rank across metrics as determined from the full results in Tab. 17 : Matrix Scaling on SVHN and CIFAR-10, Label Smoothing on CIFAR-100.

Method	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$
Baseline	$73.51_{0.57}$	$0.81_{0.02}$	$0.38_{0.01}$
COG-FC	$\mathbf{74.97}_{0.47}$	$\mathbf{0.76}_{0.02}$	$\mathbf{0.36}_{0.01}$
Temp. Scaling	$73.51_{0.57}$	$\mathbf{0.76_{0.01}}$	$0.37_{0.01}$
COG-TS	$73.73_{0.43}$	$0.81_{0.02}$	$0.38_{0.00}$
Matrix Scaling	$74.04_{0.35}$	$0.74_{0.01}$	$\mathbf{0.36}_{0.00}$
COG-MS	$73.14_{0.71}$	$0.81_{0.02}$	$0.38_{0.01}$

Table 5: Performance with a ViT backbone on CIFAR-10. Bold indicates the best value per metric

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

		# Extra	End	Calibration	Extra
	Method	Trainable Params.	To End	Data	Burden
Single-Task	Baseline	0	✓	✗	-
	COG-FC	0	✓	✗	1 Learning Rate
	Label Smoothing	0	✓	✗	1 Smoothing Level
	COG+LS	0	✓	✗	(a) 1 Smoothing Level (b) 1 Learning Rate
	Temp. Scaling	1	✗	✓	Calibrator Training

Appendix

Table 7: Complexity of different calibration methods in Single-Task and Multi-Task learning settings, where

K

denotes the number of classes.

		CIFAR-10		CIFAR-100		SVHN		CelebA
	Method	$\lambda_{f}$	$\lambda_{c}$	$\lambda_{f}$	$\lambda_{c}$	$\lambda_{f}$	$\lambda_{c}$	$\lambda_{f}$	$\lambda_{c}$
ResNet	COG-FC	1e-3	5e-5	1e-3	1e-5	1e-3	1e-5	1e-3	1e-5
	COG-TS	1e-3	1e-4	1e-3	1e-5	1e-3	1e-4	1e-2	5e-5
	COG-MS	1e-3	5e-5	5e-4	1e-5	5e-4	1e-5	-	-
	COG-TST	1e-3	5e-5	5e-4	1e-5	5e-4	1e-4	-	-
WRN	COG-FC	1e-3	1e-5	1e-2	1e-5	1e-3	5e-5	-	-

Appendix

Table 8: Hyperparameter configurations selected based on validation performance for the different COG variants.

\lambda_{f}

and

\lambda_{c}

denote the learning rates used for model training and calibration, respectively. A dash indicates that the corresponding component was not used for that method.

Figure 5: Reliability Diagrams for all COG variants and their corresponding calibration methods using 15 bins.

	SVHN				CIFAR-10				CIFAR-100
Method	ECE-10	ECE-15	ECE-100	ECE-500	ECE-10	ECE-15	ECE-100	ECE-500	ECE-10	ECE-15	ECE-100	ECE-500
Baseline	$3.26_{0.11}$ [8]	$3.25_{0.11}$ [8]	$3.29_{0.10}$ [8]	$3.45_{0.10}$ [8]	$7.38_{0.25}$ [8]	$7.39_{0.23}$ [8]	$7.47_{0.23}$ [8]	$7.89_{0.22}$ [8]	$26.56_{0.79}$ [7]	$26.56_{0.78}$ [7]	$26.60_{0.79}$ [7]	$26.76_{0.83}$ [7]
COG-FC	$1.14_{0.30}$ [2]	$1.16_{0.29}$ [2]	$1.32_{0.17}$ [2]	$1.86_{0.07}$ [2]	$5.32_{0.16}$ [4]	$5.32_{0.14}$ [4]	$5.49_{0.11}$ [4]	$6.22_{0.12}$ [5]	$8.00_{0.35}$ [2]	$8.00_{0.36}$ [2]	$8.13_{0.37}$ [2]	$9.20_{0.28}$ [2]
Temp Scaling	$2.19_{0.09}$ [7]	$2.20_{0.09}$ [7]	$2.26_{0.08}$ [7]	$2.57_{0.05}$ [7]	$5.40_{0.29}$ [5]	$5.41_{0.28}$ [5]	$5.57_{0.26}$ [5]	$6.21_{0.14}$ [4]	$18.88_{0.41}$ [6]	$18.88_{0.41}$ [6]	$18.91_{0.42}$ [6]	$19.26_{0.43}$ [6]
COG-TS	$1.23_{0.14}$ [4]	$1.24_{0.14}$ [4]	$1.41_{0.11}$ [4]	$1.95_{0.11}$ [4]	$5.52_{0.40}$ [6]	$5.52_{0.40}$ [6]	$5.66_{0.40}$ [6]	$6.40_{0.40}$ [6]	$11.14_{2.05}$ [3]	$11.14_{2.05}$ [3]	$11.21_{2.03}$ [3]	$11.96_{1.77}$ [3]
Matrix Scaling	$\mathbf{0.71_{0.16}}$ [1]	$\mathbf{0.73_{0.15}}$ [1]	$\mathbf{1.05_{0.07}}$ [1]	$\mathbf{1.84_{0.09}}$ [1]	$\mathbf{0.73_{0.12}}$ [1]	$\mathbf{0.89_{0.10}}$ [1]	$\mathbf{1.76_{0.07}}$ [1]	$\mathbf{3.74_{0.22}}$ [1]	$50.18_{1.79}$ [8]	$50.18_{1.79}$ [8]	$50.19_{1.79}$ [8]	$50.25_{1.79}$ [8]

Appendix

Table 9: Comparison of ECE across different COG variants and their corresponding post-hoc counterparts for varying numbers of bins. We report

\text{mean}_{\text{std}}\text{ [Rank]}

	SVHN				CIFAR-10				CIFAR-100
Method	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500
Baseline	$5.03_{0.44}$ [8]	$5.57_{0.28}$ [8]	$7.22_{0.22}$ [8]	$8.85_{0.22}$ [8]	$8.98_{0.43}$ [8]	$9.65_{0.41}$ [8]	$11.89_{0.54}$ [8]	$14.58_{0.46}$ [8]	$27.69_{0.86}$ [7]	$28.27_{0.84}$ [7]	$30.69_{0.83}$ [7]	$32.47_{0.84}$ [7]
COG-FC	$1.96_{0.66}$ [2]	$2.16_{0.67}$ [2]	$3.17_{0.33}$ [2]	$5.59_{0.17}$ [2]	$6.98_{0.36}$ [4]	$7.27_{0.29}$ [4]	$9.08_{0.36}$ [4]	$11.87_{0.34}$ [4]	$9.18_{0.46}$ [2]	$9.39_{0.47}$ [2]	$10.27_{0.51}$ [2]	$12.98_{0.46}$ [2]
Temp Scaling	$3.62_{0.27}$ [7]	$3.89_{0.30}$ [7]	$4.93_{0.30}$ [7]	$6.72_{0.29}$ [7]	$7.03_{0.40}$ [6]	$7.58_{0.41}$ [6]	$9.46_{0.46}$ [7]	$12.18_{0.33}$ [6]	$20.57_{0.39}$ [6]	$21.07_{0.44}$ [6]	$22.51_{0.55}$ [6]	$24.14_{0.50}$ [6]
COG-TS	$2.03_{0.35}$ [3]	$2.23_{0.39}$ [4]	$3.33_{0.25}$ [5]	$5.83_{0.35}$ [6]	$6.99_{0.51}$ [5]	$7.49_{0.56}$ [5]	$9.20_{0.65}$ [5]	$11.97_{0.78}$ [5]	$12.34_{2.19}$ [3]	$12.56_{2.28}$ [3]	$13.51_{2.42}$ [3]	$15.85_{2.03}$ [3]
Matrix Scaling	$\mathbf{1.26_{0.25}}$ [1]	$\mathbf{1.40_{0.23}}$ [1]	$\mathbf{2.63_{0.16}}$ [1]	$\mathbf{5.34_{0.14}}$ [1]	$\mathbf{1.46_{0.25}}$ [1]	$\mathbf{1.90_{0.34}}$ [1]	$\mathbf{4.15_{0.28}}$ [1]	$\mathbf{8.52_{0.37}}$ [1]	$50.30_{1.79}$ [8]	$50.38_{1.79}$ [8]	$50.91_{1.73}$ [8]	$51.63_{1.64}$ [8]

Appendix

Table 10: Comparison of RMSCE across different COG variants and their corresponding post-hoc counterparts for varying numbers of bins. We report

\text{mean}_{\text{std}}\text{ [Rank]}

	SVHN				CIFAR-10				CIFAR-100
Method	MCE-10	MCE-15	MCE-100	MCE-500	MCE-10	MCE-15	MCE-100	MCE-500	MCE-10	MCE-15	MCE-100	MCE-500
Baseline	$41.31_{18.97}$ [8]	$44.57_{16.42}$ [8]	$67.18_{6.90}$ [6]	$86.85_{2.54}$ [8]	$36.20_{3.99}$ [8]	$38.15_{2.71}$ [7]	$62.04_{7.32}$ [4]	$89.10_{1.47}$ [8]	$48.99_{2.05}$ [7]	$52.19_{1.21}$ [7]	$68.53_{10.04}$ [7]	$83.80_{4.79}$ [7]
COG-FC	$17.27_{6.00}$ [3]	$19.03_{6.81}$ [3]	$48.96_{19.74}$ [2]	$74.04_{1.04}$ [2]	$24.65_{3.82}$ [7]	$28.05_{9.34}$ [5]	$64.03_{12.07}$ [6]	$74.84_{2.75}$ [3]	$15.22_{1.28}$ [2]	$15.74_{1.50}$ [2]	$40.79_{27.35}$ [3]	$72.55_{19.63}$ [2]
Temp Scaling	$20.24_{5.18}$ [4]	$33.59_{25.20}$ [6]	$66.79_{16.02}$ [5]	$78.23_{3.06}$ [5]	$24.28_{3.67}$ [6]	$25.01_{1.70}$ [4]	$59.11_{13.25}$ [3]	$78.52_{4.12}$ [7]	$36.93_{1.15}$ [6]	$37.08_{1.10}$ [6]	$53.48_{17.14}$ [6]	$82.83_{1.14}$ [4]
COG-TS	$33.36_{28.15}$ [7]	$35.96_{26.44}$ [7]	$80.52_{2.18}$ [8]	$80.67_{1.99}$ [7]	$22.90_{4.49}$ [2]	$\mathbf{23.01_{2.66}}$ [1]	$\mathbf{50.63_{17.00}}$ [1]	$\mathbf{74.07_{4.23}}$ [1]	$19.63_{3.45}$ [3]	$20.76_{3.28}$ [3]	$42.74_{22.05}$ [4]	$83.17_{1.57}$ [6]
Matrix Scaling	$30.13_{29.41}$ [6]	$30.16_{29.38}$ [5]	$68.95_{21.59}$ [7]	$78.93_{2.55}$ [6]	$\mathbf{13.89_{8.78}}$ [1]	$23.63_{30.26}$ [2]	$70.17_{14.95}$ [8]	$77.00_{1.27}$ [5]	$59.68_{2.69}$ [8]	$67.51_{3.68}$ [8]	$77.04_{2.03}$ [8]	$92.62_{2.22}$ [8]

Appendix

Table 11: Comparison of MCE across different COG variants and their corresponding post-hoc counterparts for varying numbers of bins. We report

\text{mean}_{\text{std}}\text{ [Rank]}

. The relative ranking of methods changes across binning choices, while the MCE values increase substantially as the number of bins increases, highlighting the sensitivity of MCE to the binning scheme.

	SVHN				CIFAR-10				CIFAR-100
Method	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	Rank $(\downarrow)$	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	Rank $(\downarrow)$	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	Rank $(\downarrow)$
Baseline	$95.77_{0.18}$	$0.32_{0.01}$	$0.07_{0.00}$	8.67	$90.50_{0.21}$	$0.64_{0.02}$	$0.17_{0.00}$	8.00	$65.04_{1.33}$	$2.98_{0.06}$	$0.60_{0.02}$	8.33
COG-FC	$95.84_{0.20}$	$\mathbf{0.16_{0.00}}$	$\mathbf{0.06_{0.00}}$	2.67	$89.94_{0.25}$	$0.38_{0.01}$	$0.16_{0.00}$	5.67	$70.46_{0.30}$	$\mathbf{1.08_{0.01}}$	$\mathbf{0.41_{0.00}}$	$\color[rgb]{0,0.75,0.16}{\mathbf{1.33}}$
COG+LS	$\mathbf{96.25_{0.15}}$	$0.21_{0.00}$	$0.07_{0.00}$	4.67	$89.92_{0.21}$	$0.38_{0.00}$	$0.16_{0.00}$	6.00	$\mathbf{72.67_{0.78}}$	$1.31_{0.04}$	$\mathbf{0.41_{0.01}}$	2.00
COG-TS	$95.86_{0.05}$	$\mathbf{0.16_{0.00}}$	$\mathbf{0.06_{0.00}}$	2.33	$89.54_{0.15}$	$0.40_{0.02}$	$0.16_{0.00}$	8.00	$67.55_{0.62}$	$1.30_{0.06}$	$0.46_{0.00}$	3.33
COG-MS	$96.02_{0.11}$	$\mathbf{0.16_{0.00}}$	$\mathbf{0.06_{0.00}}$	$\color[rgb]{0,0.75,0.16}{\mathbf{1.33}}$	$89.27_{0.26}$	$0.41_{0.01}$	$0.17_{0.00}$	10.33	$67.74_{0.56}$	$1.93_{0.95}$	$0.46_{0.01}$	4.67

Appendix

Table 12: Comparison of all models and all variants seen in this work. We report

\text{mean}_{\text{std}}

. The best value in each column is shown in bold. Rank denotes the mean rank across ACC, NLL, and Brier for a given dataset (1 = best; ties share a rank).

	SVHN				CIFAR-10				CIFAR-100
Method	ECE-10	ECE-15	ECE-100	ECE-500	ECE-10	ECE-15	ECE-100	ECE-500	ECE-10	ECE-15	ECE-100	ECE-500
Baseline	$3.26_{0.11}$	$3.25_{0.11}$	$3.29_{0.10}$	$3.45_{0.10}$	$7.38_{0.25}$	$7.39_{0.23}$	$7.47_{0.23}$	$7.89_{0.22}$	$26.56_{0.79}$	$26.56_{0.78}$	$26.60_{0.79}$	$26.76_{0.83}$
COG-FC	$1.14_{0.30}$	$1.16_{0.29}$	$1.32_{0.17}$	$1.86_{0.07}$	$5.32_{0.16}$	$5.32_{0.14}$	$5.49_{0.11}$	$6.22_{0.12}$	$8.00_{0.35}$	$8.00_{0.36}$	$8.13_{0.37}$	$9.20_{0.28}$
COG+LS	$7.44_{0.20}$	$7.51_{0.19}$	$7.82_{0.17}$	$8.32_{0.17}$	$4.90_{0.18}$	$5.10_{0.31}$	$5.65_{0.31}$	$7.17_{0.26}$	$6.81_{2.10}$	$6.94_{1.85}$	$7.95_{1.59}$	$9.33_{1.07}$
COG-TS	$1.23_{0.14}$	$1.24_{0.14}$	$1.41_{0.11}$	$1.95_{0.11}$	$5.52_{0.40}$	$5.52_{0.40}$	$5.66_{0.40}$	$6.40_{0.40}$	$11.14_{2.05}$	$11.14_{2.05}$	$11.21_{2.03}$	$11.96_{1.77}$
COG-MS	$1.37_{0.08}$	$1.38_{0.08}$	$1.51_{0.06}$	$1.97_{0.10}$	$5.67_{0.24}$	$5.68_{0.24}$	$5.84_{0.25}$	$6.61_{0.15}$	$11.46_{2.24}$	$11.46_{2.24}$	$11.55_{2.19}$	$12.43_{1.82}$

Appendix

Table 13: Comparison of ECE (

\downarrow

lower is better) for all models and all variants seen in this work for multiple choices of the number of bins. We report

\text{mean}_{\text{std}}

. The best value in each column is shown in bold.

	SVHN				CIFAR-10				CIFAR-100
Method	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500
Baseline	$5.03_{0.44}$	$5.57_{0.28}$	$7.22_{0.22}$	$8.85_{0.22}$	$8.98_{0.43}$	$9.65_{0.41}$	$11.89_{0.54}$	$14.58_{0.46}$	$27.69_{0.86}$	$28.27_{0.84}$	$30.69_{0.83}$	$32.47_{0.84}$
COG-FC	$1.96_{0.66}$	$2.16_{0.67}$	$3.17_{0.33}$	$5.59_{0.17}$	$6.98_{0.36}$	$7.27_{0.29}$	$9.08_{0.36}$	$11.87_{0.34}$	$9.18_{0.46}$	$9.39_{0.47}$	$10.27_{0.51}$	$12.98_{0.46}$
COG+LS	$7.59_{0.17}$	$7.70_{0.14}$	$8.10_{0.13}$	$9.19_{0.14}$	$5.28_{0.28}$	$5.42_{0.42}$	$6.53_{0.37}$	$9.73_{0.25}$	$9.65_{2.03}$	$10.91_{2.71}$	$11.87_{2.54}$	$14.68_{1.98}$
COG-TS	$2.03_{0.35}$	$2.23_{0.39}$	$3.33_{0.25}$	$5.83_{0.35}$	$6.99_{0.51}$	$7.49_{0.56}$	$9.20_{0.65}$	$11.97_{0.78}$	$12.34_{2.19}$	$12.56_{2.28}$	$13.51_{2.42}$	$15.85_{2.03}$
COG-MS	$2.31_{0.14}$	$2.53_{0.21}$	$3.56_{0.13}$	$5.80_{0.19}$	$7.17_{0.49}$	$7.61_{0.60}$	$9.33_{0.53}$	$12.19_{0.41}$	$12.75_{2.35}$	$13.06_{2.48}$	$14.00_{2.52}$	$16.35_{2.28}$

Appendix

Table 14: Comparison of RMSCE (

\downarrow

lower is better) for all models and all variants seen in this work for multiple choices of the number of bins. We report

\text{mean}_{\text{std}}

. The best value in each column is shown in bold.

	SVHN				CIFAR-10				CIFAR-100
Method	MCE-10	MCE-15	MCE-100	MCE-500	MCE-10	MCE-15	MCE-100	MCE-500	MCE-10	MCE-15	MCE-100	MCE-500
Baseline	$41.31_{18.97}$	$44.57_{16.42}$	$67.18_{6.90}$	$86.85_{2.54}$	$36.20_{3.99}$	$38.15_{2.71}$	$62.04_{7.32}$	$89.10_{1.47}$	$48.99_{2.05}$	$52.19_{1.21}$	$68.53_{10.04}$	$83.80_{4.79}$
COG-FC	$17.27_{6.00}$	$19.03_{6.81}$	$48.96_{19.74}$	$74.04_{1.04}$	$24.65_{3.82}$	$28.05_{9.34}$	$64.03_{12.07}$	$74.84_{2.75}$	$15.22_{1.28}$	$15.74_{1.50}$	$40.79_{27.35}$	$72.55_{19.63}$
COG+LS	$26.96_{36.96}$	$31.07_{34.26}$	$64.62_{19.09}$	$82.10_{10.28}$	$\mathbf{13.53_{4.93}}$	$\mathbf{17.21_{8.08}}$	$62.42_{17.53}$	$74.22_{1.98}$	$25.15_{4.64}$	$25.52_{5.26}$	$42.97_{10.78}$	$90.75_{7.47}$
COG-TS	$33.36_{28.15}$	$35.96_{26.44}$	$80.52_{2.18}$	$80.67_{1.99}$	$22.90_{4.49}$	$23.01_{2.66}$	$50.63_{17.00}$	$\mathbf{74.07_{4.23}}$	$19.63_{3.45}$	$20.76_{3.28}$	$42.74_{22.05}$	$83.17_{1.57}$
COG-MS	$20.63_{8.74}$	$26.11_{4.82}$	$43.57_{16.80}$	$\mathbf{73.42_{6.72}}$	$23.96_{3.41}$	$24.99_{3.38}$	$52.36_{16.96}$	$74.15_{4.45}$	$21.13_{4.24}$	$22.67_{5.13}$	$43.17_{22.18}$	$83.00_{1.05}$

Appendix

Table 15: Comparison of MCE (

\downarrow

lower is better) for all models and all variants seen in this work for multiple choices of the number of bins. We report

\text{mean}_{\text{std}}

. The best value in each column is shown in bold.

	SVHN			CIFAR-10			CIFAR-100
Method	AUROC $(\uparrow)$	FPR95 $(\downarrow)$	Rank $(\downarrow)$	AUROC $(\uparrow)$	FPR95 $(\downarrow)$	Rank $(\downarrow)$	AUROC $(\uparrow)$	FPR95 $(\downarrow)$	Rank $(\downarrow)$
Baseline	$94.78_{0.24}$	$34.43_{1.27}$	8.00	$83.97_{2.70}$	$76.39_{3.20}$	7.00	$71.92_{1.33}$	$87.67_{1.34}$	7.00
COG-FC	$96.60_{0.41}$	$21.60_{2.59}$	2.00	$87.00_{1.32}$	$73.98_{4.03}$	3.00	$68.50_{2.22}$	$85.83_{1.43}$	8.00
COG+LS	$93.80_{0.61}$	$22.58_{2.45}$	6.50	$81.93_{3.10}$	$73.43_{3.18}$	6.00	$65.48_{7.85}$	$83.71_{3.42}$	6.50
COG-TS	$96.21_{0.35}$	$24.71_{2.66}$	4.50	$86.04_{0.89}$	$77.01_{2.21}$	6.00	$67.31_{1.22}$	$88.67_{0.81}$	10.50
COG-MS	$\mathbf{96.70_{0.28}}$	$\mathbf{20.36_{2.18}}$	$\color[rgb]{0,0.75,0.16}{\mathbf{1.00}}$	$86.20_{0.56}$	$77.27_{1.74}$	6.00	$69.40_{1.55}$	$86.22_{1.06}$	8.00

Appendix

Table 16: Performance in out-of-distribution detection or all models and all variants seen in this work.

	SVHN				CIFAR-10				CIFAR-100
Method	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	Rank $(\downarrow)$	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	Rank $(\downarrow)$	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	Rank $(\downarrow)$
Baseline	$47.01_{1.06}$	$5.29_{0.23}$	$0.94_{0.02}$	8.67	$74.28_{0.59}$	$2.03_{0.11}$	$0.45_{0.01}$	6.67	$45.85_{1.36}$	$5.70_{0.21}$	$0.92_{0.02}$	7.67
COG-FC	$45.82_{1.32}$	$2.43_{0.12}$	$0.80_{0.02}$	6.00	$68.80_{0.68}$	$1.57_{0.09}$	$0.50_{0.01}$	9.00	$44.09_{0.42}$	$2.88_{0.04}$	$0.77_{0.01}$	4.00
COG+LS	$46.05_{0.57}$	$\mathbf{1.91_{0.02}}$	$0.77_{0.00}$	3.00	$69.31_{0.42}$	$1.11_{0.02}$	$0.46_{0.01}$	7.33	$44.92_{0.69}$	$2.90_{0.11}$	$0.81_{0.03}$	5.67
COG-TS	$45.59_{0.50}$	$3.35_{0.12}$	$0.88_{0.01}$	10.33	$67.98_{0.22}$	$2.36_{0.17}$	$0.55_{0.01}$	11.33	$40.99_{0.22}$	$5.23_{0.63}$	$0.93_{0.02}$	9.33
COG-MS	$46.38_{0.51}$	$2.53_{0.10}$	$0.81_{0.01}$	6.00	$67.67_{0.63}$	$1.68_{0.11}$	$0.52_{0.01}$	10.67	$43.63_{0.80}$	$4.10_{1.23}$	$0.80_{0.02}$	6.67

Appendix

Table 17: Performance under covariate shifts. Shift metrics are averaged across all considered covariate shifts.

	SVHN				CIFAR-10				CIFAR-100
Method	ECE-10	ECE-15	ECE-100	ECE-500	ECE-10	ECE-15	ECE-100	ECE-500	ECE-10	ECE-15	ECE-100	ECE-500
Baseline	$43.81_{1.01}$	$43.81_{1.01}$	$43.81_{1.01}$	$43.82_{1.01}$	$20.68_{0.56}$	$20.68_{0.56}$	$20.68_{0.56}$	$20.66_{0.56}$	$41.90_{0.89}$	$41.89_{0.89}$	$41.89_{0.89}$	$41.86_{0.89}$
COG-FC	$29.55_{2.32}$	$29.55_{2.32}$	$29.55_{2.32}$	$29.56_{2.31}$	$20.46_{0.86}$	$20.45_{0.86}$	$20.45_{0.86}$	$20.43_{0.86}$	$22.93_{0.19}$	$22.93_{0.19}$	$22.93_{0.19}$	$22.93_{0.19}$
COG+LS	$24.90_{0.61}$	$24.90_{0.61}$	$24.91_{0.61}$	$24.91_{0.61}$	$11.96_{1.14}$	$11.96_{1.14}$	$11.99_{1.10}$	$12.00_{1.09}$	$25.93_{3.92}$	$25.93_{3.92}$	$25.93_{3.92}$	$25.93_{3.92}$
COG-TS	$37.76_{0.75}$	$37.76_{0.75}$	$37.76_{0.75}$	$37.77_{0.76}$	$24.80_{0.63}$	$24.80_{0.63}$	$24.80_{0.63}$	$24.78_{0.63}$	$38.37_{2.30}$	$38.37_{2.30}$	$38.36_{2.29}$	$38.35_{2.29}$
COG-MS	$30.83_{1.39}$	$30.83_{1.39}$	$30.83_{1.39}$	$30.83_{1.39}$	$21.30_{0.92}$	$21.29_{0.92}$	$21.29_{0.92}$	$21.27_{0.92}$	$26.76_{2.95}$	$26.76_{2.95}$	$26.76_{2.95}$	$26.75_{2.95}$

Appendix

Table 18: ECE averaged across all considered covariate shifts for all models and all variants seen in this work for multiple choices of the number of bins.

	SVHN				CIFAR-10				CIFAR-100
Method	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500
Baseline	$44.25_{1.00}$	$44.43_{0.99}$	$45.42_{0.97}$	$46.12_{0.97}$	$21.71_{0.49}$	$22.21_{0.47}$	$24.29_{0.40}$	$25.44_{0.39}$	$42.51_{0.86}$	$42.80_{0.84}$	$44.18_{0.77}$	$44.98_{0.75}$
COG-FC	$30.73_{2.20}$	$31.00_{2.21}$	$31.68_{2.37}$	$31.82_{2.39}$	$21.66_{0.79}$	$22.17_{0.78}$	$23.72_{0.86}$	$24.20_{0.92}$	$24.08_{0.18}$	$24.29_{0.18}$	$24.63_{0.17}$	$24.68_{0.17}$
COG+LS	$26.50_{0.73}$	$26.71_{0.64}$	$27.18_{0.66}$	$27.31_{0.66}$	$14.31_{0.88}$	$14.26_{0.71}$	$14.75_{0.80}$	$14.81_{0.81}$	$27.11_{3.86}$	$27.88_{4.18}$	$28.26_{4.31}$	$28.29_{4.32}$
COG-TS	$38.64_{0.79}$	$38.92_{0.80}$	$40.05_{0.89}$	$40.58_{0.92}$	$25.64_{0.56}$	$26.09_{0.53}$	$27.94_{0.53}$	$28.86_{0.58}$	$39.19_{2.20}$	$39.48_{2.19}$	$40.50_{2.26}$	$40.87_{2.38}$
COG-MS	$31.92_{1.35}$	$32.18_{1.37}$	$32.90_{1.42}$	$33.04_{1.43}$	$22.41_{0.86}$	$22.90_{0.84}$	$24.41_{0.88}$	$24.89_{0.92}$	$27.80_{2.91}$	$28.04_{2.97}$	$28.52_{3.15}$	$28.60_{3.18}$

Appendix

Table 19: RMSCE averaged across all considered covariate shifts for all models and all variants seen in this work for multiple choices of the number of bins.

	SVHN				CIFAR-10				CIFAR-100
Method	MCE-10	MCE-15	MCE-100	MCE-500	MCE-10	MCE-15	MCE-100	MCE-500	MCE-10	MCE-15	MCE-100	MCE-500
Baseline	$58.64_{1.18}$	$62.43_{0.81}$	$72.23_{7.88}$	$76.60_{4.84}$	$44.30_{0.51}$	$46.83_{0.37}$	$54.91_{13.78}$	$65.07_{13.82}$	$59.43_{0.36}$	$62.38_{0.52}$	$64.12_{0.60}$	$72.86_{10.41}$
COG-FC	$45.33_{2.35}$	$46.36_{2.83}$	$47.61_{2.88}$	$58.79_{14.22}$	$37.89_{1.23}$	$38.70_{1.44}$	$47.84_{18.37}$	$71.61_{17.00}$	$33.81_{0.21}$	$34.23_{0.21}$	$40.17_{10.67}$	$49.31_{24.20}$
COG+LS	$39.56_{1.40}$	$58.27_{19.86}$	$97.99_{1.03}$	$98.13_{1.02}$	$25.85_{1.53}$	$26.18_{1.67}$	$44.33_{24.12}$	$60.95_{22.11}$	$41.05_{5.79}$	$46.22_{9.70}$	$77.67_{15.92}$	$92.00_{16.61}$
COG-TS	$54.80_{1.62}$	$57.11_{1.40}$	$59.03_{1.37}$	$74.21_{11.88}$	$52.15_{16.14}$	$54.01_{15.10}$	$61.46_{16.53}$	$77.84_{2.28}$	$53.65_{2.26}$	$55.01_{2.77}$	$55.75_{3.00}$	$57.57_{2.84}$
COG-MS	$46.55_{1.60}$	$47.67_{1.92}$	$49.19_{1.52}$	$53.37_{1.47}$	$38.17_{1.23}$	$39.19_{1.23}$	$57.12_{21.10}$	$72.99_{16.32}$	$39.09_{4.42}$	$39.43_{4.28}$	$40.26_{4.15}$	$50.85_{21.98}$

Appendix

Table 20: MCE averaged across all considered covariate shifts for all models and all variants seen in this work for multiple choices of the number of bins.

Method	ECE-10	ECE-15	ECE-100	ECE-500	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500	MCE-10	MCE-15	MCE-100	MCE-500
Baseline	$7.73_{0.66}$ [4]	$7.73_{0.66}$ [4]	$7.90_{0.66}$ [4]	$9.18_{0.77}$ [4]	$8.49_{0.79}$ [3]	$8.77_{0.86}$ [4]	$9.67_{0.89}$ [4]	$12.46_{0.87}$ [3]	$15.84_{3.92}$ [2]	$15.94_{3.67}$ [2]	$41.11_{21.89}$ [2]	$76.03_{1.74}$ [3]
COG-FC	$7.70_{0.70}$ [3]	$7.70_{0.67}$ [3]	$7.87_{0.61}$ [3]	$8.99_{0.61}$ [3]	$8.61_{0.73}$ [5]	$8.88_{0.75}$ [5]	$9.75_{0.72}$ [5]	$12.52_{0.68}$ [4]	$28.76_{28.91}$ [6]	$29.45_{28.51}$ [6]	$59.26_{23.83}$ [5]	$77.84_{1.67}$ [5]
Temp Scaling	$1.65_{0.34}$ [2]	$1.72_{0.32}$ [2]	$3.11_{0.24}$ [2]	$6.19_{0.13}$ [2]	$1.96_{0.30}$ [2]	$2.16_{0.33}$ [2]	$4.28_{0.36}$ [2]	$8.85_{0.11}$ [2]	$21.00_{5.63}$ [4]	$21.00_{5.63}$ [4]	$\mathbf{40.90_{28.62}}$ [1]	$78.78_{2.61}$ [6]
COG-TS	$8.09_{1.46}$ [6]	$8.11_{1.45}$ [6]	$8.28_{1.40}$ [6]	$9.50_{1.06}$ [6]	$8.89_{1.50}$ [6]	$9.08_{1.57}$ [6]	$10.00_{1.56}$ [6]	$12.85_{1.24}$ [6]	$\mathbf{14.22_{2.15}}$ [1]	$\mathbf{15.73_{2.69}}$ [1]	$58.58_{25.29}$ [4]	$75.97_{2.26}$ [2]
Matrix Scaling	$\mathbf{1.00_{0.13}}$ [1]	$\mathbf{1.22_{0.21}}$ [1]	$\mathbf{2.72_{0.38}}$ [1]	$\mathbf{5.87_{0.22}}$ [1]	$\mathbf{1.44_{0.19}}$ [1]	$\mathbf{1.82_{0.32}}$ [1]	$\mathbf{4.02_{0.46}}$ [1]	$\mathbf{8.68_{0.08}}$ [1]	$20.99_{10.35}$ [3]	$20.99_{10.35}$ [3]	$61.38_{28.21}$ [6]	$\mathbf{75.84_{12.01}}$ [1]
COG-MS	$7.74_{0.54}$ [5]	$7.76_{0.53}$ [5]	$7.91_{0.54}$ [5]	$9.25_{0.40}$ [5]	$8.53_{0.54}$ [4]	$8.76_{0.62}$ [3]	$9.63_{0.59}$ [3]	$12.53_{0.49}$ [5]	$27.80_{29.72}$ [5]	$28.35_{29.50}$ [5]	$56.64_{22.11}$ [3]	$77.62_{2.36}$ [4]

Appendix

Table 21: Comparison of calibration errors for ViT in CIFAR-10 for multiple choices of the number of bins.

Figure 6: Per-task metrics on CelebA. Tasks are ordered by their mean Brier. COG-FC achieves higher accuracy than Temperature Scaling and Label Smoothing on all 40 tasks. In terms of Brier, COG-FC outperforms Temperature Scaling on 32 of 40 tasks and Label Smoothing on all 40 tasks.

	CelebA
Method	ECE-10	ECE-15	ECE-100	ECE-500	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500	MCE-10	MCE-15	MCE-100	MCE-500
Baseline	$5.84_{0.08}$	$5.85_{0.08}$	$6.01_{0.08}$	$6.56_{0.08}$	$7.52_{0.16}$	$7.83_{0.14}$	$9.29_{0.11}$	$11.47_{0.10}$	$26.89_{0.74}$	$29.35_{0.69}$	$49.93_{0.46}$	$82.51_{1.17}$
COG-FC	$\mathbf{5.01}_{0.04}$	$\mathbf{5.03}_{0.03}$	$\mathbf{5.19}_{0.04}$	$\mathbf{5.77}_{0.03}$	$\mathbf{6.50}_{0.07}$	$\mathbf{6.82}_{0.06}$	$\mathbf{8.27}_{0.06}$	$\mathbf{10.56}_{0.05}$	$\mathbf{23.38}_{0.86}$	$\mathbf{25.50}_{0.54}$	$\mathbf{48.25}_{0.45}$	$\mathbf{81.91}_{0.65}$
Label Smoothing	$4.88_{0.05}$	$4.90_{0.06}$	$5.09_{0.05}$	$5.76_{0.05}$	$6.59_{0.11}$	$6.92_{0.11}$	$8.26_{0.10}$	$10.45_{0.07}$	$23.80_{0.65}$	$25.46_{0.77}$	$46.98_{1.62}$	$80.59_{0.41}$
COG+LS	$\mathbf{4.09}_{0.04}$	$\mathbf{4.11}_{0.05}$	$\mathbf{4.33}_{0.04}$	$\mathbf{5.03}_{0.04}$	$\mathbf{5.61}_{0.04}$	$\mathbf{5.94}_{0.05}$	$\mathbf{7.27}_{0.03}$	$\mathbf{9.64}_{0.05}$	$\mathbf{19.57}_{0.32}$	$\mathbf{21.82}_{0.42}$	$\mathbf{45.07}_{0.51}$	$\mathbf{79.47}_{0.56}$
Temp. Scaling	$\mathbf{3.27}_{0.09}$	$\mathbf{3.29}_{0.09}$	$\mathbf{3.53}_{0.08}$	$\mathbf{4.40}_{0.06}$	$\mathbf{4.78}_{0.17}$	$\mathbf{4.98}_{0.16}$	$\mathbf{6.06}_{0.13}$	$\mathbf{8.63}_{0.10}$	$\mathbf{17.11}_{0.81}$	$\mathbf{19.02}_{0.93}$	$\mathbf{39.36}_{0.92}$	$\mathbf{76.39}_{1.60}$

Appendix

Table 22: Calibration errors for ResNet101 on CelebA, averaged across tasks. Bold indicates the best value within each group.

Figure 7: Per-task ECE-100 for the best-performing models evaluated in the main text.

	SVHN				CIFAR-10				CIFAR-100
Method	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	Rank $(\downarrow)$	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	Rank $(\downarrow)$	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	Rank $(\downarrow)$
Baseline	$95.35_{0.20}$	$0.35_{0.01}$	$0.08_{0.00}$	9.67	$91.33_{0.60}$	$0.53_{0.02}$	$0.15_{0.01}$	7.67	$69.49_{1.15}$	$2.33_{0.11}$	$0.52_{0.02}$	7.67
COG-FC	$95.60_{0.15}$	$0.17_{0.00}$	$\mathbf{0.07_{0.00}}$	3.00	$88.88_{0.34}$	$0.36_{0.01}$	$0.17_{0.00}$	8.00	$67.89_{0.55}$	$\mathbf{1.19_{0.04}}$	$0.44_{0.01}$	3.00
COG+LS	$\mathbf{95.92_{0.19}}$	$0.22_{0.00}$	$\mathbf{0.07_{0.00}}$	3.67	$89.41_{0.30}$	$0.37_{0.01}$	$0.16_{0.00}$	7.67	$71.98_{1.04}$	$1.32_{0.04}$	$0.40_{0.01}$	2.33
COG-TS	$95.64_{0.15}$	$\mathbf{0.16_{0.00}}$	$\mathbf{0.07_{0.00}}$	2.00	$88.55_{0.11}$	$0.38_{0.01}$	$0.17_{0.00}$	9.00	$65.14_{0.57}$	$1.34_{0.06}$	$0.48_{0.01}$	5.67
COG-MS	$95.76_{0.19}$	$\mathbf{0.16_{0.01}}$	$\mathbf{0.07_{0.00}}$	$\color[rgb]{0,0.75,0.16}{\mathbf{1.33}}$	$87.74_{1.17}$	$0.40_{0.02}$	$0.18_{0.01}$	11.00	$66.31_{0.86}$	$1.35_{0.01}$	$0.47_{0.01}$	5.33

Appendix

Table 23: Comparison of all models and all variants seen in this work. We report

\text{mean}_{\text{std}}

. The best value in each column is shown in bold. Rank denotes the mean rank across ACC, NLL, and Brier for a given dataset (1 = best; ties share a rank).

	SVHN				CIFAR-10				CIFAR-100
Method	ECE-10	ECE-15	ECE-100	ECE-500	ECE-10	ECE-15	ECE-100	ECE-500	ECE-10	ECE-15	ECE-100	ECE-500
Baseline	$3.63_{0.15}$	$3.63_{0.15}$	$3.67_{0.16}$	$3.83_{0.15}$	$6.70_{0.41}$	$6.70_{0.40}$	$6.80_{0.37}$	$7.21_{0.34}$	$22.90_{0.88}$	$22.90_{0.88}$	$22.91_{0.88}$	$23.12_{0.89}$
COG-FC	$1.07_{0.10}$	$1.08_{0.09}$	$1.27_{0.08}$	$1.89_{0.08}$	$4.25_{0.33}$	$4.22_{0.34}$	$4.55_{0.28}$	$5.62_{0.26}$	$7.82_{1.49}$	$7.82_{1.49}$	$8.01_{1.46}$	$9.45_{1.19}$
COG+LS	$7.46_{0.14}$	$7.48_{0.14}$	$7.70_{0.14}$	$8.29_{0.14}$	$4.16_{0.12}$	$4.37_{0.12}$	$5.29_{0.12}$	$6.93_{0.09}$	$3.69_{0.40}$	$4.67_{0.41}$	$5.62_{0.33}$	$7.90_{0.38}$
COG-TS	$1.00_{0.07}$	$1.01_{0.08}$	$1.23_{0.11}$	$1.88_{0.08}$	$4.60_{0.17}$	$4.61_{0.18}$	$4.89_{0.13}$	$5.94_{0.14}$	$9.84_{1.33}$	$9.84_{1.33}$	$9.87_{1.33}$	$10.87_{1.17}$
COG-MS	$1.24_{0.12}$	$1.26_{0.12}$	$1.42_{0.14}$	$1.93_{0.11}$	$5.17_{0.27}$	$5.19_{0.27}$	$5.39_{0.32}$	$6.33_{0.32}$	$11.82_{1.04}$	$11.82_{1.04}$	$11.90_{1.01}$	$12.53_{0.87}$

Appendix

Table 24: Comparison of ECE (

\downarrow

lower is better) for all models and all variants seen in this work for multiple choices of the number of bins. We report

\text{mean}_{\text{std}}

. The best value in each column is shown in bold.

	SVHN				CIFAR-10				CIFAR-100
Method	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500
Baseline	$5.25_{0.27}$	$5.89_{0.27}$	$7.68_{0.18}$	$9.34_{0.12}$	$8.35_{0.61}$	$8.91_{0.55}$	$11.50_{0.42}$	$13.94_{0.43}$	$24.34_{0.91}$	$25.02_{0.98}$	$27.40_{0.90}$	$29.24_{0.84}$
COG-FC	$1.58_{0.18}$	$1.76_{0.20}$	$2.87_{0.22}$	$5.48_{0.34}$	$5.47_{0.50}$	$5.93_{0.56}$	$7.52_{0.44}$	$10.64_{0.48}$	$8.78_{1.67}$	$8.99_{1.78}$	$9.97_{1.71}$	$13.08_{1.42}$
COG+LS	$7.56_{0.14}$	$7.62_{0.14}$	$7.97_{0.17}$	$9.13_{0.18}$	$4.48_{0.08}$	$4.76_{0.11}$	$6.09_{0.21}$	$9.40_{0.25}$	$5.01_{0.30}$	$5.73_{0.45}$	$7.18_{0.53}$	$11.35_{0.37}$
COG-TS	$1.64_{0.10}$	$1.82_{0.13}$	$2.92_{0.28}$	$5.48_{0.29}$	$5.79_{0.28}$	$6.25_{0.31}$	$7.86_{0.17}$	$11.14_{0.34}$	$10.93_{1.47}$	$11.12_{1.48}$	$11.93_{1.50}$	$14.54_{1.38}$
COG-MS	$1.99_{0.14}$	$2.27_{0.19}$	$3.29_{0.24}$	$5.69_{0.27}$	$6.48_{0.50}$	$6.91_{0.34}$	$8.46_{0.58}$	$11.42_{0.58}$	$13.12_{1.15}$	$13.31_{1.22}$	$14.21_{1.23}$	$16.43_{0.99}$

Appendix

Table 25: Comparison of RMSCE (

\downarrow

lower is better) for all models and all variants seen in this work for multiple choices of the number of bins. We report

\text{mean}_{\text{std}}

. The best value in each column is shown in bold.

	SVHN				CIFAR-10				CIFAR-100
Method	MCE-10	MCE-15	MCE-100	MCE-500	MCE-10	MCE-15	MCE-100	MCE-500	MCE-10	MCE-15	MCE-100	MCE-500
Baseline	$40.91_{18.18}$	$35.91_{3.53}$	$64.66_{5.09}$	$86.39_{5.07}$	$34.74_{1.87}$	$37.27_{4.82}$	$62.85_{4.98}$	$88.29_{1.94}$	$46.89_{1.18}$	$50.50_{2.59}$	$63.17_{12.17}$	$81.55_{5.53}$
COG-FC	$29.45_{28.99}$	$31.63_{27.43}$	$50.32_{24.93}$	$76.26_{2.50}$	$15.82_{1.90}$	$19.94_{3.79}$	$64.66_{15.10}$	$73.02_{2.25}$	$13.84_{3.09}$	$14.60_{3.64}$	$23.54_{2.02}$	$85.11_{2.45}$
COG+LS	$28.77_{31.43}$	$30.17_{30.53}$	$54.89_{25.22}$	$81.89_{9.10}$	$13.48_{6.73}$	$17.64_{4.10}$	$69.21_{20.08}$	$77.80_{1.23}$	$11.97_{2.43}$	$12.48_{2.21}$	$40.61_{32.25}$	$74.45_{20.82}$
COG-TS	$46.93_{31.63}$	$56.62_{34.70}$	$74.02_{15.11}$	$80.11_{2.95}$	$20.19_{6.53}$	$32.24_{13.47}$	$61.07_{22.01}$	$74.28_{4.21}$	$17.48_{2.92}$	$18.71_{3.04}$	$26.67_{5.20}$	$71.28_{19.95}$
COG-MS	$19.20_{8.44}$	$26.31_{13.57}$	$60.68_{26.69}$	$78.79_{4.65}$	$16.12_{0.84}$	$36.00_{22.68}$	$67.60_{17.03}$	$74.53_{2.90}$	$20.77_{2.37}$	$21.35_{2.66}$	$43.62_{24.09}$	$83.56_{3.11}$

Appendix

Table 26: Comparison of MCE (

\downarrow

lower is better) for all models and all variants seen in this work for multiple choices of the number of bins. We report

\text{mean}_{\text{std}}

. The best value in each column is shown in bold.

	SVHN			CIFAR-10			CIFAR-100
Method	AUROC $(\uparrow)$	FPR95 $(\downarrow)$	Rank $(\downarrow)$	AUROC $(\uparrow)$	FPR95 $(\downarrow)$	Rank $(\downarrow)$	AUROC $(\uparrow)$	FPR95 $(\downarrow)$	Rank $(\downarrow)$
Baseline	$93.26_{0.75}$	$42.67_{3.11}$	9.50	$87.00_{1.36}$	$71.21_{4.41}$	4.50	$70.66_{2.85}$	$86.80_{1.38}$	8.50
COG-FC	$96.11_{0.34}$	$26.30_{2.79}$	2.50	$85.11_{1.13}$	$77.61_{2.59}$	9.50	$70.20_{3.78}$	$84.34_{2.25}$	8.50
COG+LS	$93.98_{0.27}$	$25.37_{1.76}$	4.00	$82.40_{2.77}$	$74.75_{4.02}$	9.00	$73.32_{2.30}$	$81.67_{1.68}$	4.00
COG-TS	$95.43_{0.48}$	$31.51_{3.42}$	4.50	$85.53_{0.57}$	$76.66_{1.60}$	8.50	$67.50_{2.59}$	$87.76_{1.51}$	10.50
COG-MS	$95.69_{0.31}$	$29.02_{2.13}$	3.50	$84.58_{1.10}$	$78.71_{3.14}$	11.00	$68.82_{4.94}$	$88.05_{1.70}$	10.50

Appendix

Table 27: Performance in out-of-distribution detection or all models and all variants seen in this work.

	SVHN				CIFAR-10				CIFAR-100
Method	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	Rank $(\downarrow)$	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	Rank $(\downarrow)$	ACC $(\uparrow)$	NLL $(\downarrow)$	Brier $(\downarrow)$	Rank $(\downarrow)$
Baseline	$44.53_{0.94}$	$5.95_{0.19}$	$1.00_{0.02}$	10.67	$74.24_{0.54}$	$1.99_{0.13}$	$0.45_{0.01}$	8.33	$46.50_{1.29}$	$5.16_{0.34}$	$0.91_{0.02}$	8.33
COG-FC	$45.34_{0.93}$	$2.34_{0.06}$	$0.81_{0.01}$	4.00	$67.42_{0.31}$	$1.28_{0.01}$	$0.50_{0.00}$	9.33	$44.03_{1.53}$	$2.71_{0.11}$	$0.76_{0.02}$	4.33
COG+LS	$\mathbf{45.48_{1.00}}$	$\mathbf{1.87_{0.03}}$	$\mathbf{0.76_{0.01}}$	$\color[rgb]{0,0.75,0.16}{\mathbf{1.00}}$	$67.67_{0.35}$	$1.10_{0.01}$	$0.47_{0.00}$	7.67	$47.04_{1.07}$	$2.59_{0.07}$	$0.71_{0.02}$	2.00
COG-TS	$44.78_{0.68}$	$3.27_{0.09}$	$0.89_{0.02}$	9.00	$66.55_{0.39}$	$1.92_{0.04}$	$0.55_{0.01}$	11.67	$45.39_{2.00}$	$3.73_{0.21}$	$0.83_{0.03}$	8.00
COG-MS	$44.79_{0.35}$	$2.53_{0.09}$	$0.83_{0.01}$	6.00	$67.22_{1.29}$	$1.47_{0.07}$	$0.51_{0.02}$	10.67	$40.61_{1.31}$	$2.87_{0.14}$	$0.79_{0.02}$	7.33

Appendix

Table 28: Performance under covariate shifts. Shift metrics are averaged across all considered covariate shifts.

	SVHN				CIFAR-10				CIFAR-100
Method	ECE-10	ECE-15	ECE-100	ECE-500	ECE-10	ECE-15	ECE-100	ECE-500	ECE-10	ECE-15	ECE-100	ECE-500
Baseline	$47.06_{0.83}$	$47.06_{0.83}$	$47.06_{0.83}$	$47.06_{0.83}$	$20.89_{0.53}$	$20.89_{0.53}$	$20.89_{0.53}$	$20.86_{0.53}$	$40.89_{1.24}$	$40.88_{1.24}$	$40.88_{1.24}$	$40.86_{1.24}$
COG-FC	$30.31_{0.98}$	$30.31_{0.98}$	$30.30_{0.98}$	$30.31_{0.98}$	$17.97_{0.28}$	$17.97_{0.28}$	$17.96_{0.28}$	$17.94_{0.28}$	$20.91_{1.30}$	$20.91_{1.30}$	$20.90_{1.30}$	$20.90_{1.30}$
COG+LS	$\mathbf{22.99_{0.56}}$	$\mathbf{22.99_{0.56}}$	$\mathbf{22.99_{0.55}}$	$\mathbf{23.00_{0.55}}$	$10.02_{0.33}$	$10.02_{0.33}$	$10.02_{0.33}$	$10.02_{0.33}$	$12.78_{2.20}$	$12.79_{2.17}$	$12.81_{2.18}$	$12.82_{2.17}$
COG-TS	$38.11_{0.94}$	$38.11_{0.94}$	$38.11_{0.94}$	$38.12_{0.94}$	$23.93_{0.32}$	$23.93_{0.32}$	$23.93_{0.32}$	$23.91_{0.32}$	$32.18_{1.10}$	$32.17_{1.10}$	$32.16_{1.10}$	$32.16_{1.10}$
COG-MS	$32.44_{1.20}$	$32.44_{1.20}$	$32.44_{1.20}$	$32.45_{1.19}$	$19.02_{0.87}$	$19.02_{0.87}$	$19.01_{0.87}$	$19.00_{0.87}$	$20.49_{1.95}$	$20.49_{1.95}$	$20.49_{1.95}$	$20.49_{1.95}$

Appendix

Table 29: ECE averaged across all considered covariate shifts for all models and all variants seen in this work for multiple choices of the number of bins.

	SVHN				CIFAR-10				CIFAR-100
Method	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500	RMSCE-10	RMSCE-15	RMSCE-100	RMSCE-500
Baseline	$47.45_{0.83}$	$47.59_{0.83}$	$48.37_{0.83}$	$49.01_{0.86}$	$21.87_{0.62}$	$22.35_{0.65}$	$24.44_{0.81}$	$25.63_{0.90}$	$41.56_{1.20}$	$41.87_{1.18}$	$43.27_{1.11}$	$44.06_{1.09}$
COG-FC	$31.40_{1.02}$	$31.66_{1.01}$	$32.16_{1.07}$	$32.25_{1.06}$	$19.19_{0.36}$	$19.65_{0.37}$	$20.75_{0.43}$	$20.93_{0.43}$	$22.07_{1.34}$	$22.26_{1.36}$	$22.59_{1.45}$	$22.63_{1.46}$
COG+LS	$\mathbf{24.11_{0.56}}$	$\mathbf{24.76_{0.65}}$	$\mathbf{25.07_{0.63}}$	$\mathbf{25.20_{0.64}}$	$11.89_{0.27}$	$11.79_{0.26}$	$12.09_{0.27}$	$12.15_{0.27}$	$14.94_{2.26}$	$15.37_{2.34}$	$15.54_{2.35}$	$15.57_{2.35}$
COG-TS	$38.94_{0.92}$	$39.21_{0.94}$	$40.27_{0.98}$	$40.71_{1.02}$	$24.84_{0.35}$	$25.29_{0.37}$	$27.01_{0.44}$	$27.74_{0.44}$	$33.18_{1.06}$	$33.52_{1.05}$	$34.54_{1.02}$	$34.81_{1.03}$
COG-MS	$33.51_{1.18}$	$33.79_{1.18}$	$34.39_{1.22}$	$34.49_{1.22}$	$20.18_{0.83}$	$20.63_{0.83}$	$21.74_{0.92}$	$22.03_{0.97}$	$21.67_{2.00}$	$21.85_{2.01}$	$22.16_{2.05}$	$22.19_{2.06}$

Appendix

Table 30: RMSCE averaged across all considered covariate shifts for all models and all variants seen in this work for multiple choices of the number of bins.

	SVHN				CIFAR-10				CIFAR-100
Method	MCE-10	MCE-15	MCE-100	MCE-500	MCE-10	MCE-15	MCE-100	MCE-500	MCE-10	MCE-15	MCE-100	MCE-500
Baseline	$60.58_{0.79}$	$64.16_{0.72}$	$68.20_{1.46}$	$74.20_{0.61}$	$44.61_{2.00}$	$47.01_{1.82}$	$60.00_{15.71}$	$76.09_{2.62}$	$59.26_{0.73}$	$61.89_{0.86}$	$68.29_{10.60}$	$69.41_{9.97}$
COG-FC	$45.30_{1.81}$	$45.53_{1.47}$	$47.61_{1.88}$	$\mathbf{51.98_{1.85}}$	$31.98_{0.97}$	$32.05_{0.80}$	$32.98_{0.97}$	$82.21_{1.30}$	$31.84_{2.22}$	$32.16_{2.21}$	$33.02_{2.18}$	$46.77_{26.24}$
COG+LS	$\mathbf{36.84_{1.36}}$	$41.11_{2.38}$	$81.65_{18.07}$	$95.96_{6.08}$	$19.74_{0.50}$	$19.88_{0.61}$	$23.76_{6.11}$	$59.58_{32.80}$	$27.12_{3.57}$	$27.44_{3.97}$	$49.32_{29.19}$	$67.27_{30.02}$
COG-TS	$54.03_{1.25}$	$56.26_{1.50}$	$58.00_{1.73}$	$69.84_{12.18}$	$49.59_{17.78}$	$50.80_{17.10}$	$66.09_{19.91}$	$79.75_{1.81}$	$47.66_{1.03}$	$47.98_{1.28}$	$48.84_{1.04}$	$50.20_{1.24}$
COG-MS	$48.16_{1.48}$	$48.68_{1.56}$	$57.82_{15.40}$	$72.49_{15.32}$	$33.43_{1.39}$	$33.61_{1.66}$	$34.62_{1.69}$	$54.18_{23.89}$	$31.79_{2.98}$	$43.52_{28.00}$	$34.01_{5.52}$	$48.23_{25.65}$

Appendix

Table 31: MCE averaged across all considered covariate shifts for all models and all variants seen in this work for multiple choices of the number of bins.

Model Calibration

Model calibration aligns predicted confidence or model outputs with observed outcomes and measurements to support more reliable decisions.

See all 214 papers

Apr 22, 2026

Too Sharp, Too Sure: When Calibration Follows Curvature

Modern neural networks can achieve high accuracy while remaining poorly calibrated, producing confidence estimates that do not match empirical correctness. Yet calibration is often treated as a post-hoc attribute. We take a different perspective: we study calibration as a training-time phenomenon on small vision tasks, and ask whether calibrated solutions can be obtained reliably by intervening on the training procedure. We identify a tight coupling between calibration, curvature, and margins during training of deep networks under multiple gradient-based methods. Empirically, Expected Calibration Error (ECE) closely tracks curvature-based sharpness throughout optimization. Mathematically, we show that both ECE and Gauss--Newton curvature are controlled, up to problem-specific constants, by the same margin-dependent exponential tail functional along the trajectory. Guided by this mechanism, we introduce a margin-aware training objective that explicitly targets robust-margin tails and local smoothness, yielding improved out-of-sample calibration across optimizers without sacrificing accuracy.

Alessandro Morosini, Matea Gjika, Tomaso Poggio +1

Massachusetts Institute of Technology, Cambridge, MA, USA.

Loss Landscape Sharpness Margin-Based Learning Model Calibration Neural Network Training Dynamics Statistical Learning Theory

Ask PDF

Oct 6, 2026

Neighborhood Smoothing for Calibration

Modern neural networks are often miscalibrated, with a tendency to overconfidence. Existing train-time calibration methods largely modify task losses or calibration penalties, leaving neighborhood structure in learned representations underexploited. We introduce graph smoothing as a general principle for train-time calibration, which encourages similar predictive distributions across neighboring samples in representation space. We analyze the effects of graph smoothing, deriving bounds that connect predictive divergence between neighboring samples to local confidence variation and to the propagation of pointwise calibration error, and characterize the conditions under which smoothing can or cannot improve calibration. In light of this analysis, we propose \modelNoSpace, a graph-based train-time regularizer that penalizes the Jensen--Shannon divergence between predictive distributions of neighboring samples. We present a thorough empirical analysis, showing that across standard calibration benchmarks, \model improves predictive quality, and the improvement is complementary to post-hoc calibration: after temperature scaling, \model attains the lowest NLL of all evaluated train-time methods in seven of the eight image and tabular settings. These findings demonstrate the value of graph smoothing over learned representations for neural network calibration.

Idan Horowitz, Avigdor Gal

Faculty of Data and Decision Sciences Technion Haifa, 3200003, IL

Model Calibration

Ask PDF

Oct 8, 2026

Do Not Train Away Uncertainty: Early Uncertainty Anchored Calibration

Deep neural networks, including large language models, have achieved remarkable performance across various tasks. However, they are prone to overconfidence during training or fine-tuning. In this work, we observe a consistent phenomenon across different models that the early model is better calibrated, while later training or fine-tuning yields marginal accuracy gains but substantially increases calibration errors. Our analysis suggests that the early model retains uncertainty awareness in both its predictions and features, which is gradually lost with continued training. To avoid training away this uncertainty awareness, we propose EUA-Cal, a novel method that exploits the Early model as an Uncertainty Anchor for Calibration. EUA-Cal introduces early prediction regularization to preserve early predictive uncertainty and prototype structure regularization to exploit uncertainty reflected in the early feature space, jointly mitigating overconfidence. Extensive experiments on image classification and multiple-choice question answering across eight diverse models demonstrate that EUA-Cal outperforms state-of-the-art calibration methods.

Yutong Xie, Jiawei Tang, Zhenglin Hua +6

School of Software Engineering, Southeast University, Nanjing, China · School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Computing Information Sciences, Saint Francis University, Hong Kong, China +2

Model Calibration Uncertainty Calibration Language Model Calibration

Ask PDF

Abstract

Model Calibration

Explore similar work

Too Sharp, Too Sure: When Calibration Follows Curvature

Neighborhood Smoothing for Calibration

Do Not Train Away Uncertainty: Early Uncertainty Anchored Calibration