We study post-calibration uncertainty for trained ensembles of classifiers. Specifically, we consider both aleatoric (label noise) and epistemic (model) uncertainty. Among the most popular and widely used calibration methods in classification are temperature scaling (i.e., pool-then-calibrate) and conformal methods. However, the main shortcoming of these calibration methods is that they do not balance the proportion of aleatoric and epistemic uncertainty. Not balancing these uncertainties can severely misrepresent predictive uncertainty, leading to overconfident predictions in some input regions while being underconfident in others. To address this shortcoming, we present a simple but powerful calibration algorithm Joint Uncertainty Calibration (JUCAL) that jointly calibrates aleatoric and epistemic uncertainty. JUCAL jointly calibrates two constants to weight and scale epistemic and aleatoric uncertainties by optimizing the negative log-likelihood (NLL) on the validation/calibration dataset. JUCAL can be applied to any trained ensemble of classifiers (e.g., transformers, CNNs, or tree-based methods), with minimal computational overhead, without requiring access to the models' internal parameters. We experimentally evaluate JUCAL on various text classification tasks, for ensembles of varying sizes and with different ensembling strategies. Our experiments show that JUCAL significantly outperforms SOTA calibration methods across all considered classification tasks, reducing NLL and predictive set size by up to 15% and 20%, respectively. Interestingly, even applying JUCAL to an ensemble of size 5 can outperform temperature-scaled ensembles of size up to 50 in terms of NLL and predictive set size, resulting in up to 10 times smaller inference costs. Thus, we propose JUCAL as a new go-to method for calibrating ensembles in classification.
Figures & tables
Figure 1: Predictive probability estimation for a synthetic 2D binary classification task. (a) Softmax outputs from a single NN. (b) Deep Ensemble. (c) & (d) show the same ensemble as in (b) but with different calibration algorithms applied to it. In all cases, the uncertainty peaks near the decision boundary, but only JUCAL sufficiently accounts for epistemic uncertainty by widening the uncertain region (bright colors) as the distance to the training data increases. This reflects the model’s limited knowledge in data-sparse regions, highlighting the ensemble’s ability to distinguish between aleatory and epistemic components.
Figure 2: Scatter plots of ensemble members’ softmax outputs for (a) binary ( K=2) and (b-e) ternary ( K=3 ) classification. Each subplot shows a different possibility of how the M=50 predictions could be arranged for a fixed input point x . Each point represents a probability vector p(y∣x,θm) over K classes estimated by an ensemble member. (a)&(b) low total predictive uncertainty; (c) very high aleatoric and low epistemic uncertainty; (d) low aleatoric and very high epistemic uncertainty; (d)&(e) high epistemic uncertainty. Theoretically (d) claims that the aleatoric uncertainty is certainly low, while (e) is uncertain about the aleatoric uncertainty, but in practice, both (d)&(e) should usually be simply interpreted as high epistemic uncertainty (see Remark A.1 ).
Figure 3: Binary classification example with X∼N(0,1) . The ensemble logits strongly agree in the center of the distribution x∈[−2,2] , but disagree more as one moves away from the center. The two subplots show the same ensemble before and after applying JUCAL to it.
Figure 4: Text Classification Results. For each of the six subplots, lower values of the metrics (displayed on the y-axis) are better. On the x-axis, we list 12 text classification datasets (a 10%-mini and a 100%-full version of 6 distinct datasets). The striped bars correspond to ensemble size M=5 , while the non-striped bars correspond to M=50 . JUCAL’s results are yellow. For all six metrics (defined in Section 5.1.1 ), we show the average and ±1 standard deviation across 5 random validation-test splits. (a) NLL normalized by the mean of JUCAL Greedy-50 on the corresponding full dataset; (b) AORAC=1−AURAC ; (c) AOROC=1−AUROC ; (d) Average set size for the coverage threshold of 99.9% for DBpedia (Full and Mini) and 99% for all other datasets; (e) Brier Score ; (f) Misclassification Rate=1−Accuracy . For more detailed results, see the corresponding tables in Appendix F .
Figure 5: Image Classification Results. For each of the six subplots, lower values of the metrics (displayed on the y-axis) are better. On the x-axis, we list distinct image classification datasets (and two hyperparameter-ablation studies for MNIST). JUCAL’s results are yellow. For all six metrics (defined in Section 5.1.1 ), we show the average and ±1 standard deviation across 10 random train-validation-test splits. (a) NLL normalized by the mean of JUCAL Greedy-5; (b) AORAC=1−AURAC ; (c) AOROC=1−AUROC ; (d) Average set size for the coverage threshold of 99% for CIFAR-10 , 90% for CIFAR-100 , and 99.9% for al variants of MNIST and Fashion-MNIST ; (e) Brier Score ; (f) Misclassification Rate=1−Accuracy .
Figure 6: Epistemic and Aleatoric Uncertainty (computed as in Section A.2.1 ) of JUCAL applied on Greedy-50 ensembles across six datasets in the metadataset. We compare the full (100%) and the mini (10%) metadataset configurations for both epistemic and aleatoric uncertainty. Bars indicate the mean uncertainty, and error bars denote one standard deviation over random seeds.
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Different possible behaviors of a higher-order distribution over the simplex △K−1 in a binary (a) and ternary (b-e) classification task. We both show the density of a higher-order distribution (such as a posterior distribution) via colors and M=50 samples from this distribution via semi-transparent black circles. Each point on the simplex △K−1 corresponds to a (first-order) distribution over the K classes. Sub-figure (a)&(b) show almost no aleatoric or epistemic uncertainty (i.e., very low aleatoric and epistemic uncertainty, leading to a low total predictive uncertainty), (c) shows almost only aleatoric uncertainty , (d) shows almost only epistemic uncertainty and (e) shows both aleatoric and epistemic uncertainty . More precisely, (e) shows epistemic uncertainty on whether the aleatoric uncertainty is large or small, whereas (d) is theoretically more certain that the aleatoric uncertainty is large; (a), (b), and theoretically (d) are more certain that the aleatoric uncertainty is low. Note that (d)’s “certaitny” on the absence of aleatoric uncertainty, should not be trusted in typical settings as discussed in Remarks A.1 and A.4 . (c) is certain that the aleatoric uncertainty is high.
More observations
Better prior
More covariates
Smaller noise
Epistemic
Decreases
Decreases
Increases (typically) / /
Decreases
Aleatoric
No effect
No effect
Decreases /
Decreases
Appendix
Table 1: Expected effects of different factors on epistemic and aleatoric uncertainty.
Figure 8: Estimated Epistemic Uncertainty for Figure 1
Figure 9: Estimated Aleatoric Uncertainty for Figure 1 . In regions of high epistemic uncertainty, one usually does not know if the aleatoric uncertainty is high or low; thus it has the possibility to be high, and averaging over all possibilities can result in quite high estimates of the aleatoric uncertainty.
Figure 10: The same ensemble without and with JUCAL calibration. The logit diversity increases as you move further OOD, but the probability-diversity decreases without JUCAL.
Figure 11: Desired behavior of a higher-order distribution over the simplex in a ternary classification task. Sub-figure (a) almost no aleatoric or epistemic uncertainty , (b) shows almost only aleatoric uncertainty , (c) shows almost only epistemic uncertainty and (d) shows both aleatoric and epistemic uncertainty
Figure 12: Illustrating the point-wise predicted Dirichlet distributions in a 1D binary classification task with class probabilities defined by p(y=1∣x)=0.5+0.5sin(x) , where x∼N(0,1) (visualized as green line) and y∼Bernoulli(p(y=1∣x)) . For each value x∈[−15,15] , we visualize the density of the corresponding Dirichlet distribution over the interval [0,1] , with black circles indicating the training data.
Figure 13: The same ensemble without and with JUCAL calibration. The logit diversity increases as you move further OOD, but the probability-diversity can simultaneously decrease if the logit diversity does not grow fast enough. JUCAL can scale the logit-diversity via c2 to prevent this.
Figure 14: For an ensemble consisting of 5 CNNs trained on CIFAR-10, we compute multiple quantities on the training, validation, and test datasets for multiple different values of c1 and c2 . We used Equations 7 and 6 from Section A.2.1 to compute aleatoric, epistemic, and total uncertainty, while there would be other alternatives too (see Section A.2 ).
Ensemble Type
DBpedia
News
SST-2
SetFit
Tweet
IMDB
G5
0.0376 ± 0.0005
0.1682 ± 0.0048
0.1359 ± 0.0051
0.5465 ± 0.0033
0.5095 ± 0.0089
0.1171 ± 0.0028
G5 p.t.c.
0.0348 ± 0.0007
0.1618 ± 0.0052
0.1208 ± 0.0040
0.5431 ± 0.0019
0.5012 ± 0.0052
0.1018 ± 0.0022
G5 JUCAL
0.0290 ± 0.0004
0.1479 ± 0.0023
0.1143 ± 0.0032
0.4965 ± 0.0013
0.4772 ± 0.0028
0.1005 ± 0.0018
G50
0.0349 ± 0.0005
0.1541 ± 0.0043
0.1137 ± 0.0039
0.531 ± 0.0016
0.4763 ± 0.0052
0.1050 ± 0.0026
G50 p.t.c.
0.0331 ± 0.0003
0.1510 ± 0.0037
0.1130 ± 0.0035
0.5309 ± 0.0016
0.4758 ± 0.0049
0.1042 ± 0.0019
G50 JUCAL
0.0288 ± 0.0004
0.1423 ± 0.0024
0.1090 ± 0.0032
0.4972 ± 0.0018
0.4680 ± 0.0045
0.0983 ± 0.0017
Appendix
Table 2: FTC-metadataset full: Negative log-likelihood ( NLLmean over data splits; mean ± 95% confidence interval half-width) on the full dataset (100%). The best mean is shown in bold, and methods not significantly different from the best (paired test, α=0.05 ) are shaded.
Ensemble Type
DBpedia
News
SST-2
SetFit
Tweet
IMDB
G5
0.9895 ± 0.0
0.981 ± 0.0011
0.984 ± 0.0005
0.8915 ± 0.0008
0.9103 ± 0.0028
0.9859 ± 0.0002
G5 p.t.c.
0.9895 ± 0.0
0.981 ± 0.0011
0.984 ± 0.0005
0.8915 ± 0.0008
0.9103 ± 0.0027
0.9859 ± 0.0002
G5 JUCAL
0.9897 ± 0.0
0.9829 ± 0.0005
0.9842 ± 0.0005
0.924 ± 0.0006
0.9211 ± 0.0006
0.9858 ± 0.0002
G50
0.9895 ± 0.0
0.981 ± 0.0008
0.9833 ± 0.0005
0.9023 ± 0.0006
0.9157 ± 0.0021
0.9838 ± 0.0003
G50 p.t.c.
0.9895 ± 0.0
0.981 ± 0.0008
0.9833 ± 0.0005
0.9023 ± 0.0006
0.9158 ± 0.0021
0.9838 ± 0.0003
G50 JUCAL
0.9897 ± 0.0
0.9835 ± 0.0005
0.9849 ± 0.0005
0.9237 ± 0.0007
0.9236 ± 0.0014
0.9855 ± 0.0002
Appendix
Table 3: FTC-metadataset full: Area Under the Rejection-Accuracy Curve (AURAC) over data splits; mean ± 95% confidence interval half-width) on the full dataset (100%). The best mean is shown in bold, and methods not significantly different from the best (paired test, α=0.05 ) are shaded.
Ensemble Type
DBpedia
News
SST-2
SetFit
Tweet
IMDB
G5
0.9998312 ± 0.0
0.9929 ± 0.0007
0.9907 ± 0.0007
0.9144 ± 0.0008
0.9316 ± 0.0019
0.9934 ± 0.0003
G5 p.t.c.
0.9998311 ± 0.0
0.9929 ± 0.0007
0.9907 ± 0.0007
0.9144 ± 0.0008
0.9316 ± 0.0018
0.9934 ± 0.0003
G5 JUCAL
0.9998758 ± 0.0
0.9943 ± 0.0003
0.9912 ± 0.0006
0.9377 ± 0.0004
0.9383 ± 0.0010
0.9934 ± 0.0002
G50
0.9998198 ± 0.0
0.9931 ± 0.0005
0.9898 ± 0.0007
0.9229 ± 0.0005
0.9369 ± 0.0014
0.9911 ± 0.0003
G50 p.t.c.
0.9998199 ± 0.0
0.9931 ± 0.0005
0.9898 ± 0.0007
0.9229 ± 0.0005
0.9369 ± 0.0014
0.9911 ± 0.0003
G50 JUCAL
0.9998785 ± 0.0
0.9948 ± 0.0004
0.9917 ± 0.0007
0.9371 ± 0.0006
0.9405 ± 0.0014
0.9930 ± 0.0004
Appendix
Table 4: FTC-metadataset full: Area under the ROC ( AUROC over data splits; mean ± 95% confidence interval half-width) on the full dataset (100%). The best mean is shown in bold, and methods not significantly different from the best (paired test, α=0.05 ) are shaded.
Ensemble Type
DBpedia
News
SST-2
SetFit
Tweet
IMDB
G5
1.2941 ± 0.0395
1.3517 ± 0.0385
1.1544 ± 0.0097
2.6642 ± 0.0228
2.3281 ± 0.0963
1.0996 ± 0.0065
G5 p.t.c.
1.3008 ± 0.0484
1.3591 ± 0.0424
1.1550 ± 0.0107
2.6567 ± 0.0209
2.3313 ± 0.0993
1.1003 ± 0.0062
G5 JUCAL
1.2270 ± 0.0438
1.2490 ± 0.0161
1.1459 ± 0.0116
2.2368 ± 0.0231
2.1286 ± 0.0722
1.1004 ± 0.0039
G50
1.3516 ± 0.0428
1.3436 ± 0.0313
1.1617 ± 0.0148
2.6519 ± 0.0237
2.2280 ± 0.0507
1.1135 ± 0.0070
G50 p.t.c.
1.3534 ± 0.0398
1.3517 ± 0.0226
1.1621 ± 0.0175
2.6514 ± 0.0490
2.2261 ± 0.0476
1.1140 ± 0.0092
G50 JUCAL
1.2072 ± 0.0358
1.2228 ± 0.0244
1.1385 ± 0.0094
2.2334 ± 0.0199
2.0633 ± 0.0291
1.1005 ± 0.0051
Appendix
Table 5: FTC-metadataset full: Set size over data splits; mean ± 95% confidence interval half-width) on the full dataset (100%). The best mean is shown in bold, and methods not significantly different from the best (paired test, α=0.05 ) are shaded. Here the coverage threshold is 99% for all but DBpedia where it is 99.9%
Ensemble Type
DBpedia
News
SST-2
SetFit
Tweet
IMDB
G5
0.0432 ± 0.0012
0.2321 ± 0.0031
0.1534 ± 0.0044
0.4067 ± 0.002
0.5311 ± 0.0065
0.1334 ± 0.0064
G5 p.t.c.
0.0341 ± 0.0008
0.2050 ± 0.0026
0.1472 ± 0.0020
0.4051 ± 0.0018
0.5294 ± 0.0062
0.1314 ± 0.0043
G5 JUCAL
0.0326 ± 0.0008
0.1966 ± 0.0026
0.1396 ± 0.002
0.3684 ± 0.0018
0.5205 ± 0.0059
0.1303 ± 0.0034
G50
0.0352 ± 0.0009
0.1967 ± 0.0032
0.1320 ± 0.0035
0.3594 ± 0.0014
0.4980 ± 0.0063
0.1258 ± 0.0020
G50 p.t.c.
0.0346 ± 0.0004
0.1964 ± 0.0031
0.1320 ± 0.0034
0.3594 ± 0.0014
0.4979 ± 0.0061
0.1255 ± 0.0014
G50 JUCAL
0.0305 ± 0.0008
0.1899 ± 0.0028
0.1309 ± 0.0034
0.3480 ± 0.0013
0.4979 ± 0.0059
0.1257 ± 0.0018
Appendix
Table 6: FTC-metadataset mini (10%): Negative log-likelihood ( NLLmean over data splits; mean ± 95% confidence interval half-width) on the full dataset (100%). The best mean is shown in bold, and methods not significantly different from the best (paired test, α=0.05 ) are shaded.
Ensemble Type
DBpedia
News
SST-2
SetFit
Tweet
IMDB
G5
0.9895 ± 0.0001
0.9769 ± 0.0002
0.979 ± 0.0008
0.9406 ± 0.0005
0.8982 ± 0.0026
0.9809 ± 0.0006
G5 p.t.c.
0.9895 ± 0.0001
0.9769 ± 0.0003
0.979 ± 0.0008
0.9407 ± 0.0005
0.8981 ± 0.0026
0.9809 ± 0.0006
G5 JUCAL
0.9895 ± 0.0001
0.9779 ± 0.0003
0.9817 ± 0.0004
0.95 ± 0.0005
0.9025 ± 0.0018
0.9819 ± 0.0005
G50
0.9893 ± 0.0001
0.9748 ± 0.0005
0.9822 ± 0.0005
0.9503 ± 0.0003
0.9091 ± 0.0018
0.9821 ± 0.0004
G50 p.t.c.
0.9893 ± 0.0001
0.9748 ± 0.0005
0.9822 ± 0.0005
0.9503 ± 0.0003
0.9091 ± 0.0018
0.9821 ± 0.0004
G50 JUCAL
0.9896 ± 0.0
0.978 ± 0.0005
0.9828 ± 0.0005
0.9554 ± 0.0002
0.9099 ± 0.0017
0.9822 ± 0.0005
Appendix
Table 7: FTC-metadataset mini (10%): Area Under the Rejection-Accuracy Curve (AURAC) over data splits; mean ± 95% confidence interval half-width) on the full dataset (100%). The best mean is shown in bold, and methods not significantly different from the best (paired test, α=0.05 ) are shaded.
Ensemble Type
DBpedia
News
SST-2
SetFit
Tweet
IMDB
G5
0.9998 ± 0.0
0.9899 ± 0.0002
0.9853 ± 0.0007
0.9539 ± 0.0004
0.9226 ± 0.0015
0.9872 ± 0.0007
G5 p.t.c.
0.9998 ± 0.0
0.9899 ± 0.0001
0.9853 ± 0.0007
0.9539 ± 0.0004
0.9225 ± 0.0015
0.9872 ± 0.0007
G5 JUCAL
0.9998 ± 0.0
0.9905 ± 0.0002
0.9874 ± 0.0005
0.9620 ± 0.0004
0.9253 ± 0.0019
0.9883 ± 0.0006
G50
0.9997 ± 0.0
0.9889 ± 0.0005
0.9878 ± 0.0008
0.9632 ± 0.0002
0.9302 ± 0.0013
0.9886 ± 0.0001
G50 p.t.c.
0.9997 ± 0.0
0.9889 ± 0.0005
0.9878 ± 0.0008
0.9632 ± 0.0002
0.9302 ± 0.0013
0.9886 ± 0.0001
G50 JUCAL
0.9998 ± 0.0
0.9907 ± 0.0003
0.9885 ± 0.0008
0.9667 ± 0.0001
0.9306 ± 0.0012
0.9886 ± 0.0002
Appendix
Table 8: FTC-metadataset mini (10%): Are under the ROC ( AUROC over data splits; mean ± 95% confidence interval half-width) on the full dataset (100%). The best mean is shown in bold, and methods not significantly different from the best (paired test, α=0.05 ) are shaded.
Ensemble Type
DBpedia
News
SST-2
SetFit
Tweet
IMDB
G5
1.3673 ± 0.0702
1.4414 ± 0.0167
1.2467 ± 0.0135
2.2989 ± 0.0027
2.2997 ± 0.0356
1.2392 ± 0.0099
G5 p.t.c.
1.4313 ± 0.0695
1.4475 ± 0.0184
1.2504 ± 0.0149
2.3124 ± 0.0136
2.3028 ± 0.0366
1.2347 ± 0.0158
G5 JUCAL
1.4522 ± 0.0567
1.4131 ± 0.0276
1.2091 ± 0.0115
1.9976 ± 0.0254
2.2110 ± 0.0277
1.2148 ± 0.0115
G50
1.6459 ± 0.0546
1.7193 ± 0.0952
1.1918 ± 0.0132
2.1899 ± 0.0061
2.1735 ± 0.0475
1.1821 ± 0.0043
G50 p.t.c.
1.6453 ± 0.0563
1.7274 ± 0.0792
1.1933 ± 0.0119
2.2008 ± 0.0059
2.1684 ± 0.0414
1.1831 ± 0.0092
G50 JUCAL
1.3105 ± 0.0232
1.4389 ± 0.0334
1.1862 ± 0.0120
1.8980 ± 0.0086
2.1470 ± 0.0444
1.1819 ± 0.0126
Appendix
Table 9: FTC-metadataset mini (10%): Set size over data splits; mean ± 95% confidence interval half-width) on the full dataset (100%). The best mean is shown in bold, and methods not significantly different from the best (paired test, α=0.05 ) are shaded. Here the coverage threshold is 99% for all but DBpedia where it is 99.9%
Figure 15: ECE Results for Text Classification. For the ECE, lower values (displayed on the y-axis) are better. On the x-axis, we list 12 text classification datasets (a 10%-mini and a 100%-full version of 6 distinct datasets). The striped bars correspond to ensemble size M=5 , while the non-striped bars correspond to M=50 . JUCAL’s results are yellow. We show the average ECE and ±1 standard deviation across 5 random validation-test splits.
Figure 16: ECE Results for Image Classification. For the ECE, lower values (displayed on the y-axis) are better. On the x-axis, we list distinct image classification datasets (and two hyperparameter-ablation studies for MNIST). JUCAL’s results are yellow. We show the average ECE and ±1 standard deviation across 10 random train-validation-test splits.
Figure 17: Conformal Set Size Results for Text Classification. For the conformal set size, lower values (displayed on the y-axis) are better. On the x-axis, we list 12 text classification datasets (a 10%-mini and a 100%-full version of 6 distinct datasets). The striped bars correspond to ensemble size M=5 , while the non-striped bars correspond to M=50 . JUCAL’s results are yellow. We show the average conformal prediction set size (for the conformal target coverage threshold of 99.9% for DBpedia (Full and Mini) and 99% for all other datasets) and ±1 standard deviation across 5 random validation-test splits.
Figure 18: Conformal Set Size Results for Image Classification. For the conformal set size, lower values (displayed on the y-axis) are better. On the x-axis, we list distinct image classification datasets (and two hyperparameter-ablation studies for MNIST). JUCAL’s results are yellow. We show the average conformal prediction set size (for the conformal target coverage threshold of 99% for CIFAR-10 , 90% for CIFAR-100 , and 99.9% for al variants of MNIST and Fashion-MNIST ) and ±1 standard deviation across 10 random train-validation-test splits.
Figure 19: Conformal Log Set Size Results for Text Classification. For the conformal log set size, lower values (displayed on the y-axis) are better. On the x-axis, we list 12 text classification datasets (a 10%-mini and a 100%-full version of 6 distinct datasets). The striped bars correspond to ensemble size M=5 , while the non-striped bars correspond to M=50 . JUCAL’s results are yellow. We show the average of the logarithm of the conformal prediction set size (for the conformal target coverage threshold of 99.9% for DBpedia (Full and Mini) and 99% for all other datasets) and ±1 standard deviation across 5 random validation-test splits.
Figure 20: Conformal Log Set Size Results for Image Classification. For the conformal log set size, lower values (displayed on the y-axis) are better. On the x-axis, we list distinct image classification datasets (and two hyperparameter-ablation studies for MNIST). JUCAL’s results are yellow. We show the average logarithmic conformal prediction set size (for the conformal target coverage threshold of 99% for CIFAR-10 , 90% for CIFAR-100 , and 99.9% for al variants of MNIST and Fashion-MNIST ) and ±1 standard deviation across 10 random train-validation-test splits.
Figure 21: Conformal Coverage Results for Text Classification. For the conformal coverage, values near the target coverage indicate better calibration. Larger values of coverage are more desirable than smaller values of coverage (unless larger coverage leads to larger set sizes). On the x-axis, we list 12 text classification datasets (a 10%-mini and a 100%-full version of 6 distinct datasets). The striped bars correspond to ensemble size M=5 , while the non-striped bars correspond to M=50 . JUCAL’s results are yellow. We show the average test-coverage (for the conformal target coverage threshold of 99.9% for DBpedia (Full and Mini) and 99% for all other datasets), and ±1 standard deviation across 5 random validation-test splits.
Figure 22: Conformal Coverage Results for Image Classification. For the conformal coverage, values near the target coverage indicate better calibration. Larger values of coverage are more desirable than smaller values of coverage (unless larger coverage leads to larger set sizes). On the x-axis, we list distinct image classification datasets (and two hyperparameter-ablation studies for MNIST). JUCAL’s results are yellow. We show the average test-coverage (for the conformal target coverage threshold of 99% for CIFAR-10 , 90% for CIFAR-100 , and 99.9% for al variants of MNIST and Fashion-MNIST ) and ±1 standard deviation across 10 random train-validation-test splits.
Figure 23: Softmax outputs visualizing the estimated predictive probabilities calibrated by JUCAL for a synthetic 2D binary classification task.
Figure 24: Softmax outputs visualizing the estimated predictive probabilities from a single neural network trained on two dataset configurations of ta synthetic 2D binary classification task.
Dataset
Classes
Members
Train Size
Valid Size
Test Size
DBpedia Full
14
25
448,000
112,000
70,000
DBpedia Mini
14
65
44,800
112,000
70,000
News Full
4
99
96,000
24,000
7,600
News Mini
4
120
9,600
24,000
7,600
SST-2 Full
2
125
43,103
13,470
10,776
SST-2 Mini
2
125
4,310
13,470
10,776
Appendix
Table 10: Summary of the underlying datasets from which the FTC-metadataset is constructed by [ 5 ] .
Dataset
Ensemble Method
Ensemble Selection Time (s)
Calibration Time (s)
DBpedia Full
JUCAL Greedy-50
17.6798 ± 0.5566
680.2392 ± 9.9481
DBpedia Full
JUCAL Greedy-5
0.6779 ± 0.5347
92.5821 ± 7.4349
DBpedia Mini
JUCAL Greedy-50
51.0481 ± 5.3242
764.0273 ± 26.3293
DBpedia Mini
JUCAL Greedy-5
0.8412 ± 0.0215
99.8790 ± 12.2445
News Full
JUCAL Greedy-50
8.8411 ± 0.2699
78.3229 ± 0.8914
News Full
JUCAL Greedy-5
0.6653 ± 0.5407
11.2228 ± 0.3444
Appendix
Table 11: Ensemble selection and calibration time (mean ± std in seconds) for JUCAL on Greedy-50 and Greedy-5 across all datasets (Full vs Mini).
Uncertainty estimation is critical for deploying machine learning models in high-stakes settings. However, classical calibration only assesses the reliability of predicted probabilities and does not evaluate whether epistemic uncertainty estimates are themselves trustworthy. This limitation is particularly relevant for second-order classification models. We introduce epistemic calibration, a principled criterion that measures whether reported epistemic uncertainty faithfully reflects the dispersion of model predictions around the ground truth. We show that epistemic calibration is a strictly stronger notion than classical calibration and captures failure modes invisible to standard metrics. We relate this work to the existing literature through an impossibility theorem that holds under the epistemic calibration hypothesis. To operationalize this concept, we propose the Expected Epistemic Calibration Error (EECE), which we prove to be a consistent estimator of a True Epistemic Calibration Error (TECE). Experiments across a broad range of uncertainty quantification methods show that epistemic calibration is a coherent and meaningful criterion and reveal substantial differences across methods, despite similar predictive performance.
Arthur Hoarau
Université de Lorraine, CentraleSupélec · Loria, CNRS, Metz, France
Deep neural networks, including large language models, have achieved remarkable performance across various tasks. However, they are prone to overconfidence during training or fine-tuning. In this work, we observe a consistent phenomenon across different models that the early model is better calibrated, while later training or fine-tuning yields marginal accuracy gains but substantially increases calibration errors. Our analysis suggests that the early model retains uncertainty awareness in both its predictions and features, which is gradually lost with continued training. To avoid training away this uncertainty awareness, we propose EUA-Cal, a novel method that exploits the Early model as an Uncertainty Anchor for Calibration. EUA-Cal introduces early prediction regularization to preserve early predictive uncertainty and prototype structure regularization to exploit uncertainty reflected in the early feature space, jointly mitigating overconfidence. Extensive experiments on image classification and multiple-choice question answering across eight diverse models demonstrate that EUA-Cal outperforms state-of-the-art calibration methods.
Yutong Xie, Jiawei Tang, Zhenglin Hua +6
School of Software Engineering, Southeast University, Nanjing, China · School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Computing Information Sciences, Saint Francis University, Hong Kong, China +2
Heterogeneous model collaboration seeks to exploit the complementary strengths of different models to balance predictive performance and inference cost. Existing approaches typically rely either on trained routers, which tie routing decisions to a fixed task and model pool, or on raw-confidence cascades, whose thresholds lack consistent reliability semantics across heterogeneous models. Consequently, these approaches adapt poorly to changing model pools and deployment budgets. We propose Calibration-Aware Uncertainty Cascades (CAUC), a simple post-hoc framework that independently calibrates each model's confidence and selects deployment policies using validation data. The resulting calibrated confidence scores establish a common reliability scale for accepting an early prediction, invoking a stronger model, or selectively combining model outputs. This unified decision criterion decouples deployment policies from any particular model pool or operating budget. We further show theoretically that calibration gives confidence thresholds an explicit selective-risk interpretation, whereas uncalibrated scores offer no comparable reliability guarantee. Extensive experiments demonstrate that, across six language benchmarks, CAUC achieves an average relative accuracy improvement of 1.9% over strong-model-only inference while avoiding approximately 47% of strong-model calls. On image classification benchmarks, it maintains or improves predictive performance while reducing measured GFLOPs by up to 57%.
Yilin Zhang, Han Jiang, Cai Xu +2
School of Computer Science and Technology, Xidian University Xi’an, China