JUCAL: Jointly Calibrating Aleatoric and Epistemic Uncertainty in Classification Tasks

Jakob Heiss; Sören Lambrecht; Jakob Weissteiner; Hanna Wutte; Žan Žurič; Josef Teichmann; Bin Yu

stat.MLFeb 23, 2026

JUCAL: Jointly Calibrating Aleatoric and Epistemic Uncertainty in Classification Tasks

Authors: Jakob Heiss, Sören Lambrecht, Jakob Weissteiner, Hanna Wutte, Žan Žurič, Josef Teichmann, Bin Yu

Organizations: UC Berkeley · ETH Zurich · UBS Zurich · Kaiju Worldwide

Abstract

We study post-calibration uncertainty for trained ensembles of classifiers. Specifically, we consider both aleatoric (label noise) and epistemic (model) uncertainty. Among the most popular and widely used calibration methods in classification are temperature scaling (i.e., pool-then-calibrate) and conformal methods. However, the main shortcoming of these calibration methods is that they do not balance the proportion of aleatoric and epistemic uncertainty. Not balancing these uncertainties can severely misrepresent predictive uncertainty, leading to overconfident predictions in some input regions while being underconfident in others. To address this shortcoming, we present a simple but powerful calibration algorithm Joint Uncertainty Calibration (JUCAL) that jointly calibrates aleatoric and epistemic uncertainty. JUCAL jointly calibrates two constants to weight and scale epistemic and aleatoric uncertainties by optimizing the negative log-likelihood (NLL) on the validation/calibration dataset. JUCAL can be applied to any trained ensemble of classifiers (e.g., transformers, CNNs, or tree-based methods), with minimal computational overhead, without requiring access to the models' internal parameters. We experimentally evaluate JUCAL on various text classification tasks, for ensembles of varying sizes and with different ensembling strategies. Our experiments show that JUCAL significantly outperforms SOTA calibration methods across all considered classification tasks, reducing NLL and predictive set size by up to 15% and 20%, respectively. Interestingly, even applying JUCAL to an ensemble of size 5 can outperform temperature-scaled ensembles of size up to 50 in terms of NLL and predictive set size, resulting in up to 10 times smaller inference costs. Thus, we propose JUCAL as a new go-to method for calibrating ensembles in classification.

Figures & tables

Figure 1: Predictive probability estimation for a synthetic 2D binary classification task. (a) Softmax outputs from a single NN. (b) Deep Ensemble. (c) & (d) show the same ensemble as in (b) but with different calibration algorithms applied to it. In all cases, the uncertainty peaks near the decision boundary, but only JUCAL sufficiently accounts for epistemic uncertainty by widening the uncertain region (bright colors) as the distance to the training data increases. This reflects the model’s limited knowledge in data-sparse regions, highlighting the ensemble’s ability to distinguish between aleatory and epistemic components.

Figure 2: Scatter plots of ensemble members’ softmax outputs for (a) binary ( $K=2)$ and (b-e) ternary ( $K=3$ ) classification. Each subplot shows a different possibility of how the $M=50$ predictions could be arranged for a fixed input point $x$ . Each point represents a probability vector $p(y|x,\theta_{m})$ over $K$ classes estimated by an ensemble member. (a)&(b) low total predictive uncertainty; (c) very high aleatoric and low epistemic uncertainty; (d) low aleatoric and very high epistemic uncertainty; (d)&(e) high epistemic uncertainty. Theoretically (d) claims that the aleatoric uncertainty is certainly low, while (e) is uncertain about the aleatoric uncertainty, but in practice, both (d)&(e) should usually be simply interpreted as high epistemic uncertainty (see Remark A.1 ).

Figure 3: Binary classification example with $X\sim\mathcal{N}(0,1)$ . The ensemble logits strongly agree in the center of the distribution $x\in[-2,2]$ , but disagree more as one moves away from the center. The two subplots show the same ensemble before and after applying JUCAL to it.

Figure 4: Text Classification Results. For each of the six subplots, lower values of the metrics (displayed on the y-axis) are better. On the x-axis, we list 12 text classification datasets (a 10%-mini and a 100%-full version of 6 distinct datasets). The striped bars correspond to ensemble size

M=5

, while the non-striped bars correspond to

M=50

. JUCAL’s results are yellow. For all six metrics (defined in Section 5.1.1 ), we show the average and

\pm 1

standard deviation across 5 random validation-test splits. (a) NLL normalized by the mean of JUCAL Greedy-50 on the corresponding full dataset; (b)

\text{AORAC}=1-\text{AURAC}

; (c)

\text{AOROC}=1-\text{AUROC}

; (d) Average set size for the coverage threshold of 99.9% for DBpedia (Full and Mini) and 99% for all other datasets; (e) Brier Score ; (f)

\text{Misclassification Rate}=1-\text{Accuracy}

. For more detailed results, see the corresponding tables in Appendix F .

Figure 5: Image Classification Results. For each of the six subplots, lower values of the metrics (displayed on the y-axis) are better. On the x-axis, we list distinct image classification datasets (and two hyperparameter-ablation studies for MNIST). JUCAL’s results are yellow. For all six metrics (defined in Section 5.1.1 ), we show the average and

\pm 1

standard deviation across 10 random train-validation-test splits. (a) NLL normalized by the mean of JUCAL Greedy-5; (b)

\text{AORAC}=1-\text{AURAC}

; (c)

\text{AOROC}=1-\text{AUROC}

; (d) Average set size for the coverage threshold of 99% for CIFAR-10 , 90% for CIFAR-100 , and 99.9% for al variants of MNIST and Fashion-MNIST ; (e) Brier Score ; (f)

\text{Misclassification Rate}=1-\text{Accuracy}

Figure 6: Epistemic and Aleatoric Uncertainty (computed as in Section A.2.1 ) of JUCAL applied on Greedy-50 ensembles across six datasets in the metadataset. We compare the full (100%) and the mini (10%) metadataset configurations for both epistemic and aleatoric uncertainty. Bars indicate the mean uncertainty, and error bars denote one standard deviation over random seeds.

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Different possible behaviors of a higher-order distribution over the simplex $\triangle_{K-1}$ in a binary (a) and ternary (b-e) classification task. We both show the density of a higher-order distribution (such as a posterior distribution) via colors and $M=50$ samples from this distribution via semi-transparent black circles. Each point on the simplex $\triangle_{K-1}$ corresponds to a (first-order) distribution over the $K$ classes. Sub-figure (a)&(b) show almost no aleatoric or epistemic uncertainty (i.e., very low aleatoric and epistemic uncertainty, leading to a low total predictive uncertainty), (c) shows almost only aleatoric uncertainty , (d) shows almost only epistemic uncertainty and (e) shows both aleatoric and epistemic uncertainty . More precisely, (e) shows epistemic uncertainty on whether the aleatoric uncertainty is large or small, whereas (d) is theoretically more certain that the aleatoric uncertainty is large; (a), (b), and theoretically (d) are more certain that the aleatoric uncertainty is low. Note that (d)’s “certaitny” on the absence of aleatoric uncertainty, should not be trusted in typical settings as discussed in Remarks A.1 and A.4 . (c) is certain that the aleatoric uncertainty is high.

	More observations	Better prior	More covariates	Smaller noise
Epistemic	Decreases	Decreases	Increases (typically) / /	Decreases
Aleatoric	No effect	No effect	Decreases /	Decreases

Appendix

Table 1: Expected effects of different factors on epistemic and aleatoric uncertainty.

Figure 8: Estimated Epistemic Uncertainty for Figure 1

Figure 9: Estimated Aleatoric Uncertainty for Figure 1 . In regions of high epistemic uncertainty, one usually does not know if the aleatoric uncertainty is high or low; thus it has the possibility to be high, and averaging over all possibilities can result in quite high estimates of the aleatoric uncertainty.

Panel of Figure 10 — Figure 10: The same ensemble without and with JUCAL calibration. The logit diversity increases as you move further OOD, but the probability-diversity decreases without JUCAL.

Figure 11: Desired behavior of a higher-order distribution over the simplex in a ternary classification task. Sub-figure (a) almost no aleatoric or epistemic uncertainty , (b) shows almost only aleatoric uncertainty , (c) shows almost only epistemic uncertainty and (d) shows both aleatoric and epistemic uncertainty

Figure 12: Illustrating the point-wise predicted Dirichlet distributions in a 1D binary classification task with class probabilities defined by $p(y=1\mid x)=0.5+0.5\sin(x)$ , where $x\sim\mathcal{N}(0,1)$ (visualized as green line) and $y\sim\mathrm{Bernoulli}(p(y=1\mid x))$ . For each value $x\in[-15,15]$ , we visualize the density of the corresponding Dirichlet distribution over the interval $[0,1]$ , with black circles indicating the training data.

Panel of Figure 13 — Figure 13: The same ensemble without and with JUCAL calibration. The logit diversity increases as you move further OOD, but the probability-diversity can simultaneously decrease if the logit diversity does not grow fast enough. JUCAL can scale the logit-diversity via $c_{2}$ to prevent this.

Figure 14: For an ensemble consisting of 5 CNNs trained on CIFAR-10, we compute multiple quantities on the training, validation, and test datasets for multiple different values of $c_{1}$ and $c_{2}$ . We used Equations 7 and 6 from Section A.2.1 to compute aleatoric, epistemic, and total uncertainty, while there would be other alternatives too (see Section A.2 ).

Ensemble Type	DBpedia	News	SST-2	SetFit	Tweet	IMDB
G5	0.0376 ± 0.0005	0.1682 ± 0.0048	0.1359 ± 0.0051	0.5465 ± 0.0033	0.5095 ± 0.0089	0.1171 ± 0.0028
G5 p.t.c.	0.0348 ± 0.0007	0.1618 ± 0.0052	0.1208 ± 0.0040	0.5431 ± 0.0019	0.5012 ± 0.0052	0.1018 ± 0.0022
G5 JUCAL	0.0290 ± 0.0004	0.1479 ± 0.0023	0.1143 ± 0.0032	0.4965 ± 0.0013	0.4772 ± 0.0028	0.1005 ± 0.0018
G50	0.0349 ± 0.0005	0.1541 ± 0.0043	0.1137 ± 0.0039	0.531 ± 0.0016	0.4763 ± 0.0052	0.1050 ± 0.0026
G50 p.t.c.	0.0331 ± 0.0003	0.1510 ± 0.0037	0.1130 ± 0.0035	0.5309 ± 0.0016	0.4758 ± 0.0049	0.1042 ± 0.0019
G50 JUCAL	0.0288 ± 0.0004	0.1423 ± 0.0024	0.1090 ± 0.0032	0.4972 ± 0.0018	0.4680 ± 0.0045	0.0983 ± 0.0017

Appendix

Table 2: FTC-metadataset full: Negative log-likelihood (

\text{NLL}_{\text{mean}}

over data splits; mean ± 95% confidence interval half-width) on the full dataset (100%). The best mean is shown in bold, and methods not significantly different from the best (paired test,

\alpha=0.05

) are shaded.

Ensemble Type	DBpedia	News	SST-2	SetFit	Tweet	IMDB
G5	0.9895 ± 0.0	0.981 ± 0.0011	0.984 ± 0.0005	0.8915 ± 0.0008	0.9103 ± 0.0028	0.9859 ± 0.0002
G5 p.t.c.	0.9895 ± 0.0	0.981 ± 0.0011	0.984 ± 0.0005	0.8915 ± 0.0008	0.9103 ± 0.0027	0.9859 ± 0.0002
G5 JUCAL	0.9897 ± 0.0	0.9829 ± 0.0005	0.9842 ± 0.0005	0.924 ± 0.0006	0.9211 ± 0.0006	0.9858 ± 0.0002
G50	0.9895 ± 0.0	0.981 ± 0.0008	0.9833 ± 0.0005	0.9023 ± 0.0006	0.9157 ± 0.0021	0.9838 ± 0.0003
G50 p.t.c.	0.9895 ± 0.0	0.981 ± 0.0008	0.9833 ± 0.0005	0.9023 ± 0.0006	0.9158 ± 0.0021	0.9838 ± 0.0003
G50 JUCAL	0.9897 ± 0.0	0.9835 ± 0.0005	0.9849 ± 0.0005	0.9237 ± 0.0007	0.9236 ± 0.0014	0.9855 ± 0.0002

Appendix

Table 3: FTC-metadataset full: Area Under the Rejection-Accuracy Curve (AURAC) over data splits; mean ± 95% confidence interval half-width) on the full dataset (100%). The best mean is shown in bold, and methods not significantly different from the best (paired test,

\alpha=0.05

) are shaded.

Ensemble Type	DBpedia	News	SST-2	SetFit	Tweet	IMDB
G5	0.9998312 ± 0.0	0.9929 ± 0.0007	0.9907 ± 0.0007	0.9144 ± 0.0008	0.9316 ± 0.0019	0.9934 ± 0.0003
G5 p.t.c.	0.9998311 ± 0.0	0.9929 ± 0.0007	0.9907 ± 0.0007	0.9144 ± 0.0008	0.9316 ± 0.0018	0.9934 ± 0.0003
G5 JUCAL	0.9998758 ± 0.0	0.9943 ± 0.0003	0.9912 ± 0.0006	0.9377 ± 0.0004	0.9383 ± 0.0010	0.9934 ± 0.0002
G50	0.9998198 ± 0.0	0.9931 ± 0.0005	0.9898 ± 0.0007	0.9229 ± 0.0005	0.9369 ± 0.0014	0.9911 ± 0.0003
G50 p.t.c.	0.9998199 ± 0.0	0.9931 ± 0.0005	0.9898 ± 0.0007	0.9229 ± 0.0005	0.9369 ± 0.0014	0.9911 ± 0.0003
G50 JUCAL	0.9998785 ± 0.0	0.9948 ± 0.0004	0.9917 ± 0.0007	0.9371 ± 0.0006	0.9405 ± 0.0014	0.9930 ± 0.0004

Appendix

Table 4: FTC-metadataset full: Area under the ROC ( AUROC over data splits; mean ± 95% confidence interval half-width) on the full dataset (100%). The best mean is shown in bold, and methods not significantly different from the best (paired test,

\alpha=0.05

) are shaded.

Ensemble Type	DBpedia	News	SST-2	SetFit	Tweet	IMDB
G5	1.2941 ± 0.0395	1.3517 ± 0.0385	1.1544 ± 0.0097	2.6642 ± 0.0228	2.3281 ± 0.0963	1.0996 ± 0.0065
G5 p.t.c.	1.3008 ± 0.0484	1.3591 ± 0.0424	1.1550 ± 0.0107	2.6567 ± 0.0209	2.3313 ± 0.0993	1.1003 ± 0.0062
G5 JUCAL	1.2270 ± 0.0438	1.2490 ± 0.0161	1.1459 ± 0.0116	2.2368 ± 0.0231	2.1286 ± 0.0722	1.1004 ± 0.0039
G50	1.3516 ± 0.0428	1.3436 ± 0.0313	1.1617 ± 0.0148	2.6519 ± 0.0237	2.2280 ± 0.0507	1.1135 ± 0.0070
G50 p.t.c.	1.3534 ± 0.0398	1.3517 ± 0.0226	1.1621 ± 0.0175	2.6514 ± 0.0490	2.2261 ± 0.0476	1.1140 ± 0.0092
G50 JUCAL	1.2072 ± 0.0358	1.2228 ± 0.0244	1.1385 ± 0.0094	2.2334 ± 0.0199	2.0633 ± 0.0291	1.1005 ± 0.0051

Appendix

Table 5: FTC-metadataset full: Set size over data splits; mean ± 95% confidence interval half-width) on the full dataset (100%). The best mean is shown in bold, and methods not significantly different from the best (paired test,

\alpha=0.05

) are shaded. Here the coverage threshold is 99% for all but DBpedia where it is 99.9%

Ensemble Type	DBpedia	News	SST-2	SetFit	Tweet	IMDB
G5	0.0432 ± 0.0012	0.2321 ± 0.0031	0.1534 ± 0.0044	0.4067 ± 0.002	0.5311 ± 0.0065	0.1334 ± 0.0064
G5 p.t.c.	0.0341 ± 0.0008	0.2050 ± 0.0026	0.1472 ± 0.0020	0.4051 ± 0.0018	0.5294 ± 0.0062	0.1314 ± 0.0043
G5 JUCAL	0.0326 ± 0.0008	0.1966 ± 0.0026	0.1396 ± 0.002	0.3684 ± 0.0018	0.5205 ± 0.0059	0.1303 ± 0.0034
G50	0.0352 ± 0.0009	0.1967 ± 0.0032	0.1320 ± 0.0035	0.3594 ± 0.0014	0.4980 ± 0.0063	0.1258 ± 0.0020
G50 p.t.c.	0.0346 ± 0.0004	0.1964 ± 0.0031	0.1320 ± 0.0034	0.3594 ± 0.0014	0.4979 ± 0.0061	0.1255 ± 0.0014
G50 JUCAL	0.0305 ± 0.0008	0.1899 ± 0.0028	0.1309 ± 0.0034	0.3480 ± 0.0013	0.4979 ± 0.0059	0.1257 ± 0.0018

Appendix

Table 6: FTC-metadataset mini (10%): Negative log-likelihood (

\text{NLL}_{\text{mean}}

over data splits; mean ± 95% confidence interval half-width) on the full dataset (100%). The best mean is shown in bold, and methods not significantly different from the best (paired test,

\alpha=0.05

) are shaded.

Ensemble Type	DBpedia	News	SST-2	SetFit	Tweet	IMDB
G5	0.9895 ± 0.0001	0.9769 ± 0.0002	0.979 ± 0.0008	0.9406 ± 0.0005	0.8982 ± 0.0026	0.9809 ± 0.0006
G5 p.t.c.	0.9895 ± 0.0001	0.9769 ± 0.0003	0.979 ± 0.0008	0.9407 ± 0.0005	0.8981 ± 0.0026	0.9809 ± 0.0006
G5 JUCAL	0.9895 ± 0.0001	0.9779 ± 0.0003	0.9817 ± 0.0004	0.95 ± 0.0005	0.9025 ± 0.0018	0.9819 ± 0.0005
G50	0.9893 ± 0.0001	0.9748 ± 0.0005	0.9822 ± 0.0005	0.9503 ± 0.0003	0.9091 ± 0.0018	0.9821 ± 0.0004
G50 p.t.c.	0.9893 ± 0.0001	0.9748 ± 0.0005	0.9822 ± 0.0005	0.9503 ± 0.0003	0.9091 ± 0.0018	0.9821 ± 0.0004
G50 JUCAL	0.9896 ± 0.0	0.978 ± 0.0005	0.9828 ± 0.0005	0.9554 ± 0.0002	0.9099 ± 0.0017	0.9822 ± 0.0005

Appendix

Table 7: FTC-metadataset mini (10%): Area Under the Rejection-Accuracy Curve (AURAC) over data splits; mean ± 95% confidence interval half-width) on the full dataset (100%). The best mean is shown in bold, and methods not significantly different from the best (paired test,

\alpha=0.05

) are shaded.

Ensemble Type	DBpedia	News	SST-2	SetFit	Tweet	IMDB
G5	0.9998 ± 0.0	0.9899 ± 0.0002	0.9853 ± 0.0007	0.9539 ± 0.0004	0.9226 ± 0.0015	0.9872 ± 0.0007
G5 p.t.c.	0.9998 ± 0.0	0.9899 ± 0.0001	0.9853 ± 0.0007	0.9539 ± 0.0004	0.9225 ± 0.0015	0.9872 ± 0.0007
G5 JUCAL	0.9998 ± 0.0	0.9905 ± 0.0002	0.9874 ± 0.0005	0.9620 ± 0.0004	0.9253 ± 0.0019	0.9883 ± 0.0006
G50	0.9997 ± 0.0	0.9889 ± 0.0005	0.9878 ± 0.0008	0.9632 ± 0.0002	0.9302 ± 0.0013	0.9886 ± 0.0001
G50 p.t.c.	0.9997 ± 0.0	0.9889 ± 0.0005	0.9878 ± 0.0008	0.9632 ± 0.0002	0.9302 ± 0.0013	0.9886 ± 0.0001
G50 JUCAL	0.9998 ± 0.0	0.9907 ± 0.0003	0.9885 ± 0.0008	0.9667 ± 0.0001	0.9306 ± 0.0012	0.9886 ± 0.0002

Appendix

Table 8: FTC-metadataset mini (10%): Are under the ROC ( AUROC over data splits; mean ± 95% confidence interval half-width) on the full dataset (100%). The best mean is shown in bold, and methods not significantly different from the best (paired test,

\alpha=0.05

) are shaded.

Ensemble Type	DBpedia	News	SST-2	SetFit	Tweet	IMDB
G5	1.3673 ± 0.0702	1.4414 ± 0.0167	1.2467 ± 0.0135	2.2989 ± 0.0027	2.2997 ± 0.0356	1.2392 ± 0.0099
G5 p.t.c.	1.4313 ± 0.0695	1.4475 ± 0.0184	1.2504 ± 0.0149	2.3124 ± 0.0136	2.3028 ± 0.0366	1.2347 ± 0.0158
G5 JUCAL	1.4522 ± 0.0567	1.4131 ± 0.0276	1.2091 ± 0.0115	1.9976 ± 0.0254	2.2110 ± 0.0277	1.2148 ± 0.0115
G50	1.6459 ± 0.0546	1.7193 ± 0.0952	1.1918 ± 0.0132	2.1899 ± 0.0061	2.1735 ± 0.0475	1.1821 ± 0.0043
G50 p.t.c.	1.6453 ± 0.0563	1.7274 ± 0.0792	1.1933 ± 0.0119	2.2008 ± 0.0059	2.1684 ± 0.0414	1.1831 ± 0.0092
G50 JUCAL	1.3105 ± 0.0232	1.4389 ± 0.0334	1.1862 ± 0.0120	1.8980 ± 0.0086	2.1470 ± 0.0444	1.1819 ± 0.0126

Appendix

Table 9: FTC-metadataset mini (10%): Set size over data splits; mean ± 95% confidence interval half-width) on the full dataset (100%). The best mean is shown in bold, and methods not significantly different from the best (paired test,

\alpha=0.05

) are shaded. Here the coverage threshold is 99% for all but DBpedia where it is 99.9%

Figure 15: ECE Results for Text Classification. For the ECE, lower values (displayed on the y-axis) are better. On the x-axis, we list 12 text classification datasets (a 10%-mini and a 100%-full version of 6 distinct datasets). The striped bars correspond to ensemble size

M=5

, while the non-striped bars correspond to

M=50

. JUCAL’s results are yellow. We show the average ECE and

\pm 1

standard deviation across 5 random validation-test splits.

Figure 16: ECE Results for Image Classification. For the ECE, lower values (displayed on the y-axis) are better. On the x-axis, we list distinct image classification datasets (and two hyperparameter-ablation studies for MNIST). JUCAL’s results are yellow. We show the average ECE and

\pm 1

standard deviation across 10 random train-validation-test splits.

Figure 17: Conformal Set Size Results for Text Classification. For the conformal set size, lower values (displayed on the y-axis) are better. On the x-axis, we list 12 text classification datasets (a 10%-mini and a 100%-full version of 6 distinct datasets). The striped bars correspond to ensemble size

M=5

, while the non-striped bars correspond to

M=50

. JUCAL’s results are yellow. We show the average conformal prediction set size (for the conformal target coverage threshold of 99.9% for DBpedia (Full and Mini) and 99% for all other datasets) and

\pm 1

standard deviation across 5 random validation-test splits.

Figure 18: Conformal Set Size Results for Image Classification. For the conformal set size, lower values (displayed on the y-axis) are better. On the x-axis, we list distinct image classification datasets (and two hyperparameter-ablation studies for MNIST). JUCAL’s results are yellow. We show the average conformal prediction set size (for the conformal target coverage threshold of 99% for CIFAR-10 , 90% for CIFAR-100 , and 99.9% for al variants of MNIST and Fashion-MNIST ) and

\pm 1

standard deviation across 10 random train-validation-test splits.

Figure 19: Conformal Log Set Size Results for Text Classification. For the conformal log set size, lower values (displayed on the y-axis) are better. On the x-axis, we list 12 text classification datasets (a 10%-mini and a 100%-full version of 6 distinct datasets). The striped bars correspond to ensemble size

M=5

, while the non-striped bars correspond to

M=50

. JUCAL’s results are yellow. We show the average of the logarithm of the conformal prediction set size (for the conformal target coverage threshold of 99.9% for DBpedia (Full and Mini) and 99% for all other datasets) and

\pm 1

standard deviation across 5 random validation-test splits.

Figure 20: Conformal Log Set Size Results for Image Classification. For the conformal log set size, lower values (displayed on the y-axis) are better. On the x-axis, we list distinct image classification datasets (and two hyperparameter-ablation studies for MNIST). JUCAL’s results are yellow. We show the average logarithmic conformal prediction set size (for the conformal target coverage threshold of 99% for CIFAR-10 , 90% for CIFAR-100 , and 99.9% for al variants of MNIST and Fashion-MNIST ) and

\pm 1

standard deviation across 10 random train-validation-test splits.

Figure 21: Conformal Coverage Results for Text Classification. For the conformal coverage, values near the target coverage indicate better calibration. Larger values of coverage are more desirable than smaller values of coverage (unless larger coverage leads to larger set sizes). On the x-axis, we list 12 text classification datasets (a 10%-mini and a 100%-full version of 6 distinct datasets). The striped bars correspond to ensemble size

M=5

, while the non-striped bars correspond to

M=50

. JUCAL’s results are yellow. We show the average test-coverage (for the conformal target coverage threshold of 99.9% for DBpedia (Full and Mini) and 99% for all other datasets), and

\pm 1

standard deviation across 5 random validation-test splits.

Figure 22: Conformal Coverage Results for Image Classification. For the conformal coverage, values near the target coverage indicate better calibration. Larger values of coverage are more desirable than smaller values of coverage (unless larger coverage leads to larger set sizes). On the x-axis, we list distinct image classification datasets (and two hyperparameter-ablation studies for MNIST). JUCAL’s results are yellow. We show the average test-coverage (for the conformal target coverage threshold of 99% for CIFAR-10 , 90% for CIFAR-100 , and 99.9% for al variants of MNIST and Fashion-MNIST ) and

\pm 1

standard deviation across 10 random train-validation-test splits.

Figure 23: Softmax outputs visualizing the estimated predictive probabilities calibrated by JUCAL for a synthetic 2D binary classification task.

Figure 24: Softmax outputs visualizing the estimated predictive probabilities from a single neural network trained on two dataset configurations of ta synthetic 2D binary classification task.

Dataset	Classes	Members	Train Size	Valid Size	Test Size
DBpedia Full	14	25	448,000	112,000	70,000
DBpedia Mini	14	65	44,800	112,000	70,000
News Full	4	99	96,000	24,000	7,600
News Mini	4	120	9,600	24,000	7,600
SST-2 Full	2	125	43,103	13,470	10,776
SST-2 Mini	2	125	4,310	13,470	10,776

Appendix

Table 10: Summary of the underlying datasets from which the FTC-metadataset is constructed by [ 5 ] .

Dataset	Ensemble Method	Ensemble Selection Time (s)	Calibration Time (s)
DBpedia Full	JUCAL Greedy-50	17.6798 $\pm$ 0.5566	680.2392 $\pm$ 9.9481
DBpedia Full	JUCAL Greedy-5	0.6779 $\pm$ 0.5347	92.5821 $\pm$ 7.4349
DBpedia Mini	JUCAL Greedy-50	51.0481 $\pm$ 5.3242	764.0273 $\pm$ 26.3293
DBpedia Mini	JUCAL Greedy-5	0.8412 $\pm$ 0.0215	99.8790 $\pm$ 12.2445
News Full	JUCAL Greedy-50	8.8411 $\pm$ 0.2699	78.3229 $\pm$ 0.8914
News Full	JUCAL Greedy-5	0.6653 $\pm$ 0.5407	11.2228 $\pm$ 0.3444

Appendix

Table 11: Ensemble selection and calibration time (mean

\pm

std in seconds) for JUCAL on Greedy-50 and Greedy-5 across all datasets (Full vs Mini).

Uncertainty Calibration

Uncertainty calibration aligns model confidence and predictive intervals with observed correctness and outcome frequencies.

See all 264 papers

Can we trust our models? Epistemic calibration in second-order classification
Jun 9, 2026Arthur HoarauUncertainty Quantification Uncertainty Calibration
Université de Lorraine, CentraleSupélec · Loria, CNRS, Metz, France
Do Not Train Away Uncertainty: Early Uncertainty Anchored Calibration
Oct 8, 2026Yutong Xie, Jiawei Tang, Zhenglin Hua +6Model Calibration Uncertainty Calibration
School of Software Engineering, Southeast University, Nanjing, China · School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Computing Information Sciences, Saint Francis University, Hong Kong, China +2
Calibration-Aware Uncertainty Cascades for Efficient Heterogeneous Model Collaboration
Sep 10, 2026Yilin Zhang, Han Jiang, Cai Xu +2Selective Prediction Cost-Aware Inference
School of Computer Science and Technology, Xidian University Xi’an, China

Jun 9, 2026J/K move · Enter open · S save

Can we trust our models? Epistemic calibration in second-order classification

Arthur Hoarau

Université de Lorraine, CentraleSupélec · Loria, CNRS, Metz, France

Uncertainty Quantification Uncertainty Calibration Epistemic Uncertainty

Open paper AskPDF arXiv

Uncertainty estimation is critical for deploying machine learning models in high-stakes settings. However, classical calibration only assesses the reliability of predicted probabilities and does not evaluate whether epistemic uncertainty estimates are themselves trustworthy. This limitation is particularly relevant for second-order classification models. We introduce epistemic calibration, a principled criterion that measures whether reported epistemic uncertainty faithfully reflects the dispersion of model predictions around the ground truth. We show that epistemic calibration is a strictly stronger notion than classical calibration and captures failure modes invisible to standard metrics. We relate this work to the existing literature through an impossibility theorem that holds under the epistemic calibration hypothesis. To operationalize this concept, we propose the Expected Epistemic Calibration Error (EECE), which we prove to be a consistent estimator of a True Epistemic Calibration Error (TECE). Experiments across a broad range of uncertainty quantification methods show that epistemic calibration is a coherent and meaningful criterion and reveal substantial differences across methods, despite similar predictive performance.

Abstract

Uncertainty Calibration

Explore similar work

Can we trust our models? Epistemic calibration in second-order classification

Do Not Train Away Uncertainty: Early Uncertainty Anchored Calibration

Calibration-Aware Uncertainty Cascades for Efficient Heterogeneous Model Collaboration