Epistemic Learning from Imprecise Annotation

Kaizheng Wang; Siu Lun Chau

cs.LGSep 28, 2026

Epistemic Learning from Imprecise Annotation

Authors: Kaizheng Wang, Siu Lun Chau

Organizations: Epistemic Intelligence & Computation Lab, College of Computing & Data Science, Nanyang Technological University, Singapore

Abstract

Imprecise annotations may support several plausible labelling distributions, yet learning methods often resolve this ambiguity into a single predictive distribution. This can obscure what the annotation evidence leaves unresolved. We introduce epistemic learning from credal supervision, a framework that uses convex sets of plausible labelling distributions, called credal sets, as supervision and learns sets of predictive distributions. We instantiate the framework with the pessimistic--optimistic credal classifier (POCC), which combines a shared backbone with two classification heads trained to minimise worst-case and best-case losses over the supervision sets. Their outputs define a predictive credal set whose spread provides an uncertainty score. We also show how credal labels can be obtained through a simple relaxation of existing probabilistic labels, reducing commitment to their precise probability assignments. This construction admits closed-form inner optimisation under cross-entropy loss, enabling efficient training. Assuming the supervision sets contain the true conditional label distributions, and other regularity assumptions, we establish a finite-sample generalisation bound for the averaged predictor with an explicit penalty for supervision imprecision. We evaluate POCC using human annotator disagreement and teacher predictions, alongside label smoothing as a controlled proxy for annotation imprecision. Across these settings, POCC achieves a favourable balance of predictive accuracy, calibration, and uncertainty-based selective classification versus competitive baselines.

Figures & tables

Figure 1 : Overview of epistemic learning from credal supervision. Top: the framework relates credal supervision to sets of predictive distributions. Bottom left: practical construction of credal labels from imprecise annotations. Bottom right: POCC uses a shared backbone and pessimistic and optimistic heads to form a predictive credal set.

Method	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$
EDL	$85.0\scriptstyle{\pm 0.4}$	$9.5\scriptstyle{\pm 0.3}$	$96.7\scriptstyle{\pm 0.1}$	$33.1\scriptstyle{\pm 8.3}$
LbBnn	$\mathbf{87.0\scriptstyle{\pm 0.8}}$	$6.6\scriptstyle{\pm 0.3}$	$95.9\scriptstyle{\pm 0.3}$	$54.8\scriptstyle{\pm 6.3}$
Decali	$84.3\scriptstyle{\pm 1.0}$	$\underline{1.0\scriptstyle{\pm 0.2}}$	$94.9\scriptstyle{\pm 0.5}$	$35.4\scriptstyle{\pm 4.5}$
DAPPr	$86.6\scriptstyle{\pm 0.3}$	$6.9\scriptstyle{\pm 0.2}$	$\mathbf{97.2\scriptstyle{\pm 0.1}}$	$68.3\scriptstyle{\pm 5.9}$
Ours	$\mathbf{87.0\scriptstyle{\pm 0.4}}$	$2.9\scriptstyle{\pm 0.3}$	$\underline{97.1\scriptstyle{\pm 0.2}}$	$\mathbf{86.0\scriptstyle{\pm 6.1}}$
MCDO	$85.6\scriptstyle{\pm 0.7}$	$\mathbf{0.9\scriptstyle{\pm 0.1}}$	$96.6\scriptstyle{\pm 0.3}$	$71.0\scriptstyle{\pm 10.6}$

Table 1 : CIFAR-10H results (%). Best in bold; second best underlined.

Method	CIFAR-10				CIFAR-100
Method	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$
EDL	$93.4\scriptstyle{\pm 0.2}$	$9.0\scriptstyle{\pm 0.2}$	$99.1\scriptstyle{\pm 0.1}$	$51.6\scriptstyle{\pm 3.8}$	$33.1\scriptstyle{\pm 5.7}$	$\underline{23.3\scriptstyle{\pm 3.9}}$	$65.8\scriptstyle{\pm 6.1}$	$30.2\scriptstyle{\pm 2.9}$
LbBnn	$\underline{95.1\scriptstyle{\pm 0.1}}$	$28.1\scriptstyle{\pm 0.2}$	$\underline{99.2\scriptstyle{\pm 0.0}}$	$54.2\scriptstyle{\pm 3.3}$	$\underline{73.1\scriptstyle{\pm 0.2}}$	$63.1\scriptstyle{\pm 0.2}$	$63.4\scriptstyle{\pm 0.4}$	$31.9\scriptstyle{\pm 2.4}$
Decali	$93.5\scriptstyle{\pm 0.1}$	$8.5\scriptstyle{\pm 0.1}$	$98.0\scriptstyle{\pm 0.1}$	$29.7\scriptstyle{\pm 2.5}$	$71.0\scriptstyle{\pm 0.2}$	$27.8\scriptstyle{\pm 0.2}$	$73.3\scriptstyle{\pm 0.3}$	$64.2\scriptstyle{\pm 1.8}$
DAPPr	$93.4\scriptstyle{\pm 0.2}$	$14.6\scriptstyle{\pm 0.2}$	$99.0\scriptstyle{\pm 0.0}$	$42.8\scriptstyle{\pm 3.1}$	$72.5\scriptstyle{\pm 0.4}$	$52.8\scriptstyle{\pm 0.3}$	$78.3\scriptstyle{\pm 0.3}$	$55.1\scriptstyle{\pm 1.6}$
Ours	$\mathbf{95.5\scriptstyle{\pm 0.1}}$	$\mathbf{3.0\scriptstyle{\pm 0.1}}$	$\mathbf{99.5\scriptstyle{\pm 0.0}}$	$\mathbf{99.9\scriptstyle{\pm 0.3}}$	$\mathbf{77.2\scriptstyle{\pm 0.2}}$	$\mathbf{10.6\scriptstyle{\pm 0.4}}$	$\mathbf{90.6\scriptstyle{\pm 0.2}}$	$\mathbf{100.0\scriptstyle{\pm 0.0}}$

Table 2 : Teacher-prediction results (%) with ResNet-20. Best in bold; second best underlined.

Method	CIFAR-10				CIFAR-100
Method	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$
EDL	$93.4\scriptstyle{\pm 0.3}$	$9.1\scriptstyle{\pm 0.2}$	$99.1\scriptstyle{\pm 0.1}$	$48.6\scriptstyle{\pm 4.3}$	$26.3\scriptstyle{\pm 1.2}$	$18.7\scriptstyle{\pm 1.0}$	$58.5\scriptstyle{\pm 1.6}$	$24.3\scriptstyle{\pm 0.6}$
LbBnn	$\underline{95.4\scriptstyle{\pm 0.2}}$	$29.9\scriptstyle{\pm 0.3}$	$99.1\scriptstyle{\pm 0.1}$	$53.6\scriptstyle{\pm 2.6}$	$\underline{79.3\scriptstyle{\pm 0.3}}$	$61.6\scriptstyle{\pm 0.5}$	$74.5\scriptstyle{\pm 0.9}$	$47.7\scriptstyle{\pm 1.1}$
Decali	$93.9\scriptstyle{\pm 0.3}$	$3.6\scriptstyle{\pm 0.1}$	$98.3\scriptstyle{\pm 0.2}$	$39.0\scriptstyle{\pm 5.2}$	$76.2\scriptstyle{\pm 0.3}$	$\underline{3.5\scriptstyle{\pm 0.4}}$	$90.7\scriptstyle{\pm 0.3}$	$94.0\scriptstyle{\pm 0.7}$
DAPPr	$93.9\scriptstyle{\pm 0.3}$	$8.9\scriptstyle{\pm 0.2}$	$99.1\scriptstyle{\pm 0.1}$	$58.5\scriptstyle{\pm 6.4}$	$75.1\scriptstyle{\pm 0.4}$	$35.3\scriptstyle{\pm 0.4}$	$91.9\scriptstyle{\pm 0.2}$	$76.5\scriptstyle{\pm 0.4}$
Ours	$\mathbf{95.5\scriptstyle{\pm 0.1}}$	$\mathbf{1.5\scriptstyle{\pm 0.1}}$	$\underline{99.2\scriptstyle{\pm 0.1}}$	$\mathbf{90.6\scriptstyle{\pm 4.7}}$	$\underline{79.3\scriptstyle{\pm 0.2}}$	$\mathbf{2.7\scriptstyle{\pm 0.2}}$	$\underline{93.4\scriptstyle{\pm 0.1}}$	$\mathbf{99.0\scriptstyle{\pm 0.3}}$

Table 3 : Label-smoothing results (%) at

\epsilon=0.05

over 10 runs. Best in bold; second best underlined.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	CIFAR-10H	MiceBone	TreeVersity
Backbone	ResNet-18	DenseNet-121	DenseNet-121
Classes	10	3	6
Input size	$(3\times 32\times 32)$	$(3\times 224\times 224)$	$(3\times 224\times 224)$
Batch size	128	128	128
Epochs	150	60	60
Learning rate	0.1	0.00001	0.00001

Appendix

Table A1 : Hyperparameters used for annotator-disagreement experiments.

Hyperparameter	CIFAR-10	CIFAR-100
Backbone	ResNet-18	WRN-28-4
Classes	10	100
Input size	$(3\times 32\times 32)$	$(3\times 32\times 32)$
Batch size	128	128
Epochs	200	200
Learning rate	0.1	0.1

Appendix

Table A2 : Hyperparameters used for both teacher-prediction and label-smoothing experiments.

Component	Specification
CPU	Intel Core Ultra 24C/24T, 5.8GHz Max, 36MB L3 Cache
GPU	$1\times$ NVIDIA GeForce RTX 5090 (32 GB GDDR7)
RAM	192 GB DDR4 ECC DIMM

Appendix

Table A3: Specifications of Computing Resources

	MiceBone Dataset				TreeVersity Dataset
Method	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$
EDL	$88.5\scriptstyle{\pm 0.8}$	$9.9\scriptstyle{\pm 0.8}$	$\mathbf{97.4\scriptstyle{\pm 0.3}}$	$\underline{68.7\scriptstyle{\pm 3.6}}$	$85.5\scriptstyle{\pm 0.6}$	$17.1\scriptstyle{\pm 0.7}$	$\underline{95.7\scriptstyle{\pm 0.3}}$	$58.4\scriptstyle{\pm 1.9}$
Decali	$88.7\scriptstyle{\pm 1.2}$	$\underline{5.8\scriptstyle{\pm 1.0}}$	$93.6\scriptstyle{\pm 0.9}$	$51.2\scriptstyle{\pm 6.0}$	$\mathbf{86.9\scriptstyle{\pm 0.4}}$	$\mathbf{2.3\scriptstyle{\pm 0.3}}$	$95.0\scriptstyle{\pm 0.4}$	$\underline{92.4\scriptstyle{\pm 2.2}}$
DAPPr	$\underline{89.0\scriptstyle{\pm 1.1}}$	$13.2\scriptstyle{\pm 0.9}$	$96.4\scriptstyle{\pm 0.5}$	$53.6\scriptstyle{\pm 5.1}$	$\underline{86.6\scriptstyle{\pm 0.5}}$	$17.0\scriptstyle{\pm 0.4}$	$95.5\scriptstyle{\pm 0.3}$	$61.5\scriptstyle{\pm 1.5}$
Ours	$\mathbf{89.1\scriptstyle{\pm 0.9}}$	$\mathbf{2.7\scriptstyle{\pm 0.4}}$	$\underline{97.3\scriptstyle{\pm 0.3}}$	$\mathbf{96.1\scriptstyle{\pm 4.6}}$	$86.2\scriptstyle{\pm 0.6}$	$\underline{2.5\scriptstyle{\pm 0.5}}$	$\mathbf{96.7\scriptstyle{\pm 0.2}}$	$\mathbf{97.1\scriptstyle{\pm 1.9}}$
DE	$85.5\scriptstyle{\pm 0.9}$	$7.5\scriptstyle{\pm 0.8}$	$94.3\scriptstyle{\pm 0.5}$	$23.6\scriptstyle{\pm 5.5}$	$75.7\scriptstyle{\pm 1.1}$	$6.2\scriptstyle{\pm 0.9}$	$88.6\scriptstyle{\pm 0.8}$	$24.4\scriptstyle{\pm 1.7}$

Appendix

Table A4 : Performance comparison (in

\%

) under annotator-disagreement annotation imprecision across various datasets, averaged over 10 runs.

Method	CIFAR-10				CIFAR-100
Method	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$
Teacher: ResNet-20
EDL	$93.4\scriptstyle{\pm 0.2}$	$9.0\scriptstyle{\pm 0.2}$	$99.1\scriptstyle{\pm 0.1}$	$51.6\scriptstyle{\pm 3.8}$	$33.1\scriptstyle{\pm 5.7}$	$\underline{23.3\scriptstyle{\pm 3.9}}$	$65.8\scriptstyle{\pm 6.1}$	$30.2\scriptstyle{\pm 2.9}$
LbBnn	$\underline{95.1\scriptstyle{\pm 0.1}}$	$28.1\scriptstyle{\pm 0.2}$	$\underline{99.2\scriptstyle{\pm 0.0}}$	$54.2\scriptstyle{\pm 3.3}$	$\underline{73.1\scriptstyle{\pm 0.2}}$	$63.1\scriptstyle{\pm 0.2}$	$63.4\scriptstyle{\pm 0.4}$	$31.9\scriptstyle{\pm 2.4}$
Decali	$93.5\scriptstyle{\pm 0.1}$	$8.5\scriptstyle{\pm 0.1}$	$98.0\scriptstyle{\pm 0.1}$	$29.7\scriptstyle{\pm 2.5}$	$71.0\scriptstyle{\pm 0.2}$	$27.8\scriptstyle{\pm 0.2}$	$73.3\scriptstyle{\pm 0.2}$	$64.2\scriptstyle{\pm 1.9}$
DAPPr	$93.4\scriptstyle{\pm 0.2}$	$14.6\scriptstyle{\pm 0.2}$	$99.0\scriptstyle{\pm 0.0}$	$42.8\scriptstyle{\pm 3.1}$	$72.5\scriptstyle{\pm 0.4}$	$52.8\scriptstyle{\pm 0.3}$	$78.3\scriptstyle{\pm 0.3}$	$55.1\scriptstyle{\pm 1.6}$

Appendix

Table A5 : Performance comparison (in

\%

) under teacher-prediction annotation imprecision with varying settings, averaged over 10 runs.

	Method	CIFAR-10				CIFAR-100
	Method	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$
$\epsilon=0.05$	EDL	$93.4\scriptstyle{\pm 0.3}$	$9.1\scriptstyle{\pm 0.2}$	$99.1\scriptstyle{\pm 0.1}$	$48.6\scriptstyle{\pm 4.3}$	$26.3\scriptstyle{\pm 1.2}$	$18.7\scriptstyle{\pm 1.0}$	$58.5\scriptstyle{\pm 1.6}$	$24.3\scriptstyle{\pm 0.6}$
	LbBnn	$\underline{95.4\scriptstyle{\pm 0.2}}$	$29.9\scriptstyle{\pm 0.3}$	$99.1\scriptstyle{\pm 0.1}$	$53.6\scriptstyle{\pm 2.6}$	$\underline{79.3\scriptstyle{\pm 0.3}}$	$61.6\scriptstyle{\pm 0.5}$	$74.5\scriptstyle{\pm 0.9}$	$47.7\scriptstyle{\pm 1.1}$
	Decali	$93.9\scriptstyle{\pm 0.3}$	$3.6\scriptstyle{\pm 0.1}$	$98.3\scriptstyle{\pm 0.2}$	$39.0\scriptstyle{\pm 5.2}$	$76.2\scriptstyle{\pm 0.3}$	$\underline{3.5\scriptstyle{\pm 0.4}}$	$90.7\scriptstyle{\pm 0.3}$	$94.0\scriptstyle{\pm 0.7}$
	DAPPr	$93.9\scriptstyle{\pm 0.3}$	$8.9\scriptstyle{\pm 0.2}$	$99.1\scriptstyle{\pm 0.1}$	$58.5\scriptstyle{\pm 6.4}$	$75.1\scriptstyle{\pm 0.4}$	$35.3\scriptstyle{\pm 0.4}$	$91.9\scriptstyle{\pm 0.2}$	$76.5\scriptstyle{\pm 0.4}$
	Ours	$\mathbf{95.5\scriptstyle{\pm 0.1}}$	$\mathbf{1.5\scriptstyle{\pm 0.1}}$	$\underline{99.2\scriptstyle{\pm 0.1}}$	$\mathbf{90.6\scriptstyle{\pm 4.7}}$	$\underline{79.3\scriptstyle{\pm 0.2}}$	$\mathbf{2.7\scriptstyle{\pm 0.2}}$	$\underline{93.4\scriptstyle{\pm 0.1}}$	$\mathbf{99.0\scriptstyle{\pm 0.3}}$

Appendix

Table A6 : Label-smoothing results (%) across

\epsilon

. POCC achieves the highest mean BQS in seven of eight settings; the sole exception is

\epsilon=0.1

on CIFAR-10, where it ranks second behind DE (5 members), with the gap attributable to marginal differences in accuracy and AUARC. Statistical significance tests (Table A10 , Appendix E ) confirm that POCC ranks first statistically across all settings.

Method	CIFAR-10				CIFAR-100
Method	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$
Teacher: ResNet-20
EDL	$93.3\scriptstyle{\pm 0.2}$	$9.1\scriptstyle{\pm 0.3}$	$99.1\scriptstyle{\pm 0.1}$	$1.2\scriptstyle{\pm 2.4}$	$33.7\scriptstyle{\pm 5.0}$	$23.7\scriptstyle{\pm 3.4}$	$66.5\scriptstyle{\pm 5.5}$	$14.3\scriptstyle{\pm 3.1}$
LbBnn	$\underline{95.4\scriptstyle{\pm 0.1}}$	$1.5\scriptstyle{\pm 0.1}$	$\underline{99.5\scriptstyle{\pm 0.0}}$	$\underline{91.4\scriptstyle{\pm 2.7}}$	$\underline{77.3\scriptstyle{\pm 0.3}}$	$19.5\scriptstyle{\pm 0.5}$	$90.3\scriptstyle{\pm 0.2}$	$81.3\scriptstyle{\pm 0.7}$
Decali	$93.8\scriptstyle{\pm 0.2}$	$1.5\scriptstyle{\pm 0.1}$	$99.2\scriptstyle{\pm 0.0}$	$42.2\scriptstyle{\pm 8.0}$	$74.3\scriptstyle{\pm 0.4}$	$\mathbf{2.4\scriptstyle{\pm 0.3}}$	$89.1\scriptstyle{\pm 0.1}$	$92.9\scriptstyle{\pm 1.2}$
DAPPr	$94.0\scriptstyle{\pm 0.2}$	$7.0\scriptstyle{\pm 0.1}$	$99.2\scriptstyle{\pm 0.1}$	$27.2\scriptstyle{\pm 7.4}$	$75.4\scriptstyle{\pm 0.3}$	$39.7\scriptstyle{\pm 0.3}$	$90.1\scriptstyle{\pm 0.3}$	$61.6\scriptstyle{\pm 1.2}$

Appendix

Table A7 : Performance comparison (in

\%

) using the teachers’ original predictions (

T=1.0

) as the source of annotation imprecision, averaged over 10 runs. These results consistently confirm the robustness of POCC in achieving the highest BQS scores across settings.

Method

CIFAR-10H

MiceBone

TreeVersity

ACC

1-ECE

AUARC

BQS-ST

ACC

1-ECE

AUARC

BQS-ST

ACC

1-ECE

AUARC

BQS-ST

EDL

-5.0

-6.0

-1.0

15.2

-1.0

-2.0

\mathbf{3.0}

\underline{58.3}

-2.0

-3.0

\underline{1.0}

30.4

LbBnn

\underline{3.0}

-2.0

-4.0

47.8

–

Decali

-5.0

\mathbf{5.0}

-6.0

33.3

\underline{1.0}

\underline{2.0}

-4.0

52.8

\mathbf{3.0}

\mathbf{3.0}

-2.0

\underline{75.0}

DAPPr

1.0

-4.0

\mathbf{5.0}

61.6

\mathbf{2.0}

-4.0

\underline{0.0}

52.4

\mathbf{3.0}

-3.0

\underline{1.0}

54.2

Ours

\mathbf{4.0}

\underline{2.0}

\mathbf{5.0}

\mathbf{90.9}

\mathbf{2.0}

\mathbf{4.0}

\mathbf{3.0}

\mathbf{100.0}

\underline{0.0}

\mathbf{3.0}

\mathbf{4.0}

\mathbf{85.7}

Appendix

Table A8 : Performance comparison (in

\%

\uparrow

) under annotator disagreements.

	Method	CIFAR-10				CIFAR-100
	Method	ACC	1-ECE	AUARC	BQS-ST	ACC	1-ECE	AUARC	BQS-ST
ResNet-20	EDL	$-3.0$	$0.0$	$0.0$	$33.3$	$-6.0$	$\underline{4.0}$	$-5.0$	$27.8$
	LbBnn	$\underline{4.0}$	$-6.0$	$\underline{4.0}$	$\underline{53.7}$	$\underline{4.0}$	$-6.0$	$-5.0$	$27.8$
	Decali	$-3.0$	$\underline{3.0}$	$-6.0$	$25.0$	$-3.0$	$2.0$	$-2.0$	$39.6$
	DAPPr	$-3.0$	$-4.0$	$-3.0$	$13.9$	$1.0$	$-4.0$	$0.0$	$40.2$
	Ours	$\mathbf{6.0}$	$\mathbf{6.0}$	$\mathbf{6.0}$	$\mathbf{100.0}$	$\mathbf{6.0}$	$\mathbf{6.0}$	$\mathbf{6.0}$	$\mathbf{100.0}$

Appendix

Table A9 : Performance comparison (in

\%

\uparrow

) under teacher-prediction annotation imprecision.

	Method	CIFAR-10				CIFAR-100
	Method	ACC	1-ECE	AUARC	BQS-ST	ACC	1-ECE	AUARC	BQS-ST
$\epsilon=0.05$	EDL	$-6.0$	$-4.0$	$-1.0$	$19.4$	$-6.0$	$-2.0$	$-6.0$	$11.1$
	LbBnn	$\mathbf{5.0}$	$-6.0$	$-2.0$	$44.4$	$\underline{3.0}$	$-6.0$	$-4.0$	$30.6$
	Decali	$-1.0$	$\underline{3.0}$	$-6.0$	$40.2$	$-1.0$	$\underline{3.0}$	$-2.0$	$50.0$
	DAPPr	$-2.0$	$-2.0$	$\underline{2.0}$	$45.5$	$-4.0$	$-4.0$	$1.0$	$30.6$
	Ours	$\mathbf{5.0}$	$\mathbf{6.0}$	$1.0$	$\mathbf{86.1}$	$\underline{3.0}$	$\mathbf{6.0}$	$\underline{4.0}$	$\mathbf{86.1}$

Appendix

Table A10 : Performance comparison (in

\%

\uparrow

) under label-smoothing annotation imprecision.

Figure A6 : Pairwise one-sided Wilcoxon signed-rank test results on various datasets under label-smoothing imprecision. Cell $(i,j)$ indicates whether row method $i$ significantly outperforms column method $j$ ; coloured cells show the scaled $p$ -value ( $\times 10^{3}$ ). Orange ( $\blacksquare$ ) mark comparisons involving our method; dark navy ( $\blacksquare$ ) mark all other significant pairs.

Method	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$	BQS-ST $\uparrow$
CIFAR-10H
EDL	$84.8\scriptstyle{\pm 0.2}$	$9.5\scriptstyle{\pm 0.5}$	$96.6\scriptstyle{\pm 0.1}$	$41.4\scriptstyle{\pm 9.2}$	$33.3$
LbBnn	$85.9\scriptstyle{\pm 0.5}$	$6.9\scriptstyle{\pm 0.4}$	$96.2\scriptstyle{\pm 0.3}$	$53.5\scriptstyle{\pm 8.9}$	$48.5$
Decali	$82.7\scriptstyle{\pm 0.9}$	$9.5\scriptstyle{\pm 0.5}$	$95.5\scriptstyle{\pm 0.4}$	$3.4\scriptstyle{\pm 5.1}$	$0.0$
DAPPr	$85.5\scriptstyle{\pm 0.4}$	$6.2\scriptstyle{\pm 0.4}$	$96.6\scriptstyle{\pm 0.2}$	$59.5\scriptstyle{\pm 7.5}$	$62.9$
Ours	$\mathbf{87.0\scriptstyle{\pm 0.4}}$	$\underline{2.9\scriptstyle{\pm 0.3}}$	$\mathbf{97.1\scriptstyle{\pm 0.2}}$	$\mathbf{95.7\scriptstyle{\pm 1.1}}$	$\mathbf{93.9}$

Appendix

Table A11 : Performance comparison (in

\%

) under annotator disagreement, where baseline methods are trained on collapsed hard labels derived from imprecise annotations.

	Method	ACC $\uparrow$	ECE $\downarrow$	Sp(EU,CE) $\uparrow$	BQS $\uparrow$	BQS-ST $\uparrow$
CIFAR-10H	EDL	$85.0\scriptstyle{\pm 0.4}$	$9.5\scriptstyle{\pm 0.3}$	$\mathbf{95.3\scriptstyle{\pm 0.2}}$	$40.8\scriptstyle{\pm 6.4}$	$33.3$
	LbBnn	$87.0\scriptstyle{\pm 0.8}$	$6.6\scriptstyle{\pm 0.3}$	$88.8\scriptstyle{\pm 0.7}$	$51.7\scriptstyle{\pm 4.4}$	$47.8$
	Decali	$84.3\scriptstyle{\pm 1.0}$	$\underline{1.0\scriptstyle{\pm 0.2}}$	$85.6\scriptstyle{\pm 0.9}$	$35.4\scriptstyle{\pm 4.5}$	$33.3$
	DAPPr	$86.6\scriptstyle{\pm 0.3}$	$6.9\scriptstyle{\pm 0.2}$	$\mathbf{95.3\scriptstyle{\pm 0.2}}$	$68.5\scriptstyle{\pm 5.6}$	$61.6$
	Ours	$\mathbf{87.0\scriptstyle{\pm 0.4}}$	$2.9\scriptstyle{\pm 0.3}$	$94.4\scriptstyle{\pm 0.5}$	$\mathbf{84.9\scriptstyle{\pm 5.7}}$	$\mathbf{81.8}$
	MCDO	$85.6\scriptstyle{\pm 0.7}$	$\mathbf{0.9\scriptstyle{\pm 0.1}}$	$90.4\scriptstyle{\pm 0.4}$	$63.8\scriptstyle{\pm 8.7}$	$56.6$

Appendix

Table A12 : Annotator-disagreement results (%). Best in bold; second best underlined.

Method	CIFAR-10					CIFAR-100
Method	ACC	ECE	Sp(EU,CE)	BQS	BQS-ST	ACC	ECE	Sp(EU,CE)	BQS	BQS-ST
Teacher: ResNet-20
EDL	$93.4\scriptstyle{\pm 0.2}$	$9.0\scriptstyle{\pm 0.2}$	$\mathbf{98.9\scriptstyle{\pm 0.1}}$	$\underline{60.7\scriptstyle{\pm 2.9}}$	$\underline{50.0}$	$33.1\scriptstyle{\pm 5.7}$	$\underline{23.3\scriptstyle{\pm 3.9}}$	$62.7\scriptstyle{\pm 5.6}$	$56.6\scriptstyle{\pm 1.0}$	$50.0$
LbBnn	$\underline{95.1\scriptstyle{\pm 0.1}}$	$28.1\scriptstyle{\pm 0.2}$	$96.0\scriptstyle{\pm 0.1}$	$51.3\scriptstyle{\pm 2.9}$	$44.1$	$\underline{73.1\scriptstyle{\pm 0.2}}$	$63.1\scriptstyle{\pm 0.2}$	$-43.4\scriptstyle{\pm 1.4}$	$30.2\scriptstyle{\pm 0.3}$	$27.8$
Decali	$93.5\scriptstyle{\pm 0.1}$	$8.5\scriptstyle{\pm 0.1}$	$88.7\scriptstyle{\pm 0.4}$	$29.7\scriptstyle{\pm 2.5}$	$25.0$	$71.0\scriptstyle{\pm 0.2}$	$27.8\scriptstyle{\pm 0.2}$	$6.1\scriptstyle{\pm 0.4}$	$65.7\scriptstyle{\pm 0.6}$	$36.1$
DAPPr	$93.4\scriptstyle{\pm 0.2}$	$14.6\scriptstyle{\pm 0.2}$	$97.6\scriptstyle{\pm 0.1}$	$49.8\scriptstyle{\pm 2.4}$	$29.8$	$72.5\scriptstyle{\pm 0.4}$	$52.8\scriptstyle{\pm 0.3}$	$42.8\scriptstyle{\pm 0.7}$	$61.7\scriptstyle{\pm 0.5}$	$36.1$

Appendix

Table A13 : Teacher-prediction results (%). Best in bold; second best underlined.

Method	CIFAR-10					CIFAR-100
Method	ACC	ECE	Sp(EU,CE)	BQS	BQS-ST	ACC	ECE	Sp(EU,CE)	BQS	BQS-ST
$\epsilon=0.05$
EDL	$93.4\scriptstyle{\pm 0.3}$	$9.1\scriptstyle{\pm 0.2}$	$\mathbf{99.0\scriptstyle{\pm 0.1}}$	$58.9\scriptstyle{\pm 1.9}$	$38.9$	$26.3\scriptstyle{\pm 1.2}$	$18.7\scriptstyle{\pm 1.0}$	$58.6\scriptstyle{\pm 1.8}$	$47.3\scriptstyle{\pm 0.6}$	$16.7$
LbBnn	$\underline{95.4\scriptstyle{\pm 0.2}}$	$29.9\scriptstyle{\pm 0.3}$	$64.8\scriptstyle{\pm 2.6}$	$40.3\scriptstyle{\pm 3.8}$	$45.5$	$\underline{79.3\scriptstyle{\pm 0.3}}$	$61.6\scriptstyle{\pm 0.5}$	$-5.8\scriptstyle{\pm 4.0}$	$32.7\scriptstyle{\pm 0.3}$	$25.0$
Decali	$93.9\scriptstyle{\pm 0.3}$	$3.6\scriptstyle{\pm 0.1}$	$52.3\scriptstyle{\pm 6.5}$	$39.5\scriptstyle{\pm 5.6}$	$40.2$	$76.2\scriptstyle{\pm 0.3}$	$\underline{3.5\scriptstyle{\pm 0.4}}$	$76.6\scriptstyle{\pm 0.9}$	$93.1\scriptstyle{\pm 0.7}$	$50.0$
DAPPr	$93.9\scriptstyle{\pm 0.3}$	$8.9\scriptstyle{\pm 0.2}$	$\underline{98.9\scriptstyle{\pm 0.1}}$	$67.1\scriptstyle{\pm 4.0}$	$56.6$	$75.1\scriptstyle{\pm 0.4}$	$35.3\scriptstyle{\pm 0.4}$	$\mathbf{87.6\scriptstyle{\pm 0.4}}$	$78.4\scriptstyle{\pm 0.2}$	$44.4$

Appendix

Table A14 : Label-smoothing results (%). Best in bold; second best underlined.

Metric	CIFAR-10H	MiceBone	TreeVersity
MMI	$97.1\scriptstyle{\pm 0.2}$	$97.3\scriptstyle{\pm 0.3}$	$96.7\scriptstyle{\pm 0.2}$
H-Diff	$96.6\scriptstyle{\pm 0.2}$	$95.0\scriptstyle{\pm 0.7}$	$95.7\scriptstyle{\pm 0.2}$

Appendix

Table A15: Comparison of MMI and

\mathrm{H_{diff}}

on AUARC (

\uparrow

) in selective classification under annotator-disagreement annotation imprecision.

	ShuffleNetV2-0.5x		ResNet-20		MobileNetV2-x0-5
Dataset	MMI	H-Diff	MMI	H-Diff	MMI	H-Diff
CIFAR-10	$99.4\scriptstyle{\pm 0.0}$	$99.1\scriptstyle{\pm 0.0}$	$99.5\scriptstyle{\pm 0.0}$	$99.3\scriptstyle{\pm 0.0}$	$99.5\scriptstyle{\pm 0.0}$	$99.2\scriptstyle{\pm 0.0}$
CIFAR-100	$90.2\scriptstyle{\pm 0.3}$	$83.0\scriptstyle{\pm 0.7}$	$90.6\scriptstyle{\pm 0.2}$	$84.5\scriptstyle{\pm 0.6}$	$90.9\scriptstyle{\pm 0.2}$	$83.7\scriptstyle{\pm 0.4}$

Appendix

Table A16: Comparison of MMI and

\mathrm{H_{diff}}

on AUARC (

\uparrow

) in selective classification under annotation imprecision from teacher model predictions.

Dataset	Smoothing $\epsilon$	MMI	H-Diff
CIFAR-10	0.05	$99.2\scriptstyle{\pm 0.1}$	$99.1\scriptstyle{\pm 0.1}$
	0.10	$98.8\scriptstyle{\pm 0.2}$	$98.6\scriptstyle{\pm 0.2}$
	0.15	$98.8\scriptstyle{\pm 0.2}$	$98.1\scriptstyle{\pm 0.2}$
	0.20	$98.7\scriptstyle{\pm 0.1}$	$97.3\scriptstyle{\pm 0.2}$
CIFAR-100	0.05	$93.4\scriptstyle{\pm 0.1}$	$92.2\scriptstyle{\pm 0.1}$
	0.10	$93.3\scriptstyle{\pm 0.1}$	$91.5\scriptstyle{\pm 0.2}$

Appendix

Table A17: Comparison of MMI and

\mathrm{H_{diff}}

on AUARC (

\uparrow

) in selective classification under annotation imprecision from label smoothing.

	SVHN	Places	FMNIST	ImageNet
EDL	$0.735\scriptstyle{\pm 0.004}$	$0.717\scriptstyle{\pm 0.004}$	$0.721\scriptstyle{\pm 0.007}$	$0.701\scriptstyle{\pm 0.004}$
Decali	$\mathbf{0.854\scriptstyle{\pm 0.011}}$	$\mathbf{0.832\scriptstyle{\pm 0.012}}$	$\mathbf{0.846\scriptstyle{\pm 0.014}}$	$\mathbf{0.807\scriptstyle{\pm 0.011}}$
DAPPr	$0.730\scriptstyle{\pm 0.007}$	$0.707\scriptstyle{\pm 0.006}$	$0.699\scriptstyle{\pm 0.009}$	$0.680\scriptstyle{\pm 0.006}$
Ours	$\underline{0.840\scriptstyle{\pm 0.009}}$	$\underline{0.816\scriptstyle{\pm 0.009}}$	$\underline{0.830\scriptstyle{\pm 0.013}}$	$\underline{0.785\scriptstyle{\pm 0.007}}$
Our-Ens	$\mathbf{1.000\scriptstyle{\pm 0.000}}$	$\mathbf{0.992\scriptstyle{\pm 0.001}}$	$\mathbf{0.997\scriptstyle{\pm 0.001}}$	$\mathbf{0.990\scriptstyle{\pm 0.001}}$
MCDO	$0.957\scriptstyle{\pm 0.009}$	$0.930\scriptstyle{\pm 0.009}$	$0.945\scriptstyle{\pm 0.009}$	$0.904\scriptstyle{\pm 0.010}$

Appendix

Table A18: AUROC (%) for OOD detection using epistemic uncertainty, with CIFAR-10 (models trained using CIFAR-10H) as the ID dataset, averaged over 10 runs. Methods are grouped into single-model approaches (above the line) and multiple-forward-pass approaches, including an ensemble of our POCCs (Our-Ens), MCDO, and DE (below the line). Bold and underlined entries denote the best and second-best results within each group, respectively.

	SVHN	Places	FMNIST	ImageNet
EDL	$0.717\scriptstyle{\pm 0.008}$	$0.696\scriptstyle{\pm 0.006}$	$0.693\scriptstyle{\pm 0.010}$	$0.670\scriptstyle{\pm 0.005}$
Decali	$\mathbf{0.750\scriptstyle{\pm 0.003}}$	$\underline{0.713\scriptstyle{\pm 0.003}}$	$\mathbf{0.737\scriptstyle{\pm 0.005}}$	$\mathbf{0.694\scriptstyle{\pm 0.002}}$
DAPPr	$0.609\scriptstyle{\pm 0.004}$	$0.605\scriptstyle{\pm 0.003}$	$0.603\scriptstyle{\pm 0.003}$	$0.601\scriptstyle{\pm 0.003}$
Ours	$\underline{0.743\scriptstyle{\pm 0.003}}$	$\mathbf{0.721\scriptstyle{\pm 0.003}}$	$\underline{0.730\scriptstyle{\pm 0.006}}$	$\underline{0.688\scriptstyle{\pm 0.003}}$
Our-Ens	$\mathbf{1.000\scriptstyle{\pm 0.000}}$	$\mathbf{0.989\scriptstyle{\pm 0.001}}$	$\mathbf{0.997\scriptstyle{\pm 0.000}}$	$\mathbf{0.980\scriptstyle{\pm 0.001}}$
MCDO	$0.944\scriptstyle{\pm 0.001}$	$0.902\scriptstyle{\pm 0.002}$	$0.940\scriptstyle{\pm 0.004}$	$0.852\scriptstyle{\pm 0.002}$

Appendix

Table A19: AUROC (%) for OOD detection using epistemic uncertainty, with CIFAR-10 (model trained using the ResNet-20 as the teacher model with

T=2.5

) as the ID dataset, averaged over 10 runs. Methods are grouped into single-model approaches (above the line) and multiple-forward-pass approaches, including an ensemble of our POCCs (Our-Ens), MCDO, and DE (below the line). Bold and underlined entries denote the best and second-best results within each group, respectively.

	SVHN	Places	FMNIST	ImageNet
EDL	$0.715\scriptstyle{\pm 0.008}$	$0.696\scriptstyle{\pm 0.006}$	$0.691\scriptstyle{\pm 0.008}$	$0.671\scriptstyle{\pm 0.004}$
Decali	$\underline{0.814\scriptstyle{\pm 0.020}}$	$\underline{0.780\scriptstyle{\pm 0.026}}$	$\underline{0.784\scriptstyle{\pm 0.020}}$	$\underline{0.755\scriptstyle{\pm 0.013}}$
DAPPr	$0.744\scriptstyle{\pm 0.003}$	$0.706\scriptstyle{\pm 0.002}$	$0.720\scriptstyle{\pm 0.007}$	$0.664\scriptstyle{\pm 0.002}$
Ours	$\mathbf{0.872\scriptstyle{\pm 0.009}}$	$\mathbf{0.836\scriptstyle{\pm 0.008}}$	$\mathbf{0.846\scriptstyle{\pm 0.007}}$	$\mathbf{0.798\scriptstyle{\pm 0.010}}$
Our-Ens	$\mathbf{0.999\scriptstyle{\pm 0.000}}$	$\mathbf{0.989\scriptstyle{\pm 0.001}}$	$\mathbf{0.997\scriptstyle{\pm 0.000}}$	$\mathbf{0.976\scriptstyle{\pm 0.001}}$
MCDO	$0.920\scriptstyle{\pm 0.002}$	$0.891\scriptstyle{\pm 0.002}$	$0.906\scriptstyle{\pm 0.005}$	$0.861\scriptstyle{\pm 0.003}$

Appendix

Table A20: AUROC (%) for OOD detection using epistemic uncertainty, with CIFAR-10 (model trained using label smoothing

\varepsilon=0.05

Explore similar work

May 25, 2025stat.ML

Optimal Conformal Prediction under Epistemic Uncertainty

Conformal prediction (CP) is a widely used frequentist framework to quantify uncertainty by constructing prediction sets with user-specified marginal coverage guarantees. In practice, CP is typically applied on top of probabilistic classifiers, which are able to express aleatoric but not epistemic uncertainty. In this paper, we consider the question of how to optimally employ CP on top of a more expressive formalism, namely credal sets, which can express both aleatoric and epistemic uncertainty. More specifically, we propose probabilistic Bernoulli prediction sets (BPS) and derive a variant that achieves conditional coverage for valid credal sets while remaining minimal in expected size. We then address the more realistic scenario in which the validity of the credal sets is not guaranteed. Assuming access to calibration data with ground-truth distributions over labels, we apply conformal risk control to BPS and derive a PAC-style guarantee: with high probability over the data, the achieved conditional coverage is at least the desired level. We validate our theoretical findings empirically over various datasets.

Alireza Javanmardi, Soroush H. Zargarbashi, Santo M. A. R. Thies +3

1LMU Munich · 2MCML · 3CISPA +3

Online Conformal Prediction Predictive Uncertainty Conformal Risk Control Uncertainty Integrity

Ask PDF

Mar 28, 2026cs.AI

Quantification of Credal Uncertainty: A Distance-Based Approach

Credal sets, i.e., closed convex sets of probability measures, provide a natural framework to represent aleatoric and epistemic uncertainty in machine learning. Yet how to quantify these two types of uncertainty for a given credal set, particularly in multiclass classification, remains underexplored. In this paper, we propose a distance-based approach to quantify total, aleatoric, and epistemic uncertainty for credal sets. Concretely, we introduce a family of such measures within the framework of Integral Probability Metrics (IPMs). The resulting quantities admit clear semantic interpretations, satisfy natural theoretical desiderata, and remain computationally tractable for common choices of IPMs. We instantiate the framework with the total variation distance and obtain simple, efficient uncertainty measures for multiclass classification. In the binary case, this choice recovers established uncertainty measures, for which a principled multiclass generalization has so far been missing. Empirical results confirm practical usefulness, with favorable performance at low computational cost.

Xabier Gonzalez-Garcia, Siu Lun Chau, Julian Rodemann +6

Dept. of Statistics, CS & Mathematics, Public Univ. of Navarre · EPIC Lab, Nanyang Technological Univ. Singapore · Rational Intelligence Lab, CISPA Helmholtz Center for Information Security +7

Uncertainty Multiclass Classification Quantifying Distance Integral Integrity

Ask PDF

Jun 9, 2026cs.LG

Can we trust our models? Epistemic calibration in second-order classification

Uncertainty estimation is critical for deploying machine learning models in high-stakes settings. However, classical calibration only assesses the reliability of predicted probabilities and does not evaluate whether epistemic uncertainty estimates are themselves trustworthy. This limitation is particularly relevant for second-order classification models. We introduce epistemic calibration, a principled criterion that measures whether reported epistemic uncertainty faithfully reflects the dispersion of model predictions around the ground truth. We show that epistemic calibration is a strictly stronger notion than classical calibration and captures failure modes invisible to standard metrics. We relate this work to the existing literature through an impossibility theorem that holds under the epistemic calibration hypothesis. To operationalize this concept, we propose the Expected Epistemic Calibration Error (EECE), which we prove to be a consistent estimator of a True Epistemic Calibration Error (TECE). Experiments across a broad range of uncertainty quantification methods show that epistemic calibration is a coherent and meaningful criterion and reveal substantial differences across methods, despite similar predictive performance.

Arthur Hoarau

Université de Lorraine, CentraleSupélec · Loria, CNRS, Metz, France

Dynamic Epistemic Logic Calibrated Uncertainty Uncertainty

Ask PDF