Epistemic Learning from Imprecise Annotation

Kaizheng Wang; Siu Lun Chau

cs.LGSep 28, 2026

Epistemic Learning from Imprecise Annotation

Authors: Kaizheng Wang, Siu Lun Chau

Organizations: Epistemic Intelligence & Computation Lab, College of Computing & Data Science, Nanyang Technological University, Singapore

Abstract

Imprecise annotations may support several plausible labelling distributions, yet learning methods often resolve this ambiguity into a single predictive distribution. This can obscure what the annotation evidence leaves unresolved. We introduce epistemic learning from credal supervision, a framework that uses convex sets of plausible labelling distributions, called credal sets, as supervision and learns sets of predictive distributions. We instantiate the framework with the pessimistic--optimistic credal classifier (POCC), which combines a shared backbone with two classification heads trained to minimise worst-case and best-case losses over the supervision sets. Their outputs define a predictive credal set whose spread provides an uncertainty score. We also show how credal labels can be obtained through a simple relaxation of existing probabilistic labels, reducing commitment to their precise probability assignments. This construction admits closed-form inner optimisation under cross-entropy loss, enabling efficient training. Assuming the supervision sets contain the true conditional label distributions, and other regularity assumptions, we establish a finite-sample generalisation bound for the averaged predictor with an explicit penalty for supervision imprecision. We evaluate POCC using human annotator disagreement and teacher predictions, alongside label smoothing as a controlled proxy for annotation imprecision. Across these settings, POCC achieves a favourable balance of predictive accuracy, calibration, and uncertainty-based selective classification versus competitive baselines.

Figures & tables

Figure 1 : Overview of epistemic learning from credal supervision. Top: the framework relates credal supervision to sets of predictive distributions. Bottom left: practical construction of credal labels from imprecise annotations. Bottom right: POCC uses a shared backbone and pessimistic and optimistic heads to form a predictive credal set.

Method	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$
EDL	$85.0\scriptstyle{\pm 0.4}$	$9.5\scriptstyle{\pm 0.3}$	$96.7\scriptstyle{\pm 0.1}$	$33.1\scriptstyle{\pm 8.3}$
LbBnn	$\mathbf{87.0\scriptstyle{\pm 0.8}}$	$6.6\scriptstyle{\pm 0.3}$	$95.9\scriptstyle{\pm 0.3}$	$54.8\scriptstyle{\pm 6.3}$
Decali	$84.3\scriptstyle{\pm 1.0}$	$\underline{1.0\scriptstyle{\pm 0.2}}$	$94.9\scriptstyle{\pm 0.5}$	$35.4\scriptstyle{\pm 4.5}$
DAPPr	$86.6\scriptstyle{\pm 0.3}$	$6.9\scriptstyle{\pm 0.2}$	$\mathbf{97.2\scriptstyle{\pm 0.1}}$	$68.3\scriptstyle{\pm 5.9}$
Ours	$\mathbf{87.0\scriptstyle{\pm 0.4}}$	$2.9\scriptstyle{\pm 0.3}$	$\underline{97.1\scriptstyle{\pm 0.2}}$	$\mathbf{86.0\scriptstyle{\pm 6.1}}$
MCDO	$85.6\scriptstyle{\pm 0.7}$	$\mathbf{0.9\scriptstyle{\pm 0.1}}$	$96.6\scriptstyle{\pm 0.3}$	$71.0\scriptstyle{\pm 10.6}$

Table 1 : CIFAR-10H results (%). Best in bold; second best underlined.

Method	CIFAR-10				CIFAR-100
Method	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$
EDL	$93.4\scriptstyle{\pm 0.2}$	$9.0\scriptstyle{\pm 0.2}$	$99.1\scriptstyle{\pm 0.1}$	$51.6\scriptstyle{\pm 3.8}$	$33.1\scriptstyle{\pm 5.7}$	$\underline{23.3\scriptstyle{\pm 3.9}}$	$65.8\scriptstyle{\pm 6.1}$	$30.2\scriptstyle{\pm 2.9}$
LbBnn	$\underline{95.1\scriptstyle{\pm 0.1}}$	$28.1\scriptstyle{\pm 0.2}$	$\underline{99.2\scriptstyle{\pm 0.0}}$	$54.2\scriptstyle{\pm 3.3}$	$\underline{73.1\scriptstyle{\pm 0.2}}$	$63.1\scriptstyle{\pm 0.2}$	$63.4\scriptstyle{\pm 0.4}$	$31.9\scriptstyle{\pm 2.4}$
Decali	$93.5\scriptstyle{\pm 0.1}$	$8.5\scriptstyle{\pm 0.1}$	$98.0\scriptstyle{\pm 0.1}$	$29.7\scriptstyle{\pm 2.5}$	$71.0\scriptstyle{\pm 0.2}$	$27.8\scriptstyle{\pm 0.2}$	$73.3\scriptstyle{\pm 0.3}$	$64.2\scriptstyle{\pm 1.8}$
DAPPr	$93.4\scriptstyle{\pm 0.2}$	$14.6\scriptstyle{\pm 0.2}$	$99.0\scriptstyle{\pm 0.0}$	$42.8\scriptstyle{\pm 3.1}$	$72.5\scriptstyle{\pm 0.4}$	$52.8\scriptstyle{\pm 0.3}$	$78.3\scriptstyle{\pm 0.3}$	$55.1\scriptstyle{\pm 1.6}$
Ours	$\mathbf{95.5\scriptstyle{\pm 0.1}}$	$\mathbf{3.0\scriptstyle{\pm 0.1}}$	$\mathbf{99.5\scriptstyle{\pm 0.0}}$	$\mathbf{99.9\scriptstyle{\pm 0.3}}$	$\mathbf{77.2\scriptstyle{\pm 0.2}}$	$\mathbf{10.6\scriptstyle{\pm 0.4}}$	$\mathbf{90.6\scriptstyle{\pm 0.2}}$	$\mathbf{100.0\scriptstyle{\pm 0.0}}$

Table 2 : Teacher-prediction results (%) with ResNet-20. Best in bold; second best underlined.

Method	CIFAR-10				CIFAR-100
Method	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$
EDL	$93.4\scriptstyle{\pm 0.3}$	$9.1\scriptstyle{\pm 0.2}$	$99.1\scriptstyle{\pm 0.1}$	$48.6\scriptstyle{\pm 4.3}$	$26.3\scriptstyle{\pm 1.2}$	$18.7\scriptstyle{\pm 1.0}$	$58.5\scriptstyle{\pm 1.6}$	$24.3\scriptstyle{\pm 0.6}$
LbBnn	$\underline{95.4\scriptstyle{\pm 0.2}}$	$29.9\scriptstyle{\pm 0.3}$	$99.1\scriptstyle{\pm 0.1}$	$53.6\scriptstyle{\pm 2.6}$	$\underline{79.3\scriptstyle{\pm 0.3}}$	$61.6\scriptstyle{\pm 0.5}$	$74.5\scriptstyle{\pm 0.9}$	$47.7\scriptstyle{\pm 1.1}$
Decali	$93.9\scriptstyle{\pm 0.3}$	$3.6\scriptstyle{\pm 0.1}$	$98.3\scriptstyle{\pm 0.2}$	$39.0\scriptstyle{\pm 5.2}$	$76.2\scriptstyle{\pm 0.3}$	$\underline{3.5\scriptstyle{\pm 0.4}}$	$90.7\scriptstyle{\pm 0.3}$	$94.0\scriptstyle{\pm 0.7}$
DAPPr	$93.9\scriptstyle{\pm 0.3}$	$8.9\scriptstyle{\pm 0.2}$	$99.1\scriptstyle{\pm 0.1}$	$58.5\scriptstyle{\pm 6.4}$	$75.1\scriptstyle{\pm 0.4}$	$35.3\scriptstyle{\pm 0.4}$	$91.9\scriptstyle{\pm 0.2}$	$76.5\scriptstyle{\pm 0.4}$
Ours	$\mathbf{95.5\scriptstyle{\pm 0.1}}$	$\mathbf{1.5\scriptstyle{\pm 0.1}}$	$\underline{99.2\scriptstyle{\pm 0.1}}$	$\mathbf{90.6\scriptstyle{\pm 4.7}}$	$\underline{79.3\scriptstyle{\pm 0.2}}$	$\mathbf{2.7\scriptstyle{\pm 0.2}}$	$\underline{93.4\scriptstyle{\pm 0.1}}$	$\mathbf{99.0\scriptstyle{\pm 0.3}}$

Table 3 : Label-smoothing results (%) at

\epsilon=0.05

over 10 runs. Best in bold; second best underlined.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	CIFAR-10H	MiceBone	TreeVersity
Backbone	ResNet-18	DenseNet-121	DenseNet-121
Classes	10	3	6
Input size	$(3\times 32\times 32)$	$(3\times 224\times 224)$	$(3\times 224\times 224)$
Batch size	128	128	128
Epochs	150	60	60
Learning rate	0.1	0.00001	0.00001

Appendix

Table A1 : Hyperparameters used for annotator-disagreement experiments.

Hyperparameter	CIFAR-10	CIFAR-100
Backbone	ResNet-18	WRN-28-4
Classes	10	100
Input size	$(3\times 32\times 32)$	$(3\times 32\times 32)$
Batch size	128	128
Epochs	200	200
Learning rate	0.1	0.1

Appendix

Table A2 : Hyperparameters used for both teacher-prediction and label-smoothing experiments.

Component	Specification
CPU	Intel Core Ultra 24C/24T, 5.8GHz Max, 36MB L3 Cache
GPU	$1\times$ NVIDIA GeForce RTX 5090 (32 GB GDDR7)
RAM	192 GB DDR4 ECC DIMM

Appendix

Table A3: Specifications of Computing Resources

	MiceBone Dataset				TreeVersity Dataset
Method	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$
EDL	$88.5\scriptstyle{\pm 0.8}$	$9.9\scriptstyle{\pm 0.8}$	$\mathbf{97.4\scriptstyle{\pm 0.3}}$	$\underline{68.7\scriptstyle{\pm 3.6}}$	$85.5\scriptstyle{\pm 0.6}$	$17.1\scriptstyle{\pm 0.7}$	$\underline{95.7\scriptstyle{\pm 0.3}}$	$58.4\scriptstyle{\pm 1.9}$
Decali	$88.7\scriptstyle{\pm 1.2}$	$\underline{5.8\scriptstyle{\pm 1.0}}$	$93.6\scriptstyle{\pm 0.9}$	$51.2\scriptstyle{\pm 6.0}$	$\mathbf{86.9\scriptstyle{\pm 0.4}}$	$\mathbf{2.3\scriptstyle{\pm 0.3}}$	$95.0\scriptstyle{\pm 0.4}$	$\underline{92.4\scriptstyle{\pm 2.2}}$
DAPPr	$\underline{89.0\scriptstyle{\pm 1.1}}$	$13.2\scriptstyle{\pm 0.9}$	$96.4\scriptstyle{\pm 0.5}$	$53.6\scriptstyle{\pm 5.1}$	$\underline{86.6\scriptstyle{\pm 0.5}}$	$17.0\scriptstyle{\pm 0.4}$	$95.5\scriptstyle{\pm 0.3}$	$61.5\scriptstyle{\pm 1.5}$
Ours	$\mathbf{89.1\scriptstyle{\pm 0.9}}$	$\mathbf{2.7\scriptstyle{\pm 0.4}}$	$\underline{97.3\scriptstyle{\pm 0.3}}$	$\mathbf{96.1\scriptstyle{\pm 4.6}}$	$86.2\scriptstyle{\pm 0.6}$	$\underline{2.5\scriptstyle{\pm 0.5}}$	$\mathbf{96.7\scriptstyle{\pm 0.2}}$	$\mathbf{97.1\scriptstyle{\pm 1.9}}$
DE	$85.5\scriptstyle{\pm 0.9}$	$7.5\scriptstyle{\pm 0.8}$	$94.3\scriptstyle{\pm 0.5}$	$23.6\scriptstyle{\pm 5.5}$	$75.7\scriptstyle{\pm 1.1}$	$6.2\scriptstyle{\pm 0.9}$	$88.6\scriptstyle{\pm 0.8}$	$24.4\scriptstyle{\pm 1.7}$

Appendix

Table A4 : Performance comparison (in

\%

) under annotator-disagreement annotation imprecision across various datasets, averaged over 10 runs.

Method	CIFAR-10				CIFAR-100
Method	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$
Teacher: ResNet-20
EDL	$93.4\scriptstyle{\pm 0.2}$	$9.0\scriptstyle{\pm 0.2}$	$99.1\scriptstyle{\pm 0.1}$	$51.6\scriptstyle{\pm 3.8}$	$33.1\scriptstyle{\pm 5.7}$	$\underline{23.3\scriptstyle{\pm 3.9}}$	$65.8\scriptstyle{\pm 6.1}$	$30.2\scriptstyle{\pm 2.9}$
LbBnn	$\underline{95.1\scriptstyle{\pm 0.1}}$	$28.1\scriptstyle{\pm 0.2}$	$\underline{99.2\scriptstyle{\pm 0.0}}$	$54.2\scriptstyle{\pm 3.3}$	$\underline{73.1\scriptstyle{\pm 0.2}}$	$63.1\scriptstyle{\pm 0.2}$	$63.4\scriptstyle{\pm 0.4}$	$31.9\scriptstyle{\pm 2.4}$
Decali	$93.5\scriptstyle{\pm 0.1}$	$8.5\scriptstyle{\pm 0.1}$	$98.0\scriptstyle{\pm 0.1}$	$29.7\scriptstyle{\pm 2.5}$	$71.0\scriptstyle{\pm 0.2}$	$27.8\scriptstyle{\pm 0.2}$	$73.3\scriptstyle{\pm 0.2}$	$64.2\scriptstyle{\pm 1.9}$
DAPPr	$93.4\scriptstyle{\pm 0.2}$	$14.6\scriptstyle{\pm 0.2}$	$99.0\scriptstyle{\pm 0.0}$	$42.8\scriptstyle{\pm 3.1}$	$72.5\scriptstyle{\pm 0.4}$	$52.8\scriptstyle{\pm 0.3}$	$78.3\scriptstyle{\pm 0.3}$	$55.1\scriptstyle{\pm 1.6}$

Appendix

Table A5 : Performance comparison (in

\%

) under teacher-prediction annotation imprecision with varying settings, averaged over 10 runs.

	Method	CIFAR-10				CIFAR-100
	Method	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$
$\epsilon=0.05$	EDL	$93.4\scriptstyle{\pm 0.3}$	$9.1\scriptstyle{\pm 0.2}$	$99.1\scriptstyle{\pm 0.1}$	$48.6\scriptstyle{\pm 4.3}$	$26.3\scriptstyle{\pm 1.2}$	$18.7\scriptstyle{\pm 1.0}$	$58.5\scriptstyle{\pm 1.6}$	$24.3\scriptstyle{\pm 0.6}$
	LbBnn	$\underline{95.4\scriptstyle{\pm 0.2}}$	$29.9\scriptstyle{\pm 0.3}$	$99.1\scriptstyle{\pm 0.1}$	$53.6\scriptstyle{\pm 2.6}$	$\underline{79.3\scriptstyle{\pm 0.3}}$	$61.6\scriptstyle{\pm 0.5}$	$74.5\scriptstyle{\pm 0.9}$	$47.7\scriptstyle{\pm 1.1}$
	Decali	$93.9\scriptstyle{\pm 0.3}$	$3.6\scriptstyle{\pm 0.1}$	$98.3\scriptstyle{\pm 0.2}$	$39.0\scriptstyle{\pm 5.2}$	$76.2\scriptstyle{\pm 0.3}$	$\underline{3.5\scriptstyle{\pm 0.4}}$	$90.7\scriptstyle{\pm 0.3}$	$94.0\scriptstyle{\pm 0.7}$
	DAPPr	$93.9\scriptstyle{\pm 0.3}$	$8.9\scriptstyle{\pm 0.2}$	$99.1\scriptstyle{\pm 0.1}$	$58.5\scriptstyle{\pm 6.4}$	$75.1\scriptstyle{\pm 0.4}$	$35.3\scriptstyle{\pm 0.4}$	$91.9\scriptstyle{\pm 0.2}$	$76.5\scriptstyle{\pm 0.4}$
	Ours	$\mathbf{95.5\scriptstyle{\pm 0.1}}$	$\mathbf{1.5\scriptstyle{\pm 0.1}}$	$\underline{99.2\scriptstyle{\pm 0.1}}$	$\mathbf{90.6\scriptstyle{\pm 4.7}}$	$\underline{79.3\scriptstyle{\pm 0.2}}$	$\mathbf{2.7\scriptstyle{\pm 0.2}}$	$\underline{93.4\scriptstyle{\pm 0.1}}$	$\mathbf{99.0\scriptstyle{\pm 0.3}}$

Appendix

Table A6 : Label-smoothing results (%) across

\epsilon

. POCC achieves the highest mean BQS in seven of eight settings; the sole exception is

\epsilon=0.1

on CIFAR-10, where it ranks second behind DE (5 members), with the gap attributable to marginal differences in accuracy and AUARC. Statistical significance tests (Table A10 , Appendix E ) confirm that POCC ranks first statistically across all settings.

Method	CIFAR-10				CIFAR-100
Method	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$
Teacher: ResNet-20
EDL	$93.3\scriptstyle{\pm 0.2}$	$9.1\scriptstyle{\pm 0.3}$	$99.1\scriptstyle{\pm 0.1}$	$1.2\scriptstyle{\pm 2.4}$	$33.7\scriptstyle{\pm 5.0}$	$23.7\scriptstyle{\pm 3.4}$	$66.5\scriptstyle{\pm 5.5}$	$14.3\scriptstyle{\pm 3.1}$
LbBnn	$\underline{95.4\scriptstyle{\pm 0.1}}$	$1.5\scriptstyle{\pm 0.1}$	$\underline{99.5\scriptstyle{\pm 0.0}}$	$\underline{91.4\scriptstyle{\pm 2.7}}$	$\underline{77.3\scriptstyle{\pm 0.3}}$	$19.5\scriptstyle{\pm 0.5}$	$90.3\scriptstyle{\pm 0.2}$	$81.3\scriptstyle{\pm 0.7}$
Decali	$93.8\scriptstyle{\pm 0.2}$	$1.5\scriptstyle{\pm 0.1}$	$99.2\scriptstyle{\pm 0.0}$	$42.2\scriptstyle{\pm 8.0}$	$74.3\scriptstyle{\pm 0.4}$	$\mathbf{2.4\scriptstyle{\pm 0.3}}$	$89.1\scriptstyle{\pm 0.1}$	$92.9\scriptstyle{\pm 1.2}$
DAPPr	$94.0\scriptstyle{\pm 0.2}$	$7.0\scriptstyle{\pm 0.1}$	$99.2\scriptstyle{\pm 0.1}$	$27.2\scriptstyle{\pm 7.4}$	$75.4\scriptstyle{\pm 0.3}$	$39.7\scriptstyle{\pm 0.3}$	$90.1\scriptstyle{\pm 0.3}$	$61.6\scriptstyle{\pm 1.2}$

Appendix

Table A7 : Performance comparison (in

\%

) using the teachers’ original predictions (

T=1.0

) as the source of annotation imprecision, averaged over 10 runs. These results consistently confirm the robustness of POCC in achieving the highest BQS scores across settings.

Method

CIFAR-10H

MiceBone

TreeVersity

ACC

1-ECE

AUARC

BQS-ST

ACC

1-ECE

AUARC

BQS-ST

ACC

1-ECE

AUARC

BQS-ST

EDL

-5.0

-6.0

-1.0

15.2

-1.0

-2.0

\mathbf{3.0}

\underline{58.3}

-2.0

-3.0

\underline{1.0}

30.4

LbBnn

\underline{3.0}

-2.0

-4.0

47.8

–

Decali

-5.0

\mathbf{5.0}

-6.0

33.3

\underline{1.0}

\underline{2.0}

-4.0

52.8

\mathbf{3.0}

\mathbf{3.0}

-2.0

\underline{75.0}

DAPPr

1.0

-4.0

\mathbf{5.0}

61.6

\mathbf{2.0}

-4.0

\underline{0.0}

52.4

\mathbf{3.0}

-3.0

\underline{1.0}

54.2

Ours

\mathbf{4.0}

\underline{2.0}

\mathbf{5.0}

\mathbf{90.9}

\mathbf{2.0}

\mathbf{4.0}

\mathbf{3.0}

\mathbf{100.0}

\underline{0.0}

\mathbf{3.0}

\mathbf{4.0}

\mathbf{85.7}

Appendix

Table A8 : Performance comparison (in

\%

\uparrow

) under annotator disagreements.

	Method	CIFAR-10				CIFAR-100
	Method	ACC	1-ECE	AUARC	BQS-ST	ACC	1-ECE	AUARC	BQS-ST
ResNet-20	EDL	$-3.0$	$0.0$	$0.0$	$33.3$	$-6.0$	$\underline{4.0}$	$-5.0$	$27.8$
	LbBnn	$\underline{4.0}$	$-6.0$	$\underline{4.0}$	$\underline{53.7}$	$\underline{4.0}$	$-6.0$	$-5.0$	$27.8$
	Decali	$-3.0$	$\underline{3.0}$	$-6.0$	$25.0$	$-3.0$	$2.0$	$-2.0$	$39.6$
	DAPPr	$-3.0$	$-4.0$	$-3.0$	$13.9$	$1.0$	$-4.0$	$0.0$	$40.2$
	Ours	$\mathbf{6.0}$	$\mathbf{6.0}$	$\mathbf{6.0}$	$\mathbf{100.0}$	$\mathbf{6.0}$	$\mathbf{6.0}$	$\mathbf{6.0}$	$\mathbf{100.0}$

Appendix

Table A9 : Performance comparison (in

\%

\uparrow

) under teacher-prediction annotation imprecision.

	Method	CIFAR-10				CIFAR-100
	Method	ACC	1-ECE	AUARC	BQS-ST	ACC	1-ECE	AUARC	BQS-ST
$\epsilon=0.05$	EDL	$-6.0$	$-4.0$	$-1.0$	$19.4$	$-6.0$	$-2.0$	$-6.0$	$11.1$
	LbBnn	$\mathbf{5.0}$	$-6.0$	$-2.0$	$44.4$	$\underline{3.0}$	$-6.0$	$-4.0$	$30.6$
	Decali	$-1.0$	$\underline{3.0}$	$-6.0$	$40.2$	$-1.0$	$\underline{3.0}$	$-2.0$	$50.0$
	DAPPr	$-2.0$	$-2.0$	$\underline{2.0}$	$45.5$	$-4.0$	$-4.0$	$1.0$	$30.6$
	Ours	$\mathbf{5.0}$	$\mathbf{6.0}$	$1.0$	$\mathbf{86.1}$	$\underline{3.0}$	$\mathbf{6.0}$	$\underline{4.0}$	$\mathbf{86.1}$

Appendix

Table A10 : Performance comparison (in

\%

\uparrow

) under label-smoothing annotation imprecision.

Figure A6 : Pairwise one-sided Wilcoxon signed-rank test results on various datasets under label-smoothing imprecision. Cell $(i,j)$ indicates whether row method $i$ significantly outperforms column method $j$ ; coloured cells show the scaled $p$ -value ( $\times 10^{3}$ ). Orange ( $\blacksquare$ ) mark comparisons involving our method; dark navy ( $\blacksquare$ ) mark all other significant pairs.

Method	ACC $\uparrow$	ECE $\downarrow$	AUARC $\uparrow$	BQS $\uparrow$	BQS-ST $\uparrow$
CIFAR-10H
EDL	$84.8\scriptstyle{\pm 0.2}$	$9.5\scriptstyle{\pm 0.5}$	$96.6\scriptstyle{\pm 0.1}$	$41.4\scriptstyle{\pm 9.2}$	$33.3$
LbBnn	$85.9\scriptstyle{\pm 0.5}$	$6.9\scriptstyle{\pm 0.4}$	$96.2\scriptstyle{\pm 0.3}$	$53.5\scriptstyle{\pm 8.9}$	$48.5$
Decali	$82.7\scriptstyle{\pm 0.9}$	$9.5\scriptstyle{\pm 0.5}$	$95.5\scriptstyle{\pm 0.4}$	$3.4\scriptstyle{\pm 5.1}$	$0.0$
DAPPr	$85.5\scriptstyle{\pm 0.4}$	$6.2\scriptstyle{\pm 0.4}$	$96.6\scriptstyle{\pm 0.2}$	$59.5\scriptstyle{\pm 7.5}$	$62.9$
Ours	$\mathbf{87.0\scriptstyle{\pm 0.4}}$	$\underline{2.9\scriptstyle{\pm 0.3}}$	$\mathbf{97.1\scriptstyle{\pm 0.2}}$	$\mathbf{95.7\scriptstyle{\pm 1.1}}$	$\mathbf{93.9}$

Appendix

Table A11 : Performance comparison (in

\%

) under annotator disagreement, where baseline methods are trained on collapsed hard labels derived from imprecise annotations.

	Method	ACC $\uparrow$	ECE $\downarrow$	Sp(EU,CE) $\uparrow$	BQS $\uparrow$	BQS-ST $\uparrow$
CIFAR-10H	EDL	$85.0\scriptstyle{\pm 0.4}$	$9.5\scriptstyle{\pm 0.3}$	$\mathbf{95.3\scriptstyle{\pm 0.2}}$	$40.8\scriptstyle{\pm 6.4}$	$33.3$
	LbBnn	$87.0\scriptstyle{\pm 0.8}$	$6.6\scriptstyle{\pm 0.3}$	$88.8\scriptstyle{\pm 0.7}$	$51.7\scriptstyle{\pm 4.4}$	$47.8$
	Decali	$84.3\scriptstyle{\pm 1.0}$	$\underline{1.0\scriptstyle{\pm 0.2}}$	$85.6\scriptstyle{\pm 0.9}$	$35.4\scriptstyle{\pm 4.5}$	$33.3$
	DAPPr	$86.6\scriptstyle{\pm 0.3}$	$6.9\scriptstyle{\pm 0.2}$	$\mathbf{95.3\scriptstyle{\pm 0.2}}$	$68.5\scriptstyle{\pm 5.6}$	$61.6$
	Ours	$\mathbf{87.0\scriptstyle{\pm 0.4}}$	$2.9\scriptstyle{\pm 0.3}$	$94.4\scriptstyle{\pm 0.5}$	$\mathbf{84.9\scriptstyle{\pm 5.7}}$	$\mathbf{81.8}$
	MCDO	$85.6\scriptstyle{\pm 0.7}$	$\mathbf{0.9\scriptstyle{\pm 0.1}}$	$90.4\scriptstyle{\pm 0.4}$	$63.8\scriptstyle{\pm 8.7}$	$56.6$

Appendix

Table A12 : Annotator-disagreement results (%). Best in bold; second best underlined.

Method	CIFAR-10					CIFAR-100
Method	ACC	ECE	Sp(EU,CE)	BQS	BQS-ST	ACC	ECE	Sp(EU,CE)	BQS	BQS-ST
Teacher: ResNet-20
EDL	$93.4\scriptstyle{\pm 0.2}$	$9.0\scriptstyle{\pm 0.2}$	$\mathbf{98.9\scriptstyle{\pm 0.1}}$	$\underline{60.7\scriptstyle{\pm 2.9}}$	$\underline{50.0}$	$33.1\scriptstyle{\pm 5.7}$	$\underline{23.3\scriptstyle{\pm 3.9}}$	$62.7\scriptstyle{\pm 5.6}$	$56.6\scriptstyle{\pm 1.0}$	$50.0$
LbBnn	$\underline{95.1\scriptstyle{\pm 0.1}}$	$28.1\scriptstyle{\pm 0.2}$	$96.0\scriptstyle{\pm 0.1}$	$51.3\scriptstyle{\pm 2.9}$	$44.1$	$\underline{73.1\scriptstyle{\pm 0.2}}$	$63.1\scriptstyle{\pm 0.2}$	$-43.4\scriptstyle{\pm 1.4}$	$30.2\scriptstyle{\pm 0.3}$	$27.8$
Decali	$93.5\scriptstyle{\pm 0.1}$	$8.5\scriptstyle{\pm 0.1}$	$88.7\scriptstyle{\pm 0.4}$	$29.7\scriptstyle{\pm 2.5}$	$25.0$	$71.0\scriptstyle{\pm 0.2}$	$27.8\scriptstyle{\pm 0.2}$	$6.1\scriptstyle{\pm 0.4}$	$65.7\scriptstyle{\pm 0.6}$	$36.1$
DAPPr	$93.4\scriptstyle{\pm 0.2}$	$14.6\scriptstyle{\pm 0.2}$	$97.6\scriptstyle{\pm 0.1}$	$49.8\scriptstyle{\pm 2.4}$	$29.8$	$72.5\scriptstyle{\pm 0.4}$	$52.8\scriptstyle{\pm 0.3}$	$42.8\scriptstyle{\pm 0.7}$	$61.7\scriptstyle{\pm 0.5}$	$36.1$

Appendix

Table A13 : Teacher-prediction results (%). Best in bold; second best underlined.

Method	CIFAR-10					CIFAR-100
Method	ACC	ECE	Sp(EU,CE)	BQS	BQS-ST	ACC	ECE	Sp(EU,CE)	BQS	BQS-ST
$\epsilon=0.05$
EDL	$93.4\scriptstyle{\pm 0.3}$	$9.1\scriptstyle{\pm 0.2}$	$\mathbf{99.0\scriptstyle{\pm 0.1}}$	$58.9\scriptstyle{\pm 1.9}$	$38.9$	$26.3\scriptstyle{\pm 1.2}$	$18.7\scriptstyle{\pm 1.0}$	$58.6\scriptstyle{\pm 1.8}$	$47.3\scriptstyle{\pm 0.6}$	$16.7$
LbBnn	$\underline{95.4\scriptstyle{\pm 0.2}}$	$29.9\scriptstyle{\pm 0.3}$	$64.8\scriptstyle{\pm 2.6}$	$40.3\scriptstyle{\pm 3.8}$	$45.5$	$\underline{79.3\scriptstyle{\pm 0.3}}$	$61.6\scriptstyle{\pm 0.5}$	$-5.8\scriptstyle{\pm 4.0}$	$32.7\scriptstyle{\pm 0.3}$	$25.0$
Decali	$93.9\scriptstyle{\pm 0.3}$	$3.6\scriptstyle{\pm 0.1}$	$52.3\scriptstyle{\pm 6.5}$	$39.5\scriptstyle{\pm 5.6}$	$40.2$	$76.2\scriptstyle{\pm 0.3}$	$\underline{3.5\scriptstyle{\pm 0.4}}$	$76.6\scriptstyle{\pm 0.9}$	$93.1\scriptstyle{\pm 0.7}$	$50.0$
DAPPr	$93.9\scriptstyle{\pm 0.3}$	$8.9\scriptstyle{\pm 0.2}$	$\underline{98.9\scriptstyle{\pm 0.1}}$	$67.1\scriptstyle{\pm 4.0}$	$56.6$	$75.1\scriptstyle{\pm 0.4}$	$35.3\scriptstyle{\pm 0.4}$	$\mathbf{87.6\scriptstyle{\pm 0.4}}$	$78.4\scriptstyle{\pm 0.2}$	$44.4$

Appendix

Table A14 : Label-smoothing results (%). Best in bold; second best underlined.

Metric	CIFAR-10H	MiceBone	TreeVersity
MMI	$97.1\scriptstyle{\pm 0.2}$	$97.3\scriptstyle{\pm 0.3}$	$96.7\scriptstyle{\pm 0.2}$
H-Diff	$96.6\scriptstyle{\pm 0.2}$	$95.0\scriptstyle{\pm 0.7}$	$95.7\scriptstyle{\pm 0.2}$

Appendix

Table A15: Comparison of MMI and

\mathrm{H_{diff}}

on AUARC (

\uparrow

) in selective classification under annotator-disagreement annotation imprecision.

	ShuffleNetV2-0.5x		ResNet-20		MobileNetV2-x0-5
Dataset	MMI	H-Diff	MMI	H-Diff	MMI	H-Diff
CIFAR-10	$99.4\scriptstyle{\pm 0.0}$	$99.1\scriptstyle{\pm 0.0}$	$99.5\scriptstyle{\pm 0.0}$	$99.3\scriptstyle{\pm 0.0}$	$99.5\scriptstyle{\pm 0.0}$	$99.2\scriptstyle{\pm 0.0}$
CIFAR-100	$90.2\scriptstyle{\pm 0.3}$	$83.0\scriptstyle{\pm 0.7}$	$90.6\scriptstyle{\pm 0.2}$	$84.5\scriptstyle{\pm 0.6}$	$90.9\scriptstyle{\pm 0.2}$	$83.7\scriptstyle{\pm 0.4}$

Appendix

Table A16: Comparison of MMI and

\mathrm{H_{diff}}

on AUARC (

\uparrow

) in selective classification under annotation imprecision from teacher model predictions.

Dataset	Smoothing $\epsilon$	MMI	H-Diff
CIFAR-10	0.05	$99.2\scriptstyle{\pm 0.1}$	$99.1\scriptstyle{\pm 0.1}$
	0.10	$98.8\scriptstyle{\pm 0.2}$	$98.6\scriptstyle{\pm 0.2}$
	0.15	$98.8\scriptstyle{\pm 0.2}$	$98.1\scriptstyle{\pm 0.2}$
	0.20	$98.7\scriptstyle{\pm 0.1}$	$97.3\scriptstyle{\pm 0.2}$
CIFAR-100	0.05	$93.4\scriptstyle{\pm 0.1}$	$92.2\scriptstyle{\pm 0.1}$
	0.10	$93.3\scriptstyle{\pm 0.1}$	$91.5\scriptstyle{\pm 0.2}$

Appendix

Table A17: Comparison of MMI and

\mathrm{H_{diff}}

on AUARC (

\uparrow

) in selective classification under annotation imprecision from label smoothing.

	SVHN	Places	FMNIST	ImageNet
EDL	$0.735\scriptstyle{\pm 0.004}$	$0.717\scriptstyle{\pm 0.004}$	$0.721\scriptstyle{\pm 0.007}$	$0.701\scriptstyle{\pm 0.004}$
Decali	$\mathbf{0.854\scriptstyle{\pm 0.011}}$	$\mathbf{0.832\scriptstyle{\pm 0.012}}$	$\mathbf{0.846\scriptstyle{\pm 0.014}}$	$\mathbf{0.807\scriptstyle{\pm 0.011}}$
DAPPr	$0.730\scriptstyle{\pm 0.007}$	$0.707\scriptstyle{\pm 0.006}$	$0.699\scriptstyle{\pm 0.009}$	$0.680\scriptstyle{\pm 0.006}$
Ours	$\underline{0.840\scriptstyle{\pm 0.009}}$	$\underline{0.816\scriptstyle{\pm 0.009}}$	$\underline{0.830\scriptstyle{\pm 0.013}}$	$\underline{0.785\scriptstyle{\pm 0.007}}$
Our-Ens	$\mathbf{1.000\scriptstyle{\pm 0.000}}$	$\mathbf{0.992\scriptstyle{\pm 0.001}}$	$\mathbf{0.997\scriptstyle{\pm 0.001}}$	$\mathbf{0.990\scriptstyle{\pm 0.001}}$
MCDO	$0.957\scriptstyle{\pm 0.009}$	$0.930\scriptstyle{\pm 0.009}$	$0.945\scriptstyle{\pm 0.009}$	$0.904\scriptstyle{\pm 0.010}$

Appendix

Table A18: AUROC (%) for OOD detection using epistemic uncertainty, with CIFAR-10 (models trained using CIFAR-10H) as the ID dataset, averaged over 10 runs. Methods are grouped into single-model approaches (above the line) and multiple-forward-pass approaches, including an ensemble of our POCCs (Our-Ens), MCDO, and DE (below the line). Bold and underlined entries denote the best and second-best results within each group, respectively.

	SVHN	Places	FMNIST	ImageNet
EDL	$0.717\scriptstyle{\pm 0.008}$	$0.696\scriptstyle{\pm 0.006}$	$0.693\scriptstyle{\pm 0.010}$	$0.670\scriptstyle{\pm 0.005}$
Decali	$\mathbf{0.750\scriptstyle{\pm 0.003}}$	$\underline{0.713\scriptstyle{\pm 0.003}}$	$\mathbf{0.737\scriptstyle{\pm 0.005}}$	$\mathbf{0.694\scriptstyle{\pm 0.002}}$
DAPPr	$0.609\scriptstyle{\pm 0.004}$	$0.605\scriptstyle{\pm 0.003}$	$0.603\scriptstyle{\pm 0.003}$	$0.601\scriptstyle{\pm 0.003}$
Ours	$\underline{0.743\scriptstyle{\pm 0.003}}$	$\mathbf{0.721\scriptstyle{\pm 0.003}}$	$\underline{0.730\scriptstyle{\pm 0.006}}$	$\underline{0.688\scriptstyle{\pm 0.003}}$
Our-Ens	$\mathbf{1.000\scriptstyle{\pm 0.000}}$	$\mathbf{0.989\scriptstyle{\pm 0.001}}$	$\mathbf{0.997\scriptstyle{\pm 0.000}}$	$\mathbf{0.980\scriptstyle{\pm 0.001}}$
MCDO	$0.944\scriptstyle{\pm 0.001}$	$0.902\scriptstyle{\pm 0.002}$	$0.940\scriptstyle{\pm 0.004}$	$0.852\scriptstyle{\pm 0.002}$

Appendix

Table A19: AUROC (%) for OOD detection using epistemic uncertainty, with CIFAR-10 (model trained using the ResNet-20 as the teacher model with

T=2.5

) as the ID dataset, averaged over 10 runs. Methods are grouped into single-model approaches (above the line) and multiple-forward-pass approaches, including an ensemble of our POCCs (Our-Ens), MCDO, and DE (below the line). Bold and underlined entries denote the best and second-best results within each group, respectively.

	SVHN	Places	FMNIST	ImageNet
EDL	$0.715\scriptstyle{\pm 0.008}$	$0.696\scriptstyle{\pm 0.006}$	$0.691\scriptstyle{\pm 0.008}$	$0.671\scriptstyle{\pm 0.004}$
Decali	$\underline{0.814\scriptstyle{\pm 0.020}}$	$\underline{0.780\scriptstyle{\pm 0.026}}$	$\underline{0.784\scriptstyle{\pm 0.020}}$	$\underline{0.755\scriptstyle{\pm 0.013}}$
DAPPr	$0.744\scriptstyle{\pm 0.003}$	$0.706\scriptstyle{\pm 0.002}$	$0.720\scriptstyle{\pm 0.007}$	$0.664\scriptstyle{\pm 0.002}$
Ours	$\mathbf{0.872\scriptstyle{\pm 0.009}}$	$\mathbf{0.836\scriptstyle{\pm 0.008}}$	$\mathbf{0.846\scriptstyle{\pm 0.007}}$	$\mathbf{0.798\scriptstyle{\pm 0.010}}$
Our-Ens	$\mathbf{0.999\scriptstyle{\pm 0.000}}$	$\mathbf{0.989\scriptstyle{\pm 0.001}}$	$\mathbf{0.997\scriptstyle{\pm 0.000}}$	$\mathbf{0.976\scriptstyle{\pm 0.001}}$
MCDO	$0.920\scriptstyle{\pm 0.002}$	$0.891\scriptstyle{\pm 0.002}$	$0.906\scriptstyle{\pm 0.005}$	$0.861\scriptstyle{\pm 0.003}$

Appendix

Table A20: AUROC (%) for OOD detection using epistemic uncertainty, with CIFAR-10 (model trained using label smoothing

\varepsilon=0.05

Explore similar work

Optimal Conformal Prediction under Epistemic Uncertainty
May 25, 2025Alireza Javanmardi, Soroush H. Zargarbashi, Santo M. A. R. Thies +3Online Conformal Prediction Predictive Uncertainty
1LMU Munich · 2MCML · 3CISPA +3
Quantification of Credal Uncertainty: A Distance-Based Approach
Mar 28, 2026Xabier Gonzalez-Garcia, Siu Lun Chau, Julian Rodemann +6Uncertainty Multiclass Classification
Dept. of Statistics, CS & Mathematics, Public Univ. of Navarre · EPIC Lab, Nanyang Technological Univ. Singapore · Rational Intelligence Lab, CISPA Helmholtz Center for Information Security +7
Can we trust our models? Epistemic calibration in second-order classification
Jun 9, 2026Arthur HoarauDynamic Epistemic Logic Calibrated Uncertainty
Université de Lorraine, CentraleSupélec · Loria, CNRS, Metz, France

May 25, 2025, stat.MLJ/K move · Enter open · S save

Optimal Conformal Prediction under Epistemic Uncertainty

Alireza Javanmardi, Soroush H. Zargarbashi, Santo M. A. R. Thies, Willem Waegeman+2

1LMU Munich · 2MCML · 3CISPA+3

Online Conformal Prediction Predictive Uncertainty Conformal Risk Control Uncertainty Integrity

Open paper AskPDF arXiv

Conformal prediction (CP) is a widely used frequentist framework to quantify uncertainty by constructing prediction sets with user-specified marginal coverage guarantees. In practice, CP is typically applied on top of probabilistic classifiers, which are able to express aleatoric but not epistemic uncertainty. In this paper, we consider the question of how to optimally employ CP on top of a more expressive formalism, namely credal sets, which can express both aleatoric and epistemic uncertainty. More specifically, we propose probabilistic Bernoulli prediction sets (BPS) and derive a variant that achieves conditional coverage for valid credal sets while remaining minimal in expected size. We then address the more realistic scenario in which the validity of the credal sets is not guaranteed. Assuming access to calibration data with ground-truth distributions over labels, we apply conformal risk control to BPS and derive a PAC-style guarantee: with high probability over the data, the achieved conditional coverage is at least the desired level. We validate our theoretical findings empirically over various datasets.

Abstract

Explore similar work

Optimal Conformal Prediction under Epistemic Uncertainty

Quantification of Credal Uncertainty: A Distance-Based Approach

Can we trust our models? Epistemic calibration in second-order classification