Ensemble-Based Dirichlet Modeling for Predictive Uncertainty and Selective Classification

Courtney Franzen; Farhad Pourkamali-Anaraki

stat.MLApr 7, 2026

Ensemble-Based Dirichlet Modeling for Predictive Uncertainty and Selective Classification

Authors: Courtney Franzen, Farhad Pourkamali-Anaraki

Organizations: Department of Mathematical and Statistical Science, University of Colorado Denver, Denver, 80208, CO, USA,. · Department of Mathematical and Statistical Science, University of Colorado Denver, Denver, 80208, CO, USA.

Abstract

Neural network classifiers trained with cross-entropy loss achieve strong predictive accuracy but lack the capability to provide inherent predictive uncertainty estimates, thus requiring external techniques to obtain these estimates. In addition, softmax scores for the true class can vary substantially across independent training runs, which limits the reliability of uncertainty-based decisions in downstream tasks. Evidential Deep Learning aims to address these limitations by producing uncertainty estimates in a single pass, but evidential training is highly sensitive to design choices including loss formulation, prior regularization, and activation functions. Therefore, this work introduces an alternative Dirichlet parameter estimation strategy by applying a method of moments estimator to ensembles of softmax outputs, with an optional maximum-likelihood refinement step. This ensemble-based construction decouples uncertainty estimation from the fragile evidential loss design while also mitigating the variability of single-run cross-entropy training, producing explicit Dirichlet predictive distributions. Across multiple datasets, we show that the improved stability and predictive uncertainty behavior of these ensemble-derived Dirichlet estimates translate into stronger performance in downstream uncertainty-guided applications such as prediction confidence scoring and selective classification.

Figures & tables

Figure 1: Frequency plots of softmax scores for the true class for two representative samples from a 10-class image classification task across 100 randomly initialized classifiers trained with a cross-entropy loss function. The left plot (automobile) shows a difficult sample with highly variable scores, while the right plot (deer) shows an easier sample whose scores remain concentrated but still exhibit nontrivial variability.

Figure 2: Overview of the method of moments Dirichlet construction. An ensemble of CE-trained models produces multiple softmax probability vectors for a fixed input. These are treated as empirical samples on the probability simplex, from which class-wise means and variances are computed and matched to a Dirichlet distribution, yielding per-input concentration parameters $\hat{\boldsymbol{\alpha}}$ .

Figure 3: Diagram of an EDL Network. A neural network maps an input feature vector $\mathbf{x}$ to class-specific logits, which are transformed by a nonnegative activation function to produce evidence values. These evidence values are converted into Dirichlet concentration parameters, optionally including an additive prior offset. The resulting parameter vector defines a Dirichlet distribution over categorical class probabilities from which predictive confidence and uncertainty measures can be derived.

Panel of Figure 4 — Figure 4: Training dynamics for Dirichlet classifiers on CIFAR-100 trained from random initialization. With a linearly annealed KL term, MSE+KL training converges to stable accuracy with controlled growth of the total concentration parameter $\alpha_{0}$ . Removing KL annealing induces collapse: accuracy degrades to chance ( $1\%$ for $K=100$ ) while $\alpha_{0}$ grows explosively (mean $\alpha_{0}\approx 10^{7}$ ), yielding degenerate confidence estimates despite decreasing loss. Notably, the collapsed regime achieves lower ECE values than the successful run ( $\text{ECE}_{\text{fail}}$ $\approx$ $0.0025$ versus $\text{ECE}_{\text{succ}}$ $\approx$ $0.075$ ), which would typically suggest better calibration, illustrating that low ECE can mask training collapse.

Figure 5: Example confidence diagnostics. The confidence histogram (left) shows the distribution of confidence scores for correct and incorrect predictions. The reliability diagram (right) compares empirical accuracy to predicted confidence; the dashed diagonal denotes perfect calibration, with points above indicating underconfidence and below indicating overconfidence.

Panel of Figure 6 — Figure 6: Comparison of successful versus failed EDL training variance histograms for correct and incorrect predictions. Successful training (left) shows clearer separation between the distributions, enabling selective classification thresholding, while failed training (right) shows substantial overlap within a condensed variance range.

Figure 7: Risk–coverage curves for 50 independently initialized classifiers trained with cross-entropy loss on a medical dataset. Each curve corresponds to a separately initialized model. The variability in the low-coverage regime illustrates instability in confidence ordering across CE training runs. At a fixed target risk, $r$ $=$ $0.1$ , the achieved coverage ranges from $56.39\%$ to $69.26\%$ , yielding a spread of $12.87$ percentage points, further quantifying this variability.

Model	Base Loss	Regularization	Activation	$\boldsymbol{\alpha}$
MSE Only	MSE	None	Softplus	$\boldsymbol{\alpha}>1e-6$
MSE Clamp	MSE	KL (annealed)	Softplus	$\boldsymbol{\alpha}>1e-6$
MSE Soft Adapt	MSE	KL (annealed)	Adaptive Softplus	$\boldsymbol{\alpha}>1e-6$
MSE Plus One	MSE	KL (annealed)	Softplus	$\boldsymbol{\alpha}=\mathbf{e}+\mathbf{1}$
Exponential	MSE	KL (annealed)	Exponential	$\boldsymbol{\alpha}=\mathbf{e}+\mathbf{1}$
Digamma	Digamma	Log-evidence	Softplus	$\boldsymbol{\alpha}=\mathbf{e}+\mathbf{1}$

Table 1: Summary of EDL formulations used for comparison.

Dataset	Domain	Samples	Classes
CIFAR-10	Natural images	60,000	10
CIFAR-100	Natural images	60,000	100
DermaMNIST	Dermatology images	10,015	7
APTOS 2019	Retinal fundus images	3,662	5

Table 2: Summary of datasets used in the experiments.

Figure 8: Predictive performance metrics (accuracy, macro F1, and negative log-likelihood) for the evaluated EDL configurations on CIFAR-100.

Panel of Figure 9 — Figure 9: Confidence diagnostics for Dirichlet classifiers on CIFAR-100. Each subfigure corresponds to a different training configuration where the left panel shows the confidence histogram and the right panel shows the reliability diagram. Confidence histograms are computed from the mean predictions across runs, and the red annotation reports the percentage of incorrect predictions with confidence greater than $0.8$ . Reliability diagrams summarize calibration behavior across runs, where shaded bands denote one standard deviation and the dashed line indicates perfect calibration.

Figure 10: Predictive performance metrics (accuracy, macro F1, and negative log-likelihood) for the evaluated EDL configurations on DermaMNIST.

Panel of Figure 11 — Figure 11: Confidence diagnostics for Dirichlet classifiers on DermaMNIST. Each subfigure corresponds to a different training configuration where the left panel shows the confidence histogram and the right panel shows the reliability diagram. Confidence histograms are computed from the mean predictions across runs, and the red annotation reports the percentage of incorrect predictions with confidence greater than $0.8$ . Reliability diagrams summarize calibration behavior across runs, where shaded bands denote one standard deviation and the dashed line indicates perfect calibration.

Figure 12: Predictive performance metrics (accuracy, macro F1, and negative log-likelihood) for the evaluated ensemble-based Dirichlet estimators on CIFAR-100.

Figure 13: Reliability diagrams for ensemble-based Dirichlet estimators on CIFAR-100. The plots compare calibration behavior of method of moments and likelihood-refined Dirichlet parameter estimates across ensemble sizes $M$ , where $M$ denotes the number of independently trained models aggregated in the ensemble. Each curve plots empirical accuracy against predicted confidence, where perfect calibration corresponds to the diagonal.

Figure 14: CIFAR-100 confidence histograms for moment-based and likelihood-refined Dirichlet estimators across ensemble sizes. Histograms are conditioned on prediction correctness, with a vertical dashed line at the $0.8$ confidence threshold and the percentage of incorrect predictions above this threshold reported.

Figure 15: Predictive performance metrics (accuracy, macro F1, and negative log-likelihood) for the evaluated ensemble-based Dirichlet estimators on DermaMNIST.

Figure 16: Reliability diagrams for ensemble-based Dirichlet estimators on DermaMNIST. The plots compare calibration behavior of method of moments and likelihood-refined Dirichlet parameter estimates across ensemble sizes $M$ , where $M$ denotes the number of independently trained models aggregated in the ensemble. Each curve plots empirical accuracy against predicted confidence, where perfect calibration corresponds to the diagonal.

Figure 17: DermaMNIST confidence histograms for moment-based and likelihood-refined Dirichlet estimators across ensemble sizes. Histograms are conditioned on prediction correctness, with a vertical dashed line at the $0.8$ confidence threshold and the percentage of incorrect predictions above this threshold reported.

Panel of Figure 18 — Figure 18: Variance-based selective classification on DermaMNIST using ensemble-based Dirichlet parameters estimated via the method of moments. Left: Risk–coverage curve with target risk $r=0.1$ (dashed line) selected on a held-out calibration set. Right: Test set Dirichlet total predictive variance distributions, separated by correctness, with the corresponding abstention threshold $\tau$ indicated by a dashed line.

Panel of Figure 19 — Figure 19: Variance-based selective classification on DermaMNIST for evidential Dirichlet models trained with Digamma (left column) and Exponential (right column) losses. (a) Risk–coverage curves with target risk $r$ $=$ $0.1$ (dashed line) and thresholds selected on a held-out calibration set. (b) Test set Dirichlet total predictive variance distributions, separated by correctness, with the corresponding abstention threshold indicated.

Model	Full Test				Retained Test
	$N$	Acc	F1	NLL	$N$	Acc	F1	NLL
Digamma	1002	0.7605	0.3963	1.7061	634	0.9132	0.4346	1.1109
Exponential	1002	0.7515	0.4366	0.7215	559	0.9141	0.5204	0.3496
MoM Alphas ( $M{=}50$ )	1002	0.7974	0.5916	0.5552	701	0.8987	0.6292	0.3124

Table 3: DermaMNIST performance before and after variance-based abstention. Retained test metrics are computed after enforcing

r{=}0.1

via a calibration-selected threshold. Best values per column are shown in bold. Here

N

denotes the number of samples evaluated in each test subset, and

M

denotes the number of independently trained models aggregated in the ensemble.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 20: Predictive performance metrics (accuracy, macro F1, and negative log-likelihood) for the evaluated EDL configurations on CIFAR-10.

Panel of Figure 21 — Figure 21: Confidence diagnostics for Dirichlet classifiers on CIFAR-10. Each subfigure corresponds to a different training configuration where the left panel shows the confidence histogram and the right panel shows the reliability diagram. Confidence histograms are computed from the mean predictions across runs, and the red annotation reports the percentage of incorrect predictions with confidence greater than $0.8$ . Reliability diagrams summarize calibration behavior across runs, where shaded bands denote one standard deviation and the dashed line indicates perfect calibration.

Figure 22: Predictive performance metrics (accuracy, macro F1, and negative log-likelihood) for the evaluated EDL configurations on APTOS.

Panel of Figure 23 — Figure 23: Confidence diagnostics for Dirichlet classifiers on APTOS. Each subfigure corresponds to a different training configuration where the left panel shows the confidence histogram and the right panel shows the reliability diagram. Confidence histograms are computed from the mean predictions across runs, and the red annotation reports the percentage of incorrect predictions with confidence greater than $0.8$ . Reliability diagrams summarize calibration behavior across runs, where shaded bands denote one standard deviation and the dashed line indicates perfect calibration.

Figure 24: Predictive performance metrics (accuracy, macro F1, and negative log-likelihood) for the evaluated ensemble-based Dirichlet estimators on CIFAR-10.

Figure 25: Reliability diagrams for ensemble-based Dirichlet estimators on CIFAR-10. The plots compare calibration behavior of method of moments and likelihood-refined Dirichlet parameter estimates across ensemble sizes $M$ , where $M$ denotes the number of independently trained models aggregated in the ensemble. Each curve plots empirical accuracy against predicted confidence, where perfect calibration corresponds to the diagonal.

Figure 26: CIFAR-10 confidence histograms for moment-based and likelihood-refined Dirichlet estimators across ensemble sizes. Histograms are conditioned on prediction correctness, with a vertical dashed line at the $0.8$ confidence threshold and the percentage of incorrect predictions above this threshold reported.

Figure 27: Predictive performance metrics (accuracy, macro F1, and negative log-likelihood) for the evaluated ensemble-based Dirichlet estimators on APTOS.

Figure 28: Reliability diagrams for ensemble-based Dirichlet estimators on APTOS. The plots compare calibration behavior of method of moments and likelihood-refined Dirichlet parameter estimates across ensemble sizes $M$ , where $M$ denotes the number of independently trained models aggregated in the ensemble. Each curve plots empirical accuracy against predicted confidence, where perfect calibration corresponds to the diagonal.

Figure 29: APTOS confidence histograms for moment-based and likelihood-refined Dirichlet estimators across ensemble sizes. Histograms are conditioned on prediction correctness, with a vertical dashed line at the $0.8$ confidence threshold and the percentage of incorrect predictions above this threshold reported.

Uncertainty Quantification

Uncertainty quantification estimates how reliable predictions and scientific inferences are, and how uncertainty propagates into decisions.

See all 649 papers

Plug-in Losses for Evidential Deep Learning: A Simplified Framework for Uncertainty Estimation that Includes the Softmax Classifier
May 21, 2026Berk Hayta, Hannah Laus, Simon Mittermaier +1Evidential Deep Learning
TU Munich · TU Munich & MCML · Infineon Technologies +1
Possibilistic Predictive Uncertainty for Deep Learning
May 1, 2026Yao Ni, Jeremie Houssineau, Yew-Soon Ong +1Uncertainty Quantification Epistemic Uncertainty
Nanyang Technological University · A*STAR · The University of New South Wales +1
Evidential Transformation Network: Turning Pretrained Models into Evidential Models for Post-hoc Uncertainty Estimation
Apr 9, 2026Yongchan Chun, Chanhee Park, Jeongho Yoon +2Uncertainty Quantification Evidential Deep Learning
Korea University · Konkuk University

May 21, 2026J/K move · Enter open · S save

Plug-in Losses for Evidential Deep Learning: A Simplified Framework for Uncertainty Estimation that Includes the Softmax Classifier

Berk Hayta, Hannah Laus, Simon Mittermaier, Felix Krahmer

TU Munich · TU Munich & MCML · Infineon Technologies+1

Evidential Deep Learning

Open paper AskPDF arXiv

Real-world sensor-based learning systems require uncertainty estimation that is both reliable and computationally efficient. Evidential Deep Learning (EDL) provides single-pass uncertainty estimation by modeling the class probabilities via Dirichlet distributions, where the Dirichlet parameters are predicted by a learned neural network mapping. However, this approach can lead to computational challenges, as Dirichlet expected objectives are more complex than standard supervised learning losses, complicating their analysis and implementation. We address this issue by approximating the objective of the first-order empirical risk minimization problem induced by EDL with a plug-in loss evaluated at the Dirichlet mean and show that, under mild assumptions, the approximation error decays with growing evidence for a broad class of loss functions, including mean-squared error and cross-entropy loss. As a special case, our analysis provides justification for the use of softmax in the context of uncertainty estimation, since under a particular evidence-to-Dirichlet mapping, our framework includes the standard softmax classifier. We validate the proposed simplified objectives on the Google Speech Commands dataset and show that they achieve predictive accuracy and selective prediction performance comparable to classical EDL, while being simpler to implement using standard deep learning losses and training pipelines. To the best of our knowledge, this empirical analysis is the first to obtain coverage-accuracy trade-offs for speech recognition tasks through EDL.

Abstract

Uncertainty Quantification

Explore similar work

Plug-in Losses for Evidential Deep Learning: A Simplified Framework for Uncertainty Estimation that Includes the Softmax Classifier

Possibilistic Predictive Uncertainty for Deep Learning

Evidential Transformation Network: Turning Pretrained Models into Evidential Models for Post-hoc Uncertainty Estimation