Ensemble-Based Dirichlet Modeling for Predictive Uncertainty and Selective Classification
Authors: Courtney Franzen, Farhad Pourkamali-Anaraki
Organizations: Department of Mathematical and Statistical Science, University of Colorado Denver, Denver, 80208, CO, USA,. · Department of Mathematical and Statistical Science, University of Colorado Denver, Denver, 80208, CO, USA.
Neural network classifiers trained with cross-entropy loss achieve strong predictive accuracy but lack the capability to provide inherent predictive uncertainty estimates, thus requiring external techniques to obtain these estimates. In addition, softmax scores for the true class can vary substantially across independent training runs, which limits the reliability of uncertainty-based decisions in downstream tasks. Evidential Deep Learning aims to address these limitations by producing uncertainty estimates in a single pass, but evidential training is highly sensitive to design choices including loss formulation, prior regularization, and activation functions. Therefore, this work introduces an alternative Dirichlet parameter estimation strategy by applying a method of moments estimator to ensembles of softmax outputs, with an optional maximum-likelihood refinement step. This ensemble-based construction decouples uncertainty estimation from the fragile evidential loss design while also mitigating the variability of single-run cross-entropy training, producing explicit Dirichlet predictive distributions. Across multiple datasets, we show that the improved stability and predictive uncertainty behavior of these ensemble-derived Dirichlet estimates translate into stronger performance in downstream uncertainty-guided applications such as prediction confidence scoring and selective classification.
Figures & tables
Figure 1: Frequency plots of softmax scores for the true class for two representative samples from a 10-class image classification task across 100 randomly initialized classifiers trained with a cross-entropy loss function. The left plot (automobile) shows a difficult sample with highly variable scores, while the right plot (deer) shows an easier sample whose scores remain concentrated but still exhibit nontrivial variability.
Figure 2: Overview of the method of moments Dirichlet construction. An ensemble of CE-trained models produces multiple softmax probability vectors for a fixed input. These are treated as empirical samples on the probability simplex, from which class-wise means and variances are computed and matched to a Dirichlet distribution, yielding per-input concentration parameters α^ .
Figure 3: Diagram of an EDL Network. A neural network maps an input feature vector x to class-specific logits, which are transformed by a nonnegative activation function to produce evidence values. These evidence values are converted into Dirichlet concentration parameters, optionally including an additive prior offset. The resulting parameter vector defines a Dirichlet distribution over categorical class probabilities from which predictive confidence and uncertainty measures can be derived.
Figure 4: Training dynamics for Dirichlet classifiers on CIFAR-100 trained from random initialization. With a linearly annealed KL term, MSE+KL training converges to stable accuracy with controlled growth of the total concentration parameter α0 . Removing KL annealing induces collapse: accuracy degrades to chance ( 1% for K=100 ) while α0 grows explosively (mean α0≈107 ), yielding degenerate confidence estimates despite decreasing loss. Notably, the collapsed regime achieves lower ECE values than the successful run ( ECEfail≈0.0025 versus ECEsucc≈0.075 ), which would typically suggest better calibration, illustrating that low ECE can mask training collapse.
Figure 5: Example confidence diagnostics. The confidence histogram (left) shows the distribution of confidence scores for correct and incorrect predictions. The reliability diagram (right) compares empirical accuracy to predicted confidence; the dashed diagonal denotes perfect calibration, with points above indicating underconfidence and below indicating overconfidence.
Figure 6: Comparison of successful versus failed EDL training variance histograms for correct and incorrect predictions. Successful training (left) shows clearer separation between the distributions, enabling selective classification thresholding, while failed training (right) shows substantial overlap within a condensed variance range.
Figure 7: Risk–coverage curves for 50 independently initialized classifiers trained with cross-entropy loss on a medical dataset. Each curve corresponds to a separately initialized model. The variability in the low-coverage regime illustrates instability in confidence ordering across CE training runs. At a fixed target risk, r=0.1 , the achieved coverage ranges from 56.39% to 69.26% , yielding a spread of 12.87 percentage points, further quantifying this variability.
Model
Base Loss
Regularization
Activation
α
MSE Only
MSE
None
Softplus
α>1e−6
MSE Clamp
MSE
KL (annealed)
Softplus
α>1e−6
MSE Soft Adapt
MSE
KL (annealed)
Adaptive Softplus
α>1e−6
MSE Plus One
MSE
KL (annealed)
Softplus
α=e+1
Exponential
MSE
KL (annealed)
Exponential
α=e+1
Digamma
Digamma
Log-evidence
Softplus
α=e+1
Table 1: Summary of EDL formulations used for comparison.
Dataset
Domain
Samples
Classes
CIFAR-10
Natural images
60,000
10
CIFAR-100
Natural images
60,000
100
DermaMNIST
Dermatology images
10,015
7
APTOS 2019
Retinal fundus images
3,662
5
Table 2: Summary of datasets used in the experiments.
Figure 8: Predictive performance metrics (accuracy, macro F1, and negative log-likelihood) for the evaluated EDL configurations on CIFAR-100.
Figure 9: Confidence diagnostics for Dirichlet classifiers on CIFAR-100. Each subfigure corresponds to a different training configuration where the left panel shows the confidence histogram and the right panel shows the reliability diagram. Confidence histograms are computed from the mean predictions across runs, and the red annotation reports the percentage of incorrect predictions with confidence greater than 0.8 . Reliability diagrams summarize calibration behavior across runs, where shaded bands denote one standard deviation and the dashed line indicates perfect calibration.
Figure 10: Predictive performance metrics (accuracy, macro F1, and negative log-likelihood) for the evaluated EDL configurations on DermaMNIST.
Figure 11: Confidence diagnostics for Dirichlet classifiers on DermaMNIST. Each subfigure corresponds to a different training configuration where the left panel shows the confidence histogram and the right panel shows the reliability diagram. Confidence histograms are computed from the mean predictions across runs, and the red annotation reports the percentage of incorrect predictions with confidence greater than 0.8 . Reliability diagrams summarize calibration behavior across runs, where shaded bands denote one standard deviation and the dashed line indicates perfect calibration.
Figure 12: Predictive performance metrics (accuracy, macro F1, and negative log-likelihood) for the evaluated ensemble-based Dirichlet estimators on CIFAR-100.
Figure 13: Reliability diagrams for ensemble-based Dirichlet estimators on CIFAR-100. The plots compare calibration behavior of method of moments and likelihood-refined Dirichlet parameter estimates across ensemble sizes M , where M denotes the number of independently trained models aggregated in the ensemble. Each curve plots empirical accuracy against predicted confidence, where perfect calibration corresponds to the diagonal.
Figure 14: CIFAR-100 confidence histograms for moment-based and likelihood-refined Dirichlet estimators across ensemble sizes. Histograms are conditioned on prediction correctness, with a vertical dashed line at the 0.8 confidence threshold and the percentage of incorrect predictions above this threshold reported.
Figure 15: Predictive performance metrics (accuracy, macro F1, and negative log-likelihood) for the evaluated ensemble-based Dirichlet estimators on DermaMNIST.
Figure 16: Reliability diagrams for ensemble-based Dirichlet estimators on DermaMNIST. The plots compare calibration behavior of method of moments and likelihood-refined Dirichlet parameter estimates across ensemble sizes M , where M denotes the number of independently trained models aggregated in the ensemble. Each curve plots empirical accuracy against predicted confidence, where perfect calibration corresponds to the diagonal.
Figure 17: DermaMNIST confidence histograms for moment-based and likelihood-refined Dirichlet estimators across ensemble sizes. Histograms are conditioned on prediction correctness, with a vertical dashed line at the 0.8 confidence threshold and the percentage of incorrect predictions above this threshold reported.
Figure 18: Variance-based selective classification on DermaMNIST using ensemble-based Dirichlet parameters estimated via the method of moments. Left: Risk–coverage curve with target risk r=0.1 (dashed line) selected on a held-out calibration set. Right: Test set Dirichlet total predictive variance distributions, separated by correctness, with the corresponding abstention threshold τ indicated by a dashed line.
Figure 19: Variance-based selective classification on DermaMNIST for evidential Dirichlet models trained with Digamma (left column) and Exponential (right column) losses. (a) Risk–coverage curves with target risk r=0.1 (dashed line) and thresholds selected on a held-out calibration set. (b) Test set Dirichlet total predictive variance distributions, separated by correctness, with the corresponding abstention threshold indicated.
Model
Full Test
Retained Test
N
Acc
F1
NLL
N
Acc
F1
NLL
Digamma
1002
0.7605
0.3963
1.7061
634
0.9132
0.4346
1.1109
Exponential
1002
0.7515
0.4366
0.7215
559
0.9141
0.5204
0.3496
MoM Alphas ( M=50 )
1002
0.7974
0.5916
0.5552
701
0.8987
0.6292
0.3124
Table 3: DermaMNIST performance before and after variance-based abstention. Retained test metrics are computed after enforcing r=0.1 via a calibration-selected threshold. Best values per column are shown in bold. Here N denotes the number of samples evaluated in each test subset, and M denotes the number of independently trained models aggregated in the ensemble.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 20: Predictive performance metrics (accuracy, macro F1, and negative log-likelihood) for the evaluated EDL configurations on CIFAR-10.
Figure 21: Confidence diagnostics for Dirichlet classifiers on CIFAR-10. Each subfigure corresponds to a different training configuration where the left panel shows the confidence histogram and the right panel shows the reliability diagram. Confidence histograms are computed from the mean predictions across runs, and the red annotation reports the percentage of incorrect predictions with confidence greater than 0.8 . Reliability diagrams summarize calibration behavior across runs, where shaded bands denote one standard deviation and the dashed line indicates perfect calibration.
Figure 22: Predictive performance metrics (accuracy, macro F1, and negative log-likelihood) for the evaluated EDL configurations on APTOS.
Figure 23: Confidence diagnostics for Dirichlet classifiers on APTOS. Each subfigure corresponds to a different training configuration where the left panel shows the confidence histogram and the right panel shows the reliability diagram. Confidence histograms are computed from the mean predictions across runs, and the red annotation reports the percentage of incorrect predictions with confidence greater than 0.8 . Reliability diagrams summarize calibration behavior across runs, where shaded bands denote one standard deviation and the dashed line indicates perfect calibration.
Figure 24: Predictive performance metrics (accuracy, macro F1, and negative log-likelihood) for the evaluated ensemble-based Dirichlet estimators on CIFAR-10.
Figure 25: Reliability diagrams for ensemble-based Dirichlet estimators on CIFAR-10. The plots compare calibration behavior of method of moments and likelihood-refined Dirichlet parameter estimates across ensemble sizes M , where M denotes the number of independently trained models aggregated in the ensemble. Each curve plots empirical accuracy against predicted confidence, where perfect calibration corresponds to the diagonal.
Figure 26: CIFAR-10 confidence histograms for moment-based and likelihood-refined Dirichlet estimators across ensemble sizes. Histograms are conditioned on prediction correctness, with a vertical dashed line at the 0.8 confidence threshold and the percentage of incorrect predictions above this threshold reported.
Figure 27: Predictive performance metrics (accuracy, macro F1, and negative log-likelihood) for the evaluated ensemble-based Dirichlet estimators on APTOS.
Figure 28: Reliability diagrams for ensemble-based Dirichlet estimators on APTOS. The plots compare calibration behavior of method of moments and likelihood-refined Dirichlet parameter estimates across ensemble sizes M , where M denotes the number of independently trained models aggregated in the ensemble. Each curve plots empirical accuracy against predicted confidence, where perfect calibration corresponds to the diagonal.
Figure 29: APTOS confidence histograms for moment-based and likelihood-refined Dirichlet estimators across ensemble sizes. Histograms are conditioned on prediction correctness, with a vertical dashed line at the 0.8 confidence threshold and the percentage of incorrect predictions above this threshold reported.
Real-world sensor-based learning systems require uncertainty estimation that is both reliable and computationally efficient. Evidential Deep Learning (EDL) provides single-pass uncertainty estimation by modeling the class probabilities via Dirichlet distributions, where the Dirichlet parameters are predicted by a learned neural network mapping. However, this approach can lead to computational challenges, as Dirichlet expected objectives are more complex than standard supervised learning losses, complicating their analysis and implementation. We address this issue by approximating the objective of the first-order empirical risk minimization problem induced by EDL with a plug-in loss evaluated at the Dirichlet mean and show that, under mild assumptions, the approximation error decays with growing evidence for a broad class of loss functions, including mean-squared error and cross-entropy loss. As a special case, our analysis provides justification for the use of softmax in the context of uncertainty estimation, since under a particular evidence-to-Dirichlet mapping, our framework includes the standard softmax classifier. We validate the proposed simplified objectives on the Google Speech Commands dataset and show that they achieve predictive accuracy and selective prediction performance comparable to classical EDL, while being simpler to implement using standard deep learning losses and training pipelines. To the best of our knowledge, this empirical analysis is the first to obtain coverage-accuracy trade-offs for speech recognition tasks through EDL.
Berk Hayta, Hannah Laus, Simon Mittermaier +1
TU Munich · TU Munich & MCML · Infineon Technologies +1
Deep neural networks achieve impressive results across diverse applications, yet their overconfidence on unseen inputs necessitates reliable epistemic uncertainty modeling. Existing methods for uncertainty modeling face a fundamental dilemma: Bayesian approaches provide principled estimates but remain computationally prohibitive, while efficient second-order predictors lack rigorous connections between their specific objectives and epistemic uncertainty quantification. To resolve this dilemma, we introduce Dirichlet-approximated possibilistic posterior predictions (DAPPr), a principled framework grounded in possibility theory. We define a possibilistic posterior over parameters, project it to the prediction space via supremum operators, and approximate the projected posterior using learnable Dirichlet possibility functions. This projection-and-approximation strategy yields a simple training objective with closed-form solutions. Despite its simplicity, extensive experiments across diverse benchmarks show that DAPPr achieves competitive or superior uncertainty quantification performance over state-of-the-art second-order predictors while maintaining both principled derivation and computational efficiency. Code is available at https://github.com/MaxwellYaoNi/DAPPr.
Yao Ni, Jeremie Houssineau, Yew-Soon Ong +1
Nanyang Technological University · A*STAR · The University of New South Wales +1
Pretrained models have become standard in both vision and language, yet they typically do not provide reliable measures of confidence. Existing uncertainty estimation methods, such as deep ensembles and MC dropout, are often too computationally expensive to deploy in practice. Evidential Deep Learning (EDL) offers a more efficient alternative, but it requires models to be trained to output evidential quantities from the start, which is rarely true for pretrained networks. To enable EDL-style uncertainty estimation in pretrained models, we propose the Evidential Transformation Network (ETN), a lightweight post-hoc module that converts a pretrained predictor into an evidential model. ETN operates in logit space: it learns a sample-dependent affine transformation of the logits and interprets the transformed outputs as parameters of a Dirichlet distribution for uncertainty estimation. We evaluate ETN on image classification and large language model question-answering benchmarks under both in-distribution and out-of-distribution settings. ETN consistently improves uncertainty estimation over post-hoc baselines while preserving accuracy and adding only minimal computational overhead.