cs.LGMar 11, 2026

Beyond Accuracy: Reliability and Uncertainty Estimation in Convolutional Neural Networks

Authors: Sanne Ruijs, Alina Kosiakova, Farrukh Javed

Organizations: Department of Economics, Lund University, Sweden · Department of Statistics, Lund University, Sweden

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Deep neural networks (DNNs) have become integral to a wide range of scientific and practical applications due to their flexibility and strong predictive performance. Despite their accuracy, however, DNNs frequently exhibit poor calibration, often assigning overly confident probabilities to incorrect predictions. This limitation underscores the growing need for integrated mechanisms that provide reliable uncertainty estimation. In this article, we compare two prominent approaches for uncertainty quantification: a Bayesian approximation via Monte Carlo Dropout and the nonparametric Conformal Prediction framework. Both methods are assessed using two convolutional neural network architectures; H-CNN VGG16 and GoogLeNet, trained on the Fashion-MNIST dataset. The empirical results show that although H-CNN VGG16 attains higher predictive accuracy, it tends to exhibit pronounced overconfidence, whereas GoogLeNet yields better-calibrated uncertainty estimates. Conformal Prediction additionally demonstrates consistent validity by producing statistically guaranteed prediction sets, highlighting its practical value in high-stakes decision-making contexts. Overall, the findings emphasize the importance of evaluating model performance beyond accuracy alone and contribute to the development of more reliable and trustworthy deep learning systems.

Figures & tables

ModelMetricBaselineBayesian
H-CNN VGG16Accuracy (Best)92.99%92.47%
Accuracy (5-Fold Avg.)92.62%92.29%
Duration (s)12,342.6513,417.11
Trainable Parameters90,312,274
Non-Trainable Params2,944
Optimizer Parameters90,312,276
Table 1: Performance summary for H-CNN VGG16 and GoogLeNet architectures
Figure 1: Confusion Matrix for H-CNN VGG16 and GoogLeNet
Figure 2: Training vs. Validation Loss and Accuracy for H-CNN VGG16 (Fold 2)
Figure 3: Training vs. Validation Loss and Accuracy for GoogLeNet (Fold 2)
Figure 4: Sparsity vs. Threshold. Note. Sparsity as a percentage of total trainable weights
Weight Range# Weights% TotalCum. %
 (1)(2)(1)(2)(1)(2)
<0.0000134,4271,5480.040.030.040.03
0.00001–0.00005134,7574,8120.150.080.190.11
0.00005–0.0001165,4506,0330.180.100.370.21
0.0001–0.00051,297,61348,0001.440.801.811.01
0.0005–0.0011,609,45459,5121.781.003.592.01
Table 2: Sparsity Comparison Across Thresholds for (1) Bayesian H-CNN VGG16 and (2) GoogLeNet
ArchitectureECE (Baseline)ECE (Bayesian)
H-CNN VGG165.66%5.61%
GoogLeNet2.82%1.37%
Table 3: Comparison of ECE before and after Bayesian modeling
Figure 5: H-CNN VGG16 & GoogLeNet Calibration Scores
Set Size1234
GoogLeNet Count6,4311,3771857
H-CNN VGG16 Count7,551398447
Table 4: Prediction set sizes for GoogLeNet and H-CNN VGG16 models
Figure 6: Prediction Set Size Per Class H-CNN VGG16
Figure 7: Overall Confidence Intervals Bayesian
Figure 8: Uncertainty Decomposition for H-CNN VGG16 & GoogLe Net
Figure 9: Entropy per Class for Correct vs. Incorrect Predictions for H-CNN VGG16 & GoogLeNet
Figure 10: Predictive Entropy by Class for H-CNN VGG16 & GoogLeNet
Figure 11: Prediction Set Vs. Predictive Entropy for H-CNN VGG16 & GoogLeNet
Figure 12: Heatmap for H-CNN VGG16 & GoogLeNet Bayesian
Figure 13: Accuracy Vs. Loss H-CNN VGG16 (Fold 1, 3-5)
Figure 14: Accuracy Vs. Loss H-CNN VGG16 Bayesian (Fold 1-5)
Figure 15: Accuracy Vs. Loss GoogLeNet (Fold 1, 3-5)
Figure 16: Accuracy Vs. Loss GoogLeNet Bayesian(Fold 1-5)
Figure 17: Sparsity vs Threshold VGG16 & GoogLeNetBayesian Note. Sparsity as a percentage of total weights
Figure 18: Predictive Entropy for H-CNN VGG16 & GoogLeNet
Figure 19: Classwise Confidence Interval for H-CNN VGG16 & GoogLeNet
Figure 20: Classwise Confidence Interval GoogLeNet
Weight RangeNumber of WeightsPercentage of Total (%)Cumulative (%)
 VGG16GoogLeNetVGG16GoogLeNetVGG16GoogLeNet
<< 0.0000133,4191,5460.040.030.040.03
0.00001–0.00005133,3275,8190.150.080.180.11
0.00005–0.0001164,0216,0240.180.100.370.21
0.0001–0.00051,299,67747,9581.440.801.811.01
0.0005–0.0011,610,64159.8151.781.003.592.01
Table 5: Sparsity Comparison Across Thresholds for Baseline H-CNN VGG16 and GoogLeNet

Explore similar work

CardsList

  1. Dirichlet-Based Monte Carlo Dropout for Uncertainty Estimation in Neural Networks

    May 22, 2026Rouaa Hoblos, Noura Dridi, Noureddine Zerhouni +1Evidential Deep LearningUncertainty Quantification

    FEMTO-ST · Universit´e Marie et Louis Pasteur, SUPMICROTECH, CNRS, Institut FEMTO-ST, F-25000 Besan¸con, France

  2. Enhancing Deep Neural Network Reliability with Refinement and Calibration

    May 22, 2026Ramya Hebbalaguppe, Ajay Shastry, Soumya Suvra Ghosal +1Probability CalibrationUncertainty Calibration

    SIT, Indian Institute of Technology Delhi, New Delhi, India

  3. Uncertainty Quantification in CNN Through the Bootstrap of Convex Neural Networks

    Apr 11, 2026Hongfei Du, Emre Barut, Fang JinUncertainty QuantificationConvolutional Neural Networks

    The George Washington University · Amazon.com, Inc.