Uncertainty Identifies Difficult Samples Across Methods: A Multi-Task Study on a Heterogeneous Skin Lesion Dataset

Leon Koole; Jiapan Guo; Matias Valdenegro-Toro

cs.CVAug 14, 2026

Uncertainty Identifies Difficult Samples Across Methods: A Multi-Task Study on a Heterogeneous Skin Lesion Dataset

Authors: Leon Koole, Jiapan Guo, Matias Valdenegro-Toro

Organizations: Bernoulli Institute, University of Groningen, Netherlands

Abstract

Skin lesion classifiers can be confidently wrong on the cases that matter most, so knowing when a prediction should not be trusted is clinically as useful as the prediction. We study uncertainty quantification on a dataset pooled from many ISIC sources, with a shared backbone and two jointly learned heads: a binary malignant versus non-malignant head and a five-class diagnostic head. Five UQ methods (MC Dropout, DropConnect, Flipout, Deep Ensembles, DUQ) are compared on accuracy, calibration, uncertainty decomposition, and risk-coverage. Difficulty is largely method-agnostic: even methods with narrow entropy distributions rank the same samples as hard (per-sample entropy correlations of $0.54$ to $0.91$ ). The choice of method matters more for calibration and uncertainty decomposition, where Deep Ensembles is the clear winner, than for finding difficult cases. The ranking is also good enough that deferring the most uncertain cases removes a disproportionate share of errors, supporting uncertainty-based selective referral, evaluated here in-distribution only.

Figures & tables

Panel of Figure 1 — Figure 1: One example test image per consensus difficulty tier on H1, where the tier counts how many of the five UQ methods misclassify the sample, from Tier 0 (all five correct) to Tier 5 (all five wrong).

	H1 (binary)					H2 (multiclass)
Method	Acc	ECE	Epi%	AUROC	AURC	Acc	ECE	Epi%	AUROC	AURC
Baseline	0.911	0.035	–	0.881	0.016	0.870	0.059	–	0.868	0.028
Deep Ensembles	0.926	0.013	16.1	0.889	0.012	0.887	0.019	23.7	0.884	0.022
MC Dropout	0.927	0.043	38.7	0.877	0.014	0.902	0.059	43.0	0.877	0.020
DropConnect	0.927	0.057	2.6	0.885	0.014	0.902	0.078	3.3	0.878	0.020
Flipout	0.868	0.020	0.0	0.842	0.034	0.763	0.057	0.1	0.833	0.075

Table 1: Test-set comparison, both heads. ECE = expected calibration error. Epi% = epistemic share of predictive entropy (mutual information over predictive entropy). AURC = area under the risk–coverage curve. Lower ECE/AURC is better. “–” marks no epistemic decomposition (single-pass DUQ and the no-UQ Baseline).

Panel of Figure 2 — Figure 2: Mean predictive entropy (left) and overconfidence (right) by LOMO difficulty tier, H1. The tier-4 entropy dip with peaking overconfidence marks systematically misleading cases (confidently wrong, not uncertain).

Panel of Figure 3 — Figure 3: Left shows spearman rank correlation of per-sample predictive entropy between method pairs, H1 (lower triangle) and H2 (upper triangle). High correlations across very different methods indicate difficulty is mostly carried by the sample. Right shows Risk–coverage curves, H1. Lower error at a given coverage means uncertainty ranks errors better, which selective referral exploits [ 19 , 6 ] . is a softmax baseline.

Variable	df	$\eta^{2}$	Partial $\eta^{2}$	$p$	$\eta^{2}$	Partial $\eta^{2}$
	Binary Head (H1)			Multiclass Head (H2)
Confirmation type	4	0.262	0.285		0.197	0.211
Secondary diagnosis	4	0.048	0.067		0.008	0.01
Attribution	10	0.017	0.025		0.026	0.034
Image type	2	0.007	0.01		0.011	0.015
Anatomical site	7	0.007	0.01		0.015	0.02

Table 2: Multivariate regression of predictive entropy on metadata variables (Type II SS, Flipout). p-values are denoted as: *:

p<0.05

, **:

p<0.01

, ***:

p<0.001

Variable	df	$\eta^{2}$	Part. $\eta^{2}$	$p$	$\eta^{2}$	Part. $\eta^{2}$	$p$	$\eta^{2}$	Part. $\eta^{2}$	$p$	$\eta^{2}$	Part. $\eta^{2}$
	DUQ						Deep Ens.
	H1			H2			H1			H2
Confirmation type	4	0.069	0.073		0.103	0.106		0.143	0.152		0.176	0.188
Secondary diagnosis	4	0.021	0.024		0.003	0.003	0.033	0.027	0.033		0.006	0.008
Attribution	10	0.015	0.017		0.012	0.014		0.021	0.025		0.023	0.029
Image type	2	0.006	0.007		0.005	0.005		0.008	0.01		0.013	0.017

Table 3: Multivariate regression of predictive entropy on metadata variables (Type II SS) for DUQ and Deep Ensembles). p-values are denoted as: *:

p<0.05

, **:

p<0.01

, ***:

p<0.001

Variable	df	$\eta^{2}$	Part. $\eta^{2}$	$p$	$\eta^{2}$	Part. $\eta^{2}$	$p$	$\eta^{2}$	Part. $\eta^{2}$	$p$	$\eta^{2}$	Part. $\eta^{2}$
	MC-Dropout						MC-DropConnect
	H1			H2			H1			H2
Confirmation type	4	0.025	0.026		0.031	0.033		0.025	0.026		0.028	0.029
Secondary diagnosis	4	0.004	0.04	0.003	0.011	0.012		0.004	0.005	0.003	0.005	0.006
Attribution	10	0.013	0.013		09.01	0.011		0.024	0.025		0.017	0.018
Image type	2	0.003	0.003	0.003	0.005	0.005		0.012	0.013		0.01	0.01

Table 4: Multivariate regression of predictive entropy on metadata variables (Type II SS) for MC-Dropout and MC-DropConnect). p-values are denoted as: *:

p<0.05

, **:

p<0.01

, ***:

p<0.001

UQ Method	Test Set	Indet. Set	$\Delta\%$	Test Set	Indet. Set	$\Delta\%$
	Binary (H1)			Multiclass (H2)
Deep Ensembles	0.160	0.392	145.9%	0.236	0.651	176.3%
MC-Dropout	0.071	0.233	228.3%	0.096	0.388	253.4%
MC-DropConnect	0.039	0.144	268.5%	0.050	0.202	305.6%
Flipout	0.255	0.420	64.2%	0.457	0.838	83.6%
DUQ	0.134	0.337	151.8%	0.204	0.497	144.2%

Table 5: Mean predictive entropy for test and indeterminate sets, with relative increase (

\Delta\%

). A larger

\Delta\%

means the method responds more to sample difficulty.

	MC-DC	MC-DO	DUQ	Deep Ens.	Flipout.
MC-DC	–	0.72 [0.71, 0.73]	0.69 [0.68, 0.70]	0.74 [0.73, 0.75]	0.53 [0.52, 0.55]
MC-DO	0.70 [0.69, 0.71]	–	0.70 [0.69, 0.71]	0.79 [0.78, 0.80]	0.61 [0.59, 0.62]
DUQ	0.68 [0.67, 0.69]	0.75 [0.74, 0.76]	–	0.85 [0.85, 0.86]	0.66 [0.64, 0.67]
Deep Ens.	0.72 [0.71, 0.74]	0.79 [0.78, 0.80]	0.91 [0.90, 0.91]	–	0.75 [0.74, 0.76]
Flipout	0.54 [0.52, 0.56]	0.63 [0.61, 0.65]	0.75 [0.74, 0.76]	0.79 [0.78, 0.80]	–

Table 6: Reproduction of Table 3(a) with 95% confidence intervals using the Fisher approximation. H1 (lower triangle) and H2 (upper triangle).

Method	H1	H2	H1	H2	H1	H2
	Original		Black bg ( $\Delta$ )		+markers removed ( $\Delta$ )
Deep Ensembles	0.504	0.475	$-0.074$	$+0.068$	$-0.114$	$+0.075$
MC Dropout	0.224	0.234	$-0.066$	$-0.045$	$-0.097$	$-0.099$
DropConnect	0.283	0.286	$-0.219$	$-0.188$	$-0.158$	$-0.154$
Flipout	0.590	0.805	$-0.119$	$-0.106$	$-0.135$	$-0.099$
DUQ	0.408	0.462	$-0.092$	$-0.052$	$-0.174$	$-0.052$

Table 7: Mean predictive entropy on the 16 Prince Alfred images and its change after editing the background. Lower entropy means the model became more confident. “Black bg” replaces the white background with black. “+markers removed” additionally erases the surgical markers.

Panel of Figure 4 — Figure 4: Examples of visually similar benign–malignant lesion pairs from the ISIC archive. Each column shows a benign (top) and malignant (bottom) lesion paired by similarity in a neural network’s learned feature representation. The visual diversity across columns demonstrates intra-class variation.

Panel of Figure 5 — Figure 5: Six melanoma images from different data sources and imaging modalities. The images differ substantially in background color, magnification, lighting, and resolution due to differences in clinical equipment and imaging protocols.

Panel of Figure 6 — Figure 6: Five indeterminate-set samples with the highest predictive entropy on the multiclass head, ranked by Deep Ensembles.

Explore similar work

Uncertainty-Aware and Explainable Ensemble Deep Learning Framework for Multi-Class Skin Lesion Classification
Aug 11, 2026Rofiqul Islam, Lilatul FerdouseSkin Lesion Classification Deep Ensembles
Department of Computer Science and Physics, Wilfrid Laurier University, Waterloo, ON, Canada
Uncertainty-Guided Dual-Domain Learning for Reliable Skin Lesion Segmentation
May 10, 2026Duwei Dai, Caixia Dong, Guowei Dai +6Medical Image Segmentation Skin Lesion Segmentation
Department of Dermatology, the Second Affiliated Hospital of Xi’an Jiaotong University, Xi’an, 710004, Shaanxi, China. · National-Local Joint Engineering Research Center of Biodiagnosis & Biotherapy, the Second Affiliated Hospital of Xi’an Jiaotong University, Xi’an, 710004, Shaanxi, China. · School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, 710049, Shaanxi, China. +3
SpatialUQ: Post-Hoc Uncertainty Quantification from Spatial Consistency in Black-Box Vision Models
Oct 7, 2026Md Kawsher Mahbub, Milon Biswas, Mirza Niaz Morshed +1Uncertainty Quantification Uncertainty Calibration
NPI University of Bangladesh · Towson University · University of Asia Pacific

Aug 11, 2026J/K move · Enter open · S save

Uncertainty-Aware and Explainable Ensemble Deep Learning Framework for Multi-Class Skin Lesion Classification

Rofiqul Islam, Lilatul Ferdouse

Department of Computer Science and Physics, Wilfrid Laurier University, Waterloo, ON, Canada

Skin Lesion Classification Deep Ensembles Medical Image Classification Explainable Medical Image Analysis

Open paper AskPDF arXiv

Skin cancer diagnosis from dermoscopic images remains challenging due to high intra-class variability, inter-class similarity, class imbalance, and the limited interpretability of deep learning models. This paper proposes an uncertainty-aware and explainable deep learning framework for multi-class skin lesion classification. The framework combines a vision transformer model (MaxViT-Tiny) with CNN-based models (ConvNeXt-Tiny and EfficientNetV2-B0) through deep ensemble learning. Monte Carlo (MC) Dropout estimates predictive uncertainty and identifies unreliable predictions, while Grad-CAM++, an explainable AI (XAI) technique, provides visual explanations by highlighting lesion regions that influence model decisions. Evaluated on the HAM10000 dataset, the framework achieves 96% accuracy and 99% ROC-AUC under uncertainty-aware filtering (entropy < 1.0, confidence >= 0.7), with macro-average precision, recall, and F1-score of 94%, 95%, and 95%, respectively, and 96% weighted-average scores across all three metrics. The results demonstrate accurate, interpretable, and uncertainty-aware skin lesion classification for trustworthy computer-aided diagnosis.

Abstract

Explore similar work

Uncertainty-Aware and Explainable Ensemble Deep Learning Framework for Multi-Class Skin Lesion Classification

Uncertainty-Guided Dual-Domain Learning for Reliable Skin Lesion Segmentation

SpatialUQ: Post-Hoc Uncertainty Quantification from Spatial Consistency in Black-Box Vision Models