Beyond Performance Metrics: Uncertainty Mapping of Label Ambiguity in Fazekas Score Prediction

Susanne Schmid; Johanna Ospel; Richard Frayne; Roberto Souza

cs.CVSep 15, 2026

Beyond Performance Metrics: Uncertainty Mapping of Label Ambiguity in Fazekas Score Prediction

Authors: Susanne Schmid, Johanna Ospel, Richard Frayne, Roberto Souza

Organizations: Schulich School of Engineering, Department of Electrical and Software Engineering, University of Calgary, Calgary, T2N 1N4, AB, Canada · Hotchkiss Brain Institute, University of Calgary, Calgary, T2N 1N4, AB, Canada · Department of Radiology and Clinical Neurosciences, University of Calgary, Calgary, T2N 1N4, AB, Canada · Calgary Image Processing and Analysis Centre, Foothills Medical Centre, Calgary, T2N 2T9, AB, Canada ·

Abstract

Reference labels used to train medical image classification models are not always as certain as they may appear, and this uncertainty has implications on performance metrics. In this study, we propose a framework to analyze model performance for periventricular Fazekas score prediction that goes beyond conventional metrics. The Fazekas score is an ordinal visual rating scale used to assess the severity of white matter hyperintensities and is known to be affected by inter-rater variability. While the best Fazekas score prediction model achieved a Matthews correlation coefficient (MCC) of 0.70, performance varied across data splits and loss functions, making interpretation of model capabilities difficult. Rather than interpreting epistemic uncertainty of a model's prediction as an isolated scalar value, our approach of uncertainty mapping relates uncertainty to its position within the learned feature representation. This highlights regions of class-boundary transitions where cases appear more ambiguous and misclassifications are more likely. It also identifies potential label disagreement, including low-uncertainty misclassified cases that expert review found to be inconsistent with the original reference Fazekas score. Therefore, uncertainty mapping allows model behaviour to be examined in relation to class separation and potential model-label disagreement. Loss function choice also influenced the uncertainty profile, with some models showing clearer class separation and more localized uncertainty in ambiguous regions than others. These findings suggest that uncertainty mapping for Fazekas score predictions can support model interpretation and targeted dataset review when reference labels are affected by ambiguity/ inter-rater variability.

Figures & tables

Figure 1: Distribution of the periventricular Fazekas score (PVWMH) by age. With increasing age, the WMHs become more prevalent. By the age of 80 years, all individuals have some extent of WMHs.

Figure 2: Two-channel 3D volume input to the convolutional neural network architecture. Multiple dropout layers were included to enable uncertainty estimation using Monte Carlo dropout.

Figure 3: Uncertainty Mapping: Feature Visualization combined with epistemic uncertainty estimation obtained through Monte Carlo dropout.

Figure 4: Visual comparison of Layerwise-Relevance Propagation (LRP) patterns for four representative subjects with periventricular Fazekas scores of 0, 1, 2, and 3. LRP heatmaps are shown for models trained with different loss functions from the fold with the best overall performance. While performance differed substantially between models, these differences were not clearly reflected in the heatmaps. Red regions indicate areas supporting the model prediction, whereas blue regions indicate areas contributing against the prediction or toward an alternative class. The subject with Fazekas score 1 was incorrectly predicted by the GCE-trained model.

Loss function	Accuracy	MCC	Cohen’s $\kappa$	F1: Class 0	F1: Class 1	F1: Class 2	F1: Class 3
CCE	0.73 $\pm$ 0.05	0.46 $\pm$ 0.09	0.45 $\pm$ 0.09	0.54 $\pm$ 0.11	0.80 $\pm$ 0.05	0.65 $\pm$ 0.13	0.33 $\pm$ 0.44
GCE	0.68 $\pm$ 0.06	0.39 $\pm$ 0.07	0.37 $\pm$ 0.08	0.44 $\pm$ 0.19	0.76 $\pm$ 0.07	0.61 $\pm$ 0.14	0.24 $\pm$ 0.43
OLL	0.73 $\pm$ 0.04	0.47 $\pm$ 0.10	0.46 $\pm$ 0.11	0.53 $\pm$ 0.16	0.80 $\pm$ 0.04	0.66 $\pm$ 0.12	0.31 $\pm$ 0.46
OLL clip	0.74 $\pm$ 0.05	0.49 $\pm$ 0.10	0.48 $\pm$ 0.10	0.57 $\pm$ 0.10	0.81 $\pm$ 0.04	0.66 $\pm$ 0.14	0.40 $\pm$ 0.46

Table 1: Nested cross-validation results: Average of all 20 runs

\pm

the standard deviation.

Figure 5: Distribution of epistemic uncertainty per Fazekas score class for the baseline model trained with CCE, achieving an MCC of 0.59. Misclassified cases are highlighted in red, while correct predictions are gray.

Figure 6: $t$ -SNE dimension reduction of the penultimate layer for CCE baseline model (MCC=0.59). Each point represents one subject from the test set. The fill colour indicated the reference label, while the edge colour indicated the model predictions. Hence, points with different fill and edge colours correspond to misclassified samples, while homogeneous points are correctly predicted.

Panel of Figure 7 — Figure 7: Uncertainty results combined with the $t$ -SNE analysis. The shapes represent the reference Fazekas score, while the color scale represents epistemic uncertainty. In a), epistemic uncertainty is shown for all samples, while in b), the same results are visualized only for incorrect predictions; correct predictions are shown in grey. Dashed ellipses indicate Fazekas score region and were added to guide visual interpretation.

Panel of Figure 8 — Figure 8: Epistemic uncertainty estimation visualized combined with the corresponding model feature space representation for the four different loss functions analyzed (a) CCE, (b) GCE, (c) OLL, and (d) OLL clip. The color scale represents epistemic uncertainty, while the shape corresponds to the reference Fazekas score.

Panel of Figure 9 — Figure 9: Agreement between the predictions of the CCE and OLL clip model is shown for both models uncertainty mapping. Coloured points indicate the epistemic uncertainty of test samples where the two models disagreed. Grey data points reflect subjects both models agreed on. The shape corresponds to the reference label.

Selection Group	Cases Reviewed	Revised	Observed revisions
Inconsistency	4	4	(A) 0 $\rightarrow$ 2; (B) 0 $\rightarrow$ 2; (C) 0 $\rightarrow$ 1; (D)1 $\rightarrow$ 3
Ambiguous boundary	4	2	(E) 1-; (H) 1 $\rightarrow$ 2; (G) 1 $\rightarrow$ 3-
Confident	4	1	(I) 0 $\rightarrow$ 1; (J) 1+

Table 2: Target re-evaluation of the selected 12 cases summarized by group. Case letters correspond to the samples shown in Figure 10 . Label revisions are reported as original reference label

\rightarrow

reassessed label. The symbols

+

and

-

indicate directional uncertainty in the reassessment score, where

+

indicates that the score is at the higher end and

-

indicates that the score is at the lower end.

Figure 10: Identification of cases selected for targeted re-evaluation using the OLL clip uncertainty map. Three groups were selected: (1) label-inconsistency cases (A-D); (2) ambiguous boundary cases (E-H), selected from regions of elevated uncertainty near transitions between neighbouring Fazekas scores, and (3) confident agreement cases (I-L), defined as low-uncertainty correctly classified samples located within well-defined Fazekas score regions.

Figure 11: Label-inconsistency cases (misclassified despite low epistemic uncertainty) for the OLL clip model based on Figure 10 .

Medical Image Classification

Medical image classification uses computational models to identify diseases, grade severity, and support decisions from clinical images.

See all 508 papers

Does Model Uncertainty Track Human Ambiguity? Evidence from Multi-Annotator Vision Benchmarks
Sep 28, 2026Manya Singh, Arjun PakrashiUncertainty Quantification
School of Computer Science, University College Dublin, Ireland
A Decision-Aware Evaluation Protocol for Uncertainty in Medical Image Segmentation
Oct 15, 2026Saket Maganti, Shreyank N GowdaUncertainty Quantification Segmentation Uncertainty Estimation
Independent Researcher · School of Computer Science, University of Nottingham, Nottingham, UK
Uncertainty-Aware Learning from Multi-Expert Interval Targets
Sep 8, 2026Samira Alkaee Taleghan, Younghyun Koo, Andrew P. Barrett +1Uncertainty Decomposition Label Distribution Learning
University of Colorado Denver Denver, Colorado, USA · National Snow and Ice Data Center (NSIDC) CIRES, University of Colorado Boulder Boulder, Colorado, USA

Sep 28, 2026J/K move · Enter open · S save

Does Model Uncertainty Track Human Ambiguity? Evidence from Multi-Annotator Vision Benchmarks

Manya Singh, Arjun Pakrashi

School of Computer Science, University College Dublin, Ireland

Uncertainty Quantification

Open paper AskPDF arXiv

Human-model alignment is critical for trustworthy AI-assisted decision-making systems. Yet, most work evaluates model predictions against single ground-truth labels, overlooking that humans themselves often disagree on labels, a signal of genuine ambiguity. We investigate whether models struggle on the same instances that humans find difficult. We measure this on two vision datasets (FER+ and CIFAR-10H) where multiple human annotations per image capture human disagreement patterns. We evaluate eight pretrained models across three architectures (ResNet, EfficientNet, MobileNetV3) in two parts: first, whether model uncertainty (softmax confidence, entropy) correlates with human disagreement, and second, whether predictive multiplicity measures (inter-model disagreement, Jensen-Shannon divergence) do. We find that it does not: alignment is weak in both dimensions. At the discrete label level, 50.4% of CIFAR-10H images and 33.5% of FER+ images receive multiple valid classifications from humans, while the models converge on only one. These instances represent a critical failure case where humans perceive ambiguity and would request expert review, yet models decide confidently. At the continuous score level, single-model uncertainty correlates weakly with human disagreement (

ρ= 0.24--0.55

), and predictive multiplicity provides only modest improvement. Widely-used uncertainty quantification methods do not reliably identify instances humans find ambiguous. Model uncertainty should not be treated as a trustworthy signal by default for decision-making in high-stakes scenarios.

Abstract

Medical Image Classification

Explore similar work

Does Model Uncertainty Track Human Ambiguity? Evidence from Multi-Annotator Vision Benchmarks

A Decision-Aware Evaluation Protocol for Uncertainty in Medical Image Segmentation

Uncertainty-Aware Learning from Multi-Expert Interval Targets