Beyond Boundary Noise: Aggregated Aleatoric Uncertainty Fails to Capture Presence Ambiguity in 3D Lung Nodule Segmentation
Authors: Simon Baur, Arne Schernich, Ekin Böke, Wojciech Samek, Jackie Ma
Organizations: Fraunhofer Heinrich-Hertz-Institut, 10587 Berlin, Germany · Technische Universität Berlin, 10623 Berlin, Germany · The Berlin Institute for the Foundations of Learning and Data (BIFOLD), 10587 Berlin, Germany
Uncertainty estimation is critical for the safe clinical deployment of deep learning in medical image segmentation, with aleatoric uncertainty theoretically designed to capture irreducible data ambiguity. However, whether entropy-based measures reflect clinically meaningful ambiguity, i.e. case-level disagreement about whether a pathology is present at all, remains poorly understood. Contrary to most prior work, which focused on pixel-wise boundary disagreement, we systematically evaluate how well aleatoric uncertainty captures presence ambiguity. Our evaluation spans 3D lung nodule segmentation across four architectures with Monte Carlo dropout and deep ensembles, on LIDC-IDRI and an external validation cohort (LNDb). We find that entropy-based uncertainty maps align with boundary noise and minor drawing variation but carry insufficient discriminative signal for presence ambiguity. In contrast, a lightweight supervised ambiguity head trained on frozen segmentation features substantially outperforms all entropy-aggregation-based baselines across architectures, metrics, and both cohorts, and matches or exceeds methods that explicitly model ambiguity under disagreement supervision (Probabilistic U-Net, Annotator-Confusion 3D-UNet). A qualitative feature-space analysis shows that presence ambiguity is already encoded in the frozen encoder features of pixel-wise-trained networks, only to be discarded by the segmentation output and its entropy aggregation. Our findings expose a fundamental mismatch between the theoretical promise of aleatoric uncertainty and its practical behavior, and suggest that practitioners should not rely on entropy-based uncertainty as a proxy for clinical ambiguity in safety-critical applications.
Figures & tables
Figure 1: Visualization of a non-ambiguous (top row) and ambiguous (bottom row) lung nodule patch from the LIDC-IDRI dataset. Each column shows: the CT image with annotator segmentations overlaid as colored circles (left), the aleatoric uncertainty map derived from MC-Dropout (center), and the mean predicted segmentation mask (right). We can clearly see that the ambiguous sample (where annotators disagree on whether a structure is present at all) produces virtually no aleatoric uncertainty signal.
Figure 2: Presence ambiguity detection across disagreement levels on LIDC and LNDb. Radar plots compare pixel-wise AU aggregation (mean of Ensemble and MC-Dropout) against post-hoc Ambiguity Head, averaged over four architectures. Entropy-based aggregation stays near-random across all disagreement levels, while the lightweight head recovers a highly discriminative signal from frozen segmentation features.
Figure 3: Sorted performance metrics, averaged across model architectures and ambiguity thresholds. Aleatoric uncertainty (AU) baselines appear twice, differing in the region over which voxel-wise uncertainty is aggregated into a score: the ground-truth mask (hatched) or the predicted mask (solid). Using the predicted mask consistently improves both the unsupervised entropy-aggregation methods and the supervised disagreement-modelling baselines (Prob-3DUnet, Ann-Conf-3DUnet). The Ambiguity Head surpasses all baseline variants across metrics in almost all cases, except for Ann-Conf-3DUnet (prediction mask aggregation) where it performs on par. Notably, this performance also holds in the external validation cohort (LNDb). Standard deviations are omitted here, as averaging over disagreement levels, models, and runs renders them uninformative. Non-aggregated metrics and standard deviations per disagreement level and model are reported in Appendix A.2.
Figure 4: Ordinal correlation between uncertainty scores and ambiguity severity. Scatter panels (left, mid) show an exemplary distribution of uncertainty scores (UNet-3D). The summary (right) reports Spearman correlations ( ρ ) across both cohorts, all backbones, and all methods. The Ambiguity Head stratifies the full spectrum of disagreement severity rather than only binary ambiguity, whereas pixel-wise AU aggregation and the supervised baselines track severity only weakly or inconsistently.
Figure 5: UMAP projection of frozen model features (1vs3). The clustering indicates presence ambiguity is already well-represented in the latent space, i.e. Ambiguity Head does not learn from scratch but recovers a signal the standard pipeline fails to transport.
AU (Ensemble)
AU (MC-Dropout)
Ambiguity Head
Split
Model
AUROC
AUPRC
F1
AUROC
AUPRC
F1
AUROC
AUPRC
F1
LIDC 1vs3
Attn-3DUnet
0.45 ± 0.02
0.53 ± 0.02
0.40 ± 0.03
0.48 ± 0.04
0.54 ± 0.02
0.38 ± 0.09
0.86 ± 0.04
0.87 ± 0.03
0.78 ± 0.05
3D-ResUnet
0.48 ± 0.02
0.57 ± 0.02
0.44 ± 0.03
0.49 ± 0.03
0.57 ± 0.03
0.47 ± 0.04
0.90 ± 0.01
0.89 ± 0.03
0.82 ± 0.04
SegFormer3D
0.43 ± 0.04
0.58 ± 0.04
0.32 ± 0.09
0.42 ± 0.12
0.53 ± 0.08
0.23 ± 0.17
0.61 ± 0.04
0.62 ± 0.03
0.57 ± 0.04
3D-Unet
0.39 ± 0.02
0.45 ± 0.01
0.32 ± 0.04
0.42 ± 0.07
0.47 ± 0.03
0.31 ± 0.17
0.91 ± 0.02
0.92 ± 0.01
0.83 ± 0.02
LIDC 2vs2
Attn-3DUnet
0.52 ± 0.01
0.42 ± 0.01
0.38 ± 0.09
0.52 ± 0.03
0.40 ± 0.01
0.41 ± 0.09
0.76 ± 0.04
0.66 ± 0.04
0.60 ± 0.06
Table 1: Presence ambiguity detection performance on disagreement level.
Split
Model
AUROC
AUPRC
F1
LIDC 1vs3
Ambiguity Head (mean)
0.82 ± 0.14
0.82 ± 0.14
0.75 ± 0.12
Ann-Conf-3DUnet
0.54 ± 0.09
0.68 ± 0.09
0.55 ± 0.08
Prob-3DUnet
0.49 ± 0.03
0.56 ± 0.01
0.48 ± 0.10
LIDC 2vs2
Ambiguity Head (mean)
0.75 ± 0.08
0.65 ± 0.10
0.61 ± 0.06
Ann-Conf-3DUnet
0.63 ± 0.12
0.63 ± 0.14
0.51 ± 0.13
Prob-3DUnet
0.48 ± 0.01
0.40 ± 0.02
0.40 ± 0.03
Table 2: Presence ambiguity detection performance compared with supervised competitors.
LIDC
LNDb
Method
AUROC
AUPRC
F1
AUROC
AUPRC
F1
Ambiguity Head
0.74
0.67
0.63
0.67
0.66
0.66
Ann-Conf-3DUnet
0.73
0.60
0.62
0.65
0.60
0.66
Prob-3DUnet
0.64
0.56
0.60
0.62
0.61
0.55
MC-Dropout
0.58
0.53
0.49
0.54
0.55
0.56
Ensemble
0.54
0.51
0.42
0.49
0.52
0.47
Table 3: Presence ambiguity detection performance across architectures and disagreement thresholds, on LIDC and the external LNDb cohort for aggregation of UAU(x) on the prediction mask instead of ground truth mask.
Method
Backbone
LIDC ρ
LNDb ρ
AU Aggregation
Ann. Confusion
0.43 ± 0.08
0.08 ± 0.03
3D-ResUnet
0.32 ± 0.02
0.16 ± 0.01
Prob-UNet
0.29 ± 0.03
0.18 ± 0.01
Attn-3DUnet
0.21 ± 0.02
0.18 ± 0.01
3D-Unet
0.15 ± 0.04
0.16 ± 0.01
SegFormer3D
−0.01± 0.03
0.02 ± 0.02
Table 4: Ordinal correlation (Spearman ρ ) between uncertainty scores and disagreement severity for aggregation of UAU(x) on the prediction mask instead of ground truth mask.
ICube, Institut Strauss, DKFZ · ICube Laboratory, CNRS UMR-7357, University of Strasbourg, Strasbourg, France · CLCC Institut-Strauss, Strasbourg, France +6