Rethinking Uncertainty in Segmentation: From Estimation to Decision

Saket Maganti

cs.CVApr 14, 2026

Rethinking Uncertainty in Segmentation: From Estimation to Decision

Authors: Saket Maganti

Abstract

In medical image segmentation, uncertainty estimates are often reported but rarely used to guide decisions. We study the missing step: how uncertainty maps are converted into actionable policies such as accepting, flagging, or deferring predictions. We formulate segmentation as a two-stage pipeline, estimation followed by decision, and show that optimizing uncertainty alone fails to capture most of the achievable safety gains. Using retinal vessel segmentation benchmarks (DRIVE, STARE, CHASE_DB1), we evaluate two uncertainty sources (Monte Carlo Dropout and Test-Time Augmentation) combined with three deferral strategies, and introduce a simple confidence-aware deferral rule that prioritizes uncertain and low-confidence predictions. Our results show that the best method and policy combination removes up to 80 percent of segmentation errors at only 25 percent pixel deferral, while achieving strong cross-dataset robustness. We further show that calibration improvements do not translate to better decision quality, highlighting a disconnect between standard uncertainty metrics and real-world utility. These findings suggest that uncertainty should be evaluated based on the decisions it enables, rather than in isolation.

Figures & tables

Figure 1: Overview of the two-stage decision pipeline. Stage 1 produces a segmentation prediction and uncertainty estimate. Stage 2 applies a deferral policy to produce an accept/defer decision per pixel. The two stages are modular: any uncertainty method can pair with any deferral policy.

Figure 2: Error reduction vs. deferral rate. Each point pairs an uncertainty method (MC Dropout in blue, TTA in orange) with a deferral policy (marker shape). TTA + adaptive dominates the upper-left (79.5% error reduction at 25% deferral); TTA + confidence-aware is the Pareto optimum at low budgets (55% at only 12%). Every TTA configuration outperforms every MC Dropout configuration—the decision layer reshapes, not merely refines, what the uncertainty estimator delivers.

Method	Dice	AUC	IoU	ECE	Unc-AUROC	ERR	Def. %	Time (s)
Deterministic	0.736	0.944	—	0.245	0.500	—	—	0.08
MC Drop. (global)	0.763	0.965	0.617	0.035	0.722	26.5%	33.4%	2.65
MC Drop. (adaptive)	0.763	0.965	0.617	0.035	0.722	42.4%	25.0%	2.65
MC Drop. (conf-aware)	0.763	0.965	0.617	0.035	0.722	48.8%	10.9%	2.65
TTA (global)	0.768	0.969	0.624	0.035	0.881	55.1%	13.2%	0.86
TTA (adaptive)	0.768	0.969	0.624	0.035	0.881	79.5%	25.0%	0.86

Table 1: Main results on the DRIVE test set (20 images). TTA matches the ensemble on segmentation quality while producing the best uncertainty (Unc-AUROC 0.881) and the largest error reduction (79.5% at 25% deferral) at 3.1

\times

the speed of MC Dropout. Best values per column in bold; runtime is per image on a single GPU. ERR

=

error reduction ratio; Def. %

=

fraction of pixels deferred.

Figure 3: Uncertainty maps from MC Dropout and TTA. TTA uncertainty concentrates on boundary and thin-vessel regions where errors are most frequent, while MC Dropout produces more spatially diffuse uncertainty.

Uncertainty	Deferral	Error before	Error after	ERR	Def. %	ERR / Def. %
MC Dropout	Global	0.0658	0.0484	26.5%	33.4%	0.79
MC Dropout	Adaptive	0.0658	0.0379	42.4%	25.0%	1.70
MC Dropout	Conf-aware	0.0658	0.0337	48.8%	10.9%	4.49
TTA	Global	0.0640	0.0287	55.1%	13.2%	4.19
TTA	Adaptive	0.0640	0.0131	79.5%	25.0%	3.18
TTA	Conf-aware	0.0640	0.0288	55.1%	12.1%	4.56

Table 2: Deferral strategy comparison on the DRIVE test set. Confidence-aware deferral dominates at low review budgets—it delivers 4.56 percentage points of error reduction per percentage point deferred, roughly

5.8\times

the efficiency of the MC Dropout global baseline. Adaptive deferral produces the largest absolute error removal (79.5% at 25% deferral). Bold marks the best value in each column within its uncertainty group.

Figure 4: Error rates before and after deferral across all configurations. TTA+adaptive achieves 80% error reduction; TTA+conf-aware achieves 55% error reduction at roughly half the review budget. The choice of both uncertainty method and deferral policy substantially reshapes the practical review workflow.

Panel of Figure 5 — Figure 5: Deferral behavior and case-level uncertainty analysis. Left panel shows TTA deferral modes; right panel confirms that uncertainty quality holds across the difficulty spectrum.

Figure 6: Risk-coverage curves for MC Dropout and TTA. TTA reaches clinically acceptable Dice (0.82) at substantially higher coverage than MC Dropout, requiring far less human review to meet the clinical bar.

Method	$T$	ECE (pre)	ECE (post)	$\Delta$ ECE	Unc-AUROC
MC Dropout	1.20	0.035	0.043	$+0.008$	0.722
TTA	1.35	0.035	0.039	$+0.005$	0.881

Table 3: Effect of temperature scaling on calibration and deferral. Both ECE values increase after calibration while Unc-AUROC is unchanged, confirming that rank-preserving transforms cannot improve a discriminative metric.

T

is the learned temperature.

Panel of Figure 7 — Figure 7: Reliability diagrams for MC Dropout (left) and TTA (right), with and without temperature scaling. Both models are already close to the diagonal before calibration, leaving little room for temperature scaling to improve.

Dataset	Dice	AUC	ECE	Unc-AUROC	AUCC (Dice)
DRIVE (in-domain)	0.791	0.960	0.043	0.850	0.793
STARE	0.727	0.949	0.038	0.846	0.773
CHASE_DB1	0.726	0.952	0.035	0.835	0.766

Table 4: Cross-dataset evaluation (zero-shot transfer from DRIVE). Segmentation quality drops by 6–7 Dice points under domain shift, yet uncertainty quality (Unc-AUROC) holds within 1–2 points of the in-domain value. Uncertainty transfers more robustly than segmentation—deferral remains reliable exactly when it matters most.

Figure 8: Qualitative results across datasets. Errors concentrate at vessel boundaries (column d). TTA uncertainty tracks these regions (column e), and the deferral policy routes them for review (column f) while retaining the large majority of correctly segmented pixels.

Shift level	Dice	AUC	ECE	Unc-AUROC	ERR (adaptive)
None	0.764	0.965	0.037	0.765	45.6%
Mild	0.517	0.840	0.056	0.648	7.9%
Strong	0.186	0.758	0.094	0.655	25.0%

Table 5: Effect of synthetic distribution shift. Uncertainty degrades more slowly than segmentation: Dice falls from 0.764 to 0.186 under strong shift (

-

75% relative) while Unc-AUROC only drops from 0.765 to 0.655 (

-

14% relative). The clean baseline is in bold ; the strong-shift row is in italics .

Configuration	Dice	ECE	Unc-AUROC	Time (s)	Unc/s
MC Drop. $T{=}5$	0.790	0.045	0.811	0.24	3.38
MC Drop. $T{=}10$	0.792	0.042	0.836	0.47	1.78
MC Drop. $T{=}20$	0.791	0.043	0.832	0.93	0.89
MC Drop. $T{=}30$	0.791	0.044	0.838	1.42	0.59
Ensemble $N{=}2$	0.768	0.032	0.803	1.37	0.59
Ensemble $N{=}5$	0.771	0.031	0.833	3.90	0.21

Table 6: Compute–quality trade-off across MC Dropout passes and ensemble sizes. Uncertainty quality plateaus after

T{=}10

; ensembles are well-calibrated but slower than TTA at lower Unc-AUROC. TTA (

K{=}6

, bottom row) dominates the frontier: highest Unc-AUROC at the lowest multi-pass cost. Best value per column in bold.

Uncertainty	Deferral	5%	10%	20%	30%
MC Dropout	Global	8.2%	14.7%	23.5%	28.1%
MC Dropout	Conf-aware	21.4%	38.6%	52.1%	57.3%
TTA	Global	15.3%	28.1%	44.8%	56.2%
TTA	Adaptive	18.7%	35.9%	61.3%	75.4%
TTA	Conf-aware	24.8%	43.2%	60.7%	68.1%

Table 7: Error reduction at fixed deferral budgets. Confidence-aware deferral dominates at low review budgets (5–10%), the regime most relevant for clinical workflows, while adaptive thresholding wins at higher budgets (20–30%). Bold marks the best value per column.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Image	Dice	Unc-AUROC (MC)	Unc-AUROC (TTA)	Uncertainty (mean)
Hardest	0.721	0.812	0.913	0.019
Q1	0.748	0.789	0.895	0.015
Median	0.765	0.770	0.878	0.012
Q3	0.782	0.752	0.866	0.010
Easiest	0.809	0.718	0.842	0.007

Appendix

Table 8: Per-image Dice and Unc-AUROC for 5 representative DRIVE test images, ranked by difficulty. Harder images (lower Dice) yield higher Unc-AUROC, consistent with the cross-dataset finding: uncertainty quality improves when the model makes more errors. Best Unc-AUROC in bold.

Panel of Figure 9 — Figure 9: Deferral maps comparing global and confidence-aware strategies. Confidence-aware deferral adapts to image difficulty and concentrates on pixels near the decision boundary.

Component	MC Drop. ( $T{=}30$ )	TTA ( $K{=}6$ )	Ensemble ( $N{=}5$ )	Determ.
Preprocessing	0.005 s	0.005 s	0.005 s	0.005 s
Forward passes	0.780 s	0.110 s	3.850 s	0.065 s
Aggregation	0.030 s	0.012 s	0.040 s	—
Deferral scoring	0.005 s	0.003 s	0.005 s	—
Total	0.820 s	0.130 s	3.900 s	0.070 s

Appendix

Table 9: Per-image runtime breakdown on DRIVE (mean over 20 images). Forward passes dominate total cost; aggregation and deferral scoring are negligible. Best (fastest) per row in bold.

Comparison	Metric	$p$ - value
MC Drop. vs. Deterministic	Dice	$<0.001$ **
MC Drop. vs. Deterministic	AUC	$<0.001$ **
MC Drop. vs. Deterministic	Unc-AUROC	$<0.001$ **
MC Drop. vs. TTA	Dice	$0.142$
MC Drop. vs. TTA	Unc-AUROC	$<0.001$ **
MC Drop. vs. Ensemble	Dice	$<0.001$ **

Appendix

Table 10: Paired

t

-test

p

-values for pairwise method comparisons on the DRIVE test set. The Dice gap between MC Dropout and TTA is not significant (

p{=}0.142

); the Unc-AUROC gap is highly significant (

p{<}0.001

). ** denotes

p{<}0.01

Bin range	% pixels	Avg conf	Avg acc	Gap	Gap (post-cal)
$[0.00,0.07)$	82.3%	0.013	0.016	0.003	0.004
$[0.07,0.13)$	2.1%	0.095	0.112	0.017	0.015
$[0.13,0.20)$	1.3%	0.163	0.186	0.023	0.019
$[0.20,0.33)$	1.4%	0.261	0.278	0.017	0.013
$[0.33,0.47)$	1.1%	0.398	0.405	0.007	0.005
$[0.47,0.60)$	0.9%	0.531	0.522	0.009	0.008

Appendix

Table 11: Bin-level calibration analysis for MC Dropout (15-bin ECE). Most miscalibration concentrates in the low-confidence bins (0.07–0.20), where the model underestimates accuracy by 1–2 points. Temperature scaling reduces mid-range gaps by 2–4 percentage points but has minimal effect on the dominant bins near 0 and 1, which explains why overall ECE is largely unchanged. Bold marks the largest pre-calibration gap.

Symbol	Meaning	Symbol	Meaning
$\mathbf{x}$	Input image ( $\mathbb{R}^{H\times W\times C}$ )	$\bar{p}_{ij}$	Mean prediction across passes
$\mathbf{y}$	Ground truth mask ( $\{0,1\}^{H\times W}$ )	$\mathrm{MI}_{ij}$	Mutual information (epistemic)
$\hat{\mathbf{p}}$	Predicted probability map ( $[0,1]^{H\times W}$ )	$H(p)$	Binary entropy
$\hat{y}_{ij}$	Hard prediction: $\mathbb{1}[\hat{p}_{ij}>0.5]$	$\delta_{ij}$	Deferral decision (1=accept, 0=defer)
$\mathbf{u}$	Uncertainty map ( $\mathbb{R}_{\geq 0}^{H\times W}$ )	$\tau$	Deferral threshold (global)
$T$	Number of MC Dropout forward passes	$\tau_{n}$	Deferral threshold, image $n$ (adaptive)

Appendix

Table 12: Summary of notation used throughout the paper.

Explore similar work

SegWithU: Uncertainty as Perturbation Energy for Single-Forward-Pass Risk-Aware Medical Image Segmentation
Apr 16, 2026Tianhao Fu, Austin Wang, Charles Chen +2Image Segmentation Uncertainty Quantification
University of Toronto, Toronto, ON, Canada · Vector Institute, Toronto, ON, Canada · Project Neura, Toronto, ON, Canada +4
U-SEG: Uncertainty in SEGmentation -- A systematic multi-variable exploration
May 14, 2026Michael Smith, Frank P. FerrieImage Segmentation Panoptic Segmentation
Centre for Intelligent Machines, McGill University
Principle-Guided Supervision for Interpretable Uncertainty in Medical Image Segmentation
May 9, 2026An Sui, Yuzhu Li, Gunter Schumann +2Image Segmentation Uncertainty Quantification
School of Data Science, Fudan University, Shanghai, China · Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China · National Heart and Lung Institute, Imperial College London, London, United Kingdom

Apr 16, 2026J/K move · Enter open · S save

SegWithU: Uncertainty as Perturbation Energy for Single-Forward-Pass Risk-Aware Medical Image Segmentation

Tianhao Fu, Austin Wang, Charles Chen, Roby Aldave-Garza+1

University of Toronto, Toronto, ON, Canada · Vector Institute, Toronto, ON, Canada · Project Neura, Toronto, ON, Canada+4

Image Segmentation Uncertainty Quantification Selective Prediction Medical Image Segmentation Segmentation Uncertainty Estimation Uncertainty Calibration

Open paper AskPDF arXiv

Reliable uncertainty estimation is critical for medical image segmentation, where automated contours feed downstream quantification and clinical decision support. Many strong uncertainty methods require repeated inference, while efficient single-forward-pass alternatives often provide weaker failure ranking or rely on restrictive feature-space assumptions. We present

\textbf{SegWithU}

, a post-hoc framework that augments a frozen pretrained segmentation backbone with a lightweight uncertainty head. SegWithU taps intermediate backbone features and models uncertainty as perturbation energy in a compact probe space using rank-1 posterior probes. It produces two voxel-wise uncertainty maps: a calibration-oriented map for probability tempering and a ranking-oriented map for error detection and selective prediction. Across ACDC, BraTS2024, and LiTS, SegWithU is the strongest and most consistent single-forward-pass baseline, achieving AUROC/AURC of

0.9838/2.4885

0.9946/0.2660

, and

0.9925/0.8193

, respectively, while preserving segmentation quality. These results suggest that perturbation-based uncertainty modeling is an effective and practical route to reliability-aware medical segmentation. Source code is available at https://github.com/ProjectNeura/SegWithU.

Abstract

Explore similar work

SegWithU: Uncertainty as Perturbation Energy for Single-Forward-Pass Risk-Aware Medical Image Segmentation

U-SEG: Uncertainty in SEGmentation -- A systematic multi-variable exploration

Principle-Guided Supervision for Interpretable Uncertainty in Medical Image Segmentation