A Decision-Aware Evaluation Protocol for Uncertainty in Medical Image Segmentation

Saket Maganti; Shreyank N Gowda

cs.CVOct 15, 2026

A Decision-Aware Evaluation Protocol for Uncertainty in Medical Image Segmentation

Authors: Saket Maganti, Shreyank N Gowda

Organizations: Independent Researcher · School of Computer Science, University of Nottingham, Nottingham, UK

Abstract

Medical segmentation uncertainty is often evaluated with calibration error or pixel-level error-detection scores, but these metrics do not show whether uncertainty improves downstream review decisions. We propose a decision-aware evaluation protocol that separates uncertainty estimation from deferral policy and evaluates their combination under fixed review budgets: which pixels, regions, or images should be accepted, and which should be reviewed. The protocol reports fixed-budget error reduction, risk-coverage behaviour, region-level review proxies, structural failure modes, probability-derived baselines, and sensitivity to review assumptions. We instantiate it across retinal vessel, skin lesion, and polyp segmentation with multiple backbones and uncertainty estimators. Under matched inference budgets and canonical multi-layer MC Dropout, test-time augmentation (TTA) gives a stronger error-detection signal across matched MC--TTA comparisons, including architecture and dropout-placement checks. However, estimator rankings are not universal. For example, probability-derived scores remain competitive, deep ensembles can outperform TTA, learned deferral helps mainly when the base estimator is weak, and rankings can change with review unit, uncertainty target, correction assumptions, and cost model. Structural and second-observer analyses further show that boundary and thin-structure failures are partly tied to annotation ambiguity. These findings support evaluating uncertainty as an estimator-policy-review configuration, not as a standalone score map. Code: https://anonymous.4open.science/r/Uncertainity-in-Medical-Image-Segmentation/README.md

Figures & tables

Figure 1: Decision-aware evaluation separates uncertainty estimation from the review decision . A trained segmentation backbone produces $\hat{p}$ and $\hat{y}$ ; an estimator, ensemble, or probability-derived baseline produces a score map $u$ ; and a deferral policy ranks pixels or regions under budget $\beta$ . ERR@ $\beta$ , risk–coverage AURC, region error capture, and review-assumption sensitivity are measured after this decision, not from the uncertainty map alone.

Dataset	Train	Val / fit	Test	Split status
DRIVE	16	4	20	official test
FIVES	520	120	160	self-defined
ISIC 2018	1686	389	519	self-defined
Kvasir-SEG	650	150	200	self-defined

Table 1: Dataset splits and fitting data. DRIVE uses the official 20-image test set with a self-defined train/validation split. FIVES, ISIC, and Kvasir use fixed 65/15/20 splits. Learned heads and adaptive policies are fit only on validation images.

Comparison family	Positive	BH sig.	Mean $\Delta$
Matched budget, anchor	24/24	24/24	+0.035
10 MC realizations	24/24	24/24	+0.033
Complementary backbone	24/24	24/24	+0.063
Dropout placement	72/72	72/72	+0.075
Dropout rate, DRIVE only	24/24	18/24	+0.015
Training seed, DRIVE only	15/18	10/18	+0.009

Table 2: Matched-budget MC–TTA robustness. Each row is a paired image-level comparison family with BH correction applied within that family. Positive cells favour TTA.

Dataset	Func.	MC-T6	MC-T50	TTA-T6	Ens.	$\Delta_{\mathrm{Ens}}$
DRIVE	Var	0.839	0.857	0.858	0.806	$-0.052^{*}$
DRIVE	MI	0.852	0.868	0.868	0.799	$-0.069^{*}$
FIVES	Var	0.852	0.878	0.858	0.881	$+0.023^{*}$
FIVES	MI	0.859	0.882	0.872	0.886	$+0.014^{*}$
ISIC	Var	0.803	0.840	0.867	0.896	$+0.029^{*}$
ISIC	MI	0.821	0.851	0.874	0.894	$+0.019^{*}$

Table 3: Estimator family changes the ranking. Mean image-level error-detection AUROC on the same held-out images. TTA uses six transforms.

\Delta

values are paired differences against TTA-T6; positive means the competitor is better. Asterisks denote BH-corrected significance within the comparison family.

Figure 2: The score ranking is not the decision ranking. Per cell, TTA raw uncertainty and the probability baseline are compared under a score-level metric (left: error-detection AUROC) and a decision-level metric (right: fraction of all errors captured at a

20\%

review budget). Rankings can flip at the decision point.

Figure 3: Decision-layer sensitivity on the DRIVE anchor. Fraction of tested settings in which changing one review assumption changes the preferred estimator. Predictions are held fixed, so reversals isolate the decision layer. These are finite-grid sensitivity summaries, not population estimates.

Figure 4: DRIVE second-observer analysis. Uncertainty is near chance on pixels where the two observers disagree; changing the target definition changes the AUROC; and boundary conclusions depend on the morphological radius, while thin-structure effects are more stable.

Audit	Main result
Learned deferral	Capt.@20% improves in 16/18 cell–estimator settings; DRIVE uses only 4 validation images for fitting.
Transfer vs raw target $u$	8/18 significantly better, 7/18 significantly worse, 3/18 n.s.
Transfer vs in-domain refit	4/18 significantly better, 9/18 significantly worse, 5/18 n.s.; refitting is usually safer.
DRIVE second observer	Global MC–TTA differences are n.s.; TTA gains concentrate on thin structures ( $+0.072$ variance, $+0.075$ MI).
Human-disagreement target	MI AUROC is 0.813 for MC and 0.817 for TTA, but all scores are near chance on disputed pixels.

Table 4: Policy, transfer, and annotation audits. Compact summary of decision-layer results. Learned-deferral counts use the deployable validation-fit protocol; transfer uses source-validation fitting and held-out target tests.

Coordinate	What should be reported
Estimator	MC/TTA/ensemble/probability score and uncertainty functional.
Compute	Number of passes or ensemble members; matched or deliberately swept.
Backbone	Segmentation checkpoint and whether the comparator changes it.
Policy	Raw ranking, adaptive threshold, learned head, or conformal rule.
Fitting data	Validation images used for policy fitting; no test-label fitting.
Review unit	Pixel, block, patch, connected component, region, or image.

Table 5: Minimum reporting checklist for decision-aware uncertainty evaluation. The estimator alone is insufficient; the reported result should identify the full configuration that turns uncertainty into a review decision.

Dataset	Architecture	Wave	Train	Val	Test	Split status	MC	TTA	Ens.	Head fit	Head eval
DRIVE	U-Net	canonical + Waves A/C/D/E/F	16	4	20	official test, self-defined train/val	20	20	20	4	20
DRIVE	MiT-B2 TransUNet	Wave B (complementary architecture)	16	4	20	official test, self-defined train/val	20	20	0	0	0
FIVES	MiT-B2 encoder + U-Net decoder	canonical + Waves A/C/D/E/F	520	120	160	self-defined	160	160	160	120	160
FIVES	U-Net	Wave B (complementary architecture)	520	120	160	self-defined	160	160	0	0	0
ISIC	U-Net	canonical + Waves A/C/D/E/F	1686	389	519	self-defined	519	519	519	389	519
ISIC	MiT-B2 TransUNet	Wave B (complementary architecture)	1686	389	519	self-defined	519	519	0	0	0

Table 6: Exact split and analysis counts. DRIVE uses the official 20-image test half but a self-defined 16/4 train/validation subdivision. FIVES, ISIC, and Kvasir use self-defined 65/15/20 splits (seed 42). Zero denotes an analysis not run for that architecture arm.

		$T{=}2$		$T{=}4$		$T{=}6$
Dataset	Func.	MC	$\Delta$	MC	$\Delta$	MC	$\Delta$
DRIVE	Var	0.805	+0.017 *	0.831	+0.019 *	0.839	+0.019 *
DRIVE	MI	0.807	+0.016 *	0.843	+0.017 *	0.852	+0.016 *
FIVES	Var	0.829	+0.004 *	0.844	+0.008 *	0.852	+0.007 *
FIVES	MI	0.837	+0.010 *	0.853	+0.014 *	0.859	+0.014 *
ISIC	Var	0.761	+0.056 *	0.793	+0.060 *	0.803	+0.064 *

Table 7: Matched inference budgets, canonical multi-layer MC Dropout. Mean image-level error-detection AUROC; MC and TTA draw the same

T

passes from the same checkpoint on the same images.

\Delta

is the paired per-image mean TTA

{-}

MC; * marks

q<0.05

after BH correction across this family of 24. TTA leads in all 24 and every bootstrap interval excludes zero.

Axis	TTA	BH	mean $\Delta$	range $\Delta$
Matched budget, anchor (4 ds)	24/24	24/24	+0.035	[+0.004, +0.065]
10 MC realizations (4 ds)	24/24	24/24	+0.033	[+0.003, +0.063]
Complementary backbone (4 ds)	24/24	24/24	+0.063	[+0.015, +0.133]
Dropout placement (4 ds)	72/72	72/72	+0.075	[+0.004, +0.250]
Dropout rate $p$ (DRIVE)	24/24	18/24	+0.015	[+0.003, +0.038]
Training seed (DRIVE)	15/18	10/18	+0.009	[-0.004, +0.020]

Table 8: Robustness across every configuration axis raised in review. Each row is an independent family of paired image-level comparisons with its own BH correction; “TTA” counts positive point estimates and “BH” counts

q<0.05

. The matched-budget, replication, architecture, and placement families are uniformly positive; the two DRIVE-only sweeps show smaller effects, non-significant cells, and three tiny seed reversals.

Figure 5: Complementary architecture audit. DRIVE, ISIC, and Kvasir are evaluated with a MiT-B2 TransUNet variant, while FIVES is evaluated with a U-Net variant. The paired TTA–MC effect remains positive in all budget–functional comparisons after the architecture swap.

Figure 6: Unequal compute changes the MC–TTA comparison. TTA is held at six passes while MC Dropout is increased to 10, 20, 30, and 50 stochastic passes. DRIVE approaches a tie, FIVES crosses to MC, and ISIC/Kvasir remain TTA-favourable in the tested settings.

Figure 7: Estimator family can change the winner. Five-member deterministic deep ensembles outperform TTA on FIVES and ISIC, underperform it on DRIVE, and are statistically indistinguishable from TTA on Kvasir after correction.

		mean image AUROC				$\Delta$ vs TTA-T6
Dataset	Func.	MC T6	MC T50	TTA T6	Ens. 5	MC T50	Ens. 5
DRIVE	Var	0.839	0.857	0.858	0.806	-0.001	-0.052 *
DRIVE	MI	0.852	0.868	0.868	0.799	+0.000	-0.069 *
FIVES	Var	0.852	0.878	0.858	0.881	+0.019 *	+0.023 *
FIVES	MI	0.859	0.882	0.872	0.886	+0.010 *	+0.014 *
ISIC	Var	0.803	0.840	0.867	0.896	-0.028 *	+0.029 *

Table 9: No estimator family wins everywhere once budget is unconstrained. Mean image AUROC, same images and backbone throughout; TTA uses 6 passes.

\Delta

columns are paired differences against TTA-T6 , so positive means the competitor is better; * is

q<0.05

within each column’s family. At

8\times

the passes MC ties TTA on DRIVE and overtakes it on FIVES; deep ensembles beat TTA on FIVES and ISIC and lose on DRIVE.

Dataset	Train	Val / fit imgs	Test	Capt.@20 wins
DRIVE	16	4	20	2/2
FIVES	520	120	160	2/4
ISIC	1686	389	519	4/4
Kvasir	650	150	200	8/8
Total	–	–	–	16/18

Table 10: Learned-deferral accounting. The deployable head is fitted on the held-out validation split and evaluated on test. The separate five-fold test-distribution cross-fit is retained only as a diagnostic upper bound.

Cell	Arm	Fit imgs	Eval imgs	Learned	Plain $u$	$\Delta$	95% CI	$q$
drive/unet	MC	4	20	0.796	0.623	+0.173 ∗	[+0.133,+0.218]	$<.001$
drive/unet	TTA	4	20	0.830	0.817	+0.013 ∗	[+0.010,+0.015]	$<.001$
fives/unet	MC	120	160	0.861	0.790	+0.071 ∗	[+0.059,+0.084]	$<.001$
fives/unet	TTA	120	160	0.924	0.928	-0.004 ∗	[-0.004,-0.003]	$<.001$
fives/unetpp	MC	120	160	0.838	0.778	+0.061 ∗	[+0.053,+0.069]	$<.001$
fives/unetpp	TTA	120	160	0.908	0.909	-0.001	[-0.002,+0.001]	0.337

Table 11: Deployable validation-fitted learned deferral at Capt.@20%. The head is fitted only on each cell’s validation split and evaluated on test.

\Delta=

learned minus plain uncertainty; ∗ denotes BH

q<.05

within the corresponding estimator family.

Arm	Source	Target	$n$	Transfer	Reference	$\Delta$	95% CI	$q$	Verdict
A cross domain	kvasir/unetpp	fives/unetpp	160	0.0023	0.0022	+0.0002 ∗	[+0.0001,+0.0002]	$<.001$	Reference
A cross domain	fives/unetpp	kvasir/unetpp	200	0.0236	0.0082	+0.0154 ∗	[+0.0102,+0.0212]	$<.001$	Reference
A cross domain	kvasir/deeplabv3plus	isic/deeplabv3plus	519	0.0074	0.0081	-0.0008 ∗	[-0.0011,-0.0004]	$<.001$	Transfer
A cross domain	isic/deeplabv3plus	kvasir/deeplabv3plus	200	0.0083	0.0084	-0.0001	[-0.0002,+0.0000]	0.086	n.s.
A cross domain	drive/unet	fives/unet	160	0.0019	0.0018	+0.0001 ∗	[+0.0000,+0.0001]	$<.001$	Reference
A cross domain	fives/unet	drive/unet	20	0.0173	0.0151	+0.0021 ∗	[+0.0019,+0.0024]	$<.001$	Reference

Table 12: Clean TTA policy transfer versus raw target uncertainty (AURC, 18 pairs). Source policies are fitted on source validation data and evaluated on the full target test set. AURC is lower-is-better, so negative

\Delta=

transfer minus reference favours transfer. ∗ denotes BH

q<.05

across the 18-pair family.

Arm	Source	Target	$n$	Transfer	Reference	$\Delta$	95% CI	$q$	Verdict
A cross domain	kvasir/unetpp	fives/unetpp	160	0.0023	0.0021	+0.0002 ∗	[+0.0002,+0.0003]	$<.001$	Reference
A cross domain	fives/unetpp	kvasir/unetpp	200	0.0236	0.0082	+0.0153 ∗	[+0.0102,+0.0211]	$<.001$	Reference
A cross domain	kvasir/deeplabv3plus	isic/deeplabv3plus	519	0.0074	0.0059	+0.0015 ∗	[+0.0007,+0.0023]	$<.001$	Reference
A cross domain	isic/deeplabv3plus	kvasir/deeplabv3plus	200	0.0083	0.0082	+0.0001	[-0.0000,+0.0002]	0.100	n.s.
A cross domain	drive/unet	fives/unet	160	0.0019	0.0020	-0.0001 ∗	[-0.0002,-0.0001]	$<.001$	Transfer
A cross domain	fives/unet	drive/unet	20	0.0173	0.0126	+0.0047 ∗	[+0.0039,+0.0056]	$<.001$	Reference

Table 13: Clean TTA policy transfer versus an in-domain refit (AURC, 18 pairs). Negative

\Delta=

transfer minus reference favours transfer. The mixed result shows that target-domain refitting is usually safer than transferring the source policy. ∗ denotes BH

q<.05

across the 18-pair family.

Figure 8: Decision-layer reversal rates on the DRIVE anchor. Predictions are held fixed while one review assumption is varied. These are finite-grid sensitivity summaries, not population rates.

ID	Experiment	Status	Data	Method	Metric	Effect	CI
CPU-001	exp01_drive_annotation_disagreement	PASS	DRIVE	mc_mutual_information	mean_image_auroc_human_disagreement	0.8131	[0.8007, 0.8253]
CPU-002	exp01_drive_annotation_disagreement	PASS	DRIVE	tta_mutual_information	mean_image_auroc_human_disagreement	0.8166	[0.8056, 0.8271]
CPU-003	exp02_imperfect_reviewer	PASS	DRIVE	all_estimators	ranking_reversal_fraction_vs_perfect_reviewer	0.1600	–
CPU-004	exp03_review_unit_sensitivity	PASS	DRIVE	all_estimators	fraction_review_unit_budget_cells_reversing_pixel_winner	0.6286	–
CPU-005	exp04_calibration	PASS_WITH_LIMITATION	DRIVE	mc	best_calibrated_probability_minus_best_sampling_error_auroc	-0.0506	–
CPU-006	exp04_calibration	PASS_WITH_LIMITATION	DRIVE	tta	best_calibrated_probability_minus_best_sampling_error_auroc	-0.0315	–

Table 14: CPU decision-assumption campaign. All analyses reuse stored prediction artifacts. PASS_WITH_LIMITATION is retained rather than collapsed into PASS; the separately planned cross-domain EXP14 is blocked because comparable non-DRIVE validation maps were not retained and is not treated as a null result.

Figure 9: DRIVE second-observer audit. Disagreement is the XOR of the two manual annotations inside the DRIVE field of view. Human disagreement is concentrated near vessel boundaries, while uncertainty is near chance on the disputed pixels themselves.

Score	Region	MC	TTA	$\Delta$	95% CI	$q$
variance	Global	0.857	0.859	+0.001	[-0.005,+0.009]	0.862
variance	DisAgree	0.474	0.473	-0.001	[-0.008,+0.006]	0.971
variance	Agree	0.922	0.924	+0.002	[-0.001,+0.006]	0.444
variance	Boundary $r=2$	0.714	0.717	+0.003	[-0.000,+0.005]	0.164
variance	Thin $r=1$	0.514	0.585	+0.072 ∗	[+0.061,+0.084]	$<.001$
MI	Global	0.868	0.868	+0.000	[-0.005,+0.007]	0.971

Table 15: DRIVE second-observer MC–TTA audit (

n=20

). Scores detect model error with respect to observer 2.

\Delta=

TTA

-

MC. Global MC/TTA differences are negligible; the robust positive difference is concentrated on thin structures. Second-observer evidence is DRIVE-only.

Dataset	Region	$r$	$n$	Region frac.	MC	TTA	$\Delta$	95% CI
DRIVE	Boundary	1	20	0.154	0.621	0.625	+0.005	[+0.002,+0.007]
DRIVE	Boundary	2	20	0.267	0.714	0.717	+0.003	[-0.000,+0.005]
DRIVE	Boundary	3	20	0.356	0.761	0.760	-0.001	[-0.004,+0.002]
DRIVE	Boundary	4	20	0.429	0.786	0.783	-0.003	[-0.007,+0.000]
DRIVE	Thin	1	20	0.071	0.514	0.585	+0.072	[+0.061,+0.084]
DRIVE	Thin	2	20	0.107	0.624	0.694	+0.070	[+0.059,+0.082]

Table 16: Boundary/thin-region radius sweep (variance AUROC).

\Delta=

TTA

-

MC. The DRIVE boundary sign changes at radii 3–4, whereas the thin-structure advantage remains positive. Other datasets are included to expose radius sensitivity rather than canonize one radius.

Figure 10: Representative qualitative examples across domains. Examples make the ranking problem concrete rather than supporting a new quantitative claim. High uncertainty tends to concentrate around difficult boundaries, small structures, and local segmentation failures, while different estimators allocate review mass differently.

Uncertainty Quantification

Uncertainty quantification estimates how reliable predictions and scientific inferences are, and how uncertainty propagates into decisions.

See all 647 papers

Rethinking Uncertainty in Segmentation: From Estimation to Decision
Apr 14, 2026Saket MagantiImage Segmentation Segmentation Uncertainty Estimation
SegWithU: Uncertainty as Perturbation Energy for Single-Forward-Pass Risk-Aware Medical Image Segmentation
Apr 16, 2026Tianhao Fu, Austin Wang, Charles Chen +2Image Segmentation Uncertainty Quantification
University of Toronto, Toronto, ON, Canada · Vector Institute, Toronto, ON, Canada · Project Neura, Toronto, ON, Canada +4
Trustworthy Medical Segmentation: Uncertainty-Aware U-Net Evaluation Under Clinical Image Degradation
Jul 22, 2026Pranav Kaliaperumal, Manisha KaliaperumalImage Corruption Robustness U-Net
Department of Computer Science University of Colorado Denver Aurora, CO, USA · Department of Biology (Pre-Medical) Creighton University Omaha, NE, USA

Apr 14, 2026J/K move · Enter open · S save

Rethinking Uncertainty in Segmentation: From Estimation to Decision

Saket Maganti

Image Segmentation Segmentation Uncertainty Estimation

Open paper AskPDF arXiv

In medical image segmentation, uncertainty estimates are often reported but rarely used to guide decisions. We study the missing step: how uncertainty maps are converted into actionable policies such as accepting, flagging, or deferring predictions. We formulate segmentation as a two-stage pipeline, estimation followed by decision, and show that optimizing uncertainty alone fails to capture most of the achievable safety gains. Using retinal vessel segmentation benchmarks (DRIVE, STARE, CHASE_DB1), we evaluate two uncertainty sources (Monte Carlo Dropout and Test-Time Augmentation) combined with three deferral strategies, and introduce a simple confidence-aware deferral rule that prioritizes uncertain and low-confidence predictions. Our results show that the best method and policy combination removes up to 80 percent of segmentation errors at only 25 percent pixel deferral, while achieving strong cross-dataset robustness. We further show that calibration improvements do not translate to better decision quality, highlighting a disconnect between standard uncertainty metrics and real-world utility. These findings suggest that uncertainty should be evaluated based on the decisions it enables, rather than in isolation.

Abstract

Uncertainty Quantification

Explore similar work

Rethinking Uncertainty in Segmentation: From Estimation to Decision

SegWithU: Uncertainty as Perturbation Energy for Single-Forward-Pass Risk-Aware Medical Image Segmentation

Trustworthy Medical Segmentation: Uncertainty-Aware U-Net Evaluation Under Clinical Image Degradation