Segmentation Confidence for Arbitrary CNNs

Baris Oguz; Xing Yao; Claudia Tawil; Alison Pouch; Brett Byram; Gabriel Arenas; Nadav Schwartz; Ipek Oguz

cs.CVOct 27, 2026

Segmentation Confidence for Arbitrary CNNs

Authors: Baris Oguz, Xing Yao, Claudia Tawil, Alison Pouch, Brett Byram, Gabriel Arenas, Nadav Schwartz, Ipek Oguz

Organizations: University of Pennsylvania · Vanderbilt University

Abstract

Convolutional neural networks (CNN) are widely used for medical image segmentation. However, they lack an inherent measure of confidence. Existing methods for estimating prediction uncertainty often require a custom network architecture and/or dedicated training setups and need to be trained from scratch. In addition, the deduction of a single confidence score per input from the uncertainty maps is not always straightforward. In this study, we explore voxel-based uncertainty estimation methods that can be added to existing segmentation pipelines without training from scratch. We also propose a novel method to estimate a single confidence score from the uncertainty maps. Our evaluations on different ROIs and modalities suggest that our confidence score can predict segmentation performance in unseen data.

Figures & tables

Figure 1: a) 3DUS placenta input, b) ground truth, c) uncertainty map $u(x)$ , d) thresholded segmentation mask $M_{0}(x)$ .

Figure 3: a) Segmentations with low uncertainty/high confidence. b) Segmentations with high uncertainty/low confidence. c) 1D illustration of $V_{i}$ . The blue curve is overall more uncertain than the red curve, and thus has a lower $V_{i}$ . d) Illustration of $V_{i+1}/V_{i}$ . The blue curve has a (locally) slower change in uncertainty, and thus a smaller $V_{i+1}/V_{i}$ .

Figure 4: Dice-confidence and Dice-VVC correlations. Top, placenta; middle, anterior hippocampus; bottom, posterior hippocampus. 2D models with flip aug., stdev uncertainty. Blue/red dots denote inliers/outliers (Sec. 2.5 ), with shades corresponding to individual folds. Confidence has stronger correlation with Dice than VVC across each ROI/fold.

Figure 5: Posterior hippocampus Dice, confidence, VVC. 3D models with Gaussian noise aug., stdev+entropy uncertainty. Blue/red dots denote inliers/outliers (Sec. 2.5 ).

Figure 6: Candidate selection strategies for manual segmentation. Selecting images with the worst confidence significantly outperforms random selection, and approaches the oracle strategy (selection by worst Dice).

Segmentation Uncertainty Estimation

Segmentation uncertainty estimation measures where pixel- or object-level masks may be wrong, ambiguous, or unreliable.

See all 100 papers

SegWithU: Uncertainty as Perturbation Energy for Single-Forward-Pass Risk-Aware Medical Image Segmentation
Apr 16, 2026Tianhao Fu, Austin Wang, Charles Chen +2Image Segmentation Uncertainty Quantification
University of Toronto, Toronto, ON, Canada · Vector Institute, Toronto, ON, Canada · Project Neura, Toronto, ON, Canada +4
Rethinking Uncertainty in Segmentation: From Estimation to Decision
Apr 14, 2026Saket MagantiImage Segmentation Segmentation Uncertainty Estimation
A Decision-Aware Evaluation Protocol for Uncertainty in Medical Image Segmentation
Oct 15, 2026Saket Maganti, Shreyank N GowdaUncertainty Quantification Segmentation Uncertainty Estimation
Independent Researcher · School of Computer Science, University of Nottingham, Nottingham, UK

Apr 16, 2026J/K move · Enter open · S save

SegWithU: Uncertainty as Perturbation Energy for Single-Forward-Pass Risk-Aware Medical Image Segmentation

Tianhao Fu, Austin Wang, Charles Chen, Roby Aldave-Garza+1

University of Toronto, Toronto, ON, Canada · Vector Institute, Toronto, ON, Canada · Project Neura, Toronto, ON, Canada+4

Image Segmentation Uncertainty Quantification Selective Prediction Medical Image Segmentation Segmentation Uncertainty Estimation Uncertainty Calibration

Open paper AskPDF arXiv

Reliable uncertainty estimation is critical for medical image segmentation, where automated contours feed downstream quantification and clinical decision support. Many strong uncertainty methods require repeated inference, while efficient single-forward-pass alternatives often provide weaker failure ranking or rely on restrictive feature-space assumptions. We present

\textbf{SegWithU}

, a post-hoc framework that augments a frozen pretrained segmentation backbone with a lightweight uncertainty head. SegWithU taps intermediate backbone features and models uncertainty as perturbation energy in a compact probe space using rank-1 posterior probes. It produces two voxel-wise uncertainty maps: a calibration-oriented map for probability tempering and a ranking-oriented map for error detection and selective prediction. Across ACDC, BraTS2024, and LiTS, SegWithU is the strongest and most consistent single-forward-pass baseline, achieving AUROC/AURC of

0.9838/2.4885

0.9946/0.2660

, and

0.9925/0.8193

, respectively, while preserving segmentation quality. These results suggest that perturbation-based uncertainty modeling is an effective and practical route to reliability-aware medical segmentation. Source code is available at https://github.com/ProjectNeura/SegWithU.

Abstract

Segmentation Uncertainty Estimation

Explore similar work

SegWithU: Uncertainty as Perturbation Energy for Single-Forward-Pass Risk-Aware Medical Image Segmentation

Rethinking Uncertainty in Segmentation: From Estimation to Decision

A Decision-Aware Evaluation Protocol for Uncertainty in Medical Image Segmentation