Better than Average: Spatially-Aware Aggregation of Segmentation Uncertainty Improves Downstream Performance

Vanessa Emanuela Guarino; Claudia Winklmayr; Jannik Franzen; Josef Lorenz Rumberger; Manuel Pfeuffer; Sonja Greven; Klaus Maier-Hein; Carsten T. Lüth

Figure 1: Aggregation strategies (AggS). (a) An AggS reduces an uncertainty map to a single scalar score. (b) Empirical evaluation of AggSs. Choosing an appropriate AggS is challenging and highly task-dependent. Hence, we benchmark different strategies in the context of OoD- and failure detection over different datasets. (c) Limitations of individual AggSs. Pixel-wise strategies, in particular, have key shortcomings e.g . 1. AVG ignores spatial structure ; 2. AQA lacks proportional invariance ; and 3. ATA is not monotonic (see Supp. A for more details). (d) Subtypes of AggSs. Beyond pixel-wise AggSs, we also explore spatially-aware approaches—such as prediction-based and spatial AggSs, which measure the fraction of uncertainty mass within structured regions of the uncertainty map. Finally, we consider Meta-AggSs, which combine intensity-based and spatial strategies by fitting a Gaussian Mixture Model (GMM).

Figure 2: Structural diversity of datasets. The scatterplot illustrates structural diversity by projecting uncertainty maps into the space of two SMR scores: MOR and EDS. High MOR indicates clustered uncertainty (low: noise), while high EDS reflects edge-localized uncertainty (low: flat regions). Notably, EDS effectively identifies OoD samples in CAR-ID/CAR-CS.

Figure 3: Performance on OoD Detection. Higher AUROC (0.6–1.0) indicates better iD–OoD separation. AggSs are ranked by their mean AUROC, computed over 500 bootstrap samples per dataset and then averaged across datasets for stability. Ranking robustness is assessed via one-sided Wilcoxon signed-rank tests at 5%, showing that prediction-based BCA and ICA, along with GMM-based methods, form a statistically dominant tier (

p<0.05

). Numbers after AggS labels indicate method-specific parameters (cf. Supp. A.1 ). Full analysis, including additional UQ methods and confidence intervals, is in Supp. C .

Figure 4: GMM Robustness for OoD Detection. (a) Fitting a GMM enables spatial methods to act as AggSs, capturing subtle non-linear changes in uncertainty maps. (b) Qualitative examples showing that effective AggSs for iD–OoD separation are data-dependent. (c) GMM-All AUROC: leave-one-out (top) and individual fitting (bottom) on a subset (CAR-CS, LIZ-IG, WEED-Hand, and WORM-Nem split). Using all AggSs generally matches or outperforms individual ones in 3 cases, with minimal impact from removing specific ones. Exceptions occur when a feature dominates ( e.g . , EDS for CAR-CS, marked as Tukey outlier) or when features lack discriminative power. (d) GMM-All AUROC: absolute SHAP values. Positive values indicate that an AggS aids GMM-All in separating iD–OoD; bi-directional or null contributions reduce performance (e.g., in LIZ-IG). SHAP values are shown for the same subsets as in (c).

Figure 5: Performance on FD. (a) Exemplary Selective Risk-Coverage curves . (b) E-AURC scores . Lower values indicate better alignment between uncertainty and prediction errors. AggSs are ranked by their mean E-AURC, computed over 500 bootstrap samples per dataset and then averaged across datasets for stability. Ranking robustness is assessed via one-sided Wilcoxon signed-rank tests at 5%, showing that prediction-based QFR shows statistically significant improvement over all other AggSs (

p<0.001

), followed by BCA and GMM-based methods (

p<0.05

). Full analysis, including additional UQ methods and confidence intervals, is in Supp. C .

	PF	M	PI	L
Global Average (AVG)	✓	✓	✗	✗
Above-Threshold Average (ATA)	✗	✗	✓	✗
Above-Quantile Average (AQA)	✗	✓	✗	✗
Patch-Level Maximum (PLM)	✗	✓	✓	✓
Weighted Class Average (WCA)	✗	✓	✗ ∗	✗

Table A.1 : Overview of key properties of selected AggSs. PF: Parameter-free, M: Monotonic, PI: Proportion invariant, L: Local. ∗ WCA becomes proportion invariant when the background class is excluded in specific use cases where foreground classes exhibit high uncertainty and the background class has low uncertainty.

Figure A.1 : Non-monotonicity of ATA. Left is shown a

3\times 3

uncertainty map. If we increase each pixel value by

\Delta

the ATA score (threshold

T=0.7

) will drop at a value of

\Delta=0.1

, since the amount of pixels affecting the average suddenly increases as their values pass the threshold.

Figure B.1 : Spatial decomposition using Moran’s I. For an uncertainty map

U

we compute the pixelwise local spatial measure w.r.t. to Moran’s I which captures noise (

I=0

) and clusters (

I=1

). The local spatial measures can be interpreted as weights of a weight matrix

W

where high weights correspond to high spatial auto-correlation. The pixelwise product of

U

and

W

is

U_{\text{high}}

while the pixelwise product of

U

and

1-W

is

U_{\text{low}}

, which yields a matrix decomposition of the original uncertainty map.

	a AUROC
BCA	.72 $\pm$ .06	.79 $\pm$ .05	.89 $\pm$ .01	.57 $\pm$ .05	.82 $\pm$ .07	.68 $\pm$ .02	.68 $\pm$ .02	.58 $\pm$ .06	.77 $\pm$ .06	.95 $\pm$ .02	5.2
GMM-All	.84 $\pm$ .05	.87 $\pm$ .05	1.00 $\pm$ .00	.86 $\pm$ .03	.77 $\pm$ .05	.45 $\pm$ .03	.44 $\pm$ .03	.95 $\pm$ .03	1.00 $\pm$ .00	1.00 $\pm$ .00	5.2
ICA	.60 $\pm$ .07	.83 $\pm$ .05	.84 $\pm$ .02	.57 $\pm$ .05	.82 $\pm$ .07	.71 $\pm$ .02	.59 $\pm$ .03	.57 $\pm$ .06	.77 $\pm$ .06	.95 $\pm$ .02	5.9
GMM-Int	.79 $\pm$ .06	.83 $\pm$ .06	.73 $\pm$ .02	.86 $\pm$ .03	.78 $\pm$ .05	.49 $\pm$ .03	.43 $\pm$ .03	.91 $\pm$ .04	.98 $\pm$ .01	1.00 $\pm$ .00	6.0
QFR	.62 $\pm$ .07	.86 $\pm$ .04	.62 $\pm$ .02	.54 $\pm$ .05	.89 $\pm$ .05	.68 $\pm$ .03	.57 $\pm$ .03	.58 $\pm$ .06	.67 $\pm$ .06	.91 $\pm$ .03	6.6
GMM-Spa	.93 $\pm$ .03	.66 $\pm$ .07	1.00 $\pm$ .00	.67 $\pm$ .04	.72 $\pm$ .07	.49 $\pm$ .03	.41 $\pm$ .03	.85 $\pm$ .05	.89 $\pm$ .04	.90 $\pm$ .03	7.1

Table C.1 : Performance of AggSs on uncertainty maps generated with MC Dropout (MCD) in OoD and failure detection. Columns are color-coded with a red-to-green gradient, where white represents the mean (for AUROC this is equivalent to random guessing). (a) Higher AUROC values (greener cells, 0.6–1.0) indicate better iD vs. OoD separation, while (b) lower E-AURC values (greener) indicate better alignment between uncertainty estimates and prediction errors. AggSs are ranked from best (top) to worst (bottom) based on their average metric, first computed across 500 bootstrap samples per dataset, and then averaged across datasets to ensure stable rankings (rightmost column). Each value is reported with the standard deviation across bootstrap samples. These tables complement Figures 3 and 5 b.

	a AUROC
GMM-All	.81 $\pm$ .05	.85 $\pm$ .05	1.00 $\pm$ .00	.92 $\pm$ .03	.55 $\pm$ .08	.55 $\pm$ .03	.45 $\pm$ .03	.93 $\pm$ .03	.99 $\pm$ .01	1.00 $\pm$ .00	5.4
BCA	.87 $\pm$ .04	.75 $\pm$ .06	.90 $\pm$ .01	.60 $\pm$ .05	.64 $\pm$ .09	.66 $\pm$ .02	.73 $\pm$ .02	.62 $\pm$ .06	.69 $\pm$ .06	.86 $\pm$ .04	5.6
ICA	.83 $\pm$ .05	.78 $\pm$ .06	.80 $\pm$ .02	.61 $\pm$ .05	.64 $\pm$ .09	.70 $\pm$ .02	.66 $\pm$ .02	.61 $\pm$ .06	.68 $\pm$ .06	.86 $\pm$ .03	5.9
QFR	.85 $\pm$ .05	.81 $\pm$ .06	.56 $\pm$ .02	.60 $\pm$ .05	.62 $\pm$ .08	.73 $\pm$ .02	.71 $\pm$ .02	.60 $\pm$ .06	.66 $\pm$ .06	.89 $\pm$ .03	5.9
GMM-Int	.81 $\pm$ .05	.76 $\pm$ .06	.80 $\pm$ .02	.91 $\pm$ .03	.58 $\pm$ .07	.52 $\pm$ .03	.46 $\pm$ .03	.94 $\pm$ .03	.99 $\pm$ .01	1.00 $\pm$ .00	6.3
GMM-Spa	.84 $\pm$ .06	.73 $\pm$ .07	1.00 $\pm$ .00	.70 $\pm$ .05	.58 $\pm$ .07	.51 $\pm$ .02	.52 $\pm$ .03	.89 $\pm$ .03	.82 $\pm$ .05	.87 $\pm$ .04	6.7

Table C.2 : AggSs performance on uncertainty maps generated with Maximum Softmax Probability (MSP) in OoD and failure detection. Color coding and metric interpretation are as detailed in Table A.1. (a)-(b) AggSs are ranked from best (top) to worst (bottom) based on their average metric, first computed across 500 bootstrap samples per dataset, and then averaged across datasets to ensure stable rankings. Each value is reported with the std. deviation across bootstrap samples.

	a AUROC
GMM-All	.83 $\pm$ .05	.84 $\pm$ .06	.99 $\pm$ .00	.86 $\pm$ .03	.70 $\pm$ .07	4.8
QFR	.90 $\pm$ .04	.81 $\pm$ .06	.67 $\pm$ .02	.77 $\pm$ .04	.88 $\pm$ .04	4.8
GMM-Spa	.93 $\pm$ .03	.72 $\pm$ .06	1.00 $\pm$ .00	.83 $\pm$ .03	.70 $\pm$ .06	5.0
BCA	.89 $\pm$ .04	.76 $\pm$ .06	.93 $\pm$ .01	.71 $\pm$ .04	.82 $\pm$ .06	5.2
ICA	.88 $\pm$ .05	.79 $\pm$ .06	.89 $\pm$ .01	.70 $\pm$ .04	.82 $\pm$ .07	5.4
GMM-Int	.79 $\pm$ .05	.76 $\pm$ .06	.79 $\pm$ .02	.89 $\pm$ .03	.68 $\pm$ .06	6.4

Table C.3 : AggSs performance on ensembling heatmaps in OoD and failure detection. Color coding and metric interpretation are as detailed in Table A.1. (a)-(b) AggSs are ranked from best (top) to worst (bottom) based on their average metric, first computed across 500 bootstrap samples per dataset, and then averaged across datasets to ensure stable rankings. Each value is reported with the std. deviation across bootstrap samples.

	a AUROC
QFR	.85 $\pm$ .05	.88 $\pm$ .04	.68 $\pm$ .04	.81 $\pm$ .07	4.5
BCA	.81 $\pm$ .06	.87 $\pm$ .04	.70 $\pm$ .04	.73 $\pm$ .09	5.0
GMM-All	.72 $\pm$ .07	.88 $\pm$ .05	.89 $\pm$ .03	.73 $\pm$ .05	5.0
ICA	.74 $\pm$ .06	.88 $\pm$ .04	.70 $\pm$ .04	.72 $\pm$ .09	5.5
AQA .9	.81 $\pm$ .05	.74 $\pm$ .06	.95 $\pm$ .02	.51 $\pm$ .06	6.5
PLM 20	.73 $\pm$ .06	.78 $\pm$ .06	.95 $\pm$ .02	.50 $\pm$ .07	6.8

Table C.4 : AggSs performance on TTA heatmaps AggSs in OoD and failure detection. Color coding and metric interpretation are as detailed in Table A.1. (a)-(b) AggSs are ranked from best (top) to worst (bottom) based on their average metric, first computed across 500 bootstrap samples per dataset, and then averaged across datasets to ensure stable rankings. Each value is reported with the std. deviation across bootstrap samples.

Figure D.1 : Illustration of dataset diversity as captured by different aggregation measures. Unlike the projection of uncertainty maps into EDS–MOR space (cf. Fig. 2 , with non-highlighted datasets shown in gray), projections onto spaces defined by randomly chosen pairs of pixel-intensities or prediction-aware AggSs do not reveal a clear separation of the datasets. Nevertheless, the two-dimensional configuration of CAR-ID and CAR-CS points suggests that a separation might emerge in a multidimensional space incorporating all AggSs.

Dataset	iD Samples	OoD Samples
ARC-BC	25	50
ARC-Nuc	25	50
LIDC-Mal	53	93
LIDC-Tex	84	20
WORM-Nem	25	47
WORM-Pro	25	82

Table D.1 : Sample counts for iD and OoD splits across evaluation datasets.

Figure G.1 : Effect of PCA. Example of feature collapse resulting from computing a 1D PCA projection in 2D space using only iD samples, and subsequently applying this projection to both iD and OoD samples. The resulting histogram (inset) illustrates that this approach yields poor separability between iD and OoD data.

Figure G.2 : Data Normalization. AUROC scores from our OOD detection experiment are shown for four different normalization methods: Beta CDF Gaussianizer, Standard Scaler, Quantile Transformer, and no normalization. The three bar plots represent the performance of the methods GMM-All , GMM-Int , and GMM-Spa under each normalization setting.

Figure G.3 : Utilizing spatial measures as AggSs. Histograms of individual AggSs for CAR-ID and CAR-CS are shown along with: 1. a GMM fitted on an an iD subset of the individual score (solid line), and 2. the diagonal component of a GMM fitted on all scores (dotted line). This empirically shows that spatial measures can be leveraged by a GMM to detect OoD samples, even when the shift from iD to OoD is toward lower aggregated values.

Figure G.4 : GMM Robustness for OoD Detection for all datasets. (a) GMM-All AUROC: leave-one-out (top) and individual fitting (bottom). Using all AggSs generally matches or outperforms individual ones in 6 of 10 cases, with minimal impact from removing specific AggSs. Exceptions occur when a feature dominates ( e.g . , EDS for CAR-CS or ICA in LIDC-MAL, marked in red as Tukey outliers) or when features lack discriminative power between iD and OoD samples ( e.g . , LIZ-SG). The most discriminative AggSs for ARC-BC, ARC-Nuc, CAR-CS, LIDC-MAL, and WEED-Hand are typically spatial mass ratios or prediction-based (proportion-invariant). (b) GMM-All AUROC: absolute SHAP values. Positive values indicate that an AggS helps GMM-All separate iD–OoD, confirming the results observed in (a); bi-directional or null contributions reduce performance (e.g., in LIZ-IG).

Figure G.5 : AggSs behavior under gradual perturbations. For varying perturbation levels of a) ARC-Nuc and b) ARC-BC we display the sample-wise behavior of selected AggSs, i.e. an increase with increasing perturbation parameter manipulation is marked as light blue connection, a decrease as red arrow. The overall fraction between de- and increases is inset in the upper left corner of each AggSs plot.

Better than Average: Spatially-Aware Aggregation of Segmentation Uncertainty Improves Downstream Performance

Abstract

Figures & tables

Uncertainty Quantification

U-SEG: Uncertainty in SEGmentation -- A systematic multi-variable exploration

Rethinking Uncertainty Quantification and Entanglement in Image Segmentation

Rethinking Uncertainty in Segmentation: From Estimation to Decision

Abstract

Uncertainty Quantification

Explore similar work

U-SEG: Uncertainty in SEGmentation -- A systematic multi-variable exploration

Rethinking Uncertainty Quantification and Entanglement in Image Segmentation

Rethinking Uncertainty in Segmentation: From Estimation to Decision