Evaluation of Uncertainty-Aware Multi-Software Ensembles for Hippocampal Segmentation

Gabriel Oliveira-Stahl; Anna Schroder; James Moggridge; Hamza A. Salhab; Caroline Micallef; Josephine Barnes; M. Jorge Cardoso; Carole H. Sudre

cs.CVOct 22, 2026

Evaluation of Uncertainty-Aware Multi-Software Ensembles for Hippocampal Segmentation

Authors: Gabriel Oliveira-Stahl, Anna Schroder, James Moggridge, Hamza A. Salhab, Caroline Micallef, Josephine Barnes, M. Jorge Cardoso, Carole H. Sudre,

Organizations: Hawkes Institute, Department of Computer Science, University College London, UK · Lysholm Department of Neuroradiology, National Hospital for Neurology and Neurosurgery, University College London Hospitals NHS Foundation Trust, London, UK · Department of Brain Repair and Rehabilitation, UCL Institute of Neurology, University College London, UK · Dementia Research Centre, University College London, UK ·

Abstract

Accurate hippocampal segmentation can be a useful tool for diagnosing and monitoring neurological conditions such as Alzheimer's disease and epilepsy. While numerous automated segmentation methods exist, their clinical adoption remains limited. Reliable uncertainty assessment can enhance trust and facilitate clinical translation. This study evaluates ensembles of five heterogeneous hippocampal segmentation methods - InnerEye, ASHS, FastSurfer, HippoSeg, and FreeSurfer - across two dementia datasets and one epilepsy dataset. The sub-ensemble containing InnerEye, FastSurfer, and HippoSeg emerged as both accurate and efficient, highlighting the feasibility of balancing computational cost and performance. Additionally, ensemble-derived uncertainty quantification with sample variance, mutual information, and predictive entropy is shown to reduce inaccurate segmentations by flagging low-confidence cases, potentially providing a mechanism for automatically escalating ambiguous cases for expert assessment. Lastly, this study explores the downstream clinical utility of ensemble-derived uncertainty, finding a significant correlation with motion artefact severity and a modest contribution to Alzheimer's disease classification.

Figures & tables

Tool	Method	Training Data / Atlas	References
ASHS (AS)	Atlas-based (Multi-atlas)	Healthy controls, MCI	( Xie and et al., 2016 )
HippoSeg (HI)	Atlas-based (Multi-atlas)	Healthy controls, Epilepsy	( Vos and et al., 2020 ) ( Winston et al., 2013 )
FreeSurfer (FR)	Atlas-based (Probabilistic)	Healthy controls, AD	( Fischl, 2012 )
FastSurfer (FA)	Deep learning (CNN)	Healthy controls, AD, ASD	( Henschel et al., 2020 )
InnerEye (IN)	Deep learning (CNN)	Healthy controls, MCI, AD	( Schroder et al., 2026 ) ( Microsoft, 2024 )

Table 1: Overview of selected hippocampal segmentation tools, their methodology, and the primary clinical cohorts comprising their training data or atlases. Abbreviations: CNN = convolutional neural network; ASD = Autism Spectrum Disorder.

Figure 1: Visualisation of uncertainty maps. Uncertainty quantified as sample variance, mutual information, and predictive entropy, all min-max normalised. Dark red line represents the gold standard boundary. Top row: High-Dice example. Bottom row: Low-Dice example.

Figure 2: Balance between accuracy, CO 2 emission, and uncertainty calibration. Benchmarking on ADNI dataset, processed on dedicated node on HPC cluster. Error bars indicate bootstrapped confidence intervals. Winning Triad : InnerEye + FastSurfer + HippoSeg, achieving comparable performance to full ensemble, while reducing computational cost by approximately 82 %. Abbreviations: HI = HippoSeg, FA = FastSurfer, FR = FreeSurfer, IN = InnerEye, AS = ASHS. AECE values colour-coded with lower values in brighter green.

Figure 3: Segmentation performance comparison. Ensemble ( Triad ) = InnerEye + FastSurfer + HippoSeg. a) Dice score comparison. Mean averaging significantly outperformed all other methods. b) Hausdorff Distance comparison. Blue background: Silent failure detection. Entropy-based filtering caught $\sim$ 92% of failed segmentations (defined as cases with DSC $<$ 0.82; see Methods Section 2.6.1 ) . The aggregated uncertainty from the Ensemble Averaged was used to discard the most uncertain hippocampi, resulting in 35% excluded predictions when using entropy, and $\sim$ 26% exclusions with variance and mutual information (averaged across datasets). * indicates significant difference to Ensemble Averaged.

Figure 4: Reliability diagram. The averaged ensemble ( Triad ) achieved the best calibration (AECE = 0.05) across all datasets. a) ADNI dataset. b) Local Epilepsy dataset. c) Local Dementia dataset. Black axes: calibration curve. Blue axis: distribution of confidence scores. Stacked bars show the percentage of voxels in each confidence bin for the respective segmentation software. Analysis includes union of predicted segmentations with a one-voxel margin.

Figure 5: Uncertainty correlation with image quality. Uncertainty increased significantly with motion artefact severity (Spearman $\rho=0.30$ – $0.36$ , $p<0.01$ ). Each point represents one hippocampus (n = 180, 164, and 280 for scores 1, 2, and 3, respectively). Quality scores were assigned by expert neuroradiologists, with 1 = good, 2 = medium, and 3 = bad. Uncertainty values are min-max normalized across all observations. * indicates significant pairwise differences (Wilcoxon signed-rank, p < 0.05, Holm-adjusted).

Figure 6: CN vs. AD classification: AUC improvement over baseline with 95% CI (5 $\times$ 5 repeated stratified cross-validation). Combining volume with all uncertainty metrics improved CN vs. AD classification by +0.06 AUC ( $p<0.001$ ). Y-axis (top to bottom): Vol_ensemble_mean = baseline (TIV-normalized bilateral hippocampal volume); Full_LR/Full_mean = all features (volumes + uncertainty metrics, L/R separate or averaged); Vol+UQ variants = volume combined with uncertainty; Vol_all_tools_mean = per-tool volumes; Vol_LR_separate = left/right volumes; Vol_raw_mean = unnormalized volume; UQ variants = uncertainty only (variance, mutual information), aggregated by mean/sum/max over ROI or whole image. All volume-based sets include age and sex as covariates.

Figure 7: Confusion matrices for CN vs. AD classification using logistic regression. The Full_mean model reclassified five additional subjects compared to the baseline with volume alone. a) Baseline Vol_ensemble_mean: TIV-normalised hippocampal volume with age and sex covariates. $AUC=0.811$ , and $BAcc=0.753$ b) Full_mean: volumes combined with voxel-wise uncertainty metrics. $AUC=0.872$ , and $BAcc=0.804$ .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S2: Confusion matrices for CN vs. MCI classification using logistic regression. a) Baseline Vol_ensemble_mean: TIV-normalised hippocampal volume with age and sex covariates. $AUC=0.691$ , and $BAcc=0.650$ . b) Full_mean: volumes combined with voxel-wise uncertainty metrics. $AUC=0.660$ , and $BAcc=0.638$ .

Figure S4: Confusion matrices for MCI vs. AD classification using logistic regression. a) Baseline Vol_ensemble_mean: TIV-normalised hippocampal volume with age and sex covariates. $AUC=0.698$ , and $BAcc=0.646$ . b) Full_mean: volumes combined with voxel-wise uncertainty metrics. $AUC=0.700$ , and $BAcc=0.677$ .

Figure S5: Axial T1-weighted slices of one subject (sub-000103) imaged under the three MR-ART conditions, illustrating the image quality levels: (a) good (artefact score 1, still acquisition), (b) medium (score 2) and (c) bad (score 3, clinically unusable), the latter two acquired during head motion. Quality scores were assigned by a neuroradiologist as part of the MR-ART dataset ( Nárai et al., 2022 )

Dataset	Tool	$n$	ECE	AECE	AECE fg	AECE bg
ADNI	InnerEye	60	0.080 [0.073, 0.088]	0.078 [0.071, 0.086]	0.208 [0.196, 0.220]	0.154 [0.146, 0.162]
ADNI	FastSurfer	60	0.148 [0.136, 0.161]	0.147 [0.134, 0.160]	0.104 [0.096, 0.113]	0.414 [0.404, 0.424]
ADNI	HippoSeg	60	0.051 [0.045, 0.058]	0.051 [0.045, 0.058]	0.345 [0.324, 0.367]	0.134 [0.128, 0.141]
ADNI	Triad (Ensemble)	60	0.047 [0.044, 0.050]	0.046 [0.043, 0.049]	0.231 [0.219, 0.243]	0.100 [0.096, 0.105]
Local Epilepsy	InnerEye	16	0.097 [0.082, 0.112]	0.096 [0.081, 0.110]	0.246 [0.225, 0.266]	0.142 [0.135, 0.150]
Local Epilepsy	FastSurfer	16	0.178 [0.156, 0.202]	0.177 [0.156, 0.202]	0.058 [0.051, 0.066]	0.499 [0.482, 0.519]

Appendix

Table S1: Supplementary calibration metrics by dataset and tool. Values are per-hippocampus means with 95% bootstrap confidence intervals over hippocampi (1000 resamples, percentile method). Headline binning uses

R=15

. ECE is equal-width and sample-weighted; AECE is adaptive equal-mass ECE; AECE fg and AECE bg are computed separately on true foreground and true background voxels.

Medical Image Segmentation

Medical image segmentation delineates organs, lesions, and other structures in scans to support measurement, diagnosis, treatment planning, and research.

See all 788 papers

U-SEG: Uncertainty in SEGmentation -- A systematic multi-variable exploration
May 14, 2026Michael Smith, Frank P. FerrieImage Segmentation Panoptic Segmentation
Centre for Intelligent Machines, McGill University
Lost in the Folds: When Cross-Validation Is Not a Deep Ensemble for Uncertainty Estimation
May 18, 2026Tristan Kirscher, Markus Bujotzek, Yannick Kirchhoff +5Deep Ensembles Medical Image Segmentation
ICube, Institut Strauss, DKFZ · ICube Laboratory, CNRS UMR-7357, University of Strasbourg, Strasbourg, France · CLCC Institut-Strauss, Strasbourg, France +6
Rethinking Uncertainty in Segmentation: From Estimation to Decision
Apr 14, 2026Saket MagantiImage Segmentation Segmentation Uncertainty Estimation

May 14, 2026J/K move · Enter open · S save

U-SEG: Uncertainty in SEGmentation -- A systematic multi-variable exploration

Michael Smith, Frank P. Ferrie

Centre for Intelligent Machines, McGill University

Image Segmentation Panoptic Segmentation Deep Ensembles Segmentation Uncertainty Estimation Semantic Segmentation Uncertainty Calibration

Open paper AskPDF arXiv

In this study, we explore in depth a few under-studied topics at the intersection of uncertainty estimation and segmentation. Prior work has shown that the quality of uncertainty estimates can be very sensitive to a range of variables. As one of the main uses of uncertainty estimation is to help identify and deal with prediction errors in practical scenarios, any factors that affect this must be clearly identified. For example, do more challenging domains or different datasets and architectures result in worse performance when using uncertainty estimates? Can prior frames in a video sequence in fact provide useful uncertainty estimates comparable to other approaches? Is it possible to combine uncertainty estimation approaches, taking advantage of sample diversity, to get better estimates? Finally, when might it make sense to use an ensemble-based uncertainty estimate over a deterministic network? We address these questions by creating a framework for and executing a large scale study across many variables such as datasets, backbones, and downstream tasks, for both semantic and panoptic segmentation. We find that a) the more challenging task of panoptic segmentation usually results in worse performance while high performance variance between datasets and backbones indicates that generalization is not guaranteed, b) time series samples can be useful for specific configurations, but in many cases are not worth the cost, c) sample diversity shows the most promise in the downstream task of calibration, but otherwise fails to beat simpler alternatives, d) a deterministic approach is adequate for some downstream tasks, but ensembles allow for significant improvements if the right conditions can be achieved in deployment.

Abstract

Medical Image Segmentation

Explore similar work

U-SEG: Uncertainty in SEGmentation -- A systematic multi-variable exploration

Lost in the Folds: When Cross-Validation Is Not a Deep Ensemble for Uncertainty Estimation

Rethinking Uncertainty in Segmentation: From Estimation to Decision