Delving Aleatoric Uncertainty in Medical Image Segmentation via Vision Foundation Models

Ruiyang Li; Fang Liu; Licheng Jiao; Xinglin Xie; Jiayao Hao; Shuo Li; Xu Liu; Jingyi Yang

cs.AIApr 13, 2026

Delving Aleatoric Uncertainty in Medical Image Segmentation via Vision Foundation Models

Authors: Ruiyang Li, Fang Liu, Licheng Jiao, Xinglin Xie, Jiayao Hao, Shuo Li, Xu Liu, Jingyi Yang, +3 more

Organizations: Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education, International Research Center for Intelligent Perception and Computation, Joint International Research Laboratory of Intelligent Perception and Computation, School of Artificial Intelligence, Xidian University, Xi’an 710071, China

Abstract

Medical image segmentation supports clinical workflows by precisely delineating anatomical structures and lesions. However, medical image datasets medical image datasets suffer from acquisition noise and annotation ambiguity, causing pervasive data uncertainty that substantially undermines model robustness. Existing research focuses primarily on model architectural improvements and predictive reliability estimation, while systematic exploration of the intrinsic data uncertainty remains insufficient. To address this gap, this work proposes leveraging the universal representation capabilities of visual foundation models to estimate inherent data uncertainty. Specifically, we analyze the feature diversity of the model's decoded representations and quantify their singular value energy to define the semantic perception scale for each class, thereby measuring sample difficulty and aleatoric uncertainty. Based on this foundation, we design two uncertainty-driven application strategies: (1) the aleatoric uncertainty-aware data filtering mechanism to eliminate potentially noisy samples and enhance model learning quality; (2) the dynamic uncertainty-aware optimization strategy that adaptively adjusts class-specific loss weights during training based on the semantic perception scale, combined with a label denoising mechanism to improve training stability. Experimental results on five public datasets encompassing CT and MRI modalities and involving multi-organ and tumor segmentation tasks demonstrate that our method achieves significant and robust performance improvements across various mainstream network architectures, revealing the broad application potential of aleatoric uncertainty in medical image understanding and segmentation tasks.

Figures & tables

Figure 1: Visualization of low (high) difficulty training samples scored by MedSAM2 via derived Aleatoric Uncertainty Values (AUV). Lower AUV indicates easier-to-learn samples.

Figure 2: Singular value decay (left) and cumulative energy (right), illustrating the disparities in feature distribution across four samples of varying difficulty from the LiTS (Tumor) dataset.

Figure 3: Illustrative pipeline of the quantifying aleatoric uncertainty (orange part in (a)). Two applications of aleatoric uncertainty study in medical image segmentation tasks: aleatoric uncertainty-aware data filter (a) and dynamic uncertainty-aware optimization strategy (b).

Quantifying AUV via Data Variance.
Quantification	Data	LiTS		TotalSeg		WORD		FeTA		KiTS23		Mean
Strategy		Dice(%)	mIoU(%)	Dice(%)	mIoU(%)	Dice(%)	mIoU(%)	Dice(%)	mIoU(%)	Dice(%)	mIoU(%)
Baseline	100%	82.96	73.24	80.34	72.71	79.03	69.85	84.30	74.63	70.15	63.76	75.10
Variance [ 42 ]	95%	83.00 $\uparrow$	73.27 $\uparrow$	80.29 $\downarrow$	72.66 $\downarrow$	81.43 $\uparrow$	71.99 $\uparrow$	84.50 $\uparrow$	74.76 $\uparrow$	68.05 $\downarrow$	61.97 $\downarrow$	75.19 $\uparrow$
	90%	81.72 $\downarrow$	72.15 $\downarrow$	80.37 $\uparrow$	72.93 $\uparrow$	81.67 $\uparrow$	72.15 $\uparrow$	84.52 $\uparrow$	74.78 $\uparrow$	69.96 $\downarrow$	63.66 $\downarrow$	75.39 $\uparrow$
Quantifying AUV via Task-specific Models.

Table 1: Performance of aleatoric uncertainty-aware data filtering strategies based on multiple medical foundation models and two baseline strategies across five medical segmentation datasets. Here, ‘95%’ indicates the removal of ‘5%’ of noisy samples.

Method	DUO	LiTS		TotalSeg		WORD		FeTA		KiTS23		Mean
		Dice(%)	mIoU(%)	Dice(%)	mIoU(%)	Dice(%)	mIoU(%)	Dice(%)	mIoU(%)	Dice(%)	mIoU(%)
nnU-Net [ 11 ]	$\times$	82.96	73.24	80.34	72.71	79.03	69.85	84.30	74.63	70.15	63.76	75.10
	$\checkmark$	84.13 $\uparrow$	74.73 $\uparrow$	80.82 $\uparrow$	73.14 $\uparrow$	79.63 $\uparrow$	70.38 $\uparrow$	84.63 $\uparrow$	74.91 $\uparrow$	70.51 $\uparrow$	63.97 $\uparrow$	75.69 $\uparrow$
Swin-UNETR [ 8 ]	$\times$	80.54	71.10	77.38	70.03	77.26	68.26	84.15	74.48	66.89	60.79	73.09
	$\checkmark$	81.25 $\uparrow$	71.72 $\uparrow$	77.94 $\uparrow$	70.51 $\uparrow$	77.90 $\uparrow$	68.85 $\uparrow$	84.53 $\uparrow$	74.82 $\uparrow$	67.57 $\uparrow$	61.32 $\uparrow$	73.64 $\uparrow$
U-Mamba [ 26 ]	$\times$	81.24	71.72	79.27	71.74	78.93	69.76	84.42	74.72	67.85	61.67	74.13

Table 2: Performance of the Dynamic Uncertainty-aware Optimization (DUO) strategy on three baseline networks.

Figure 4: The comparison of segmentation boundaries from three baseline networks on tumor (LiTS) and multi-organ (WORD) dataset. Ground truth (red) is compared against the baseline (blue) and the DUO strategy (orange).

Method	90 % data		+DUO
	Liver	Tumor	Liver	Tumor
Fisher [ 6 ]	96.58	70.17 +0.86	96.59	70.01 +0.70
MD [ 4 ]	96.56	69.78 +0.47	96.57	69.28 -0.03
EAOA [ 43 ]	96.61	70.41 +1.10	$-$	$-$
$\mathcal{S}(\cdot)$	96.61	74.28 +4.97	96.60	71.66 +2.35

Table 3: Comparison of different uncertainty quantification methods on tumor segmentation (Dice %) in two medical applications of the LiTS dataset.

Figure 5: The correspondence between the feature vectors $Z$ of the liver and tumors predicted by MedSAM2 and the aleatoric uncertainty values is visualized through t-SNE.

Properties	Pearson	P-value	Spearman	P-value
Fisher [ 6 ]	0.4402	0.0880	0.6623	0.0052
MD [ 4 ]	0.3620	0.1683	0.4618	0.0718
$\mathcal{S}(\cdot)$	0.6267	0.0094	0.7471	0.0009

Table 4: Correlation analysis between quantified values from different methods and predicted Dice scores

strategy	90 % data		95 % data
	Dice (%)	mIoU (%)	Dice (%)	mIoU (%)
(a)	73.35 +4.04	63.44 +2.99	69.47 +0.16	59.43 +0.12
(b)	72.38 +2.53	62.32 +1.87	69.76 +0.45	59.73 +0.42
(c)	74.28 +4.97	63.69 +3.24	70.03 +0.72	59.88 +0.57
(d)	73.74 +4.43	63.29 +2.84	70.26 +0.95	60.07 +0.76

Table 5: The impact of different data filtering strategies on tumor segmentation performance in LiTS.

Figure 6: The histograms of AUV based on two different image inputs for both entire and specific class analyses in the LiTS dataset.

EN	RW	Params(M)	Memory(M)	Liver	Tumor
$\times$	$\times$	30.80	6105	96.61	69.31
$\checkmark$	$\times$	30.80	6105	96.60	69.87
$\times$	$\checkmark$	30.81	6287	96.59	71.37
$\checkmark$	$\checkmark$	30.81	6287	96.61	71.66

Table 6: Ablation study on the design of dynamic uncertainty-Aware optimization in the LiTS Dataset.

Figure 7: Validation Loss and local Dice score curves of the dynamic uncertainty-aware optimization strategy during training.

Explore similar work

Apr 16, 2026

SegWithU: Uncertainty as Perturbation Energy for Single-Forward-Pass Risk-Aware Medical Image Segmentation

Reliable uncertainty estimation is critical for medical image segmentation, where automated contours feed downstream quantification and clinical decision support. Many strong uncertainty methods require repeated inference, while efficient single-forward-pass alternatives often provide weaker failure ranking or rely on restrictive feature-space assumptions. We present

\textbf{SegWithU}

, a post-hoc framework that augments a frozen pretrained segmentation backbone with a lightweight uncertainty head. SegWithU taps intermediate backbone features and models uncertainty as perturbation energy in a compact probe space using rank-1 posterior probes. It produces two voxel-wise uncertainty maps: a calibration-oriented map for probability tempering and a ranking-oriented map for error detection and selective prediction. Across ACDC, BraTS2024, and LiTS, SegWithU is the strongest and most consistent single-forward-pass baseline, achieving AUROC/AURC of

0.9838/2.4885

0.9946/0.2660

, and

0.9925/0.8193

, respectively, while preserving segmentation quality. These results suggest that perturbation-based uncertainty modeling is an effective and practical route to reliability-aware medical segmentation. Source code is available at https://github.com/ProjectNeura/SegWithU.

Tianhao Fu, Austin Wang, Charles Chen +2

University of Toronto, Toronto, ON, Canada · Vector Institute, Toronto, ON, Canada · Project Neura, Toronto, ON, Canada +4

Image Segmentation Uncertainty Quantification Selective Prediction Medical Image Segmentation Segmentation Uncertainty Estimation Uncertainty Calibration

Ask PDF

Jun 17, 2026

Quantification of Uncertainty with Adversarial Models in Medical Image Segmentation

Reliable pixel-level uncertainty quantification holds the potential to transform clinical workflows by enabling high-fidelity longitudinal monitoring and distinguishing true pathological changes from artifacts. Ideally, these models provide the stability required for critical treatment planning and surgical intervention. However, standard deep learning models often suffer from miscalibration, yielding overconfident predictions that mask underlying vulnerabilities at subtle pathological boundaries. To address this, we propose QUAM-SM, a post-hoc framework using targeted adversarial search to identify "adversarially fragile" pixels. By actively seeking perturbations that expose predictive instability, our method highlights regions where decisions are most vulnerable to being flipped. Importantly, the framework disentangles epistemic uncertainty from aleatoric uncertainty. Experiments on two public datasets with multiple expert annotations demonstrate that QUAM-SM outperforms both standard and recent uncertainty estimation approaches in terms of reliability and boundary sensitivity. Code is available at https://github.com/HanaJebril/quam_sm

Hana Jebril, Thomas Pinetz, Günter Klambauer +1

Institute of Artificial Intelligence, Center for Medical Data Science, Medical University of Vienna, Austria · Comprehensive Center for AI in Medicine, Medical University of Vienna, Austria · ELLIS Unit Linz, LIT AI Lab and Institute for Machine Learning, Johannes Kepler University Linz, Austria +1

Uncertainty Quantification Medical Image Segmentation Segmentation Uncertainty Estimation

Ask PDF

Apr 14, 2026

Rethinking Uncertainty in Segmentation: From Estimation to Decision

In medical image segmentation, uncertainty estimates are often reported but rarely used to guide decisions. We study the missing step: how uncertainty maps are converted into actionable policies such as accepting, flagging, or deferring predictions. We formulate segmentation as a two-stage pipeline, estimation followed by decision, and show that optimizing uncertainty alone fails to capture most of the achievable safety gains. Using retinal vessel segmentation benchmarks (DRIVE, STARE, CHASE_DB1), we evaluate two uncertainty sources (Monte Carlo Dropout and Test-Time Augmentation) combined with three deferral strategies, and introduce a simple confidence-aware deferral rule that prioritizes uncertain and low-confidence predictions. Our results show that the best method and policy combination removes up to 80 percent of segmentation errors at only 25 percent pixel deferral, while achieving strong cross-dataset robustness. We further show that calibration improvements do not translate to better decision quality, highlighting a disconnect between standard uncertainty metrics and real-world utility. These findings suggest that uncertainty should be evaluated based on the decisions it enables, rather than in isolation.

Saket Maganti

Image Segmentation Segmentation Uncertainty Estimation

Ask PDF