Delving Aleatoric Uncertainty in Medical Image Segmentation via Vision Foundation Models

Ruiyang Li; Fang Liu; Licheng Jiao; Xinglin Xie; Jiayao Hao; Shuo Li; Xu Liu; Jingyi Yang

cs.AIApr 13, 2026

Delving Aleatoric Uncertainty in Medical Image Segmentation via Vision Foundation Models

Authors: Ruiyang Li, Fang Liu, Licheng Jiao, Xinglin Xie, Jiayao Hao, Shuo Li, Xu Liu, Jingyi Yang, +3 more

Organizations: Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education, International Research Center for Intelligent Perception and Computation, Joint International Research Laboratory of Intelligent Perception and Computation, School of Artificial Intelligence, Xidian University, Xi’an 710071, China

Abstract

Medical image segmentation supports clinical workflows by precisely delineating anatomical structures and lesions. However, medical image datasets medical image datasets suffer from acquisition noise and annotation ambiguity, causing pervasive data uncertainty that substantially undermines model robustness. Existing research focuses primarily on model architectural improvements and predictive reliability estimation, while systematic exploration of the intrinsic data uncertainty remains insufficient. To address this gap, this work proposes leveraging the universal representation capabilities of visual foundation models to estimate inherent data uncertainty. Specifically, we analyze the feature diversity of the model's decoded representations and quantify their singular value energy to define the semantic perception scale for each class, thereby measuring sample difficulty and aleatoric uncertainty. Based on this foundation, we design two uncertainty-driven application strategies: (1) the aleatoric uncertainty-aware data filtering mechanism to eliminate potentially noisy samples and enhance model learning quality; (2) the dynamic uncertainty-aware optimization strategy that adaptively adjusts class-specific loss weights during training based on the semantic perception scale, combined with a label denoising mechanism to improve training stability. Experimental results on five public datasets encompassing CT and MRI modalities and involving multi-organ and tumor segmentation tasks demonstrate that our method achieves significant and robust performance improvements across various mainstream network architectures, revealing the broad application potential of aleatoric uncertainty in medical image understanding and segmentation tasks.

Figures & tables

Figure 1: Visualization of low (high) difficulty training samples scored by MedSAM2 via derived Aleatoric Uncertainty Values (AUV). Lower AUV indicates easier-to-learn samples.

Figure 2: Singular value decay (left) and cumulative energy (right), illustrating the disparities in feature distribution across four samples of varying difficulty from the LiTS (Tumor) dataset.

Figure 3: Illustrative pipeline of the quantifying aleatoric uncertainty (orange part in (a)). Two applications of aleatoric uncertainty study in medical image segmentation tasks: aleatoric uncertainty-aware data filter (a) and dynamic uncertainty-aware optimization strategy (b).

Quantifying AUV via Data Variance.
Quantification	Data	LiTS		TotalSeg		WORD		FeTA		KiTS23		Mean
Strategy		Dice(%)	mIoU(%)	Dice(%)	mIoU(%)	Dice(%)	mIoU(%)	Dice(%)	mIoU(%)	Dice(%)	mIoU(%)
Baseline	100%	82.96	73.24	80.34	72.71	79.03	69.85	84.30	74.63	70.15	63.76	75.10
Variance [ 42 ]	95%	83.00 $\uparrow$	73.27 $\uparrow$	80.29 $\downarrow$	72.66 $\downarrow$	81.43 $\uparrow$	71.99 $\uparrow$	84.50 $\uparrow$	74.76 $\uparrow$	68.05 $\downarrow$	61.97 $\downarrow$	75.19 $\uparrow$
	90%	81.72 $\downarrow$	72.15 $\downarrow$	80.37 $\uparrow$	72.93 $\uparrow$	81.67 $\uparrow$	72.15 $\uparrow$	84.52 $\uparrow$	74.78 $\uparrow$	69.96 $\downarrow$	63.66 $\downarrow$	75.39 $\uparrow$
Quantifying AUV via Task-specific Models.

Table 1: Performance of aleatoric uncertainty-aware data filtering strategies based on multiple medical foundation models and two baseline strategies across five medical segmentation datasets. Here, ‘95%’ indicates the removal of ‘5%’ of noisy samples.

Method	DUO	LiTS		TotalSeg		WORD		FeTA		KiTS23		Mean
		Dice(%)	mIoU(%)	Dice(%)	mIoU(%)	Dice(%)	mIoU(%)	Dice(%)	mIoU(%)	Dice(%)	mIoU(%)
nnU-Net [ 11 ]	$\times$	82.96	73.24	80.34	72.71	79.03	69.85	84.30	74.63	70.15	63.76	75.10
	$\checkmark$	84.13 $\uparrow$	74.73 $\uparrow$	80.82 $\uparrow$	73.14 $\uparrow$	79.63 $\uparrow$	70.38 $\uparrow$	84.63 $\uparrow$	74.91 $\uparrow$	70.51 $\uparrow$	63.97 $\uparrow$	75.69 $\uparrow$
Swin-UNETR [ 8 ]	$\times$	80.54	71.10	77.38	70.03	77.26	68.26	84.15	74.48	66.89	60.79	73.09
	$\checkmark$	81.25 $\uparrow$	71.72 $\uparrow$	77.94 $\uparrow$	70.51 $\uparrow$	77.90 $\uparrow$	68.85 $\uparrow$	84.53 $\uparrow$	74.82 $\uparrow$	67.57 $\uparrow$	61.32 $\uparrow$	73.64 $\uparrow$
U-Mamba [ 26 ]	$\times$	81.24	71.72	79.27	71.74	78.93	69.76	84.42	74.72	67.85	61.67	74.13

Table 2: Performance of the Dynamic Uncertainty-aware Optimization (DUO) strategy on three baseline networks.

Figure 4: The comparison of segmentation boundaries from three baseline networks on tumor (LiTS) and multi-organ (WORD) dataset. Ground truth (red) is compared against the baseline (blue) and the DUO strategy (orange).

Method	90 % data		+DUO
	Liver	Tumor	Liver	Tumor
Fisher [ 6 ]	96.58	70.17 +0.86	96.59	70.01 +0.70
MD [ 4 ]	96.56	69.78 +0.47	96.57	69.28 -0.03
EAOA [ 43 ]	96.61	70.41 +1.10	$-$	$-$
$\mathcal{S}(\cdot)$	96.61	74.28 +4.97	96.60	71.66 +2.35

Table 3: Comparison of different uncertainty quantification methods on tumor segmentation (Dice %) in two medical applications of the LiTS dataset.

Figure 5: The correspondence between the feature vectors $Z$ of the liver and tumors predicted by MedSAM2 and the aleatoric uncertainty values is visualized through t-SNE.

Properties	Pearson	P-value	Spearman	P-value
Fisher [ 6 ]	0.4402	0.0880	0.6623	0.0052
MD [ 4 ]	0.3620	0.1683	0.4618	0.0718
$\mathcal{S}(\cdot)$	0.6267	0.0094	0.7471	0.0009

Table 4: Correlation analysis between quantified values from different methods and predicted Dice scores

strategy	90 % data		95 % data
	Dice (%)	mIoU (%)	Dice (%)	mIoU (%)
(a)	73.35 +4.04	63.44 +2.99	69.47 +0.16	59.43 +0.12
(b)	72.38 +2.53	62.32 +1.87	69.76 +0.45	59.73 +0.42
(c)	74.28 +4.97	63.69 +3.24	70.03 +0.72	59.88 +0.57
(d)	73.74 +4.43	63.29 +2.84	70.26 +0.95	60.07 +0.76

Table 5: The impact of different data filtering strategies on tumor segmentation performance in LiTS.

Figure 6: The histograms of AUV based on two different image inputs for both entire and specific class analyses in the LiTS dataset.

EN	RW	Params(M)	Memory(M)	Liver	Tumor
$\times$	$\times$	30.80	6105	96.61	69.31
$\checkmark$	$\times$	30.80	6105	96.60	69.87
$\times$	$\checkmark$	30.81	6287	96.59	71.37
$\checkmark$	$\checkmark$	30.81	6287	96.61	71.66

Table 6: Ablation study on the design of dynamic uncertainty-Aware optimization in the LiTS Dataset.

Figure 7: Validation Loss and local Dice score curves of the dynamic uncertainty-aware optimization strategy during training.

Explore similar work

SegWithU: Uncertainty as Perturbation Energy for Single-Forward-Pass Risk-Aware Medical Image Segmentation
Apr 16, 2026Tianhao Fu, Austin Wang, Charles Chen +2Image Segmentation Uncertainty Quantification
University of Toronto, Toronto, ON, Canada · Vector Institute, Toronto, ON, Canada · Project Neura, Toronto, ON, Canada +4
Quantification of Uncertainty with Adversarial Models in Medical Image Segmentation
Jun 17, 2026Hana Jebril, Thomas Pinetz, Günter Klambauer +1Uncertainty Quantification Medical Image Segmentation
Institute of Artificial Intelligence, Center for Medical Data Science, Medical University of Vienna, Austria · Comprehensive Center for AI in Medicine, Medical University of Vienna, Austria · ELLIS Unit Linz, LIT AI Lab and Institute for Machine Learning, Johannes Kepler University Linz, Austria +1
Rethinking Uncertainty in Segmentation: From Estimation to Decision
Apr 14, 2026Saket MagantiImage Segmentation Segmentation Uncertainty Estimation

Apr 16, 2026J/K move · Enter open · S save

SegWithU: Uncertainty as Perturbation Energy for Single-Forward-Pass Risk-Aware Medical Image Segmentation

Tianhao Fu, Austin Wang, Charles Chen, Roby Aldave-Garza+1

University of Toronto, Toronto, ON, Canada · Vector Institute, Toronto, ON, Canada · Project Neura, Toronto, ON, Canada+4

Image Segmentation Uncertainty Quantification Selective Prediction Medical Image Segmentation Segmentation Uncertainty Estimation Uncertainty Calibration

Open paper AskPDF arXiv

Reliable uncertainty estimation is critical for medical image segmentation, where automated contours feed downstream quantification and clinical decision support. Many strong uncertainty methods require repeated inference, while efficient single-forward-pass alternatives often provide weaker failure ranking or rely on restrictive feature-space assumptions. We present

\textbf{SegWithU}

, a post-hoc framework that augments a frozen pretrained segmentation backbone with a lightweight uncertainty head. SegWithU taps intermediate backbone features and models uncertainty as perturbation energy in a compact probe space using rank-1 posterior probes. It produces two voxel-wise uncertainty maps: a calibration-oriented map for probability tempering and a ranking-oriented map for error detection and selective prediction. Across ACDC, BraTS2024, and LiTS, SegWithU is the strongest and most consistent single-forward-pass baseline, achieving AUROC/AURC of

0.9838/2.4885

0.9946/0.2660

, and

0.9925/0.8193

, respectively, while preserving segmentation quality. These results suggest that perturbation-based uncertainty modeling is an effective and practical route to reliability-aware medical segmentation. Source code is available at https://github.com/ProjectNeura/SegWithU.

Abstract

Explore similar work

SegWithU: Uncertainty as Perturbation Energy for Single-Forward-Pass Risk-Aware Medical Image Segmentation

Quantification of Uncertainty with Adversarial Models in Medical Image Segmentation

Rethinking Uncertainty in Segmentation: From Estimation to Decision