DeferredSeg:A Multi-Expert Deferral Framework for Medical Image Segmentation

Qiuyu Tian; Haoliang Sun; Yunshan Wang; Yinghuan Shi; Yilong Yin

cs.CVApr 14, 2026

DeferredSeg:A Multi-Expert Deferral Framework for Medical Image Segmentation

Authors: Qiuyu Tian, Haoliang Sun, Yunshan Wang, Yinghuan Shi, Yilong Yin

Abstract

Segmentation models based on deep neural networks demonstrate strong generalization for medical image segmentation. However, they often exhibit overconfidence or underconfidence, leading to unreliable confidence scores for segmentation masks, especially in ambiguous regions. This undermines the trustworthiness required for clinical deployment. Motivated by the learning-to-defer (L2D) paradigm, we introduce DeferredSeg, a deferral-aware segmentation framework, i.e., a Human--AI collaboration system that determines whether to defer predictions to human experts in specific regions. DeferredSeg extends the base segmentor with an aggregated deferral predictor and additional routing channels that dynamically route each pixel to either the base segmentor or a human expert. To train this routing efficiently, we introduce a pixel-wise surrogate collaboration loss that supervises deferral decisions. In addition, to preserve spatial coherence within deferral regions, we propose a spatial-coherence loss that enforces smooth deferral masks, thereby enhancing reliability. Beyond single-expert deferral, we further extend the framework to a multi-expert setting by introducing multiple discrepancy experts for collaborative decision-making. To prevent overloading or underutilizing individual experts, we further design a load-balancing penalty that evenly distributes workload across expert branches. We evaluate DeferredSeg on three challenging medical datasets using MedSAM and CENet as the base segmentor for fair comparison. Experimental results show that DeferredSeg consistently outperforms the baseline, demonstrating its effectiveness for trustworthy dense medical segmentation. Moreover, the proposed framework is model-agnostic and can be readily applied to other segmentation architectures.

Figures & tables

Figure 1: Overview of DeferredSeg . A deferral predictor produces a pixel-wise deferral map that triages image regions between human-expert branches and the base segmentation model, enabling spatially adaptive Human–AI collaboration for deferral.

Figure 2: Overall pipeline of the proposed DeferredSeg framework, illustrated with a MedSAM-based instantiation. The deferral function $f_{\perp}$ denotes the routing head that outputs per-pixel routing logits. The routing probability maps $\mathcal{M}_{j}$ denote softmax-normalized branch-assignment probabilities. The binary deferral mask $\bar{\mathcal{M}}$ denotes the hard model/expert routing decision derived from the routing probability maps. The aggregated deferral predictor denotes the auxiliary CNN-based predictor that outputs $\tilde{\mathcal{M}}$ , a single-channel aggregated expert-assignment probability map used for spatial-coherence supervision.

Table 3

Dataset	Metric	CENet	Ours (3 experts)
			System	Expert	Model
Prostate (PROMISE12)	DSC	68.96 $\pm$ 0.16	88.85 $\pm$ 0.07	89.80 $\pm$ 0.82	71.98 $\pm$ 6.69
	Jaccard	54.62 $\pm$ 0.13	80.47 $\pm$ 0.03	82.01 $\pm$ 1.22	59.89 $\pm$ 3.87
	Sens.	75.20 $\pm$ 0.10	92.79 $\pm$ 2.38	94.24 $\pm$ 0.87	72.41 $\pm$ 7.32
Liver (LiTS)	DSC	93.61 $\pm$ 0.10	95.16 $\pm$ 0.48	94.19 $\pm$ 1.00	94.75 $\pm$ 1.86
	Jaccard	89.52 $\pm$ 0.07	91.73 $\pm$ 0.71	89.04 $\pm$ 0.97	91.95 $\pm$ 2.70

Table 6: CENet baseline versus CENet with L2D under the three-expert setting (mean

\pm

standard deviation, %). Row-wise best values are bold .

Table 5

Panel of Figure 3 — Figure 3: (a) Qualitative results under the 1-expert and 3-expert settings. (b) Deferral heatmap, where brighter regions indicate higher expert-assignment probability. (c) System Jaccard heatmap over coherence weight $\lambda_{1}$ and balance weight $\lambda_{2}$ , with the best performance obtained at $\lambda_{1}{=}1$ and $\lambda_{2}{=}5$ .

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: DeferredSeg interactive expert-annotation interface. The page shows (top) model prediction and routing results, (middle) per-expert canvases for interactive corrections, and (bottom) the fused segmentation and ground-truth comparison.

Dataset

Metric

nnUNet v2

CENet

MedSAM

PatchCorr

Ours (1 expert)

Ours (3 experts)

(MICCAI 2024)

(MICCAI 2025)

(NC 2024)

(Fine-tuned)

(MIDL 2024)

System

Expert

Model

System

Expert

Model

Prostate (PROMISE12)

DSC

90.49

\pm

0.12

68.96

\pm

0.16

84.16

86.17

\pm

1.53

92.24

\pm

0.03

97.06

\pm

0.05

97.88

\pm

0.01

82.80

\pm

2.75

96.00

\pm

0.14

95.09

\pm

0.19

87.17

\pm

1.99

Jaccard

82.70

\pm

0.20

54.62

\pm

0.13

73.01

76.08

\pm

2.32

85.98

\pm

0.05

94.38

\pm

0.08

95.85

\pm

0.01

79.56

\pm

2.68

92.46

\pm

0.24

90.80

\pm

0.32

86.80

\pm

1.91

Sens.

91.78

\pm

0.53

75.20

\pm

0.10

76.70

84.97

\pm

6.97

94.45

\pm

0.04

95.33

\pm

0.09

97.01

\pm

0.00

83.55

\pm

1.37

96.42

\pm

0.04

95.57

\pm

0.04

99.05

\pm

0.64

Liver (LiTS)

DSC

96.62

\pm

0.05

93.61

\pm

0.10

91.64

91.69

\pm

0.06

92.86

\pm

0.00

97.96

\pm

0.15

95.38

\pm

0.09

95.07

\pm

0.14

96.15

\pm

0.51

97.17

\pm

0.31

93.69

\pm

0.98

Jaccard

93.48

\pm

0.10

89.52

\pm

0.07

86.43

86.49

\pm

0.07

88.83

\pm

0.01

96.30

\pm

0.16

91.34

\pm

0.07

94.13

\pm

0.13

93.70

\pm

0.64

95.16

\pm

0.35

91.01

\pm

1.04

Appendix

Table 10: Comparison with fully automatic reference baselines and the expert-assisted PatchCorr baseline across multiple datasets, organs, and evaluation metrics (%). Row-wise best values are bold .

Dataset	Metric	CENet	MedSAM	MedSAM	PatchCorr	Ours (1 expert)			Ours (3 experts)
		MICCAI 2025	NC 2024	Fine-tuned	MIDL 2024	System	Expert	Model	System	Expert	Model
Chaksu	DSC	97.20 $\pm$ 0.06	94.97	96.76 $\pm$ 0.08	90.37 $\pm$ 0.07	97.36 $\pm$ 0.05	97.37 $\pm$ 0.04	96.50 $\pm$ 0.10	97.17 $\pm$ 0.04	97.27 $\pm$ 0.05	96.98 $\pm$ 0.15
	Jaccard	94.64 $\pm$ 0.08	90.68	93.89 $\pm$ 0.13	82.78 $\pm$ 0.04	94.87 $\pm$ 0.01	94.87 $\pm$ 0.03	93.56 $\pm$ 0.08	94.60 $\pm$ 0.07	94.74 $\pm$ 0.11	94.49 $\pm$ 0.09
	Sens.	96.67 $\pm$ 0.15	91.51	96.69 $\pm$ 0.11	84.11 $\pm$ 0.12	96.30 $\pm$ 0.04	96.50 $\pm$ 0.04	94.55 $\pm$ 0.05	96.74 $\pm$ 0.25	95.72 $\pm$ 0.10	96.84 $\pm$ 0.29
Brain Growth (QUBIQ)	DSC	86.76 $\pm$ 0.05	66.98	83.15 $\pm$ 0.50	82.10 $\pm$ 0.47	86.68 $\pm$ 0.93	92.32 $\pm$ 0.86	84.87 $\pm$ 0.29	89.33 $\pm$ 1.34	87.45 $\pm$ 2.06	89.50 $\pm$ 1.79
	Jaccard	79.10 $\pm$ 0.08	50.57	71.28 $\pm$ 0.67	72.37 $\pm$ 0.75	78.52 $\pm$ 1.37	86.11 $\pm$ 1.51	73.96 $\pm$ 0.38	81.51 $\pm$ 1.89	78.18 $\pm$ 3.34	81.05 $\pm$ 2.45

Appendix

Table 11: Complete real-expert evaluation across Chaksu, QUBIQ, and CURVAS (mean

\pm

standard deviation, %). Fully automatic methods are included as model-only references, while PatchCorr is included as an expert-assisted baseline. Row-wise best full-image results are in bold . Expert- and Model-branch results are shown in gray.

Held-out site	Cases	MedSAM Zero-shot			MedSAM-FT			Ours-1E			Ours-3E
Held-out site	Cases	DSC	Jaccard	Sens.	DSC	Jaccard	Sens.	DSC	Jaccard	Sens.	DSC	Jaccard	Sens.
Site A	15	84.88	74.31	89.45	86.10	76.16	81.23	87.16	77.24	82.46	91.48	84.95	85.30
Site B	15	84.58	74.04	86.73	81.39	69.33	71.81	84.15	72.64	76.41	93.61	88.19	89.99
Site C	15	83.46	72.23	88.00	84.81	74.00	77.87	86.11	75.61	80.11	92.33	86.18	85.79
Site D	5	80.89	68.57	89.24	86.29	76.23	80.47	88.80	79.86	79.63	93.05	87.26	87.83
Center average	–	83.45	72.29	88.35	84.65	73.93	77.84	86.56	76.34	79.65	92.62	86.65	87.23

Appendix

Table 12: Leave-one-center-out cross-center validation on PROMISE12. Values are reported as percentages. MedSAM Zero-shot denotes direct inference using the original MedSAM checkpoint. MedSAM-FT denotes supervised fine-tuning on the training centers. The best result for each metric is highlighted in bold.

Held-out site	Ours-1E		Ours-3E
Held-out site	Deferred	Model	Deferred	E1	E2	E3	Model
Site A	2.53	97.47	6.06	0.62	4.31	1.13	93.94
Site B	2.17	97.83	6.25	0.44	4.90	0.91	93.75
Site C	1.95	98.05	5.34	0.52	3.84	0.99	94.66
Site D	1.71	98.29	4.34	0.37	3.09	0.88	95.66
Center average	2.09	97.91	5.50	0.49	4.04	0.98	94.50

Appendix

Table 13: Deferral ratio in leave-one-center-out cross-center validation on PROMISE12. Values are reported as percentages. Deferred denotes the percentage of pixels routed to expert branches.

Figure 6: Additional qualitative visualization results on different datasets, anatomical structures, and imaging modalities.

Dataset	Cases	One expert				Three experts
		vs. MedSAM-FT		vs. PatchCorr		vs. MedSAM-FT		vs. PatchCorr
		W/T/L	$p$	W/T/L	$p$	W/T/L	$p$	W/T/L	$p$
PROMISE12	30	29/0/1	$<.001$	27/0/3	$<.001$	29/0/1	$<.001$	27/0/3	$<.001$
LiTS-Liver	28	27/0/1	$<.001$	25/0/3	$<.001$	15/0/13	n.s.	21/0/7	$<.01$
LiTS-Tumor	27	26/0/1	$<.001$	25/0/2	$<.001$	25/0/2	$<.001$	24/0/3	$<.001$
AMOS22-Spleen	120	96/0/24	$<.001$	94/0/26	$<.001$	99/0/21	$<.001$	98/0/22	$<.001$

Appendix

Table 14: Detailed statistical significance analysis of the MedSAM-based one- and three-expert configurations using one-sided paired Wilcoxon signed-rank tests on case-level DSC. Each configuration is compared with fine-tuned MedSAM (MedSAM-FT) and PatchCorr on the same test cases. W/T/L denotes the numbers of test cases in which DeferredSeg obtains higher/equal/lower case-level DSC than the corresponding baseline, respectively. n.s. denotes

p\geq 0.05

Dataset	Cases	W/T/L	$p$
PROMISE12	30	29/0/1	$<.001$
LiTS-Liver	28	20/0/8	$<.05$
LiTS-Tumor	27	22/0/5	$<.01$
AMOS22-Spleen	120	98/0/22	$<.001$
Right Kidney	118	101/0/17	$<.001$
Chaksu	273	206/0/67	$<.05$

Appendix

Table 15: Statistical significance analysis of the CENet-based instantiation of DeferredSeg . One-sided paired Wilcoxon signed-rank tests are performed on case-level DSC between CENet with DeferredSeg and the corresponding fine-tuned CENet baseline under the same backbone. W/T/L denotes the numbers of test cases in which CENet with DeferredSeg obtains higher/equal/lower case-level DSC than CENet-FT, respectively. n.s. denotes

p\geq 0.05

J	Loss	DSC			Jaccard			Sens.
		System	Expert	Model	System	Expert	Model	System	Expert	Model
1	DC	96.86 $\pm$ 0.92	95.67 $\pm$ 1.48	94.55 $\pm$ 0.89	94.00 $\pm$ 1.73	91.78 $\pm$ 2.72	93.67 $\pm$ 1.01	96.80 $\pm$ 1.89	95.32 $\pm$ 2.90	96.49 $\pm$ 1.71
1	+ SC	97.17 $\pm$ 0.63	96.09 $\pm$ 0.88	96.08 $\pm$ 0.52	94.57 $\pm$ 1.17	92.55 $\pm$ 1.62	95.14 $\pm$ 0.54	97.29 $\pm$ 1.24	96.01 $\pm$ 1.76	97.41 $\pm$ 1.32
3	DC	95.39 $\pm$ 1.64	93.49 $\pm$ 2.63	94.35 $\pm$ 1.41	91.51 $\pm$ 2.95	88.23 $\pm$ 4.57	93.58 $\pm$ 1.46	97.13 $\pm$ 2.23	95.76 $\pm$ 3.51	98.42 $\pm$ 1.84
	+ SC	95.79 $\pm$ 1.71	93.98 $\pm$ 2.58	94.16 $\pm$ 0.76	92.14 $\pm$ 3.07	88.98 $\pm$ 4.51	93.38 $\pm$ 0.70	97.18 $\pm$ 2.31	95.79 $\pm$ 3.55	97.39 $\pm$ 3.01
	+ LB	96.59 $\pm$ 1.46	95.06 $\pm$ 2.49	95.72 $\pm$ 0.84	93.52 $\pm$ 2.69	90.78 $\pm$ 4.45	94.70 $\pm$ 0.89	97.16 $\pm$ 2.19	95.78 $\pm$ 3.52	97.94 $\pm$ 1.55

Appendix

Table 16: Analysis of each loss component through ablation. These components are mutually complementary and indispensable.

Group	$\lambda_{1}$	$\lambda_{2}$	System			Expert			Model
			Jaccard	DSC	Sens.	Jaccard	DSC	Sens.	Jaccard	DSC	Sens.
A	0.10	0.10	91.26	95.35	94.74	87.45	93.21	91.99	92.71	93.32	98.55
B	0.10	1.00	91.12	95.25	94.73	87.51	93.25	92.00	93.80	94.43	99.90
C	0.10	5.00	91.04	95.11	94.76	87.42	93.09	91.99	93.66	94.47	95.20
D	0.10	10.00	91.36	95.43	94.67	87.89	93.50	92.02	93.12	93.95	99.23
E	1.00	0.10	91.37	95.44	94.66	87.96	93.54	91.99	93.02	93.84	99.39

Appendix

Table 17: Effect of varying coherence (

\lambda_{1}

) and balance (

\lambda_{2}

) weights on System / Expert / Model performance (%).

Figure 7: Complete set of heatmaps for all branches and metrics.

Group	$\lambda_{1}$	$\lambda_{2}$	System
			DSC	Jaccard	Sens.
A	0.10	0.10	94.15	88.95	92.97
B	0.10	5.00	95.82	91.98	94.10
C	1.00	1.00	95.95	92.22	94.77
D	1.00	5.00	96.15	93.70	95.47
E	1.00	10.00	96.01	92.33	94.88

Appendix

Table 18: Effect of varying coherence (

\lambda_{1}

) and balance (

\lambda_{2}

) weights on System performance on LiTS-Liver (%).

Expert	FG	BG	BD
E1	92	98	97
E2	91	99	96
E3	93	97	98
E4	90	97	99
E5	94	99	99

Appendix

Table 19: Settings of scalability-specific synthetic expert pool.

J	Experts	System			Expert			Model
		Jaccard	DSC	Sens.	Jaccard	DSC	Sens.	Jaccard	DSC	Sens.
1	E1	91.04 $\pm$ 0.16	95.24 $\pm$ 0.10	95.54 $\pm$ 0.06	91.00 $\pm$ 0.19	95.21 $\pm$ 0.10	97.01 $\pm$ 0.02	80.81 $\pm$ 2.70	83.94 $\pm$ 2.84	85.12 $\pm$ 1.20
2	E1, E6	89.03 $\pm$ 0.72	94.10 $\pm$ 0.42	93.58 $\pm$ 0.81	91.91 $\pm$ 0.09	95.73 $\pm$ 0.05	97.04 $\pm$ 0.00	84.07 $\pm$ 5.16	89.49 $\pm$ 5.01	91.85 $\pm$ 2.05
3	E1, E2, E3	92.45 $\pm$ 0.21	95.99 $\pm$ 0.13	96.38 $\pm$ 0.05	90.73 $\pm$ 0.26	95.05 $\pm$ 0.15	95.51 $\pm$ 0.01	85.21 $\pm$ 2.68	85.59 $\pm$ 2.66	98.76 $\pm$ 0.97
4	E1, E2, E3, E4	94.06 $\pm$ 0.46	96.83 $\pm$ 0.18	97.43 $\pm$ 0.04	92.02 $\pm$ 0.18	95.50 $\pm$ 0.12	96.94 $\pm$ 0.01	74.45 $\pm$ 3.00	74.62 $\pm$ 2.98	99.50 $\pm$ 0.44
5	E1, E2, E3, E4, E5	94.40 $\pm$ 0.45	96.88 $\pm$ 0.26	97.46 $\pm$ 0.07	93.13 $\pm$ 0.36	96.39 $\pm$ 0.21	96.92 $\pm$ 1.42	78.31 $\pm$ 1.58	85.84 $\pm$ 3.13	98.48 $\pm$ 0.03

Appendix

Table 22: System / Expert / Model performance with increasing number of complementary experts (mean

\pm

std, %).

Method	Params (M)	FLOPs (G)	Time (ms/image)
MedSAM	90.7372	69.2878	16.1161 $\pm$ 0.0223
DeferredSeg (1 expert)	91.3125	69.4532	16.8139 $\pm$ 0.0157
DeferredSeg (3 experts)	91.5926	69.4676	17.0741 $\pm$ 0.0086

Appendix

Table 23: Additional computational-cost comparison with the base MedSAM. FLOPs and inference time are measured with an input resolution of

1024\times 1024

, a batch size of 1, and FP32 precision on the same GPU. Inference time is reported as the mean and standard deviation over five repeated runs.

Explore similar work

Dec 1, 2025

Diffusion Model in Latent Space for Medical Image Segmentation Task

Medical image segmentation is crucial for clinical diagnosis and treatment planning. Traditional methods typically produce a single segmentation mask, failing to capture inherent uncertainty. Recent generative models enable the creation of multiple plausible masks per image, mimicking the collaborative interpretation of several clinicians. However, these approaches remain computationally heavy. We propose MedSegLatDiff, a diffusion based framework that combines a variational autoencoder (VAE) with a latent diffusion model for efficient medical image segmentation. The VAE compresses the input into a low dimensional latent space, reducing noise and accelerating training, while the diffusion process operates directly in this compact representation. We further replace the conventional MSE loss with weighted cross entropy in the VAE mask reconstruction path to better preserve tiny structures such as small nodules. MedSegLatDiff is evaluated on ISIC-2018 (skin lesions), CVC-Clinic (polyps), and LIDC-IDRI (lung nodules). It achieves state of the art or highly competitive Dice and IoU scores while simultaneously generating diverse segmentation hypotheses and confidence maps. This provides enhanced interpretability and reliability compared to deterministic baselines, making the model particularly suitable for clinical deployment.

Ngoc Huynh Trinh, Hai Toan Nguyen, Son Ba Luong +1

Institute for Artificial Intelligence, University of Engineering and Technology, Vietnam National University, Hanoi, Vietnam

Latent Diffusion Models Medical Image Segmentation Segmentation Uncertainty Estimation

Ask PDF

Apr 14, 2026

Rethinking Uncertainty in Segmentation: From Estimation to Decision

In medical image segmentation, uncertainty estimates are often reported but rarely used to guide decisions. We study the missing step: how uncertainty maps are converted into actionable policies such as accepting, flagging, or deferring predictions. We formulate segmentation as a two-stage pipeline, estimation followed by decision, and show that optimizing uncertainty alone fails to capture most of the achievable safety gains. Using retinal vessel segmentation benchmarks (DRIVE, STARE, CHASE_DB1), we evaluate two uncertainty sources (Monte Carlo Dropout and Test-Time Augmentation) combined with three deferral strategies, and introduce a simple confidence-aware deferral rule that prioritizes uncertain and low-confidence predictions. Our results show that the best method and policy combination removes up to 80 percent of segmentation errors at only 25 percent pixel deferral, while achieving strong cross-dataset robustness. We further show that calibration improvements do not translate to better decision quality, highlighting a disconnect between standard uncertainty metrics and real-world utility. These findings suggest that uncertainty should be evaluated based on the decisions it enables, rather than in isolation.

Saket Maganti

Image Segmentation Segmentation Uncertainty Estimation

Ask PDF

Jun 21, 2026

Human and AI collaboration for pulmonary nodule segmentation

Medical expert annotators are scarce, and blind reliance on artificial intelligence (AI) can be misleading, motivating approaches in which humans, particularly junior medical trainees or even non-medical personnel, collaborate with AI to achieve robust medical segmentation. Although the Segment Anything Model (SAM) shows promise for general-purpose image segmentation, its performance in human-AI collaboration for specialized medical tasks has not been thoroughly evaluated. Here we present Hi-Seg, a human-in-the-loop segmentation framework for pulmonary nodules built on SAM. Humans iteratively refine prompts through trial-and-error learning and semantic reasoning, progressively guiding SAM toward higher-quality masks. Using chest CT scans from 1,179 patients across 12 centers, we conducted the first large-scale external validation of collaborative human-SAM segmentation. Across all annotator groups, Hi-Seg achieved a mean Dice score of almost 85%, outperforming five state-of-the-art deep learning models by 10-22% and 13 SAM variants by 1-29%. Hi-Seg improved segmentation accuracy while reducing annotation time for medical annotators, and briefly trained non-medical annotators achieved performance comparable to that of the junior medical student. These findings suggest that human-in-the-loop segmentation can reduce clinician workload, enable scalable crowdsourced annotation, and transform clinical workflows by facilitating the safe and efficient integration of foundation models into routine clinical practice.

Hongqiao Dong, Wenhao Chi, Ruobing Liang +9

State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing, China · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing, China · Yau Mathematical Sciences Center, Tsinghua University, Beijing, China +5

Interactive Medical Image Segmentation Medical Imaging Foundation Models Promptable Image Segmentation Human-in-the-Loop Annotation Medical Image Segmentation

Ask PDF