DeferredSeg:A Multi-Expert Deferral Framework for Medical Image Segmentation

Qiuyu Tian; Haoliang Sun; Yunshan Wang; Yinghuan Shi; Yilong Yin

cs.CVApr 14, 2026

DeferredSeg:A Multi-Expert Deferral Framework for Medical Image Segmentation

Authors: Qiuyu Tian, Haoliang Sun, Yunshan Wang, Yinghuan Shi, Yilong Yin

Abstract

Segmentation models based on deep neural networks demonstrate strong generalization for medical image segmentation. However, they often exhibit overconfidence or underconfidence, leading to unreliable confidence scores for segmentation masks, especially in ambiguous regions. This undermines the trustworthiness required for clinical deployment. Motivated by the learning-to-defer (L2D) paradigm, we introduce DeferredSeg, a deferral-aware segmentation framework, i.e., a Human--AI collaboration system that determines whether to defer predictions to human experts in specific regions. DeferredSeg extends the base segmentor with an aggregated deferral predictor and additional routing channels that dynamically route each pixel to either the base segmentor or a human expert. To train this routing efficiently, we introduce a pixel-wise surrogate collaboration loss that supervises deferral decisions. In addition, to preserve spatial coherence within deferral regions, we propose a spatial-coherence loss that enforces smooth deferral masks, thereby enhancing reliability. Beyond single-expert deferral, we further extend the framework to a multi-expert setting by introducing multiple discrepancy experts for collaborative decision-making. To prevent overloading or underutilizing individual experts, we further design a load-balancing penalty that evenly distributes workload across expert branches. We evaluate DeferredSeg on three challenging medical datasets using MedSAM and CENet as the base segmentor for fair comparison. Experimental results show that DeferredSeg consistently outperforms the baseline, demonstrating its effectiveness for trustworthy dense medical segmentation. Moreover, the proposed framework is model-agnostic and can be readily applied to other segmentation architectures.

Figures & tables

Figure 1: Overview of DeferredSeg . A deferral predictor produces a pixel-wise deferral map that triages image regions between human-expert branches and the base segmentation model, enabling spatially adaptive Human–AI collaboration for deferral.

Figure 2: Overall pipeline of the proposed DeferredSeg framework, illustrated with a MedSAM-based instantiation. The deferral function $f_{\perp}$ denotes the routing head that outputs per-pixel routing logits. The routing probability maps $\mathcal{M}_{j}$ denote softmax-normalized branch-assignment probabilities. The binary deferral mask $\bar{\mathcal{M}}$ denotes the hard model/expert routing decision derived from the routing probability maps. The aggregated deferral predictor denotes the auxiliary CNN-based predictor that outputs $\tilde{\mathcal{M}}$ , a single-channel aggregated expert-assignment probability map used for spatial-coherence supervision.

Table 3

Dataset	Metric	CENet	Ours (3 experts)
			System	Expert	Model
Prostate (PROMISE12)	DSC	68.96 $\pm$ 0.16	88.85 $\pm$ 0.07	89.80 $\pm$ 0.82	71.98 $\pm$ 6.69
	Jaccard	54.62 $\pm$ 0.13	80.47 $\pm$ 0.03	82.01 $\pm$ 1.22	59.89 $\pm$ 3.87
	Sens.	75.20 $\pm$ 0.10	92.79 $\pm$ 2.38	94.24 $\pm$ 0.87	72.41 $\pm$ 7.32
Liver (LiTS)	DSC	93.61 $\pm$ 0.10	95.16 $\pm$ 0.48	94.19 $\pm$ 1.00	94.75 $\pm$ 1.86
	Jaccard	89.52 $\pm$ 0.07	91.73 $\pm$ 0.71	89.04 $\pm$ 0.97	91.95 $\pm$ 2.70

Table 6: CENet baseline versus CENet with L2D under the three-expert setting (mean

\pm

standard deviation, %). Row-wise best values are bold .

Table 5

Panel of Figure 3 — Figure 3: (a) Qualitative results under the 1-expert and 3-expert settings. (b) Deferral heatmap, where brighter regions indicate higher expert-assignment probability. (c) System Jaccard heatmap over coherence weight $\lambda_{1}$ and balance weight $\lambda_{2}$ , with the best performance obtained at $\lambda_{1}{=}1$ and $\lambda_{2}{=}5$ .

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: DeferredSeg interactive expert-annotation interface. The page shows (top) model prediction and routing results, (middle) per-expert canvases for interactive corrections, and (bottom) the fused segmentation and ground-truth comparison.

Dataset

Metric

nnUNet v2

CENet

MedSAM

PatchCorr

Ours (1 expert)

Ours (3 experts)

(MICCAI 2024)

(MICCAI 2025)

(NC 2024)

(Fine-tuned)

(MIDL 2024)

System

Expert

Model

System

Expert

Model

Prostate (PROMISE12)

DSC

90.49

\pm

0.12

68.96

\pm

0.16

84.16

86.17

\pm

1.53

92.24

\pm

0.03

97.06

\pm

0.05

97.88

\pm

0.01

82.80

\pm

2.75

96.00

\pm

0.14

95.09

\pm

0.19

87.17

\pm

1.99

Jaccard

82.70

\pm

0.20

54.62

\pm

0.13

73.01

76.08

\pm

2.32

85.98

\pm

0.05

94.38

\pm

0.08

95.85

\pm

0.01

79.56

\pm

2.68

92.46

\pm

0.24

90.80

\pm

0.32

86.80

\pm

1.91

Sens.

91.78

\pm

0.53

75.20

\pm

0.10

76.70

84.97

\pm

6.97

94.45

\pm

0.04

95.33

\pm

0.09

97.01

\pm

0.00

83.55

\pm

1.37

96.42

\pm

0.04

95.57

\pm

0.04

99.05

\pm

0.64

Liver (LiTS)

DSC

96.62

\pm

0.05

93.61

\pm

0.10

91.64

91.69

\pm

0.06

92.86

\pm

0.00

97.96

\pm

0.15

95.38

\pm

0.09

95.07

\pm

0.14

96.15

\pm

0.51

97.17

\pm

0.31

93.69

\pm

0.98

Jaccard

93.48

\pm

0.10

89.52

\pm

0.07

86.43

86.49

\pm

0.07

88.83

\pm

0.01

96.30

\pm

0.16

91.34

\pm

0.07

94.13

\pm

0.13

93.70

\pm

0.64

95.16

\pm

0.35

91.01

\pm

1.04

Appendix

Table 10: Comparison with fully automatic reference baselines and the expert-assisted PatchCorr baseline across multiple datasets, organs, and evaluation metrics (%). Row-wise best values are bold .

Dataset	Metric	CENet	MedSAM	MedSAM	PatchCorr	Ours (1 expert)			Ours (3 experts)
		MICCAI 2025	NC 2024	Fine-tuned	MIDL 2024	System	Expert	Model	System	Expert	Model
Chaksu	DSC	97.20 $\pm$ 0.06	94.97	96.76 $\pm$ 0.08	90.37 $\pm$ 0.07	97.36 $\pm$ 0.05	97.37 $\pm$ 0.04	96.50 $\pm$ 0.10	97.17 $\pm$ 0.04	97.27 $\pm$ 0.05	96.98 $\pm$ 0.15
	Jaccard	94.64 $\pm$ 0.08	90.68	93.89 $\pm$ 0.13	82.78 $\pm$ 0.04	94.87 $\pm$ 0.01	94.87 $\pm$ 0.03	93.56 $\pm$ 0.08	94.60 $\pm$ 0.07	94.74 $\pm$ 0.11	94.49 $\pm$ 0.09
	Sens.	96.67 $\pm$ 0.15	91.51	96.69 $\pm$ 0.11	84.11 $\pm$ 0.12	96.30 $\pm$ 0.04	96.50 $\pm$ 0.04	94.55 $\pm$ 0.05	96.74 $\pm$ 0.25	95.72 $\pm$ 0.10	96.84 $\pm$ 0.29
Brain Growth (QUBIQ)	DSC	86.76 $\pm$ 0.05	66.98	83.15 $\pm$ 0.50	82.10 $\pm$ 0.47	86.68 $\pm$ 0.93	92.32 $\pm$ 0.86	84.87 $\pm$ 0.29	89.33 $\pm$ 1.34	87.45 $\pm$ 2.06	89.50 $\pm$ 1.79
	Jaccard	79.10 $\pm$ 0.08	50.57	71.28 $\pm$ 0.67	72.37 $\pm$ 0.75	78.52 $\pm$ 1.37	86.11 $\pm$ 1.51	73.96 $\pm$ 0.38	81.51 $\pm$ 1.89	78.18 $\pm$ 3.34	81.05 $\pm$ 2.45

Appendix

Table 11: Complete real-expert evaluation across Chaksu, QUBIQ, and CURVAS (mean

\pm

standard deviation, %). Fully automatic methods are included as model-only references, while PatchCorr is included as an expert-assisted baseline. Row-wise best full-image results are in bold . Expert- and Model-branch results are shown in gray.

Held-out site	Cases	MedSAM Zero-shot			MedSAM-FT			Ours-1E			Ours-3E
Held-out site	Cases	DSC	Jaccard	Sens.	DSC	Jaccard	Sens.	DSC	Jaccard	Sens.	DSC	Jaccard	Sens.
Site A	15	84.88	74.31	89.45	86.10	76.16	81.23	87.16	77.24	82.46	91.48	84.95	85.30
Site B	15	84.58	74.04	86.73	81.39	69.33	71.81	84.15	72.64	76.41	93.61	88.19	89.99
Site C	15	83.46	72.23	88.00	84.81	74.00	77.87	86.11	75.61	80.11	92.33	86.18	85.79
Site D	5	80.89	68.57	89.24	86.29	76.23	80.47	88.80	79.86	79.63	93.05	87.26	87.83
Center average	–	83.45	72.29	88.35	84.65	73.93	77.84	86.56	76.34	79.65	92.62	86.65	87.23

Appendix

Table 12: Leave-one-center-out cross-center validation on PROMISE12. Values are reported as percentages. MedSAM Zero-shot denotes direct inference using the original MedSAM checkpoint. MedSAM-FT denotes supervised fine-tuning on the training centers. The best result for each metric is highlighted in bold.

Held-out site	Ours-1E		Ours-3E
Held-out site	Deferred	Model	Deferred	E1	E2	E3	Model
Site A	2.53	97.47	6.06	0.62	4.31	1.13	93.94
Site B	2.17	97.83	6.25	0.44	4.90	0.91	93.75
Site C	1.95	98.05	5.34	0.52	3.84	0.99	94.66
Site D	1.71	98.29	4.34	0.37	3.09	0.88	95.66
Center average	2.09	97.91	5.50	0.49	4.04	0.98	94.50

Appendix

Table 13: Deferral ratio in leave-one-center-out cross-center validation on PROMISE12. Values are reported as percentages. Deferred denotes the percentage of pixels routed to expert branches.

Figure 6: Additional qualitative visualization results on different datasets, anatomical structures, and imaging modalities.

Dataset	Cases	One expert				Three experts
		vs. MedSAM-FT		vs. PatchCorr		vs. MedSAM-FT		vs. PatchCorr
		W/T/L	$p$	W/T/L	$p$	W/T/L	$p$	W/T/L	$p$
PROMISE12	30	29/0/1	$<.001$	27/0/3	$<.001$	29/0/1	$<.001$	27/0/3	$<.001$
LiTS-Liver	28	27/0/1	$<.001$	25/0/3	$<.001$	15/0/13	n.s.	21/0/7	$<.01$
LiTS-Tumor	27	26/0/1	$<.001$	25/0/2	$<.001$	25/0/2	$<.001$	24/0/3	$<.001$
AMOS22-Spleen	120	96/0/24	$<.001$	94/0/26	$<.001$	99/0/21	$<.001$	98/0/22	$<.001$

Appendix

Table 14: Detailed statistical significance analysis of the MedSAM-based one- and three-expert configurations using one-sided paired Wilcoxon signed-rank tests on case-level DSC. Each configuration is compared with fine-tuned MedSAM (MedSAM-FT) and PatchCorr on the same test cases. W/T/L denotes the numbers of test cases in which DeferredSeg obtains higher/equal/lower case-level DSC than the corresponding baseline, respectively. n.s. denotes

p\geq 0.05

Dataset	Cases	W/T/L	$p$
PROMISE12	30	29/0/1	$<.001$
LiTS-Liver	28	20/0/8	$<.05$
LiTS-Tumor	27	22/0/5	$<.01$
AMOS22-Spleen	120	98/0/22	$<.001$
Right Kidney	118	101/0/17	$<.001$
Chaksu	273	206/0/67	$<.05$

Appendix

Table 15: Statistical significance analysis of the CENet-based instantiation of DeferredSeg . One-sided paired Wilcoxon signed-rank tests are performed on case-level DSC between CENet with DeferredSeg and the corresponding fine-tuned CENet baseline under the same backbone. W/T/L denotes the numbers of test cases in which CENet with DeferredSeg obtains higher/equal/lower case-level DSC than CENet-FT, respectively. n.s. denotes

p\geq 0.05

J	Loss	DSC			Jaccard			Sens.
		System	Expert	Model	System	Expert	Model	System	Expert	Model
1	DC	96.86 $\pm$ 0.92	95.67 $\pm$ 1.48	94.55 $\pm$ 0.89	94.00 $\pm$ 1.73	91.78 $\pm$ 2.72	93.67 $\pm$ 1.01	96.80 $\pm$ 1.89	95.32 $\pm$ 2.90	96.49 $\pm$ 1.71
1	+ SC	97.17 $\pm$ 0.63	96.09 $\pm$ 0.88	96.08 $\pm$ 0.52	94.57 $\pm$ 1.17	92.55 $\pm$ 1.62	95.14 $\pm$ 0.54	97.29 $\pm$ 1.24	96.01 $\pm$ 1.76	97.41 $\pm$ 1.32
3	DC	95.39 $\pm$ 1.64	93.49 $\pm$ 2.63	94.35 $\pm$ 1.41	91.51 $\pm$ 2.95	88.23 $\pm$ 4.57	93.58 $\pm$ 1.46	97.13 $\pm$ 2.23	95.76 $\pm$ 3.51	98.42 $\pm$ 1.84
	+ SC	95.79 $\pm$ 1.71	93.98 $\pm$ 2.58	94.16 $\pm$ 0.76	92.14 $\pm$ 3.07	88.98 $\pm$ 4.51	93.38 $\pm$ 0.70	97.18 $\pm$ 2.31	95.79 $\pm$ 3.55	97.39 $\pm$ 3.01
	+ LB	96.59 $\pm$ 1.46	95.06 $\pm$ 2.49	95.72 $\pm$ 0.84	93.52 $\pm$ 2.69	90.78 $\pm$ 4.45	94.70 $\pm$ 0.89	97.16 $\pm$ 2.19	95.78 $\pm$ 3.52	97.94 $\pm$ 1.55

Appendix

Table 16: Analysis of each loss component through ablation. These components are mutually complementary and indispensable.

Group	$\lambda_{1}$	$\lambda_{2}$	System			Expert			Model
			Jaccard	DSC	Sens.	Jaccard	DSC	Sens.	Jaccard	DSC	Sens.
A	0.10	0.10	91.26	95.35	94.74	87.45	93.21	91.99	92.71	93.32	98.55
B	0.10	1.00	91.12	95.25	94.73	87.51	93.25	92.00	93.80	94.43	99.90
C	0.10	5.00	91.04	95.11	94.76	87.42	93.09	91.99	93.66	94.47	95.20
D	0.10	10.00	91.36	95.43	94.67	87.89	93.50	92.02	93.12	93.95	99.23
E	1.00	0.10	91.37	95.44	94.66	87.96	93.54	91.99	93.02	93.84	99.39

Appendix

Table 17: Effect of varying coherence (

\lambda_{1}

) and balance (

\lambda_{2}

) weights on System / Expert / Model performance (%).

Figure 7: Complete set of heatmaps for all branches and metrics.

Group	$\lambda_{1}$	$\lambda_{2}$	System
			DSC	Jaccard	Sens.
A	0.10	0.10	94.15	88.95	92.97
B	0.10	5.00	95.82	91.98	94.10
C	1.00	1.00	95.95	92.22	94.77
D	1.00	5.00	96.15	93.70	95.47
E	1.00	10.00	96.01	92.33	94.88

Appendix

Table 18: Effect of varying coherence (

\lambda_{1}

) and balance (

\lambda_{2}

) weights on System performance on LiTS-Liver (%).

Expert	FG	BG	BD
E1	92	98	97
E2	91	99	96
E3	93	97	98
E4	90	97	99
E5	94	99	99

Appendix

Table 19: Settings of scalability-specific synthetic expert pool.

J	Experts	System			Expert			Model
		Jaccard	DSC	Sens.	Jaccard	DSC	Sens.	Jaccard	DSC	Sens.
1	E1	91.04 $\pm$ 0.16	95.24 $\pm$ 0.10	95.54 $\pm$ 0.06	91.00 $\pm$ 0.19	95.21 $\pm$ 0.10	97.01 $\pm$ 0.02	80.81 $\pm$ 2.70	83.94 $\pm$ 2.84	85.12 $\pm$ 1.20
2	E1, E6	89.03 $\pm$ 0.72	94.10 $\pm$ 0.42	93.58 $\pm$ 0.81	91.91 $\pm$ 0.09	95.73 $\pm$ 0.05	97.04 $\pm$ 0.00	84.07 $\pm$ 5.16	89.49 $\pm$ 5.01	91.85 $\pm$ 2.05
3	E1, E2, E3	92.45 $\pm$ 0.21	95.99 $\pm$ 0.13	96.38 $\pm$ 0.05	90.73 $\pm$ 0.26	95.05 $\pm$ 0.15	95.51 $\pm$ 0.01	85.21 $\pm$ 2.68	85.59 $\pm$ 2.66	98.76 $\pm$ 0.97
4	E1, E2, E3, E4	94.06 $\pm$ 0.46	96.83 $\pm$ 0.18	97.43 $\pm$ 0.04	92.02 $\pm$ 0.18	95.50 $\pm$ 0.12	96.94 $\pm$ 0.01	74.45 $\pm$ 3.00	74.62 $\pm$ 2.98	99.50 $\pm$ 0.44
5	E1, E2, E3, E4, E5	94.40 $\pm$ 0.45	96.88 $\pm$ 0.26	97.46 $\pm$ 0.07	93.13 $\pm$ 0.36	96.39 $\pm$ 0.21	96.92 $\pm$ 1.42	78.31 $\pm$ 1.58	85.84 $\pm$ 3.13	98.48 $\pm$ 0.03

Appendix

Table 22: System / Expert / Model performance with increasing number of complementary experts (mean

\pm

std, %).

Method	Params (M)	FLOPs (G)	Time (ms/image)
MedSAM	90.7372	69.2878	16.1161 $\pm$ 0.0223
DeferredSeg (1 expert)	91.3125	69.4532	16.8139 $\pm$ 0.0157
DeferredSeg (3 experts)	91.5926	69.4676	17.0741 $\pm$ 0.0086

Appendix

Table 23: Additional computational-cost comparison with the base MedSAM. FLOPs and inference time are measured with an input resolution of

1024\times 1024

, a batch size of 1, and FP32 precision on the same GPU. Inference time is reported as the mean and standard deviation over five repeated runs.

Explore similar work

Diffusion Model in Latent Space for Medical Image Segmentation Task
Dec 1, 2025Ngoc Huynh Trinh, Hai Toan Nguyen, Son Ba Luong +1Latent Diffusion Models Medical Image Segmentation
Institute for Artificial Intelligence, University of Engineering and Technology, Vietnam National University, Hanoi, Vietnam
Rethinking Uncertainty in Segmentation: From Estimation to Decision
Apr 14, 2026Saket MagantiImage Segmentation Segmentation Uncertainty Estimation
Human and AI collaboration for pulmonary nodule segmentation
Jun 21, 2026Hongqiao Dong, Wenhao Chi, Ruobing Liang +9Interactive Medical Image Segmentation Medical Imaging Foundation Models
State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing, China · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing, China · Yau Mathematical Sciences Center, Tsinghua University, Beijing, China +5

Dec 1, 2025J/K move · Enter open · S save

Diffusion Model in Latent Space for Medical Image Segmentation Task

Ngoc Huynh Trinh, Hai Toan Nguyen, Son Ba Luong, Quoc Long Tran

Institute for Artificial Intelligence, University of Engineering and Technology, Vietnam National University, Hanoi, Vietnam

Latent Diffusion Models Medical Image Segmentation Segmentation Uncertainty Estimation

Open paper AskPDF arXiv

Medical image segmentation is crucial for clinical diagnosis and treatment planning. Traditional methods typically produce a single segmentation mask, failing to capture inherent uncertainty. Recent generative models enable the creation of multiple plausible masks per image, mimicking the collaborative interpretation of several clinicians. However, these approaches remain computationally heavy. We propose MedSegLatDiff, a diffusion based framework that combines a variational autoencoder (VAE) with a latent diffusion model for efficient medical image segmentation. The VAE compresses the input into a low dimensional latent space, reducing noise and accelerating training, while the diffusion process operates directly in this compact representation. We further replace the conventional MSE loss with weighted cross entropy in the VAE mask reconstruction path to better preserve tiny structures such as small nodules. MedSegLatDiff is evaluated on ISIC-2018 (skin lesions), CVC-Clinic (polyps), and LIDC-IDRI (lung nodules). It achieves state of the art or highly competitive Dice and IoU scores while simultaneously generating diverse segmentation hypotheses and confidence maps. This provides enhanced interpretability and reliability compared to deterministic baselines, making the model particularly suitable for clinical deployment.

Abstract

Explore similar work

Diffusion Model in Latent Space for Medical Image Segmentation Task

Rethinking Uncertainty in Segmentation: From Estimation to Decision

Human and AI collaboration for pulmonary nodule segmentation