Domain adaptive segmentation (DAS) is a promising paradigm for delineating intracellular structures from various large-scale electron microscopy (EM) without incurring extensive annotated data in each domain. However, the prevalent unsupervised domain adaptation (UDA) strategies often demonstrate limited and biased performance, which hinders their practical applications. In this study, we explore sparse points and local human preferences as weak labels in the target domain, thereby presenting a more realistic yet annotation-efficient setting. Specifically, we develop Prefer-DAS, which pioneers sparse promptable learning and local preference alignment. The Prefer-DAS is a promptable multitask model that integrates self-training and prompt-guided contrastive learning. Unlike SAM-like methods, the Prefer-DAS allows for the use of full, partial, and even no point prompts during both training and inference stages and thus enables interactive segmentation. Instead of using image-level human preference alignment for segmentation, we introduce Local direct Preference Optimization (LPO), plug-and-play solutions for alignment with spatially varying human feedback. To address potential missing feedback, we also introduce Unsupervised Preference Optimization (UPO), which leverages self-learned preferences. As a result, the Prefer-DAS model can effectively perform both weakly-supervised and unsupervised DAS, depending on the availability of points and human preferences. Comprehensive experiments on four challenging DAS tasks demonstrate that our model outperforms SAM-like methods as well as unsupervised and weakly-supervised DAS methods in both automatic and interactive segmentation modes, highlighting strong generalizability and flexibility. Additionally, the performance of our model is very close to or even exceeds that of supervised models.
Figures & tables
Fig. 1: A visualization of local human preferences. While selecting preferred segmentation at the image level usually poses challenges for human raters and results in reward misspecification in preference optimization, we gather local preferences at the patch level. Here, an image of small size and 2 × 2 patches are used solely for illustrative purposes.
Fig. 2: Illustration of our Prefer-DAS model for weakly-supervised domain adaptive segmentation. Our framework comprises two stages. The first stage performs promptable domain-adaptive segmentation for UDA or WDA segmentation, depending on the availability of sparse points; the second stage performs locally supervised or unsupervised preference learning, depending on the availability of human preferences.
Types
Methods
Training
H → R
R → H
H → Lucchi++
H → Stem
Point Prompts
Dice
AJI
PQ
Dice
AJI
PQ
Dice
AJI
PQ
Dice
AJI
PQ
Automatic Segmentation Mode
SAM-like
SAM [ 2 ]
0%
32.0
14.3
30.0
20.8
11.4
18.7
21.5
10.3
11.3
19.0
0.9
0.3
SAM-Med2D [ 9 ]
15.9
-
-
22.5
-
-
23.1
12.7
2.6
27.9
13.0
3.3
Med-SAM Adapter [ 10 ] †
75.5
56.6
27.5
72.4
55.0
33.4
64.6
44.8
19.5
27.8
12.9
7.1
UDA
UALR [ 18 ]
0%
86.3
71.6
53.7
83.8
69.7
60.0
81.3
68.1
63.2
65.5
38.7
28.7
TABLE I: Quantitative comparison. Leading UDA, WDA, and SAM-like approaches in both automatic and interactive modes are compared. Our Prefer-DAS is flexible to conduct both UDA and WDA, depending on the availability of the training point prompts and human preferences, and can utilize sparse point prompts during inference. For interactive segmentation, indicated by a ”+”, full center points for all mitochondria are used as testing prompts to fulfill the requirements of SAM-like methods.
Fig. 3: Visual comparison of segmentation results in both automatic and interactive modes. Despite the Med-SAM Adapter+ and WeSAM+ being fine-tuned with the source EM data and utilizing full point prompts during inference, they still show severe false positive segmentation.
Pseudo-labeling
Training
PCL
LPO
H → R
R → H
Det.
Seg.
Prompts
Dice
PQ
Dice
PQ
I
88.6
68.7
78.1
55.5
II
✓
89.2
70.4
87.4
68.4
III
✓
89.5
70.0
87.7
68.3
IV
✓
✓
90.4
71.8
88.5
68.8
V
✓
✓
✓
92.7
74.1
88.9
69.0
TABLE II: Ablation study of the proposed model.
Fig. 4: Box plot of our Prefer-DAS using different DPOs. Global and local refer to image-level and patch-level preferences, respectively. ”w/o” indicates the Prefer-DAS using no preferences.
Fig. 5: Qualitative comparison of different strategies for preference learning. GPO refers to learning using image-level human preferences.
DPO
Negative(s)
H → R
R → H
H → Lucchi++
Single
Multiple
Dice
PQ
Dice
PQ
Dice
PQ
GPO
✓
94.0
78.2
91.1
73.3
93.2
80.5
✓
94.3
78.7
91.5
74.2
93.4
81.7
LPO
✓
94.2
78.0
91.3
74.7
93.9
82.3
✓
94.6
79.6
92.2
75.8
94.1
82.3
TABLE III: Effectiveness of using multiple negatives.
Fig. 6: Effectiveness of preference learning in correcting defective and biased segmentation. Red: Ground truth; Cyan: CAFA; Blue: Prompt-DAS; Yellow: Prefer-DAS (UPO); Green: Prefer-DAS (LPO). The H → Lucchi++ task is used for evaluation. Both CAFA and Prompt-DAS reveal two types of segmentation defects: 1) incomplete segmentation and 2) misalignment with target annotations, which primarily arises from the domain inconsistency in annotating inner and outer membrane edges.
Model
Testing
H → R
R → H
H → Lucchi++
Prompts
Dice
PQ
Dice
PQ
Dice
PQ
Prompt-DAS
0%
93.3
74.5
89.2
69.1
88.2
73.7
15%
93.5
74.2
90.0
69.8
88.6
74.7
50%
93.5
74.2
90.4
70.9
88.6
74.8
100%
93.5
74.2
90.8
72.3
88.7
74.8
0%
93.7
77.5
90.1
71.2
92.6
80.9
TABLE IV: Influence of the quantity of inference point prompts on interactive segmentation. Different from SAM, our models can utilize partial point prompts.
College of Medicine and Biological Information Engineering, Northeastern University · Engineering Research Center of Medical Imaging and Intelligent Analysis, Ministry of Education · Key Laboratory of Medical Image Computing, Ministry of Education, Northeastern University