Medical image segmentation is vital for clinical diagnosis and quantitative analysis, yet remains challenging due to the heterogeneity of imaging modalities and the high cost of pixel-level annotations. Although general interactive segmentation models like SAM have achieved remarkable progress, their transfer to medical imaging still faces two key bottlenecks: (i) the lack of adaptive mechanisms for modality- and anatomy-specific tasks, which limits generalization in out-of-distribution medical scenarios; and (ii) current medical adaptation methods fine-tune on large, heterogeneous datasets without selection, leading to noisy supervision, higher cost, and negative transfer. To address these issues, we propose SegMoTE, an efficient and adaptive framework for medical image segmentation. SegMoTE preserves SAM's original prompt interface, efficient inference, and zero-shot generalization while introducing only a small number of learnable parameters to dynamically adapt across modalities and tasks. In addition, we design a progressive prompt tokenization mechanism that enables fully automatic segmentation, significantly reducing annotation dependence. Trained on MedSeg-HQ, a curated dataset less than 1% of existing large-scale datasets, SegMoTE achieves SOTA performance across diverse imaging modalities and anatomical tasks. It represents the first efficient, robust, and scalable adaptation of general segmentation models to the medical domain under extremely low annotation cost, advancing the practical deployment of foundation vision models in clinical applications.
Figures & tables
Figure 1: SegMoTE vs. Previous Works. The heterogeneous data X is first processed by the encoder ε to extract the feature representation f . (a) Previous methods typically perform full fine-tuning of the mask decoder or parameter-efficient fine-tuning, leading to distribution shift from the pretrained model. (b) SegMoTE introduces a token-level mixture of experts mechanism that dynamically selects modality-adaptive expert tokens while keeping the mask decoder frozen. The process is guided by the load balancing loss Lbalance , is constrained using the squared coefficient of variation (CV2) . This design preserves SAM’s original capability, enhances modality-specific adaptability, and maintains a lightweight architecture.
Figure 2: Overview of the proposed SegMoTE framework. SegMoTE extends SAM by introducing a token-level expert routing mechanism to enable adaptive multimodal medical image segmentation. The frozen SAM encoder extracts modality-agnostic embedding representations, while the progressive prompt tokenization transforms latent features into semantically aligned feature tokens. These tokens interact with the decoder layers and MoTE for dynamic expert selection and adaptive token updates.
Figure 3: Architecture of the Progressive Prompt Tokenization. By randomly selecting mask and text prompts as foreground priors, the learnable query Q captures the relationship between the foreground and background by performing attention on the normalized image features.
Figure 4: Feature distribution comparison across datasets. Feature embeddings extracted from the frozen SAM encoder are visualized after dimensionality reduction. The red boxes highlight regions where features exhibit overlap and unsmooth transitions. Compared with other datasets, MedSeg-HQ presents smoother and more continuous feature distributions, indicating higher consistency.
Figure 5: In-domain segmentation results across datasets. (a) and (b) show the Dice coefficient comparisons under single click and bounding box interactions, respectively. (c) illustrates the training dataset sizes used by different methods, where SegMoTE achieves performance improvement by optimizing the annotation quality of the data.
Dataset
Category
SAM
SAM 2
MedSAM
Huang et al.
SAM-Med2D
IMIS
Ours
ISLES
Ischemic Stroke Lesion
55.00
59.17
58.02
56.54
67.93
71.24
77.30
SegThor
Esophagus
68.94
77.69
38.59
68.47
73.92
80.82
81.84
Heart
80.28
81.73
68.09
78.21
80.34
72.31
74.51
Aorta
74.52
76.16
53.73
86.25
83.76
88.67
87.40
Trachea
82.48
83.64
55.97
81.78
78.18
80.28
80.53
Average
76.55
79.81
54.09
78.67
79.06
80.52
83.39
Table 1: Performance comparison on three out-of-domain datasets. Experimental results of SegMoTE in comparison to other methods under bounding-box interactions. Best results are highlighted in red , and second best results are highlighted in blue .
Dataset
SAM
SAM2
SAM-Med2D
IMIS
Ours
AMOS (CT)
76.05
78.84
80.71
83.56
85.16
AMOS (MRI)
72.63
73.46
76.55
79.72
80.27
BTCV
77.82
78.79
80.52
82.24
84.51
CHAOS (T1)
82.67
83.30
86.14
86.92
89.00
ISIC2018
86.15
87.46
88.32
88.93
93.02
SZ-CXR
86.72
87.58
88.72
92.03
95.04
Table 2: Experimental results on multiple datasets under bounding-box interactions. Our method involves unfreezing the original decoder parameters and training them along with the MoTE and PPT mechanisms on the MedSeg-HQ dataset.
Method
Image Resolution
Learnable Params
Training #GPU
Batch Size
SAM (Large)
1024×1024
1191 M
A100 × 128
128
MedSAM (Base)
1024×1024
93 M
A100 × 20
128
IMIS (Base)
1024×1024
29 M
RTX4090 × 72
2
SegMoTE (Base)
512×512
17 M
RTX4090 × 8
10
(-MoTE)
512×512
10 M
–
–
(-PPT)
512×512
7 M
–
–
Table 3: Comparison of SegMoTE with existing interactive segmentation benchmarks in terms of model size.
Figure 6: (a) Expert token selection statistics across datasets, the numbers above the bars in the histogram represent the sample size of the current dataset. and (b) (c) experiments with different numbers of experts and activations.
Figure 7: Visualization of expert token routing across datasets in the MoTE framework. Red boxes highlight the target segmentation areas, and the green checkmark indicates the selected token.
Dataset
ISIC2017
ISIC2018
SZ-CXR
ISLES
Q=2
77.62
87.68
77.63
65.28
Q=4
77.23
87.79
77.24
64.97
Q=8
77.36
87.71
77.51
65.00
w/o PPT
76.66
84.87
75.20
59.00
Table 4: Ablation study on query token number. Analysis of different query token sizes (Q size) under various prompt types.