Authors: Ruicheng Zhang, Jianhui Lei, Kaiwen Shen, Haowei Guo, Jun Zhou, Bin Chen, Mengtang Li, Shen Zhao, Shuo Li
Organizations: School of intelligent systems engineering, Sun Yat-sen University, Guangzhou 510006, China · Affiliated Hangzhou First People’s Hospital, Zhejiang University School of Medicine, Zhejiang, China · School of Engineering, Case Western Reserve University, Cleveland, USA
Deep snake is a promising family of instance segmentation methods that accurately predicts object-level contours, thereby overcoming common pixel-level misclassification issues such as mask cavities and jagged edges in semantic segmentation approaches. However, existing deep snake methods face challenges in handling complex morphological variations, accurately capturing fine-grained organ details, and correcting base detection errors. To mitigate these limitations, we propose a cohesive Text-prompted spatiotEmporal dual-heAd Mamba Snake (TEAMS), a novel vision-language Mamba snake framework with three key innovations: (1) A Spatiotemporal Snake Evolution Strategy (SSES) is introduced to tackle complex morphological variations by capturing bidirectional spatial dependencies along the snake contour and temporal dynamics across evolution steps in a state space model. (2) A Contour Morphology-Aware Mamba (CMAM) is proposed to quantify local contour morphologies to modulate the structured attention mask in the Mamba2 SSD dual form, which extends Mamba's capability to perceive the relative importance of its input sequence elements for better delineation of fine-grained organ details. (3) A Text-prompted Collaborative Dual-Head Snake (TCDHS) is designed to incorporate cues from textual prompts and transfer the evolved contour information to the base detection head, which enhances the deep snake workflow and mitigates wrong detections. Comprehensive evaluations on five datasets covering different organs and imaging modalities demonstrate that TEAMS outperforms existing semantic and deep snake segmentation methods (e.g., relative mDice/mBF improvements of 6.9%/9.1% in a spinal dataset), underscoring its potential as a reliable tool across diverse medical image segmentation scenarios.
Figures & tables
Fig. 1: Deep snake has the potential to address illogical errors in semantic segmentation caused by pixel misclassifications.
Fig. 2: Limitations of existing deep snake workflow: susceptibility to morphological variations, inadequate modeling of fine-grained organ details, and inability to correct base detection errors.
Fig. 3: Innovation and advantage of our TEAMS: TEAMS is a new deep snake method that captures spatial and temporal context during snake evolution (in SSES), dynamically adapts contour smoothness to fit geometric details (in CMAM), integrates cues from textual prompts, and feeds evolved contour features to enhance base detection (in TCDHS) with dual-head consistency feedback.
Fig. 4: The workflow of TEAMS. It first extracts image/text features via a feature pyramid backbone and ClinicalBERT, and fuses them with cross-modal self-attention for base detection (Fig. 4 (a)) and contour initialization; it then evolves contours with a dedicated snake evolution head (Fig. 4 (b)), where SSES captures spatial and temporal contexts (Fig. 4 (b1)) while CMAM adaptively regulates contour smoothness (Fig. 4 (b2)) for accurate snake evolution. Finally, TCDHS performs post-evolution detection and designs a dual-head detection consistency loss to align the detections of the base and post-evolution branches (Fig. 4 (b3)).
Fig. 5: The Spatiotemporal Snake Evolution Strategy (SSES). Fig. 5 (a) shows the bidirectional spatial state space module. Fig. 5 (b) shows the temporal trajectory state aggregation module.
Fig. 6: The Contour Morphology-Aware Mamba (CMAM). Fig. 6 (a) shows how CMAM quantifies contour morphology complexity. Fig. 6 (b) shows how the CM-SSD is designed to leverage the morphological complexity for modulating the structured attention mask.
Fig. 7: The Text-prompted Collaborative Dual-Head Snake (TCDHS). In Fig. 7 (a), task-level text prompts are fused with multi-scale visual features to condition the base detection head, which predicts Clsbase , Boxbase , and initializes P0 . In Fig. 7 (b), organ-level text prompts are aligned with RoI features to form multi-modal feature maps Fcsm for snake evolution. In Fig. 7 (c), the evolved contours P are converted into contour-aware heatmaps to augment features for a post-evolution detector. In Fig. 7 (d), a dual-head detection consistency loss is designed to align { Clsbase,Boxbase } with { Clspost,Boxpost }, transferring the contour information to the base detector.
Fig. 8: TEAMS achieves high instance segmentation performance on five challenging datasets of different imaging modalities and target organs (Column 11). It also outperforms state-of-the-art semantic segmentation (Columns 3 ∼ 8) and deep snake (Columns 9/10) methods.
Dataset ID/ Name
Modality / Target organs
Dataset size
Target class numbers
1. MR_AVBCE- Extended
MRI / Spine
∼ 480 scans, ∼ 1233 slices
50
2. VerSe
CT / Spine
∼ 374 scans, ∼ 8000 slices
26
3. BTCV
CT / Abdomen
30 scans, ∼ 900 slices
8
4. RAOS
CT / Abdomen
∼ 317 scans, ∼ 7500 slices
19
5. PanNuke
Microscopy / Nuclei
∼ 7900 image patches
5
Table 1: Overview of the five evaluation datasets.
Method
MR_AVBCE- Extended
VerSe19(Pu)
VerSe19(Pr)
VerSe20(Pu)
VerSe20(Pr)
BTCV
RAOS
PanNuke
mIoU
mDice
mBF
mIoU
mDice
mBF
mIoU
mDice
mBF
mIoU
mDice
mBF
mIoU
mDice
mBF
mIoU
mDice
mBF
mIoU
mDice
mBF
mPQ
TEAMS
69.2 ∗ ± 0.5
77.4 ∗ ± 0.4
69.6 ∗ ± 0.6
86.5 ∗ ± 0.4
92.5 ∗ ± 0.2
82.2 ∗ ± 0.4
85.9 ∗ ± 0.2
91.6 ± 0.4
81.5 ∗ ± 0.4
87.0 ∗ ± 0.6
92.9 ∗ ± 0.3
80.7 ∗ ± 0.5
85.8 ∗ ± 0.4
91.8 ∗ ± 0.2
80.9 ∗ ± 0.3
85.3 ± 0.2
91.7 ∗ ± 0.1
79.8 ∗ ± 0.4
79.8 ∗ ± 0.3
87.9 ∗ ± 0.4
75.9 ∗ ± 0.7
41.1 ± 0.8
nnU-Net v2 ( Isensee et al. (2021) )
62.7 ± 1.1
70.1 ± 1.0
61.8 ± 1.3
82.2 ± 0.5
89.7 ± 0.3
76.8 ± 0.6
81.0 ± 0.3
88.8 ± 0.4
75.6 ± 0.8
80.2 ± 0.4
88.1 ± 0.3
73.9 ± 0.8
81.4 ± 0.4
89.5 ± 0.3
75.4 ± 0.8
81.9 ± 0.6
86.0 ± 0.9
71.4 ± 0.7
73.8 ± 0.2
85.7 ± 0.3
68.6 ± 0.8
34.5 ± 1.1
UNETR ( Hatamizadeh et al. (2022) )
56.1 ± 1.1
67.3 ± 1.1
57.7 ± 1.2
78.6 ± 0.5
85.3 ± 0.6
71.6 ± 0.9
79.7 ± 0.7
85.5 ± 0.6
71.3 ± 0.5
78.0 ± 0.6
85.3 ± 0.5
71.8 ± 0.7
79.2 ± 0.5
86.4 ± 0.4
72.5 ± 0.9
81.0 ± 0.6
84.1 ± 0.7
69.6 ± 1.1
74.1 ± 0.7
85.9 ± 0.8
68.1 ± 1.0
31.4 ± 1.3
TransUNet ( Chen et al. (2021) )
55.8 ± 1.0
66.7 ± 1.0
56.4 ± 1.3
78.5 ± 0.7
84.5 ± 0.7
70.1 ± 0.9
77.8 ± 0.6
84.2 ± 0.6
69.5 ± 0.7
78.4 ± 0.6
85.0 ± 0.7
71.2 ± 0.7
76.8 ± 0.6
83.9 ± 0.6
69.1 ± 0.9
79.0 ± 0.9
85.3 ± 0.9
70.2 ± 1.0
72.3 ± 0.6
84.1 ± 0.6
67.2 ± 1.1
35.8 ± 1.2
Swin-Unet ( Cao et al. (2023a) )
56.4 ± 1.0
67.0 ± 0.9
57.3 ± 1.4
80.8 ± 0.6
87.3 ± 0.3
73.4 ± 1.1
80.4 ± 0.6
85.7 ± 0.5
72.6 ± 0.7
79.5 ± 0.7
86.6 ± 0.6
72.9 ± 0.5
79.8 ± 0.6
85.1 ± 0.5
71.7 ± 0.8
80.8 ± 0.6
83.9 ± 0.7
67.8 ± 0.9
73.0 ± 0.8
83.6 ± 0.8
66.7 ± 0.9
31.3 ± 1.2
Table 2: Superiority of TEAMS over 14 SOTA methods on five datasets. Results are reported as mean ± standard deviation. The best results are highlighted in bold (this annotation holds for the subsequent tables), and the second-best results are underlined. Statistical significance is marked with * when p ¡ 0.05 compared with the strongest competing method under two-sided paired t-tests across folds or repeated runs.
Fig. 9: TEAMS consistently achieves strong results for most organ categories on the two most challenging datasets (Datasets 1 and 4). TEAMS (solid lines) outperforms the best semantic segmentation approach (dashed lines) for each organ.
Model Configuration
mIoU
mDice
mBF
Baseline
58.2
67.5
58.8
+ SSES
62.2( ⇑ 6.9%)
72.9( ⇑ 8.0%)
62.8( ⇑ 6.8%)
+ CMAM
59.1( ⇑ 1.5%)
71.3( ⇑ 5.6%)
60.3( ⇑ 2.6%)
+ Text prompts in TCDHS
60.7( ⇑ 4.3%)
71.3( ⇑ 5.6%)
61.4( ⇑ 4.4%)
+ Consistency feedback in TCDHS
61.3( ⇑ 5.3%)
71.6( ⇑ 6.1%)
60.8( ⇑ 3.4%)
+ TCDHS
61.9( ⇑ 6.3%)
72.4( ⇑ 7.3%)
62.5( ⇑ 6.3%)
Table 3: Ablation study demonstrating the effect of SSES, CMAM, text prompts, detection consistency feedback, and full TCDHS.
Fig. 10: The contributions of each core module. When each core module is enabled, the evolution failures are mitigated (a1 VS a2), the boundary details are better delineated (b1 VS b2), and the wrong detections are avoided (c1 VS c2).
Fig. 11: The necessity of Mamba modeling in deep snakes. Fig. 11 (a)/(b) show SSES/CMAM (green) better benefits overall performance than hyperparameter tuning (blue). Fig. 11 (c) shows SSES (c1) better mitigates evolution failure than increasing iteration numbers (c2). Fig. 11 (d) shows CMAM (d1) better delineates fine-grained organ details than increasing contour points (d2).
Temporal modeling strategy
mIoU
mDice
mBF
Standard temporal feature aggregation
66.6
73.6
66.5
GRU-based recurrent modeling
66.1
72.9
62.9
SSES
69.2
77.4
69.6
Table 4: SSES achieves superior performance compared with existing temporal modeling strategies.
Temporal modeling method
mIoU
mDice
mBF
Trainable params (M)
Inference time (s)
GPU mem- ory (MB)
Attention-based temporal fusion
63.0
73.9
63.1
17.67
1.25
3066
Exponential decay aggregation (ours)
69.2
77.4
69.6
15.74
1.17
2852
Table 5: The superiority of exponential decay aggregation over attention-based temporal fusion.
Dataset
Method
mIoU
mDice
mBF
mPQ
MR_AVBCE- Extended
Learned alternative
61.9
69.9
60.4
–
Our CMAM
69.2
77.4
69.6
–
VerSe
Learned alternative
80.4
83.1
71.0
–
Our CMAM
87.0
92.9
80.7
–
BTCV
Learned alternative
79.7
82.4
70.6
–
Our CMAM
85.3
91.7
79.8
–
Table 6: The morphology-guided learnable structured attention mask modeling in CMAM is better than the learned alternative.
Variant
mIoU
mDice
mBF
Convolution-based snake evolution
58.9
66.0
60.9
Transformer-based snake evolution
63.5
72.8
64.9
Mamba-based snake evolution (ours)
69.2
77.4
69.6
Table 7: Mamba-based snake evolution outperforms convolution-based and Transformer-based alternatives.
Fig. 12: The benefits of the text prompts. The task-level text prompts help to better distinguish different organs (a1 VS a2), and the organ-level text prompts help the snake to better evolve to target organs (b1 VS b2).
Setting
mIoU
mDice
mBF
No prompt
65.8
73.6
64.3
Random prompt
63.3
71.9
62.8
Incorrect prompt
63.0
71.3
62.1
Different correct template 1
68.9
77.4
68.8
Different correct template 2
68.5
76.7
68.5
Correct prompt / TEAMS
69.2
77.4
69.6
Table 8: Ablation study on different textual prompt settings validates that textual prompts benefit snake segmentation.
Method
Params (M)
Inference time (s)
Peak GPU memory (MB)
mDice/ mBF
Conference version
12.48
0.83
1856
71.3/63.6
Journal version (TEAMS)
15.74 (trainable) + 137.43 (frozen)
1.17
2852
77.4/69.6
Table 9: The computational costs of TEAMS and the conference version on MR_AVBCE-Extended.
Fig. 13: Representative failure cases of TEAMS under annotation ambiguity. The solid contours denote the ground truth, and the dashed contours denote the evolved snake predictions. Red arrows indicate ambiguous vessel regions near the liver boundary: the vessel is excluded from the liver ground truth in Fig. 13 (a1), but included in the liver ground truth in Fig. 13 (a2). Such inconsistent annotations may lead to suboptimal segmentation results.
Medical image segmentation delineates organs, lesions, and other structures in scans to support measurement, diagnosis, treatment planning, and research.
The Segment Anything Model (SAM) is a promising foundation model for medical image segmentation, but its prompt-driven design is not directly suitable for fully automatic multi-class anatomical segmentation. Without manual point, box, or mask prompts, SAM faces a dual over-smoothing problem: deep image features may lose fine local structural evidence, while mask tokens may become semantically ambiguous. We propose SAM-DCE, a decoupled context enhancement framework for prompt-free medical image segmentation. SAM-DCE introduces Spatial Representation Compensation (SRC) to recover anatomical evidence from intermediate ViT features and Semantic Token Calibration (STC) to improve the category consistency of foreground mask tokens. Rather than replacing the pretrained decoder or fusing output masks, SAM-DCE refines the spatial and semantic representations before recombining them through SAM's original token-to-mask mechanism. On the Synapse multi-organ CT segmentation dataset, SAM-DCE achieved a mean Dice score of 83.86% and an HD95 of 16.04 mm, outperforming the representative CNN-, Transformer-, and SAM-based baselines evaluated here. Ablation experiments further showed that SRC and STC provide complementary improvements, supporting the proposed spatial--semantic factorization for prompt-free SAM adaptation.
Yingzhen Hu, Yiheng Zhong, Aijing Yang +2
Mohamed bin Zayed University of AI, Abu Dhabi, UAE · Xi’an Jiaotong-Liverpool University, Suzhou, China · University of Liverpool, Liverpool, United Kingdom
Medical image segmentation remains challenging due to limited fine-grained annotations, complex anatomical structures, and image degradation from noise, low contrast, or illumination variation. We propose TAMISeg, a text-guided segmentation framework that incorporates clinical language prompts and semantic distillation as auxiliary semantic cues to enhance visual understanding and reduce reliance on pixel-level fine-grained annotations. TAMISeg integrates three core components: a consistency-aware encoder pretrained with strong perturbations for robust feature extraction, a semantic encoder distillation module with supervision from a frozen DINOv3 teacher to enhance semantic discriminability, and a scale-adaptive decoder that segments anatomical structures across different spatial scales. Experiments on the Kvasir-SEG, MosMedData+, and QaTa-COV19 datasets demonstrate that TAMISeg consistently outperforms existing uni-modal and multi-modal methods in both qualitative and quantitative evaluations. Code will be made publicly available at https://github.com/qczggaoqiang/TAMISeg.
Qiang Gao, Yi Wang, Yong Zhang +4
Department of Data Science and AI, Monash University, Melbourne, Australia · College of Computer Science, Chongqing University, Chongqing, China · Chongqing University Central Hospital, School of Medicine, Chongqing, China
Recently Mamba-based methods have shown promise in abdominal organ segmentation. However, existing approaches neglect cross-channel anatomical semantic collaboration and lack explicit boundary-aware feature fusion mechanisms. To address these limitations, we propose CS-MUNet with two purpose-built modules. The Boundary-Aware State Mamba module employs a Bayesian-attention framework to generate pixel-level boundary posterior maps, injected directly into Mamba's core scan parameters to embed boundary awareness into the SSM state transition mechanism, while dual-branch weight allocation enables complementary modulation between global and local structural representations. The Channel Mamba State Aggregation module redefines the channel dimension as the SSM sequence dimension to explicitly model cross-channel anatomical semantic collaboration in a data-driven manner. Experiments on two public benchmarks demonstrate that CS-MUNet consistently outperforms state-of-the-art methods across multiple metrics, establishing a new SSM modeling paradigm that jointly addresses channel semantic collaboration and boundary-aware feature fusion for abdominal multi-organ segmentation.
Yuyang Zheng, Mingda Zhang, Jianglong Qin +4
School of Software Engineering, Yunnan University, KunMing, 650500, Yunnan, China. · The School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, 518172, Guangdong, China. · Yunnan Provincial Key Laboratory of Software Engineering, Yunnan University, Kunming, 650500, Yunnan, China.