Segment Anything Model Based Context Guided Mamba Unet for Medical Image Segmentation
Authors: Chenchen Wang, You Zhou
Organizations: School of Computer Science and Informatics, Cardiff University, Cardiff, UK · Department of Infection and Immunity, School of Medicine, Cardiff University, Cardiff, UK
Medical image segmentation is crucial for clinical diagnosis and treatment planning, requiring both global contextual understanding and fine-grained structural precision. While the Segment Anything Model (SAM) demonstrates strong generalization capabilities, its direct application to medical images often lacks domain-specific adaptability. Meanwhile, State Space Models (SSMs)-based methods, such as Mamba, efficiently capture long-range dependencies but may struggle with preserving fine anatomical details. To address these challenges, we propose a segment anything model based context guided mamba unet, SAMBA-UNet, a novel framework that integrates SAM's context knowledge with Mamba's hierarchical feature learning to enhance both global representation and local detail refinement. Specifically, we introduce the Hierarchical Context Enhancement Module (HCEM), an attention-based fusion module that deeply incorporates the multi-level features of SAM and Mamba to improve detail perception and segmentation precision. Additionally, we employ LoRA and Adapter-based fine-tuning to adapt SAM to medical imaging tasks, further enhancing its domain-specific segmentation capability and robustness. We conducted a comprehensive study on several medical segmentation datasets, including Synapse, ISIC17, and ISIC18, evaluating our method across multiple metrics. Experimental results demonstrate that SAMBA-UNet achieves state-of-the-art performance, effectively balancing generalization, segmentation accuracy, and computational efficiency. Our code will be released upon acceptance.
Medical image segmentation is fundamental for biomedical discovery. Existing methods lack generalizability and demand extensive, time-consuming manual annotation for new clinical application. Here, we propose MedSAM-3, a text promptable medical segmentation model for medical image and video segmentation. By fine-tuning the Segment Anything Model (SAM) 3 architecture on medical images paired with semantic conceptual labels, our MedSAM-3 enables medical Promptable Concept Segmentation (PCS), allowing precise targeting of anatomical structures via open-vocabulary text descriptions rather than solely geometric prompts. We further introduce the MedSAM-3 Agent, a framework that integrates Multimodal Large Language Models (MLLMs) to perform complex reasoning and iterative refinement in an agent-in-the-loop workflow. Comprehensive experiments across diverse medical imaging modalities, including X-ray, MRI, Ultrasound, CT, and video, demonstrate that our approach significantly outperforms existing specialist and foundation models. We will release our code and model at https://github.com/Joey-S-Liu/MedSAM3.
Anglin Liu, Xu R. Cao, Yifan Shen +4
The Hong Kong University of Science and Technology (Guangzhou) · University of Illinois Urbana-Champaign · Southeast University +1
The Segment Anything Model (SAM) is a promising foundation model for medical image segmentation, but its prompt-driven design is not directly suitable for fully automatic multi-class anatomical segmentation. Without manual point, box, or mask prompts, SAM faces a dual over-smoothing problem: deep image features may lose fine local structural evidence, while mask tokens may become semantically ambiguous. We propose SAM-DCE, a decoupled context enhancement framework for prompt-free medical image segmentation. SAM-DCE introduces Spatial Representation Compensation (SRC) to recover anatomical evidence from intermediate ViT features and Semantic Token Calibration (STC) to improve the category consistency of foreground mask tokens. Rather than replacing the pretrained decoder or fusing output masks, SAM-DCE refines the spatial and semantic representations before recombining them through SAM's original token-to-mask mechanism. On the Synapse multi-organ CT segmentation dataset, SAM-DCE achieved a mean Dice score of 83.86% and an HD95 of 16.04 mm, outperforming the representative CNN-, Transformer-, and SAM-based baselines evaluated here. Ablation experiments further showed that SRC and STC provide complementary improvements, supporting the proposed spatial--semantic factorization for prompt-free SAM adaptation.
Yingzhen Hu, Yiheng Zhong, Aijing Yang +2
Mohamed bin Zayed University of AI, Abu Dhabi, UAE · Xi’an Jiaotong-Liverpool University, Suzhou, China · University of Liverpool, Liverpool, United Kingdom
Deep learning has achieved remarkable success in medical image segmentation, often reaching expert-level accuracy in delineating tumors and tissues. However, most existing approaches remain task-specific, showing strong performance on individual datasets but limited generalization across diverse imaging modalities. Moreover, many methods focus primarily on the encoder, relying on large pretrained backbones that increase computational complexity. In this paper, we propose a decoder-centric approach for generalized 2D medical image segmentation. The proposed Deco-Mamba follows a U-Net-like structure with a Transformer-CNN-Mamba design. The encoder combines a CNN block and Transformer backbone for efficient feature extraction, while the decoder integrates our novel Co-Attention Gate (CAG), Vision State Space Module (VSSM), and deformable convolutional refinement block to enhance multi-scale contextual representation. Additionally, a windowed distribution-aware KL-divergence loss is introduced for deep supervision across multiple decoding stages. Extensive experiments on diverse medical image segmentation benchmarks yield state-of-the-art performance and strong generalization capability while maintaining moderate model complexity. The source code will be released upon acceptance.
Fares Bougourzi, Fadi Dornaika, Abdenour Hadid
IEMN UMR CNRS 8520, Université Polytechnique Hauts-de-France, 59313 Valenciennes, France · University of the Basque Country (UPV/EHU), Donostia-San Sebastián, Spain · Sorbonne Center for Artificial Intelligence, Sorbonne University Abu Dhabi, Abu Dhabi, UAE