SpectralMamba-UNet: Frequency-Disentangled State Space Modeling for Texture-Structure Consistent Medical Image Segmentation

Fuhao Zhang; Lei Liu; Jialin Zhang; Ya-Nan Zhang; Nan Mu

cs.CVFeb 26, 2026

SpectralMamba-UNet: Frequency-Disentangled State Space Modeling for Texture-Structure Consistent Medical Image Segmentation

Authors: Fuhao Zhang, Lei Liu, Jialin Zhang, Ya-Nan Zhang, Nan Mu

Organizations: College of Computer Science, Sichuan Normal University, Chengdu, Sichuan 610068, China · Zhejiang University & Ant Group, Hangzhou, Zhejiang, China · State Key Laboratory of Public Big Data, Guizhou University, Guiyang, Guizhou 550025, China

Abstract

Accurate medical image segmentation requires effective modeling of both global anatomical structures and fine-grained boundary details. Recent state space models (e.g., Vision Mamba) offer efficient long-range dependency modeling. However, their one-dimensional serialization weakens local spatial continuity and high-frequency representation. To this end, we propose SpectralMamba-UNet, a novel frequency-disentangled framework to decouple the learning of structural and textural information in the spectral domain. Our Spectral Decomposition and Modeling (SDM) module applies discrete cosine transform to decompose low- and high-frequency features, where low frequency contributes to global contextual modeling via a frequency-domain Mamba and high frequency preserves boundary-sensitive details. To balance spectral contributions, we introduce a Spectral Channel Reweighting (SCR) mechanism to form channel-wise frequency-aware attention, and a Spectral-Guided Fusion (SGF) module to achieve adaptively multi-scale fusion in the decoder. Experiments on five public benchmarks demonstrate consistent improvements across diverse modalities and segmentation targets, validating the effectiveness and generalizability of our approach.

Figures & tables

Figure 1: Architecture of SpectralMamba-UNet. SDM performs spectral decomposition, SCR reweights frequency responses, and SGF enables frequency-guided decoder fusion.

Model	DSC $\uparrow$	HD95 $\downarrow$	Aorta	Gallbladder	Kidney (L)	Kidney (R)	Liver	Pancreas	Spleen	Stomach
Res-UNet	77.27	20.86	88.58	55.60	80.54	77.96	94.18	61.68	88.05	71.59
TransUNet	77.48	31.69	87.23	63.13	81.87	77.02	94.08	55.86	85.08	75.62
Swin-Transformer	79.13	21.55	85.47	66.53	83.28	79.61	94.29	56.58	90.66	76.60
UltraLight VM-UNet	38.12	56.28	58.96	–	–	–	87.69	31.50	73.48	53.35
VM-UNet	81.08	19.21	86.40	69.41	86.16	82.76	94.17	58.80	89.51	81.40
SpectralMamba-UNet	81.10	15.31	89.79	63.14	85.84	81.36	94.91	69.69	88.86	75.09

Table 1: Quantitative comparison on the Synapse dataset (multi-organ CT segmentation). Best results are shown in bold and second-best in underline .

Model	ACDC				DRIVE		EAT		IA
Model	mDSC $\uparrow$	RV	Myo	LV	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$
Res-UNet	91.25	90.57	89.10	94.07	81.96	3.05	88.33	1.33	84.14	34.28
TransUNet	89.71	88.86	84.53	95.73	81.81	6.09	86.36	1.64	80.84	61.19
Swin-Transformer	90.00	88.55	85.62	95.83	71.54	4.94	84.41	1.58	84.49	21.52
UltraLight VM-UNet	78.84	76.68	75.53	84.32	78.57	9.64	76.78	2.76	85.23	23.90
VM-UNet	90.56	87.77	87.89	95.02	80.96	8.99	87.06	1.46	81.86	33.88

Table 2: Quantitative comparison on ACDC (cardiac MRI), DRIVE (retinal vessels), EAT (epicardial adipose tissue), and IA (intracranial aneurysm) datasets. Best results are shown in bold and second-best in underline .

Figure 2: Qualitative comparison on Synapse, ACDC, EAT, IA, and DRIVE (left to right). Compared with representative baselines (c–g), SpectralMamba-UNet (h) produces sharper boundaries and improved topological consistency.

Variant	Synapse		ACDC		DRIVE		EAT		IA
Variant	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$	DSC $\uparrow$	HD95 $\downarrow$
Baseline	77.27	20.86	91.25	–	81.96	3.05	88.33	1.33	84.14	34.28
+Freq	77.76	17.08	91.83	–	83.05	2.70	88.87	1.71	85.63	22.76
+Spatial Mamba	77.51	22.07	91.76	–	83.07	2.58	88.40	1.54	84.66	33.03
+Freq+SCR+SGF	78.72	18.48	91.82	–	83.48	2.16	89.45	1.58	84.61	36.26
+SDM	78.05	22.19	91.77	–	83.39	2.39	90.16	1.27	84.40	26.78

Table 3: Ablation study of different components. +Freq denotes spectral-domain convolution (without Mamba), +Spatial Mamba denotes spatial-domain Mamba modeling, and +Freq+SCR+SGF represents spectral decomposition with channel reweighting and guided fusion. Best results are shown in bold .

Figure 3: Qualitative comparison of ablation variants. From left to right: input image, ground truth, Baseline, +Freq, +Spatial Mamba, +Freq+SCR+SGF, +SDM, and the complete SpectralMamba-UNet. The full model produces clearer boundaries and improved structural continuity across datasets.

Medical Image Segmentation

Medical image segmentation delineates organs, lesions, and other structures in scans to support measurement, diagnosis, treatment planning, and research.

See all 788 papers

USEMA: a Scalable Efficient Mamba Like Attention for Medical Image Segmentation
May 11, 2026Elisha Dayag, Nhat Thanh Tran, Jack XinImage Segmentation Hybrid CNN-Transformer Architectures
University of California Irvine, Irvine CA 92697
USP-Mamba: Unmixing-Derived Spectral and Structural Prompting for Hyperspectral Image Super-Resolution
Aug 3, 2026Shi Chen, Jie Zhang, Yicong ZhouMamba Hyperspectral Image Super-Resolution
Department of Computer Science, University of Macau, Macau, China
SpectraFlow: Unifying Structural Pretraining and Frequency Adaptation for Medical Image Segmentation
May 14, 2026Zhiquan Chen, Haitao Wang, Guowei Zou +1Flow Matching Image Segmentation
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China · Guangdong Key Laboratory of Big Data Analysis and Processing, Guangzhou, China

May 11, 2026J/K move · Enter open · S save

USEMA: a Scalable Efficient Mamba Like Attention for Medical Image Segmentation

Elisha Dayag, Nhat Thanh Tran, Jack Xin

University of California Irvine, Irvine CA 92697

Image Segmentation Hybrid CNN-Transformer Architectures Self-Attention U-Net Efficient Attention Medical Image Segmentation

Open paper AskPDF arXiv

Accurate medical image segmentation is an integral part of the medical image analysis pipeline that requires the ability to merge local and global information. While vision transformers are able to capture global interactions using vanilla self-attention, their quadratic computational complexity in the input size remains a struggle for medical image segmentation tasks. Motivated by the dispersion property of vanilla self-attention and recent development of Mamba form of attention, Scalable and Efficient Mamba like Attention (SEMA) utilizes token localization via local window attention to avoid dispersion and maintain focusing, complemented by theoretically consistent arithmetic averaging to capture global aspect of attention. In this work, we present USEMA, a hybrid UNet architecture that merges the local feature extraction ability of convolutional neural networks (CNNs) with SEMA attention. We conduct experiments with USEMA across a variety of modalities and image sizes, demonstrating improved computational efficiency compared to transformer based models using full self-attention, and superior segmentation performance relative to purely convolution and Mamba-based models.

Abstract

Medical Image Segmentation

Explore similar work

USEMA: a Scalable Efficient Mamba Like Attention for Medical Image Segmentation

USP-Mamba: Unmixing-Derived Spectral and Structural Prompting for Hyperspectral Image Super-Resolution

SpectraFlow: Unifying Structural Pretraining and Frequency Adaptation for Medical Image Segmentation