SEMamba++: A General Speech Restoration Framework Leveraging Global, Local, and Periodic Spectral Patterns

Yongjoon Lee; Jung-Woo Choi

eess.ASMar 12, 2026

SEMamba++: A General Speech Restoration Framework Leveraging Global, Local, and Periodic Spectral Patterns

Authors: Yongjoon Lee, Jung-Woo Choi

Organizations: Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Korea

Abstract

General speech restoration demands techniques that can interpret complex speech structures under various distortions. While State-Space Models like SEMamba have advanced the state-of-the-art in speech denoising, they are not inherently optimized for critical speech characteristics, such as spectral periodicity or multi-resolution frequency analysis. In this work, we introduce an architecture tailored to incorporate speech-specific features as inductive biases. In particular, we propose the Global, Local, and Periodic (GLP) module, a frequency feature extraction block that effectively and efficiently leverages the properties of frequency bins. Then, we design a multi-resolution parallel time-frequency dual-processing block to capture diverse spectral patterns, and a learnable mapping to further enhance model performance. With all our ideas combined, the proposed SEMamba++ achieves the best performance among multiple baseline models while remaining computationally efficient.

Figures & tables

Figure 1: Various TFDP processing methods. ds and us denote downsampling and upsampling, respectively.

Figure 2: Overall architecture of SEMamba++ (a) and Frequency GLP (b). In (a), ”Mag.”, ”Comp.”, and ”Decomp.” refer to magnitude, compression, and decompression, respectively. ”Conv” and ”TrConv” indicate the 2d convolution and transposed convolution that down- and upsample along the frequency axis. (b) describes Frequency GLP with frequency dimension $F^{\prime}$ .

		In-domain						Out-of-domain
Model	Method	VCTK-GSR test						URGENT 2025 val						URGENT 2025 test
		Perceptual quality			Signal fidelity			Perceptual quality			Signal fidelity			Perceptual quality
		$\text{SCOREQ}^{\uparrow}$	$\text{UTMOS}^{\uparrow}$	$\text{OVRL}^{\uparrow}$	$\text{PESQ}^{\uparrow}$	$\text{LSD}^{\downarrow}$	$\text{LPS}^{\uparrow}$	$\text{SCOREQ}^{\uparrow}$	$\text{UTMOS}^{\uparrow}$	$\text{OVRL}^{\uparrow}$	$\text{PESQ}^{\uparrow}$	$\text{LSD}^{\downarrow}$	$\text{LPS}^{\uparrow}$	$\text{SCOREQ}^{\uparrow}$	$\text{UTMOS}^{\uparrow}$	$\text{OVRL}^{\uparrow}$
MP-SENet	GAN	1.87	2.17	2.82	2.01	1.22	0.42	1.59	1.90	2.88	1.63	1.78	0.59	1.38	1.71	2.81
SEMamba	GAN	1.89	2.34	2.88	2.13	1.19	0.45	1.66	2.06	2.92	1.67	1.83	0.60	1.51	1.88	2.84
USEMamba	Reg.	1.87	2.30	2.94	1.90	1.21	0.42	1.77	2.05	2.98	1.53	1.51	0.60	1.60	1.88	2.91

Table 1: Objective evaluation results on VCTK-GSR test , URGENT 2025 val , and URGENT 2025 test . Reg. denotes the training with regressive loss. Official checkpoints have been utilized for models with ∗ .

Model	Params (M)	RTF	SIG	BAK	OVRL	PESQ
Degraded	-		2.89	3.02	2.47	1.91
MP-SENet	2.3	0.022	3.27	3.89	2.94	1.83
SEMamba	1.7	0.013	3.21	3.88	2.88	1.68
USEMamba	3.9	0.045	3.36	3.99	3.06	1.73
Universe++	42.8	0.023	2.58	3.83	2.32	1.11
Universe++ ∗	42.8	0.023	3.10	3.86	2.80	1.77

Table 2: Efficiency and objective performance evaluation results on AATC Challenge 2025. Official checkpoints have been utilized for models with ∗ .

Model	Methods	Params (M)	Training data (hours)	SIG	BAK	OVRL
Degraded	-			3.053	2.509	2.255
ANYENHANCE †	MGM	363.6	3.7k	3.488	3.977	3.161
MaskSR-M †	MLM	145	800	3.430	4.025	3.136
LLaSE-G1 †	LM	1072	500	3.472	3.996	3.177
VoiceFixer ∗	Reg.	111.7	44	3.291	3.960	2.992
USEMamba	Reg.	3.9	44	3.239	3.937	2.923

Table 3: Joint denoising and reverberation performance, evaluated by DNSMOS on DNS 2020 test real recordings. All models are trained for solving general speech restoration. Diff. and Reg. denote the training with diffusion and regressive loss, respectively. ∗ indicates models from official checkpoints, and † represents model performance reported in the source papers.

Figure 3: Gradient visualization of outputs from different branches in the proposed multi-branch method. (a) and (b) represent the magnitude spectrogram of the clean and the degraded speech, respectively. Resolution-wise visualizations of the gradient-weighted magnitude spectrogram are illustrated in (1), (2), and (3). The top resolution has a frequency dimension equal to $F^{\prime}$ .

Model	Degradation types
	Noise (dB)				Bandwidth limitation (kHz)
	-15	-10	0	10	1	2	4	6
Degraded	1.42	1.46	1.6	2.25	1.96	2.75	3.73	3.95
MP-SENet	1.94	2.34	3.33	3.82	1.85	2.80	3.73	3.99
SEMamba	2.06	2.51	3.48	3.89	1.85	2.79	3.72	3.97
USEMamba	1.98	2.38	3.35	3.83	1.52	2.89	3.85	4.04

Table 4: UTMOS evaluation on VCTK-GSR test dataset across different degradation intensities. The gray areas represent the evaluation results on unseen degradation intensities.

Method	VCTK-GSR test	URGENT test	DNS test
MR-Parallel	0.036	0.036	0.037
MR-Sequential	0.045	0.047	0.046

Table 5: Intersection Over Union (IoU) values of influential gradients across multiple resolutions.

Figure 4: The ratio of gradient norms under different degradation types and intensity. $\mathcal{R}>1$ indicates the larger contribution of the Global Periodicity module than the Local module.

Figure 5: Softplus functions with the learnable $\beta$ for each frequency bin. The black dotted line denotes the ReLU function.

Design	Params (M)	RTF	In-domain		Out-of-domain
			VCTK-GSR test		URGENT test		DNS test
			UTMOS	OVRL	UTMOS	OVRL	UTMOS	OVRL
Mamba	2.7	0.025	3.55	3.12	2.50	3.04	2.70	3.06
Transformer	2.9	0.027	2.95	2.95	2.24	2.95	2.40	3.04
Conformer	2.4	0.028	3.45	3.06	2.46	3.04	2.93	3.17
SpatialNet	2.6	0.024	3.46	3.10	2.36	2.98	2.68	3.11

Table 6: Comparison and ablation studies on various frequency mixing modules using SEMamba++ as a backbone.

Design	Params (M)	RTF	In-domain		Out-of-domain
			VCTK-GSR test		URGENT test		DNS test
			UTMOS	OVRL	UTMOS	OVRL	UTMOS	OVRL
MR-Parallel (Ours)	2.7	0.021	3.55	3.14	2.61	3.13	3.02	3.21
MR-Sequential	2.7	0.021	3.54	3.12	2.53	3.08	2.91	3.21
SR	9.0	0.034	3.53	3.11	2.42	2.85	2.19	2.48
w/o down.	2.8	0.028	3.50	3.09	2.44	2.95	2.72	3.0

Table 7: Ablation studies on the proposed multi-resolution parallel TFDP processing.

\times

2 and

\times

4 denote downsampling by factors of 2 and 4, respectively.

Design	Params (M)	RTF	In-domain		Out-of-domain
			VCTK-GSR test		URGENT test		DNS test
			UTMOS	PESQ	UTMOS	OVRL	UTMOS	OVRL
SEMamba	1.7	0.013	2.34	2.13	1.79	2.80	1.57	2.75
LMask. $\mathrel{\mathchoice{\mkern 2.0mu\hbox{\hbox to8.19pt{\vbox to5.03pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{{}{}{}{}\lxSVG@discardpath\lxSVG@discardpath@clipped{M 0 0 L 0 6.96 L 11.33 6.96 L 11.33 0 Z} {{{}{}{{}}{} {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 0 0)} \pgfsys@hbox{59}\lxSVG@closescope }}}} {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}}{\mkern 2.0mu\hbox{\hbox to8.19pt{\vbox to5.03pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{{}{}{}{}\lxSVG@discardpath\lxSVG@discardpath@clipped{M 0 0 L 0 6.96 L 11.33 6.96 L 11.33 0 Z} {{{}{}{{}}{} {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 0 0)} \pgfsys@hbox{59}\lxSVG@closescope }}}} {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}}{\mkern 2.0mu\hbox{\hbox to5.01pt{\vbox to3.5pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{{}{}{}{}\lxSVG@discardpath\lxSVG@discardpath@clipped{M 0 0 L 0 4.84 L 6.93 4.84 L 6.93 0 Z} {{{}{}{{}}{} {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 0 0)} \pgfsys@hbox{59}\lxSVG@closescope }}}} {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}}{\mkern 2.0mu\hbox{\hbox to3.58pt{\vbox to2.5pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{{}{}{}{}\lxSVG@discardpath\lxSVG@discardpath@clipped{M 0 0 L 0 3.46 L 4.95 3.46 L 4.95 0 Z} {{{}{}{{}}{} {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 0 0)} \pgfsys@hbox{59}\lxSVG@closescope }}}} {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}}}$ LMap.	1.7	0.013	2.30	2.22	1.82	2.87	1.82	2.90
Metric. $\mathrel{\mathchoice{\mkern 2.0mu\hbox{\hbox to8.19pt{\vbox to5.03pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{{}{}{}{}\lxSVG@discardpath\lxSVG@discardpath@clipped{M 0 0 L 0 6.96 L 11.33 6.96 L 11.33 0 Z} {{{}{}{{}}{} {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 0 0)} \pgfsys@hbox{59}\lxSVG@closescope }}}} {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}}{\mkern 2.0mu\hbox{\hbox to8.19pt{\vbox to5.03pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{{}{}{}{}\lxSVG@discardpath\lxSVG@discardpath@clipped{M 0 0 L 0 6.96 L 11.33 6.96 L 11.33 0 Z} {{{}{}{{}}{} {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 0 0)} \pgfsys@hbox{59}\lxSVG@closescope }}}} {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}}{\mkern 2.0mu\hbox{\hbox to5.01pt{\vbox to3.5pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{{}{}{}{}\lxSVG@discardpath\lxSVG@discardpath@clipped{M 0 0 L 0 4.84 L 6.93 4.84 L 6.93 0 Z} {{{}{}{{}}{} {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 0 0)} \pgfsys@hbox{59}\lxSVG@closescope }}}} {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}}{\mkern 2.0mu\hbox{\hbox to3.58pt{\vbox to2.5pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{{}{}{}{}\lxSVG@discardpath\lxSVG@discardpath@clipped{M 0 0 L 0 3.46 L 4.95 3.46 L 4.95 0 Z} {{{}{}{{}}{} {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 0 0)} \pgfsys@hbox{59}\lxSVG@closescope }}}} {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}}}$ Vocoder	1.7	0.013	3.32	1.71	2.28	2.91	2.14	2.88
TFMamba-large	2.7	0.018	3.31	1.75	2.23	2.84	2.31	2.85

Table 8: Ablation studies on the model design choices. LMask., LMap., and Metric. refer to learnable masking, mapping, and MetricGAN, respectively.

Speech Enhancement

Speech enhancement improves intelligibility and listening quality by suppressing noise, reverberation, and competing speakers in recorded or live audio.

See all 128 papers

RT-SEMamba: Real-Time Speech Enhancement Mamba via Progressive Knowledge Distillation
Aug 12, 2026Rong Chao, Sung-Feng Huang, Moreno La Quatra +4Mamba Speech Enhancement
Academia Sinica, Taiwan · National Taiwan University, Taiwan · NVIDIA +2
TF-MossFormer: Integrating Convolution Gated Local-Global Attentions for Enhanced Time-Frequency Domain Monaural Speech Separation
Jul 23, 2026Shengkui Zhao, Zexu Pan, Haoxu Wang +3Hybrid CNN-Transformer Architectures Frequency-Domain Feature Learning
Token Foundry, Alibaba Group, Singapore
DualSpecSE: A Dual-Path Speech Enhancement Network Integrating Mel and Complex Spectrograms
Sep 12, 2026Xingchen Li, Ziqian Wang, Zikai Liu +4Speech Enhancement
Huawei Technologies Co., Ltd., China

Aug 12, 2026J/K move · Enter open · S save

RT-SEMamba: Real-Time Speech Enhancement Mamba via Progressive Knowledge Distillation

Rong Chao, Sung-Feng Huang, Moreno La Quatra, Sabato Marco Siniscalchi+3

Academia Sinica, Taiwan · National Taiwan University, Taiwan · NVIDIA+2

Mamba Speech Enhancement Knowledge Distillation

Open paper AskPDF arXiv

We present RT-SEMamba, a fully causal speech enhancement (SE) model built upon causal time-frequency Mamba blocks. Unlike Transformer-based architectures that rely on a growing key-value cache, Mamba propagates a fixed-size recurrent state per layer, enabling memory- and bandwidth-efficient long-form inference. We further introduce a progressive knowledge distillation (KD) strategy that compresses an 8-layer teacher into a shallow 1-layer student by jointly distilling complex spectral outputs and intermediate representations. On Voicebank-DEMAND, the 8-layer RT-SEMamba achieves 3.32 PESQ with a 25 ms algorithmic latency constraint, and the distilled 1-layer student improves over a naive 1-layer baseline from 3.06 to 3.18 PESQ while preserving the same steady-state RTF, delivering a 2.64x speedup over the teacher. These results demonstrate that state-space models with progressive KD provide a competitive quality-latency trade-off for real-time SE.

Abstract

Speech Enhancement

Explore similar work

RT-SEMamba: Real-Time Speech Enhancement Mamba via Progressive Knowledge Distillation

TF-MossFormer: Integrating Convolution Gated Local-Global Attentions for Enhanced Time-Frequency Domain Monaural Speech Separation

DualSpecSE: A Dual-Path Speech Enhancement Network Integrating Mel and Complex Spectrograms