SEMamba++: A General Speech Restoration Framework Leveraging Global, Local, and Periodic Spectral Patterns
Organizations: Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Korea
Abstract
General speech restoration demands techniques that can interpret complex speech structures under various distortions. While State-Space Models like SEMamba have advanced the state-of-the-art in speech denoising, they are not inherently optimized for critical speech characteristics, such as spectral periodicity or multi-resolution frequency analysis. In this work, we introduce an architecture tailored to incorporate speech-specific features as inductive biases. In particular, we propose the Global, Local, and Periodic (GLP) module, a frequency feature extraction block that effectively and efficiently leverages the properties of frequency bins. Then, we design a multi-resolution parallel time-frequency dual-processing block to capture diverse spectral patterns, and a learnable mapping to further enhance model performance. With all our ideas combined, the proposed SEMamba++ achieves the best performance among multiple baseline models while remaining computationally efficient.
Figures & tables
| In-domain | Out-of-domain | |||||||||||||||
| Model | Method | VCTK-GSR test | URGENT 2025 val | URGENT 2025 test | ||||||||||||
| Perceptual quality | Signal fidelity | Perceptual quality | Signal fidelity | Perceptual quality | ||||||||||||
| MP-SENet | GAN | 1.87 | 2.17 | 2.82 | 2.01 | 1.22 | 0.42 | 1.59 | 1.90 | 2.88 | 1.63 | 1.78 | 0.59 | 1.38 | 1.71 | 2.81 |
| SEMamba | GAN | 1.89 | 2.34 | 2.88 | 2.13 | 1.19 | 0.45 | 1.66 | 2.06 | 2.92 | 1.67 | 1.83 | 0.60 | 1.51 | 1.88 | 2.84 |
| USEMamba | Reg. | 1.87 | 2.30 | 2.94 | 1.90 | 1.21 | 0.42 | 1.77 | 2.05 | 2.98 | 1.53 | 1.51 | 0.60 | 1.60 | 1.88 | 2.91 |
| Model | Params (M) | RTF | SIG | BAK | OVRL | PESQ |
| Degraded | - | 2.89 | 3.02 | 2.47 | 1.91 | |
| MP-SENet | 2.3 | 0.022 | 3.27 | 3.89 | 2.94 | 1.83 |
| SEMamba | 1.7 | 0.013 | 3.21 | 3.88 | 2.88 | 1.68 |
| USEMamba | 3.9 | 0.045 | 3.36 | 3.99 | 3.06 | 1.73 |
| Universe++ | 42.8 | 0.023 | 2.58 | 3.83 | 2.32 | 1.11 |
| Universe++ ∗ | 42.8 | 0.023 | 3.10 | 3.86 | 2.80 | 1.77 |
| Model | Methods | Params (M) | Training data (hours) | SIG | BAK | OVRL |
| Degraded | - | 3.053 | 2.509 | 2.255 | ||
| ANYENHANCE † | MGM | 363.6 | 3.7k | 3.488 | 3.977 | 3.161 |
| MaskSR-M † | MLM | 145 | 800 | 3.430 | 4.025 | 3.136 |
| LLaSE-G1 † | LM | 1072 | 500 | 3.472 | 3.996 | 3.177 |
| VoiceFixer ∗ | Reg. | 111.7 | 44 | 3.291 | 3.960 | 2.992 |
| USEMamba | Reg. | 3.9 | 44 | 3.239 | 3.937 | 2.923 |
| Model | Degradation types | |||||||
| Noise (dB) | Bandwidth limitation (kHz) | |||||||
| -15 | -10 | 0 | 10 | 1 | 2 | 4 | 6 | |
| Degraded | 1.42 | 1.46 | 1.6 | 2.25 | 1.96 | 2.75 | 3.73 | 3.95 |
| MP-SENet | 1.94 | 2.34 | 3.33 | 3.82 | 1.85 | 2.80 | 3.73 | 3.99 |
| SEMamba | 2.06 | 2.51 | 3.48 | 3.89 | 1.85 | 2.79 | 3.72 | 3.97 |
| USEMamba | 1.98 | 2.38 | 3.35 | 3.83 | 1.52 | 2.89 | 3.85 | 4.04 |
| Method | VCTK-GSR test | URGENT test | DNS test |
| MR-Parallel | 0.036 | 0.036 | 0.037 |
| MR-Sequential | 0.045 | 0.047 | 0.046 |
| Design | Params (M) | RTF | In-domain | Out-of-domain | ||||
| VCTK-GSR test | URGENT test | DNS test | ||||||
| UTMOS | OVRL | UTMOS | OVRL | UTMOS | OVRL | |||
| Mamba | 2.7 | 0.025 | 3.55 | 3.12 | 2.50 | 3.04 | 2.70 | 3.06 |
| Transformer | 2.9 | 0.027 | 2.95 | 2.95 | 2.24 | 2.95 | 2.40 | 3.04 |
| Conformer | 2.4 | 0.028 | 3.45 | 3.06 | 2.46 | 3.04 | 2.93 | 3.17 |
| SpatialNet | 2.6 | 0.024 | 3.46 | 3.10 | 2.36 | 2.98 | 2.68 | 3.11 |
| Design | Params (M) | RTF | In-domain | Out-of-domain | ||||
| VCTK-GSR test | URGENT test | DNS test | ||||||
| UTMOS | OVRL | UTMOS | OVRL | UTMOS | OVRL | |||
| MR-Parallel (Ours) | 2.7 | 0.021 | 3.55 | 3.14 | 2.61 | 3.13 | 3.02 | 3.21 |
| MR-Sequential | 2.7 | 0.021 | 3.54 | 3.12 | 2.53 | 3.08 | 2.91 | 3.21 |
| SR | 9.0 | 0.034 | 3.53 | 3.11 | 2.42 | 2.85 | 2.19 | 2.48 |
| w/o down. | 2.8 | 0.028 | 3.50 | 3.09 | 2.44 | 2.95 | 2.72 | 3.0 |
| Design | Params (M) | RTF | In-domain | Out-of-domain | ||||
| VCTK-GSR test | URGENT test | DNS test | ||||||
| UTMOS | PESQ | UTMOS | OVRL | UTMOS | OVRL | |||
| SEMamba | 1.7 | 0.013 | 2.34 | 2.13 | 1.79 | 2.80 | 1.57 | 2.75 |
| LMask. \mathrel{\mathchoice{\mkern 2.0mu\hbox{\hbox to8.19pt{\vbox to5.03pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{{}{}{}{}\lxSVG@discardpath\lxSVG@discardpath@clipped{M 0 0 L 0 6.96 L 11.33 6.96 L 11.33 0 Z} {{{}{}{{}}{} {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 0 0)} \pgfsys@hbox{59}\lxSVG@closescope }}}} {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}}{\mkern 2.0mu\hbox{\hbox to8.19pt{\vbox to5.03pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{{}{}{}{}\lxSVG@discardpath\lxSVG@discardpath@clipped{M 0 0 L 0 6.96 L 11.33 6.96 L 11.33 0 Z} {{{}{}{{}}{} {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 0 0)} \pgfsys@hbox{59}\lxSVG@closescope }}}} {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}}{\mkern 2.0mu\hbox{\hbox to5.01pt{\vbox to3.5pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{{}{}{}{}\lxSVG@discardpath\lxSVG@discardpath@clipped{M 0 0 L 0 4.84 L 6.93 4.84 L 6.93 0 Z} {{{}{}{{}}{} {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 0 0)} \pgfsys@hbox{59}\lxSVG@closescope }}}} {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}}{\mkern 2.0mu\hbox{\hbox to3.58pt{\vbox to2.5pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{{}{}{}{}\lxSVG@discardpath\lxSVG@discardpath@clipped{M 0 0 L 0 3.46 L 4.95 3.46 L 4.95 0 Z} {{{}{}{{}}{} {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 0 0)} \pgfsys@hbox{59}\lxSVG@closescope }}}} {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}}} LMap. | 1.7 | 0.013 | 2.30 | 2.22 | 1.82 | 2.87 | 1.82 | 2.90 |
| Metric. \mathrel{\mathchoice{\mkern 2.0mu\hbox{\hbox to8.19pt{\vbox to5.03pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{{}{}{}{}\lxSVG@discardpath\lxSVG@discardpath@clipped{M 0 0 L 0 6.96 L 11.33 6.96 L 11.33 0 Z} {{{}{}{{}}{} {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 0 0)} \pgfsys@hbox{59}\lxSVG@closescope }}}} {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}}{\mkern 2.0mu\hbox{\hbox to8.19pt{\vbox to5.03pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{{}{}{}{}\lxSVG@discardpath\lxSVG@discardpath@clipped{M 0 0 L 0 6.96 L 11.33 6.96 L 11.33 0 Z} {{{}{}{{}}{} {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 0 0)} \pgfsys@hbox{59}\lxSVG@closescope }}}} {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}}{\mkern 2.0mu\hbox{\hbox to5.01pt{\vbox to3.5pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{{}{}{}{}\lxSVG@discardpath\lxSVG@discardpath@clipped{M 0 0 L 0 4.84 L 6.93 4.84 L 6.93 0 Z} {{{}{}{{}}{} {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 0 0)} \pgfsys@hbox{59}\lxSVG@closescope }}}} {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}}{\mkern 2.0mu\hbox{\hbox to3.58pt{\vbox to2.5pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{{}{}{}{}\lxSVG@discardpath\lxSVG@discardpath@clipped{M 0 0 L 0 3.46 L 4.95 3.46 L 4.95 0 Z} {{{}{}{{}}{} {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 0 0)} \pgfsys@hbox{59}\lxSVG@closescope }}}} {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}}} Vocoder | 1.7 | 0.013 | 3.32 | 1.71 | 2.28 | 2.91 | 2.14 | 2.88 |
| TFMamba-large | 2.7 | 0.018 | 3.31 | 1.75 | 2.23 | 2.84 | 2.31 | 2.85 |
Speech Enhancement
Speech enhancement improves intelligibility and listening quality by suppressing noise, reverberation, and competing speakers in recorded or live audio.
See all 128 papers