Asymmetric Encoder-Decoder Based on Time-Frequency Correlation for Speech Separation

Ui-Hyeop Shin; Hyung-Min Park

eess.ASMar 31, 2026

Asymmetric Encoder-Decoder Based on Time-Frequency Correlation for Speech Separation

Authors: Ui-Hyeop Shin, Hyung-Min Park

Organizations: The authors are with the Department of Electronic Engineering, Sogang University, Seoul 04107, Republic of Korea

Abstract

Speech separation in realistic acoustic environments remains challenging because overlapping speakers, background noise, and reverberation must be resolved simultaneously. Although recent time-frequency (TF) domain models have shown strong performance, most still rely on late-split architectures, where speaker disentanglement is deferred to the final stage, creating an information bottleneck and weakening discriminability under adverse conditions. To address this issue, we propose SR-CorrNet, an asymmetric encoder-decoder framework that introduces the separation-reconstruction (SepRe) strategy into a TF dual-path backbone. The encoder performs coarse separation from mixture observations, while the weight-shared decoder progressively reconstructs speaker-discriminative features with cross-speaker interaction, enabling stage-wise refinement. To complement this architecture, we formulate speech separation as a structured correlation-to-filter problem: spatio-spectro-temporal correlations computed from the observations are used as input features, and the corresponding deep filters are estimated to recover target signals. We further incorporate an attractor-based dynamic split module to adapt the number of output streams to the actual speaker configuration. Experimental results on WSJ0-{2,3,4,5}Mix, WHAMR!, and LibriCSS demonstrate consistent improvements across anechoic, noisy-reverberant, and real-recorded conditions in both single- and multi-channel settings, highlighting the effectiveness of TF-domain SepRe with correlation-based filter estimation for speech separation.

Figures & tables

Panel of Figure 1 — Fig. 1: Block diagrams of (a) late-split and (b) early-split schemes.

Fig. 2: Illustration of two-stage multi-channel separation structure.

Fig. 3: Architecture of SR-CorrNet. The multi-band and multi-frame vectors of multi-channel observation $\tilde{\mathbf{x}}_{tf}\in\mathbb{R}^{M(2L+1)(2I+1)}$ is used for the computation of correlation and output filtering. From $\tilde{\mathbf{x}}_{tf}$ , the correlation module computes a latent representation of correlations $\mathbf{E}_{tf}^{(0)}\in\mathbb{R}^{C}$ as an input for TF-Encoder. Then, $B_{E}$ stacks of TF-Encoder process the features to $\mathbf{E}_{tf}^{(B_{E})}$ and split into speaker-specific features $\mathbf{D}_{k,tf}^{(0)}$ for the TF-Decoder. In the TF-Decoder, the separated features are reconstructed $B_{D}$ times and the reconstructed features $\mathbf{D}_{k,tf}^{(B_{D})}$ are used to predict a final multi-channel multi-tap filter $\mathbf{w}_{k,tf}$ for $Y_{k,tf}$ .

Panel of Figure 4 — Fig. 4: Block diagrams of (a) correlation and (b) filter modules. In the correlation module, the correlation operator performs correlation operation of ( 8 ) with normalization by ( 9 ) or ( 10 ). Then, the complex correlations are flattend to the real-valued vector of $\mathbb{R}^{2M(2L+1)(2I+1)}$ .

Panel of Figure 5 — Fig. 5: Block diagrams of (a) common unit module for time and frequency modules and (b) speaker interaction module.

Fig. 6: Block diagram of the attractor-based dynamic split module. The encoder feature is added to added 2-d positional embeddings and flattened, and provided as key-value input to the attractor decoder. The decoder consists of stacked Transformer decoder blocks with learnable query vectors $\mathbf{q}_{k}$ . The attractors $\mathbf{a}_{k}$ are used to estimate speaker existence probabilities $p_{k}$ .

Fig. 7: Illustration of CSS scheme. In our experiment, we set the chunk-size of $V_{h}=1.2\ s,V=0.8\ s$ , and $V_{f}=0.4\ s$ , respectively.

System	Params. (M)	MACs (G)	SI-SNRi (dB)	SDRi (dB)
Conv-TasNet [ 2 ]	5.1	10.5	15.3	15.6
SuDoRM-RF [ 65 ]	6.4	10.1	18.9	-
DPRNN [ 3 ]	2.6	88.5	18.8	19.0
DPTNet [ 66 ]	2.7	102.5	20.2	20.3
Sepformer [ 4 ]	26.0	86.9	20.4	20.5
A-FRCNN [ 67 ]	6.1	125.0	18.3	18.6

TABLE I: Evaluation on WSJ0-2mix dataset.

System	Split	Prior	$K$
System	Type	$K$	2	3	4	5
Sepformer [ 4 ]	F	Yes	20.4	17.6	-	-
WaveSplit [ 57 ]	F	Yes	21.0	17.3	-	-
Mossformer(L) [ 71 ]	F	Yes	22.8	21.2	-	-
SepEDA 2/3 [ 72 ]	D	Yes	21.5	19.9	-	-
SepTDA 2/3 [ 45 ]	D	Yes	24.0	23.7	-	-

TABLE II: Evaluation of SI-SNRi (dB) results on WSJ0-

K

mix dataset. “Prior

K

” denotes whether the number of speakers in the mixture is given in advance

System	Domain	SI-SNRi (dB)	SDRi (dB)
WaveSplit + DM † [ 57 ]	Time	13.2	12.2
SuDoRM-RF + DM [ 65 ]	Time	13.5	-
Sepformer + DM [ 74 ]	Time	14.0	13.0
QDPN + DM [ 70 ]	Time	14.4	-
Mossformer2 + DM [ 75 ]	Time	17.0	-
SepReformer-L + DM [ 31 ]	Time	17.2	16.0

TABLE III: Evaluation on WHAMR! (1ch)

System	Domain	SI-SNRi (dB)	SDRi (dB)
MC-TasNet [ 76 ]	Time	12.1	-
MC-TasNet with spk. extraction [ 77 ]	Time	13.4	-
TF-GridNet (DNN 1 ) [ 29 ]	TF	18.6	17.0
TF-GridNet (DNN 1 +WF+DNN 2 ) [ 29 ]	TF	19.8	16.6
SpatialNet [ 30 ]	TF	20.2	18.5
SR-CorrNet-B	TF	21.8	20.1

TABLE IV: Evaluation on WHAMR! (2ch)

Enc-Dec Depth	Input-Output	Context Length	SI-SNRi (dB)
( $B_{E}$ , $B_{D}$ )	Configuration	( $\tilde{L},\tilde{I}$ )	1ch	2ch
SepRe (2,4)	Corr.-Filtering	(7,3)	19.7	21.8
SepRe (3,3)	Corr.-Filtering	(7,3)	19.4	21.5
Enc.-only (6,0)	Corr.-Filtering	(7,3)	18.3	20.2
Enc.-only (6,0)	Corr.-Filtering	(3,3)	18.1	20.1
Enc.-only (6,0)	Corr.-Filtering	(7,1)	18.2	20.1

TABLE V: Ablation studies of SR-CorrNet on WHAMR!

Method	Stage configuration	Utterance-wise						Continuous
Method	Stage configuration	0S	0L	10	20	30	40	0S	0L	10	20	30	40
No Processing	-	11.8	11.7	18.8	27.2	35.6	43.3	15.4	11.5	21.7	27.0	34.3	40.5
Oracle Sound	-	4.9	5.1	-	-	-	-	-	-	-	-	-	-
Single-channel Evaluation
BLSTM [ 51 ]	SISO	12.7	12.1	17.6	23.2	30.5	35.6	17.6	16.3	20.9	26.1	32.6	36.1
Conformer [ 15 ]	SISO	12.9	12.2	15.1	20.1	24.3	27.6	13.3	11.7	16.3	20.7	25.6	29.3

TABLE VI: WER(%)

\downarrow

results for utterance-wise and continuous input evaluation on the LibriCSS. SC and LOC denote speaker counting [ 27 ] and localization for stream merging [ 44 ] .

System	#Params. (M)	MACs (G/s)	RTF
MIMO (TF-GridNet)	5.6	171.8	0.872
MIMO (TF-CorrNet)	5.1	44.5	0.234
MIMO-BF-MISO (TF-GridNet)	11.2	512.2	1.688
MIMO-BF-MISO (TF-CorrNet)	10.3	131.8	0.517
MISO (SR-CorrNet- fix 2 )	7.4	170.4	0.315

TABLE VII: Comparison of Computational resources based Real-time factor(RTF) and MACs. RTFs are computed based on average processing times of 2.4 s (chunk-size

V_{h}+V+V_{f}

) divided by 0.8 s(

=V

) on GeForce RTX 4090.

Speech Separation

Speech separation isolates individual voices from overlapping speech, supporting clearer communication, transcription, and speaker-aware audio processing.

See all 36 papers

TF-MossFormer: Integrating Convolution Gated Local-Global Attentions for Enhanced Time-Frequency Domain Monaural Speech Separation
Jul 23, 2026Shengkui Zhao, Zexu Pan, Haoxu Wang +3Hybrid CNN-Transformer Architectures Frequency-Domain Feature Learning
Token Foundry, Alibaba Group, Singapore
Cross-Talk Speech Reduction, by Separation, for Separation
May 19, 2026Zhong-Qiu Wang, Samuele CornellSpeech Processing Speech Separation
Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen 518055, China · Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA
SLM-SS: Token-Based Speech Separation with Speech Language Models
Oct 20, 2026Chenda Li, Tianhua Li, Yiming Chen +4Speech Separation Speech Language Models
Auditory Cognition and Computational Acoustics Lab, Shanghai Jiao Tong University, Shanghai, 200240, P. R. China

Jul 23, 2026J/K move · Enter open · S save

TF-MossFormer: Integrating Convolution Gated Local-Global Attentions for Enhanced Time-Frequency Domain Monaural Speech Separation

Shengkui Zhao, Zexu Pan, Haoxu Wang, Biao Tian+2

Token Foundry, Alibaba Group, Singapore

Hybrid CNN-Transformer Architectures Frequency-Domain Feature Learning Transformer Attention Audio Source Separation Speech Separation

Open paper AskPDF arXiv

Transformers with global attention capture long-range dependencies but can miss the fine-grained local continuity crucial for speech separation. We propose TF-MossFormer, a time-frequency transformer that combines local and global attention to jointly model short- and long-range contexts for monaural speech separation. At its core is a content-aware sliding-window attention mechanism that dynamically adapts receptive fields for stronger local interactions, avoiding the rigidity of static convolutions. Unlike time-domain chunk-based methods, TF-MossFormer leverages the 2D spectrogram to model structure along both time and frequency axes. Convolutional gating between attention layers further improves feature selection and information flow. TF-MossFormer achieves SI-SDRi of 22.6, 24.0, and 24.4 dB on WSJ0-2Mix with 5.9M, 16.9M, and 25.4M parameters, respectively, outperforming prior approaches.

Abstract

Speech Separation

Explore similar work

TF-MossFormer: Integrating Convolution Gated Local-Global Attentions for Enhanced Time-Frequency Domain Monaural Speech Separation

Cross-Talk Speech Reduction, by Separation, for Separation

SLM-SS: Token-Based Speech Separation with Speech Language Models