Multi-channel speech separation in dynamic environments is challenging as time-varying spatial and spectral features evolve at different temporal scales. Existing methods typically employ sequential architectures, forcing a single network stream to simultaneously model both feature types, creating an inherent modeling conflict. In this paper, we propose a dual-branch parallel spectral-spatial (PS2) architecture that separately processes spectral and spatial features through parallel streams. The spectral branch uses a bi-directional long short-term memory (BLSTM)-based frequency module, a Mamba-based temporal module, and a self-attention module to model spectral features. The spatial branch employs bi-directional gated recurrent unit (BGRU) networks to process spatial features that encode the evolving geometric relationships between sources and microphones. Features from both branches are integrated through a cross-attention fusion mechanism that adaptively weights their contributions. Experimental results demonstrate that the PS2 outperforms existing state-of-the-art (SOTA) methods by 1.6-2.2 dB in scale-invariant signal-to-distortion ratio (SI-SDR) for moving speaker scenarios, with robust separation quality under different reverberation times (RT60), noise levels, and source movement speeds. Even with fast source movements, the proposed model maintains SI-SDR improvements of over 13 dB. These improvements are consistently observed across multiple datasets, including WHAMR! and our generated WSJ0-Demand-6ch-Move dataset.
Figures & tables
Fig. 1: Visualization of temporal evolution in spectral and spatial features for moving versus static sound source scenarios. Top row: magnitude spectrogram (left) and inter-channel time difference (ITD) (right) of a moving speaker ( 0.8 m/s); Bottom row: corresponding visualizations for a static speaker positioned at the initial point of the moving source trajectory. The magnitude spectrograms use the first microphone channel of noise-free reverberant speech (RT 60 : 250 ms) simulated in a 10 m ×10 m ×3 m room with two microphones spaced by 15 cm at 16 kHz sampling rate. The ITD is visualized by computing the time difference between the two channels for each time-frequency bin. For each channel, the complex-valued spectrogram is normalized by its magnitude, yielding a phase-only spectrogram. The inter-channel phase difference is then computed for each bin and transformed into a time delay.
Fig. 2: The illustration of the proposed PS2 system.
Fig. 3: The architecture of the proposed PS2 system.
Fig. 4: The core components of PS2 system: (a) spectral branch, (b) spatial branch, and (c) cross-attention fusion module.
#
Dual
Frequency
Temporal
Self-attention
Spatial branch
Fusion
# Params
FLOPs
SI-SDR
PESQ
eSTOI
branch
module
module
module
activation
method
(M)
(G/s)
(dB)
Unproc.
–
–
–
–
–
–
–
–
-1.5
1.24
0.527
1
✓
BLSTM
Mamba
✓
PReLU
Cross-attention
8.4
66.8
13.4
3.20
0.897
2
✗
BLSTM
Mamba
✓
–
–
3.0
66.2
11.9
2.88
0.859
2+
✗
BLSTM
Mamba
✓
–
–
9.4
181.9
12.1
2.90
0.862
3
✓
BLSTM
Mamba
✓
Tanh
Cross-attention
8.4
66.9
12.4
2.93
0.878
TABLE I: Ablation experiments.
Systems
#Param
Config.
SDR
SI-SDR
SIR
SAR
PESQ
eSTOI
(M)
(dB)
(dB)
(dB)
(dB)
Unproc.
-
Multi
-1.4
-1.5
0.1
7.6
1.24
0.358
Single
-1.5
-1.6
0.1
7.5
1.21
0.347
TF-GridNet 2 [ 15 ]
14.5
Multi
12.1
11.8
23.1
12.6
2.85
0.857
Single
9.2
8.8
19.5
9.8
2.32
0.752
SpatialNet(small) 3 [ 17 ]
1.6
Multi
10.8
10.4
19.7
11.7
2.52
0.815
TABLE II: Results on WSJ0-Demand-6ch-Move dataset under multi-channel and single-channel conditions.
TABLE III: Results on WSJ0-Demand-6ch-Static dataset under multi-channel and single-channel conditions.
Systems
SDR(dB)
SI-SDR(dB)
PESQ
ESTOI
unproc.
-3.5
-6.1
1.41
0.317
Conv-TasNet [ 4 , 46 ]
–
2.2
–
–
BLSTM-TasNet [ 46 ]
–
3.0
–
–
SISO 1 [ 58 ]
6.2
4.2
1.79
0.594
DPRNN [ 6 ]
–
4.2
–
–
DPTNET [ 7 ]
7.6
6.0
–
–
TABLE IV: Results on WHAMR! (1ch).
Systems
SDR(dB)
SI-SDR(dB)
PESQ
ESTOI
unproc.
-3.5
-6.1
1.41
0.317
MC-ConvTasNet [ 11 , 12 ]
–
6.0
–
–
MC-ConvTasNet with speaker extraction [ 12 ]
–
7.3
–
–
TF-GridNet [ 15 ]
14.8
13.7
3.16
0.868
SpatialNet(small) [ 17 ]
13.1
11.8
2.93
0.826
SpatialNet(large) [ 17 ]
15.0
14.1
3.16
0.870
TABLE V: Results on WHAMR! (2ch).
Fig. 5: Multi-conditional evaluation results under different acoustic conditions. SI-SDR performance across varying (a) reverberation times (RT60), (b) input SNR levels, (c) source movement speeds, (d) inter-source angles, and (e) signal durations.