Regional target speaker extraction (RSE) aims to extract speech originating from a specified angular region, relaxing the requirement for precise target speaker location when performing location based speech extraction. However, existing RSE methods typically assume a fixed microphone array configuration and microphone number, limiting their applicability to unseen microphone geometries and varying numbers of microphones. In this work, we propose a geometry-aware RSE framework capable of extract with varying microphone array geometry and microphone numbers. Specifically, we introduce an attention-based multichannel front-end to encode inter-microphone phase information, together with a spatial audio reconstruction-based auxiliary objective to improve the front-end spatial representations quality. Experimental results demonstrate that the proposed framework improves SI-SDRi by 5.1 dB over geometry-aware baselines across microphone arrays with four-microphones, while maintaining a performance gap of less than 0.6 dB compared with a tetrahedral geometry-specific baseline when evaluated on the tetrahedral microphone array.
Figures & tables
Model
Microphone
0 Target
1 Target
2 Target
Geometry
Decay
SI-SDR
SI-SDRi
PESQ
STOI
SI-SDR
SI-SDRi
PESQ
STOI
Geometry Specific Models
DSENet Tetra
Dynamic-4
-23.64
-32.61
-26.83
1.09
0.31
-32.56
-37.10
1.08
0.42
Tetrahedral
-31.64
5.03
11.36
1.77
0.75
10.09
5.63
2.40
0.84
DSENet NAO
Dynamic-12
-23.45
-33.10
-26.86
1.11
0.30
-30.56
-35.03
1.12
0.40
NAO Robot
-27.55
6.04
12.31
1.85
0.78
10.58
6.08
2.47
0.86
Table 1: Model Performance in Decay ↓ , SI-SDR ↑ , SI-SDRi ↑ , PESQ ↑ , STOI ↑ on Simulated Data across different Microphone Array Configuration and Target Speaker Number.
Model
Tetrahedral
12 Mic NAO Robot
simulated
real
simulated
real
DSENet
11.36
8.40
12.31
9.57
DPARNet-RSE
3.52
5.19
3.84
6.11
LCMV
0.25
1.12
-0.03
-2.58
FoVNet
4.83
3.85
5.21
4.96
Ours
10.79
9.15
11.97
9.08
Table 2: Model Performance comparison between Simulated and Real Data under single target speaker scenario.
Figure 1: Balanced Accuracy under different microphone aparture scaling and target speaker angular width. Dashed lines indicate the Balanced Accuracy of the geometry specific DSENet model at each target speaker angular width.
Figure 2: Visualization of model extraction performance before and after scaling the microphone array aperture size.
We present a multi-channel target speaker extraction (TSE) framework that is guided by relative spatial cues rather than explicit speaker-dependent references, such as clean enrollment utterances or precise directions of arrival (DOA). The proposed approach encodes the target speaker through discrete, interpretable relations between concurrent speakers, such as left and right or near and far, referred to as the Relative Position Cue (RPC). This relational conditioning effectively leverages spatial information while avoiding explicit DOA estimation, thereby enabling more robust operation under realistic conditions. We evaluate the proposed method across multiple datasets encompassing diverse acoustic conditions, including directional and diffuse babble noise, varying numbers of microphones, mixtures with different numbers of speakers, and dynamic scenarios involving moving speakers. Experimental results demonstrate that the RPC-based approach consistently achieves strong extraction performance and generalizes well across all tested configurations, establishing it as a practical and interpretable alternative for spatially informed speaker extraction.
Multi-channel speech enhancement (SE) systems exhibit superior performance over single-channel methods but are constrained to fixed microphone array configurations. This restricts their real-world deployment across devices with diverse array geometries. While recent array-agnostic SE methods address variable microphone numbers and permutations, they largely fail to exploit explicit array geometry priors when available, missing a crucial cue for optimal spatial filtering. A Geometry-Aware Dynamic Convolution (Geo-DConv) framework is proposed, which explicitly leverages microphone coordinates to transform standard fixed-array SE models into robust array-invariant systems. Experiments are conducted on the recent real-recorded RealMAN multi-channel speech dataset. Results demonstrate that the proposed architecture enables two widely used fixed-array models to adapt to array-invariant settings, with consistent performance improvements across diverse array topologies.
Zhenglong Liu, Wangyou Zhang, Chenda Li +1
Auditory Cognition and Computational Acoustics Lab Shanghai Jiao Tong University, Shanghai, China · VUI Labs
Target speaker extraction (TSE) aims to recover a target speaker's speech from a mixture using a reference utterance as a cue. Most TSE systems adopt conditional auto-encoder architectures with one-step inference. Inspired by test-time scaling, we propose a training-free multi-step inference method that enables iterative refinement with a frozen pretrained model. At each step, new candidates are generated by interpolating the original mixture and the previous estimate, and the best candidate is selected for further refinement until convergence. Experiments show that, when ground-truth target speech is available, optimizing an intrusive metric (SI-SDRi) yields consistent gains across multiple evaluation metrics. Without ground truth, optimizing non-intrusive metrics (UTMOS or SpkSim) improves the corresponding metric but may hurt others. We therefore introduce joint metric optimization to balance these objectives, enabling controllable extraction preferences for practical deployment.
Zhenghai You, Ying Shi, Lantian Li +1
Beijing University of Posts and Telecommunications, China · Center for Speech and Language Technologies, BNRist, Tsinghua University, China