RadarVox: Radar-Audio Multimodal Cocktail-Party Speech Separation with Speaker-Aware Cross-Modal Matching
Authors: Yanlin Xu, Yiwei Ru, Mupei Li, Yongji Liu, Jie Wang, Zhenan Sun
Organizations: New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences (CASIA), Beijing 100190, China · School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS), Beijing 100049, China · Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · Beijing University of Posts and Telecommunications, Beijing 100876, China · School of Information Science and Technology, Dalian Maritime University, Dalian 116026, China
In embodied voice interaction, cocktail-party speech perception requires both speech separation and speaker attribution across machine-generated and human speech sources. However, conventional audio-only blind source separation remains permutation ambiguous, making the correspondence between separated streams and physical speakers unclear. This paper presents RadarVox, a radar-audio multimodal benchmark for identity-aware cocktail-party speech separation. RadarVox provides acoustic mixtures and source-level radar displacement signals from loudspeaker-emitted and human speech, enabling source-aware speaker assignment. FMCW radar captures laryngeal mechanical motion, providing speaker-specific spatial-motion cues unavailable to a single-channel microphone. We inject radar-derived priors into a DPRNN separator via gated fusion and learn a speaker-aware cross-modal matcher to associate unordered speech streams with radar-observed speakers. Experiments on multi-speaker mixtures show that radar cues provide complementary benefits, achieving scale-invariant signal-to-distortion ratio (SI-SDR) values of 9.75 dB and 7.13 dB in two- and three-speaker scenarios, respectively. More importantly, the proposed method improves ordered SI-SDR by more than 13 dB over audio-only methods while achieving over 80% speaker assignment accuracy.
Figures & tables
Fig. 1: An example application scenario for speaker-aware speech perception. In daily environments with multiple speakers and ambient noise, a robot needs to perceive speaker-specific speech and provide personalized services accordingly.
TABLE I: Comparison of Existing Speech Datasets related to cocktail-party problem Across Modalities and Tasks
Fig. 2: Overview of the Proposed Radar–Audio Speech Perception System. (a) Radar acquisition of loudspeaker vibrations to simulate agent-agent communication, and acquisition of human laryngeal vibrations to simulate agent-human interaction. (b) The transmitted and received radar chirps are preprocessed to localize the speaker and extract the corresponding laryngeal micro-motion signal. (c) The proposed two-stage model for identity-aware speech separation.
Fig. 3: Recording devices and scenarios of the loudspeaker. (a) Radar devices. (b) Recording scenarios of loudspeaker.
Fig. 4: Recording scenarios of the speaker. (a) Orientation of radar. (b) Recording scenarios of the speaker.
Parameter
Value
Ramp End Time
83 μ s
Idle Time
5 μ s
Frequency Slope
42 MHz/ μ s
Loop Count (No. of Chirps per Frame)
255
Frame Periodicity
23 ms
TABLE II: Radar System Parameter Configuration
Fig. 5: Schematic of the Radar Preprocessing Pipeline for Loudspeaker Data. The six sequential steps are illustrated from left to right.
Preprocessing Strategy
SI-SDR ↑
PESQ ↑
STOI ↑
EMA Clutter Cancellation [ 39 ]
-38.10
1.027
0.177
VMD-based Denoising [ 47 ]
-56.37
1.367
0.178
Wavelet Shrinkage Denoising [ 47 ]
-42.76
1.033
0.066
Kalman Smoothing [ 51 ]
-41.06
1.020
0.158
Phase-aligned Coherent Accumulation
-37.95
1.040
0.104
TABLE III: Speech reconstruction performance for a loudspeaker under different radar preprocessing strategies.
Sampling Rate (Hz)
SI-SDR (dB)
STOI
PESQ
SNR (dB)
WER
1000
-51.29
0.169
1.036
-3.02
1.00
7000
-32.63
0.048
1.033
-2.76
1.00
11087
-37.95
0.104
1.040
-2.82
0.41
TABLE IV: Performance Under Different Sampling Rates
Fig. 6: Distributions of loudspeaker’s recordings across voice categories and emotions, measured by total duration. (a) Timbre-category distribution. (b) Emotion distribution.
Fig. 7: Dataset composition and SNR distributions. (a) Split-wise distribution of original single-speaker recordings and synthesized two-speaker and three-speaker mixtures. (b) SNR distributions under different speaker configurations.
Fig. 8: Overall Framework of the Two-Stage Radar-Audio Speech Separation and Cross-Modal Matching Network. Here, N denotes the number of speakers and i indexes the corresponding speaker.
Modality
Method
SI-SDR ↑
STOI ↑
PESQ ↑
Input (audio)
3.36
0.5983
2.045
Input (radio)
-47.90
0.0968
1.206
Audio-Only
Conv-TasNet
11.79
0.723
2.841
DPRNN
12.05
0.726
2.776
Radar-Only
WaveEar
-38.56
0.125
1.365
RadioMic
-50.55
0.037
1.371
TABLE V: Results for single-speaker speech recovery and enhancement
2-person mix (clean)
2-person mix (noisy)
Model
SI-SDR ↑
SIR ↑
STOI ↑
PESQ ↑
SI-SDR ↑
SIR ↑
STOI ↑
PESQ ↑
Input
0.00
0.00
0.585
1.859
-1.90
1.90
0.470
1.612
Conv-TasNet
8.72 / -5.98
20.90 / 6.20
0.705 / 0.735
2.507 / 2.540
5.99 / -7.94
22.08 / 8.16
0.608 / 0.629
2.038 / 2.010
DPRNN
9.16 / -6.44
22.13 / 6.52
0.718 / 0.748
2.542 / 2.611
6.28 / -8.36
23.09 / 8.46
0.622 / 0.644
2.026 / 2.043
RADIOSES
8.75 / 0.60
20.54 / 1.31
0.707 / 0.586
2.478 / 1.890
5.99 / -0.46
21.49 / 2.18
0.614 / 0.501
2.021 / 1.671
RadarVox
9.75 / 7.11
21.69 / 18.98
0.728 / 0.675
2.643 / 2.502
6.84 / 4.51
23.53 / 20.46
0.634 / 0.589
2.110 / 1.985
TABLE VI: Speech separation results for two-speaker and three-speaker mixtures. Values before and after the slash denote PIT and ordered evaluations, respectively.
Fig. 9: Speech quality heatmaps under different window and hop sizes. In the SI-SDR, STOI, and PESQ heatmaps, lighter colors consistently indicate better performance.
Fig. 10: Speech quality evaluation under different FFT size
Spk.
Setting
Clean
Noisy
SI-SDR
SIR
STOI
PESQ
SI-SDR
SIR
STOI
PESQ
2spk
w/o radar
9.16
22.13
0.718
2.542
6.28
23.09
0.622
2.026
w/ Radar
9.75
21.69
0.728
2.643
6.84
23.53
0.634
2.110
3spk
w/o radar
6.85
20.85
0.655
2.278
5.16
21.97
0.600
2.014
w/ Radar
7.13
22.15
0.690
2.356
5.49
23.92
0.636
2.070
TABLE VII: Ablation of Stage I radar integration under PIT evaluation.
Spk.
Setting
Clean
Noisy
SI-SDR
SIR
STOI
PESQ
SI-SDR
SIR
STOI
PESQ
2spk
w/o Radar
-6.44
6.52
0.748
2.611
-8.36
8.46
0.644
2.043
w/ Radar
-0.03
-0.01
0.588
1.899
-0.96
0.00
0.499
1.669
3spk
w/o Radar
-14.69
7.32
0.227
1.440
-16.24
8.12
0.213
1.403
w/ Radar
-15.27
7.36
0.293
1.614
-17.41
8.39
0.276
1.523
TABLE VIII: Ablation of Stage I radar integration under ordered evaluation.
Spk.
Setting
Clean
Noisy
SI-SDR
SIR
STOI
PESQ
Match Acc.
SI-SDR
SIR
STOI
PESQ
Match Acc.
2spk
w/o Matcher
-0.03
-0.01
0.588
1.899
49.9%
-0.96
0.00
0.499
1.669
49.9%
w/ Matcher
7.11
18.98
0.675
2.502
88.2%
4.51
20.46
0.589
1.985
88.9%
3spk
w/o Matcher
-15.27
7.36
0.293
1.614
16.3%
-17.41
8.39
0.276
1.523
16.4%
w/ Matcher
5.43
17.83
0.630
2.278
82.8%
3.97
19.51
0.582
2.012
82.5%
TABLE IX: Ablation of Stage II matcher learning under ordered evaluation.
Humans have the remarkable ability to engage in spontaneous informal conversations and selectively attend to individual speakers while filtering out competing speech from nearby conversations. This "cocktail party" scenario still presents severe challenges to speech recognition systems. The CHiME-9 MCoRec task provides a testbed where systems must recognize groups of speakers and transcribe each of their conversations from audio-visual input. In this work, we analyze a diverse set of systems, representing different design directions for addressing the cocktail-party scenario, where the best system achieves up to 57% relative error reduction. We identify three main strategies: (1) explicit or implicit audio-visual target speech separation, (2) improved audio-visual speech recognition for each target speaker, and (3) the use of large language models to group speakers into conversations and enhance conversational consistency. Our analysis shows that these directions address complementary failure modes of the cocktail-party problem, and that high speech overlap alone does not explain performance differences, challenging the common assumption that overlap is the primary source of difficulty in cocktail-party recognition.
Thai-Binh Nguyen, Zhaolin Li, Jan Niehues +1
Karlsruhe Institute of Technology Karlsruhe, Germany · Carnegie Mellon University Pittsburgh, USA
Self-supervised learning (SSL) is standard for speech representation learning, but mainstream models are designed around single-speaker audio, limiting their usefulness in multi-speakers scenarios. We present SepRQ, an open-source SSL framework that replaces masked prediction with a pseudo-source-separation objective over frozen random-projection codebooks. By adopting a novel mask-free, multiresolution approach, SepRQ achieves state-of-the-art performance in Speaker Diarization and Speech Separation on the SUPERB benchmark, surpassing WavLM and other cocktail-party derived SSLs at both Base and Large scales, while requiring only 85.68M inference parameters. SepRQ also demonstrates strong performance across target-speaker tasks requiring enrollment (such as Target-Speaker Automatic Speech Recognition), and on the challenging multi-domain DIHARD 3 diarization dataset. Notably, we report strong separation capabilities on three-speaker mixtures (WSJ0-3Mix), where current SSL literature struggles. While cocktail-party SSLs remain scarce and closed-source, limited to C-HuBERT and the enrollment-based SA-WavLM, we open-source SepRQ to the community.
Séverin Baroudi, Hervé Bredin, Ricard Marxer
Univ Toulon, Aix Marseille Univ, CNRS, LIS, Toulon, France · CNRS, ILLS, Montr´eal, Canada · pyannoteAI, Toulouse, France
In long-form multi-party conversations, highly imbalanced speaker activity and frequent overlap make it difficult to identify "who spoke when and what". Sliding-window continuous speech separation (CSS) mitigates sparse supervision, but often suffers from cross-window speaker inconsistency and residual crosstalk, which in practice requires diarization for reliable speaker attribution. Motivated by the stability of speakers' directions of arrival (DOAs) in meetings, we propose PATSE, a multi-channel Position-Aware Target Speaker Extraction front-end that uses DOA as a spatial prior to directly extract the speech of each target speaker. PATSE combines a DOA-guided spatial encoder and conditioner to generate speaker-attributed streams, from which speaker activity can be inferred via simple post-processing (e.g., VAD) without explicit diarization. Experiments on both replayed and real conversations show consistent ASR gains outperforming CSS and diarization-based pipelines.
Yichi Wang, Junzhe Chen, Wangjin Zhou +1
Graduate School of Informatics, Kyoto University, Kyoto, Japan