RadarVox: Radar-Audio Multimodal Cocktail-Party Speech Separation with Speaker-Aware Cross-Modal Matching

Yanlin Xu; Yiwei Ru; Mupei Li; Yongji Liu; Jie Wang; Zhenan Sun

eess.ASSep 20, 2026

RadarVox: Radar-Audio Multimodal Cocktail-Party Speech Separation with Speaker-Aware Cross-Modal Matching

Authors: Yanlin Xu, Yiwei Ru, Mupei Li, Yongji Liu, Jie Wang, Zhenan Sun

Organizations: New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences (CASIA), Beijing 100190, China · School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS), Beijing 100049, China · Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · Beijing University of Posts and Telecommunications, Beijing 100876, China ·

Abstract

In embodied voice interaction, cocktail-party speech perception requires both speech separation and speaker attribution across machine-generated and human speech sources. However, conventional audio-only blind source separation remains permutation ambiguous, making the correspondence between separated streams and physical speakers unclear. This paper presents RadarVox, a radar-audio multimodal benchmark for identity-aware cocktail-party speech separation. RadarVox provides acoustic mixtures and source-level radar displacement signals from loudspeaker-emitted and human speech, enabling source-aware speaker assignment. FMCW radar captures laryngeal mechanical motion, providing speaker-specific spatial-motion cues unavailable to a single-channel microphone. We inject radar-derived priors into a DPRNN separator via gated fusion and learn a speaker-aware cross-modal matcher to associate unordered speech streams with radar-observed speakers. Experiments on multi-speaker mixtures show that radar cues provide complementary benefits, achieving scale-invariant signal-to-distortion ratio (SI-SDR) values of 9.75 dB and 7.13 dB in two- and three-speaker scenarios, respectively. More importantly, the proposed method improves ordered SI-SDR by more than 13 dB over audio-only methods while achieving over 80% speaker assignment accuracy.

Figures & tables

Fig. 1: An example application scenario for speaker-aware speech perception. In daily environments with multiple speakers and ambient noise, a robot needs to perceive speaker-specific speech and provide personalized services accordingly.

Dataset	Sensing Modality	Task	Speech Source	Per-Speaker Text Coverage	Dataset Size	Corpus	Public Available
LibriMix [ 23 ]	Audio	Speech Separation	Human	Not applicable	364 h (921 speakers set)	LibriSpeech train-360	✓
Radio2Speech [ 7 ]	Radar	Speech Recovery	Loudspeaker	LJSpeech corpus + TIMIT corpus	27.5 h (loudspeaker playback)	LJSpeech + TIMIT	✗
Wavesdropper [ 8 ]	Radar	Closed-set Word Recognition	Human	57 fixed words	- (23 speakers)	57-word list	✗
WaveEar [ 9 ]	Radar	Speech Recovery	Human	5 shared fixed passages ( $\sim$ 1.5k words), 3 repetitions	$\sim$ 12 h (21 speakers)	5 reading materials	✗
Wavoice [ 10 ]	Radar–Audio	Speech Recognition	Human	40 shared commands,40 repetitions	- (20 speakers)	ok-google.io + Google speech commands	✗
mmMUSE [ 11 ]	Radar–Audio	Speech Enhancement	Human	15 commands + 13 sentences,10 repetitions	15 h (46 speakers)	ok-google.io + TIMIT subsets	✗

TABLE I: Comparison of Existing Speech Datasets related to cocktail-party problem Across Modalities and Tasks

Fig. 2: Overview of the Proposed Radar–Audio Speech Perception System. (a) Radar acquisition of loudspeaker vibrations to simulate agent-agent communication, and acquisition of human laryngeal vibrations to simulate agent-human interaction. (b) The transmitted and received radar chirps are preprocessed to localize the speaker and extract the corresponding laryngeal micro-motion signal. (c) The proposed two-stage model for identity-aware speech separation.

Panel of Figure 4 — Fig. 3: Recording devices and scenarios of the loudspeaker. (a) Radar devices. (b) Recording scenarios of loudspeaker.

Panel of Figure 5 — Fig. 4: Recording scenarios of the speaker. (a) Orientation of radar. (b) Recording scenarios of the speaker.

Parameter	Value
Ramp End Time	83 $\mu$ s
Idle Time	5 $\mu$ s
Frequency Slope	42 MHz/ $\mu$ s
Loop Count (No. of Chirps per Frame)	255
Frame Periodicity	23 ms

TABLE II: Radar System Parameter Configuration

Fig. 5: Schematic of the Radar Preprocessing Pipeline for Loudspeaker Data. The six sequential steps are illustrated from left to right.

Preprocessing Strategy	SI-SDR $\uparrow$	PESQ $\uparrow$	STOI $\uparrow$
EMA Clutter Cancellation [ 39 ]	-38.10	1.027	0.177
VMD-based Denoising [ 47 ]	-56.37	1.367	0.178
Wavelet Shrinkage Denoising [ 47 ]	-42.76	1.033	0.066
Kalman Smoothing [ 51 ]	-41.06	1.020	0.158
Phase-aligned Coherent Accumulation	-37.95	1.040	0.104

TABLE III: Speech reconstruction performance for a loudspeaker under different radar preprocessing strategies.

Sampling Rate (Hz)	SI-SDR (dB)	STOI	PESQ	SNR (dB)	WER
1000	-51.29	0.169	1.036	-3.02	1.00
7000	-32.63	0.048	1.033	-2.76	1.00
11087	-37.95	0.104	1.040	-2.82	0.41

TABLE IV: Performance Under Different Sampling Rates

Fig. 6: Distributions of loudspeaker’s recordings across voice categories and emotions, measured by total duration. (a) Timbre-category distribution. (b) Emotion distribution.

Panel of Figure 11 — Fig. 7: Dataset composition and SNR distributions. (a) Split-wise distribution of original single-speaker recordings and synthesized two-speaker and three-speaker mixtures. (b) SNR distributions under different speaker configurations.

Fig. 8: Overall Framework of the Two-Stage Radar-Audio Speech Separation and Cross-Modal Matching Network. Here, $N$ denotes the number of speakers and $i$ indexes the corresponding speaker.

Modality	Method	SI-SDR $\uparrow$	STOI $\uparrow$	PESQ $\uparrow$
Input (audio)		3.36	0.5983	2.045
Input (radio)		-47.90	0.0968	1.206
Audio-Only	Conv-TasNet	11.79	0.723	2.841
Audio-Only	DPRNN	12.05	0.726	2.776
Radar-Only	WaveEar	-38.56	0.125	1.365
Radar-Only	RadioMic	-50.55	0.037	1.371

TABLE V: Results for single-speaker speech recovery and enhancement

	2-person mix (clean)				2-person mix (noisy)
Model	SI-SDR $\uparrow$	SIR $\uparrow$	STOI $\uparrow$	PESQ $\uparrow$	SI-SDR $\uparrow$	SIR $\uparrow$	STOI $\uparrow$	PESQ $\uparrow$
Input	0.00	0.00	0.585	1.859	-1.90	1.90	0.470	1.612
Conv-TasNet	8.72 / -5.98	20.90 / 6.20	0.705 / 0.735	2.507 / 2.540	5.99 / -7.94	22.08 / 8.16	0.608 / 0.629	2.038 / 2.010
DPRNN	9.16 / -6.44	22.13 / 6.52	0.718 / 0.748	2.542 / 2.611	6.28 / -8.36	23.09 / 8.46	0.622 / 0.644	2.026 / 2.043
RADIOSES	8.75 / 0.60	20.54 / 1.31	0.707 / 0.586	2.478 / 1.890	5.99 / -0.46	21.49 / 2.18	0.614 / 0.501	2.021 / 1.671
RadarVox	9.75 / 7.11	21.69 / 18.98	0.728 / 0.675	2.643 / 2.502	6.84 / 4.51	23.53 / 20.46	0.634 / 0.589	2.110 / 1.985

TABLE VI: Speech separation results for two-speaker and three-speaker mixtures. Values before and after the slash denote PIT and ordered evaluations, respectively.

Fig. 9: Speech quality heatmaps under different window and hop sizes. In the SI-SDR, STOI, and PESQ heatmaps, lighter colors consistently indicate better performance.

Fig. 10: Speech quality evaluation under different FFT size

Spk.	Setting	Clean				Noisy
Spk.	Setting	SI-SDR	SIR	STOI	PESQ	SI-SDR	SIR	STOI	PESQ
2spk	w/o radar	9.16	22.13	0.718	2.542	6.28	23.09	0.622	2.026
2spk	w/ Radar	9.75	21.69	0.728	2.643	6.84	23.53	0.634	2.110
3spk	w/o radar	6.85	20.85	0.655	2.278	5.16	21.97	0.600	2.014
3spk	w/ Radar	7.13	22.15	0.690	2.356	5.49	23.92	0.636	2.070

TABLE VII: Ablation of Stage I radar integration under PIT evaluation.

Spk.	Setting	Clean				Noisy
Spk.	Setting	SI-SDR	SIR	STOI	PESQ	SI-SDR	SIR	STOI	PESQ
2spk	w/o Radar	-6.44	6.52	0.748	2.611	-8.36	8.46	0.644	2.043
2spk	w/ Radar	-0.03	-0.01	0.588	1.899	-0.96	0.00	0.499	1.669
3spk	w/o Radar	-14.69	7.32	0.227	1.440	-16.24	8.12	0.213	1.403
3spk	w/ Radar	-15.27	7.36	0.293	1.614	-17.41	8.39	0.276	1.523

TABLE VIII: Ablation of Stage I radar integration under ordered evaluation.

Spk.	Setting	Clean					Noisy
Spk.	Setting	SI-SDR	SIR	STOI	PESQ	Match Acc.	SI-SDR	SIR	STOI	PESQ	Match Acc.
2spk	w/o Matcher	-0.03	-0.01	0.588	1.899	49.9%	-0.96	0.00	0.499	1.669	49.9%
2spk	w/ Matcher	7.11	18.98	0.675	2.502	88.2%	4.51	20.46	0.589	1.985	88.9%
3spk	w/o Matcher	-15.27	7.36	0.293	1.614	16.3%	-17.41	8.39	0.276	1.523	16.4%
3spk	w/ Matcher	5.43	17.83	0.630	2.278	82.8%	3.97	19.51	0.582	2.012	82.5%

TABLE IX: Ablation of Stage II matcher learning under ordered evaluation.

Speech Separation

Speech separation isolates individual voices from overlapping speech, supporting clearer communication, transcription, and speaker-aware audio processing.

See all 36 papers

From Speech to Interaction: Analyzing Multimodal Systems in Cocktail-Party Scenarios
Aug 9, 2026Thai-Binh Nguyen, Zhaolin Li, Jan Niehues +1Target Speaker Extraction Automatic Speech Recognition
Karlsruhe Institute of Technology Karlsruhe, Germany · Carnegie Mellon University Pittsburgh, USA
SepRQ : Self-Supervised Speech Mixture Representation Learning via Mask-Free, Multi-Scale Source Separation
Oct 3, 2026Séverin Baroudi, Hervé Bredin, Ricard MarxerSelf-Supervised Speech Representation Learning Speech Separation
Univ Toulon, Aix Marseille Univ, CNRS, LIS, Toulon, France · CNRS, ILLS, Montr´eal, Canada · pyannoteAI, Toulouse, France
Position-Aware Target Speaker Extraction for Long-Form Multi-Party Conversations: A Diarization-Free Framework for ASR
Jun 28, 2026Yichi Wang, Junzhe Chen, Wangjin Zhou +1Target Speaker Extraction Speech Processing
Graduate School of Informatics, Kyoto University, Kyoto, Japan

Aug 9, 2026J/K move · Enter open · S save

From Speech to Interaction: Analyzing Multimodal Systems in Cocktail-Party Scenarios

Thai-Binh Nguyen, Zhaolin Li, Jan Niehues, Alexander Waibel

Karlsruhe Institute of Technology Karlsruhe, Germany · Carnegie Mellon University Pittsburgh, USA

Target Speaker Extraction Automatic Speech Recognition Audio-Visual Speech Recognition Speaker Diarization Speech Separation

Open paper AskPDF arXiv

Humans have the remarkable ability to engage in spontaneous informal conversations and selectively attend to individual speakers while filtering out competing speech from nearby conversations. This "cocktail party" scenario still presents severe challenges to speech recognition systems. The CHiME-9 MCoRec task provides a testbed where systems must recognize groups of speakers and transcribe each of their conversations from audio-visual input. In this work, we analyze a diverse set of systems, representing different design directions for addressing the cocktail-party scenario, where the best system achieves up to 57% relative error reduction. We identify three main strategies: (1) explicit or implicit audio-visual target speech separation, (2) improved audio-visual speech recognition for each target speaker, and (3) the use of large language models to group speakers into conversations and enhance conversational consistency. Our analysis shows that these directions address complementary failure modes of the cocktail-party problem, and that high speech overlap alone does not explain performance differences, challenging the common assumption that overlap is the primary source of difficulty in cocktail-party recognition.

Abstract

Speech Separation

Explore similar work

From Speech to Interaction: Analyzing Multimodal Systems in Cocktail-Party Scenarios

SepRQ : Self-Supervised Speech Mixture Representation Learning via Mask-Free, Multi-Scale Source Separation

Position-Aware Target Speaker Extraction for Long-Form Multi-Party Conversations: A Diarization-Free Framework for ASR