Organizations: Google Mountain View, CA, USA · University of Michigan Ann Arbor, MI, USA · Columbia University New York, NY, USA · Google San Francisco, CA, USA
In Extended Reality (XR), complex acoustic environments often overwhelm users, compromising both scene awareness and social engagement due to entangled sound sources. We introduce MoXaRt, a real-time XR system that uses audio-visual cues to separate these sources and enable fine-grained sound interaction. MoXaRt's core is a cascaded architecture that performs coarse, audio-only separation in parallel with visual detection of sources (e.g., faces, instruments). These visual anchors then guide refinement networks to isolate individual sources, separating complex mixes of up to 5 concurrent sources (e.g., 2 voices + 3 instruments) with ~2 second processing latency. We validate MoXaRt through a technical evaluation on a new dataset of 30 one-minute recordings featuring concurrent speech and music, and a 22-participant user study. Empirical results indicate that our system significantly enhances speech intelligibility, yielding a 36.2% (p < 0.01) increase in listening comprehension within adversarial acoustic environments while substantially reducing cognitive load (p < 0.001), thereby paving the way for more perceptive and socially adept XR experiences.
Figures & tables
Figure 1. MoXaRt is a real-time XR system that (a) uses audio-visual cues to separate sound sources from a single audio channel and (b) enables fine-grained, interactive control over the user’s soundscape.
Figure 2. The user interface of MoXaRt and walkthrough of its internal implementation. (a) The user can identify separated sound sources through visualizations and adjust the volume of each source for remixing. (b) Video captured from the Quest 3 headset is streamed to the PC for model inference, and the remixed audio is streamed back. (c) To ensure continuous playback of remixed audio, MoXaRt operates in a three-stage pipeline: Video and Audio Capture, Model Inference, and Playback.
Figure 3. The cascaded architecture of MoXaRt for multi-modal sound separation. The system pipeline consists of two main stages: First, a Coarse Sound Separation module processes the raw audio inputs to produce initial coarse speech and music mixtures. Next, these mixtures, along with outputs from the Face Detection Network and Instrument Detection Network , are fed into two parallel refinement branches: Speech Refinement and Music Refinement . The Speech Refinement branch further disambiguates the coarse speech into individual speaker tracks and their corresponding spatial coordinates. The Music Refinement branch utilizes an ensemble of Band Split Roformer (BSR) models, dynamically activated by the detected instruments, to output isolated instrument tracks. This structured approach enables precise source separation by leveraging visual context only when specialized refinement is required.
Figure 4. Interactive Music Experiences. MoXaRt enables users to actively control their listening experience during live music events. (a) The user can act as a real-time audio engineer, selectively adjusting the gain of individual instruments, such as the violin, cello, and piano in a trio—to create a personalized mix. (b) In a concert setting, the user can enhance acoustic focus by isolating the musicians’ performance from distracting audience chatter.
Figure 5. Enhanced Social Interaction. MoXaRt facilitates communication in complex and noisy social settings. (a) In an environment with multiple simultaneous conversations, a user can selectively amplify a target conversation to maintain focus. (b) The system can improve speech intelligibility in chaotic environments by isolating a conversation partner’s voice from loud background noise, such as live music.
Figure 6. Downstream AI Assistance. The separated streams from MoXaRt can serve as clean, audio-visually aligned input for LLM assistance ( e.g., Gemini, Chatgpt.) (a) The system can isolate individual speakers in a multi-lingual environment, enabling real-time, multi-person transcription and translation. (b) The system’s output could be used to create a queryable log of interactions, allowing a user to ask questions about past events ( e.g., “What did Peter say about this evening?”) and receive a contextually grounded response.
Reference System
Architecture
Target Domain
Object Selection
Max Sources
Real-Time
Sound of Pixels
End-to-End
Music Only
✓
2
✓
DAVIS
End-to-End
Universal †
✗
3
✗
AudioScopeV2
Cascaded
Universal
✗
4
✓
AV-MossFormer2
End-to-End
Speech Only
✗
3
✗
MoXaRt (Offline)
Cascaded
Universal
✓
8
✗
MoXaRt (Real-Time)
Cascaded
Universal
✓
8
✓
Table 1. Comparison of MoXaRt against baselines and audio-visual sound separation benchmarks.
Figure 7. Data collection setup. (a) A Meta Quest 3 with two attached Rode Wireless GO II microphones captures the input mono-audio, while a stationary iPhone records a third-person view of the scene. (b) The three sound sources (two instruments and one speaker), each close-mic’d to record their individual ground-truth audio signal.
Method
WER † ↓
DNSMOS-SIG ↑
DNSMOS-BAK ↑
DNSMOS-OVRL ↑
Raw Audio
0.6297 ( ± 0.4371)
3.7740 ( ± 0.1977)
2.4796 ( ± 0.3928)
2.8944 ( ± 0.1673)
Sound of Pixels
0.8636 ( ± 0.1787)
3.3212 ( ± 0.2467)
2.9535 ( ± 0.2935)
2.7291 ( ± 0.2004)
DAVIS
0.7259 ( ± 0.2484)
3.3212 ( ± 0.2467)
2.9535 ( ± 0.2935)
2.7291 ( ± 0.2004)
AudioScopeV2
0.5263 ( ± 0.3358)
3.5027 ( ± 0.5029)
3.0236 ( ± 0.4497)
2.9349 ( ± 0.3074)
AV-Mossformer2 †
0.3956 ( ± 0.2847)
3.7245 ( ± 0.2813)
3.2564 ( ± 0.4728)
3.0759 ( ± 0.2892)
MoXaRt Offline
0.3824 ( ± 0.2775)
3.6625 ( ± 0.3059)
3.019 ( ± 0.6078)
2.9949 ( ± 0.2585)
Table 2. Objective evaluation of separation accuracy and quality.
Figure 8. User study results. (a) Average Listening Comprehension Performance per scenario and the overall average. MoXaRt led to a statistically significant 36.2% improvement in scores compared to the baseline (p = 0.0058). (b) Average subjective ratings (on a 10-point scale, higher scores indicate better experience) per dimension for the five core experiential dimensions and per scenario and the overall average. Participants rated their experience significantly higher with MoXaRt compared to the baseline (all p < 0.001). Error bars on the average represent the standard deviation.
Figure 9. Model Performance vs. Real-time Analysis. (a) The size of the context window for model input ( window_size ) determines the trade-off between WER and inference time. (b) Reducing chunk_size can optimize interactive latency to as low as 0.82s, at the expense of audio separation quality.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 10. Data collection setup: A Meta Quest 3 with 2 attached Rode Wireless GO II microphones captures the mono audio, while a stationary iPhone records a third-person view of the scene.
In embodied voice interaction, cocktail-party speech perception requires both speech separation and speaker attribution across machine-generated and human speech sources. However, conventional audio-only blind source separation remains permutation ambiguous, making the correspondence between separated streams and physical speakers unclear. This paper presents RadarVox, a radar-audio multimodal benchmark for identity-aware cocktail-party speech separation. RadarVox provides acoustic mixtures and source-level radar displacement signals from loudspeaker-emitted and human speech, enabling source-aware speaker assignment. FMCW radar captures laryngeal mechanical motion, providing speaker-specific spatial-motion cues unavailable to a single-channel microphone. We inject radar-derived priors into a DPRNN separator via gated fusion and learn a speaker-aware cross-modal matcher to associate unordered speech streams with radar-observed speakers. Experiments on multi-speaker mixtures show that radar cues provide complementary benefits, achieving scale-invariant signal-to-distortion ratio (SI-SDR) values of 9.75 dB and 7.13 dB in two- and three-speaker scenarios, respectively. More importantly, the proposed method improves ordered SI-SDR by more than 13 dB over audio-only methods while achieving over 80% speaker assignment accuracy.
Yanlin Xu, Yiwei Ru, Mupei Li +3
New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences (CASIA), Beijing 100190, China · School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS), Beijing 100049, China · Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China +2
Hearables are becoming ubiquitous, yet their sound controls remain blunt: users can either enable global noise suppression or focus on a single target sound. Real-world acoustic scenes, however, contain many simultaneous sources that users may want to adjust independently. We introduce Aurchestra, the first system to provide fine-grained, real-time soundscape control on resource-constrained hearables. Our system has two key components: (1) a dynamic interface that surfaces only active sound classes and (2) a real-time, on-device multi-output extraction network that generates separate streams for each selected class, achieving robust performance for upto 5 overlapping target sounds, and letting users mix their environment by customizing per-class volumes, much like an audio engineer mixes tracks. We optimize the model architecture for multiple compute-limited platforms and demonstrate real-time performance on 6 ms streaming audio chunks. Across real-world environments in previously unseen indoor and outdoor scenarios, our system enables expressive per-class sound control and achieves substantial improvements in target-class enhancement and interference suppression. Our results show that the world need not be heard as a single, undifferentiated stream: with Aurchestra, the soundscape becomes truly programmable.
Seunghyun Oh, Malek Itani, Aseem Gauri +1
Paul G. Allen School of Computer Science and Engineering, University of Washington · Hearvana AI
Target speech extraction (TSE) aims to recover a target speaker's voice from a mixture. While recent text-prompted approaches have shown promise, most approaches assume fully overlapped mixtures, limiting insight into behavior across realistic overlap ratios. We introduce VorTEX (Various overlap ratio for Target speech EXtraction), a text-prompted TSE architecture with a Decoupled Adaptive Multi-branch (DAM) Fusion block that separates primary extraction from auxiliary regularization pathways. To enable controlled analysis, we construct PORTE, a two-speaker dataset spanning overlap ratios from 0% to 100%. We further propose Suppression Ratio on Energy (SuRE), a diagnostic metric that detects suppression behavior not captured by conventional measures. Experiments show that existing models exhibit suppression or residual interference under overlap, whereas VorTEX achieves the highest separation fidelity across 20-100% overlap (e.g., 5.50 dB at 20% and 2.04 dB at 100%) while maintaining zero SuRE, indicating robust extraction without suppression-driven artifacts.
Ro-hoon Oh, Jihwan Seol, Bugeun Kim
Department of Artificial Intelligence, Chung-Ang University, Republic of Korea