cs.SDMar 11, 2026

MoXaRt: Audio-Visual Object-Guided Sound Interaction for XR

Authors: Tianyu Xu, Sieun Kim, Qianhui Zheng, Ruoyu Xu, Tejasvi Ravi, Anuva Kulkarni, Katrina Passarella-Ward, Junyi Zhu, 

Organizations: Google Mountain View, CA, USA · University of Michigan Ann Arbor, MI, USA · Columbia University New York, NY, USA · Google San Francisco, CA, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

In Extended Reality (XR), complex acoustic environments often overwhelm users, compromising both scene awareness and social engagement due to entangled sound sources. We introduce MoXaRt, a real-time XR system that uses audio-visual cues to separate these sources and enable fine-grained sound interaction. MoXaRt's core is a cascaded architecture that performs coarse, audio-only separation in parallel with visual detection of sources (e.g., faces, instruments). These visual anchors then guide refinement networks to isolate individual sources, separating complex mixes of up to 5 concurrent sources (e.g., 2 voices + 3 instruments) with ~2 second processing latency. We validate MoXaRt through a technical evaluation on a new dataset of 30 one-minute recordings featuring concurrent speech and music, and a 22-participant user study. Empirical results indicate that our system significantly enhances speech intelligibility, yielding a 36.2% (p < 0.01) increase in listening comprehension within adversarial acoustic environments while substantially reducing cognitive load (p < 0.001), thereby paving the way for more perceptive and socially adept XR experiences.

Figures & tables

Figure 1. MoXaRt is a real-time XR system that (a) uses audio-visual cues to separate sound sources from a single audio channel and (b) enables fine-grained, interactive control over the user’s soundscape.
Figure 2. The user interface of MoXaRt and walkthrough of its internal implementation. (a) The user can identify separated sound sources through visualizations and adjust the volume of each source for remixing. (b) Video captured from the Quest 3 headset is streamed to the PC for model inference, and the remixed audio is streamed back. (c) To ensure continuous playback of remixed audio, MoXaRt operates in a three-stage pipeline: Video and Audio Capture, Model Inference, and Playback.
Figure 3. The cascaded architecture of MoXaRt for multi-modal sound separation. The system pipeline consists of two main stages: First, a Coarse Sound Separation module processes the raw audio inputs to produce initial coarse speech and music mixtures. Next, these mixtures, along with outputs from the Face Detection Network and Instrument Detection Network , are fed into two parallel refinement branches: Speech Refinement and Music Refinement . The Speech Refinement branch further disambiguates the coarse speech into individual speaker tracks and their corresponding spatial coordinates. The Music Refinement branch utilizes an ensemble of Band Split Roformer (BSR) models, dynamically activated by the detected instruments, to output isolated instrument tracks. This structured approach enables precise source separation by leveraging visual context only when specialized refinement is required.
Figure 4. Interactive Music Experiences. MoXaRt enables users to actively control their listening experience during live music events. (a) The user can act as a real-time audio engineer, selectively adjusting the gain of individual instruments, such as the violin, cello, and piano in a trio—to create a personalized mix. (b) In a concert setting, the user can enhance acoustic focus by isolating the musicians’ performance from distracting audience chatter.
Figure 5. Enhanced Social Interaction. MoXaRt facilitates communication in complex and noisy social settings. (a) In an environment with multiple simultaneous conversations, a user can selectively amplify a target conversation to maintain focus. (b) The system can improve speech intelligibility in chaotic environments by isolating a conversation partner’s voice from loud background noise, such as live music.
Figure 6. Downstream AI Assistance. The separated streams from MoXaRt can serve as clean, audio-visually aligned input for LLM assistance ( e.g., Gemini, Chatgpt.) (a) The system can isolate individual speakers in a multi-lingual environment, enabling real-time, multi-person transcription and translation. (b) The system’s output could be used to create a queryable log of interactions, allowing a user to ask questions about past events ( e.g., “What did Peter say about this evening?”) and receive a contextually grounded response.
Reference SystemArchitectureTarget DomainObject SelectionMax SourcesReal-Time
Sound of PixelsEnd-to-EndMusic Only✓2✓
DAVISEnd-to-EndUniversal †✗3✗
AudioScopeV2CascadedUniversal✗4✓
AV-MossFormer2End-to-EndSpeech Only✗3✗
MoXaRt (Offline)CascadedUniversal✓8✗
MoXaRt (Real-Time)CascadedUniversal✓8✓
Table 1. Comparison of MoXaRt against baselines and audio-visual sound separation benchmarks.
Figure 7. Data collection setup. (a) A Meta Quest 3 with two attached Rode Wireless GO II microphones captures the input mono-audio, while a stationary iPhone records a third-person view of the scene. (b) The three sound sources (two instruments and one speaker), each close-mic’d to record their individual ground-truth audio signal.
MethodWER † ↓\downarrowDNSMOS-SIG ↑\uparrowDNSMOS-BAK ↑\uparrowDNSMOS-OVRL ↑\uparrow
Raw Audio0.6297 ( ±\pm 0.4371)3.7740 ( ±\pm 0.1977)2.4796 ( ±\pm 0.3928)2.8944 ( ±\pm 0.1673)
Sound of Pixels0.8636 ( ±\pm 0.1787)3.3212 ( ±\pm 0.2467)2.9535 ( ±\pm 0.2935)2.7291 ( ±\pm 0.2004)
DAVIS0.7259 ( ±\pm 0.2484)3.3212 ( ±\pm 0.2467)2.9535 ( ±\pm 0.2935)2.7291 ( ±\pm 0.2004)
AudioScopeV20.5263 ( ±\pm 0.3358)3.5027 ( ±\pm 0.5029)3.0236 ( ±\pm 0.4497)2.9349 ( ±\pm 0.3074)
AV-Mossformer2 †0.3956 ( ±\pm 0.2847)3.7245 ( ±\pm 0.2813)3.2564 ( ±\pm 0.4728)3.0759 ( ±\pm 0.2892)
MoXaRt Offline0.3824 ( ±\pm 0.2775)3.6625 ( ±\pm 0.3059)3.019 ( ±\pm 0.6078)2.9949 ( ±\pm 0.2585)
Table 2. Objective evaluation of separation accuracy and quality.
Figure 8. User study results. (a) Average Listening Comprehension Performance per scenario and the overall average. MoXaRt led to a statistically significant 36.2% improvement in scores compared to the baseline (p = 0.0058). (b) Average subjective ratings (on a 10-point scale, higher scores indicate better experience) per dimension for the five core experiential dimensions and per scenario and the overall average. Participants rated their experience significantly higher with MoXaRt compared to the baseline (all p < 0.001). Error bars on the average represent the standard deviation.
Figure 9. Model Performance vs. Real-time Analysis. (a) The size of the context window for model input ( window_size ) determines the trade-off between WER and inference time. (b) Reducing chunk_size can optimize interactive latency to as low as 0.82s, at the expense of audio separation quality.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 10. Data collection setup: A Meta Quest 3 with 2 attached Rode Wireless GO II microphones captures the mono audio, while a stationary iPhone records a third-person view of the scene.
More in this topic

Audio Source Separation

Audio source separation isolates individual speakers, instruments, or sound events from mixtures for listening, analysis, and audio editing.

See all 66 papers

Explore similar work

CardsList

  1. RadarVox: Radar-Audio Multimodal Cocktail-Party Speech Separation with Speaker-Aware Cross-Modal Matching

    Sep 20, 2026Yanlin Xu, Yiwei Ru, Mupei Li +3Speech Separation

    New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences (CASIA), Beijing 100190, China · School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS), Beijing 100049, China · Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China +2

  2. Fine-grained Soundscape Control for Augmented Hearing

    Feb 28, 2026Seunghyun Oh, Malek Itani, Aseem Gauri +1Audio Source Separation

    Paul G. Allen School of Computer Science and Engineering, University of Washington · Hearvana AI

  3. VorTEX: Various overlap ratio for Target speech EXtraction

    Mar 16, 2026Ro-hoon Oh, Jihwan Seol, Bugeun KimTarget Speaker ExtractionSpeech Separation

    Department of Artificial Intelligence, Chung-Ang University, Republic of Korea