cs.ROOct 15, 2026

MuSTMM: Learning Sound-Triggered Mobile Manipulation across Multiple Scenes

Authors: Hao Ju, Feng Yan, Zhihao Zhong, Heng Yang, Shaofei Huang, Zhedong Zheng

Organizations: University of Macau · Enacta-AI · Hefei University of Technology

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Sound-Triggered Mobile Manipulation (STMM) requires robots to respond to sound events by navigating to the associated objects and performing appropriate physical interactions. Existing STMM evaluations primarily vary object layouts within limited architectural structures, leaving challenges to unseen scenes insufficiently explored. We introduce Multi-scene Sound-Triggered Mobile Manipulation (MuSTMM), a dataset that combines spatial audio rendering and physical interaction across 31 scenes with diverse architectures and object layouts. MuSTMM contains 14.36 million precomputed binaural Room Impulse Responses (RIRs) and a scene-disjoint split for evaluating single-event and multi-event settings in unseen scenes. We further augment the Navigation Policy Model with explicit spatial memory that complements acoustic cues with observed geometry. Specifically, we accumulate pose-aligned depth observations in a shared coordinate frame and project this memory into an egocentric Bird's-Eye View (BEV) occupancy map. Fusing map features with sound context produces a goal descriptor that provides both target-location and scene-structure priors, guiding navigation to the target region required for subsequent physical interaction. This policy achieves a 2.60% success rate boost over existing methods, while component ablations further support the effectiveness of spatial memory. MuSTMM and the navigation policy provide a testbed for the new challenge of studying sound-triggered mobile manipulation in unseen scenes. The code and dataset will be released upon acceptance.

Figures & tables

Figure 2: Overview of MuSTMM. (a) Comparison with existing datasets. In addition to physical interaction and sound rendering, MuSTMM brings structure and layout variation. (b) Structure variation: Distribution of room counts across the different scenes in MuSTMM, up to 17 . (c) Layout variation: MuSTMM contains 855 furniture assets, providing varied visual backgrounds.
Figure 3: Overview of the proposed baseline. The baseline consists of a high-level task planner and low-level policy models. Sound-triggered Task Planner receives initial observations and predicts the skill chain. Following the skill chain, policy models are sequentially activated and interact with the environment. For the navigation policy model, we propose spatially augmented goal descriptor branch that provides the policy branch with spatio-temporal context of the goal. (Color information is projected into the spatial memory for visualization only.)
Figure 4: Results on MuSTMM. (a) Success rates of task planning and skill completion. SSP achieves the best performance in single-event settings, while Qwen2.5-Omni-7B performs best in the multi-event setting. Best and second-best results are highlighted, excluding Oracle. (b) Progressive completion success rate of object relocation. (c) Progressive completion success rate of state transition. (d) Progressive completion success rate in the multi-event setting.
Figure 5: Qualitative Visualizations of Task Execution (Zoom in for a better visualization). Execution trajectories for (a) Object Relocation, (b) State Transition, and (c) Multi-event Setting. For each task, the left panel maps robot paths and sound sources from a top-down view, while the right panel shows third-person keyframes. Multi-event Setting requires sequential navigation and interaction with two distinct sound sources. More visualizations can be seen in the supplementary material.
Table 1: Navigation comparison and Ablation Study. (a) Comparison with existing navigation policies ( Chen et al., 2021 ; Zeng et al., 2026 ) . (b) Effect of memory components on the navigation policy. E.S.M. and C.S.T.M. denote Explicit Spatial Memory and Compressed Spatio-Temporal Memory, respectively. (c) Effect of input modalities on the open refrigerator policy model. (d) Effect of input modalities on the supervised setting-specific planner. S.R. denotes success rate.
MethodSoundHead RGBS.R.
I✗✓51.00
II✓✓71.70
III✓✗75.40
Table 2: Ablation study on input modalities for Setting-Specific Planner (SSP) in the multi-event setting.
Table 3: Numerical Results on the MuSTMM. We report success rates ( %\% ) for task planning ( SRplanSR_{\text{plan}} ), individual skill evaluation (Individual), conditional skill completion (skill columns), and full-task completion (CSR). The higher the better. (a) Evaluation of object relocation. (b) Evaluation of state transition. (c) Evaluation of multi-event setting. Success rates are reported in the format “First Sound Source / Second Sound Source”.
Figure 6: Failure Case Visualizations of Navigation Skill Execution (Zoom in for a better visualization). For each case, the left panel maps robot paths and sound sources from a top-down view, while the right panel shows third-person keyframes.
Figure 7: Train and test splits. ( 24.91k and 5.20k episodes) O.R. denotes Object Relocation, and S.T. denotes State Transition. Interaction type of the first sound source →\rightarrow the second one.
More in this topic

Robotic Spatial Memory

See all 21 papers

Explore similar work

CardsList
Jan 29, 2026

From Instruction to Event: Sound-Triggered Mobile Manipulation

Current mobile manipulation research predominantly follows an instruction-driven paradigm, where robots rely on predefined textual commands to execute tasks. However, this setting confines robots to a passive role, limiting robotic autonomy and the ability to react to dynamic environmental events. To address these limitations, we introduce Sound-Triggered Mobile Manipulation (STMM), where robots must actively perceive and interact with sound-emitting objects without explicit action instructions. To support STMM, we develop Habitat-Echo, a simulation platform that integrates sound rendering with physical interaction. We further propose a hierarchical baseline that translates high-level planning into low-level executions, where a task planner predicts a skill chain for policy models to execute sequentially. Experiments indicate the feasibility of perceiving auditory events and executing corresponding physical interactions without explicit instructions. Notably, in challenging multi-event scenarios, the robot successfully isolates the primary sources from overlapping acoustic interference to execute the first interactions, and subsequently proceeds to manipulate the secondary objects. These new challenges of planning-to-execution position STMM as a measurable research direction.The code and datasets will be released upon acceptance.

Hao Ju, Shaofei Huang, Hongyu Li +4

University of Macau · Beihang University · Hefei University of Technology

Robotics SimulationMobile RoboticsRobotic ManipulationRobot Task PlanningRobot Manipulation Benchmarks
AskPDF
Sep 20, 2026

OmniEcho: Audio-Visual Spatial Understanding for Omni-Modal Embodied Agents

Humans can effortlessly localize the direction of a sound source and integrate it with visual cues for reasoning, yet this remains challenging for embodied agents. In particular, it is still unclear how to effectively evaluate and model spatial audio understanding in embodied settings. To address this gap, we introduce OmniEchoBench, a unified benchmark for spatial audio-visual perception and audio-vision-language navigation. OmniEchoBench comprises six tasks over 197 real-world spatial audio-visual scenes, 2,972 question-answer pairs, and 900 navigation samples with first-order ambisonics (FOA) audio collected from 30 real-world environments. To enable scalable training supervision, we develop a controllable rendering pipeline for spatial audio. It preserves geometric consistency among sound sources, visual observations, and agent trajectories. Building on this, we propose OmniEcho, a spatially aware omni-modal model. It introduces an FOA spatial encoder alongside a pretrained semantic audio pathway. Extensive experiments show that OmniEcho achieves state-of-the-art performance on spatial audio-visual perception. For our sound-guided navigation, OmniEcho reaches a performance level close to that of traditional vision-language navigation. These results demonstrate that spatial audio can serve as a valuable signal for embodied scene reasoning and navigation, while also highlighting fine-grained spatial localization and distance estimation as important open challenges. Our code and data will be available in https://github.com/PKU-VaLuE-Lab/OmniEcho/tree/main

Ruixun Liu, Yuxuan Wang, Jiacheng Xie +10

School of Intelligence Science and Technology, Peking University · State Key Laboratory of General Artificial Intelligence, Peking University · Alibaba Token Hub, Alibaba Group +1

Audio-Visual UnderstandingSpatial Reasoning BenchmarksEmbodied NavigationAudio-Visual NavigationSpatial Audio
AskPDF
Oct 12, 2026

Listen Before Act: Audio-Conditioned Robot Manipulation with ListenVLA

Vision-language-action (VLA) models have advanced robotic manipulation by connecting visual perception and language instructions to action generation. However, many manipulation tasks depend on auditory cues that cannot be reliably inferred from visual observations and may need to be elicited through physical interaction. To address this gap, we present ListenVLA, an audio-conditioned VLA policy built on GR00T N1.7 for sound-guided manipulation. ListenVLA combines semantic audio embeddings with waveform-derived acoustic cues over a causal audio history and conditions continuous action generation through audio tokens and feature modulation. Moreover, we introduce an audio-action binding objective that uses context-matched demonstrations to link auditory differences to demonstrated action differences. The objective supervises short-horizon differences between flow-velocity predictions under different audio conditions, with shared pure-noise action inputs and fixed non-audio observations, alongside full-chunk action supervision. Furthermore, we develop LIBERO-Audio, an eight-task benchmark covering sound-based target selection, responses to auditory events, and physical interactions that reveal acoustic evidence. Experiments on LIBERO-Audio and three real-world manipulation tasks show excellent performance. Ablation studies support the contributions of binding supervision and audio modulation, while audio swap tests show that sound influences action predictions. Our code and videos are available at https://anonymous.4open.science/r/listenvla-F18D

Jiaqi Fan, Rui Xu, Jiakun Xia +7

Robotic ManipulationVision-Language-Action ModelsVLMs for Robotics
AskPDF