Sound-Triggered Mobile Manipulation (STMM) requires robots to respond to sound events by navigating to the associated objects and performing appropriate physical interactions. Existing STMM evaluations primarily vary object layouts within limited architectural structures, leaving challenges to unseen scenes insufficiently explored. We introduce Multi-scene Sound-Triggered Mobile Manipulation (MuSTMM), a dataset that combines spatial audio rendering and physical interaction across 31 scenes with diverse architectures and object layouts. MuSTMM contains 14.36 million precomputed binaural Room Impulse Responses (RIRs) and a scene-disjoint split for evaluating single-event and multi-event settings in unseen scenes. We further augment the Navigation Policy Model with explicit spatial memory that complements acoustic cues with observed geometry. Specifically, we accumulate pose-aligned depth observations in a shared coordinate frame and project this memory into an egocentric Bird's-Eye View (BEV) occupancy map. Fusing map features with sound context produces a goal descriptor that provides both target-location and scene-structure priors, guiding navigation to the target region required for subsequent physical interaction. This policy achieves a 2.60% success rate boost over existing methods, while component ablations further support the effectiveness of spatial memory. MuSTMM and the navigation policy provide a testbed for the new challenge of studying sound-triggered mobile manipulation in unseen scenes. The code and dataset will be released upon acceptance.
Figures & tables
Figure 2: Overview of MuSTMM. (a) Comparison with existing datasets. In addition to physical interaction and sound rendering, MuSTMM brings structure and layout variation. (b) Structure variation: Distribution of room counts across the different scenes in MuSTMM, up to 17 . (c) Layout variation: MuSTMM contains 855 furniture assets, providing varied visual backgrounds.
Figure 3: Overview of the proposed baseline. The baseline consists of a high-level task planner and low-level policy models. Sound-triggered Task Planner receives initial observations and predicts the skill chain. Following the skill chain, policy models are sequentially activated and interact with the environment. For the navigation policy model, we propose spatially augmented goal descriptor branch that provides the policy branch with spatio-temporal context of the goal. (Color information is projected into the spatial memory for visualization only.)
Figure 4: Results on MuSTMM. (a) Success rates of task planning and skill completion. SSP achieves the best performance in single-event settings, while Qwen2.5-Omni-7B performs best in the multi-event setting. Best and second-best results are highlighted, excluding Oracle. (b) Progressive completion success rate of object relocation. (c) Progressive completion success rate of state transition. (d) Progressive completion success rate in the multi-event setting.
Figure 5: Qualitative Visualizations of Task Execution (Zoom in for a better visualization). Execution trajectories for (a) Object Relocation, (b) State Transition, and (c) Multi-event Setting. For each task, the left panel maps robot paths and sound sources from a top-down view, while the right panel shows third-person keyframes. Multi-event Setting requires sequential navigation and interaction with two distinct sound sources. More visualizations can be seen in the supplementary material.
Table 1: Navigation comparison and Ablation Study. (a) Comparison with existing navigation policies ( Chen et al., 2021 ; Zeng et al., 2026 ) . (b) Effect of memory components on the navigation policy. E.S.M. and C.S.T.M. denote Explicit Spatial Memory and Compressed Spatio-Temporal Memory, respectively. (c) Effect of input modalities on the open refrigerator policy model. (d) Effect of input modalities on the supervised setting-specific planner. S.R. denotes success rate.
Method
Sound
Head RGB
S.R.
I
✗
✓
51.00
II
✓
✓
71.70
III
✓
✗
75.40
Table 2: Ablation study on input modalities for Setting-Specific Planner (SSP) in the multi-event setting.
Table 3: Numerical Results on the MuSTMM. We report success rates ( % ) for task planning ( SRplan ), individual skill evaluation (Individual), conditional skill completion (skill columns), and full-task completion (CSR). The higher the better. (a) Evaluation of object relocation. (b) Evaluation of state transition. (c) Evaluation of multi-event setting. Success rates are reported in the format “First Sound Source / Second Sound Source”.
Figure 6: Failure Case Visualizations of Navigation Skill Execution (Zoom in for a better visualization). For each case, the left panel maps robot paths and sound sources from a top-down view, while the right panel shows third-person keyframes.
Figure 7: Train and test splits. ( 24.91k and 5.20k episodes) O.R. denotes Object Relocation, and S.T. denotes State Transition. Interaction type of the first sound source → the second one.
School of Intelligence Science and Technology, Peking University · State Key Laboratory of General Artificial Intelligence, Peking University · Alibaba Token Hub, Alibaba Group +1