Navigating toward human callers is an important capability for rescue robots operating where visual contact is degraded or occluded. We present AcousticDiffusion, a semantically conditioned, audio-guided diffusion policy for human-directed navigation. A frozen pretrained audio recognizer processes 10.24 s windows, with speech gating and distress-aware prioritization converting recognition outputs into source-level navigation roles. Microphone-array direction-of-arrival measurements are recursively integrated into a robot-centric Bayesian bird's-eye-view belief field. Ego-motion compensation aligns successive observations, progressively constraining source position while preserving bearing-induced range uncertainty. The semantic belief, recent acoustic observations, audio features, and robot state condition a diffusion model that generates waypoint trajectories. On a synthetic-navigation validation set using recorded audio, AcousticDiffusion achieves a mean end-point bearing error of 11.20 degrees, with 91.78% of trajectories aligned within 30 degrees of the caller. Distractor rejection ranges from 89.20% to 98.99%, and the policy favors a HELP-designated caller over a competing speaker in 91.07% of windows. Deployed online on a ZSL-1 quadruped without additional retraining, it achieves a mean bearing error of 64.9 degrees, compared with 98.2 degrees for A* and 90.4 degrees for RRT, with a mean planner compute time of 6.07 ms. Despite imperfect acoustic localization, the reported mean final source distance is reduced from 3.96 m for the classical planners using ODAS-derived (Open embedded Audition System) guidance to 2.48 m, a 37.4% improvement. These results demonstrate the framework's ability to translate uncertain acoustic observations into closer approaches to human callers.
Figures & tables
Fig. 1 : Overview of AcousticDiffusion . The quadruped robot identifies a task-relevant acoustic event among competing sound classes, accumulates direction-of-arrival observations into a bird’s-eye-view sound belief, and generates a diffusion-based trajectory toward the target while avoiding obstacles.
Fig. 2 : Overview of AcousticDiffusion. Pretrained audio recognition provides semantic predictions and audio features. Sound-source bearings and odometry update a Bayesian BEV belief. Fused conditioning guides a 1-D U-Net that generates trajectories from Gaussian noise through 16 DDIM iterations. The BEV output and denoising stages are illustrative.
Fig. 3 : Qualitative visualizations from the AcousticDiffusion deployment on the quadruped robot. Both panels show the microphone-array frame ( respeaker_link ), the displayed trajectory, the learned source-head diagnostic (magenta), and the BEV attraction-field estimate (red), overlaid on the local map. Panel (b) additionally shows the ODAS bearing observation in yellow.
Source class
N
Rate (%) ↑
Caller alignment within 45∘
Female speech
160
97.29±1.30
Male speech
136
96.32±0.74
Distractor rejection
Domestic sounds
71
89.20±0.81
Laughter
72
98.61±0.00
TABLE I : Human-directed navigation. Rates are mean ± sample standard deviation over three sampling seeds. N counts windows containing each class; classes may co-occur. Laughter is treated as a non-speech distractor.
Metric
x0
ϵ
Bearing error ( ∘ ) ↓
17.42±0.07
11.20±0.33
Caller-directed (%) ↑
87.05±0.39
89.64±0.98
Within 30∘ (%) ↑
80.86±0.20
91.78±1.67
ADE (m) ↓
0.285
0.329
FDE (m) ↓
0.846
0.725
Roughness / expert ↓
136.29
76.98
TABLE II : Prediction-target ablation. Alignment values are mean ± sample standard deviation over three sampling seeds, not training runs. ADE, FDE, and roughness use 256 validation windows.
Conditioning
Bearing ( ∘ ) ↓
Within 30∘ (%) ↑
Full
11.20±0.33
91.78±1.67
No speech (prior)
20.03±0.43
79.84±0.70
No speech (zero)
24.16±0.76
73.25±1.66
No ODAS (prior)
46.76±1.30
32.58±0.97
No ODAS (zero)
24.16±0.85
79.69±2.09
Past motion only
48.27±1.04
25.14±2.16
TABLE III : Conditioning ablation: mean ± sample standard deviation over three sampling seeds. Bearing denotes end-point angular error to the caller.
Metric
AD (ours)
A*
RRT
Bearing alignment
Mean error ( ∘ ) ↓
64.9
98.2
90.4
Windowed error ( ∘ ) ↓
47.2
72.8
59.8
Windows ≤15∘ (%) ↑
32
18
25
Planner compute and timing
Mean compute (ms) ↓
6.07
0.15
23.6
TABLE IV : Comparison on trials S1–S3. AD denotes AcousticDiffusion. Windowed error is the mean of the minimum bearing errors within 10.24 s windows. Compute time measures planner execution, not end-to-end sensing latency. Bold indicates the best value, excluding the descriptive replanning period.
The ability to navigate toward sound sources extends a robot's reach beyond its visual field, enabling response to auditory events in unknown environments. To equip robots with this capability, existing methods couple acoustic and visual information through joint audio-visual learning in acoustic simulators. However, acoustic simulation is both low-fidelity and expensive, producing a domain gap that prevents reliable real-world deployment, while the discrete action spaces inherited from grid-based simulators introduce an additional kinematic gap on physical robots. To alleviate these issues, we propose BCNav, a decoupled framework that separates the acoustic module from the learned navigation policy using direction-of-arrival (DOA) estimation: an estimator provides a scalar bearing to the sound source, so the navigation policy only processes depth images and a bearing angle, two inputs whose domain gaps are well characterized. We collect shortest-path demonstrations with calibrated bearing noise injection and train the policy via imitation learning to output continuous velocity commands directly executable on ground robots. We demonstrate the method in simulation and on a physical robot, navigating unknown environments without any acoustic fine-tuning, prior mapping, or real-world audio data collection. Code is available at https://github.com/york1to/bcnav.
Yaozhong Kang, Jiang Wang, Takeshi Ashizawa +2
Department of Systems and Control Engineering, Institute of Science Tokyo, Japan
Audio-visual navigation tasks require agents to locate and navigate toward continuously vocalizing targets using only visual observations and acoustic cues. However, existing methods mainly rely on simple feature concatenation or late fusion, and lack an explicit discrete representation of the target's relative position, which limits learning efficiency and generalization. We propose Spatial-Aware Conditioned Fusion (SACF). SACF first discretizes the target's relative direction and distance from audio-visual cues, predicts their distributions, and encodes them as a compact descriptor for policy conditioning and state modeling. Then, SACF uses audio embeddings and spatial descriptors to generate channel-wise scaling and bias to modulate visual features via conditional linear transformation, producing target-oriented fused representations. SACF improves navigation efficiency with lower computational overhead and generalizes well to unheard target sounds.
Shaohang Wu, Yinfeng Yu
Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University · School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China
Audio-visual Navigation refers to an agent utilizing visual and auditory information in complex 3D environments to accomplish target localization and path planning, thereby achieving autonomous navigation. The core challenge of this task lies in the following: how the agent can break free from the dependence on training data and achieve autonomous navigation with good generalization performance when facing changes in environments and sound sources. To address this challenge, we propose an Audio Spatially-Guided Fusion for Audio-Visual Navigation method. First, we design an audio spatial feature encoder, which adaptively extracts target-related spatial state information through an audio intensity attention mechanism; based on this, we introduce an Audio Spatial State Guided Fusion (ASGF) to achieve dynamic alignment and adaptive fusion of multimodal features, effectively alleviating noise interference caused by perceptual uncertainty. Experimental results on the Replica and Matterport3D datasets indicate that our method is particularly effective on unheard tasks, demonstrating improved generalization under unknown sound source distributions.
Xinyu Zhou, Yinfeng Yu
Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University · School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China