Audio Spatially-Guided Fusion for Audio-Visual Navigation

Xinyu Zhou; Yinfeng Yu

cs.SDApr 2, 2026

Audio Spatially-Guided Fusion for Audio-Visual Navigation

Authors: Xinyu Zhou, Yinfeng Yu

Organizations: Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University · School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China

Abstract

Audio-visual Navigation refers to an agent utilizing visual and auditory information in complex 3D environments to accomplish target localization and path planning, thereby achieving autonomous navigation. The core challenge of this task lies in the following: how the agent can break free from the dependence on training data and achieve autonomous navigation with good generalization performance when facing changes in environments and sound sources. To address this challenge, we propose an Audio Spatially-Guided Fusion for Audio-Visual Navigation method. First, we design an audio spatial feature encoder, which adaptively extracts target-related spatial state information through an audio intensity attention mechanism; based on this, we introduce an Audio Spatial State Guided Fusion (ASGF) to achieve dynamic alignment and adaptive fusion of multimodal features, effectively alleviating noise interference caused by perceptual uncertainty. Experimental results on the Replica and Matterport3D datasets indicate that our method is particularly effective on unheard tasks, demonstrating improved generalization under unknown sound source distributions.

Figures & tables

Fig. 1: Comparison of navigation trajectories and model architectures. Left: Top-down view of navigation trajectories in an indoor scene (blue: our ASGF-Nav; red: baseline method). Right: (a) Common audio-visual navigation method with simple feature concatenation; (b) Our ASGF-Nav model, which integrates the ASE module to extract spatial information from audio and uses the ASGF module to dynamically fuse audio-visual features.

Fig. 2: Model architecture. Our audio spatially-guided fusion for audio-visual navigation model (ASGF-Nav) uses the ASE module to extract implicit spatial state information from binaural spectrograms, and guides the dynamic fusion of visual and audio features in the ASGF module to provide a basis for policy selection.

Method	Replica						Matterport3D
	Heard			Unheard			Heard			Unheard
	SPL $\uparrow$	SR $\uparrow$	SNA $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SNA $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SNA $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SNA $\uparrow$
Random Agent [ 16 ]	4.9	18.5	1.8	4.9	18.5	1.8	2.1	9.1	0.8	2.1	9.1	0.8
Direction Follower [ 16 ]	54.7	72.0	41.1	11.1	17.2	8.4	32.3	41.2	23.8	13.9	18.0	10.7
Gan et al. [ 15 ]	57.6	83.1	47.9	7.5	15.7	5.7	22.8	37.9	17.1	5.0	10.2	3.6
AV-WAN [ 16 ]	86.6	98.7	70.7	34.7	52.8	27.1	72.3	93.6	54.8	40.9	56.7	30.6

TABLE I: Navigation Performance Comparison with other methods.

Method	Replica (Unheard)			Matterport3D (Unheard)
Method	SPL $\uparrow$	SR $\uparrow$	SNA $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SNA $\uparrow$
SoundSpaces	34.7	50.9	16.7	25.9	40.5	12.8
w/o ASGF	38.8	53.6	21.4	31.6	40.0	20.7
w/o ASE	59.5	71.9	34.8	41.4	59.9	22.7
ASGF-Nav	63.3	76.5	36.9	52.2	66.4	29.9

TABLE II: Ablation study of ASGF-Nav components on the Unheard task.

Fig. 3: Top-down visualization of agent trajectories under the Unheard task. The color gradient from dark to light blue represents temporal progression. Compared with SoundSpaces, our method reaches the target with fewer detours.

Fig. 4: t-SNE projection of the audio features extracted by the ASE module. The left figure is colored according to the distance to the target, while the right figure is colored according to the relative angle to the target.

Audio-Visual Navigation

See all 11 papers

Spatial-Aware Conditioned Fusion for Audio-Visual Navigation
Apr 2, 2026Shaohang Wu, Yinfeng YuAudio-Visual Navigation Embodied Navigation
Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University · School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China
Reliability-Aware Geometric Fusion for Robust Audio-Visual Navigation
Apr 2, 2026Teng Liu, Yinfeng YuAudio-Visual Navigation Embodied Navigation
Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University · School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China
Two-Stage Multi-modal Learning for Dynamic Audio-Visual Navigation: From Physical Alignment to Scene Aggregation
Oct 8, 2026Jia Li, Yinfeng YuAudio-Visual Navigation Audio-Visual Alignment
School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China · Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University

Apr 2, 2026J/K move · Enter open · S save

Spatial-Aware Conditioned Fusion for Audio-Visual Navigation

Shaohang Wu, Yinfeng Yu

Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University · School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China

Audio-Visual Navigation Embodied Navigation Multimodal Fusion

Open paper AskPDF arXiv

Audio-visual navigation tasks require agents to locate and navigate toward continuously vocalizing targets using only visual observations and acoustic cues. However, existing methods mainly rely on simple feature concatenation or late fusion, and lack an explicit discrete representation of the target's relative position, which limits learning efficiency and generalization. We propose Spatial-Aware Conditioned Fusion (SACF). SACF first discretizes the target's relative direction and distance from audio-visual cues, predicts their distributions, and encodes them as a compact descriptor for policy conditioning and state modeling. Then, SACF uses audio embeddings and spatial descriptors to generate channel-wise scaling and bias to modulate visual features via conditional linear transformation, producing target-oriented fused representations. SACF improves navigation efficiency with lower computational overhead and generalizes well to unheard target sounds.

Abstract

Audio-Visual Navigation

Explore similar work

Spatial-Aware Conditioned Fusion for Audio-Visual Navigation

Reliability-Aware Geometric Fusion for Robust Audio-Visual Navigation

Two-Stage Multi-modal Learning for Dynamic Audio-Visual Navigation: From Physical Alignment to Scene Aggregation