Spatial-Aware Conditioned Fusion for Audio-Visual Navigation

Shaohang Wu; Yinfeng Yu

cs.SDApr 2, 2026

Spatial-Aware Conditioned Fusion for Audio-Visual Navigation

Authors: Shaohang Wu, Yinfeng Yu

Organizations: Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University · School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China

Abstract

Audio-visual navigation tasks require agents to locate and navigate toward continuously vocalizing targets using only visual observations and acoustic cues. However, existing methods mainly rely on simple feature concatenation or late fusion, and lack an explicit discrete representation of the target's relative position, which limits learning efficiency and generalization. We propose Spatial-Aware Conditioned Fusion (SACF). SACF first discretizes the target's relative direction and distance from audio-visual cues, predicts their distributions, and encodes them as a compact descriptor for policy conditioning and state modeling. Then, SACF uses audio embeddings and spatial descriptors to generate channel-wise scaling and bias to modulate visual features via conditional linear transformation, producing target-oriented fused representations. SACF improves navigation efficiency with lower computational overhead and generalizes well to unheard target sounds.

Figures & tables

Fig. 1: Comparison between the SoundSpaces baseline and our method.

Fig. 2: SACF for audio-visual navigation receives image and audio observations $o_{t}=\{I_{t},D_{t},A_{t}\}$ at each time step. Features are extracted via visual/audio encoders, and spatial descriptors $g_{t}$ are generated by SDLD. Subsequently, ACVF conditions on $[F_{t}^{a};g_{t}]$ to generate FiLM-style channel modulation parameters that enhance the directionality of visual representations. These fused features are input into a GRU, which outputs action $a_{t}$ and value estimates via Actor-Critic for closed-loop decision-making.

Fig. 3: SDLD fuses $F_{t}^{v}$ and $F_{t}^{a}$ into $F_{av}$ to predict distance/angle distributions, then takes the expectation to obtain $\hat{d},\hat{\theta}$ . These are encoded as compact spatial descriptors using $(\cos\hat{\theta},\sin\hat{\theta})$ .

Method	Replica						Matterport3D
	Heard			Unheard			Heard			Unheard
	SPL $\uparrow$	SR $\uparrow$	SNA $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SNA $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SNA $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SNA $\uparrow$
Random [ 11 ]	4.9	18.5	1.8	4.9	18.5	1.8	2.1	9.1	0.8	2.1	9.1	0.8
Direction Follower [ 11 ]	54.7	72.0	41.1	11.1	17.2	8.4	32.3	41.2	23.8	13.9	18.0	10.7
Frontier Waypoints [ 11 ]	44.0	63.9	35.2	6.5	14.8	5.1	30.6	42.8	22.2	10.9	16.4	8.1
Supervised Waypoints [ 11 ]	59.1	88.1	48.5	14.1	43.1	10.1	21.0	36.2	16.2	4.1	8.8	2.9

TABLE I: Performance comparison with other methods under the Depth setting. SPL, SR, SNA are percentages.

Method	Replica						Matterport3D
	Heard			Unheard			Heard			Unheard
	SPL $\uparrow$	SR $\uparrow$	SNA $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SNA $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SNA $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SNA $\uparrow$
SoundSpaces [ 1 ]	62.6	72.1	31.5	24.9	35.6	14.1	44.7	64.3	22.0	20.4	30.4	7.7
SACF (Ours)	73.5	90.1	38.3	36.0	62.2	14.2	45.6	67.7	23.2	39.6	57.7	22.8

TABLE II: Performance comparison with other methods under the RGB setting. SPL, SR, SNA are percentages.

Model	Replica		Matterport3D
Model	SR ( $\uparrow$ )	SPL ( $\uparrow$ )	SR ( $\uparrow$ )	SPL ( $\uparrow$ )
w/o SDLD and w/o ACVF	50.9	34.7	33.5	21.9
w/o SDLD	66.1	38.6	56.2	37.8
w/o ACVF	67.6	36.7	56.2	39.5
SACF (Ours)	79.1	43.9	58.3	42.4

TABLE III: Ablation Studies of the SDLD and ACVF Modules On the Unheard Setting.

Model	Replica	Matterport3D
	Params (M) $\downarrow$	Params (M) $\downarrow$
Simple Concatenation	4.35M	4.95M
Cross-Modal Spatial Attention	7.06M	7.67M
SACF (Ours)	4.50M	4.49M

TABLE IV: Parameter comparison of different audio-visual fusion mechanisms.

Fig. 4: Red boxes highlight key turning points where acoustic cues successfully guide SACF to avoid dead ends and navigate more efficiently than the baseline.

Fig. 5: Top-down navigation trajectory diagram for navigation.

Fig. 6: Convergence: rewards and SPL (0–1; tables in %).

Audio-Visual Navigation

See all 11 papers

Audio Spatially-Guided Fusion for Audio-Visual Navigation
Apr 2, 2026Xinyu Zhou, Yinfeng YuAudio-Visual Navigation Embodied Navigation
Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University · School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China
Reliability-Aware Geometric Fusion for Robust Audio-Visual Navigation
Apr 2, 2026Teng Liu, Yinfeng YuAudio-Visual Navigation Embodied Navigation
Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University · School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China
Semantic Audio-Visual Navigation in Continuous Environments
Mar 20, 2026Yichen Zeng, Hebaixu Wang, Meng Liu +4Audio-Visual Navigation Multimodal Transformer
Wuhan University · Zhongguancun Academy · Shandong Jianzhu University +4

Apr 2, 2026J/K move · Enter open · S save

Audio Spatially-Guided Fusion for Audio-Visual Navigation

Xinyu Zhou, Yinfeng Yu

Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University · School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China

Audio-Visual Navigation Embodied Navigation Uncertainty-Aware Multimodal Fusion

Open paper AskPDF arXiv

Audio-visual Navigation refers to an agent utilizing visual and auditory information in complex 3D environments to accomplish target localization and path planning, thereby achieving autonomous navigation. The core challenge of this task lies in the following: how the agent can break free from the dependence on training data and achieve autonomous navigation with good generalization performance when facing changes in environments and sound sources. To address this challenge, we propose an Audio Spatially-Guided Fusion for Audio-Visual Navigation method. First, we design an audio spatial feature encoder, which adaptively extracts target-related spatial state information through an audio intensity attention mechanism; based on this, we introduce an Audio Spatial State Guided Fusion (ASGF) to achieve dynamic alignment and adaptive fusion of multimodal features, effectively alleviating noise interference caused by perceptual uncertainty. Experimental results on the Replica and Matterport3D datasets indicate that our method is particularly effective on unheard tasks, demonstrating improved generalization under unknown sound source distributions.

Abstract

Audio-Visual Navigation

Explore similar work

Audio Spatially-Guided Fusion for Audio-Visual Navigation

Reliability-Aware Geometric Fusion for Robust Audio-Visual Navigation

Semantic Audio-Visual Navigation in Continuous Environments