Generalizable Audio-Visual Navigation via Binaural Difference Attention and Action Transition Prediction
Authors: Jia Li, Yinfeng Yu
Organizations: Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University · School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China
In Audio-Visual Navigation (AVN), agents must locate sound sources in unseen 3D environments using visual and auditory cues. However, existing methods often struggle with generalization in unseen scenarios, as they tend to overfit to semantic sound features and specific training environments. To address these challenges, we propose the Binaural Difference Attention with Action Transition Prediction (BDATP) framework, which jointly optimizes perception and policy. Specifically, the Binaural Difference Attention (BDA) module explicitly models interaural differences to enhance spatial orientation, reducing reliance on semantic categories. Simultaneously, the Action Transition Prediction (ATP) task introduces an auxiliary action prediction objective as a regularization term, mitigating environment-specific overfitting. Extensive experiments on the Replica and Matterport3D datasets demonstrate that BDATP can be seamlessly integrated into various mainstream baselines, yielding consistent and significant performance gains. Notably, our framework achieves state-of-the-art Success Rates across most settings, with a remarkable absolute improvement of up to 21.6 percentage points in Replica dataset for unheard sounds. These results underscore BDATP's superior generalization capability and its robustness across diverse navigation architectures.
Figures & tables
Fig. 1: Comparison between our BDATP and conventional AVN methods.
Fig. 2: BDATP framework overview: separate encoding of visual and auditory inputs, with pink (BDA) and yellow (ATP) modules.
Method
Replica
Matterport3D
Heard
Unheard
Heard
Unheard
SR ↑
SPL ↑
SNA ↑
SR ↑
SPL ↑
SNA ↑
SR ↑
SPL ↑
SNA ↑
SR ↑
SPL ↑
SNA ↑
Random Agent
18.5
4.9
1.8
18.5
4.9
1.8
9.1
2.1
0.8
9.1
2.1
0.8
Direction Follower
72.0
54.7
41.1
17.2
11.1
8.4
41.2
32.3
23.8
18.0
13.9
10.7
SAVi [ 26 ]
54.0
45.1
30.8
33.9
27.5
17.2
40.3
29.1
13.0
29.5
20.4
9.6
Dav-NaV [ 29 ]
85.1
72.6
54.0
58.5
45.6
33.4
82.9
61.9
46.8
55.3
42.4
31.6
TABLE I: Performance comparison on Replica and Matterport3D datasets
Fig. 3: Feature relationship visualization of the BDA module across three spatial scenarios.
Fig. 4: Action Transition Matrices of ATP.Left: Top-10 most probable transitions on AV-WaN; Right: Full transitions on AV-Nav.
Fig. 5: Trajectory comparison in unseen/unheard settings: Matterport3D (left) and Replica (right).
Model
Heard
Unheard
SR ↑
SPL ↑
SR ↑
SPL ↑
w/o BDA and ATP
88.9
64.5
47.3
34.7
w/o ATP
90.2
74.0
66.2
44.4
w/o BDA
92.2
72.9
63.4
44.3
AV-NaV + BDATP
93.1
74.5
68.6
45.0
TABLE II: Ablation study results on Replica datasets
Weight λ
Heard
Unheard
SR ↑
SPL ↑
SR ↑
SPL ↑
λ=0
88.9
64.5
47.3
34.7
λ=0.001
90.2
66.3
57.1
39.2
λ=0.01
88.9
68.8
62.9
39.1
λ=0.1 (Ours)
92.2
72.9
63.4
44.3
TABLE III: Ablation study of the auxiliary task loss weight λ on Replica
Audio-visual Navigation refers to an agent utilizing visual and auditory information in complex 3D environments to accomplish target localization and path planning, thereby achieving autonomous navigation. The core challenge of this task lies in the following: how the agent can break free from the dependence on training data and achieve autonomous navigation with good generalization performance when facing changes in environments and sound sources. To address this challenge, we propose an Audio Spatially-Guided Fusion for Audio-Visual Navigation method. First, we design an audio spatial feature encoder, which adaptively extracts target-related spatial state information through an audio intensity attention mechanism; based on this, we introduce an Audio Spatial State Guided Fusion (ASGF) to achieve dynamic alignment and adaptive fusion of multimodal features, effectively alleviating noise interference caused by perceptual uncertainty. Experimental results on the Replica and Matterport3D datasets indicate that our method is particularly effective on unheard tasks, demonstrating improved generalization under unknown sound source distributions.
Xinyu Zhou, Yinfeng Yu
Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University · School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China
Embodied Audio-Visual Navigation (AVN) requires an agent to locate sound targets by jointly leveraging visual and auditory cues. While prior work has achieved notable success with static sound sources, real-world scenarios often involve moving targets, giving rise to the more challenging problem of Dynamic Audio-Visual Navigation (D-AVN). Existing approaches largely rely on semantic auditory features, which limits generalization to unheard sound categories and hinders fine-grained spatial reasoning. To address these challenges, we propose a novel Two-Stage Multi-modal Learning framework for D-AVN. In the Physical Alignment stage, the proposed Intensity-Calibrated Geometry Encoder (ICGE) performs physics-informed alignment of sound intensity with visual geometry to capture category-agnostic spatial correspondence. In the Scene Aggregation stage, a Transformer-based Cross-Modal Scene Aggregator (CMSA) dynamically integrates multi-modal features to construct a robust state representation for policy learning. Extensive experiments on Replica and Matterport3D demonstrate that our method achieves state-of-the-art success rates, while significantly improving generalization and navigation efficiency in complex environments and under unheard sound scenarios.
Jia Li, Yinfeng Yu
School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China · Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University
Audio-Visual Navigation (AVN) requires an embodied agent to navigate toward a sound source by utilizing both vision and binaural audio. A core challenge arises in complex acoustic environments, where binaural cues become intermittently unreliable, particularly when generalizing to previously unheard sound categories. To address this, we propose RAVN (Reliability-Aware Audio-Visual Navigation), a framework that conditions cross-modal fusion on audio-derived reliability cues, dynamically calibrating the integration of audio and visual inputs. RAVN introduces an Acoustic Geometry Reasoner (AGR) that is trained with geometric proxy supervision. Using a heteroscedastic Gaussian NLL objective, AGR learns observation-dependent dispersion as a practical reliability cue, eliminating the need for geometric labels during inference. Additionally, we introduce Reliability-Aware Geometric Modulation (RAGM), which converts the learned cue into a soft gate to modulate visual features, thereby mitigating cross-modal conflicts. We evaluate RAVN on SoundSpaces using both Replica and Matterport3D environments, and the results show consistent improvements in navigation performance, with notable robustness in the challenging unheard sound setting.
Teng Liu, Yinfeng Yu
Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University · School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China