Audio-visual navigation enables embodied agents to navigate toward sound-emitting targets by leveraging both auditory and visual cues. However, most existing approaches rely on precomputed room impulse responses (RIRs) for binaural audio rendering, restricting agents to discrete grid positions and leading to spatially discontinuous observations. To establish a more realistic setting, we introduce Semantic Audio-Visual Navigation in Continuous Environments (SAVN-CE), where agents can move freely in 3D spaces and perceive temporally and spatially coherent audio-visual streams. In this setting, targets may intermittently become silent or stop emitting sound entirely, causing agents to lose goal information. To tackle this challenge, we propose MAGNet, a multimodal transformer-based model that jointly encodes spatial and semantic goal representations and integrates historical context with self-motion cues to enable memory-augmented goal reasoning. Comprehensive experiments demonstrate that MAGNet significantly outperforms state-of-the-art methods, achieving up to a 12.1% absolute improvement in success rate. These results also highlight its robustness to short-duration sounds and long-distance navigation scenarios. The code is available at https://github.com/yichenzeng24/SAVN-CE.
Figures & tables
Figure 1: Illustration of the three navigation tasks: (a) the agent is restricted to discrete grid points and the sound-emitting goal is placed arbitrarily; (b) the goal, which emits a creaking sound for a limited duration, is semantically grounded in a chair; (c) the agent moves freely using fine-grained actions and the goal sound is available only within a short temporal window.
Figure 2: Overview of the proposed SAVN-CE framework. ① The agent is randomly initialized without prior knowledge of the environment or the goal. ② It explores the environment until the goal object (a chair) starts emitting sound. ③ Leveraging multimodal cues, the agent infers the goal’s semantic category, azimuth, and distance, and navigates toward it while avoiding obstacles and acoustic distractors (e.g., a ringing phone). As the agent approaches the goal (④ → ⑤), the sound emission ceases. Elements highlighted in yellow and blue denote sound-emitting and silent periods, respectively. ⑤ During the silent period, the agent maintains goal tracking by integrating historical goal representations with current self-motion cues (e.g., the previous action and the current pose), successfully localizing and reaching the creaking chair despite the presence of visually similar objects and distractor sounds.
Figure 3: Overall architecture of MAGNet. The multimodal observation encoder extracts multimodal features from current sensory inputs and updates the scene memory accordingly. The memory-augmented goal descriptor network infers spatial and semantic representations of the goal by integrating auditory cues, self-motion cues, and historical goal embeddings stored in the episodic memory, thereby ensuring temporally consistent inference even after the goal sound ceases entirely. Conditioned on the latest scene memory embeddings, the context-aware policy network attends to the encoded memory Me to predict the next action, enabling continuous navigation toward the goal.
Clean Environments
Distracted Environments
SR ↑
SPL ↑
SNA ↑
DTG ↓
SWS ↑
SR ↑
SPL ↑
SNA ↑
DTG ↓
SWS ↑
DSR
Random
0.3
0.2
0.1
15.7
0.2
0.4
0.3
0.1
15.8
0.4
1.2
ObjectGoal
0.8
0.6
0.2
14.4
0.4
1.1
0.8
0.4
14.4
0.8
0.8
AV-Nav
21.3
17.8
13.1
10.7
4.0
13.5
10.9
7.6
12.5
3.0
6.7
SMT + Audio
24.8
21.0
16.8
10.1
5.3
16.5
13.7
9.4
11.8
5.0
6.3
SAVi
25.6
21.2
17.3
10.1
6.0
18.5
14.9
11.1
10.4
5.4
6.4
Table 1: Navigation performance on the SAVN-CE dataset under Clean and Distracted Environments .
Figure 4: Navigation trajectories of different methods under Clean Environments .
Table 2: Comparison of GDN performance for SAVi and MAGNet on the test split. Sounding and Silent denote metrics measured during the sound-emitting and silent periods, respectively. Note that a low LECD accompanied by an extremely low F≤20∘ is not informative.
Figure 5: Impact of (a) action ratio and (b) geodesic distance on the cumulative success rates of different methods under Clean Environments (solid) and Distracted Environments (dashed).
In Audio-Visual Navigation (AVN), agents must locate sound sources in unseen 3D environments using visual and auditory cues. However, existing methods often struggle with generalization in unseen scenarios, as they tend to overfit to semantic sound features and specific training environments. To address these challenges, we propose the Binaural Difference Attention with Action Transition Prediction (BDATP) framework, which jointly optimizes perception and policy. Specifically, the Binaural Difference Attention (BDA) module explicitly models interaural differences to enhance spatial orientation, reducing reliance on semantic categories. Simultaneously, the Action Transition Prediction (ATP) task introduces an auxiliary action prediction objective as a regularization term, mitigating environment-specific overfitting. Extensive experiments on the Replica and Matterport3D datasets demonstrate that BDATP can be seamlessly integrated into various mainstream baselines, yielding consistent and significant performance gains. Notably, our framework achieves state-of-the-art Success Rates across most settings, with a remarkable absolute improvement of up to 21.6 percentage points in Replica dataset for unheard sounds. These results underscore BDATP's superior generalization capability and its robustness across diverse navigation architectures.
Jia Li, Yinfeng Yu
Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University · School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China
Embodied Audio-Visual Navigation (AVN) requires an agent to locate sound targets by jointly leveraging visual and auditory cues. While prior work has achieved notable success with static sound sources, real-world scenarios often involve moving targets, giving rise to the more challenging problem of Dynamic Audio-Visual Navigation (D-AVN). Existing approaches largely rely on semantic auditory features, which limits generalization to unheard sound categories and hinders fine-grained spatial reasoning. To address these challenges, we propose a novel Two-Stage Multi-modal Learning framework for D-AVN. In the Physical Alignment stage, the proposed Intensity-Calibrated Geometry Encoder (ICGE) performs physics-informed alignment of sound intensity with visual geometry to capture category-agnostic spatial correspondence. In the Scene Aggregation stage, a Transformer-based Cross-Modal Scene Aggregator (CMSA) dynamically integrates multi-modal features to construct a robust state representation for policy learning. Extensive experiments on Replica and Matterport3D demonstrate that our method achieves state-of-the-art success rates, while significantly improving generalization and navigation efficiency in complex environments and under unheard sound scenarios.
Jia Li, Yinfeng Yu
School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China · Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University
We explore whether Omni-Language Models (OLMs) can be directly applied to zero-shot Semantic Audio-Visual Navigation (SAVN). Recent work demonstrates that even state-of-the-art specialized models still struggle to achieve generalist multimodal navigation, despite extensive task-specific training. In this paper, we introduce RAO-Nav, short for Reasoning All-in-One OLM, a deployment pipeline for zero-shot SAVN. By leveraging the rich implicit audio-visual knowledge encoded in OLMs, the embodied agent is enabled to hear'', see'', reason'', and act'' in the environment. To further elicit the built-in thinking ability of OLMs, we propose a test-time Latent Navigation Reasoning (LNR) module that can be seamlessly integrated into the decoding space. LNR encourages the model to retrieve more target-relevant observations and make effective navigation decisions. Through comprehensive experiments, we show that our framework surpasses existing state-of-the-art baselines on public SAVN benchmarks without using any training data. Moreover, we introduce a new Global Navigation Instruction setting to further evaluate the ability of OLMs to serve as embodied navigation agents. Code: https://github.com/rikeilong/OmniAV\_Nav.
Qilang Ye, Meng Liu, Yu Zhou
VCIP & TMCC & DISSec, College of Computer Science & College of Cryptology and Cyber Science, Nankai University · Zhongguancun Academy · Shandong University