Two-Stage Multi-modal Learning for Dynamic Audio-Visual Navigation: From Physical Alignment to Scene Aggregation
Authors: Jia Li, Yinfeng Yu
Organizations: School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China · Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University
Embodied Audio-Visual Navigation (AVN) requires an agent to locate sound targets by jointly leveraging visual and auditory cues. While prior work has achieved notable success with static sound sources, real-world scenarios often involve moving targets, giving rise to the more challenging problem of Dynamic Audio-Visual Navigation (D-AVN). Existing approaches largely rely on semantic auditory features, which limits generalization to unheard sound categories and hinders fine-grained spatial reasoning. To address these challenges, we propose a novel Two-Stage Multi-modal Learning framework for D-AVN. In the Physical Alignment stage, the proposed Intensity-Calibrated Geometry Encoder (ICGE) performs physics-informed alignment of sound intensity with visual geometry to capture category-agnostic spatial correspondence. In the Scene Aggregation stage, a Transformer-based Cross-Modal Scene Aggregator (CMSA) dynamically integrates multi-modal features to construct a robust state representation for policy learning. Extensive experiments on Replica and Matterport3D demonstrate that our method achieves state-of-the-art success rates, while significantly improving generalization and navigation efficiency in complex environments and under unheard sound scenarios.
Figures & tables
Fig. 1: Task comparison. Left: static sound source—target is fixed in a clean environment. Right: moving sound source—target moves with interference, creating a dynamic and more challenging task.
Fig. 2: Model architecture of the two-stage framework. Observations ( Bt,Vt ) are mapped to Gt and the acoustic memory At (this branch in Figure 2 ; not a global sound field). Stage I extracts spatial features via four encoders, where ICGE performs physics-informed physical alignment via intensity–geometry fusion. Stage II aggregates spatial tokens with CMSA to produce zN . A GRU and PPO-based Actor-Critic complete policy learning.
Encoder
Kernel, Stride
CNN 1
(8, 4), (4, 2), (3, 1)
CNN 2
(8, 4), (4, 2), (3, 2), (2, 1)
CNN 3
(8, 4), (4, 2), (3, 2), (3, 2)
CNN 4
(8, 4), (4, 2), (3, 2), (2, 1)
TABLE I: CNN encoder configurations.
Hyperparameter
Value
Optimizer
Adam
PPO clip
0.1
PPO epochs
4
Mini-batches
1
Value loss coefficient
0.5
Entropy coefficient
0.02
TABLE II: RL training hyperparameters.
Model
Replica
Matterport3D
Heard
Unheard
Heard
Unheard
DSR ↑
DSPL ↑
DSNA ↑
DSR ↑
DSPL ↑
DSNA ↑
DSR ↑
DSPL ↑
DSNA ↑
DSR ↑
DSPL ↑
DSNA ↑
AV-Nav
71.0
47.2
16.3
37.4
23.4
7.8
73.0
43.4
18.4
35.5
20.9
8.4
AV-WaN
86.8
55.0
32.6
38.2
24.1
14.2
86.7
58.3
33.6
47.2
30.0
17.4
SAVi
67.6
54.0
25.7
40.8
28.9
13.9
49.0
29.9
11.3
41.6
26.3
10.0
FSAAVN
80.1
52.5
28.8
39.2
28.9
15.8
83.9
47.8
24.4
43.3
27.6
18.7
TABLE III: Performance in Clean Environments on Replica and Matterport3D Datasets.
Model
Replica
Matterport3D
Heard
Unheard
Heard
Unheard
DSR ↑
DSPL ↑
DSNA ↑
DSR ↑
DSPL ↑
DSNA ↑
DSR ↑
DSPL ↑
DSNA ↑
DSR ↑
DSPL ↑
DSNA ↑
AV-Nav
60.4
34.0
11.5
45.4
26.0
8.7
63.2
34.4
13.4
46.2
24.6
9.3
AV-WaN
66.3
43.1
25.1
51.5
32.8
19.0
84.0
52.3
31.5
70.3
43.0
25.6
SAVi
45.5
34.2
16.1
38.3
27.7
12.7
65.6
44.0
22.0
55.5
35.1
16.0
FSAAVN
63.1
37.5
18.8
49.9
29.9
17.6
70.9
39.5
19.2
62.8
30.7
20.4
TABLE IV: Performance in Complex Environments on Replica and Matterport3D Datasets.
Model
Heard
Unheard
DSR ↑
DSPL ↑
DSNA ↑
DSR ↑
DSPL ↑
DSNA ↑
Replica
CMHM
67.6
43.9
24.5
67.3
33.9
19.1
Ours
79.2
55.6
32.6
64.0
37.6
22.3
Matterport3D
CMHM
91.9
61.6
36.2
74.0
45.8
26.0
TABLE V: Performance of Training in Complex Environments and Testing in Clean Environments.
Model
Heard
Unheard
SR ↑
SPL ↑
SNA ↑
SR ↑
SPL ↑
SNA ↑
CMHM
82.9
61.9
46.8
55.3
42.4
31.6
AGSA-AVN
70.0
54.1
30.0
36.5
26.2
13.1
IRCAM-AVN
–
–
–
61.5
37.8
52.4
DMTF-AVN
–
–
–
63.1
38.4
54.2
Ours
84.3
64.9
48.5
62.6
48.4
35.0
TABLE VI: Performance with stationary sound sources in clean environments on Matterport3D.
Fig. 3: Comparison between the pure visual feature map (a) and the feature map of the Intensity-Calibrated Geometry Encoder (b).
Fig. 4: Attention distribution across different modalities in the Cross-Modal Scene Aggregator (CMSA) on Replica.
Fig. 5: Attention distribution across different modalities in the Cross-Modal Scene Aggregator (CMSA) on Matterport3D.
Fig. 6: Cross-Modal Scene Aggregator modal utilization on Matterport3D through one episode.
Fig. 7: Visualization comparison of navigation paths between our method and the strongest baseline across different unheard sound categories.
Model Variant
Heard
Unheard
Efficiency
DSR ↑
DSPL ↑
DSNA ↑
DSR ↑
DSPL ↑
DSNA ↑
Params (M) ↓
Train FPS ↑
Latency (ms) ↓
Baseline1 ( h=512 )
90.0
62.6
34.5
49.1
29.4
16.1
13.48
35.87
3.19
Baseline2 ( h=256 )
89.3
63.2
35.5
49.3
28.7
16.0
6.16
34.68
3.15
w/ ICGE
91.2
66.4
36.5
54.2
30.6
17.2
6.62
33.65
4.43
w/ CMSA
91.5
64.2
36.0
51.8
31.1
17.8
3.63
33.83
4.39
Full Model (Ours)
93.8
67.5
37.5
63.5
32.1
17.8
3.90
32.22
5.84
TABLE VII: Ablation study and computational efficiency on the Replica dataset. Efficiency measured on an A40 GPU (5 parallel processes; batch size 1). Latency excludes simulator; Train FPS includes it.
Model Variant
Heard
DSR ↑
DSPL ↑
DSNA ↑
Baseline
90.0
62.6
34.5
Variant-L1
87.6
62.7
36.3
Variant-L2
90.8
63.0
36.9
Variant-L3
91.6
65.1
35.8
Full Model (Ours)
93.8
67.5
37.5
TABLE VIII: Ablation Study on Layer Number on Replica Dataset.
Audio-Visual Navigation (AVN) requires an embodied agent to navigate toward a sound source by utilizing both vision and binaural audio. A core challenge arises in complex acoustic environments, where binaural cues become intermittently unreliable, particularly when generalizing to previously unheard sound categories. To address this, we propose RAVN (Reliability-Aware Audio-Visual Navigation), a framework that conditions cross-modal fusion on audio-derived reliability cues, dynamically calibrating the integration of audio and visual inputs. RAVN introduces an Acoustic Geometry Reasoner (AGR) that is trained with geometric proxy supervision. Using a heteroscedastic Gaussian NLL objective, AGR learns observation-dependent dispersion as a practical reliability cue, eliminating the need for geometric labels during inference. Additionally, we introduce Reliability-Aware Geometric Modulation (RAGM), which converts the learned cue into a soft gate to modulate visual features, thereby mitigating cross-modal conflicts. We evaluate RAVN on SoundSpaces using both Replica and Matterport3D environments, and the results show consistent improvements in navigation performance, with notable robustness in the challenging unheard sound setting.
Teng Liu, Yinfeng Yu
Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University · School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China
Audio-visual Navigation refers to an agent utilizing visual and auditory information in complex 3D environments to accomplish target localization and path planning, thereby achieving autonomous navigation. The core challenge of this task lies in the following: how the agent can break free from the dependence on training data and achieve autonomous navigation with good generalization performance when facing changes in environments and sound sources. To address this challenge, we propose an Audio Spatially-Guided Fusion for Audio-Visual Navigation method. First, we design an audio spatial feature encoder, which adaptively extracts target-related spatial state information through an audio intensity attention mechanism; based on this, we introduce an Audio Spatial State Guided Fusion (ASGF) to achieve dynamic alignment and adaptive fusion of multimodal features, effectively alleviating noise interference caused by perceptual uncertainty. Experimental results on the Replica and Matterport3D datasets indicate that our method is particularly effective on unheard tasks, demonstrating improved generalization under unknown sound source distributions.
Xinyu Zhou, Yinfeng Yu
Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University · School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China
In Audio-Visual Navigation (AVN), agents must locate sound sources in unseen 3D environments using visual and auditory cues. However, existing methods often struggle with generalization in unseen scenarios, as they tend to overfit to semantic sound features and specific training environments. To address these challenges, we propose the Binaural Difference Attention with Action Transition Prediction (BDATP) framework, which jointly optimizes perception and policy. Specifically, the Binaural Difference Attention (BDA) module explicitly models interaural differences to enhance spatial orientation, reducing reliance on semantic categories. Simultaneously, the Action Transition Prediction (ATP) task introduces an auxiliary action prediction objective as a regularization term, mitigating environment-specific overfitting. Extensive experiments on the Replica and Matterport3D datasets demonstrate that BDATP can be seamlessly integrated into various mainstream baselines, yielding consistent and significant performance gains. Notably, our framework achieves state-of-the-art Success Rates across most settings, with a remarkable absolute improvement of up to 21.6 percentage points in Replica dataset for unheard sounds. These results underscore BDATP's superior generalization capability and its robustness across diverse navigation architectures.
Jia Li, Yinfeng Yu
Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University · School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China