Two-Stage Multi-modal Learning for Dynamic Audio-Visual Navigation: From Physical Alignment to Scene Aggregation

Jia Li; Yinfeng Yu

cs.SDOct 8, 2026

Two-Stage Multi-modal Learning for Dynamic Audio-Visual Navigation: From Physical Alignment to Scene Aggregation

Authors: Jia Li, Yinfeng Yu

Organizations: School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China · Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University

Abstract

Embodied Audio-Visual Navigation (AVN) requires an agent to locate sound targets by jointly leveraging visual and auditory cues. While prior work has achieved notable success with static sound sources, real-world scenarios often involve moving targets, giving rise to the more challenging problem of Dynamic Audio-Visual Navigation (D-AVN). Existing approaches largely rely on semantic auditory features, which limits generalization to unheard sound categories and hinders fine-grained spatial reasoning. To address these challenges, we propose a novel Two-Stage Multi-modal Learning framework for D-AVN. In the Physical Alignment stage, the proposed Intensity-Calibrated Geometry Encoder (ICGE) performs physics-informed alignment of sound intensity with visual geometry to capture category-agnostic spatial correspondence. In the Scene Aggregation stage, a Transformer-based Cross-Modal Scene Aggregator (CMSA) dynamically integrates multi-modal features to construct a robust state representation for policy learning. Extensive experiments on Replica and Matterport3D demonstrate that our method achieves state-of-the-art success rates, while significantly improving generalization and navigation efficiency in complex environments and under unheard sound scenarios.

Figures & tables

Fig. 1: Task comparison. Left: static sound source—target is fixed in a clean environment. Right: moving sound source—target moves with interference, creating a dynamic and more challenging task.

Fig. 2: Model architecture of the two-stage framework. Observations ( $B_{t},V_{t}$ ) are mapped to $G_{t}$ and the acoustic memory $A_{t}$ (this branch in Figure 2 ; not a global sound field). Stage I extracts spatial features via four encoders, where ICGE performs physics-informed physical alignment via intensity–geometry fusion. Stage II aggregates spatial tokens with CMSA to produce $\mathbf{z}_{N}$ . A GRU and PPO-based Actor-Critic complete policy learning.

Encoder	Kernel, Stride
CNN 1	(8, 4), (4, 2), (3, 1)
CNN 2	(8, 4), (4, 2), (3, 2), (2, 1)
CNN 3	(8, 4), (4, 2), (3, 2), (3, 2)
CNN 4	(8, 4), (4, 2), (3, 2), (2, 1)

TABLE I: CNN encoder configurations.

Hyperparameter	Value
Optimizer	Adam
PPO clip	0.1
PPO epochs	4
Mini-batches	1
Value loss coefficient	0.5
Entropy coefficient	0.02

TABLE II: RL training hyperparameters.

Model	Replica						Matterport3D
	Heard			Unheard			Heard			Unheard
	DSR $\uparrow$	DSPL $\uparrow$	DSNA $\uparrow$	DSR $\uparrow$	DSPL $\uparrow$	DSNA $\uparrow$	DSR $\uparrow$	DSPL $\uparrow$	DSNA $\uparrow$	DSR $\uparrow$	DSPL $\uparrow$	DSNA $\uparrow$
AV-Nav	71.0	47.2	16.3	37.4	23.4	7.8	73.0	43.4	18.4	35.5	20.9	8.4
AV-WaN	86.8	55.0	32.6	38.2	24.1	14.2	86.7	58.3	33.6	47.2	30.0	17.4
SAVi	67.6	54.0	25.7	40.8	28.9	13.9	49.0	29.9	11.3	41.6	26.3	10.0
FSAAVN	80.1	52.5	28.8	39.2	28.9	15.8	83.9	47.8	24.4	43.3	27.6	18.7

TABLE III: Performance in Clean Environments on Replica and Matterport3D Datasets.

Model	Replica						Matterport3D
	Heard			Unheard			Heard			Unheard
	DSR $\uparrow$	DSPL $\uparrow$	DSNA $\uparrow$	DSR $\uparrow$	DSPL $\uparrow$	DSNA $\uparrow$	DSR $\uparrow$	DSPL $\uparrow$	DSNA $\uparrow$	DSR $\uparrow$	DSPL $\uparrow$	DSNA $\uparrow$
AV-Nav	60.4	34.0	11.5	45.4	26.0	8.7	63.2	34.4	13.4	46.2	24.6	9.3
AV-WaN	66.3	43.1	25.1	51.5	32.8	19.0	84.0	52.3	31.5	70.3	43.0	25.6
SAVi	45.5	34.2	16.1	38.3	27.7	12.7	65.6	44.0	22.0	55.5	35.1	16.0
FSAAVN	63.1	37.5	18.8	49.9	29.9	17.6	70.9	39.5	19.2	62.8	30.7	20.4

TABLE IV: Performance in Complex Environments on Replica and Matterport3D Datasets.

Replica
Model	Heard			Unheard
	DSR $\uparrow$	DSPL $\uparrow$	DSNA $\uparrow$	DSR $\uparrow$	DSPL $\uparrow$	DSNA $\uparrow$
CMHM	67.6	43.9	24.5	67.3	33.9	19.1
Ours	79.2	55.6	32.6	64.0	37.6	22.3
Matterport3D
CMHM	91.9	61.6	36.2	74.0	45.8	26.0

TABLE V: Performance of Training in Complex Environments and Testing in Clean Environments.

Model	Heard			Unheard
	SR $\uparrow$	SPL $\uparrow$	SNA $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SNA $\uparrow$
CMHM	82.9	61.9	46.8	55.3	42.4	31.6
AGSA-AVN	70.0	54.1	30.0	36.5	26.2	13.1
IRCAM-AVN	–	–	–	61.5	37.8	52.4
DMTF-AVN	–	–	–	63.1	38.4	54.2
Ours	84.3	64.9	48.5	62.6	48.4	35.0

TABLE VI: Performance with stationary sound sources in clean environments on Matterport3D.

Fig. 3: Comparison between the pure visual feature map (a) and the feature map of the Intensity-Calibrated Geometry Encoder (b).

Fig. 4: Attention distribution across different modalities in the Cross-Modal Scene Aggregator (CMSA) on Replica.

Fig. 5: Attention distribution across different modalities in the Cross-Modal Scene Aggregator (CMSA) on Matterport3D.

Fig. 6: Cross-Modal Scene Aggregator modal utilization on Matterport3D through one episode.

Fig. 7: Visualization comparison of navigation paths between our method and the strongest baseline across different unheard sound categories.

Model Variant	Heard			Unheard			Efficiency
Model Variant	DSR $\uparrow$	DSPL $\uparrow$	DSNA $\uparrow$	DSR $\uparrow$	DSPL $\uparrow$	DSNA $\uparrow$	Params (M) $\downarrow$	Train FPS $\uparrow$	Latency (ms) $\downarrow$
Baseline1 ( $h=512$ )	90.0	62.6	34.5	49.1	29.4	16.1	13.48	35.87	3.19
Baseline2 ( $h=256$ )	89.3	63.2	35.5	49.3	28.7	16.0	6.16	34.68	3.15
w/ ICGE	91.2	66.4	36.5	54.2	30.6	17.2	6.62	33.65	4.43
w/ CMSA	91.5	64.2	36.0	51.8	31.1	17.8	3.63	33.83	4.39
Full Model (Ours)	93.8	67.5	37.5	63.5	32.1	17.8	3.90	32.22	5.84

TABLE VII: Ablation study and computational efficiency on the Replica dataset. Efficiency measured on an A40 GPU (5 parallel processes; batch size 1). Latency excludes simulator; Train FPS includes it.

Model Variant	Heard
	DSR $\uparrow$	DSPL $\uparrow$	DSNA $\uparrow$
Baseline	90.0	62.6	34.5
Variant-L1	87.6	62.7	36.3
Variant-L2	90.8	63.0	36.9
Variant-L3	91.6	65.1	35.8
Full Model (Ours)	93.8	67.5	37.5

TABLE VIII: Ablation Study on Layer Number on Replica Dataset.

Audio-Visual Navigation

See all 11 papers

Reliability-Aware Geometric Fusion for Robust Audio-Visual Navigation
Apr 2, 2026Teng Liu, Yinfeng YuAudio-Visual Navigation Embodied Navigation
Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University · School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China
Audio Spatially-Guided Fusion for Audio-Visual Navigation
Apr 2, 2026Xinyu Zhou, Yinfeng YuAudio-Visual Navigation Embodied Navigation
Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University · School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China
Generalizable Audio-Visual Navigation via Binaural Difference Attention and Action Transition Prediction
Apr 6, 2026Jia Li, Yinfeng YuAudio-Visual Navigation Embodied Navigation
Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University · School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China

Apr 2, 2026J/K move · Enter open · S save

Reliability-Aware Geometric Fusion for Robust Audio-Visual Navigation

Teng Liu, Yinfeng Yu

Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University · School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China

Audio-Visual Navigation Embodied Navigation Uncertainty-Aware Multimodal Fusion Multimodal Robustness

Open paper AskPDF arXiv

Audio-Visual Navigation (AVN) requires an embodied agent to navigate toward a sound source by utilizing both vision and binaural audio. A core challenge arises in complex acoustic environments, where binaural cues become intermittently unreliable, particularly when generalizing to previously unheard sound categories. To address this, we propose RAVN (Reliability-Aware Audio-Visual Navigation), a framework that conditions cross-modal fusion on audio-derived reliability cues, dynamically calibrating the integration of audio and visual inputs. RAVN introduces an Acoustic Geometry Reasoner (AGR) that is trained with geometric proxy supervision. Using a heteroscedastic Gaussian NLL objective, AGR learns observation-dependent dispersion as a practical reliability cue, eliminating the need for geometric labels during inference. Additionally, we introduce Reliability-Aware Geometric Modulation (RAGM), which converts the learned cue into a soft gate to modulate visual features, thereby mitigating cross-modal conflicts. We evaluate RAVN on SoundSpaces using both Replica and Matterport3D environments, and the results show consistent improvements in navigation performance, with notable robustness in the challenging unheard sound setting.

Abstract

Audio-Visual Navigation

Explore similar work

Reliability-Aware Geometric Fusion for Robust Audio-Visual Navigation

Audio Spatially-Guided Fusion for Audio-Visual Navigation

Generalizable Audio-Visual Navigation via Binaural Difference Attention and Action Transition Prediction