Reliability-Aware Geometric Fusion for Robust Audio-Visual Navigation

Teng Liu; Yinfeng Yu

cs.SDApr 2, 2026

Reliability-Aware Geometric Fusion for Robust Audio-Visual Navigation

Authors: Teng Liu, Yinfeng Yu

Organizations: Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University · School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China

Abstract

Audio-Visual Navigation (AVN) requires an embodied agent to navigate toward a sound source by utilizing both vision and binaural audio. A core challenge arises in complex acoustic environments, where binaural cues become intermittently unreliable, particularly when generalizing to previously unheard sound categories. To address this, we propose RAVN (Reliability-Aware Audio-Visual Navigation), a framework that conditions cross-modal fusion on audio-derived reliability cues, dynamically calibrating the integration of audio and visual inputs. RAVN introduces an Acoustic Geometry Reasoner (AGR) that is trained with geometric proxy supervision. Using a heteroscedastic Gaussian NLL objective, AGR learns observation-dependent dispersion as a practical reliability cue, eliminating the need for geometric labels during inference. Additionally, we introduce Reliability-Aware Geometric Modulation (RAGM), which converts the learned cue into a soft gate to modulate visual features, thereby mitigating cross-modal conflicts. We evaluate RAVN on SoundSpaces using both Replica and Matterport3D environments, and the results show consistent improvements in navigation performance, with notable robustness in the challenging unheard sound setting.

Figures & tables

Fig. 1: Paradigm comparison. (a) naive deterministic fusion vs. (b) our reliability-aware RAVN framework.

Fig. 2: The RAVN Framework. Multi-modal observations $o_{t}$ are encoded into $f_{a}$ and $f_{v}$ . The AGR module estimates geometric embeddings $g_{t}$ and predictive uncertainty ( $\mu_{t},\sigma_{t}$ ), supervised by Ground Truth (GT) labels via an auxiliary loss. The RAGM module then uses $g_{t}$ to dynamically modulate visual features into $f^{\prime}_{v}$ . A recurrent policy (GRU) aggregates these components into the hidden state $s_{t}$ for end-to-end action ( $a_{t}$ ) selection.

Fig. 3: RAGM module architecture. Geometric features are transformed into a reliability mask, which applies element-wise modulation to visual features to produce modified visual representations.

	Replica						Matterport3D
Method	Heard			Unheard			Heard			Unheard
	SR $\uparrow$	SPL $\uparrow$	SNA $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SNA $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SNA $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SNA $\uparrow$
Random Agent [ 17 ]	18.5	4.9	1.8	18.5	4.9	1.8	9.1	2.1	0.8	9.1	2.1	0.8
Frontier Waypoints [ 17 ]	63.9	44.0	35.2	14.8	6.5	5.1	42.8	30.6	22.2	16.4	10.9	8.1
Direction Follower [ 17 ]	72.0	54.7	41.1	17.2	11.1	8.4	41.2	32.3	23.8	18.0	13.9	10.7
Supervised Waypoints [ 17 ]	88.1	59.1	48.5	43.1	14.1	10.1	36.2	21.0	16.2	8.8	4.1	2.9

TABLE I: Quantitative comparison on Replica and Matterport3D datasets. We report Success Rate (SR), Success weighted by Path Length (SPL), and Success weighted by Number of Actions (SNA). Best results are highlighted in bold.

Fig. 4: Qualitative results. (a) Top-down trajectory comparisons between the AV-NaV baseline and our RAVN in representative Replica and Mp3D episodes, with SPL shown for each run. (b) Additional rollouts of RAVN across various layouts and start-goal configurations.

	Replica
Model Variant	Heard			Unheard
	SR( $\uparrow$ )	SPL( $\uparrow$ )	SNA( $\uparrow$ )	SR( $\uparrow$ )	SPL( $\uparrow$ )	SNA( $\uparrow$ )
AV-Nav [ 1 ]	93.0	76.1	44.7	47.3	36.8	21.5
+AGR (MSE)	93.7	76.5	49.7	47.7	36.9	21.6
+AGR (NLL)	95.2	79.0	50.1	51.9	39.1	22.1
RAVN (Ours)	97.0	80.2	52.4	53.1	40.3	23.4

TABLE II: Ablation Study on Replica Dataset. We compare different variants to validate the effectiveness of our proposed modules.

	Matterport3D
Model Variant	Heard			Unheard
	SR( $\uparrow$ )	SPL( $\uparrow$ )	SNA( $\uparrow$ )	SR( $\uparrow$ )	SPL( $\uparrow$ )	SNA( $\uparrow$ )
AV-Nav [ 1 ]	68.8	52.3	29.6	33.5	21.9	10.4
+AGR (MSE)	69.4	50.9	27.0	35.3	23.8	12.4
+AGR (NLL)	69.7	54.3	29.4	35.4	24.1	13.0
RAVN (Ours)	70.9	55.6	33.2	35.6	25.9	13.6

TABLE III: Ablation Study on Mp3d Dataset. We compare different variants to validate the effectiveness of our proposed modules.

Audio-Visual Navigation

See all 11 papers

Audio Spatially-Guided Fusion for Audio-Visual Navigation
Apr 2, 2026Xinyu Zhou, Yinfeng YuAudio-Visual Navigation Embodied Navigation
Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University · School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China
Two-Stage Multi-modal Learning for Dynamic Audio-Visual Navigation: From Physical Alignment to Scene Aggregation
Oct 8, 2026Jia Li, Yinfeng YuAudio-Visual Navigation Audio-Visual Alignment
School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China · Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University
Generalizable Audio-Visual Navigation via Binaural Difference Attention and Action Transition Prediction
Apr 6, 2026Jia Li, Yinfeng YuAudio-Visual Navigation Embodied Navigation
Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University · School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China

Apr 2, 2026J/K move · Enter open · S save

Audio Spatially-Guided Fusion for Audio-Visual Navigation

Xinyu Zhou, Yinfeng Yu

Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University · School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China

Audio-Visual Navigation Embodied Navigation Uncertainty-Aware Multimodal Fusion

Open paper AskPDF arXiv

Audio-visual Navigation refers to an agent utilizing visual and auditory information in complex 3D environments to accomplish target localization and path planning, thereby achieving autonomous navigation. The core challenge of this task lies in the following: how the agent can break free from the dependence on training data and achieve autonomous navigation with good generalization performance when facing changes in environments and sound sources. To address this challenge, we propose an Audio Spatially-Guided Fusion for Audio-Visual Navigation method. First, we design an audio spatial feature encoder, which adaptively extracts target-related spatial state information through an audio intensity attention mechanism; based on this, we introduce an Audio Spatial State Guided Fusion (ASGF) to achieve dynamic alignment and adaptive fusion of multimodal features, effectively alleviating noise interference caused by perceptual uncertainty. Experimental results on the Replica and Matterport3D datasets indicate that our method is particularly effective on unheard tasks, demonstrating improved generalization under unknown sound source distributions.

Abstract

Audio-Visual Navigation

Explore similar work

Audio Spatially-Guided Fusion for Audio-Visual Navigation

Two-Stage Multi-modal Learning for Dynamic Audio-Visual Navigation: From Physical Alignment to Scene Aggregation

Generalizable Audio-Visual Navigation via Binaural Difference Attention and Action Transition Prediction