Organizations: University of California, Irvine · University of Pennsylvania · Carnegie Mellon University · Purdue University Northwest · Michigan State University · Independent Researcher · Dartmouth College · University of California, Los Angeles · University of Notre Dame
Vision-language-action models for embodied navigation typically rely on RGB observations, limiting their applicability in lights-out industrial environments where visible illumination may be unavailable. We study illumination-independent navigation using spatial audio for target localization and depth for obstacle-aware geometry. To support this setting, we introduce DarkNav-Bench, a benchmark with 24 industrial scenes and four progressively challenging acoustic conditions. We further propose ADAPT, an audio-depth action-planning framework equipped with spatial-audio perception and optimized for closed-loop navigation through reinforcement learning (RL). Our key RL contribution, transition-aware reward shaping (TARS), defines a pose-aware shortest-action potential over a position-heading graph. Unlike conventional geodesic-progress rewards, TARS accounts for both feasible movements and in-place rotations, providing action-consistent feedback for policy optimization. Experiments show that ADAPT consistently outperforms training-based navigation baselines and zero-shot multimodal models on both seen and unseen scenes. On unseen environments, it achieves 90.6% success under Clean conditions and retains 60.0% success under the most challenging Localized Noise setting. These results demonstrate the potential of spatial audio and depth as complementary modalities for reliable RGB-free navigation in lights-out industrial environments.
Figures & tables
Figure 1: Motivation of ADAPT. Existing RGB-based VLA agents depend on visible illumination and therefore require continuous lighting in lights-out factories. ADAPT instead combines spatial audio for target direction with depth sensing for local obstacle geometry, enabling reliable and energy-efficient navigation without visible light.
Figure 2: Overview of ADAPT . The model is trained in three stages: spatial-audio adaptation, multimodal SFT for audio-depth action prediction, and PPO-based closed-loop optimization with TARS. At inference time, the final policy directly maps spatial audio and depth observations to discrete navigation actions.
Method
Clean
Prop
DN
LN
SR ↑
SPL ↑
SNA ↑
FDG ↓
SR ↑
SPL ↑
SNA ↑
FDG ↓
SR ↑
SPL ↑
SNA ↑
FDG ↓
SR ↑
SPL ↑
SNA ↑
FDG ↓
1. Oracle-assisted methods
1.1 Zero-shot (strict stopping)
Qwen 2.5-Omni Qwen (2025)
19.00
13.60
7.70
15.19
16.00
13.00
6.70
14.96
22.00
17.29
9.35
14.11
19.37
13.91
8.10
15.64
Qwen 3-Omni Xu et al. (2025)
72.00
71.68
59.48
6.00
73.00
72.76
60.87
5.18
70.00
69.83
58.02
5.50
70.20
69.78
59.31
6.53
2. Non-oracle methods
Table 1: Navigation performance in seen scenes.
Method
Clean
Prop
DN
LN
SR ↑
SPL ↑
SNA ↑
FDG ↓
SR ↑
SPL ↑
SNA ↑
FDG ↓
SR ↑
SPL ↑
SNA ↑
FDG ↓
SR ↑
SPL ↑
SNA ↑
FDG ↓
1. Oracle-assisted methods (zero-shot)
1.1 Zero-shot (strict stopping)
Qwen 2.5-Omni Qwen (2025)
14.29
11.40
6.12
16.63
12.71
10.06
5.55
16.79
15.10
12.20
6.41
16.25
17.10
14.34
7.88
16.30
Qwen 3-Omni Xu et al. (2025)
62.14
61.79
52.12
6.36
62.38
61.93
52.87
6.40
63.33
62.92
53.27
6.16
62.62
62.24
52.70
6.31
2. Non-oracle methods
Table 2: Navigation performance in unseen scenes.
Figure 3: Qualitative comparison of navigation trajectories. Each panel reports the number of executed actions, FDG, and traveled path length. More examples are included in the appendix (see Fig. 6 ).
Table 3: Ablation study under the LN setting. (a) SFT-only ablation. (b) RL ablation.
Metric
Clean
Prop
DN @ 20 dB
DN @ 10 dB
DN @ 0 dB
DN @ −10 dB
MAE ↓
35.45∘
25.03∘
28.45∘
32.02∘
34.06∘
34.17∘
Median error ↓
13.23∘
10.93∘
10.74∘
11.47∘
14.93∘
17.89∘
Table 4: DoA estimation under different difficulties.
SNR (dB)
SR ↑
SPL ↑
SNA ↑
FDG ↓
10
67.10
59.30
39.00
7.89
0
61.90
51.00
32.60
9.18
-5
52.60
41.50
27.40
12.42
-10
31.20
23.70
14.40
18.39
Table 5: Navigation performance under different SNR under the DN setting.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Material
Abs.-L
Abs.-M
Abs.-H
Scat.
Trans.-L
Trans.-M
Trans.-H
Concrete
0.05
0.07
0.08
0.05
0.015
0.002
0.001
Metal
0.20
0.07
0.06
0.05
0.200
0.025
0.010
Wood
0.11
0.07
0.06
0.05
0.070
0.014
0.005
Plastic
0.03
0.04
0.05
0.05
0.080
0.030
0.015
Rubber
0.05
0.10
0.15
0.10
0.020
0.010
0.005
Appendix
Table 6: Acoustic material parameters used for geometry-dependent sound propagation. Abs.-L/M/H denote low-, mid-, and high-frequency absorption coefficients; Scat. denotes the scattering coefficient; and Trans.-L/M/H denote low-, mid-, and high-frequency transmission coefficients.
Source
SPL snd @1 m unweighted
SPL snd @1 m A-weighted
10–90% Energy Band
Energy in 500–900 Hz
Temporal Structure
Target
95.1
93.6
560–840 Hz
96.0%
50 ms sweep, 20 Hz
Conveyor
72.0
65.6
178–477 Hz
5.8%
steady hum
Forklift
68.0
58.4
110–265 Hz
0.15%
steady hum
Reach truck
66.0
60.6
183–443 Hz
5.6%
steady hum
AGV
62.0
60.7
551–1480 Hz
74.7%
steady hum
Appendix
Table 7: Acoustic characteristics of the target and localized equipment-noise sources. SPL snd values are nominal levels at 1 m under our waveform calibration. The 10–90% energy band denotes the frequency range containing the central 80% of spectral energy, and the overlap column reports the fraction of total energy within the target chirp band (500–900 Hz).
Figure 4: Comparison of geodesic progress and TARS. (a). In-place rotations leave geodesic distance unchanged, providing no feedback on heading changes. (b). TARS uses a position–heading action graph to measure the minimum remaining action count ( N(s) ). (c). With γΦ=1 , TARS rewards turns that reduce N(s) and penalizes those that increase it, distinguishing helpful from counterproductive rotations at the same position.
Input / Information
Zero-shot w/o Oracle Info.
Zero-shot w/ Oracle Info.
Current FOA audio
✓
✓
Current depth image
✓
✓
Recent action history
✓
–
Previous sensory observations
–
–
Ground-truth source bearing
–
✓
Ground-truth source distance
–
✓
Appendix
Table 8: Input information provided to the zero-shot multimodal baselines. Oracle bearing and distance are used only in the oracle-assisted diagnostic and are never available to ADAPT at inference time.
Figure 5: DoA estimation accuracy across difficulty levels. Accuracy is the percentage of predictions with angular error within 10∘ , 20∘ , or 45∘ of target direction.
Figure 6: More qualitative comparison of navigation trajectories.
Embodied navigation is moving toward long-horizon missions, yet existing long-horizon benchmarks are largely acoustically silent, and audio-visual navigation tasks typically focus on a single goal. We introduce LH-AVLN, a benchmark for Long-Horizon Audio-Visual-Language Navigation that combines multi-goal mission execution, heterogeneous goal specifications, and persistent spatialized acoustic cues. In LH-AVLN, an agent receives a global mission of two to four goals specified by category, language description, or reference image, and navigates with RGB-D observations, pose, and binaural audio in indoor 3D environments. The benchmark supports both ordered and unordered missions, where alternating goal-associated sounds can guide non-line-of-sight search but may also become distractors as mission progress changes. We further develop PAG-Nav, a training-free reference agent that maintains a temporal uniform semantic map and performs progressive goal-state planning, using sound for search while reserving completion for visual-semantic verification. Experiments show that existing vision-language, memory-based, and audio-visual agents struggle to complete full LH-AVLN missions, and that PAG-Nav provides a stronger diagnostic baseline while leaving substantial room for future progress.
Rufeng Chen, Yue Chang, Zili Shao +4
The Hong Kong University of Science and Technology (Guangzhou) · Jilin University
The ability to navigate toward sound sources extends a robot's reach beyond its visual field, enabling response to auditory events in unknown environments. To equip robots with this capability, existing methods couple acoustic and visual information through joint audio-visual learning in acoustic simulators. However, acoustic simulation is both low-fidelity and expensive, producing a domain gap that prevents reliable real-world deployment, while the discrete action spaces inherited from grid-based simulators introduce an additional kinematic gap on physical robots. To alleviate these issues, we propose BCNav, a decoupled framework that separates the acoustic module from the learned navigation policy using direction-of-arrival (DOA) estimation: an estimator provides a scalar bearing to the sound source, so the navigation policy only processes depth images and a bearing angle, two inputs whose domain gaps are well characterized. We collect shortest-path demonstrations with calibrated bearing noise injection and train the policy via imitation learning to output continuous velocity commands directly executable on ground robots. We demonstrate the method in simulation and on a physical robot, navigating unknown environments without any acoustic fine-tuning, prior mapping, or real-world audio data collection. Code is available at https://github.com/york1to/bcnav.
Yaozhong Kang, Jiang Wang, Takeshi Ashizawa +2
Department of Systems and Control Engineering, Institute of Science Tokyo, Japan
In Audio-Visual Navigation (AVN), agents must locate sound sources in unseen 3D environments using visual and auditory cues. However, existing methods often struggle with generalization in unseen scenarios, as they tend to overfit to semantic sound features and specific training environments. To address these challenges, we propose the Binaural Difference Attention with Action Transition Prediction (BDATP) framework, which jointly optimizes perception and policy. Specifically, the Binaural Difference Attention (BDA) module explicitly models interaural differences to enhance spatial orientation, reducing reliance on semantic categories. Simultaneously, the Action Transition Prediction (ATP) task introduces an auxiliary action prediction objective as a regularization term, mitigating environment-specific overfitting. Extensive experiments on the Replica and Matterport3D datasets demonstrate that BDATP can be seamlessly integrated into various mainstream baselines, yielding consistent and significant performance gains. Notably, our framework achieves state-of-the-art Success Rates across most settings, with a remarkable absolute improvement of up to 21.6 percentage points in Replica dataset for unheard sounds. These results underscore BDATP's superior generalization capability and its robustness across diverse navigation architectures.
Jia Li, Yinfeng Yu
Joint Research Laboratory for Embodied Intelligence, Xinjiang University · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University · School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China