Navigation in unknown environments to find unforeseen objects has become increasingly feasible with capable vision and language foundation models. However, these models also introduce non-negligible inference latency, which becomes an important concern when agents must operate continuously in the real world. Most state-of-the-art methods are still developed in synchronous simulators, where the environment waits for the agent to act and inference time is effectively free. As a result, agents are often designed around the sequential execution of perception, reasoning, and action, with little regard for time constraints. Under real-time execution, where wall-clock time counts towards the task budget, the inefficiencies of these architectures become clear. We show that recent zero-shot object navigation methods suffer consistent performance degradation under such realistic timing conditions. Motivated by this observation, we propose RTNav, a simple but effective architecture that treats inference latency, asynchronous environment stepping, and bounded compute as explicit design considerations. Evaluated on real-time variants of HM3D-v1, HM3D-v2, and HM3D-OVON, RTNav improves the success rate by up to 11% and the Success weighted by Completion Time by up to 5.1 points over prior work.
Figures & tables
Fig. 1: Left: State-of-the-art navigation agents are developed in synchronous environments, where the simulator is paused when the agent computes the action. This effectively allows unbounded inference times. In reality, the world updates are asynchronous to agent actions; it continues to evolve while the agent is running inference, and inference time costs wall-clock task budget. This fundamentally changes the task. Right: The performance of zero-shot navigation agents designed in synchronous environments consistently degrades under asynchronous evaluation.
Fig. 2: RTNav Overview. (A) The agent follows a modular, asynchronous architecture. The four main modules, perception, mapping, planning and navigation read and write into a shared state bus for communication. A VLM model operates as need-based server and aids decision-making and visual reasoning when required. (B) The modules run at their own native frequency without blocking each other. (C) During frontier selection, we simply query the VLM for most promising direction given the target object. (D) The VLM is queried when candidates are proposed by the constantly running object detector.
Benchmark
Method
Venue
SR ↑
SPL ↑
DTG ↓
SCT ↑
sync
real
Δ
sync
real
Δ
sync
real
Δ
real
OVON
VLFM [ 4 ]
ICRA’24
39.8
38.2
−1.6
24.2
23.7
−0.5
3.50
3.63
+0.13
5.37
RTNav (Ours)
-
44.6
-
-
21.1
-
-
3.36
-
7.82
HM3D-v1
L3MVN [ 2 ]
IROS’23
47.9
47.7
−0.2
22.6
22.5
−0.1
4.62
4.61
−0.02
7.29
VLFM [ 4 ]
ICRA’24
52.6
51.3
−1.3
30.4
30.0
−0.4
4.12
4.21
+0.09
8.95
OpenFMNav [ 3 ]
NAACL-F’24
46.3
32.9
−13.4
20.7
17.3
−3.4
4.78
5.45
+0.66
2.99
TABLE I: Navigation performance across benchmarks. SR = Success Rate, SPL = Success weighted by Path Length, DTG = Distance to Goal, SCT = Success weighted by Completion Time. Bold indicates best performance per metric within each benchmark.
Fig. 3: When agents designed in synchronous environments are evaluated under wall-clock task budget, their performance degrades consistently.
Fig. 4: RTNav showed the minimal idle time with asynchronous modules. Left: Total completion time comprises of the driving time, turning time, and idle time. Idle time consists of any time the agent’s pose is stationary including latency, being stuck due to collision, and moving camera up/down. Right: RTNav showed 14%+ less idle time compared to all other baselines.
Fig. 5: Asynchronous architecture enabled significantly higher visual processing speed. Left: Without other modules blocking, RTNav reached over 20× unique detection frames per second than synchronous baselines. Right: RTNav processed more detection frames per actions, where synchronous agents are limited to around 1 detection per action.
Fig. 6: Blind gaps between detections were significantly shorter in RTNav. Asynchronous architecture missed less visual information.
Fig. 7: Representative snapshots from a successful trajectories of the RTNav agent in the real world. Target object: sink.
Hardware
SR ↑
SPL ↑
SCT ↑
RTX A6000
72.3
30.8
13.84
L40S
72.2
31.5
14.13
Jetson Thor
72.2
31.4
12.73
TABLE II: Sensitivity of RTNav to compute hardware.
VLM
SR ↑
SPL ↑
SCT ↑
Gemma4-12B
66.2
29.9
13.45
Qwen3.5-4B
37.8
10.5
4.56
Qwen3.5-9B
72.3
30.8
13.84
TABLE III: Effect of the VLM used for semantic frontier selection.
Fig. 8: Synchronous benchmark waits for the agent output indefinitely for the next simulation stepping, whereas the real-time benchmark runs both environment and agent node separately in their own frequencies.
TABLE V: Comparison between reproduced and reported results. SR = Success Rate, SPL = Success weighted by Path Length, DTG = Distance to Goal.
Fig. 9: Qualitative examples of VLM object verfication .
HM3D-v1
HM3D-v2
OVON
Evaluation split
val
val
val-unseen
Episodes
2,000
1,000
3,000
Target categories
6
6
49
Robot height (m)
0.88
0.88
1.41
Robot radius (m)
0.18
0.18
0.17
Camera height (m)
0.88
0.88
1.31
TABLE VI: Task and simulator configurations.
Fig. 10: Category-level success rate (SR), success weighted by path length (SPL), and success weighted by completion time (SCT) for RTNav. OVON categories with fewer than 25 episodes are omitted from the visualization. The number of evaluated episodes is shown for each category.
Fig. 11: RTNav failure modes across HM3D-v1, HM3D-v2, and HM3D-OVON. We categorize failures into incorrect STOP decisions and timeouts, with their corresponding causes.