We study language-conditioned local navigation, where a robot must navigate to a text-specified target visible in its initial observation and stop at the described location. End-to-end navigation policies require diverse training data to generalize across scenes, yet real-robot data is costly to collect and simulated data leaves a sim-to-real gap. Internet egocentric videos offer a scalable source of real-world navigation experience. Our key insight is to learn goal-directed visual dynamics from Internet egocentric videos, and use these dynamics to support robot action generation. To this end, we propose ScaleNav, a unified world-action framework with three-stage training. We first perform instruction-conditioned video pretraining on these videos to learn how observations evolve as the camera approaches instructed targets across diverse real-world scenes. Action alignment then uses action-labeled simulated data to train the model to predict executable action chunks. Joint fine-tuning subsequently optimizes all trainable components together for navigation. ScaleNav achieves success rates of 86.5% on a single-view adaptation of Short-Horizon OVON and 94.9% on our HabitatGS-LN benchmark, outperforming the evaluated baselines on both benchmarks. We further deploy and evaluate ScaleNav on a real quadruped robot, with results supporting the model's ability to generalize to real-world navigation. Code and videos are available at https://zju3dv.github.io/ScaleNav/
Figures & tables
Figure 1: Overview of ScaleNav. ScaleNav learns navigation priors from Internet egocentric videos through action-free video pretraining, then uses action-labeled simulated data for action alignment and joint fine-tuning. It jointly predicts future observations and action chunks to follow navigation instructions. The right panel illustrates deployment on a real quadruped robot.
Figure 2: Overview of ScaleNav. (a) Language-conditioned local navigation. The model predicts video and action chunks from the instruction, current observation, and memory using chunk-wise causal attention. The memory contains an initial-frame anchor and a sliding window of recent video and action chunks. (b) Three-stage training. Video pretraining uses action-free real videos, followed by action alignment and joint fine-tuning with action-labeled simulation data. During action alignment, the self-attention layers within the DiT, the action head, and the action embedding MLP (not shown) are trainable. The snowflake on the outer DiT block marks the remaining DiT layers as frozen. Joint fine-tuning updates the DiT and both heads. For clarity, the initial bidirectional adaptation in Stage 1 and Internet egocentric video replay in Stages 2 and 3 are omitted.
Method
Short-Horizon OVON †
HabitatGS-LN
Amortized latency (ms) ↓
SR (%) ↑
SPL (%) ↑
SR (%) ↑
SPL (%) ↑
LocateAnything + NavDP
74.4
37.2
82.8
61.0
184
LocateAnything + ApexNav
73.3
64.1
75.8
66.8
511
StreamVLN (zero-shot)
61.1
51.2
48.4
40.0
87
Uni-NaVid (zero-shot)
11.6
9.9
8.9
8.2
99
StreamVLN (fine-tuned)
79.6
62.9
75.8
59.5
–
Table 1: Comparison on Short-Horizon OVON and HabitatGS-LN. “Internet videos” refers to our curated Internet egocentric videos. Best results are in bold . Dashes indicate model variants not profiled separately for latency. † We evaluate a single-view subset of Short-Horizon OVON under our evaluation protocol.
Variant
Video pretraining
Pseudo- action
Anchor
Alignment
Replay
SR (%) ↑
SPL (%) ↑
w/o Internet videos
–
–
✓
✓
–
90.4
80.8
Pseudo-action training
–
✓
✓
✓
–
73.9
67.1
w/o video pretraining
–
–
✓
✓
✓
90.4
83.1
w/o initial-frame anchor
✓
–
–
✓
✓
89.8
83.1
w/o action alignment
✓
–
✓
–
✓
93.6
85.1
w/o video replay
✓
–
✓
✓
–
94.3
84.2
Table 2: Internet egocentric video utilization and component ablations on HabitatGS-LN. Video pretraining and Alignment denote Stages 1 and 2, respectively. Pseudo-action denotes action labels estimated from these videos, Anchor denotes the initial-frame anchor, and Replay denotes Internet egocentric video replay during Stages 2 and 3. Best results are highlighted in bold .
Method
SR (%) ↑
Hall
Lounge
Office
Overall
StreamVLN (fine-tuned)
25.0
60.0
40.0
41.7
Uni-NaVid (fine-tuned)
60.0
75.0
20.0
51.7
Ours (w/o Internet videos)
75.0
60.0
60.0
65.0
Ours
85.0
85.0
80.0
83.3
Table 3: Real-robot success rates (60 trials per method). Best results are in bold .
Figure 3: Qualitative results of ScaleNav in an exhibition hall (a), an open lounge area (b), and an office area (c). The robot follows language instructions to reach the specified targets.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Qualitative comparison of Wan 2.2 and our Stage 1 model with autoregressive (AR) video generation. Red boxes mark the navigation targets.
Configuration
Video pretraining
Action alignment
Joint fine-tuning
Updates
10,000
2,500
6,500
Global batch size
64
64
32
Peak learning rate
1×10−5
1×10−5
5×10−6
Video replay probability
–
10%
20%
Appendix
Table 4: Training configuration. Update counts refer to the checkpoints used in the final model’s training sequence. Video replay probabilities specify dataset sampling, not loss weights. The dash indicates that Stage 1 uses only Internet egocentric videos, rather than a replay mixture.
Steps
Unoptimized
Optimized
Speedup
4
1791
684
2.6 ×
8
3272
1034
3.2 ×
20
7731
2080
3.7 ×
50
18846
4693
4.0 ×
Appendix
Table 5: Mean per-chunk policy-call latency on an RTX 4090 (ms), including history updates. Left: unoptimized and fully optimized implementations at different denoising step counts. Right: cumulative optimizations at four steps. Speedups are relative to the unoptimized implementation at the same step count.
Figure 5: Examples from the two training datasets, paired with their navigation instructions. Frames are shown in temporal order. Red boxes mark the targets for illustration only.
Figure 6: Example tasks in HabitatGS-LN with their final navigation instructions. Red boxes mark the specified targets.
Task
StreamVLN (fine-tuned)
Uni-NaVid (fine-tuned)
Ours (w/o Internet videos)
Ours
Hall
Display counter
2/5
4/5
4/5
5/5
Rocket model
1/5
1/5
2/5
3/5
Screen on the rack
0/5
4/5
5/5
5/5
Second door
2/5
3/5
4/5
4/5
Lounge
Appendix
Table 6: Per-task real-robot results, reported as successes / trials. Best results in each row are in bold .
Figure 7: Real-robot task views and instructions, grouped by environment. Red boxes mark the targets. Target boxes and privacy masks are for illustration only and are not policy inputs.
Enabling robots to navigate open-world environments via natural language is critical for general-purpose autonomy. Yet, Vision-Language Navigation has relied on end-to-end policies trained on expensive, embodiment-specific robot data. While recent foundation models trained on vast simulation data show promise, the challenge of scaling and generalizing persists due to the limited scene diversity and visual fidelity in simulation. To address this gap, we propose ImagiNav, a novel hierarchical paradigm that formulates navigation in visual space. Instead of predicting waypoints, ImagiNav synthesizes a future egocentric video conditioned on language instructions, serving as a high-level plan, interpreted by an inverse dynamics model to extract metric trajectories for execution. By decoupling planning from robot actuation, the paradigm enables direct utilization of diverse in-the-wild navigation videos. To support this, we develop an auto-labeling data pipeline that enhances motion annotation accuracy. ImagiNav demonstrates strong zero-shot transfer to robot navigation without requiring robot demonstrations, paving the way for generalist robots that learn navigation directly from unlabeled, open-world data. The project page is available at: https://j1dan.github.io/ImagiNav
Jie Chen, Yuxin Cai, Yizhuo Wang +5
Department of Mechanical Engineering, National University of Singapore, Singapore · Institute for Infocomm Research (I2R), Agency for Science, Technology and Research (A*STAR), Singapore · Nanyang Technological University (NTU), Singapore
Open-world navigation requires robots to make decisions in complex everyday environments while adapting to flexible task requirements. Conventional navigation approaches often rely on dense 3D reconstruction and hand-crafted goal metrics, which limits their generalization across tasks and environments. Recent advances in vision-language navigation (VLN) and vision-language-action (VLA) models enable end-to-end policies conditioned on natural language, but typically require interactive training, large-scale data collection, or task-specific fine-tuning with a mobile agent. We formulate navigation as a sparse subgoal identification and reaching problem and observe that providing visual anchoring targets for high-level semantic priors enables highly efficient goal-conditioned navigation. Based on this insight, we select visual frontiers as semantic anchors and propose OpenFrontier, a navigation framework that requires no task-specific training or fine-tuning and seamlessly integrates diverse vision-language prior models. OpenFrontier enables efficient navigation with a lightweight system design, without dense 3D semantic mapping, task-specific policy training, or model fine-tuning. We evaluate OpenFrontier across multiple navigation benchmarks and demonstrate strong zero-shot performance, as well as effective real-world deployment on a mobile robot.
Esteban Padilla-Cerdio, Boyang Sun, Marc Pollefeys +1
1ETH Zurich · 2Microsoft Spatial AI Lab · University of Bonn
Vision-Language Navigation (VLN) requires long-horizon instruction-consistent planning and short-horizon local planning in current geometry. Video-action models (VAMs) offer a promising way to learn predictive representations for navigation. However, existing VAMs jointly predict visual futures and local trajectories in a single process, even though they operate at different paces: future prediction remains stable across multiple updates while local trajectories must react to fresh observations. To bridge this gap, we introduce HAVANA, a dual-system VAM that decouples predictive reasoning from reactive local planning. A slow system produces a compact language-conditioned latent plan from observation history, while a fast system combines the latest plan with the current observation to generate local trajectories. Their asynchronous execution enables rapid adaptation without repeated future prediction, and no future video is generated during inference. Experiments on OmniNavBench demonstrate state-of-the-art navigation performance over prior baselines, while real-world deployments further show robust generalization to unseen environments across different robot embodiments. Our code and checkpoints will be available at https://github.com/Anastasiawd/HAVANA.
Dongzhihan Wang, Yi Du, Chen Wang +2
Tongji University · SAIR Lab · Shanghai University