ScaleNav: Scalable Internet Video Pretraining for Language-Conditioned Local Navigation

Yihui Yang; Yuanhong Yu; Jiawei Wang; Zhongrui Yu; Jia Shi; Kun Zhan; Ruizhen Hu; Xiaowei Zhou

cs.ROOct 19, 2026

ScaleNav: Scalable Internet Video Pretraining for Language-Conditioned Local Navigation

Authors: Yihui Yang, Yuanhong Yu, Jiawei Wang, Zhongrui Yu, Jia Shi, Kun Zhan, Ruizhen Hu, Xiaowei Zhou,

Organizations: Zhejiang University · Li Auto Inc. · Shenzhen University

Abstract

We study language-conditioned local navigation, where a robot must navigate to a text-specified target visible in its initial observation and stop at the described location. End-to-end navigation policies require diverse training data to generalize across scenes, yet real-robot data is costly to collect and simulated data leaves a sim-to-real gap. Internet egocentric videos offer a scalable source of real-world navigation experience. Our key insight is to learn goal-directed visual dynamics from Internet egocentric videos, and use these dynamics to support robot action generation. To this end, we propose ScaleNav, a unified world-action framework with three-stage training. We first perform instruction-conditioned video pretraining on these videos to learn how observations evolve as the camera approaches instructed targets across diverse real-world scenes. Action alignment then uses action-labeled simulated data to train the model to predict executable action chunks. Joint fine-tuning subsequently optimizes all trainable components together for navigation. ScaleNav achieves success rates of 86.5% on a single-view adaptation of Short-Horizon OVON and 94.9% on our HabitatGS-LN benchmark, outperforming the evaluated baselines on both benchmarks. We further deploy and evaluate ScaleNav on a real quadruped robot, with results supporting the model's ability to generalize to real-world navigation. Code and videos are available at https://zju3dv.github.io/ScaleNav/

Figures & tables

Figure 1: Overview of ScaleNav. ScaleNav learns navigation priors from Internet egocentric videos through action-free video pretraining, then uses action-labeled simulated data for action alignment and joint fine-tuning. It jointly predicts future observations and action chunks to follow navigation instructions. The right panel illustrates deployment on a real quadruped robot.

Figure 2: Overview of ScaleNav. (a) Language-conditioned local navigation. The model predicts video and action chunks from the instruction, current observation, and memory using chunk-wise causal attention. The memory contains an initial-frame anchor and a sliding window of recent video and action chunks. (b) Three-stage training. Video pretraining uses action-free real videos, followed by action alignment and joint fine-tuning with action-labeled simulation data. During action alignment, the self-attention layers within the DiT, the action head, and the action embedding MLP (not shown) are trainable. The snowflake on the outer DiT block marks the remaining DiT layers as frozen. Joint fine-tuning updates the DiT and both heads. For clarity, the initial bidirectional adaptation in Stage 1 and Internet egocentric video replay in Stages 2 and 3 are omitted.

Method	Short-Horizon OVON †		HabitatGS-LN		Amortized latency (ms) $\downarrow$
Method	SR (%) $\uparrow$	SPL (%) $\uparrow$	SR (%) $\uparrow$	SPL (%) $\uparrow$	Amortized latency (ms) $\downarrow$
LocateAnything + NavDP	74.4	37.2	82.8	61.0	184
LocateAnything + ApexNav	73.3	64.1	75.8	66.8	511
StreamVLN (zero-shot)	61.1	51.2	48.4	40.0	87
Uni-NaVid (zero-shot)	11.6	9.9	8.9	8.2	99
StreamVLN (fine-tuned)	79.6	62.9	75.8	59.5	–

Table 1: Comparison on Short-Horizon OVON and HabitatGS-LN. “Internet videos” refers to our curated Internet egocentric videos. Best results are in bold . Dashes indicate model variants not profiled separately for latency.

\dagger

We evaluate a single-view subset of Short-Horizon OVON under our evaluation protocol.

Variant	Video pretraining	Pseudo- action	Anchor	Alignment	Replay	SR (%) $\uparrow$	SPL (%) $\uparrow$
w/o Internet videos	–	–	$\checkmark$	$\checkmark$	–	90.4	80.8
Pseudo-action training	–	$\checkmark$	$\checkmark$	$\checkmark$	–	73.9	67.1
w/o video pretraining	–	–	$\checkmark$	$\checkmark$	$\checkmark$	90.4	83.1
w/o initial-frame anchor	$\checkmark$	–	–	$\checkmark$	$\checkmark$	89.8	83.1
w/o action alignment	$\checkmark$	–	$\checkmark$	–	$\checkmark$	93.6	85.1
w/o video replay	$\checkmark$	–	$\checkmark$	$\checkmark$	–	94.3	84.2

Table 2: Internet egocentric video utilization and component ablations on HabitatGS-LN. Video pretraining and Alignment denote Stages 1 and 2, respectively. Pseudo-action denotes action labels estimated from these videos, Anchor denotes the initial-frame anchor, and Replay denotes Internet egocentric video replay during Stages 2 and 3. Best results are highlighted in bold .

Method	SR (%) $\uparrow$
Method	Hall	Lounge	Office	Overall
StreamVLN (fine-tuned)	25.0	60.0	40.0	41.7
Uni-NaVid (fine-tuned)	60.0	75.0	20.0	51.7
Ours (w/o Internet videos)	75.0	60.0	60.0	65.0
Ours	85.0	85.0	80.0	83.3

Table 3: Real-robot success rates (60 trials per method). Best results are in bold .

Figure 3: Qualitative results of ScaleNav in an exhibition hall (a), an open lounge area (b), and an office area (c). The robot follows language instructions to reach the specified targets.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Qualitative comparison of Wan 2.2 and our Stage 1 model with autoregressive (AR) video generation. Red boxes mark the navigation targets.

Configuration	Video pretraining	Action alignment	Joint fine-tuning
Updates	10,000	2,500	6,500
Global batch size	64	64	32
Peak learning rate	$1\times 10^{-5}$	$1\times 10^{-5}$	$5\times 10^{-6}$
Video replay probability	–	10%	20%

Appendix

Table 4: Training configuration. Update counts refer to the checkpoints used in the final model’s training sequence. Video replay probabilities specify dataset sampling, not loss weights. The dash indicates that Stage 1 uses only Internet egocentric videos, rather than a replay mixture.

Steps	Unoptimized	Optimized	Speedup
4	1791	684	2.6 $\times$
8	3272	1034	3.2 $\times$
20	7731	2080	3.7 $\times$
50	18846	4693	4.0 $\times$

Appendix

Table 5: Mean per-chunk policy-call latency on an RTX 4090 (ms), including history updates. Left: unoptimized and fully optimized implementations at different denoising step counts. Right: cumulative optimizations at four steps. Speedups are relative to the unoptimized implementation at the same step count.

Panel of Figure 5 — Figure 5: Examples from the two training datasets, paired with their navigation instructions. Frames are shown in temporal order. Red boxes mark the targets for illustration only.

Panel of Figure 6 — Figure 6: Example tasks in HabitatGS-LN with their final navigation instructions. Red boxes mark the specified targets.

Task	StreamVLN (fine-tuned)	Uni-NaVid (fine-tuned)	Ours (w/o Internet videos)	Ours
Hall
Display counter	2/5	4/5	4/5	5/5
Rocket model	1/5	1/5	2/5	3/5
Screen on the rack	0/5	4/5	5/5	5/5
Second door	2/5	3/5	4/5	4/5
Lounge

Appendix

Table 6: Per-task real-robot results, reported as successes / trials. Best results in each row are in bold .

Panel of Figure 7 — Figure 7: Real-robot task views and instructions, grouped by environment. Red boxes mark the targets. Target boxes and privacy masks are for illustration only and are not policy inputs.

Vision-Language Navigation

Vision-Language Navigation studies how embodied agents use natural-language instructions and visual observations to plan and execute routes.

See all 310 papers

ImagiNav: Scalable Embodied Navigation via Generative Visual Prediction and Inverse Dynamics
Mar 14, 2026Jie Chen, Yuxin Cai, Yizhuo Wang +5Robot Navigation Vision-Language Navigation
Department of Mechanical Engineering, National University of Singapore, Singapore · Institute for Infocomm Research (I2R), Agency for Science, Technology and Research (A*STAR), Singapore · Nanyang Technological University (NTU), Singapore
OpenFrontier: General Navigation with Visual-Language Grounded Frontiers
Mar 5, 2026Esteban Padilla-Cerdio, Boyang Sun, Marc Pollefeys +1Robot Navigation Vision-Language Navigation
1ETH Zurich · 2Microsoft Spatial AI Lab · University of Bonn
HAVANA: A Hierarchical Video-Action Model for Vision-Language Navigation
Oct 16, 2026Dongzhihan Wang, Yi Du, Chen Wang +2Vision-Language Navigation Visual Robot Navigation
Tongji University · SAIR Lab · Shanghai University

Mar 14, 2026J/K move · Enter open · S save

ImagiNav: Scalable Embodied Navigation via Generative Visual Prediction and Inverse Dynamics

Jie Chen, Yuxin Cai, Yizhuo Wang, Ruofei Bai+4

Department of Mechanical Engineering, National University of Singapore, Singapore · Institute for Infocomm Research (I2R), Agency for Science, Technology and Research (A*STAR), Singapore · Nanyang Technological University (NTU), Singapore

Robot Navigation Vision-Language Navigation Data-Efficient Robot Learning

Open paper AskPDF arXiv

Enabling robots to navigate open-world environments via natural language is critical for general-purpose autonomy. Yet, Vision-Language Navigation has relied on end-to-end policies trained on expensive, embodiment-specific robot data. While recent foundation models trained on vast simulation data show promise, the challenge of scaling and generalizing persists due to the limited scene diversity and visual fidelity in simulation. To address this gap, we propose ImagiNav, a novel hierarchical paradigm that formulates navigation in visual space. Instead of predicting waypoints, ImagiNav synthesizes a future egocentric video conditioned on language instructions, serving as a high-level plan, interpreted by an inverse dynamics model to extract metric trajectories for execution. By decoupling planning from robot actuation, the paradigm enables direct utilization of diverse in-the-wild navigation videos. To support this, we develop an auto-labeling data pipeline that enhances motion annotation accuracy. ImagiNav demonstrates strong zero-shot transfer to robot navigation without requiring robot demonstrations, paving the way for generalist robots that learn navigation directly from unlabeled, open-world data. The project page is available at: https://j1dan.github.io/ImagiNav

Abstract

Vision-Language Navigation

Explore similar work

ImagiNav: Scalable Embodied Navigation via Generative Visual Prediction and Inverse Dynamics

OpenFrontier: General Navigation with Visual-Language Grounded Frontiers

HAVANA: A Hierarchical Video-Action Model for Vision-Language Navigation