Exposing the Long-tail in Embodied Urban Navigation via Scalable Learning from In-the-Wild Videos

Bingyi Xia; Han Bao; Zhewei Chen; Hanjing Ye; Jingwen Yu; Yuhan Pang; Wenjun Xu; Jiankun Wang

cs.ROAug 17, 2026

Exposing the Long-tail in Embodied Urban Navigation via Scalable Learning from In-the-Wild Videos

Authors: Bingyi Xia, Han Bao, Zhewei Chen, Hanjing Ye, Jingwen Yu, Yuhan Pang, Wenjun Xu, Jiankun Wang

Organizations: Southern University of Science and Technology · The Hong Kong University of Science and Technology · Peng Cheng Laboratory

Abstract

Learning embodied urban navigation policies from real-world data is constrained by the cost of task-specific data collection and the limited coverage of rare yet safety-critical scenarios. To address these challenges, we present a scalable framework for learning point-goal urban navigation from web-scale in-the-wild egocentric videos while systematically exposing its long tail. The framework automatically annotates uncurated web videos with metric trajectories and structured navigation semantics, which are then used to train a vision-language-action policy for interpretable navigation planning. We characterize the long tail based on model performance and the distribution of perception-motion patterns, and employ reflection-based analysis to diagnose recurring failure modes. Experiments on web-video data and real-world urban navigation tasks demonstrate effective knowledge transfer from unconstrained videos and reveal coherent long-tail structures beyond aggregate navigation performance.

Figures & tables

Figure 1: Overview of the proposed framework for scalable urban navigation VLA models. Our approach first collect worldwide web-scale street-walking videos as the dataset. A VLA model is then trained to perform end-to-end urban navigatioin planning. Finally, the reflection driven by a teacher model is introduced to analyze failure taxonomies.

Figure 2: Geographic distribution of our in-the-wild street-walking video collection.

Dataset	Duration	Locations	Annotation
SCAND	8.7 h	1	-
SiT	0.4 h	1	3D tracks and semantic maps
GND	11 h	10	Semantic map
Citywalker	200 h	1	Caption and instruction
Ours	500 h	30	Navigation reasoning CoT

Table 1: Comparison of urban navigation datasets based on real-world collection and in-the-wild videos.

Figure 3: The two-stage pipeline for automated video processing. Step 1 illustrates the annotation process consisting of Qwen3.6VL-27B and other visual foundation models. Step 2 depicts the long-tail data analysis process. Long-tail samples are defined using distributional rarity and model-dependent trajectory error, and a teacher model performs reflective reasoning for structured failure attribution.

Figure 4: Detailed architecture of the proposed WILD-Nav model. It processes temporal video streams through a vision encoder, while a Large Language Model backbone processes the fused tokens to generate CoT reasoning. Finally, the action expert decodes the planning token into a trajectory.

Video Test Set
Method	Meta(%) $\uparrow$	ADE(m) $\downarrow$	FDE(m) $\downarrow$	MAOE( ∘ ) $\downarrow$
UrbanNav	–	0.157	0.136	6.30
SocialNav-origin	–	0.906	1.123	16.53
SocialNav	–	0.470	0.728	12.65
WILD-Nav-inst	96.5	0.094	0.071	5.90
WILD-Nav-think	93.4	0.086	0.068	5.32

Table 2: Comparison of navigation planning results on the video and real-world test sets. – denotes the unavailable test.

Panel of Figure 5 — Figure 5: Overview of long-tail discovery through feature rarity and model-dependent difficulty. Left: Distribution of joint perception–motion rarity scores. Right: Distributions of ADE and FDE.

Panel of Figure 6 — Figure 6: Failure attribution obtained by privileged reflection over the hard set. (a) Visualization of six representative HDBSCAN clusters in the hard set; gray points denote all remaining samples. (b) Cluster-specific reflection failure patterns. Bubble size represents failure support, and color indicates enrichment over the global baseline.

	Test-all		Test-hard
Method	ADE $\downarrow$	FDE $\downarrow$	ADE $\downarrow$	FDE $\downarrow$	Meta (%) $\uparrow$
WILD-Nav-inst	0.094	0.074	0.286	0.247	94.88
+Random	0.091	0.072	0.277	0.231	95.10
+Hard	0.121	0.102	0.230	0.165	96.83
+Hard+Random	0.093	0.079	0.217	0.160	96.73

Table 3: Ablation of incremental fine-tuning data on the Long-Tail Test Set. The fixed tail is the union of the worst 3% ADE and FDE samples selected by WILD-Nav-inst.

Figure 7: Real-world navigation performance of our VLA model. Our model generates safe and reasonable trajectories (red) for various navigation tasks.

Embodied Navigation

Embodied navigation studies how robots perceive, map, and move through environments to reach goals and follow instructions.

See all 138 papers

ScaleNav: Scalable Internet Video Pretraining for Language-Conditioned Local Navigation
Oct 19, 2026Yihui Yang, Yuanhong Yu, Jiawei Wang +7Vision-Language Navigation Robot Learning from Video
Zhejiang University · Li Auto Inc. · Shenzhen University
Implicit Geometry Representations for Vision-and-Language Navigation from Web Videos
Mar 10, 2026Mingfei Han, Haihong Hao, Liang Ma +6Vision-Language Navigation 3D Scene Representation
Department of Computer Vision, Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates. · School of Information Science and Technology, University of Science and Technology of China. · Shenzhen Campus of Sun Yat-Sen University.
Lifelong Embodied Navigation Learning
Mar 6, 2026Xudong Wang, Jiahua Dong, Baichen Liu +3Embodied Navigation Continual Learning for LLM Agents
State Key Laboratory of Robotics and Intelligent Systems, Shenyang Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Mohamed bin Zayed University of Artificial Intelligence

Oct 19, 2026J/K move · Enter open · S save

ScaleNav: Scalable Internet Video Pretraining for Language-Conditioned Local Navigation

Yihui Yang, Yuanhong Yu, Jiawei Wang, Zhongrui Yu+6

Zhejiang University · Li Auto Inc. · Shenzhen University

Vision-Language Navigation Robot Learning from Video

Open paper AskPDF arXiv

We study language-conditioned local navigation, where a robot must navigate to a text-specified target visible in its initial observation and stop at the described location. End-to-end navigation policies require diverse training data to generalize across scenes, yet real-robot data is costly to collect and simulated data leaves a sim-to-real gap. Internet egocentric videos offer a scalable source of real-world navigation experience. Our key insight is to learn goal-directed visual dynamics from Internet egocentric videos, and use these dynamics to support robot action generation. To this end, we propose ScaleNav, a unified world-action framework with three-stage training. We first perform instruction-conditioned video pretraining on these videos to learn how observations evolve as the camera approaches instructed targets across diverse real-world scenes. Action alignment then uses action-labeled simulated data to train the model to predict executable action chunks. Joint fine-tuning subsequently optimizes all trainable components together for navigation. ScaleNav achieves success rates of 86.5% on a single-view adaptation of Short-Horizon OVON and 94.9% on our HabitatGS-LN benchmark, outperforming the evaluated baselines on both benchmarks. We further deploy and evaluate ScaleNav on a real quadruped robot, with results supporting the model's ability to generalize to real-world navigation. Code and videos are available at https://zju3dv.github.io/ScaleNav/

Abstract

Embodied Navigation

Explore similar work

ScaleNav: Scalable Internet Video Pretraining for Language-Conditioned Local Navigation

Implicit Geometry Representations for Vision-and-Language Navigation from Web Videos

Lifelong Embodied Navigation Learning