cs.CVFeb 5, 2026

Sparse Video Generation Propels Real-World Beyond-the-View Vision-Language Navigation

Authors: Hai Zhang, Siqi Liang, Li Chen, Yuxian Li, Yukuan Xu, Yichao Zhong, Fu Zhang, Hongyang Li

Organizations: The University of Hong Kong

Abstract

Why must vision-language navigation be bound to detailed and verbose language instructions? While such details ease decision-making, they fundamentally contradict the goal for navigation in the real-world. Ideally, agents should possess the autonomy to navigate in unknown environments guided solely by simple and high-level intents. Realizing this ambition introduces a formidable challenge: Beyond-the-View Navigation (BVN), where agents must locate distant, unseen targets without dense and step-by-step guidance. Existing large language model (LLM)-based methods, though adept at following dense instructions, often suffer from short-sighted behaviors due to their reliance on short-horimzon supervision. Simply extending the supervision horizon, however, destabilizes LLM training. In this work, we identify that video generation models inherently benefit from long-horizon supervision to align with language instructions, rendering them uniquely suitable for BVN tasks. Capitalizing on this insight, we propose introducing the video generation model into this field for the first time. Yet, the prohibitive latency for generating videos spanning tens of seconds makes real-world deployment impractical. To bridge this gap, we propose SparseVideoNav, achieving sub-second trajectory inference guided by a generated sparse future spanning a 20-second horizon. This yields a remarkable 27x speed-up compared to the unoptimized counterpart. Extensive real-world zero-shot experiments demonstrate that SparseVideoNav achieves 2.5x the success rate of state-of-the-art LLM baselines on BVN tasks and marks the first realization of such capability in challenging night scenes.

Explore similar work

CardsList
  1. NavDreamer: Video Models as Zero-Shot 3D Navigators

    Feb 10, 2026Xijie Huang, Weiqi Gai, Tianyue Wu +5Vision-Language NavigationModel-Based Planning

  2. SFCo-Nav: Efficient Zero-Shot Visual Language Navigation via Collaboration of Slow LLM and Fast Attributed Graph Alignment

    Mar 2, 2026Chaoran Xiong, Litao Wei, Xinhao Hu +5Vision-Language NavigationRobot Navigation

  3. HAVANA: A Hierarchical Video-Action Model for Vision-Language Navigation

    Oct 16, 2026Dongzhihan Wang, Yi Du, Chen Wang +2Vision-Language NavigationVisual Robot Navigation