HAVANA: A Hierarchical Video-Action Model for Vision-Language Navigation

Dongzhihan Wang; Yi Du; Chen Wang; Xiuxian Li; Liang Xu

cs.CVOct 16, 2026

HAVANA: A Hierarchical Video-Action Model for Vision-Language Navigation

Authors: Dongzhihan Wang, Yi Du, Chen Wang, Xiuxian Li, Liang Xu

Organizations: Tongji University · SAIR Lab · Shanghai University

Abstract

Vision-Language Navigation (VLN) requires long-horizon instruction-consistent planning and short-horizon local planning in current geometry. Video-action models (VAMs) offer a promising way to learn predictive representations for navigation. However, existing VAMs jointly predict visual futures and local trajectories in a single process, even though they operate at different paces: future prediction remains stable across multiple updates while local trajectories must react to fresh observations. To bridge this gap, we introduce HAVANA, a dual-system VAM that decouples predictive reasoning from reactive local planning. A slow system produces a compact language-conditioned latent plan from observation history, while a fast system combines the latest plan with the current observation to generate local trajectories. Their asynchronous execution enables rapid adaptation without repeated future prediction, and no future video is generated during inference. Experiments on OmniNavBench demonstrate state-of-the-art navigation performance over prior baselines, while real-world deployments further show robust generalization to unseen environments across different robot embodiments. Our code and checkpoints will be available at https://github.com/Anastasiawd/HAVANA.

Figures & tables

Figure 2: Overview of the HAVANA training pipeline. Left: The slow system processes an RGB observation history, a navigation instruction, and four learnable time-conditioned queries with a video reasoner. Its contextual and query states form the understanding (UND) stream. The query states are also decoded by a supervised MLP into four egocentric waypoints ( waypoint prediction ). A clean conditioning latent and noisy future latents form the generation (GEN) stream. UND uses causal self-attention; GEN jointly attends to both streams within each sample. Future latents are supervised by flow matching. Right: RGB, depth, and UND latents condition a 1D U-Net for flow-matching-based prediction of a 24-step trajectory.

Method	Embodiment	OSR $\uparrow$	CSR $\uparrow$	SGC $\uparrow$	SPL $\uparrow$
AstraNav-World (w/ Action Former)	NVIDIA Carter	13.38	19.70	36.94	9.94
	Unitree Aliengo	19.16	22.79	39.86	11.57
	Unitree H1	11.73	18.59	36.60	9.46
DualVLN	NVIDIA Carter	24.82	23.34	40.84	18.28
	Unitree Aliengo	23.34	23.82	41.28	16.21
	Unitree H1	6.96	13.03	30.74	5.08

Table 1: Quantitative comparison with baselines across different robot embodiments. All metrics are reported in %.

Figure 3: Illustration of inference timelines. Each rectangle marks the wall-clock interval of one inference call. AstraNav-AF is shown as a full decision pipeline, while HAVANA and DualVLN retain their native scheduling. S1 and S2 denote the slow and fast systems in each method.

Figure 4: Qualitative comparison of predictive world modeling. The top two rows show Matterport3D scenes and the bottom two show GRScenes. Compared with Cosmos3-Edge and SparseVideoNav, HAVANA tends to predict larger hypothetical viewpoint changes under the same instruction. We provide a diagnostic analysis of directional consistency in Sec. 4.3.

Figure 5: Zero-shot real-world deployment on wheeled and quadruped robots. HAVANA runs closed-loop without real-world fine-tuning: at each step it predicts a short-horizon trajectory in the robot-centric frame, which is continuously updated as new observations arrive. Cyan curves denote MPPI candidate trajectories, while yellow curves indicate the selected trajectory.

Figure 6: Effect of instruction-ranking (IR) on directional grounding. Each column shows the same observation queried with paired counterfactual instructions that differ only in the turn direction. The colored curves denote the corresponding predicted trajectories, and $\Delta$ measures the absolute difference between their endpoint yaw angles. Top: w/ IR; bottom: w/o IR. Blue: turn left ; orange: turn right .

Metric	w/o Denoise	Denoise	$\Delta$
Waypoint ADE (Random)	0.3030	0.3101	-2.37%
Waypoint ADE (Turn)	0.3891	0.3861	+0.76%
Fast Traj. ADE (Fresh)	0.1671	0.1658	+0.81%
Fast Traj. ADE (Cached)	0.2321	0.2116	+8.84%

Table 2: Effect of future-video denoising supervision on offline prediction. We report Average Displacement Error (ADE

\downarrow

) for long-horizon waypoints and short-horizon trajectories. Cached evaluates previously generated latent plans, and

\Delta

denotes relative improvement over the baseline without denoising.

Metric	w/o IR	w/ IR
Directional accuracy	0.7109	0.8672
Both variants correct	0.4219	0.7344

Table 3: Effect of instruction-ranking (IR) supervision on directional grounding. Directional accuracy (

\uparrow

) measures whether the endpoint yaw follows the instructed direction, while Both variants correct (

\uparrow

) requires correct predictions for both counterfactual prompts on the same observation.

Vision-Language Navigation

Vision-Language Navigation studies how embodied agents use natural-language instructions and visual observations to plan and execute routes.

See all 310 papers

FutureNav: Unified World-Action Modeling for Vision-and-Language Navigation
Jun 29, 2026Lingfeng Zhang, Zeying Gong, Xiaoshuai Hao +7World Models Vision-Language Navigation
Tsinghua University · Pengcheng Laboratory · Xiaomi EV +2
HAM-VLN: Harnessing Hierarchical Agentic Memory for Zero-Shot Vision-and-Language Navigation
Jul 31, 2026An Liu, Bingxi Liu, Hongyu Ding +6Memory-Augmented VLMs Vision-Language Navigation
Institute of Automation, Chinese Academy of Sciences, Beijing, China · Southern University of Science and Technology, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China +3
AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation
May 21, 2026Wenxuan Guo, Xiuwei Xu, Yichen Liu +7Vision-Language Navigation VLM Reasoning
Tsinghua University, China

Jun 29, 2026J/K move · Enter open · S save

FutureNav: Unified World-Action Modeling for Vision-and-Language Navigation

Lingfeng Zhang, Zeying Gong, Xiaoshuai Hao, Haoxiang Fu+6

Tsinghua University · Pengcheng Laboratory · Xiaomi EV+2

World Models Vision-Language Navigation World Models for VLA Models

Open paper AskPDF arXiv

Vision-and-language navigation (VLN) in continuous environments requires an agent to ground instructions in egocentric observations while maintaining spatial understanding across long action sequences. Recent navigation foundation models have shown strong progress by scaling vision-language models, but they often learn navigation primarily as direct action generation, without explicitly modeling world states or predicting their future evolution. We introduce FutureNav, a VLM-based unified world-action modeling framework for vision-and-language navigation. Specifically, FutureNav jointly encodes text, visual, and spatial features and feeds them into the LLM, and optimizes four objectives for simultaneous world and action modeling: an action policy objective for navigation action prediction, inverse and forward dynamics objectives for modeling state transitions, and a future generation objective for predicting future spatial states. This unified architecture strengthens action prediction while explicitly modeling the world, without sacrificing inference speed. Extensive experiments show that, with only a 4B-scale backbone, FutureNav achieves state-of-the-art performance on multiple VLN benchmarks and substantially outperforms prior VLN methods, paving the way toward future world-action models for VLN. We will release the code and models to support future research.

Abstract

Vision-Language Navigation

Explore similar work

FutureNav: Unified World-Action Modeling for Vision-and-Language Navigation

HAM-VLN: Harnessing Hierarchical Agentic Memory for Zero-Shot Vision-and-Language Navigation

AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation