WorldMAP: Bootstrapping Vision-Language Navigation Trajectory Prediction with Generative World Models

Hongjin Chen; Shangyun Jiang; Tonghua Su; Chen Gao; Xinlei Chen; Yong Li; Zhibo Chen

cs.AIApr 9, 2026

WorldMAP: Bootstrapping Vision-Language Navigation Trajectory Prediction with Generative World Models

Authors: Hongjin Chen, Shangyun Jiang, Tonghua Su, Chen Gao, Xinlei Chen, Yong Li, Zhibo Chen

Organizations: Harbin Institute of Technology · Zhongguancun Academy · Shandong University · Tsinghua University ·

Abstract

Vision-language models (VLMs) and generative world models are opening new opportunities for embodied navigation. VLMs are increasingly used as direct planners or trajectory predictors, while world models support look-ahead reasoning by imagining future views. Yet predicting a reliable trajectory from a single egocentric observation remains challenging. Current VLMs often generate unstable trajectories, and world models, though able to synthesize plausible futures, do not directly provide the grounded signals needed for navigation learning. This raises a central question: how can generated futures be turned into supervision for grounded trajectory prediction? We present WorldMAP, a teacher--student framework that converts world-model-generated futures into persistent semantic-spatial structure and planning-derived supervision. Its world-model-driven teacher builds semantic-spatial memory from generated videos, grounds task-relevant targets and obstacles, and produces trajectory pseudo-labels through explicit planning. A lightweight student with a multi-hypothesis trajectory head is then trained to predict navigation trajectories directly from vision-language inputs. On Target-Bench, WorldMAP achieves the best ADE and FDE among compared methods, reducing ADE by 18.0% and FDE by 42.1% relative to the best competing baseline, while lifting a small open-source VLM to DTW performance competitive with proprietary models. More broadly, the results suggest that, in embodied navigation, the value of world models may lie less in supplying action-ready imagined evidence than in synthesizing structured supervision for navigation learning.

Figures & tables

Fig. 1: Teacher–student distillation in WorldMAP. A world-model-driven teacher converts generated futures into grounded training signals for a lightweight vision-language student, which learns to predict navigation trajectories directly from the observation and instruction.

Fig. 2: Motivation and design rationale of WorldMAP. Top : VLM-only prediction and world-model-augmented reasoning remain unreliable for single-observation trajectory prediction. Bottom : WorldMAP organizes navigation around a World-Memory-Action-Perception decomposition, inspired by LeCun’s architecture for autonomous machine intelligence [ 1 ] , and converts generated futures into persistent semantic-spatial structure and planning-derived supervision for a lightweight student predictor.

Fig. 3: Architecture of WorldMAP. A world-model-driven teacher converts generated futures into semantic-spatial memory, task-aware grounding, and cost-BEV planning, and then produces trajectory pseudo-labels for student training. At inference time, only the lightweight student is used to predict the final navigation trajectory.

Fig. 4: Teacher-generated trajectory pseudo-labels from world-model futures. The teacher grounds targets and obstacles across generated views, projects them into a shared BEV planning space, constructs a cost map, and runs FMM to obtain trajectory pseudo-labels for student training.

Method	Method Family	ADE $\downarrow$	FDE $\downarrow$	DTW $\downarrow$
GPT-5.4	Proprietary	94.65	150.52	66.49
o3		112.14	177.27	57.30
Gemini-3-Pro		51.27	67.19	31.63
Qwen3-VL-4B	Open-source	140.42	256.00	136.53
Qwen3-VL-8B		183.93	339.58	177.33
Qwen3-VL-32B		151.21	298.65	108.06

TABLE I: Main results in projected 2D image space on Target-Bench. ADE, FDE, and normalized DTW are computed on projected trajectories in the real first-frame image. Lower is better for all metrics. MindJourney † follows the best-performing configuration reported in the original paper [ 24 ] , using an o-series reasoning model with SVC [ 37 ] , while WorldMAP uses Qwen3-VL-8B as the student backbone.

Fig. 5: Qualitative comparison on Target-Bench. Each row shows one instruction and the projected trajectories from the ground truth and competing methods in the real first-frame image. WorldMAP more consistently follows traversable floor geometry and stops closer to the intended target, whereas baselines are more prone to drift, overshoot, or pass through non-traversable regions.

WorldMAP student backbone	ADE $\downarrow$	FDE $\downarrow$	DTW $\downarrow$
Qwen3-VL-4B	45.07	42.84	33.56
Qwen3-VL-8B	42.06	38.87	31.95

TABLE II: Ablation on WorldMAP student backbone scale.

Fig. 6: VLM-based quality assessment of world-model pseudo-labels. Generated trajectories are evaluated using five criteria: task success, stop accuracy, collision safety, trajectory naturalness, and cross-view consistency. Each trajectory is then categorized as usable, borderline, or reject, and these quality tiers are used in the supervision ablation below. The bottom panel summarizes the resulting quality distribution across world models.

Training supervision	ADE $\downarrow$	FDE $\downarrow$	DTW $\downarrow$
WM pseudo-labels only (usable / borderline)	95.98	141.10	88.25
Train GT only	78.34	121.75	75.09
Train GT + WM pseudo-labels (usable / borderline)	42.85	40.97	31.78
Train GT + WM pseudo-labels (usable)	42.06	38.87	31.95

TABLE III: Ablation on student training supervision.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 7: Representative examples of projecting Target-Bench 3D SLAM trajectories onto 2D real-world first frames.

Fig. 8: Illustration of our homography protocol.

Fig. 9: Statistics of the processed data.

Fig. 10: Intermediate stages of the teacher pipeline.

Fig. 11: Additional qualitative comparisons across models.

Vision-Language Navigation

Vision-Language Navigation studies how embodied agents use natural-language instructions and visual observations to plan and execute routes.

See all 310 papers

FutureNav: Unified World-Action Modeling for Vision-and-Language Navigation
Jun 29, 2026Lingfeng Zhang, Zeying Gong, Xiaoshuai Hao +7World Models Vision-Language Navigation
Tsinghua University · Pengcheng Laboratory · Xiaomi EV +2
LiteNWM: Efficient Latent World Models for Onboard Visual Navigation in the Wild
Oct 8, 2026Linkai Liu, Yuntian Zhang, Zhenshan Bing +5Robot Navigation Latent World Models
Nanjing University · Imperial College London · Sony AI +1
WALT: Learning World-Model-Aligned Latent Trajectories for Autonomous Driving
Sep 24, 2026Mingkai Jia, Jiaxin Guo, Zhijian Shu +5World Model-Based Planning Autonomous Driving Planning
The Hong Kong University of Science and Technology. · Horizon Robotics. · The Chinese University of Hong Kong. +2

Jun 29, 2026J/K move · Enter open · S save

FutureNav: Unified World-Action Modeling for Vision-and-Language Navigation

Lingfeng Zhang, Zeying Gong, Xiaoshuai Hao, Haoxiang Fu+6

Tsinghua University · Pengcheng Laboratory · Xiaomi EV+2

World Models Vision-Language Navigation World Models for VLA Models

Open paper AskPDF arXiv

Vision-and-language navigation (VLN) in continuous environments requires an agent to ground instructions in egocentric observations while maintaining spatial understanding across long action sequences. Recent navigation foundation models have shown strong progress by scaling vision-language models, but they often learn navigation primarily as direct action generation, without explicitly modeling world states or predicting their future evolution. We introduce FutureNav, a VLM-based unified world-action modeling framework for vision-and-language navigation. Specifically, FutureNav jointly encodes text, visual, and spatial features and feeds them into the LLM, and optimizes four objectives for simultaneous world and action modeling: an action policy objective for navigation action prediction, inverse and forward dynamics objectives for modeling state transitions, and a future generation objective for predicting future spatial states. This unified architecture strengthens action prediction while explicitly modeling the world, without sacrificing inference speed. Extensive experiments show that, with only a 4B-scale backbone, FutureNav achieves state-of-the-art performance on multiple VLN benchmarks and substantially outperforms prior VLN methods, paving the way toward future world-action models for VLN. We will release the code and models to support future research.

Abstract

Vision-Language Navigation

Explore similar work

FutureNav: Unified World-Action Modeling for Vision-and-Language Navigation

LiteNWM: Efficient Latent World Models for Onboard Visual Navigation in the Wild

WALT: Learning World-Model-Aligned Latent Trajectories for Autonomous Driving