Policy-Guided World Model Planning for Language-Conditioned Visual Navigation

Amirhosein Chahe; Lifeng Zhou

cs.ROMar 26, 2026

Policy-Guided World Model Planning for Language-Conditioned Visual Navigation

Authors: Amirhosein Chahe, Lifeng Zhou

Organizations: Drexel University Philadelphia PA 19104, USA

Abstract

Navigating to a visually specified goal given natural language instructions remains a fundamental challenge in embodied AI. Existing approaches either rely on reactive policies that struggle with long-horizon planning, or employ world models that suffer from poor action initialization in high-dimensional spaces. We present PiJEPA, a two-stage framework that combines the strengths of learned navigation policies with latent world model planning for instruction-conditioned visual navigation. In the first stage, we finetune an Octo-based generalist policy, augmented with a frozen pretrained vision encoder (DINOv2 or V-JEPA-2), on the CAST navigation dataset to produce an informed action distribution conditioned on the current observation and language instruction. In the second stage, we use this policy-derived distribution to warm-start Model Predictive Path Integral (MPPI) planning over a separately trained JEPA world model, which predicts future latent states in the embedding space of the same frozen encoder. By initializing the MPPI sampling distribution from the policy prior rather than from an uninformed Gaussian, our planner converges faster to high-quality action sequences that reach the goal. We systematically study the effect of the vision encoder backbone, comparing DINOv2 and V-JEPA-2, across both the policy and world model components. Experiments on real-world navigation tasks demonstrate that PiJEPA significantly outperforms both standalone policy execution and uninformed world model planning, achieving improved goal-reaching accuracy and instruction-following fidelity.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of PiJEPA. Top: The Octo policy, finetuned with a frozen vision encoder ( $E_{\phi}$ ), takes the current latent observation $z_{t}$ and instruction $\ell$ as input and produces action chunk samples via its diffusion head. These are transformed from the global frame to the world model’s local body frame. Middle: The policy’s statistics $(\mu_{\pi},\sigma_{\pi})$ warm-starts MPPI, which iteratively optimizes the action distribution over $J$ iterations. Bottom: The JEPA world model predictor ( $P_{\psi}$ ), trained with the same frozen encoder, autoregressively predicts future latent states. The MPPI candidates are scored by unrolling the world model and evaluating the latent-space distance to the encoded goal $z_{g}$ (Algorithm 1 ).

Panel of Figure 2 — Figure 2: Qualitative trajectory comparison. Given a start observation, goal image, and language instruction (left), we compare trajectories produced by each method under two encoder backbones (right). The black curve (GT) shows the ground-truth path; colored curves show MPPI (red), Octo policy (blue), Octo-WM scoring (orange), and PiJEPA (green). Stars mark final positions. PiJEPA most closely tracks the ground truth in both settings.

	ATE XY (m)			ATE Hdg ( ∘ )			RPE XY (m)		RPE Hdg ( ∘ )
	RMSE	Mean	Final	RMSE	Mean	Final	RMSE	Mean	RMSE	Mean
Octo	1.98	1.65	3.19	20.08	16.76	28.55	0.61	0.57	8.29	6.94
MPPI	1.85	1.48	3.23	18.79	15.68	27.86	0.55	0.51	5.70	5.11
Octo-WM	1.80	1.43	3.15	22.16	18.60	29.16	0.56	0.51	7.51	6.45
PiJEPA	1.78	1.42	3.12	19.89	16.52	28.63	0.56	0.51	7.05	6.07

Table 1: DINOv2 ViT-S on CAST validation episodes. Best in bold . All

\downarrow

	ATE XY (m)			ATE Hdg ( ∘ )			RPE XY (m)		RPE Hdg ( ∘ )
	RMSE	Mean	Final	RMSE	Mean	Final	RMSE	Mean	RMSE	Mean
Octo	1.72	1.36	3.02	18.72	15.64	27.72	0.53	0.48	6.09	5.34
MPPI	1.87	1.50	3.29	19.15	15.98	28.50	0.55	0.52	5.75	5.15
Octo-WM	1.67	1.35	2.87	20.63	17.27	28.06	0.54	0.49	6.93	5.99
PiJEPA	1.65	1.32	2.88	19.13	15.95	27.93	0.53	0.49	6.46	5.62

Table 2: V-JEPA-2 ViT-L on CAST validation episodes. Best in bold . All

\downarrow

Panel of Figure 3 — Figure 3: Failure case analysis. The language instruction “Follow the building” is inherently ambiguous, as multiple buildings are visible in the scene. The Octo policy (blue) misinterprets the referent and veers toward a different building, illustrating how vague instructions can mislead reactive policies that lack long-horizon reasoning. Meanwhile, the world model fails to make meaningful progress because it becomes stuck in its rollouts, predicting nearly identical latent states, which causes the planner to stagnate. The WM Pred. row confirms this directly. The predicted observations remain largely unchanged across the planning horizon. PiJEPA (green) partially mitigates both issues by grounding the planner with a policy-derived prior, though it still undershoots the ground-truth trajectory.

Embodied Navigation

Embodied navigation studies how robots perceive, map, and move through environments to reach goals and follow instructions.

See all 138 papers

Mar 23, 2026

Language-Conditioned World Modeling for Visual Navigation

Goal-conditioned visual navigation has been a long-standing testbed for embodied AI. We study a natural language-conditioned variant, language-conditioned visual navigation (LCVN), in which an embodied agent must follow a natural language instruction given only an initial egocentric observation. Without access to goal images, the agent must rely on language to shape its perception and continuous control. We introduce the LCVN Dataset, a benchmark of 39,016 trajectories and 117,048 human-verified instructions spanning diverse environments and instruction styles. Building on this benchmark, we study two complementary paradigms: (i) latent-imagination policy learning, in which a diffusion-based world model (LCVN-WM) imagines future observations and an actor-critic agent (LCVN-AC) learns its policy entirely within the imagined latent space; and (ii) unified autoregressive prediction, in which a single multimodal backbone (LCVN-Uni) jointly predicts actions and observations in one forward pass over a shared token sequence. Experiments show that two paradigms offer complementary strengths: latent imagination produces more temporally coherent rollouts, whereas unified prediction generalizes better to unseen environments. Targeted ablations further isolate the contributions of language guidance, conditioning signals, and instruction style, clarifying when language grounding versus dynamics modeling is the performance bottleneck. Together, these findings position LCVN as a testbed for studying how language, imagination, and decision-making interact in embodied agents.

Yifei Dong, Fengyi Wu, Yilong Dai +8

University of Washington · National University of Singapore · Clemson University +2

World Model Learning Vision-Language Navigation Visual RL VLN Benchmarks

Ask PDF

Jun 11, 2026

NavWAM: A Navigation World Action Model for Goal-Conditioned Visual Navigation

Goal-conditioned visual navigation requires a robot to act under partial observability by anticipating how its motion will change the future egocentric view and whether that change brings it closer to the goal. Navigation world models provide such visual foresight, but they remain prediction modules that require an external planner to convert predicted futures into closed-loop control. We propose Navigation World Action Model (NavWAM), a diffusion-transformer policy that turns navigation world-model prediction into executable action by representing future observations, goal-progress values, and action chunks in a shared latent sequence. By learning future prediction jointly with the action and value targets that determine closed-loop behavior, NavWAM makes visual foresight directly usable for robot control. We build NavWAM through simulation pretraining and real-robot adaptation, and evaluate it on image-goal navigation against planning-based world models and a representative direct navigation policy. Across offline benchmarks and closed-loop real-robot deployment, NavWAM improves over planning-based world-model baselines in our evaluations while using the default policy mode without CEM-style action search. Project page: https://dachii-azm.github.io/navwam/

Daichi Azuma, Taiki Miyanishi, Koya Sakamoto +6

The University of Tokyo · National Institute of Informatics · AIRoA +1

World Models for Robotics Robot Navigation Visuomotor Policy Learning Diffusion Policy Image-Goal Navigation

Ask PDF

Aug 4, 2026

UniNav: A Unified World-Action Diffusion Model for Visual Navigation

Image-goal visual navigation is a fundamental capability for embodied agents. Existing navigation policies efficiently predict waypoint trajectories but lack visual foresight, while navigation world models can anticipate future observations but often require costly planning rollouts. We present UniNav, a unified world-action model that generates future visual observations and continuous waypoint trajectories through a single diffusion process. Given history frames and a goal image, UniNav jointly denoises visual and waypoint tokens within a single transformer, unifying future prediction and action generation in a shared framework. To improve spatial grounding, we incorporate geometry-aware camera tokens. We also train on both trajectory-labeled navigation data and video-only data, enabling the model to benefit from diverse videos without waypoint annotations. Based on this unified framework, we introduce two variants: UniNav-Full jointly predicts interpretable future observations and their corresponding trajectories, while UniNav-Fast removes future-image tokens at inference for efficient trajectory prediction. Experiments on navigation benchmarks show that UniNav outperforms the strongest baseline in ATE across all datasets. With one-step inference, UniNav-Fast achieves a latency of 0.1s without a substantial accuracy drop. Code will be released.

Changqing Zhou, Yueru Luo, Zeyu Jiang +1

1The Hong Kong University of Science and Technology (Guangzhou) · 2The Chinese University of Hong Kong, Shenzhen

Visual Navigation Trajectory Generation Diffusion-Based Planning Robot Navigation World Models Visuomotor Policy Learning+1

Ask PDF

Abstract

Embodied Navigation

Explore similar work

Language-Conditioned World Modeling for Visual Navigation

NavWAM: A Navigation World Action Model for Goal-Conditioned Visual Navigation

UniNav: A Unified World-Action Diffusion Model for Visual Navigation