Hydra: A Navigation World Action Model with Discrete Latent Planning and Continuous Flow-Matching Execution

Mohammad Nazeri; Alexandyr Card; Samira Huber; Anuj Pokhrel; Yujun Wang; Ruben Hammele; Daeun Song; Sören Pirk

cs.ROAug 29, 2026

Hydra: A Navigation World Action Model with Discrete Latent Planning and Continuous Flow-Matching Execution

Authors: Mohammad Nazeri, Alexandyr Card, Samira Huber, Anuj Pokhrel, Yujun Wang, Ruben Hammele, Daeun Song, Sören Pirk, +1 more

Organizations: George Mason University · Kiel University · Ludwig Maximilian University Munich · Ewha Womans University

Abstract

World models let robots imagine possible futures, but exploiting this capability for real-time planning is bottlenecked by a representation misalignment: generative models and planners operate on decoupled manifolds, requiring computationally expensive decoding of every candidate back to the high-dimensional observation space for evaluation. In this paper, we present Hydra, a discrete World Action Model that tackles this by establishing a unified latent manifold over visual states, physical poses, and control actions. By compressing this manifold through modality-specific Vector-Quantized bottlenecks, Hydra yields discrete vocabularies of kinodynamic intents and visual states. This enables Discrete Latent Planning (DLP), where candidates are sampled directly from the shared manifold and ranked by a Kinematic-Perceptual Cost within the discrete latent space. To bridge discrete planning with the continuous commands required for physical actuation, Hydra pairs DLP with conditional Flow Matching to map selected intents to smooth execution trajectories. Evaluated on two physical robotic platforms, Hydra outperforms state-of-the-art navigation world models in goal-directed planning, while matching or exceeding the closed-loop execution capabilities of leading reactive navigation policies.

Figures & tables

Figure 1: Moving the planner into the model’s own latent space. Top: NWM uses an external planner to sample unconstrained trajectories, decoding them to pixels for evaluation ( red ), wasting compute on infeasible candidates. Bottom: Hydra plans directly within its discrete latent manifold, then executes via continuous Flow Matching, achieving $>500\times$ faster planning on a robot.

Figure 2: Comparison of Hydra architecture with Visual Navigation Transformer (ViNT), Navigation World Model (NWM), and VertiFormer in terms of input modalities, conditioning, and the model output. Hydra builds on top of VertiFormer and discretizes the output into codebooks to enable the model to search the discrete latent space for a plausible path.

Figure 3: Initial intended future pose/action tokens are generated unconditionally, then Gaussian noise is added to become the exploratory seed ‘intents’. Then these future intents go through VQ codebooks, and we sample the top-

k

codes. After calculating the KPC cost, the lowest cost is chosen to seed the next iteration of searching. The best trajectory is decoded using flow matching heads to continuous robot poses and actions at the end for robot execution.

Model	FVD $\downarrow$	Time To Generate
NWM/B	22.10 $\pm$ 10.71	3.5h
VertiFormer	81.03 $\pm$ 12.50	2m
Hydra	19.15 $\pm$ 5.11	4m

Table 1: Comparison of Video Synthesis Quality. 100 clips of 16-seconds generated at 4 FPS.

Model	Params	Search Space	Iterations	Samples	Len.	Plan Time $\downarrow$	Unobs. (SR) $\uparrow$	Obs. (SR) $\uparrow$	Corner Turn (SR) $\uparrow$
NWM/B	200M	Continuous (CEM)	3	18	12	$>$ 500.0s	0	0	0
VertiFormer	27.11M	Continuous (MPPI)	1	18	12	$\sim$ 0.7s	40	0	0
Hydra-NoVQ	141.60M	Continuous (MPPI)	1	18	12	$\sim$ 1.2s	20	10	10
Hydra	143.29M	Discrete (DLP)	3	18	12	$\sim$ 0.9s	100	80	80

Table 2: Planning Performance. While VertiFormer with MPPI runs quickly, it suffers from sample inefficiency and is blind to obstacles (0% SR), as is the continuous variant, Hydra-NoVQ. Hydra ’s DLP achieves high success rates while maintaining real-time execution speeds compared to NWM.

		Dense Goal Images (2Hz)			Sparse Waypoints (turns only)
Scenario	Metric	GNM	ViNT	NoMaD	Hydra (CFG)	Hydra (Sampling)	VertiFormer	Hydra (Planning)
Obstructed Goal	Success Rate (%) $\uparrow$	90	80	100	90	50	100	90
	Traversal Time (s) $\downarrow$	36.4 $\pm$ 7.9	34.4 $\pm$ 6.8	37.4 $\pm$ 7.7	45.0 $\pm$ 8.0	48.4 $\pm$ 18.5	21.0 $\pm$ 1.05	37.2 $\pm$ 21.0
	Interventions $\downarrow$	1.1 $\pm$ 0.3	1.2 $\pm$ 0.6	1.1 $\pm$ 0.3	1.0 $\pm$ 0.7	0.8 $\pm$ 0.8	1.0 $\pm$ 0.5	1.2 $\pm$ 0.6
Long Distance	Success Rate (%) $\uparrow$	0	0	0	100	100	100	40
	Traversal Time (s) $\downarrow$	-	-	-	138 $\pm$ 7.7	107 $\pm$ 8.6	109 $\pm$ 7.6	119 $\pm$ 3.8

Table 3: Local Planning Evaluation across 10 Physical Trials per Scenario. Hydra drastically reduces physical interventions by leveraging its predictive foresight to move around topological traps that ensnare reactive baselines. GNM, ViNT, and NoMaD are conditioned on dense goal-image sequences at 2Hz; Hydra ’s local-following variants (CFG, Sampling) are conditioned on sparse, more widely spaced waypoints.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

\RowStyle Method	Obs. Input	Goal Input	Cond. Input	Output	Nav. Policy
GNM [ 58 ]	I	I	I	P	Reactive
VINT [ 59 ]	I	I	I	P	Reactive
NOMAD [ 62 ]	I	I	I	P	Reactive
VertiFormer [ 46 ]	I, A, P	I, P	A or P	I, A, P	Planning
NWM [ 6 ]	I	I	P	I	Planning
Hydra	I, A, P	I, P	A or P	I, A, P	Planning

Appendix

Table 4: Comparison with visually-grounded robot navigation baselines. I : Image, P : Pose, A : Action.

Model	Params	DreamSim $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$
NWM/B	200M	0.2480 $\pm$ 0.1265	0.5168 $\pm$ 0.1147	10.69
VertiFormer	27.11M	0.3738 $\pm$ 0.0560	0.5417 $\pm$ 0.0757	18.29
Hydra	143.29M	0.1103 $\pm$ 0.0345	0.2809 $\pm$ 0.0679	16.20

Appendix

Table 5: Offline single-step image quality on a held-out subset of the RECON dataset. Hydra achieves competitive accuracy with fewer parameters and NFEs.

Model	Params	SCAND		TartanDrive		SACSoN		Mean
		ADE $\downarrow$	FDE $\downarrow$	ADE $\downarrow$	FDE $\downarrow$	ADE $\downarrow$	FDE $\downarrow$	ADE $\downarrow$	FDE $\downarrow$
GNM	8.6M	0.16	0.27	0.67	1.12	0.13	0.21	0.32	0.53
ViNT	29.6M	0.15	0.25	0.80	1.33	0.16	0.26	0.37	0.61
NoMaD	19.0M	0.24	0.43	1.311	2.35	0.22	0.39	0.59	1.06
VertiFormer	27.11M	0.10	0.18	0.43	0.80	0.10	0.19	0.21	0.39
Hydra	143.29M	0.12	0.20	0.48	0.85	0.12	0.21	0.24	0.42

Appendix

Table 6: Offline trajectory forecasting evaluation on a held-out subset across three navigation datasets (SCAND, TartanDrive, and SACSoN). Hydra achieves competitive policy accuracy while retaining the planning capability that reactive baselines lack.

Figure 4: Comparison of a 16-second video generated from a held-out subset of RECON. While NWM’s continuous latents drift from the commanded actions over time, Hydra ’s discrete manifold maintains action adherence over time.

Scenario	SR (%) $\uparrow$	Time (s) $\downarrow$
Frontal, $\lambda_{\text{img}}=1$	80	24.7 $\pm$ 5.8
Blind corner, $\lambda_{\text{img}}=1$	20	30.6 $\pm$ 12.1
Blind corner, $\lambda_{\text{img}}=0$	70	34.5 $\pm$ 8.9

Appendix

Table 7: Effect of the image-goal cost term (

\lambda_{\text{img}}

) conditioned on goal visibility. The same term that improves performance when the goal is visible degrades it when the goal is not, motivating conditioning

\mathcal{C}_{\text{img}}

on goal visibility rather than applying it unconditionally.

	Spot		Jackal
Configuration	Time (s) $\downarrow$	Collisions $\downarrow$	Time (s) $\downarrow$	Collisions $\downarrow$
Original (avoidance costs + $\mathcal{C}_{\text{geo}}$ )	30.6 $\pm$ 12.1	0.20 $\pm$ 0.42	23.8 $\pm$ 0.39	0.20 $\pm$ 0.13
$\lambda_{\text{prior}}=0$	33.9 $\pm$ 13.0	0.70 $\pm$ 0.48	21.2 $\pm$ 0.26	0.70 $\pm$ 0.59
$\lambda_{\text{vq}}=0$	30.3 $\pm$ 10.0	0.50 $\pm$ 0.53	20.9 $\pm$ 0.34	0.60 $\pm$ 0.28
$\lambda_{\text{entropy}}=0$	36.2 $\pm$ 18.9	0.40 $\pm$ 0.52	26.6 $\pm$ 0.45	0.80 $\pm$ 0.61

Appendix

Table 8: Ablation of obstacle-avoidance cost terms (

\lambda_{\text{prior}}

\lambda_{\text{vq}}

\lambda_{\text{entropy}}

), across two robot embodiments: a blind corner turn (Spot) and frontal obstacle avoidance (Jackal). Each avoidance term degrades performance when removed on both platforms, indicating the mechanism is embodiment-agnostic.

Figure 5: 1-step spatial coverage of the learned pose codebook

\mathcal{V}_{p}

. Unlike normalized baselines, the discrete manifold explicitly captures the absolute physical scaling of different robotic platforms, ranging from low-speed indoor maneuvers (dense central clusters) to high-speed off-road dynamics (sparse outer bounds).

Figure 6: Live RViz visualization of Hydra ’s Discrete Latent Planning search on the physical robot in a never-before-seen environment. Left: the robot’s current ego-centric observation. Middle: a subset of Hydra ’s predicted future frames. Right: the corresponding candidate trajectories ( green ) rendered as a branching tree in the robot’s local frame; each branch is one topological maneuver considered before $\mathcal{J}_{\text{KPC}}$ selects the lowest-cost ( purple ) candidate for execution.

Figure 7: Manifold Rejection and Predictive Entropy During Impending Collision. Top / Bottom insets: Comparison of Ground Truth vs. Hydra ’s imagination at points along a rollout commanded to drive straight toward a physical occlusion. Both the spatially-averaged VQ quantization error ( $\mathcal{C}_{vq}$ , solid blue, left axis) and the predictive entropy of the visual token distribution ( $\mathcal{C}_{ent}$ , dashed orange, right axis) rise together as the commanded trajectory approaches the occlusion, and drop together immediately after the model hallucinates a navigable resolution, two independently computed signals converging on the same collision-relevant moments. The smaller early fluctuation (first two insets) is a false positive triggered by visually complex but traversable foliage, not a genuine obstacle.

Figure 8: Vanishing humans in future predictions on SCAND. Top: ground-truth future frames, in which a pedestrian remains visible throughout. Bottom: Hydra ’s predicted frames for the same sequence, in which the pedestrian is smoothed into the background.

Figure 9: Goal-blind seed initialization causes a missed turn. The green trajectory is the path selected by the DLP cost among the sampled candidates at convergence; the seed distribution that generated the candidates was conditioned only on local scene context, not on the goal, delaying goal-aware correction until it is too late to complete the turn.

Panel of Figure 10 — Figure 10: LPIPS deviation from geometric grounding. Note that these are not generated images.

Figure 11: Physical experiment setup with obstacles.

\RowStyle Parameter	Value
\Block [l]1-2 Training & Optimization
Optimizer	AdamW
Learning Rate	$2\times 10^{-4}$
Weight Decay	$0.08$
Batch Size	8
Gradient Accumulation Steps	32

Appendix

Table 9: Hyperparameters and architectural details of Hydra . The model utilizes a Transformer backbone combined with dimensionally constrained, modality-specific VQ codebooks and Flow Matching decoders.

Figure 12: Early Training Dynamics (2.1K Steps). A comparison between the model’s early-stage generation, both with (middle) and without (bottom) the deterministic anchor head. While the bottom row suffers from spatial smearing, the model utilizing the deterministic head shows a better ability to construct structural details, like the building’s edges against the sky, even after very limited training.

$\lambda_{\text{img}}$	$\lambda_{\text{prior}}$	$\lambda_{\text{ent}}$	$\lambda_{\text{vq}}$	$\gamma$	# Candidates	Iterations	$top_{k}$ samples
0.02	0.5	0.5	1.0	0.85	18	3	10

Appendix

Table 10: DLP Hyperparameters.

Explore similar work

World Action Planner: Generalizable Decision-Making with Action-Conditioned World Models
Jul 30, 2026Xiangcheng Zhang, Yilun DuEfficient World-Action Model World Model Planning
Harvard University
Delta-JEPA: Learning Action-Sensitive World Models via Latent Difference Decoding
Jun 30, 2026Zhenghao Zhang, Yuanxiang Wang, Zhenyu Guan +11Latent World Models Video Joint Embedding Predictive Architecture
School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing · Institute of Information Engineering, Chinese Academy of Sciences, Beijing · School of Computer Science and Technology, Harbin Institute of Technology, Weihai +2
Representation World Model: Learning States, Transition and Executable Plans in Representation
Sep 24, 2026Yijun Yuan, Weicheng Zheng, Weibang Wang +7World Model Planning World Models
IIIS, Tsinghua University

Jul 30, 2026, cs.AIJ/K move · Enter open · S save

World Action Planner: Generalizable Decision-Making with Action-Conditioned World Models

Xiangcheng Zhang, Yilun Du

Harvard University

Efficient World-Action Model World Model Planning Robot Planning Decisions

Open paper AskPDF arXiv

Building generalizable agents for diverse applications remains a fundamental challenge. While imitation learning-based policies succeed in specific training environments, they often fail to generalize to novel scenes and tasks. In this work, we propose World Action Planner, a robot planning system that leverages the reasoning capabilities of Vision-Language Models (VLMs) and the physical grounding of a multi-task pose-image conditioned world model. Our system enables an agent to propose initial action plans and iteratively refine them via optimization and search, reasoning over imagined world model rollouts. We demonstrate that our approach achieves superior performance across compositional tasks, new layouts, and zero-shot generalization scenarios, significantly outperforming state-of-the-art end-to-end policy models such as VLAs and WAMs. Project website at worldactionplanner.github.io

Abstract

Explore similar work

World Action Planner: Generalizable Decision-Making with Action-Conditioned World Models

Delta-JEPA: Learning Action-Sensitive World Models via Latent Difference Decoding

Representation World Model: Learning States, Transition and Executable Plans in Representation