Hierarchical Planning with Latent World Models

Wancong Zhang; Basile Terver; Artem Zholus; Soham Chitnis; Harsh Sutaria; Mido Assran; Randall Balestriero; Amir Bar

cs.LGApr 3, 2026

Hierarchical Planning with Latent World Models

Authors: Wancong Zhang, Basile Terver, Artem Zholus, Soham Chitnis, Harsh Sutaria, Mido Assran, Randall Balestriero, Amir Bar, +2 more

Organizations: FAIR at Meta · New York University · Mila - Québec AI Institute · Brown University

Abstract

World models are a promising path to zero-shot embodied control through planning. However, existing world model planners struggle on long-horizon, multi-stage tasks: prediction errors compound and naive search is exponential in the planning horizon. Hierarchy mitigates both by decomposing tasks into shorter, tractable subproblems; yet prior hierarchical approaches either amortize control into task-specific policies (hierarchical RL) or assume low-dimensional states and known dynamics (classical hierarchical MPC). We present Hierarchical Planning with Latent World Models (HWM), an architecture and planning paradigm for hierarchical model predictive control directly on visual world models trained solely via next-latent prediction. HWM learns world models at multiple temporal scales within a shared latent space, so predictions from the long horizon model serve as subgoals for the short horizon model via latent matching, without task-specific rewards or hierarchical policies. To keep long-horizon search tractable, HWM learns an action encoder that compresses primitive action chunks into latent macro-actions. On real-world Franka manipulation, HWM solves pick-and-place from a single goal image at 70% success vs. 0% for single-level planning. Across simulated push manipulation and maze navigation, HWM consistently improves performance on long-horizon tasks while requiring up to 3x less planning compute.

Figures & tables

Panel of Figure 1 — Figure 1: Top: Hierarchical planning in latent space. A high-level planner optimizes macro-actions using a long-horizon world model to reach the goal; the first predicted latent state from the high-level rollout serves as the subgoal for a low-level planner, which optimizes primitive actions with a short-horizon world model. Both levels replan as new observations arrive, in a closed-loop MPC fashion. Solid borders denote ground-truth observations; others are decoder reconstructions shown for interpretability only. Bottom Left: HWM improves long-horizon planning on 3 latent world model architectures across 4 task suites. Bottom Right: Success rate versus normalized planning time. HWM matches or exceeds performance with up to $3\times$ lower planning time.

Panel of Figure 2 — Figure 2: Hierarchical planning in latent space. A high-level planner optimizes macro actions using a long-horizon latent world model to reach the final goal. The first high level latent prediction serves as a subgoal for low-level planning, where a short-horizon world model optimizes primitive actions to reach this subgoal.

Figure 3: The high-level world model captures long-horizon dynamics using macro-actions encoded from chunks of low-level actions. It is causal and trained with a latent prediction loss.

Figure 4: Executions on pick-&-place and drawer manipulation with a dual-gripper Franka. Dotted red columns indicate subgoals inferred by the high-level planner and decoded for visualization. Executions on additional tasks are shown in appendix I .

Method	P&P Cup	P&P Box	Drawer
Octo	0%	0%	43%
$\pi_{0}$ -FAST-DROID	52%	18%	–
$\pi_{0.5}$ -DROID	68%	36%	–
VJEPA2-AC	0%	0%	30%
VJEPA2-AC (hierarchy)	70%	60%	70%

Table 1: Zero-shot robotic manipulation with Franka , evaluated end-to-end from a single goal image. Each rate is computed over

N=5

trials per (start, goal) configuration; per-configuration breakdowns and 95% Clopper–Pearson confidence intervals are reported in Table 16 . The flat VJEPA2-AC planner fails on tasks with non-greedy objectives (pick-&-place, multi-stage drawer); HWM solves them without manual decomposition.

Method	$d=25$	$d=50$	$d=75$
GCIQL	$40\%$	$25\%$	$7.5\%$
HIQL	$55\%$	$30\%$	$20\%$
HILP	$25\%$	$13\%$	$0\%$
DINO-WM	$84\%$	$55\%$	$17\%$
DINO-WM (hier.)	89%	78%	61%

Table 2: Push-T Performance Across Task Horizons . In each trial, a random start–goal pair

d

steps apart is sampled from a validation trajectory.

Panel of Figure 5 — Figure 5: Success rate vs. test-time compute. We vary the planning budget by sweeping CEM/MPPI samples and iterations (see appendix C ). Hierarchical planning achieves higher success while using up to $3\times$ less test-time compute than flat planners. Left: Push-T success rate vs. planning time ( $d=50$ ). Right: Diverse maze success rate vs. planning time ( $D\!\in\![9,12]$ ).

Method	$D\!\in\![5,8]$	$D\!\in\![9,12]$	$D\!\in\![13,16]$
GCIQL	85%	40%	33%
HIQL	88%	73%	48%
HILP	48%	20%	10%
PLDM	100%	$84\%$	$44\%$
PLDM (hier.)	100%	95%	83%

Table 3: Diverse Maze Performance Across Task Horizons . Agents are evaluated zero-shot on navigation tasks in held-out test maps. For each task, start and goal positions are randomly sampled such that their shortest-path distance is

D

Figure 6: Prediction error (

L_{1}

) as a function of prediction horizon. For short horizons (

\leq

1 s), the low-level world model is more accurate. For longer horizons (

\geq

1.5 s), a single-step prediction from the high-level world model outperforms autoregressive rollouts of the low-level model, reflecting reduced error accumulation.

Table 4: Learned macro-actions versus alternative high-level action representations. (a) Concatenated primitive actions: planning success rates on medium/hard Diverse Maze tasks. (b) Handcrafted delta-pose actions, which summarize each action sequence by its net end-effector displacement: alignment of the offline plan with expert behavior on Franka (mean

\pm

SE), measured by cosine similarity and

\ell_{1}

distance.

Method	Zero-shot	Representation	Training	Hierarchical Interface
HWM (ours)	✓	pixels $\rightarrow$ latent	Next-latent pred.	Subgoal matching (latent)
CAVIN [ 20 ]	✓	structured state	Learned dynamics	Subgoal matching (state)
LAT [ 21 ]	✓	structured state	Learned dynamics	Skill-conditioned policy
Safe HMPC [ 22 ]	$\times$	structured state	Analytical dynamics	Reference tracking
IQL-TD-MPC [ 42 ]	$\times$	state $\rightarrow$ latent	Offline RL + MPC	Goal-conditioned policy
Director,Puppeteer [ 14 , 19 ]	$\times$	pixels $\rightarrow$ latent	Model-based RL	Goal-conditioned policy

Table 5: Comparison of hierarchical model-based methods. HWM is the only entry supporting zero-shot MPC directly from pixels, using world models pretrained via next-latent prediction, with predictions from coarser models serving as subgoals for finer-scale MPC via latent matching.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

World Model	Epochs	Batch Size	$\gamma_{tf}$	$\gamma_{roll}$	context $T$
VJEPA2-AC (Low-Level)	200	256	1.0	1.0	16
VJEPA2-AC (High-Level)	120	768	1.0	0.0	3

Appendix

Table 6: Training hyperparameters for Droid + Roboset.

\gamma_{tf}

and

\gamma_{roll}

denote the coefficients for teacher-forcing and rollout losses, respectively, and context

T

counts observation frames, including the initial frame. All other hyperparameters follow those used by VJEPA2-AC.

World Model	Epochs	Batch Size	$\gamma_{tf}$	$\gamma_{roll}$	context $T$
DINO-WM (Low-Level)	100	256	1.0	0.0	4
DINO-WM (High-Level)	500	128	1.0	0.0	5

Appendix

Table 7: Training hyperparameters for Push-T.

\gamma_{tf}

and

\gamma_{roll}

denote the coefficients for teacher-forcing and rollout losses, respectively, and context

T

counts frames including the initial observation. All other hyperparameters follow those used by DINO-WM on Push-T.

World Model	Epochs	Batch Size	lr	$\alpha$	$\beta$	$\lambda$	$\omega$	$\gamma_{tf}$	$\gamma_{roll}$	pred $T$	$\mathcal{L}_{\text{proprio}}$
PLDM (Low-Level)	3	128	0.018	29.4	17.9	2.80	4.81	0.0	1.0	14	2.42
PLDM (High-Level)	5	128	0.018	–	–	–	–	0.0	1.0	6	1

Appendix

Table 8: Training hyperparameters for Diverse Maze.

\alpha

\beta

\lambda

, and

\omega

are parameters in the PLDM objective used to prevent representation collapse.

\mathcal{L}_{\text{proprio}}

coef. denotes the coefficient on the proprioceptive prediction loss.

\gamma_{tf}

and

\gamma_{roll}

are the coefficients for teacher-forcing and rollout losses, and pred

T

is the number of autoregressive prediction steps during training, excluding the initial state. For the high-level world model, we reuse the encoder trained at the low level and keep it frozen, eliminating additional representation learning.

Table 15

Figure 7: Success rate as a function of test-time compute for Push-T. Left : Setting

d=50

. Right : Setting

d=75

Figure 8: Success rate as a function of test-time compute for Diverse Maze. Left : Setting

D\in[9,12]

. Right : Setting

D\in[13,16]

Table 13: Capacity-controlled comparison between flat and hierarchical world models. Flat models with comparable or larger parameter counts fail to match hierarchical performance, and sometimes degrade with increased capacity. In contrast, hierarchical models consistently outperform flat baselines on Push-T and Diverse Maze, indicating that gains arise from hierarchical structure rather than model capacity.

High-level stride	6	8	10	12	14
Success Rate ( $D\!\in\![13,16]$ )	50%	55%	77%	78%	75%

Appendix

Table 14: Sensitivity of hierarchical planning performance to the high-level temporal stride on Diverse Maze.

Panel of Figure 9 — Figure 9: Effect of macro-action dimension on hierarchical planning. Left: Performance as a function of macro-action dimension $d$ . When the latent space has very low capacity, the high-level planner fails to produce valid plans. As $d$ increases, success improves; however, beyond a moderate dimensionality, proposed subgoals become harder for the low-level planner to execute, as indicated by a drop in action cosine similarity to expert behavior. Right: Qualitative rollouts illustrating this trade-off. Moderate latent dimensionality biases the planner toward reachable, greedy subgoals, yielding the best overall performance.

Planner	Subgoal	Cup	Box
VJEPA2-AC	None	0%	0%
HWM	Predicted	70%	60%
VJEPA2-AC	Manual	80%	80%
HWM	Manual	80%	80%
Octo	Manual	20%	10%

Appendix

Table 15: Manual subgoals on Franka. Success rates (%) over 50 trials per object (10 configurations, 5 per configuration).

Figure 10: High-level plans in Diverse Maze. The agent is shown as a green ball and the goal as a yellow star. The pink trajectory is generated by the high-level planner, while the teal trajectory is generated by the flat planner. In these trials, the high-level planner successfully produces a sequence of subgoals that lead to the goal, whereas the flat planner often struggles due to the increased search complexity of long-horizon prediction.

Figure 11: Initial and goal states for all evaluation tasks. (a) Drawer Tasks: 7 subtasks involving opening and closing drawers at various positions. (b) Pick-&-Place Cup Tasks: 10 subtasks requiring the robot to pick up a cup and place it at different target locations. Each panel shows the initial state (left) and goal state (right) for a given task.

Table 16: Per-configuration success counts (successes / trials) on Franka, with 95% Clopper–Pearson confidence intervals on aggregate rates. Configurations correspond to the start–goal pairs visualized in appendix G .

Figure 12: Executions on pick-&-place and drawer manipulation with a dual-gripper Franka arm. Dotted red columns indicate subgoals inferred by the high-level planner and decoded for visualization. Rows representing failed episodes are labeled with "Failure". Additional execution videos can be found in https://kevinghst.github.io/HWM/ .

Figure 13: Executions on Push-T task. Dotted red columns indicate subgoals inferred by the high-level planner and decoded for visualization.

	GCIQL			HIQL				HILP
	lr	expectile	rand goal $p$	lr	expectile	rand goal $p$	alpha	lr	expectile
Push-T( $d=25$ )	$3\mathrm{e}{-5}$	0.6	0.0	$3\mathrm{e}{-5}$	0.9	0.0	3.0	$3\mathrm{e}{-5}$	0.6
Push-T( $d=50$ )	$3\mathrm{e}{-5}$	0.6	0.3	$3\mathrm{e}{-5}$	0.9	0.3	1.0	$3\mathrm{e}{-5}$	0.6
Push-T( $d=75$ )	$3\mathrm{e}{-5}$	0.9	0.6	$3\mathrm{e}{-5}$	0.9	0.3	1.0	$3\mathrm{e}{-5}$	0.6
Maze( $D\in[5,8]$ )	$3\mathrm{e}{-5}$	0.8	0.0	$3\mathrm{e}{-5}$	0.6	0.3	1.0	$3\mathrm{e}{-5}$	0.9
Maze( $D\in[9,12]$ )	$3\mathrm{e}{-5}$	0.6	0.3	$3\mathrm{e}{-5}$	0.8	0.6	3.0	$3\mathrm{e}{-5}$	0.9

Appendix

Table 17: Hyperparameters. Hyperparameters used for GCIQL, HIQL, and HILP on the Push-T and Diverse Maze tasks.

Explore similar work

Beyond a single latent space: a dual-latent world model for long-horizon planning
Sep 29, 2026Delin Zhao, Zhengrong Yue, Shaobin Zhuang +6World Model-Based Planning Action-Conditioned World Models
Nanjing University · Shenzhen University of Advanced Technology · Shanghai Jiao Tong University +1
H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning
Oct 5, 2026Wancong Zhang, Basile Terver, Michael Rabbat +2World Models for Robotics World Models
NYU · Advanced Machine Intelligence · INRIA Paris +1
Enhancing Latent World Models with Associative Memory
Oct 14, 2026Yuwei Sun, Siyu ZhuWorld Model Learning World Models for Robotics
Shanghai Academy of AI for Science · Fudan University

Sep 29, 2026J/K move · Enter open · S save

Beyond a single latent space: a dual-latent world model for long-horizon planning

Delin Zhao, Zhengrong Yue, Shaobin Zhuang, Junlin He+5

Nanjing University · Shenzhen University of Advanced Technology · Shanghai Jiao Tong University+1

World Model-Based Planning Action-Conditioned World Models Long-Horizon Planning Latent World Models

Open paper AskPDF arXiv

Latent world models often struggle with long-horizon planning despite accurate short-term predictions. Recursive rollouts accumulate errors, while distance concentration in high-dimensional latent spaces can weaken goal discrimination. We introduce the Dual-Latent World Model (Dual-WM), which separates local execution and long-range planning through distinct state representations and dynamics models. The low-level model predicts action-conditioned transitions, while the high-level model uses learned macro-actions to plan over longer temporal spans. We also propose Long-Horizon Representation Learning with Weighted Rollout (LoRe), which supervises self-generated predictions at both levels. An analysis of recursive error propagation motivates exponential horizon weights with separate decay rates for the two temporal scales. During planning, the high-level model generates latent subgoals that the low-level model refines into actions for precise execution. We evaluate from-scratch Dual-WM on five goal-conditioned visual control tasks against the task-wise strongest baselines without actor-guided proposals. At goal offsets of 50 and 100 environment steps, mean success increases from 75.9% to 84.4% and from 61.4% to 69.5%, respectively. At offset 100, Dual-WM outperforms these baselines on all five tasks and improves mean success over LeWM by 30.8 percentage points. Ablations and supporting analyses provide evidence of more informative representations for goal evaluation and greater consistency under recursive prediction. These results highlight the value of separating temporal roles and training across multiple horizons for reliable latent planning. Our core implementation is available at https://github.com/DeLin1001/Dual-WM-Official.

Abstract

Explore similar work

Beyond a single latent space: a dual-latent world model for long-horizon planning

H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning

Enhancing Latent World Models with Associative Memory