World models are a promising path to zero-shot embodied control through planning. However, existing world model planners struggle on long-horizon, multi-stage tasks: prediction errors compound and naive search is exponential in the planning horizon. Hierarchy mitigates both by decomposing tasks into shorter, tractable subproblems; yet prior hierarchical approaches either amortize control into task-specific policies (hierarchical RL) or assume low-dimensional states and known dynamics (classical hierarchical MPC). We present Hierarchical Planning with Latent World Models (HWM), an architecture and planning paradigm for hierarchical model predictive control directly on visual world models trained solely via next-latent prediction. HWM learns world models at multiple temporal scales within a shared latent space, so predictions from the long horizon model serve as subgoals for the short horizon model via latent matching, without task-specific rewards or hierarchical policies. To keep long-horizon search tractable, HWM learns an action encoder that compresses primitive action chunks into latent macro-actions. On real-world Franka manipulation, HWM solves pick-and-place from a single goal image at 70% success vs. 0% for single-level planning. Across simulated push manipulation and maze navigation, HWM consistently improves performance on long-horizon tasks while requiring up to 3x less planning compute.
Figures & tables
Figure 1: Top: Hierarchical planning in latent space. A high-level planner optimizes macro-actions using a long-horizon world model to reach the goal; the first predicted latent state from the high-level rollout serves as the subgoal for a low-level planner, which optimizes primitive actions with a short-horizon world model. Both levels replan as new observations arrive, in a closed-loop MPC fashion. Solid borders denote ground-truth observations; others are decoder reconstructions shown for interpretability only. Bottom Left: HWM improves long-horizon planning on 3 latent world model architectures across 4 task suites. Bottom Right: Success rate versus normalized planning time. HWM matches or exceeds performance with up to 3× lower planning time.
Figure 2: Hierarchical planning in latent space. A high-level planner optimizes macro actions using a long-horizon latent world model to reach the final goal. The first high level latent prediction serves as a subgoal for low-level planning, where a short-horizon world model optimizes primitive actions to reach this subgoal.
Figure 3: The high-level world model captures long-horizon dynamics using macro-actions encoded from chunks of low-level actions. It is causal and trained with a latent prediction loss.
Figure 4: Executions on pick-&-place and drawer manipulation with a dual-gripper Franka. Dotted red columns indicate subgoals inferred by the high-level planner and decoded for visualization. Executions on additional tasks are shown in appendix I .
Method
P&P Cup
P&P Box
Drawer
Octo
0%
0%
43%
π0 -FAST-DROID
52%
18%
–
π0.5 -DROID
68%
36%
–
VJEPA2-AC
0%
0%
30%
VJEPA2-AC (hierarchy)
70%
60%
70%
Table 1: Zero-shot robotic manipulation with Franka , evaluated end-to-end from a single goal image. Each rate is computed over N=5 trials per (start, goal) configuration; per-configuration breakdowns and 95% Clopper–Pearson confidence intervals are reported in Table 16 . The flat VJEPA2-AC planner fails on tasks with non-greedy objectives (pick-&-place, multi-stage drawer); HWM solves them without manual decomposition.
Method
d=25
d=50
d=75
GCIQL
40%
25%
7.5%
HIQL
55%
30%
20%
HILP
25%
13%
0%
DINO-WM
84%
55%
17%
DINO-WM (hier.)
89%
78%
61%
Table 2: Push-T Performance Across Task Horizons . In each trial, a random start–goal pair d steps apart is sampled from a validation trajectory.
Figure 5: Success rate vs. test-time compute. We vary the planning budget by sweeping CEM/MPPI samples and iterations (see appendix C ). Hierarchical planning achieves higher success while using up to 3× less test-time compute than flat planners. Left: Push-T success rate vs. planning time ( d=50 ). Right: Diverse maze success rate vs. planning time ( D∈[9,12] ).
Method
D∈[5,8]
D∈[9,12]
D∈[13,16]
GCIQL
85%
40%
33%
HIQL
88%
73%
48%
HILP
48%
20%
10%
PLDM
100%
84%
44%
PLDM (hier.)
100%
95%
83%
Table 3: Diverse Maze Performance Across Task Horizons . Agents are evaluated zero-shot on navigation tasks in held-out test maps. For each task, start and goal positions are randomly sampled such that their shortest-path distance is D .
Figure 6: Prediction error ( L1 ) as a function of prediction horizon. For short horizons ( ≤ 1 s), the low-level world model is more accurate. For longer horizons ( ≥ 1.5 s), a single-step prediction from the high-level world model outperforms autoregressive rollouts of the low-level model, reflecting reduced error accumulation.
Table 4: Learned macro-actions versus alternative high-level action representations. (a) Concatenated primitive actions: planning success rates on medium/hard Diverse Maze tasks. (b) Handcrafted delta-pose actions, which summarize each action sequence by its net end-effector displacement: alignment of the offline plan with expert behavior on Franka (mean ± SE), measured by cosine similarity and ℓ1 distance.
Method
Zero-shot
Representation
Training
Hierarchical Interface
HWM (ours)
✓
pixels → latent
Next-latent pred.
Subgoal matching (latent)
CAVIN [ 20 ]
✓
structured state
Learned dynamics
Subgoal matching (state)
LAT [ 21 ]
✓
structured state
Learned dynamics
Skill-conditioned policy
Safe HMPC [ 22 ]
×
structured state
Analytical dynamics
Reference tracking
IQL-TD-MPC [ 42 ]
×
state → latent
Offline RL + MPC
Goal-conditioned policy
Director,Puppeteer [ 14 , 19 ]
×
pixels → latent
Model-based RL
Goal-conditioned policy
Table 5: Comparison of hierarchical model-based methods. HWM is the only entry supporting zero-shot MPC directly from pixels, using world models pretrained via next-latent prediction, with predictions from coarser models serving as subgoals for finer-scale MPC via latent matching.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
World Model
Epochs
Batch Size
γtf
γroll
context T
VJEPA2-AC (Low-Level)
200
256
1.0
1.0
16
VJEPA2-AC (High-Level)
120
768
1.0
0.0
3
Appendix
Table 6: Training hyperparameters for Droid + Roboset. γtf and γroll denote the coefficients for teacher-forcing and rollout losses, respectively, and context T counts observation frames, including the initial frame. All other hyperparameters follow those used by VJEPA2-AC.
World Model
Epochs
Batch Size
γtf
γroll
context T
DINO-WM (Low-Level)
100
256
1.0
0.0
4
DINO-WM (High-Level)
500
128
1.0
0.0
5
Appendix
Table 7: Training hyperparameters for Push-T. γtf and γroll denote the coefficients for teacher-forcing and rollout losses, respectively, and context T counts frames including the initial observation. All other hyperparameters follow those used by DINO-WM on Push-T.
World Model
Epochs
Batch Size
lr
α
β
λ
ω
γtf
γroll
pred T
Lproprio
PLDM (Low-Level)
3
128
0.018
29.4
17.9
2.80
4.81
0.0
1.0
14
2.42
PLDM (High-Level)
5
128
0.018
–
–
–
–
0.0
1.0
6
1
Appendix
Table 8: Training hyperparameters for Diverse Maze. α , β , λ , and ω are parameters in the PLDM objective used to prevent representation collapse. Lproprio coef. denotes the coefficient on the proprioceptive prediction loss. γtf and γroll are the coefficients for teacher-forcing and rollout losses, and pred T is the number of autoregressive prediction steps during training, excluding the initial state. For the high-level world model, we reuse the encoder trained at the low level and keep it frozen, eliminating additional representation learning.
Table 15
Figure 7: Success rate as a function of test-time compute for Push-T. Left : Setting d=50 . Right : Setting d=75 .
Figure 8: Success rate as a function of test-time compute for Diverse Maze. Left : Setting D∈[9,12] . Right : Setting D∈[13,16] .
Table 13: Capacity-controlled comparison between flat and hierarchical world models. Flat models with comparable or larger parameter counts fail to match hierarchical performance, and sometimes degrade with increased capacity. In contrast, hierarchical models consistently outperform flat baselines on Push-T and Diverse Maze, indicating that gains arise from hierarchical structure rather than model capacity.
High-level stride
6
8
10
12
14
Success Rate ( D∈[13,16] )
50%
55%
77%
78%
75%
Appendix
Table 14: Sensitivity of hierarchical planning performance to the high-level temporal stride on Diverse Maze.
Figure 9: Effect of macro-action dimension on hierarchical planning. Left: Performance as a function of macro-action dimension d . When the latent space has very low capacity, the high-level planner fails to produce valid plans. As d increases, success improves; however, beyond a moderate dimensionality, proposed subgoals become harder for the low-level planner to execute, as indicated by a drop in action cosine similarity to expert behavior. Right: Qualitative rollouts illustrating this trade-off. Moderate latent dimensionality biases the planner toward reachable, greedy subgoals, yielding the best overall performance.
Planner
Subgoal
Cup
Box
VJEPA2-AC
None
0%
0%
HWM
Predicted
70%
60%
VJEPA2-AC
Manual
80%
80%
HWM
Manual
80%
80%
Octo
Manual
20%
10%
Appendix
Table 15: Manual subgoals on Franka. Success rates (%) over 50 trials per object (10 configurations, 5 per configuration).
Figure 10: High-level plans in Diverse Maze. The agent is shown as a green ball and the goal as a yellow star. The pink trajectory is generated by the high-level planner, while the teal trajectory is generated by the flat planner. In these trials, the high-level planner successfully produces a sequence of subgoals that lead to the goal, whereas the flat planner often struggles due to the increased search complexity of long-horizon prediction.
Figure 11: Initial and goal states for all evaluation tasks. (a) Drawer Tasks: 7 subtasks involving opening and closing drawers at various positions. (b) Pick-&-Place Cup Tasks: 10 subtasks requiring the robot to pick up a cup and place it at different target locations. Each panel shows the initial state (left) and goal state (right) for a given task.
Table 16: Per-configuration success counts (successes / trials) on Franka, with 95% Clopper–Pearson confidence intervals on aggregate rates. Configurations correspond to the start–goal pairs visualized in appendix G .
Figure 12: Executions on pick-&-place and drawer manipulation with a dual-gripper Franka arm. Dotted red columns indicate subgoals inferred by the high-level planner and decoded for visualization. Rows representing failed episodes are labeled with "Failure". Additional execution videos can be found in https://kevinghst.github.io/HWM/ .
Figure 13: Executions on Push-T task. Dotted red columns indicate subgoals inferred by the high-level planner and decoded for visualization.
GCIQL
HIQL
HILP
lr
expectile
rand goal p
lr
expectile
rand goal p
alpha
lr
expectile
Push-T( d=25 )
3e−5
0.6
0.0
3e−5
0.9
0.0
3.0
3e−5
0.6
Push-T( d=50 )
3e−5
0.6
0.3
3e−5
0.9
0.3
1.0
3e−5
0.6
Push-T( d=75 )
3e−5
0.9
0.6
3e−5
0.9
0.3
1.0
3e−5
0.6
Maze( D∈[5,8] )
3e−5
0.8
0.0
3e−5
0.6
0.3
1.0
3e−5
0.9
Maze( D∈[9,12] )
3e−5
0.6
0.3
3e−5
0.8
0.6
3.0
3e−5
0.9
Appendix
Table 17: Hyperparameters. Hyperparameters used for GCIQL, HIQL, and HILP on the Push-T and Diverse Maze tasks.
Latent world models often struggle with long-horizon planning despite accurate short-term predictions. Recursive rollouts accumulate errors, while distance concentration in high-dimensional latent spaces can weaken goal discrimination. We introduce the Dual-Latent World Model (Dual-WM), which separates local execution and long-range planning through distinct state representations and dynamics models. The low-level model predicts action-conditioned transitions, while the high-level model uses learned macro-actions to plan over longer temporal spans. We also propose Long-Horizon Representation Learning with Weighted Rollout (LoRe), which supervises self-generated predictions at both levels. An analysis of recursive error propagation motivates exponential horizon weights with separate decay rates for the two temporal scales. During planning, the high-level model generates latent subgoals that the low-level model refines into actions for precise execution. We evaluate from-scratch Dual-WM on five goal-conditioned visual control tasks against the task-wise strongest baselines without actor-guided proposals. At goal offsets of 50 and 100 environment steps, mean success increases from 75.9% to 84.4% and from 61.4% to 69.5%, respectively. At offset 100, Dual-WM outperforms these baselines on all five tasks and improves mean success over LeWM by 30.8 percentage points. Ablations and supporting analyses provide evidence of more informative representations for goal evaluation and greater consistency under recursive prediction. These results highlight the value of separating temporal roles and training across multiple horizons for reliable latent planning. Our core implementation is available at https://github.com/DeLin1001/Dual-WM-Official.
Delin Zhao, Zhengrong Yue, Shaobin Zhuang +6
Nanjing University · Shenzhen University of Advanced Technology · Shanghai Jiao Tong University +1
Long-horizon planning with latent world models requires reasoning across timescales and levels of abstraction. Existing task-agnostic JEPA world models predict and plan at a single timescale or with multiple horizons in one shared latent space. We introduce H-JEPA, an end-to-end recipe for training a hierarchy of action-conditioned JEPAs in which each level predicts farther ahead in its own learned latent space. Planning proceeds top-down: the top level optimizes progress toward the goal, and each level's predictions become subgoals for the planner below it. When factors in the data evolve at separated timescales, higher levels discard fast, unpredictable detail and retain slower task-relevant state. Across four simulated navigation and manipulation environments, hierarchical planning improves over a flat JEPA; on Visual AntMaze, a three-level hierarchy raises success from 18% to 73% using less planner compute. Ablations attribute these gains to both temporal decomposition and higher-level goal representations. With inverse-dynamics supervision, the approach extends to diverse real-robot videos from DROID, where hierarchy improves offline planning fidelity at lower planner compute.
Wancong Zhang, Basile Terver, Michael Rabbat +2
NYU · Advanced Machine Intelligence · INRIA Paris +1
We investigate whether temporal hierarchy can improve LeWorldModel on long-horizon goal-conditioned control. We introduce Hi-LeWM, an extension that freezes the pretrained low-level LeWM and adds high-level planning over latent subgoals. We evaluate Hi-LeWM on PushT and Cube across increasing goal offsets. Hierarchy does not automatically improve performance: at short horizons, the best configuration uses a one-step high-level horizon, while longer horizons reveal a mismatch between the learned high-level action space and the inference-time search distribution. Experiments with true future latent subgoals show that the frozen low-level controller can execute well-aligned intermediate targets, indicating that high-level subgoal generation is the main bottleneck. Unconstrained search can select latent macro-actions that appear favorable under the learned model but produce poor control targets. Constraining search around macro-actions encoded from training trajectories, with appropriate subgoal execution timing, recovers useful hierarchical regimes, improving over flat LeWM by +11.3 percentage points at medium-range horizons and +14.7 percentage points at the longest PushT horizon. Overall, temporal abstraction can benefit compact frozen LeWM, but only when high-level search remains compatible with the low-level controller
Niccolò Caselli, Francesco Massafra, Samuele Punzo +3