Planning with world models enables test-time adaptation for embodied agents, offering superior generalization over direct policy learning. However, long-horizon tasks remain hindered by cumulative errors and search complexity. Existing hierarchical methods often use rigid macro-actions or latent skills, leading to brittle representations and limited fine-grained control. We introduce ID-WM (Implicit Dynamics World Models), a hierarchical world model that decouples state-space reachability from local control. Our framework utilizes a high-level model to sample feasible transitions across the state manifold and a low-level world model as an inverse dynamics solver to ground these transitions into precise actions. This allows the planner to identify reachable subgoals without being constrained by low-level execution details. By prioritizing local physical feasibility over global behavioral patterns, ID-WM synthesizes novel long-horizon plans from short segments, transcending the specific behavioral modes seen during training. We demonstrate its effectiveness across three navigation and manipulation suites, achieving a 38.5% improvement over state-of-the-art policy and diffusion-based planners. Crucially, by leveraging this hierarchical structure to mitigate compounding errors, our approach yields a 135% performance gain over single-level world model MPC, particularly in complex, unseen configurations for long-horizon tasks. Videos can be viewed at https://id-wm.github.io
Figures & tables
Figure 1: Overview of ID-WM . Our hierarchical planning framework decouples state-space reachability from local control, eliminating the reliance on task-specific value functions or pre-trained policies. Planning proceeds in two interleaved stages: (1) high-level planning, which discovers a sequence of temporally abstract subgoals spaced Δ steps apart, and (2) low-level control, which leverages a world model as an inverse dynamics solver to extract executable actions between each subgoal.
Figure 2: Architecture of ID-WM . Given an initial observation o0 and a goal og , we optimize for both the sequence of subgoals s0,…,sTΔ and the executable actions at in an interleaved fashion. Here, fhi denotes the diffusion-based high-level world model and flo denotes the deterministic, transformer-based low-level world model, both of which predict in latent space.
Figure 3: We evaluate ID-WM on three environment suites (Wall, PushT, Maze); from left to right: Wall, PushT, and the four maze sizes (U-, Medium, Large, Giant).
Figure 4: Diverse neighboring states generated with the high-level WM of ID-WM .
Figure 5: High-level subgoals planned by ID-WM . The planned sequence successfully guides the agent to the target goal while ensuring that consecutive subgoals satisfy local reachability constraints.
PushT
PushT-Aug
Method
Fix
Rand.
Fix
Rand.
HWM
0.38
0.20
0.42
0.16
ID-WM
1.00
0.77
1.00
1.00
Table 2: Success rate of ID-WM and HWM on PushT environments with fixed (Fix) and random (Rand.) goals.
Figure 6: Diverse plans generated by ID-WM . The top and bottom rows illustrate distinct strategies: counterclockwise and clockwise rotations of the T-block, both resulting in successful task completion.
Figure 7: Planning performance with increasing task complexity. In a), we test ID-WM ’s ability on Maze with increasing sizes. In b), we test ID-WM on increasingly longer horizon PushT tasks.
Figure 8: Subgoals generated by ID-WM on a visual PushT task. The first column denotes the starting state, and the last shaded column denotes the desired goal state. ID-WM generates reliable, sequential subgoals that drive steady progress toward the goal state without any degenerate states.
Method
d=25
d=50
d=75
Random Goal
DINO-WM
0.84
0.16
0.00
0.00
HWM
0.96
0.46
0.14
0.02
ID-WM
0.90
0.60
0.50
0.26
Table 3: Success rate on vision-based PushT. For d∈{25,50,75} the goal is d steps away; Random Goal samples both the initial and goal states randomly.
Figure 9: Examples of sampled initial and goal state pairs for the Wall, PushT, Giant Maze, Large Maze, and U-Maze environments
Figure 10: Success rate with respect to CEM samples and noise regularization.
Δ \ #Subgoals
3
4
5
6
8
12
25
1
0.06
0.04
0.06
0.08
0.04
0.10
0.06
2
0.14
0.14
0.28
0.50
0.64
0.48
0.24
3
0.50
0.56
0.64
0.82
0.76
0.74
0.42
4
0.60
0.76
0.86
0.86
1.00
0.90
0.44
5
0.74
0.84
0.96
0.98
0.94
0.76
0.38
Table 4: Success rate on PushT-Aug (Random Goal) when sweeping the subgoal horizon Δ (rows) and the number of subgoals per high-level plan (columns). The default configuration ( Δ=5 , 5 subgoals) is underlined.
Method
PushT-Aug (Rand.)
Medium Maze
HIQL
307.4
34.1
GCBC
1014.9
336.8
Diffuser
N/A
287.8
H-Diffuser
713.9
169.4
SL-WM
312.5
229.5
ID-WM
234.7
163.9
Table 5: Average steps to success over successful trajectories (lower is better). N/A indicates that no trajectory succeeded.
Table 15
Method
Samples
Elites
Replans
Horizon
SR
SL-WM (CEM budget matched)
300
30
15
25
0.21
SL-WM (inference time matched)
300
30
45
25
0.28
ID-WM
300
30
15
5 (each level)
0.77
Table 8: Success rate on PushT (Random Goal) when matching either the CEM budget or the wall-clock inference time of SL-WM to ID-WM .
Table 17Table 18Table 19
Name
Value
lr
3e−4
Discount factor γ
0.99
Expectile τ
0.7
High-level α
3.0
Low-level α
3.0
Subgoal steps ksub
25
Table 15: Shared hyperparameters for HIQL across all evaluation environments.
Table 21
Model
Execution
PushT
PushT-Aug
Fix Goal
Random Goal
Fix Goal
Random Goal
SR
Cov
SR
Cov
SR
Cov
SR
Cov
Diffuser
execute plan + replan
0.00
0.03
0.00
0.03
0.28
0.37
0.06
0.19
Diffuser
execute one action + replan
0.00
0.11
0.00
0.06
0.04
0.27
0.02
0.15
Diffuser
execute plan
0.00
0.05
0.00
0.03
0.00
0.09
0.00
0.13
Table 18: Diffuser results across execution settings and PushT variants.
Model
Execution
PushT
PushT-Aug
Fix Goal
Random Goal
Fix Goal
Random Goal
SR
Cov
SR
Cov
SR
Cov
SR
Cov
H-Diffuser
execute plan + replan
0.38
0.28
0.00
0.08
0.56
0.51
0.46
0.36
H-Diffuser
execute one action + replan
0.00
0.25
0.02
0.11
0.56
0.56
0.26
0.50
H-Diffuser
execute plan
0.04
0.09
0.00
0.06
0.02
0.17
0.00
0.10
Table 19: H-Diffuser results across execution settings and PushT variants.
Figure 11: Training-time convergence of predicted future-state distributions for the PushT environment
Figure 12: Visualization of the induced state mappings by ID-WM and naive WM.
World models are a promising path to zero-shot embodied control through planning. However, existing world model planners struggle on long-horizon, multi-stage tasks: prediction errors compound and naive search is exponential in the planning horizon. Hierarchy mitigates both by decomposing tasks into shorter, tractable subproblems; yet prior hierarchical approaches either amortize control into task-specific policies (hierarchical RL) or assume low-dimensional states and known dynamics (classical hierarchical MPC). We present Hierarchical Planning with Latent World Models (HWM), an architecture and planning paradigm for hierarchical model predictive control directly on visual world models trained solely via next-latent prediction. HWM learns world models at multiple temporal scales within a shared latent space, so predictions from the long horizon model serve as subgoals for the short horizon model via latent matching, without task-specific rewards or hierarchical policies. To keep long-horizon search tractable, HWM learns an action encoder that compresses primitive action chunks into latent macro-actions. On real-world Franka manipulation, HWM solves pick-and-place from a single goal image at 70% success vs. 0% for single-level planning. Across simulated push manipulation and maze navigation, HWM consistently improves performance on long-horizon tasks while requiring up to 3x less planning compute.
Wancong Zhang, Basile Terver, Artem Zholus +7
FAIR at Meta · New York University · Mila - Québec AI Institute +1
World models trained via latent dynamics offer a scalable path for embodied planning, yet existing methods suffer from representational collapse, computational overhead, and limited long-horizon capability. We introduce MemWM, a memory-enhanced latent world model that augments JEPA-style prediction with a content-addressable memory bank. MemWM learns high-level representations directly from observations and uses them to condition predictions via adaptive layer normalization, complementing low-level action-conditioned dynamics within a shared latent space. To prevent memory slots from collapsing into identical representations, we propose two complementary losses that encourage structural diversity across slots and balanced utilization across time steps. On PushT, TwoRoom, Reacher, and OGBCube, MemWM attains an average success rate of 89.9%, matching DINO-WM (89.3%) while using 4× fewer trainable parameters and no pretrained encoder, and improving over LeWM by 4.1%.
Yuwei Sun, Siyu Zhu
Shanghai Academy of AI for Science · Fudan University
Recently, world models have emerged as a promising paradigm for building intelligent agents by learning predictive models that estimate future environment states conditioned on observations and actions. In particular, JEPA-style latent world models provide an efficient alternative to pixel space prediction by learning action-conditioned dynamics in compact representation spaces. However, existing latent world models typically rely on one-step prediction and must be recursively rolled out for long-horizon planning, which leads to compounding errors and a mismatch between training objectives and downstream planning tasks. To address this limitation, we propose Variable-length Latent World Models (VLWMs), a framework that learns to predict future latent states conditioned on action sequences of variable lengths. Instead of training only on one-step transitions, VLWMs directly model temporally extended dynamics, allowing the same predictor to evaluate action plans over different horizons. We further introduce a curriculum training strategy that progressively expands the action horizon, stabilizing optimization from short-range dynamics to long-range prediction. At test time, we design planning methods tailored to VLWMs to better exploit their variable-length predictive capabilities. Experiments on long-horizon control tasks show that VLWMs significantly improve latent space world models, achieving 13% average improvement over the state-of-the-art LeWM across different datasets, with especially large gains on tasks requiring extended planning. These results suggest that VLWM provides a simple yet effective paradigm for improving long-horizon prediction and planning in latent world models.
Tianqi Du, Qi Zhang, Yifei Wang +1
State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Amazon AGI SF Lab · Institute for Artificial Intelligence, Peking University