Hierarchical Joint-Embedding Predictive Architectures (H-JEPA) enable long-horizon visual control but rely on costly test-time gradient descent for planning. Our analysis reveals that iterative optimization consumes up to 97.7% of episode runtime and that low predicted rollout costs do not necessarily translate into successful control. We further identify a planning-horizon asymmetry between long-horizon top-level composition and short two-step subordinate planning. To address these challenges, we propose WARP-JEPA, a hierarchical amortized planning framework that replaces online gradient optimization with learned multi-candidate trajectory proposers while keeping the world models frozen. WARP-JEPA combines a lightweight world-model-in-the-loop causal Transformer at the top level with parallel residual MLPs at subordinate levels. The proposers are trained using model-based rollout costs and best-of-K behavior cloning, enabling efficient candidate selection without test-time gradient optimization. Across 30 frozen H-JEPA models on four benchmarks, WARP-JEPA achieves 9.6-18.8x faster episode inference on long-horizon tasks (2.9-3.0x on Cube) and improves task success by up to 33.3 percentage points. These results demonstrate that amortized planning can substantially improve both inference efficiency and control performance without retraining the world models. Code is available at https://github.com/hideinsewer/WARP-JEPA.
Figures & tables
Figure 1: Online gradient descent vs . WARP-JEPA on all four benchmarks (in parentheses: success gain and speed-up of TM over Online GD, L2, Tab. 1 ). Top: multi-modal candidate selection escapes local minima on Push-T and FourRoom. Bottom: real closed-loop rollouts of MM on Visual AntMaze and OGBench Cube (full rollout strips in App. Fig. S1).
Figure 2: Architecture of WARP-JEPA (asymmetric TM shown). Blue: frozen H-JEPA modules; orange: learned proposers. (a) A standard 1-layer causal Transformer decoder TϕL [ 33 ] cross-attends to the context tokens M ( Eq. 4 ); each query joins the previous macro-action with the state imagined by the frozen PL , which also scores the KL plans with zero extra rollouts. (b) Proposers are trained with relaxed WTA rollout cost plus best-of- K behavior cloning ( Eq. 3 ) through frozen Pl . (c) A residual MLP proposes Kl two-step bridges in parallel, verified by one batched rollout of Pl .
Table 1: Main results on 30 frozen H-JEPA models. Success rate (%, mean ± s.e. over three seeds; 250 tasks per seed on Push-T, 50 elsewhere) and time per episode (s; seed 42, 10 tasks) of Online GD and the three WARP-JEPA instantiations (MM, TT, TM: e.g. MMM, TTT, TMM at L3). Bold: best success per column. Speed-up: range over the three instantiations. Per-seed results, paired tests and all timings: App. Tab. S2, S4 and S7.
Level-1 proposer
Succ. (%)
ΔGD (pp)
Ep. (s)
Top: Online GD
Online GD
40.0
ref.
25.3
MLP (top: 10 GD it.)
38.8
−1.2
5.55
Top: MLP
10.1M, 30.5 ms (30.0)
Online GD
49.6 †
+9.6
6.64
MLP (MM)
56.4 ‡
+16.4
1.43
Table 2: Architecture ablations on Push-T L2 (seed 42, 250 paired tasks; H2=8 , H1=2 ). Group rows: top-level proposer, its parameters and the time of one top-level call ( K2=32 ; scoring part in parentheses; breakdown in App. Tab. S11). †p<0.05 , ‡p<0.01 vs . Online GD (McNemar). All 14 variants: App. Tab. S6.
Figure 3: (a) Speed and success of Online GD (hollow) and WARP-JEPA-TM (filled) on all ten settings of Tab. 1 (mean over three seeds; time: seed 42). (b) Best-of- K selection over the MM top-level proposals on Push-T L2 (150 tasks; Tab. 3 ): success grows with K , matches Online GD at K=8 and exceeds averaging the proposals ( MeanHead ) and random shooting.
Offline
Closed loop
Top-level solver
C2
MSE
Succ. (%)
Time (s)
Expert actions
215.0
0.00
–
–
Online GD, 30–90 it.
16.9
4.23
42.0
44.3
Random shooting, K=32
275.0
6.74
0.0 ‡
2.33
MLP proposals ( λbc=0.05 )
Mean of 32 (MeanHead)
314.8
0.97
28.0 ‡
2.21
Table 3: Mechanism analysis on Push-T L2 (released checkpoint; level 1: MLP, except for Online GD). Offline: median terminal cost C2 and first-action MSE on 192 validation segments. Closed loop: 150 tasks, RTX 4090; ‡p<0.01 vs . MM. ∗ Reversed on 200 new tasks ( 49.5% vs . 53.5% ). All 18 variants: App. Tab. S12.
Solver time per level
Depth
Planner
Top
Middle
L1
Rest
Total
L2
Online GD
47.6
–
28.3
2.9
78.9
TM
1.38
–
1.39
2.92
5.69
L3
Online GD
32.8
19.9
28.4
3.1
84.3
TMM
0.81
1.06
1.40
2.96
6.23
L4
Online GD
19.3
39.7
30.4
3.0
92.5
Table 4: AntMaze time per episode (s) by level (seed 42, 10 tasks). Middle: levels 2 to L−1 ; Rest: encoding, simulation, overhead. All settings: App. Tab. S8.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Depth
Horizons H1,…,HL
GD candidates per level
Tasks / seed
Tasks
Push-T
L2
2, 8
50, 400
250
750
FourRoom
L2
2, 12
100, 800
50
150
AntMaze
L2
4, 9
25, 200
50
150
Cube
L2
2, 2
400, 100
50
150
Push-T
L3
2, 2, 4
25, 50, 200
250
750
FourRoom
L3
2, 2, 6
50, 200, 200
50
150
Appendix
Table S1 : Hierarchies and Online GD budgets (official configurations; horizons and candidates from level 1 up). Online GD: AdamW, 30–90 iterations per solve (early stop, patience 5), replanning every 10 steps.
Figure S1 : Qualitative closed-loop visual rollouts across all four benchmarks (L2, seed 42, WARP-JEPA = MM; step-aligned frames from t=0 to Tsucc alongside target observation og ). (a) Push-T (Task #0): Online GD stalls with the T-block misaligned, and MeanHead (averaging the K=32 proposal heads without world-model selection) averages incompatible left/right modes and pushes the block away from the target, whereas WARP-JEPA selects a feasible mode and aligns the T-block with og at t=80 . (b) FourRoom (Task #9): Online GD gets trapped against the first interior wall, while WARP-JEPA routes through both doorway subgoals to reach the bottom-right room ( t=129 ). (c) Visual AntMaze (Task #3): Online GD stalls in the initial corridor, whereas WARP-JEPA rounds the corner into the purple goal corridor ( t=128 ). (d) OGBench Cube (Task #1): Online GD misses the target placement, whereas WARP-JEPA grasps and places the red cube onto the target ( t=14 ).
Setting
Planner
Seed 42
Seed 43
Seed 44
Mean
Pooled
Push-T L2
Online GD
40.0
44.8
37.6
40.8 ±2.1
306/750
L1-only (GD 10 / MLP)
38.8
37.2
37.2
37.7 ±0.5
283/750
MM
56.4
52.4
58.8
55.9 ±1.9
419/750
TT
65.6
64.0
60.0
63.2 ±1.7
474/750
TM
66.4
66.8
60.8
64.7 ±1.9
485/750
FourRoom L2
Online GD
90.0
84.0
76.0
83.3 ±4.1
125/150
Appendix
Table S2 : Success rate (%) per model seed. Tasks per seed: Push-T 250 (50 official + 200 held-out), others 50. Mean ± s.e. over seeds; pooled successes. MM, TT, TM: the three WARP-JEPA instantiations (MMM/TTT/TMM at L3, MMMM/TTTT/TMMM at L4).
Planner
Official (%)
Succ.
Held-out (%)
Succ.
Online GD
40.7 ±5.2
61/150
40.8 ±1.4
245/600
L1-only (GD 10 / MLP)
35.3 ±1.8
53/150
38.3 ±0.4
230/600
MM
54.0 ±2.0
81/150
56.3 ±1.9
338/600
TT
66.0 ±3.1
99/150
62.5 ±1.4
375/600
TM
64.0 ±3.1
96/150
64.8 ±1.7
389/600
Appendix
Table S3 : Push-T L2: official vs . held-out tasks (3 seeds). Held-out tasks: official procedure on the validation episodes, task seeds 1003–1006.
Setting
MM vs. Online GD
TT vs. Online GD
TM vs. Online GD
TM vs. MM
TT vs. TM
Push-T L2
+15.1 ( 1.8×10−11 )
+22.4 ( 1.4×10−22 )
+23.9 ( 9.6×10−24 )
+8.8 ( 1.7×10−5 )
-1.5 (0.46)
FourRoom L2
+8.7 (0.041)
+16.7 ( 6.0×10−8 )
+16.7 ( 6.0×10−8 )
+8.0 ( 4.9×10−4 )
+0.0 (1.00)
AntMaze L2
+33.3 ( 2.9×10−9 )
+32.7 ( 3.2×10−10 )
+33.3 ( 1.8×10−10 )
+0.0 (1.00)
-0.7 (1.00)
Cube L2
+32.7 ( 6.8×10−10 )
+27.3 ( 1.3×10−8 )
+26.0 ( 1.8×10−6 )
-6.7 (0.14)
+1.3 (0.83)
Push-T L3
+1.2 (0.52)
+3.1 (0.086)
+2.7 (0.13)
+1.5 (0.42)
+0.4 (0.88)
FourRoom L3
+7.3 (0.0034)
+8.0 ( 4.9×10−4 )
+8.0 ( 4.9×10−4 )
+0.7 (1.00)
+0.0 (1.00)
Appendix
Table S4 : Paired differences for Table 1 (pp; exact two-sided McNemar p over matched (seed, task) pairs in parentheses).
Benchmark
L1-only vs. GD
MM vs. GD
MM vs. L1-only
TM vs. MM
TT vs. TM
L1-only speed-up
Push-T
-3.1 (0.17)
+15.1 ( 1.8×10−11 )
+18.1 ( 1.8×10−17 )
+8.8 ( 1.7×10−5 )
-1.5 (0.46)
4.6×
FourRoom
-7.3 (0.061)
+8.7 (0.041)
+16.0 ( 1.2×10−4 )
+8.0 ( 4.9×10−4 )
+0.0 (1.00)
4.1×
AntMaze
+27.3 ( 9.6×10−8 )
+33.3 ( 2.9×10−9 )
+6.0 (0.26)
+0.0 (1.00)
-0.7 (1.00)
4.2×
Cube
+2.0 (0.79)
+32.7 ( 6.8×10−10 )
+30.7 ( 6.9×10−9 )
-6.7 (0.14)
+1.3 (0.83)
2.5×
Appendix
Table S5 : Per-level amortization on L2 (3 seeds; Push-T 750 tasks, others 150). L1-only: MLP at level 1, 10 warm-started GD iterations at the top level. Differences in pp (McNemar p ); last column: L1-only speed-up over Online GD.
Top
L1
Official
Held-out
Succ. (%)
vs. MM
vs. Online GD
Time (s)
GD
GD
20/50
80/200
40.0
-16.4 ( 5.5×10−5 )
–
25.27
GD 10
MLP
19/50
78/200
38.8
-17.6 ( 4.9×10−6 )
-1.2 (0.84)
5.55
MLP
GD
24/50
100/200
49.6
-6.8 (0.040)
+9.6 (0.020)
6.64
MLP
GD (bypass 10)
23/50
97/200
48.0
-8.4 (0.014)
+8.0 (0.062)
5.48
MLP
MLP
28/50
113/200
56.4
–
+16.4 ( 5.5×10−5 )
1.43
tf_parallel (4L)
MLP
31/50
130/200
64.4
+8.0 (0.022)
+24.4 ( 2.7×10−10 )
1.34
Appendix
Table S6 : All Push-T ablation variants (L2, seed 42, 250 paired tasks). Differences in pp (McNemar p ). Time: s/episode, first 10 official tasks, from three studies (main: Online GD, L1-only, MM; mechanism: 4-layer; multi-benchmark: 1-layer); repeated runs differ by up to about 10%.
Setting
Online GD
L1-only
MM
TT
TM
TM speed-up
GD share
Push-T L2
25.27
5.55
1.43
1.37
1.34
18.8×
97.7
FourRoom L2
45.08
10.89
2.76
2.93
2.93
15.4×
97.2
AntMaze L2
78.87
18.75
5.23
5.70
5.69
13.9×
96.9
Cube L2
3.44
1.37
1.18
1.16
1.19
2.9×
79.9
Push-T L3
20.34
–
1.49
1.49
1.48
13.7×
97.7
FourRoom L3
36.61
–
3.02
3.19
3.10
11.8×
96.6
Appendix
Table S7 : Time per episode (s) (seed 42, first 10 official tasks, sequential runs on one H200, including encoding and simulation). GD share: fraction of Online GD time spent in the planner (%).
Setting
Planner
Level 4
Level 3
Level 2
Level 1
Rest
Total
Push-T L2
Online GD
–
–
17.22
7.31
0.74
25.27
TM
–
–
0.31
0.45
0.59
1.34
FourRoom L2
Online GD
–
–
32.57
10.95
1.55
45.08
TM
–
–
0.83
0.64
1.46
2.93
AntMaze L2
Online GD
–
–
47.64
28.33
2.91
78.87
TM
–
–
1.38
1.39
2.92
5.69
Appendix
Table S8 : Solver time per level (s/episode). Rest: encoding, simulation and code outside the per-level solvers.
Setting
Level 4
Level 3
Level 2
Level 1
Online GD iterations
Push-T L2
–
–
0 (0/70)
30 (30/100)
39.3, 36.2
FourRoom L2
–
–
0 (0/190)
14 (30/220)
36.8, 32.2
AntMaze L2
–
–
0 (0/310)
14 (50/360)
33.8, 49.5
Cube L2
–
–
0 (0/10)
75 (30/40)
32.1, 38.6
Push-T L3
–
0 (0/60)
14 (10/70)
30 (30/100)
40.8, 35.7, 46.5
FourRoom L3
–
0 (0/170)
11 (20/190)
14 (30/220)
33.7, 31.8, 31.2
Appendix
Table S9 : Solves that still run gradient descent. Percentage (count) of solves per level that are H-JEPA’s end-of-episode bypass solves (no parent macro-action), where all WARP-JEPA instantiations run 10 GD iterations; all other solves use the proposers. Counts are identical for MM, TT and TM. Last column: mean Online GD iterations per solve, top level first.
Benchmark
Level
Architecture
Layers
Params (M)
Models
Training (h)
Push-T
top
MLP
–
9.26–10.06
9
0.8–3.0
top
tf_parallel
4
10.14
1
2.1
top
tf_ar_action
4
10.14
1
4.1
top
tf_ar_wm
1
3.43–3.44
9
1.0–3.6
top
tf_ar_wm
4
10.14
1
4.2
subordinate
MLP
–
9.33–9.53
18
0.3–2.4
Appendix
Table S10 : Proposer parameter counts and training times across all benchmarks, depths, and architectures : 30k AdamW steps, batch 256, learning rate 3×10−4 , 1k warm-up steps, cosine decay; top level K=32 , λbc=0.05 ; subordinate levels K=16 , λbc=1 . Training: wall-clock time per run on one H200 (range over models).
Top proposer
Params (M)
Prop.
In-loop rollout
Scoring
Total
MLP
10.06
0.57
0.00
29.96
30.53
tf_parallel (4L)
10.14
4.95
0.00
30.06
35.24
tf_ar_action (4L)
10.14
25.89
0.00
29.79
56.30
tf_ar_wm (4L)
10.14
26.38
30.73
0.14
57.89
tf_ar_wm (1L, TM)
3.44
10.39
30.95
0.13
41.47
Appendix
Table S11 : Latency of one top-level call (ms, mean; Push-T, seed 42, H2=8 , 32 candidates, 4-layer Transformers). Proposer: forward pass; in-loop rollout: predictor calls inside the autoregressive loop; scoring: rollout and cost for selection ( tf_ar_wm reuses its in-loop rollout).
Offline
Closed loop
Top-level solver / variant
Med. C2
Mean C2
MSE 1st
MSE all
Expl. 150
vs. MM (pp, p )
Conf. 200
Pooled 350
Time (s)
Reference and search baselines
Expert (ground truth)
215.0
216.1
0.00
0.00
–
–
–
–
–
Prior mean (zero action)
372.7
391.4
3.96
3.79
–
–
–
–
–
Random shoot, K=32
275.0
270.7
6.74
7.22
0.0
−48.0 ( 4.2×10−22 )
–
–
2.33
Random shoot, K=400
218.0
206.5
6.53
7.76
–
–
–
–
–
Appendix
Table S12 : Complete top-level mechanism study on Push-T L2 (released H-JEPA checkpoint, seed 42; level 1 is the learned MLP except for Online GD). Offline : terminal predictor cost C2 (median and mean) and macro-action MSE (first step and all 8 steps, prior-std units) over 192 validation segments ( H2=8 ). Closed loop (success, %): exploratory 150-task suite (official 50 + held-out seeds 1001–1002) with the difference to MM in pp (exact McNemar p ), confirmatory 200-task suite (held-out seeds 1003–1006), pooled 350 tasks; time per episode (s, RTX 4090).
Long-horizon planning with latent world models requires reasoning across timescales and levels of abstraction. Existing task-agnostic JEPA world models predict and plan in a single latent space, often at a single timescale. We introduce H-JEPA, an end-to-end recipe for training a hierarchy of action-conditioned JEPAs in which each level predicts farther ahead in its own learned latent space. Planning proceeds top-down: the top level optimizes progress toward the goal, and each level's predictions become subgoals for the planner below it. When factors in the data evolve at separated timescales, higher levels discard fast, unpredictable detail and retain slower task-relevant state. Across four simulated navigation and manipulation environments, hierarchical planning improves over a flat JEPA; on Visual AntMaze, a three-level hierarchy raises success from 18% to 73% using less planner compute. Ablations attribute these gains to both temporal decomposition and higher-level goal representations. With inverse-dynamics supervision, the approach extends to diverse real-robot videos from DROID, where hierarchy improves offline planning fidelity at lower planner compute.
Wancong Zhang, Basile Terver, Michael Rabbat +2
NYU · Advanced Machine Intelligence · INRIA Paris +1
Joint-embedding predictive architectures (JEPAs) learn compact latent world models without pixel reconstruction. Existing JEPA pipelines typically train visual representations and action-conditioned dynamics before adding a control policy or solver. We introduce LeWAM, an end-to-end JEPA for control with: (1) a joint training objective over next latent state prediction and action flow matching, and (2) an isotropic Gaussian constraint on the latent state, enabling a single model to support direct policy execution and efficient planning by evaluating sampled actions with its learned latent dynamics. Compared with prior JEPAs in visual planning, LeWAM improves success rate from 28.6% to 89.7% on contact-rich tasks and from 10.9% to 46.2% on long-horizon tasks, and plans 32.3 times faster. On behavior cloning tasks in simulation and physical robots, LeWAM is also competitive with established methods. Interpretability analyses show that LeWAM learns control-relevant representations and action distributions effectively guiding test-time planning and control.
Minghao Fu, Tavis Siebert, Eryk Halicki +1
UC San Diego · ETH Zurich · University of British Columbia +1
World models are a promising path to zero-shot embodied control through planning. However, existing world model planners struggle on long-horizon, multi-stage tasks: prediction errors compound and naive search is exponential in the planning horizon. Hierarchy mitigates both by decomposing tasks into shorter, tractable subproblems; yet prior hierarchical approaches either amortize control into task-specific policies (hierarchical RL) or assume low-dimensional states and known dynamics (classical hierarchical MPC). We present Hierarchical Planning with Latent World Models (HWM), an architecture and planning paradigm for hierarchical model predictive control directly on visual world models trained solely via next-latent prediction. HWM learns world models at multiple temporal scales within a shared latent space, so predictions from the long horizon model serve as subgoals for the short horizon model via latent matching, without task-specific rewards or hierarchical policies. To keep long-horizon search tractable, HWM learns an action encoder that compresses primitive action chunks into latent macro-actions. On real-world Franka manipulation, HWM solves pick-and-place from a single goal image at 70% success vs. 0% for single-level planning. Across simulated push manipulation and maze navigation, HWM consistently improves performance on long-horizon tasks while requiring up to 3x less planning compute.
Wancong Zhang, Basile Terver, Artem Zholus +7
FAIR at Meta · New York University · Mila - Québec AI Institute +1