Hierarchical Joint-Embedding Predictive Architectures (H-JEPA) enable long-horizon visual control but rely on costly test-time gradient descent for planning. Our analysis reveals that iterative optimization consumes up to 97.7% of episode runtime and that low predicted rollout costs do not necessarily translate into successful control. We further identify a planning-horizon asymmetry between long-horizon top-level composition and short two-step subordinate planning. To address these challenges, we propose WARP-JEPA, a hierarchical amortized planning framework that replaces online gradient optimization with learned multi-candidate trajectory proposers while keeping the world models frozen. WARP-JEPA combines a lightweight world-model-in-the-loop causal Transformer at the top level with parallel residual MLPs at subordinate levels. The proposers are trained using model-based rollout costs and best-of-K behavior cloning, enabling efficient candidate selection without test-time gradient optimization. Across 30 frozen H-JEPA models on four benchmarks, WARP-JEPA achieves 9.6-18.8x faster episode inference on long-horizon tasks (2.9-3.0x on Cube) and improves task success by up to 33.3 percentage points. These results demonstrate that amortized planning can substantially improve both inference efficiency and control performance without retraining the world models. Code is available at https://github.com/hideinsewer/WARP-JEPA.
Figures & tables
Figure 1: Online gradient descent vs . WARP-JEPA on all four benchmarks (in parentheses: success gain and speed-up of TM over Online GD, L2, Tab. 1 ). Top: multi-modal candidate selection escapes local minima on Push-T and FourRoom. Bottom: real closed-loop rollouts of MM on Visual AntMaze and OGBench Cube (full rollout strips in App. Fig. S1).
Figure 2: Architecture of WARP-JEPA (asymmetric TM shown). Blue: frozen H-JEPA modules; orange: learned proposers. (a) A standard 1-layer causal Transformer decoder TϕL [ 33 ] cross-attends to the context tokens M ( Eq. 4 ); each query joins the previous macro-action with the state imagined by the frozen PL , which also scores the KL plans with zero extra rollouts. (b) Proposers are trained with relaxed WTA rollout cost plus best-of- K behavior cloning ( Eq. 3 ) through frozen Pl . (c) A residual MLP proposes Kl two-step bridges in parallel, verified by one batched rollout of Pl .
Table 1: Main results on 30 frozen H-JEPA models. Success rate (%, mean ± s.e. over three seeds; 250 tasks per seed on Push-T, 50 elsewhere) and time per episode (s; seed 42, 10 tasks) of Online GD and the three WARP-JEPA instantiations (MM, TT, TM: e.g. MMM, TTT, TMM at L3). Bold: best success per column. Speed-up: range over the three instantiations. Per-seed results, paired tests and all timings: App. Tab. S2, S4 and S7.
Level-1 proposer
Succ. (%)
ΔGD (pp)
Ep. (s)
Top: Online GD
Online GD
40.0
ref.
25.3
MLP (top: 10 GD it.)
38.8
−1.2
5.55
Top: MLP
10.1M, 30.5 ms (30.0)
Online GD
49.6 †
+9.6
6.64
MLP (MM)
56.4 ‡
+16.4
1.43
Table 2: Architecture ablations on Push-T L2 (seed 42, 250 paired tasks; H2=8 , H1=2 ). Group rows: top-level proposer, its parameters and the time of one top-level call ( K2=32 ; scoring part in parentheses; breakdown in App. Tab. S11). †p<0.05 , ‡p<0.01 vs . Online GD (McNemar). All 14 variants: App. Tab. S6.
Figure 3: (a) Speed and success of Online GD (hollow) and WARP-JEPA-TM (filled) on all ten settings of Tab. 1 (mean over three seeds; time: seed 42). (b) Best-of- K selection over the MM top-level proposals on Push-T L2 (150 tasks; Tab. 3 ): success grows with K , matches Online GD at K=8 and exceeds averaging the proposals ( MeanHead ) and random shooting.
Offline
Closed loop
Top-level solver
C2
MSE
Succ. (%)
Time (s)
Expert actions
215.0
0.00
–
–
Online GD, 30–90 it.
16.9
4.23
42.0
44.3
Random shooting, K=32
275.0
6.74
0.0 ‡
2.33
MLP proposals ( λbc=0.05 )
Mean of 32 (MeanHead)
314.8
0.97
28.0 ‡
2.21
Table 3: Mechanism analysis on Push-T L2 (released checkpoint; level 1: MLP, except for Online GD). Offline: median terminal cost C2 and first-action MSE on 192 validation segments. Closed loop: 150 tasks, RTX 4090; ‡p<0.01 vs . MM. ∗ Reversed on 200 new tasks ( 49.5% vs . 53.5% ). All 18 variants: App. Tab. S12.
Solver time per level
Depth
Planner
Top
Middle
L1
Rest
Total
L2
Online GD
47.6
–
28.3
2.9
78.9
TM
1.38
–
1.39
2.92
5.69
L3
Online GD
32.8
19.9
28.4
3.1
84.3
TMM
0.81
1.06
1.40
2.96
6.23
L4
Online GD
19.3
39.7
30.4
3.0
92.5
Table 4: AntMaze time per episode (s) by level (seed 42, 10 tasks). Middle: levels 2 to L−1 ; Rest: encoding, simulation, overhead. All settings: App. Tab. S8.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Depth
Horizons H1,…,HL
GD candidates per level
Tasks / seed
Tasks
Push-T
L2
2, 8
50, 400
250
750
FourRoom
L2
2, 12
100, 800
50
150
AntMaze
L2
4, 9
25, 200
50
150
Cube
L2
2, 2
400, 100
50
150
Push-T
L3
2, 2, 4
25, 50, 200
250
750
FourRoom
L3
2, 2, 6
50, 200, 200
50
150
Appendix
Table S1 : Hierarchies and Online GD budgets (official configurations; horizons and candidates from level 1 up). Online GD: AdamW, 30–90 iterations per solve (early stop, patience 5), replanning every 10 steps.
Figure S1 : Qualitative closed-loop visual rollouts across all four benchmarks (L2, seed 42, WARP-JEPA = MM; step-aligned frames from t=0 to Tsucc alongside target observation og ). (a) Push-T (Task #0): Online GD stalls with the T-block misaligned, and MeanHead (averaging the K=32 proposal heads without world-model selection) averages incompatible left/right modes and pushes the block away from the target, whereas WARP-JEPA selects a feasible mode and aligns the T-block with og at t=80 . (b) FourRoom (Task #9): Online GD gets trapped against the first interior wall, while WARP-JEPA routes through both doorway subgoals to reach the bottom-right room ( t=129 ). (c) Visual AntMaze (Task #3): Online GD stalls in the initial corridor, whereas WARP-JEPA rounds the corner into the purple goal corridor ( t=128 ). (d) OGBench Cube (Task #1): Online GD misses the target placement, whereas WARP-JEPA grasps and places the red cube onto the target ( t=14 ).
Setting
Planner
Seed 42
Seed 43
Seed 44
Mean
Pooled
Push-T L2
Online GD
40.0
44.8
37.6
40.8 ±2.1
306/750
L1-only (GD 10 / MLP)
38.8
37.2
37.2
37.7 ±0.5
283/750
MM
56.4
52.4
58.8
55.9 ±1.9
419/750
TT
65.6
64.0
60.0
63.2 ±1.7
474/750
TM
66.4
66.8
60.8
64.7 ±1.9
485/750
FourRoom L2
Online GD
90.0
84.0
76.0
83.3 ±4.1
125/150
Appendix
Table S2 : Success rate (%) per model seed. Tasks per seed: Push-T 250 (50 official + 200 held-out), others 50. Mean ± s.e. over seeds; pooled successes. MM, TT, TM: the three WARP-JEPA instantiations (MMM/TTT/TMM at L3, MMMM/TTTT/TMMM at L4).
Planner
Official (%)
Succ.
Held-out (%)
Succ.
Online GD
40.7 ±5.2
61/150
40.8 ±1.4
245/600
L1-only (GD 10 / MLP)
35.3 ±1.8
53/150
38.3 ±0.4
230/600
MM
54.0 ±2.0
81/150
56.3 ±1.9
338/600
TT
66.0 ±3.1
99/150
62.5 ±1.4
375/600
TM
64.0 ±3.1
96/150
64.8 ±1.7
389/600
Appendix
Table S3 : Push-T L2: official vs . held-out tasks (3 seeds). Held-out tasks: official procedure on the validation episodes, task seeds 1003–1006.
Setting
MM vs. Online GD
TT vs. Online GD
TM vs. Online GD
TM vs. MM
TT vs. TM
Push-T L2
+15.1 ( 1.8×10−11 )
+22.4 ( 1.4×10−22 )
+23.9 ( 9.6×10−24 )
+8.8 ( 1.7×10−5 )
-1.5 (0.46)
FourRoom L2
+8.7 (0.041)
+16.7 ( 6.0×10−8 )
+16.7 ( 6.0×10−8 )
+8.0 ( 4.9×10−4 )
+0.0 (1.00)
AntMaze L2
+33.3 ( 2.9×10−9 )
+32.7 ( 3.2×10−10 )
+33.3 ( 1.8×10−10 )
+0.0 (1.00)
-0.7 (1.00)
Cube L2
+32.7 ( 6.8×10−10 )
+27.3 ( 1.3×10−8 )
+26.0 ( 1.8×10−6 )
-6.7 (0.14)
+1.3 (0.83)
Push-T L3
+1.2 (0.52)
+3.1 (0.086)
+2.7 (0.13)
+1.5 (0.42)
+0.4 (0.88)
FourRoom L3
+7.3 (0.0034)
+8.0 ( 4.9×10−4 )
+8.0 ( 4.9×10−4 )
+0.7 (1.00)
+0.0 (1.00)
Appendix
Table S4 : Paired differences for Table 1 (pp; exact two-sided McNemar p over matched (seed, task) pairs in parentheses).
Benchmark
L1-only vs. GD
MM vs. GD
MM vs. L1-only
TM vs. MM
TT vs. TM
L1-only speed-up
Push-T
-3.1 (0.17)
+15.1 ( 1.8×10−11 )
+18.1 ( 1.8×10−17 )
+8.8 ( 1.7×10−5 )
-1.5 (0.46)
4.6×
FourRoom
-7.3 (0.061)
+8.7 (0.041)
+16.0 ( 1.2×10−4 )
+8.0 ( 4.9×10−4 )
+0.0 (1.00)
4.1×
AntMaze
+27.3 ( 9.6×10−8 )
+33.3 ( 2.9×10−9 )
+6.0 (0.26)
+0.0 (1.00)
-0.7 (1.00)
4.2×
Cube
+2.0 (0.79)
+32.7 ( 6.8×10−10 )
+30.7 ( 6.9×10−9 )
-6.7 (0.14)
+1.3 (0.83)
2.5×
Appendix
Table S5 : Per-level amortization on L2 (3 seeds; Push-T 750 tasks, others 150). L1-only: MLP at level 1, 10 warm-started GD iterations at the top level. Differences in pp (McNemar p ); last column: L1-only speed-up over Online GD.
Top
L1
Official
Held-out
Succ. (%)
vs. MM
vs. Online GD
Time (s)
GD
GD
20/50
80/200
40.0
-16.4 ( 5.5×10−5 )
–
25.27
GD 10
MLP
19/50
78/200
38.8
-17.6 ( 4.9×10−6 )
-1.2 (0.84)
5.55
MLP
GD
24/50
100/200
49.6
-6.8 (0.040)
+9.6 (0.020)
6.64
MLP
GD (bypass 10)
23/50
97/200
48.0
-8.4 (0.014)
+8.0 (0.062)
5.48
MLP
MLP
28/50
113/200
56.4
–
+16.4 ( 5.5×10−5 )
1.43
tf_parallel (4L)
MLP
31/50
130/200
64.4
+8.0 (0.022)
+24.4 ( 2.7×10−10 )
1.34
Appendix
Table S6 : All Push-T ablation variants (L2, seed 42, 250 paired tasks). Differences in pp (McNemar p ). Time: s/episode, first 10 official tasks, from three studies (main: Online GD, L1-only, MM; mechanism: 4-layer; multi-benchmark: 1-layer); repeated runs differ by up to about 10%.
Setting
Online GD
L1-only
MM
TT
TM
TM speed-up
GD share
Push-T L2
25.27
5.55
1.43
1.37
1.34
18.8×
97.7
FourRoom L2
45.08
10.89
2.76
2.93
2.93
15.4×
97.2
AntMaze L2
78.87
18.75
5.23
5.70
5.69
13.9×
96.9
Cube L2
3.44
1.37
1.18
1.16
1.19
2.9×
79.9
Push-T L3
20.34
–
1.49
1.49
1.48
13.7×
97.7
FourRoom L3
36.61
–
3.02
3.19
3.10
11.8×
96.6
Appendix
Table S7 : Time per episode (s) (seed 42, first 10 official tasks, sequential runs on one H200, including encoding and simulation). GD share: fraction of Online GD time spent in the planner (%).
Setting
Planner
Level 4
Level 3
Level 2
Level 1
Rest
Total
Push-T L2
Online GD
–
–
17.22
7.31
0.74
25.27
TM
–
–
0.31
0.45
0.59
1.34
FourRoom L2
Online GD
–
–
32.57
10.95
1.55
45.08
TM
–
–
0.83
0.64
1.46
2.93
AntMaze L2
Online GD
–
–
47.64
28.33
2.91
78.87
TM
–
–
1.38
1.39
2.92
5.69
Appendix
Table S8 : Solver time per level (s/episode). Rest: encoding, simulation and code outside the per-level solvers.
Setting
Level 4
Level 3
Level 2
Level 1
Online GD iterations
Push-T L2
–
–
0 (0/70)
30 (30/100)
39.3, 36.2
FourRoom L2
–
–
0 (0/190)
14 (30/220)
36.8, 32.2
AntMaze L2
–
–
0 (0/310)
14 (50/360)
33.8, 49.5
Cube L2
–
–
0 (0/10)
75 (30/40)
32.1, 38.6
Push-T L3
–
0 (0/60)
14 (10/70)
30 (30/100)
40.8, 35.7, 46.5
FourRoom L3
–
0 (0/170)
11 (20/190)
14 (30/220)
33.7, 31.8, 31.2
Appendix
Table S9 : Solves that still run gradient descent. Percentage (count) of solves per level that are H-JEPA’s end-of-episode bypass solves (no parent macro-action), where all WARP-JEPA instantiations run 10 GD iterations; all other solves use the proposers. Counts are identical for MM, TT and TM. Last column: mean Online GD iterations per solve, top level first.
Benchmark
Level
Architecture
Layers
Params (M)
Models
Training (h)
Push-T
top
MLP
–
9.26–10.06
9
0.8–3.0
top
tf_parallel
4
10.14
1
2.1
top
tf_ar_action
4
10.14
1
4.1
top
tf_ar_wm
1
3.43–3.44
9
1.0–3.6
top
tf_ar_wm
4
10.14
1
4.2
subordinate
MLP
–
9.33–9.53
18
0.3–2.4
Appendix
Table S10 : Proposer parameter counts and training times across all benchmarks, depths, and architectures : 30k AdamW steps, batch 256, learning rate 3×10−4 , 1k warm-up steps, cosine decay; top level K=32 , λbc=0.05 ; subordinate levels K=16 , λbc=1 . Training: wall-clock time per run on one H200 (range over models).
Top proposer
Params (M)
Prop.
In-loop rollout
Scoring
Total
MLP
10.06
0.57
0.00
29.96
30.53
tf_parallel (4L)
10.14
4.95
0.00
30.06
35.24
tf_ar_action (4L)
10.14
25.89
0.00
29.79
56.30
tf_ar_wm (4L)
10.14
26.38
30.73
0.14
57.89
tf_ar_wm (1L, TM)
3.44
10.39
30.95
0.13
41.47
Appendix
Table S11 : Latency of one top-level call (ms, mean; Push-T, seed 42, H2=8 , 32 candidates, 4-layer Transformers). Proposer: forward pass; in-loop rollout: predictor calls inside the autoregressive loop; scoring: rollout and cost for selection ( tf_ar_wm reuses its in-loop rollout).
Offline
Closed loop
Top-level solver / variant
Med. C2
Mean C2
MSE 1st
MSE all
Expl. 150
vs. MM (pp, p )
Conf. 200
Pooled 350
Time (s)
Reference and search baselines
Expert (ground truth)
215.0
216.1
0.00
0.00
–
–
–
–
–
Prior mean (zero action)
372.7
391.4
3.96
3.79
–
–
–
–
–
Random shoot, K=32
275.0
270.7
6.74
7.22
0.0
−48.0 ( 4.2×10−22 )
–
–
2.33
Random shoot, K=400
218.0
206.5
6.53
7.76
–
–
–
–
–
Appendix
Table S12 : Complete top-level mechanism study on Push-T L2 (released H-JEPA checkpoint, seed 42; level 1 is the learned MLP except for Online GD). Offline : terminal predictor cost C2 (median and mean) and macro-action MSE (first step and all 8 steps, prior-std units) over 192 validation segments ( H2=8 ). Closed loop (success, %): exploratory 150-task suite (official 50 + held-out seeds 1001–1002) with the difference to MM in pp (exact McNemar p ), confirmatory 200-task suite (held-out seeds 1003–1006), pooled 350 tasks; time per episode (s, RTX 4090).