JEPA-based world models predict action consequences in latent space, but representative planning approaches rely on candidate-action search, requiring repeated evaluation of action sequences and incurring substantial online computational cost. Direct action generation avoids this search, yet how to leverage latent predictions to guide action generation given task goals remains a key challenge in translating learned physical dynamics into effective decisions. We introduce WA-JEPA, a world-action model that couples action generation, action-conditioned latent prediction, and goal-aware feedback. The model first generates an action proposal and predicts the resulting future scenes and proprioceptive states, then refines the proposal through two complementary feedback paths, using the task goal as a reference. Goal-discrepancy residual feedback (GDRF) uses differences between the latent representations of predicted scenes and the task goal to guide chunk-wise action refinement. Future-scene context feedback (FSCF) preserves fine-grained predicted scene representations to guide subsequent action refinement. Jointly learned from offline demonstrations, this process iteratively updates the same plan before execution, making future predictions directly participate in action generation without candidate-action search. We evaluate short-horizon goal reaching, long-horizon tasks, and full-task execution. The results show that goal-aware predictive feedback supports competitive task performance while maintaining a compact model and low online computational cost. Code is available at https://github.com/Dovisya/WA-JEPA.
Figures & tables
Figure 1: Overview of WA-JEPA. The model iteratively refines an action plan before execution through a proposal–prediction–goal-aware feedback loop. Goal-discrepancy residual feedback guides chunk-wise action corrections, while future-scene context feedback coordinates updates across the plan. Both paths refine the same pending plan; new observations inform subsequent decisions as the process repeats until task completion.
Figure 2: Visual goal-reaching success on TwoRoom, PushT, and Cube. WA-JEPA achieves 100%, 96%, and 100%, respectively.
Figure 3: Short-horizon computational cost on an NVIDIA A40. (a) Mean complete-plan latency (ms) and (b) neural arithmetic (GFLOPs). Both panels use logarithmic axes. (c) Total PushT model parameters (millions) on a linear axis, split by task-training status.
PushT
TwoRoom
Cube
Mean
Method
Std.
Pert.
Time ↓
Std.
Pert.
Time ↓
Std.
Pert.
Time ↓
Std.
Pert.
Time ↓
LeWM (final goal)
2.34
2.73
1.77×
1.56
1.56
2.07×
7.03
2.73
1.29×
3.65
2.34
1.71×
WA-JEPA (final goal)
4.69
3.52
0.33×
89.84
89.84
0.47×
35.16
19.14
0.70×
43.23
37.50
0.50×
Direct replay
51.56
17.97
0.66×
99.22
100.00
0.53×
82.03
47.66
0.87×
77.60
55.21
0.69×
AP (LeWM)
69.53
64.84
1.99×
46.88
46.88
2.18×
46.88
29.69
1.33×
54.43
47.14
1.83×
AP (WA-JEPA)
56.25
46.48
1.00×
100.00
100.00
1.00×
84.38
52.34
1.00×
80.21
66.28
1.00×
Table 1: Long-range success (%) and relative decision time under AP. Timing pools both start conditions and reports decision-time ratios relative to AP (WA-JEPA). Bold marks the best success among the listed variants.
Method
Head
Push-T
LIBERO-Goal
Params. (M) ↓
Mem. (MiB) ↓
FLOPs (G) ↓
Latency (ms) ↓
WebSSL (CLS) [ 25 ]
VQ-BeT
59±1
95±1
332.7 / 308.0
665.0 / 647.3
162.5 / 324.6
19.20 / 17.49
Patch Policy [ 25 ]
VQ-BeT
68±3
94±1
334.0 / 308.7
894.5 / 2552.6
255.2 / 413.5
26.49 / 81.35
ResNet-18 Patch [ 25 ]
ACT
64±3
93±2
66.5 / 66.7
144.0 / 152.6
6.1 / 11.9
11.87 / 12.01
OpenVLA-OFT [ 10 ]
MLP
59±2
95
7608.4 / 7709.2
14837.7 / 15331.6
4202.6 / 8766.6
91.13 / 147.16
Fast-WAM [ 21 ]
Flow matching
—
96
12406.3
23959.7
2545.1
438.53
RynnVLA-002 [ 4 ]
Regression
—
96
7150.3
15429.3
16297.3
436.55
Table 2: Full-task performance and computational cost. Scores are percentages (mean ± available SD). Cost pairs denote Push-T / LIBERO-Goal ; unpaired costs refer only to LIBERO-Goal . Memory is peak allocated GPU memory. Bold marks the best values within each task.
Figure 4: Predictive-feedback ablations on PushT. (a) Zero to three feedback updates. (b) Variants without GDRF or FSCF. Curves show 20 unsmoothed checkpoints from one training seed; enlarged markers and Best labels indicate peak success.
Figure 5: Prediction–execution correspondence on PushT. (a) Predicted and executed scene deviations from the outcomes of demonstrated actions, measured by feature RMSE across chunk boundaries for proposals from stages 1–4. (b) Reduction in terminal goal-feature MSE relative to stage 1, comparing predicted and executed improvements for stages 2–4; positive values indicate improvement. Each point represents one stage proposal in one case.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Updates
PushT
TwoRoom
Cube
0
16 / 45.37
100 / 13.99
100 / 2.42
1
52 / 32.53
100 / 14.14
96 / 2.47
2
68 / 24.42
100 / 14.13
100 / 2.44
3
96 / 18.30
100 / 14.08
100 / 2.41
Appendix
Table S1 : Short-horizon task performance of a fixed model with different numbers of feedback updates.
Task
Start
Feedback updates
0
1
2
3
PushT
Std.
4.69
17.97
32.81
56.25
Pert.
5.86
16.02
28.52
46.48
TwoRoom
Std.
99.22
100.00
100.00
100.00
Pert.
100.00
100.00
100.00
100.00
Cube
Std.
79.69
80.47
82.03
84.38
Appendix
Table S2 : Fixed-checkpoint AP (WA-JEPA) success (%) after 0–3 feedback updates. Mean averages the three tasks equally.
Task / feedback updates
0
1
2
3
Push-T coverage
31.00
60.30
59.65
70.78
LIBERO-Goal success
87
88
94
97
Appendix
Table S3 : Full-task performance of a fixed model with different numbers of feedback updates. Each setting uses 100 cases per task; all values are percentages.
Feedback input
Best
Final
Last 5
Goal difference (ours)
96
94
94.4
Goal-Concat
92
92
91.6
Appendix
Table S4 : Goal-input ablation on PushT. Success rates are percentages over 50 fixed cases. Best and Final refer to the best of 20 checkpoints and epoch 20; Last 5 averages epochs 16–20 within the same training run.
Task
Method
Mean (ms)
SD (ms)
PushT
WA-JEPA
32.13
0.36
LeWM
1047.36
2.65
DINO-WM + CEM
98924.91
34.01
TwoRoom
WA-JEPA
32.62
0.42
LeWM
351.06
1.54
DINO-WM + CEM
32962.39
22.51
Appendix
Table S5 : Complete-plan latency underlying Fig. 3 (a). Mean and standard deviation (SD) are in milliseconds; WA-JEPA and LeWM use 20 timed calls over ten input pairs; DINO-WM retains ten recorded calls. All use three warmups.
Method / candidates / iterations
PushT
TwoRoom
Cube
Success (%)
Latency (ms)
Success (%)
Latency (ms)
Success (%)
Latency (ms)
LeWM / 30 / 3
50
73.35
76
74.85
62
72.46
LeWM / 100 / 3
58
73.32
82
76.93
72
79.44
LeWM / 300 / 1
14
36.84
68
36.21
42
37.23
LeWM / 300 / 3
58
105.12
84
106.83
68
105.91
LeWM / 300 / 10
94
345.24
90
351.06
72
352.28
Appendix
Table S6 : Locally reevaluated success and latency under different LeWM CEM search budgets, compared with WA-JEPA. Latency measures one complete plan on an A40. Dashes denote untested settings.
Method
Push-T
LIBERO-Goal
WebSSL (CLS, VQ-BeT)
5/5
10/1
Patch Policy (VQ-BeT)
5/5
10/1
ResNet-18 Patch (ACT)
1/5
1/5
OpenVLA-OFT
1/5
1/8
Fast-WAM
—
1/32
RynnVLA-002
—
2/5
Appendix
Table S7 : History frames / output actions per cost-measurement call. Each task uses its own deployment configuration; costs are not normalized by output length.
Representation
Shape
Current scene Z0 ; goal G,G
B×8×192
Context memory M
B×9L×192
Action tokens Q(r)
B×5×192
Action proposal A(r)
B×5×(5da)
Future scene Z(r)
B×5×8×192
Future state P(r)
B×5×dp
Appendix
Table S8 : Core tensors for the continuous-head goal-reaching model. B : batch size; L : history frames; H : action chunks; b : controls per chunk. The full-task VQ-BeT configuration is specified separately in App. C.3.3 .
Setting
Protocol
Short horizon
25 / up to 25 actions; 50 initial–goal pairs per task. Goal: recorded image at +25 steps. Metric: physical goal success.
5 / 1 actions; 100 rollouts per run. Goal: task goal. Metric: final coverage.
Full-task LIBERO
5 / 1 actions; 10×10 rollouts per run. Goal: end of a training demonstration. Metric: task success on LIBERO-Goal.
Appendix
Table S9 : WA-JEPA execution and evaluation protocols. Predicted / executed lengths count environment actions. Four internal stages refine the same pending plan without new observations.
Department of Computing, The Hong Kong Polytechnic University · Department of Land Surveying and Geospatial Science, The Hong Kong Polytechnic University