JEPA-based world models predict action consequences in latent space, but representative planning approaches rely on candidate-action search, requiring repeated evaluation of action sequences and incurring substantial online computational cost. Direct action generation avoids this search, yet how to leverage latent predictions to guide action generation given task goals remains a key challenge in translating learned physical dynamics into effective decisions. We introduce WA-JEPA, a world-action model that couples action generation, action-conditioned latent prediction, and goal-aware feedback. The model first generates an action proposal and predicts the resulting future scenes and proprioceptive states, then refines the proposal through two complementary feedback paths, using the task goal as a reference. Goal-discrepancy residual feedback (GDRF) uses differences between the latent representations of predicted scenes and the task goal to guide chunk-wise action refinement. Future-scene context feedback (FSCF) preserves fine-grained predicted scene representations to guide subsequent action refinement. Jointly learned from offline demonstrations, this process iteratively updates the same plan before execution, making future predictions directly participate in action generation without candidate-action search. We evaluate short-horizon goal reaching, long-horizon tasks, and full-task execution. The results show that goal-aware predictive feedback supports competitive task performance while maintaining a compact model and low online computational cost. Code is available at https://github.com/Dovisya/WA-JEPA.
Figures & tables
Figure 1: Overview of WA-JEPA. The model iteratively refines an action plan before execution through a proposal–prediction–goal-aware feedback loop. Goal-discrepancy residual feedback guides chunk-wise action corrections, while future-scene context feedback coordinates updates across the plan. Both paths refine the same pending plan; new observations inform subsequent decisions as the process repeats until task completion.
Figure 2: Visual goal-reaching success on TwoRoom, PushT, and Cube. WA-JEPA achieves 100%, 96%, and 100%, respectively.
Figure 3: Short-horizon computational cost on an NVIDIA A40. (a) Mean complete-plan latency (ms) and (b) neural arithmetic (GFLOPs). Both panels use logarithmic axes. (c) Total PushT model parameters (millions) on a linear axis, split by task-training status.
PushT
TwoRoom
Cube
Mean
Method
Std.
Pert.
Time ↓
Std.
Pert.
Time ↓
Std.
Pert.
Time ↓
Std.
Pert.
Time ↓
LeWM (final goal)
2.34
2.73
1.77×
1.56
1.56
2.07×
7.03
2.73
1.29×
3.65
2.34
1.71×
WA-JEPA (final goal)
4.69
3.52
0.33×
89.84
89.84
0.47×
35.16
19.14
0.70×
43.23
37.50
0.50×
Direct replay
51.56
17.97
0.66×
99.22
100.00
0.53×
82.03
47.66
0.87×
77.60
55.21
0.69×
AP (LeWM)
69.53
64.84
1.99×
46.88
46.88
2.18×
46.88
29.69
1.33×
54.43
47.14
1.83×
AP (WA-JEPA)
56.25
46.48
1.00×
100.00
100.00
1.00×
84.38
52.34
1.00×
80.21
66.28
1.00×
Table 1: Long-range success (%) and relative decision time under AP. Timing pools both start conditions and reports decision-time ratios relative to AP (WA-JEPA). Bold marks the best success among the listed variants.
Method
Head
Push-T
LIBERO-Goal
Params. (M) ↓
Mem. (MiB) ↓
FLOPs (G) ↓
Latency (ms) ↓
WebSSL (CLS) [ 25 ]
VQ-BeT
59±1
95±1
332.7 / 308.0
665.0 / 647.3
162.5 / 324.6
19.20 / 17.49
Patch Policy [ 25 ]
VQ-BeT
68±3
94±1
334.0 / 308.7
894.5 / 2552.6
255.2 / 413.5
26.49 / 81.35
ResNet-18 Patch [ 25 ]
ACT
64±3
93±2
66.5 / 66.7
144.0 / 152.6
6.1 / 11.9
11.87 / 12.01
OpenVLA-OFT [ 10 ]
MLP
59±2
95
7608.4 / 7709.2
14837.7 / 15331.6
4202.6 / 8766.6
91.13 / 147.16
Fast-WAM [ 21 ]
Flow matching
—
96
12406.3
23959.7
2545.1
438.53
RynnVLA-002 [ 4 ]
Regression
—
96
7150.3
15429.3
16297.3
436.55
Table 2: Full-task performance and computational cost. Scores are percentages (mean ± available SD). Cost pairs denote Push-T / LIBERO-Goal ; unpaired costs refer only to LIBERO-Goal . Memory is peak allocated GPU memory. Bold marks the best values within each task.
Figure 4: Predictive-feedback ablations on PushT. (a) Zero to three feedback updates. (b) Variants without GDRF or FSCF. Curves show 20 unsmoothed checkpoints from one training seed; enlarged markers and Best labels indicate peak success.
Figure 5: Prediction–execution correspondence on PushT. (a) Predicted and executed scene deviations from the outcomes of demonstrated actions, measured by feature RMSE across chunk boundaries for proposals from stages 1–4. (b) Reduction in terminal goal-feature MSE relative to stage 1, comparing predicted and executed improvements for stages 2–4; positive values indicate improvement. Each point represents one stage proposal in one case.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Updates
PushT
TwoRoom
Cube
0
16 / 45.37
100 / 13.99
100 / 2.42
1
52 / 32.53
100 / 14.14
96 / 2.47
2
68 / 24.42
100 / 14.13
100 / 2.44
3
96 / 18.30
100 / 14.08
100 / 2.41
Appendix
Table S1 : Short-horizon task performance of a fixed model with different numbers of feedback updates.
Task
Start
Feedback updates
0
1
2
3
PushT
Std.
4.69
17.97
32.81
56.25
Pert.
5.86
16.02
28.52
46.48
TwoRoom
Std.
99.22
100.00
100.00
100.00
Pert.
100.00
100.00
100.00
100.00
Cube
Std.
79.69
80.47
82.03
84.38
Appendix
Table S2 : Fixed-checkpoint AP (WA-JEPA) success (%) after 0–3 feedback updates. Mean averages the three tasks equally.
Task / feedback updates
0
1
2
3
Push-T coverage
31.00
60.30
59.65
70.78
LIBERO-Goal success
87
88
94
97
Appendix
Table S3 : Full-task performance of a fixed model with different numbers of feedback updates. Each setting uses 100 cases per task; all values are percentages.
Feedback input
Best
Final
Last 5
Goal difference (ours)
96
94
94.4
Goal-Concat
92
92
91.6
Appendix
Table S4 : Goal-input ablation on PushT. Success rates are percentages over 50 fixed cases. Best and Final refer to the best of 20 checkpoints and epoch 20; Last 5 averages epochs 16–20 within the same training run.
Task
Method
Mean (ms)
SD (ms)
PushT
WA-JEPA
32.13
0.36
LeWM
1047.36
2.65
DINO-WM + CEM
98924.91
34.01
TwoRoom
WA-JEPA
32.62
0.42
LeWM
351.06
1.54
DINO-WM + CEM
32962.39
22.51
Appendix
Table S5 : Complete-plan latency underlying Fig. 3 (a). Mean and standard deviation (SD) are in milliseconds; WA-JEPA and LeWM use 20 timed calls over ten input pairs; DINO-WM retains ten recorded calls. All use three warmups.
Method / candidates / iterations
PushT
TwoRoom
Cube
Success (%)
Latency (ms)
Success (%)
Latency (ms)
Success (%)
Latency (ms)
LeWM / 30 / 3
50
73.35
76
74.85
62
72.46
LeWM / 100 / 3
58
73.32
82
76.93
72
79.44
LeWM / 300 / 1
14
36.84
68
36.21
42
37.23
LeWM / 300 / 3
58
105.12
84
106.83
68
105.91
LeWM / 300 / 10
94
345.24
90
351.06
72
352.28
Appendix
Table S6 : Locally reevaluated success and latency under different LeWM CEM search budgets, compared with WA-JEPA. Latency measures one complete plan on an A40. Dashes denote untested settings.
Method
Push-T
LIBERO-Goal
WebSSL (CLS, VQ-BeT)
5/5
10/1
Patch Policy (VQ-BeT)
5/5
10/1
ResNet-18 Patch (ACT)
1/5
1/5
OpenVLA-OFT
1/5
1/8
Fast-WAM
—
1/32
RynnVLA-002
—
2/5
Appendix
Table S7 : History frames / output actions per cost-measurement call. Each task uses its own deployment configuration; costs are not normalized by output length.
Representation
Shape
Current scene Z0 ; goal G,G
B×8×192
Context memory M
B×9L×192
Action tokens Q(r)
B×5×192
Action proposal A(r)
B×5×(5da)
Future scene Z(r)
B×5×8×192
Future state P(r)
B×5×dp
Appendix
Table S8 : Core tensors for the continuous-head goal-reaching model. B : batch size; L : history frames; H : action chunks; b : controls per chunk. The full-task VQ-BeT configuration is specified separately in App. C.3.3 .
Setting
Protocol
Short horizon
25 / up to 25 actions; 50 initial–goal pairs per task. Goal: recorded image at +25 steps. Metric: physical goal success.
5 / 1 actions; 100 rollouts per run. Goal: task goal. Metric: final coverage.
Full-task LIBERO
5 / 1 actions; 10×10 rollouts per run. Goal: end of a training demonstration. Metric: task success on LIBERO-Goal.
Appendix
Table S9 : WA-JEPA execution and evaluation protocols. Predicted / executed lengths count environment actions. Four internal stages refine the same pending plan without new observations.
Learning latent dynamics is fundamental to building action-conditioned world models for planning. Among existing families, Joint Embedding Predictive Architectures (JEPAs) offer a natural framework by modeling transitions directly in latent space rather than at the pixel level. However, two issues remain in JEPA-based world models: (i) physical grounding lacks explicit aggregation of temporal context and (ii) action planning always overly relies on generic sampling distributions. To this end, we propose G-JEPA, which incorporates temporal context into physical grounding for latent world models and couples the learned models with prior-guided action planning. Specifically, during learning, we incorporate a shared causal temporal module that aggregates temporal context within observation and prediction sequences under physical state supervision, grounding both in task-relevant physical states. When planning, we further introduce action inference through stepwise conditioning of a generative action distribution on the current and goal latent representations together with their evolving discrepancy, yielding anchored proposals that guide model-based online refinement. Together, physical state supervision guides temporal representation learning, while the action prior guides search through the learned world model. Experiments demonstrate improved planning performance on manipulation tasks, including tasks with increasing temporal separation between initial and goal observations. Code is publicly available at https://github.com/myl11045/G-JEPA.
Yilei Man, Yang Chen, Jingcai Guo
Department of Computing, The Hong Kong Polytechnic University · Department of Land Surveying and Geospatial Science, The Hong Kong Polytechnic University
Latent world models predict the consequences of actions, but accurate prediction does not guarantee that latent distance reflects which candidate will execute successfully. We identify a decision-local prediction gap: among the few futures competing for execution, a candidate predicted closer to the goal can produce a worse realized outcome than an available alternative. We introduce D-JEPA, a decision-aligned latent world model that learns decision-relevant relations among candidate futures from executed outcomes. A bounded, permutation-equivariant operator jointly reasons over goal-relative predictive features and ordinal evidence, refining pretrained predictive geometry where action choices are most consequential. Restricted predictor adaptation and a shared ordinal interface extend this alignment across complementary predictive geometries. D-JEPA further realizes the learned decision structure in JEPA-compatible future representations, enabling deployment through native latent-distance planning. Evaluations across latent control, manipulation, pretrained action-producing models, physical robots and autonomous driving demonstrate improved action selection, including 87.89% success on PushT, a 15.04-point average gain on RoboTwin, and a 17-point gain on physical robot tasks. These results establish decision-relevant relational structure as a direct bridge between predictive world modeling and effective control.
Shuaijun Liu, Chengyu Wu, Qifu Wen +5
The Hong Kong University of Science and Technology (Guangzhou) · Boston University · Shanghai Jiao Tong University
Learning structured and control-relevant latent representations remains a key challenge for world models. Recent JEPA-based world models learn action-conditioned predictive latent dynamics from observation sequences. However, their forward-prediction objectives do not explicitly enforce reliable identifiability of robot-centric physical state from individual latents or state changes from latent pairs, which can limit downstream planning and policy performance. We propose PSG-JEPA, a physically grounded JEPA world model that shapes its latent space with two complementary grounding objectives beyond forward prediction: grounding individual latents in robot proprioceptive state, and grounding latent pairs in multi-horizon joint-angle changes. Both objectives are applied only during training, leaving the inference architecture and computational cost unchanged. To comprehensively evaluate PSG-JEPA, we conduct experiments at three levels: (1) latent identifiability via probing, (2) goal-conditioned planning on frozen latents, and (3) policy learning in simulation and on a real robot. Experiments demonstrate that our PSG-JEPA consistently outperforms state-of-the-art latent world-model baselines at all three levels.
Haodong Yan, Jiaguan Zhu, Mingyuan Jia +12
1The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · 2COCO Matrix