Predictive world models for language-guided robot control face a fundamental conditioning challenge: instructions should influence action selection without altering the predicted consequences of fixed actions under identical observations. To address this challenge, we introduce JEPA Dots (Dynamics-Oriented Task Separation), a compact dual-latent framework that jointly learns visual dynamics and language-guided control through separate dynamics and task representations. Language-conditioned task features guide a flow-matching policy, while masked attention restricts world prediction to instruction-independent dynamics history and supplied actions. In closed-loop evaluation on 40 trained LIBERO tasks, JEPA Dots achieves 84.9% overall success with approximately 39.1M parameters, compared with the published 78.7% of NanoVLA-S with 161.0M parameters under a different evaluation protocol. In the reference experiment, JEPA Dots also improves overall success from 76.0% for our task-ID-conditioned adaptation baseline to 84.9%, with LIBERO-Goal success increasing from 67.1% to 88.5%. Instruction-switching experiments further confirm that task representations and policy outputs respond to language changes, while dynamics representations and world predictions remain invariant under fixed observations and supplied actions.
Figures & tables
Figure 1: JEPA Dots separates instruction-independent dynamics representations from language-conditioned task representations within a shared visual architecture. Masked world queries read only history and supplied clean actions; the policy reads history, task goals, and noisy actions. Future-latent targets and subspace regularization supervise training, while a flow-matching policy generates action chunks for receding-horizon control.
Output
History
Clean
Goals
Noisy
Ht
actions
Gt
actions
World prediction
Yes
Prefix
No
No
Action velocity
Yes
No
Yes
Causal
Table 1: Inputs to the two branches. World prediction reads dynamics history and clean actions. Goal tokens and noisy actions condition the policy.
Model
Total (M)
Train. (M)
Spatial
Object
Goal
LIBERO-10
Overall
V-JEPA 2.1-B + FM
≈115.1
5.6
72.2
85.8
80.4
36.8
68.8
V-JEPA 2.1-L + FM
≈333.1
5.7
73.0
81.9
83.3
41.4
69.9
NanoVLA-S [ 9 ]
161.0
–
81.6
93.6
89.6
49.8
78.7
SmolVLA-240M [ 31 ]
240.0
–
87.0
93.0
88.0
63.0
82.8
JEPA Dots
≈39.1
16.4
90.4
95.8
88.5
64.8
84.9
Table 2: Comparison with other controllers. Success rates (%) and parameters in millions. FM: flow matching. DOTS and V-JEPA totals include frozen MiniLM ( ≈22.7 M); trainable counts exclude it. V-JEPA rows average three rollout streams of one trained checkpoint per size (6,000 episodes); DOTS uses the reference experiment (4,000 episodes). Published policy scores and counts retain their source protocols; – denotes an unavailable trainable count. Best success rates are shown in bold.
Figure 2: Total parameter counts. DOTS and V-JEPA 2.1 + flow matching include frozen MiniLM (approximately 22.7M). Published policy counts follow their sources. Parameter count does not measure latency.
Variant
Spatial
Object
Goal
LIBERO-10
Overall
Task-ID reference
91.0
92.6
67.1
53.1
76.0
Language-only goals
84.5
87.8
85.4
39.0
74.2
Task head, no FiLM, full
91.2
94.0
85.3
56.2
81.7
Task head + FiLM, full
90.4
91.8
86.3
57.4
81.5
Task head, no FiLM, subspace
92.2
94.0
88.2
60.6
83.8
JEPA Dots (complete)
90.4
95.8
88.5
64.8
84.9
Table 3: Reference experiment. Success rates (%) on the four LIBERO suites, with 4,000 episodes per model. Full and subspace denote full-dimensional and eight-subspace SIGReg. Values are rounded to one decimal. Best success rates are shown in bold, including displayed ties.
Variant
Spatial
Object
Goal
LIBERO-10
Overall ± SD
FiLM, full regularization
87.8
93.2
85.1
56.7
80.7 ±
0.8
No FiLM, subspace
90.3
93.9
86.8
56.7
81.9 ±
1.6
JEPA Dots (complete)
88.3
94.1
86.8
58.4
81.9 ±
2.6
Table 4: Three independent training runs. Mean success rates (%) from 12,000 episodes per variant, or 3,000 per suite. Overall entries give the mean and sample SD of the three trained-model success rates. Best means are shown in bold, including displayed ties.
Figure 3: Selected task-level FiLM gains. The eight largest positive differences from the matched no-FiLM subspace model, pooled over three runs with 300 episodes per task and model. Complete results for all 40 tasks, including decreases, appear in the supplement.
Setting
Configuration
Views and history
External and wrist RGB; two observed frames
Capture and resize
128×128 uint8; HWC to CHW; divide by 255; bilinear resize to 224×224 , align_corners=False
Image normalization
ImageNet mean (485.0,456.0,406.0)×10−3 and SD (229.0,224.0,225.0)×10−3
Encoder crop
202×202 random crop during training; center crop at evaluation
Backbone
Shared ResNet-18, jointly trained from random initialization
Latents
Separate 192D dynamics and task heads for each view
Table S1: Architecture and preprocessing.
Setting
Recorded value
Training hardware
One NVIDIA A100 per model process
Parallel experiments
Independent model processes scheduled across GPUs
Training duration
Not reported
Epochs and minibatches
60 epochs, 500 minibatches per epoch; 30,000 forward/backward minibatches
Batch and accumulation
32 per minibatch; accumulation 2; effective batch approximately 64
Optimizer updates
Approximately 15,000 per completed run
Table S2: Optimization and closed-loop evaluation.
Variant
Run A
Run B
Run C
Total
FiLM, full regularization
3259
3227
3198
9684
No FiLM, subspace
3350
3239
3242
9831
JEPA Dots
3395
3233
3198
9826
Table S3: Exact closed-loop successes. Each run has 4,000 episodes per model. Bold marks the largest count in each column.
Run
Variant
Epoch/60
Action loss ( ×10−3 )
World MSE ( ×10−4 )
A
Language-only goals
55
469.4
110.6
A
No FiLM, full
57
466.0
53.0
A
FiLM, full
56
468.7
53.9
A
No FiLM, subspace
60
482.2
45.1
A
JEPA Dots
60
485.2
44.5
A
Dense task head
58
481.2
89.9
Table S4: Selected checkpoints. Epoch numbering is one-based. World MSE is reported at the checkpoint selected by action loss. Values multiply the power of ten in each heading and are rounded to one decimal. Bold marks the lowest displayed loss within each run; latent MSE uses each model’s learned coordinates.
Figure S1 : All 40 task differences across three training runs. Orange marks positive FiLM effects, blue negative effects. Each difference uses 300 episodes per model and task.
Task
Description
No FiLM
DOTS
FiLM full
Δ pp
S0
Bowl between plate and ramekin to plate
293
296
291
+1.0
S1
Bowl beside ramekin to plate
237
224
172
−4.3
S2
Bowl at table center to plate
297
295
296
−0.7
S3
Bowl on cookie box to plate
287
285
292
−0.7
S4
Bowl in top drawer to plate
247
226
250
−7.0
S5
Bowl on ramekin to plate
254
252
255
−0.7
Table S5: Complete task counts. Each model has 300 episodes per task. Descriptions are shortened; IDs are local to each suite. Bold marks the largest success count within each task, including ties. Differences are rounded to one decimal.
Variant
Bowl to stove
Bowl to plate
Both
No FiLM, subspace
56/60
60/60
56/60
JEPA Dots
55/60
53/60
50/60
Table S6 : Same-scene instruction switching. Each of 60 scenes is evaluated separately with two goals. “Both” counts scenes where both requested rollouts succeed. Bold marks the largest count in each column.
Component or configuration
V-JEPA 2.1-B + FM
V-JEPA 2.1-L + FM
Encoder depth / width / heads
12 / 768 / 12
24 / 1,024 / 16
Frozen visual encoder
86,833,152
304,680,960
Trainable flow module
4,596,871
4,596,871
Other trainable adapters and conditioning
969,918
1,150,142
Total trainable
5,566,789
5,747,013
Vision–policy subtotal
92,399,941
310,427,973
Table S7: V-JEPA comparison components. Exact counts are shown except for MiniLM and the resulting approximate full totals. The original vision–policy totals exclude the separate frozen sentence encoder; the main table includes it consistently for DOTS and both baselines. Parameter counts are not performance rankings.
Setting
Shared configuration
Suites and demonstrations
Ten tasks each from Spatial, Object, Goal, and LIBERO-10; 50 demonstrations per task, 2,000 total.
Training / validation
45 / 5 complete demonstrations per task, or 1,800 / 200 total. No offline test split; final outcomes use simulator rollouts.
Deterministic split
Rank demonstrations by SHA-256 of the split identifier, task key, and demonstration name; hold out the first five per task. Both sizes use identical assignments. The split identifier is independent of the training initialization.
Window sampling
Task-balanced weighted sampling with replacement.
Cameras and history
External agentview_rgb and wrist eye_in_hand_rgb , at t−1 and t .
Image preparation
Native 128×128 RGB, bilinear resize to 224, 202×202 crop, then resize to 224×224 and ImageNet normalization. Random crops in training; center crops in validation and simulator evaluation.
Table S8: Data and optimization for both V-JEPA sizes. An epoch denotes a fixed sampling budget, not a complete traversal of every available window.
Setting
Configuration
Rollout count
Three evaluation streams, 50 episodes per task per stream: 150 per task, 1,500 per suite, and 6,000 per model; 12,000 across both sizes.
Initial states
Official indices 0–49 per task, reused across streams.
Episode randomization
Each episode uses the base stream identifier plus the suite-local task index multiplied by 1,000, plus the episode index.
Step budget
Ten warmup steps, then at most 600 policy-controlled environment steps; terminate earlier on success.
Action sampling
Independent Gaussian action noise and eight fixed Euler integration steps.
Execution
Predict eight actions, inverse-standardize, clip to [−1,1] , and execute the first two before replanning.
Table S9: V-JEPA evaluation protocol. All three rollout sets use the same selected weights; they are not independent training runs.
World-action models adapt pretrained video generators, and vision-language-action models build on pretrained vision-language backbones; both tie policy architecture and inference cost to large transformers designed for generation rather than control. It remains unclear how much these generation-oriented choices affect control. In this work, we introduce Latent Action JEPA (La-JEPA), a compact world-action model whose policy transformer is trained from scratch in the feature space of a pretrained V-JEPA 2.1 encoder. Following Fast-WAM, La-JEPA learns to predict future features jointly with actions during training and generates only actions at inference. Among policies without embodied pretraining, La-JEPA achieves state-of-the-art performance in clean and randomized RoboTwin 2.0 scenes and on LIBERO-Plus with correct instructions, while using far fewer parameters than the strongest competing policies. It runs substantially faster than Fast-WAM at comparable performance and outperforms the closest JEPA-based methods, LeapBot-WA and JEPA-WAM, on LIBERO-Plus and RoboTwin 2.0 without their pretrained policy backbones. On four real-robot tasks, its average success is comparable to pi0.5 and above non-embodied-pretrained baselines. Through a design study of the choices behind La-JEPA, we show that JEPA features outperform video VAE latents, most clearly out of distribution, and that training-time future prediction choices significantly impact the policy even when future prediction is skipped at inference. We correct the LIBERO-Plus instructions, as the officially released evaluator corrupts them for most variants, and re-evaluate several strong open baselines. Without a generative pretrained policy backbone, La-JEPA is better and faster than far larger policies, suggesting that robot foundation models may benefit from architectures designed for control rather than inherited from generation.
Mateo Clémente, George Z. Xue, Amin Abyaneh +5
Huawei Technologies Ltd. · University of Toronto · McGill University +1
Latent world models predict the consequences of actions, but accurate prediction does not guarantee that latent distance reflects which candidate will execute successfully. We identify a decision-local prediction gap: among the few futures competing for execution, a candidate predicted closer to the goal can produce a worse realized outcome than an available alternative. We introduce D-JEPA, a decision-aligned latent world model that learns decision-relevant relations among candidate futures from executed outcomes. A bounded, permutation-equivariant operator jointly reasons over goal-relative predictive features and ordinal evidence, refining pretrained predictive geometry where action choices are most consequential. Restricted predictor adaptation and a shared ordinal interface extend this alignment across complementary predictive geometries. D-JEPA further realizes the learned decision structure in JEPA-compatible future representations, enabling deployment through native latent-distance planning. Evaluations across latent control, manipulation, pretrained action-producing models, physical robots and autonomous driving demonstrate improved action selection, including 87.89% success on PushT, a 15.04-point average gain on RoboTwin, and a 17-point gain on physical robot tasks. These results establish decision-relevant relational structure as a direct bridge between predictive world modeling and effective control.
Shuaijun Liu, Chengyu Wu, Qifu Wen +5
The Hong Kong University of Science and Technology (Guangzhou) · Boston University · Shanghai Jiao Tong University
Controlling an agent with vision requires being able to separate useful information from irrelevant background information. JEPA-style latent world models seem like a natural approach for this, as they do not perform pixel-level reconstruction; however, they are still sensitive to these distractor signals and experience latent collapse. In this work, we introduce Controllability Factorized JEPA (CF-JEPA), a JEPA-style world model which splits the latent space into controllable and uncontrollable subspaces. This factorization allows us to capture all the distractor information into the uncontrollable region, while we use the control-relevant latent information for our task. With this, we show comparable performance across 2D and 3D control tasks under nominal conditions and improved performance under distracted conditions, where CF-JEPA is the only model that does not experience latent collapse. We also validate our model under distracted conditions for a simulated robot task, highlighting the practical application of such a scheme.
Morgan Byrd, Robert Wright, Sehoon Ha
Georgia Institute of Technology, Atlanta, GA, 30308, USA · Georgia Tech Research Institute, Atlanta, GA, 30308, USA