JEPA Dots: Dual-Latent World Model for Language-Guided Robot Control
Organizations: Z Robotics · Tsinghua University · Stanford University
Abstract
Predictive world models for language-guided robot control face a fundamental conditioning challenge: instructions should influence action selection without altering the predicted consequences of fixed actions under identical observations. To address this challenge, we introduce JEPA Dots (Dynamics-Oriented Task Separation), a compact dual-latent framework that jointly learns visual dynamics and language-guided control through separate dynamics and task representations. Language-conditioned task features guide a flow-matching policy, while masked attention restricts world prediction to instruction-independent dynamics history and supplied actions. In closed-loop evaluation on 40 trained LIBERO tasks, JEPA Dots achieves 84.9% overall success with approximately 39.1M parameters, compared with the published 78.7% of NanoVLA-S with 161.0M parameters under a different evaluation protocol. In the reference experiment, JEPA Dots also improves overall success from 76.0% for our task-ID-conditioned adaptation baseline to 84.9%, with LIBERO-Goal success increasing from 67.1% to 88.5%. Instruction-switching experiments further confirm that task representations and policy outputs respond to language changes, while dynamics representations and world predictions remain invariant under fixed observations and supplied actions.
Figures & tables
| Output | History | Clean | Goals | Noisy |
|---|---|---|---|---|
| actions | actions | |||
| World prediction | Yes | Prefix | No | No |
| Action velocity | Yes | No | Yes | Causal |
| Model | Total (M) | Train. (M) | Spatial | Object | Goal | LIBERO-10 | Overall |
|---|---|---|---|---|---|---|---|
| V-JEPA 2.1-B + FM | |||||||
| V-JEPA 2.1-L + FM | |||||||
| NanoVLA-S [ 9 ] | – | ||||||
| SmolVLA-240M [ 31 ] | – | ||||||
| JEPA Dots |
| Variant | Spatial | Object | Goal | LIBERO-10 | Overall |
|---|---|---|---|---|---|
| Task-ID reference | |||||
| Language-only goals | |||||
| Task head, no FiLM, full | |||||
| Task head + FiLM, full | |||||
| Task head, no FiLM, subspace | |||||
| JEPA Dots (complete) |
| Variant | Spatial | Object | Goal | LIBERO-10 | Overall SD | |
|---|---|---|---|---|---|---|
| FiLM, full regularization | ± | |||||
| No FiLM, subspace | ± | |||||
| JEPA Dots (complete) | ± | |||||
| Setting | Configuration |
|---|---|
| Views and history | External and wrist RGB; two observed frames |
| Capture and resize | uint8; HWC to CHW; divide by 255; bilinear resize to , align_corners=False |
| Image normalization | ImageNet mean and SD |
| Encoder crop | random crop during training; center crop at evaluation |
| Backbone | Shared ResNet-18, jointly trained from random initialization |
| Latents | Separate 192D dynamics and task heads for each view |
| Setting | Recorded value |
|---|---|
| Training hardware | One NVIDIA A100 per model process |
| Parallel experiments | Independent model processes scheduled across GPUs |
| Training duration | Not reported |
| Epochs and minibatches | 60 epochs, 500 minibatches per epoch; 30,000 forward/backward minibatches |
| Batch and accumulation | 32 per minibatch; accumulation 2; effective batch approximately 64 |
| Optimizer updates | Approximately 15,000 per completed run |
| Variant | Run A | Run B | Run C | Total |
|---|---|---|---|---|
| FiLM, full regularization | 3259 | 3227 | 3198 | 9684 |
| No FiLM, subspace | 3350 | 3239 | 3242 | 9831 |
| JEPA Dots | 3395 | 3233 | 3198 | 9826 |
| Run | Variant | Epoch/60 | Action loss ( ) | World MSE ( ) |
|---|---|---|---|---|
| A | Language-only goals | 55 | ||
| A | No FiLM, full | 57 | ||
| A | FiLM, full | 56 | ||
| A | No FiLM, subspace | 60 | ||
| A | JEPA Dots | 60 | ||
| A | Dense task head | 58 |
| Task | Description | No FiLM | DOTS | FiLM full | pp |
|---|---|---|---|---|---|
| S0 | Bowl between plate and ramekin to plate | 293 | 296 | 291 | |
| S1 | Bowl beside ramekin to plate | 237 | 224 | 172 | |
| S2 | Bowl at table center to plate | 297 | 295 | 296 | |
| S3 | Bowl on cookie box to plate | 287 | 285 | 292 | |
| S4 | Bowl in top drawer to plate | 247 | 226 | 250 | |
| S5 | Bowl on ramekin to plate | 254 | 252 | 255 |
| Variant | Bowl to stove | Bowl to plate | Both |
|---|---|---|---|
| No FiLM, subspace | 56/60 | 60/60 | 56/60 |
| JEPA Dots | 55/60 | 53/60 | 50/60 |
| Component or configuration | V-JEPA 2.1-B + FM | V-JEPA 2.1-L + FM |
|---|---|---|
| Encoder depth / width / heads | 12 / 768 / 12 | 24 / 1,024 / 16 |
| Frozen visual encoder | 86,833,152 | 304,680,960 |
| Trainable flow module | 4,596,871 | 4,596,871 |
| Other trainable adapters and conditioning | 969,918 | 1,150,142 |
| Total trainable | 5,566,789 | 5,747,013 |
| Vision–policy subtotal | 92,399,941 | 310,427,973 |
| Setting | Shared configuration |
|---|---|
| Suites and demonstrations | Ten tasks each from Spatial, Object, Goal, and LIBERO-10; 50 demonstrations per task, 2,000 total. |
| Training / validation | 45 / 5 complete demonstrations per task, or 1,800 / 200 total. No offline test split; final outcomes use simulator rollouts. |
| Deterministic split | Rank demonstrations by SHA-256 of the split identifier, task key, and demonstration name; hold out the first five per task. Both sizes use identical assignments. The split identifier is independent of the training initialization. |
| Window sampling | Task-balanced weighted sampling with replacement. |
| Cameras and history | External agentview_rgb and wrist eye_in_hand_rgb , at and . |
| Image preparation | Native RGB, bilinear resize to 224, crop, then resize to and ImageNet normalization. Random crops in training; center crops in validation and simulator evaluation. |
| Setting | Configuration |
|---|---|
| Rollout count | Three evaluation streams, 50 episodes per task per stream: 150 per task, 1,500 per suite, and 6,000 per model; 12,000 across both sizes. |
| Initial states | Official indices 0–49 per task, reused across streams. |
| Episode randomization | Each episode uses the base stream identifier plus the suite-local task index multiplied by 1,000, plus the episode index. |
| Step budget | Ten warmup steps, then at most 600 policy-controlled environment steps; terminate earlier on success. |
| Action sampling | Independent Gaussian action noise and eight fixed Euler integration steps. |
| Execution | Predict eight actions, inverse-standardize, clip to , and execute the first two before replanning. |
Language-Conditioned Robot Control
Language-conditioned robot control connects natural-language instructions to grounded, safe actions across tasks and robot embodiments.
See all 78 papers