Joint-embedding predictive architecture (JEPA) world models predict future embeddings instead of pixels and plan in latent space. Two challenges remain: 2D images leave the metric geometry that manipulation acts on implicit and entangled with appearance, and what these models imagine stays hidden, as they are mostly judged in simulation by short-horizon latent planning. We present JEPA4R, a JEPA world model that observes single-depth-camera point clouds, predicts and plans in their latent space, and acts through a policy head on its frozen latents; we call this 4D point planning. A decoder on the frozen world model shows what it imagines as 4D point clouds, with a mean point error within 1.1 cm over 200 steps. Under the policy-learning protocols of OGBench and CortexBench, JEPA4R leads pixel JEPA models on OGBench Cube-Single, Cube-Double, Cube-Triple and Scene, and ranks first on MetaWorld and Adroit Relocate with 98.4% and 88.0%, above the runner-up, R3M, at 96.0% and 65.3%. Trained only in a digital twin, JEPA4R and its policy heads run zero-shot on the real arm with both a gripper and a dexterous hand.
Figures & tables
Figure 2: JEPA4R predicts, plans and acts in the latent space of single-depth-camera point clouds and decodes what it imagines into 4D point clouds. Top (4D point planning): the encoder fθ maps Pt to zt , the predictor gϕ rolls the latent forward under actions, and the policy head πω acts on zt or, to look ahead, on imagined latents. Bottom (latent-space reconstruction): the decoder Dψ , attached to the frozen world model, turns every imagined rollout into 4D point clouds ( Eq. 2 ).
Figure 3: One scene, two observations. One OGBench Cube-Single frame seen by one camera, as pixels (left of the white line), the input of 2D JEPAs, and as a 1,200-point cloud (right), the input of JEPA4R.
Table 3Table 4
Figure 6: Key frames in simulation: OGBench Cube-Double, Scene and Puzzle, and MetaWorld Assembly. The label under each frame gives the state it shows, and the orange box marks the object that changed.
(a) Per-frame reconstruction
Latent
Mean
Median
p90
Outl. (%)
Cube
JEPA4R
0.79
0.78
0.94
2.89
1.05
Table 3: Decoding frozen latents into point clouds (OGBench Cube, cm). (a) Per frame: mean, median and 90th percentile (p90) of the bidirectional nearest-neighbor (NN) distance between decoded and observed points; Outl.: share of points with an NN distance above 3 cm; Cube: cube position error. Decoding the latent of another frame instead gives 12.31 cm. (b) Open-loop imagination; decoded true latents decode encoder, not predicted, latents, and another episode’s actions replay the actions of a different episode.
Table 4: Ablations. (a) Decoder backbone (OGBench Cube, cm). NN: mean bidirectional nearest-neighbor distance between decoded and observed points, per frame; Cube: cube position error, per frame; Step 200: NN distance of the decoded imagination at environment step 200. (b) Point-cloud encoder, (c) temporal window and (d) training hyperparameters (OGBench Scene, mean success over the five tasks, %). Context frames are spaced five environment steps apart. Default: learning rate 5×10−5 , batch size 128. LR: learning rate; † : default.
Figure 8: Real-robot deployment of JEPA4R on a RealMan RM75-B arm. Key frames of one execution per task at successive time steps, with the camera view above and the point cloud below: picking and pushing with a parallel gripper, and picking with a dexterous hand.
Task
Random
LeWM
JEPA4R
Δ
OGBench Cube
UR5e
42.8
69.4
65.2
− 4.2
RM75-B
47.0
73.8
79.8
+6.0
Table S1 : World-model planning success under LeWM’s setting (%). LeWM and JEPA4R share the predictor, objective and CEM planner; the observation and its parameter-matched encoder differ (pixels vs . depth-camera point clouds). OGBench Cube: LeWM protocol (50 starts, goal 25 steps ahead); scores average the 10 epoch checkpoints. Random: a random policy from the same starts. Δ : JEPA4R minus LeWM.
Figure S1 : Key frames on OGBench (I). Cube-Single, Cube-Double, Cube-Triple and Scene. One successful JEPA4R evaluation episode per task, from the start (left) to the first success (right); top: camera view, bottom: the point cloud JEPA4R receives.
Figure S2 : Key frames on OGBench (II). Puzzle on the 3×3 , 4×4 and 4×5 boards, shown as in Fig. S1 .
Figure S3 : Key frames on CortexBench (I). MetaWorld Assembly, Bin-pick and Button, shown as in Fig. S1 .
Figure S4 : Key frames on CortexBench (II). MetaWorld Drawer and Hammer, and Adroit Relocate, shown as in Fig. S1 .
Figure S5 : Decoded imagination on OGBench Cube-Single (I). Test episodes. The world model receives three context frames and then predicts from the recorded actions alone; top: ground-truth point clouds, bottom: point clouds decoded from the imagined latents, at environment steps 20 to 200.
Figure S6 : Decoded imagination on OGBench Cube-Single (II). Three more test episodes, shown as in Fig. S5 .
Figure S7 : Real-robot executions with the parallel gripper. Pushing cubes together and picking up a carrot, a cucumber and a chili on the RealMan RM75-B arm; top: camera view, bottom: point cloud.
Figure S8 : Real-robot executions with the dexterous hand. Pulling a bar, a bottle, a carrot and a banana out of a slotted holder, shown as in Fig. S7 .