RoboJEPA: Scaling Robotic Latent World Models

Artem Zholus; Nicolas Beltran-Velez; Jianhao Yuan; Sarath Chandar; Tushar Nagarajan; Daniel Severo; Koustuv Sinha; Michal Drozdzal

cs.AIOct 7, 2026

RoboJEPA: Scaling Robotic Latent World Models

Authors: Artem Zholus, Nicolas Beltran-Velez, Jianhao Yuan, Sarath Chandar, Tushar Nagarajan, Daniel Severo, Koustuv Sinha, Michal Drozdzal, +4 more

Organizations: FAIR at Meta · Chandar Research Lab · Mila - Quebec AI Institute · Polytechnique Montréal · Work done at Meta

Abstract

Latent world models have shown a remarkable ability to predict future states and to plan in the real world. In practice, however, we lack a principled way to estimate how their capabilities scale with model size, data, and compute, an open problem that slows progress in the field. In this work we present RoboJEPA, a world model based on the Joint Embedding Predictive Architecture (JEPA) and trained on a large-scale dataset spanning 12 robotic embodiments. We show that RoboJEPA's imagination error, the error of its latent rollouts, follows a second-order power law in compute, allowing us to predict model quality well beyond the scale at which the law is fit. We further show that downstream robotic planning performance improves predictably with compute, and that imagination error is strongly correlated with it, making it a reliable proxy for real-robot evaluation. Finally, we demonstrate that latent world models can be deployed zero-shot as robotic agents, planning toward a single goal image to solve tasks requiring long-horizon planning on real hardware. We release all model checkpoints together with our training and robot deployment code. To our knowledge, this is the first work to establish scaling laws for multi-embodiment robotic world models trained on real robot data, and RoboJEPA, at 8B parameters, is the largest JEPA predictor model trained to date.

Figures & tables

Figure 2: World-model imagination improves with scale. Decoded imagination rollouts on an AgiBot episode for RoboJEPA at $0.3$ B, $2$ B, and $8$ B parameters, compared against the ground-truth (GT) video (bottom row). Each row shows the same rollout sampled at equal-time steps; larger models track the true dynamics and preserve object and contact detail markedly better . All world models share the same encoder (V-JEPA 2.1-G) and decoder.

Figure 3: RoboJEPA training. The frozen V-JEPA 2.1 encoder $f$ ( Mur-Labadia et al., 2026 ) maps the input multiview frames $x_{t}^{(1:V)}$ to a grid of features. The predictor $g_{\theta}$ consumes per-step sequences of multiview visual ( $z_{t}^{(1:V)}$ ), action ( $a_{t}$ ) and proprioceptive state ( $s_{t}$ ) tokens, conditioned on view and embodiment identifiers, and predicts the next-step features $\hat{z}_{t+1}$ . Training minimises two losses: (left) the teacher-forcing loss , the per-token $\ell_{1}$ distance between predicted features and the next-step features from the encoder ( Equation 5 ); and (right) the rollout loss , which attempts to predict the same target sequence as the teacher-forcing loss, but given fewer input frames. The model is given the full training sequence of actions but only a prefix of the encoded features, and predicts the remaining features autoregressively, with a KV-cache. This way the model learns to make accurate long term predictions.

Dataset group	Episodes	Video hours	Action hours	Embodiments
DROID ( Khazatsky et al., 2024 )	92k	414	138	Franka
Roboset ( Bharadhwaj et al., 2024 )	73k	1244	380	Franka (Roboset)
RoboMind ( Wu et al., 2024b )	35.3k	276	98	Franka, AgileX
LeRobot ( Cadene et al., 2024 )	21.5k	182	93	SO-101
RoboCasa365 ( Nasiriany et al., 2026 )	32k	482	482	Franka + Omron (sim)
1X ( 1X Technologies, 2024 )	23.5k	90	90	1X humanoid

Table 1: Compressed view of the

23

-dataset,

26

-action-space mixture. Video hours sum the durations of all recorded videos. Action hours measure the total duration of teleoperation or policy execution, counted once regardless of the number of cameras. The per-dataset breakdown with resolutions and source FPS is in Table 7 ; sampling weights are in Table 8 ; and semantic-camera mappings are in Table 9 . We report dataset sizes that we were able to collect and verify, which may be smaller than those reported in the original papers.

Figure 4: Two-view imagination scales in simulation. First $60$ s of a RoboCasa rollout (two agent-view cameras, stacked within each cell) imagined by RoboJEPA at $0.3$ B, $2$ B, and $8$ B parameters. Larger models keep the two views mutually consistent and preserve manipulation detail markedly deeper into the rollout. All world models share the same encoder (V-JEPA 2.1-G) and decoder.

Panel of Figure 5 — Figure 5: DROID holdout (real Franka, unseen scenes).

Parametric Curve	Curve parameters		Extrapolation Error (4B and 8B), $\times 10^{-3}$
	DROID	RoboCasa	DROID	RoboCasa
$L(C)=E+A\,C^{\alpha}$	$E{=}0.182,\ A{=}108.8,\ \alpha{=}{-}0.151$	$E{=}0.075,\ A{=}23.2,\ \alpha{=}{-}0.103$	$2.0$	$3.7$
$L(C)=E+A\,C^{\alpha-\gamma\ln C}$	$E{=}0.218,\ A{=}7.96{\times}10^{-19}$ $\alpha{=}1.91,\ \gamma{=}0.02308$	$E{=}0.1709,\ A{=}1.66{\times}10^{-17}$ $\alpha{=}1.761,\ \gamma{=}0.02105$	$\mathbf{0.6}$	$\mathbf{1.4}$
BNSL ( Caballero et al., 2022 ) , $n{=}1$	Section J.1		$1.0$	$1.8$
UNSL ( Caballero et al., 2026 ) , $n{=}1$	Section J.2		$1.4$	$5.7$

Table 2: Variants of parametric curves as scaling-law candidates for the

L_{1}

rollout error of RoboJEPA world models. Curve fits use models sized

22

M–

2

B parameters; we report held-out extrapolation error at 4B and 8B for DROID and RoboCasa. The BNSL and UNSL functional forms and their fitted parameter values are in Appendix J .

Panel of Figure 8 — Figure 8: The normalized return for the pose reach task in RoboCasa environment evaluated in unseen kitchens.

Figure 11: Short-horizon prediction training on obstacle-reach. Final reward versus training compute. The reward correlates with compute, but no clean frontier emerges and smaller models often beat larger ones.

Panel of Figure 12 — Figure 12: Imagination error scaling for short- vs. long-horizon prediction training.

Figure 15: Downstream success rate on the RoboCasa push task versus training compute, for the long-horizon ( $K=10$ ) models.

Figure 16: Real-world deployment on DROID. Success rates across model sizes for three manipulation tasks; the dashed line shows their average.

Model	Goal	Grasp		Object Lift		Pick and Place
Model	Goal	Progress	Success	Progress	Success	Progress	Success
$\pi_{0}$ -FAST	Text	35	22	20	12	55	40
$\pi_{0.5}$	Text	25	5	12	0	69	53
RoboJEPA-4B	Image	60	60	54	30	49	21
RoboJEPA-8B	Image	67	67	65	50	42	27

Table 3: Real-robot task progress and success rates (%). Full results are in Table 12 .

Appendix figures & tables41 assets

Supplementary material from the paper’s appendix.

Appendix

	V-JEPA 2-AC ( Assran et al., 2025 )	RoboJEPA (ours)
Architecture
Base ViT architecture	V-JEPA 2
Training parameterisation	Feed-forward next-embedding (frame) prediction with frame-causal mask; trained with an $\ell_{1}$ latent loss.
Normalisation	LayerNorm	RMSNorm
QK-norm	No	Yes
Position embedding	3-D spatio-temporal RoPE	4-D RoPE (space–time $+$ views)

Appendix

Table 4: Architectural comparison between the action-conditioned V-JEPA 2 predictor (V-JEPA 2-AC) ( Assran et al., 2025 ) and the RoboJEPA predictor. Both share the V-JEPA 2 backbone and the next-frame latent-prediction parameterisation; RoboJEPA adds RMSNorm and QK-norm for training stability at scale, a fourth (view) RoPE axis, and explicit per-embodiment and per-view conditioning for cross-embodiment, multi-view world modeling.

$12$ default state-delta ids (one per physical platform)
#	Platform	Notes
1	Franka	DROID, RoboMind-Franka, Roboset, OXE-FMB (default)
2	Franka + Omron	RoboCasa (action-key)
3	SO-101	LeRobot
4	AgiBot dual-arm (Robotiq)	AgiBot World
5	AgiBot humanoid	AgiBot World

Appendix

Table 5: Robot-id breakdown used in this work. The first

12

rows are default state-delta ids, one per physical platform. The next

14

rows are extra (platform, action-source) or (platform, state-convention) splits. Total robot ids:

|\mathcal{E}|\!=\!12\!+\!14\!=\!26

Hyperparameter	22M	50M	100M	300M	1B	2B	4B	8B	Notation
Predictor depth	$7$	$10$	$14$	$24$	$40$	$40$	$52$	$72$	$N_{\text{layers}}$
Predictor width	$512$	$640$	$768$	$1024$	$1536$	$2048$	$2560$	$3072$	$D$
Query heads	$4$	$5$	$6$	$8$	$12$	$16$	$20$	$24$	$h_{q}$
KV heads (GQA)	$4$	$5$	$6$	$8$	$4$	$8$	$10$	$12$	$h_{kv}$
Head dim	$128$	$128$	$128$	$128$	$128$	$128$	$128$	$128$	$d_{h}$
MLP hidden ratio	$4$	$4$	$4$	$4$	$4$	$4$	$4$	$4$	$r_{\mathrm{ffn}}$

Appendix

Table 6: Hyperparameters of the RoboJEPA predictor at eight scales (zero-init view bias). Only depth

N_{\text{layers}}

, width

D

, and head counts

(h_{q},h_{kv})

change across scales; everything else (encoder, factorised RoPE on

(t,i,j,v)

, post-RoPE full-head-width view bias, padded action / state widths, local temporal window

W_{t}

, attention kernel, output projection, training loss) is held fixed. Predictor parameter counts are approximate

D^{2}N_{\text{layers}}

-dominated estimates.

Dataset	Traj.	Vid. hrs	Act. hrs	Resolution	FPS	Embodiment
DROID ( Khazatsky et al., 2024 )	92k	414	138	$720{\times}1280$	60	Franka
Roboset ( Bharadhwaj et al., 2024 )	73k	1244	380	$240{\times}424$	5	Franka (Roboset)
RoboMind Franka ( Wu et al., 2024b )	25k	92	37	$480{\times}640$	30	Franka
RoboMind AgileX ( Wu et al., 2024b )	10.3k	184	61	$480{\times}640$	30	AgileX
LeRobot ( Cadene et al., 2024 )	21.5k	182	93	$480{\times}640$	30	SO-101
RoboCasa365 ( Nasiriany et al., 2026 )	32k	482	482	$720{\times}1280$	20	Franka + Omron (sim)

Appendix

Table 7: Full per-dataset breakdown of the

23

-dataset,

26

-embodiment mixture. “Act. hrs” counts only frames with synchronized proprioception. The compressed main-text view is Table 1 ; per-rank sampling weights are in Table 8 .

Dataset (action-source)	1-view weight	2-view weight	Total	Marginal dataset weight
DROID (state-delta cartesian)	$0.0233$	$0.0306$	$0.0282$	0.1692
DROID (state-delta joints)	$0.0233$	$0.0306$	$0.0282$
DROID (action-key cartesian)	$0.0233$	$0.0306$	$0.0282$
DROID (action-key joints)	$0.0233$	$0.0306$	$0.0282$
DROID (action-key cartesian-vel)	$0.0233$	$0.0306$	$0.0282$
DROID (action-key joint-vel)	$0.0233$	$0.0306$	$0.0282$

Appendix

Table 8: Sampling weights for the 1-view and 2-view groups of the 1B run. The top block lists datasets that ship two reliably informative external cameras and feed both groups; the lower block lists single-camera sources that feed only the 1-view group. DROID and RoboCasa appear under multiple (platform, action-source) pairs — each one is a distinct embodiment id with its own per-embodiment input projection ( Section B.7 ) — but all variants share the same underlying source episodes.

Semantic id $c$	$0$	$1$	$2$	$3$	$4$	$5$	$6$
Role	Left exterior	Right exterior	Wrist / eye-in-hand	Head / front-facing	Default / unmapped	Top-down	Planar-arm overhead
DROID	left	right	wrist	—	—	—	—
Roboset	left	right	wrist	—	—	top	—
RoboMind	camera_left	camera_right	—	camera_front	—	camera_top	—
RoboMind AgileX	—	—	camera_{left,right}_wrist	camera_front	—	—	—
RoboCasa	robot0_agentview_left	robot0_agentview_right	robot0_eye_in_hand	—	—	—	—

Appendix

Table 9: Mapping from raw per-dataset camera names to the

|\mathcal{V}|{=}7

semantic camera ids consumed by the predictor’s per-block view conditioning ( Equation 9 ). Rows are datasets; columns are semantic ids; each cell shows the raw camera name(s) routed to that id by the per-dataset regex. After regex matching, only the semantic id

c\!\in\!\{0,\dots,6\}

is consumed by the predictor’s per-block view bias

U_{\ell}[c^{(v)}]

; the original camera names are not seen by the model. The Default / unmapped bucket (id

4

) collects sources that emit a single generic camera (LeRobot, OXE RoboNet) or a non-canonical camera label (OXE Bridge V2 camera{1,2} ) for which no semantic role is reliably consistent across the corpus.

Hyperparameter	Symbol	Value
Loss (defined once in Equation 5 )
Pre-loss normalization	$\mathrm{LN}_{0}$	parameter-free LayerNorm on $D_{e}$
Rollout step index	$\tau\!\in\!\{0,\dots,K_{\mathrm{ar}}\}$	—
Rollout prefix sampling	$\tau_{0}$	uniform on $\{0,\dots,T{-}K_{\mathrm{ar}}\}$
Recurrent rollout input	—	detached from autograd graph
Rollout / TF mixing	—	$\tfrac{1}{2}\!:\!\tfrac{1}{2}$

Appendix

Table 10: Additional training hyperparameters for the 1B-predictor RoboJEPA run, covering the loss, optimizer, and cooldown configuration. Settings already listed in Table 6 , including the loss type and pretraining rollout horizon, are omitted here. The loss is defined in Equation 5 , and AdamW follows Loshchilov and Hutter (2019) .

Hyperparameter	Symbol	Value
Architecture
DiT depth	$L$	32
Hidden dim	$d$	1280
Query / KV heads	–	20 / 10 (GQA, ( Ainslie et al., 2023 ) )
MLP ratio	–	4.0
Cosmos latent channels	–	16

Appendix

Table 11: Diffusion-decoder hyperparameters. The decoder is trained on frozen V-JEPA 2.1 encoder features only; the world model, V-JEPA encoder, and Cosmos tokenizer ( NVIDIA, 2025 ) are frozen, with no gradient flow between the decoder and the world model. The local temporal window

W_{\tau}{=}4

acts on Cosmos latents (themselves

4{\times}

temporally compressed) and is distinct from the predictor’s

W_{t}{=}8

-frame window over V-JEPA features ( Section 2.1 ).

Figure 17: High Resolution 720p World Model Imagination Rollout.

Figure 18: AgiBot (single view), $480{\times}640$ . Model-size comparison of world-model imaginations against ground truth; folding structure emerges only at larger scales.

Figure 19: Open X-Embodiment Bridge (single view), $480{\times}640$ . Physical contact between the gripper and the object develops only for the larger models.

Figure 20: AgiBot (two views), $480{\times}640$ . Bimanual manipulation rollout across model sizes; finer details are preserved as scale increases.

Figure 21: RoboCasa (two views), $480{\times}640$ . Simulated kitchen manipulation rollout across model sizes.

Figure 22: Open X-Embodiment Bridge (single view), $240{\times}320$ . Contact with the manipulated object develops only in the larger models.

Figure 23: Bridge-V2 (single view), $240{\times}320$ . Clear progression of imagination quality with model scale.

Figure 24: LeRobot (two views), $240{\times}320$ . Two-view manipulation rollout across model sizes.

Figure 25: Franka / DROID (two views), $240{\times}320$ , joint-space actions. Two-view manipulation rollout across model sizes.

Panel of Figure 26 — Figure 26: Real-world Franka lift the cup task. A goal image (left) and the initial observation (right) for one episode on the DROID/Franka platform, shown from both cameras: the external side camera (top) and the wrist-mounted camera (bottom). The task is specified purely by the goal image, in which the cup has been lifted off the table; from the start state the planner drives the world model toward these goal features. The two camera views are the inputs the world model is conditioned on at deployment.

Model finetuned on DROID, no in-domain finetuning
Model	Goal specification	Grasp		Object Lift		Pick and Place
Model	Goal specification	Task Progress	Success Rate	Task Progress	Success Rate	Task Progress	Success Rate
$\pi_{0}$ -FAST	text instruction	$35\%$	$22\%$	$20\%$	$12\%$	$55\%$	$40\%$
$\pi_{0.5}$	text instruction	$25\%$	$5\%$	$12\%$	$0\%$	$69\%$	$53\%$
Pretrained models, no DROID finetuning, no in-domain finetuning, a unified planner for all tasks
RoboJEPA-22M	single goal image	$30\%$	$30\%$	$34\%$	$2\%$	$30\%$	$17\%$

Appendix

Table 12: Real-world robotic capabilities on the Franka/DROID platform. Task progress and success rate on the grasp, object-lift, and pick-and-place tasks. VLA baselines (

\pi_{0}

-FAST and

\pi_{0.5}

) are conditioned on a text instruction, whereas all RoboJEPA models are conditioned on a single image goal.

Hyperparameter	Symbol	Value
Horizon and closed loop
Planning horizon (predictor steps)	$H$	$5$
Context length (encoder steps)	$L_{\mathrm{ctx}}$	$2$
Execution prefix (actions per replan)	$L_{\mathrm{exec}}$	$2$
Environment steps per episode	—	$20$
CEM search

Appendix

Table 13: Closed-loop CEM planning hyperparameters for the DROID/Franka real-robot deployment ( Section 2.2 , Algorithm 1 ). Elites are selected from the pooled

N_{\mathrm{samp}}\!\cdot\!G

samples gathered across the

G

planner GPUs, so

k

may exceed the per-GPU sample count. The planner acts in a

3

D-translation

+

1

D-gripper space with the end-effector orientation held fixed.

Panel of Figure 27 — Figure 27: RoboCasa Reach. Goal and initial observations for one Reach episode, from the external side camera (top) and the wrist camera (bottom). The goal image (left) specifies the target end-effector pose in an emptied kitchen; the planner drives the arm from the start (right) toward the goal.

Panel of Figure 28 — Figure 28: RoboCasa ObstacleReach. Goal and initial observations for one ObstacleReach episode. An appliance placed directly in front of the robot forces the end-effector to arc around it, so the straight-line path from the start (right) to the goal (left) is blocked.

Panel of Figure 29 — Figure 29: RoboCasa ObjectReach (banana). Goal and initial observations after the scripted grasp: from the start (right) the robot must carry the held banana to the goal pose (left), shown from the side (top) and wrist (bottom) cameras.

Panel of Figure 30 — Figure 30: RoboCasa ObjectReach (can). As in Figure 29 for the can object.

Panel of Figure 31 — Figure 31: RoboCasa ObjectReach (mug). As in Figure 29 for the mug object.

Panel of Figure 32 — Figure 32: RoboCasa Push. Goal and initial observations for one object-push episode, shown from the side (top) and wrist (bottom) cameras. The robot must push the object across the counter from its initial position (right) to the goal position (left). Note that the goal pose and the initial pose of the robot are the same, the only thing that changes is the object position.

Planning hyperparameters
	Symbol	Reach	ObstacleReach	ObjectReach	Push
Predictor horizon	$H$	$1$	$5$	$1$	$5$
CEM samples per iteration (total)	—	$1024$
CEM iterations (cold / warm)	$I_{0}/I_{\mathrm{step}}$	$15$ / $10$
Elite count, top- $k$	$k$	$20$
Computational cost	$d(\cdot,\cdot)$	per-token $\ell_{1}$ in $\mathrm{LN}_{0}$ space

Appendix

Table 14: Planning and environment configuration for the four RoboCasa tasks ( Section G.4 ). The table is split into the planning hyperparameters (top) and the environment parameters (bottom). Among the planning hyperparameters, only the predictor horizon

H

varies across tasks; the rest are shared. The effective CEM sample budget is

1024

candidates per iteration for every task (distributed across the planner GPUs), from which the top-

k

elites are selected.

Panel of Figure 33 — Figure 33: Real-robot Grasp task: goal and initial observations. Ten episodes (rows) of the Grasp task on the DROID/Franka platform. For each episode we show the goal image (left pair) and the initial state (right pair), each from the external side camera and the wrist camera. The goal is specified purely by these images; the planner drives the robot from the initial state toward the goal.

Panel of Figure 34 — Figure 34: Real-robot Lift task: goal and initial observations. Ten episodes (rows) of the Lift task on the DROID/Franka platform. For each episode we show the goal image (left pair) and the initial state (right pair), each from the external side camera and the wrist camera. The goal is specified purely by these images; the planner drives the robot from the initial state toward the goal.

Panel of Figure 35 — Figure 35: Real-robot Pick-and-Place task: goal and initial observations. Ten episodes (rows) of the Pick-and-Place task on the DROID/Franka platform. For each episode we show the goal image (left pair) and the initial state (right pair), each from the external side camera and the wrist camera. The goal is specified purely by these images; the planner drives the robot from the initial state toward the goal.

Configuration	Wall (ms)	GPU kernel (ms)	Dispatch (ms)	Dispatch %
1-view eager	145.6	76.7	68.9	47.3%
1-view AOT	45.0	41.9	3.1	6.8%
2-view eager	217.1	148.7	68.4	31.5%
2-view AOT	96.7	93.7	3.0	3.1%

Appendix

Table 15: Per-step decomposition into GPU kernel time and host-side dispatch overhead at

B{=}16

on H200. AOT Inductor removes essentially all dispatch (

-65

ms) and additionally speeds up GPU compute by

{\sim}1.8\times

through kernel fusion (

8{,}975\to 1{,}291

kernels for the 1-view model).

1

-view rows are retained here because dispatch behavior is qualitatively different at small sequence length, even though the deployed CEM planner runs the 2-view path.

Cache step	BF16 AOTI (ms)	FP8 torch.compile (ms)	Speedup
$0$	61.5	54.8	$1.12\times$
$4$	94.2	82.4	$1.14\times$
$7$ – $9$	${\sim}108$	${\sim}103$	$1.05\times$
average	93.1	84.1	$\mathbf{1.11\times}$

Appendix

Table 16: Static FP8 vs. BF16 AOTI on the 2-view 1B predictor at

B{=}16

, averaged across the rolling-cache trajectory.

Dataset	$E$	$A$	$\alpha$	$d$	$f$	$\beta f$
DROID	$0.201$	$0.115$	$\approx 0$	$3.15\times 10^{19}$	$0.651$	$0.14$
RoboCasa	$0.184$	$3.17$	$0.067$	$2.60\times 10^{36}$	$3.11$	$7.9\times 10^{4}$

Appendix

Table 17: Fitted BNSL (

n{=}1

, Equation 23 ) parameters for the DROID and RoboCasa holdouts.

Dataset	Fitted parameters
DROID	$a_{0}{=}0.098,\ A_{3}{=}3.71$ $K_{ND}\!:\ b{=}1.23\times 10^{11},\ c_{0}{=}({-}0.38,1.31),\ f{=}1.81,\ d{=}2.18\times 10^{9},\ c_{1}{=}(2.09,{-}0.68)$ $K_{N}\!:\ b{=}3174,\ c_{0}{=}0.50,\ f{=}0.65,\ d{=}2.76,\ c_{1}{=}0.06$ $K_{D}\!:\ b{=}1.36,\ c_{0}{=}0.064,\ f{=}{-}0.19,\ d{=}1114,\ c_{1}{=}0.27$
RoboCasa	$a_{0}{=}0.046,\ A_{3}{=}1.54$ $K_{ND}\!:\ b{=}3505,\ c_{0}{=}({-}0.12,0.48),\ f{=}0.67,\ d{=}1.23\times 10^{15},\ c_{1}{=}({-}0.75,1.81)$ $K_{N}\!:\ b{=}5.03\times 10^{5},\ c_{0}{=}0.76,\ f{=}0.55,\ d{=}1.85\times 10^{-5},\ c_{1}{=}{-}0.55$ $K_{D}\!:\ b{=}2.62\times 10^{-12},\ c_{0}{=}{-}0.92,\ f{=}0.65,\ d{=}2.52\times 10^{8},\ c_{1}{=}0.73$

Appendix

Table 18: Fitted UNSL (

n{=}1

, Equation 25 ) parameters for the DROID and RoboCasa holdouts. Each

K

component is an MBNSL kernel ( Equation 24 ); for

K_{ND}

the exponents

c_{0},c_{1}

are vectors over

(N,D)

Figure 36: Short- vs. long-horizon prediction training on DROID. Out-of-distribution imagination L1 on the DROID holdout versus training compute. Faded curves are per-size checkpoints (blue: short-horizon training, orange: long-horizon training; shade encodes model size); the bold curves are the two second-order power laws fit on each recipe’s frontier. The long-horizon training law overtakes the short-horizon training one at high compute and reaches a lower irreducible floor.

Figure 37: Short- vs. long-horizon prediction training on RoboCasa. As Figure 36 , for the RoboCasa target evaluation. The same crossover and lower floor for long-horizon training hold on the simulated suite.

Figure 38: Long-horizon training scaling law on DROID. Long-horizon prediction training alone: per-size checkpoint curves colored by parameter count, with the fitted second-order power law (dashed) on the compute-optimal frontier of the DROID holdout evaluation.

Figure 39: Long-horizon training scaling law on RoboCasa. As Figure 38 , for the RoboCasa target evaluation.

Model size	Short-horizon training	Long-horizon training
	( $K{=}2$ , $240{\times}320$ )	( $K{=}10$ , $240{\times}320+480{\times}640$ )
$22$ M	$1.56$	$6.86$
$50$ M	$3.10$	$13.11$
$100$ M	$5.74$	$23.53$
$300$ M	$15.66$	$41.9$
$1$ B	$47.77$	$176.15$

Appendix

Table 19: Per-optimizer-step training compute (in

\mathrm{PF}

, i.e.

10^{15}

FLOPs) for each model size. All values report total FLOPs per optimizer step across the full training job and are used for the compute axis of the scaling-law plots. The short-horizon training column also applies to pretraining. The short-horizon recipe is exact; the long-horizon recipe runs its rollout with a KV cache and is the estimate without the cache scaled by the measured KV-cache factor

0.36

Stage	Recipe	Total per-step PFLOPs
Pretraining / short-horizon training	$K{=}2$ , $240{\times}320$ ( $\mathrm{tp}{=}1$ )	$305$ PF
Long-horizon training	$K{=}10$ with KV cache, multi-res ( $\mathrm{tp}{=}2$ )	$1{,}003$ PF
DROID $720{\times}1280$ finetune	$3$ -view $720{\times}1280$ , $K{=}10$ with KV cache ( $\mathrm{tp}{=}4$ )	${\sim}18{,}900$ PF

Appendix

Table 20: Per-step training compute at

8

B for the three stages. All values report total FLOPs per optimizer step across the full training job. One finetune step is

{\sim}19\times

a long-horizon training step and

{\sim}62\times

a short-horizon training step. The finetune uses a longer sequence: a

720{\times}1280

image is a

45{\times}80{=}3{,}600

-token grid per view (

\times 3

views

=10{,}800

tokens, versus

{\sim}300

tokens/view at

240{\times}320

), and self-attention is

\mathcal{O}(n^{2})

in tokens.

Type	Benchmark	Metric	V-JEPA 2	V-JEPA 2.1
			ViT-g (1B)	ViT-G (2B)
Global	SSv2 action recognition	Top-1 (%) $\uparrow$	77.3	77.7
Global	K400 action recognition	Top-1 (%) $\uparrow$	87.3	87.7
Global	IN1K image classification	Top-1 (%) $\uparrow$	85.1	85.5
Global	IntPhys2 ( Bordes et al., 2025 )	VOE (%) $\uparrow$	66.0	58.8
Dense	ADE20K semantic segmentation	mIoU (%) $\uparrow$	24.4	47.9

Appendix

Table 21: Restated results from Mur-Labadia et al. (2026) (Tables 8–9) and Punzo et al. (2026) (Table 1, IntPhys2); V-JEPA 2 classification results are also reported by Assran et al. (2025) (Table 4). We distinguish global tasks (action recognition, image classification, and intuitive physics) from dense tasks (semantic segmentation and depth estimation), following the grouping in Figure 2 of Mur-Labadia et al. (2026) . Classification uses attentive probes; segmentation and depth use linear probes, with frozen encoders. IntPhys2 reports mean violation-of-expectation (VOE) accuracy over three seeds with a temporal attentive probe at each model’s best reported layer. Higher accuracy and mIoU, and lower RMSE, are better. Model sizes and pretraining recipes differ, so this is not a controlled ablation.

Explore similar work

Sep 3, 2026cs.RO

Toward Physically Grounded JEPA World Models for Goal-Conditioned Robotic Planning

Action-conditioned JEPA world models enable planning toward visually specified goals without reconstructing future pixels, yet latent prediction alone does not explicitly encourage the learned representations to retain information relevant to robotic control. We introduce an end-to-end JEPA world model that augments latent prediction with inverse dynamics (IDM) and state alignment (SA). While inverse dynamics discourages latent collapse and makes latent transitions informative of the actions that produced them, state alignment grounds consecutive representations in their associated physical configuration and motion. Across four benchmark tasks, our model attains the highest success rates on TwoRoom (100%), PushT (98%), and OGBench-Cube (87%), while performing comparably to LeWorldModel on Reacher. Our ablation further shows that adding state alignment consistently improves planning success over IDM alone across all four tasks. Although LeWorldModel, our primary baseline, attains higher average straightening on OGBench-Cube, transition-subspace analysis shows that its transition energy is concentrated in a substantially lower-dimensional subspace. Our state-aligned model exhibits a higher effective transition dimension than LeWorldModel and improves planning over IDM alone, supporting state alignment as an effective complement to inverse dynamics for robotic planning.

Muyuan Liu, Yue Huang, Zheng Liang +1

GENISOM AI, Beijing, China

Inverse Dynamics Robot Planning World Models

Ask PDF

Sep 30, 2026cs.RO

The Planning Limits of Latent World Models

World models offer a promising way to help robots understand how the physical world evolves and plan complex behaviours through imagination. Yet existing studies mainly demonstrate what these models can accomplish, leaving unclear when their predictions remain useful for planning and where they fail. We study this question using action-conditioned predictors built on five frozen self-supervised visual backbones: V-JEPA 2, V-JEPA 2.1, VideoMAEv2, VideoPrism, and DINOv2. We use frozen backbones to test representations intended to transfer across environments. We evaluate these models on diverse Meta-World manipulation tasks and real-robot interactions from BridgeData V2. We find that a world model guides action selection reliably only when the goal lies within, or slightly beyond, the trajectory it imagines during planning. With five-step rollouts, the length the predictor was trained on, the world model ranks actions reliably only for targets five to ten control steps ahead, whereas task goals lie 16 to 53 steps away. Neither an 81-fold larger predictor nor longer-rollout training extends this range; the encoder affects both range and closed-loop success, with V-JEPA 2.1 performing most consistently. More fundamentally, the limit persists under perfect prediction: using the real simulator, success falls from 92% to 41% as the target moves from five to twenty steps ahead of a five-step rollout. Planning therefore requires either longer imagined trajectories or closer subgoals. For distant goals, pure imagination succeeds in 23% of episodes, planning with feedback (MPC) raises success to 30%, imagining as far as the goal to 47%, and nearby expert subgoals to 76%. Used within its plannable range, a world model can also improve a vision-language-action (VLA) policy: choosing among eight actions the VLA proposes raises its success from 65% to 77% across 16 different tasks.

Ali Alrasheed, Basim Azam, Naveed Akhtar

The University of Melbourne, Australia

Latent World Models World Models Action Prediction World Planning Fundamental Limits+1

Ask PDF

Aug 7, 2026cs.RO

Is Forward Prediction Enough? Physical State Grounding for JEPA World Models

Learning structured and control-relevant latent representations remains a key challenge for world models. Recent JEPA-based world models learn action-conditioned predictive latent dynamics from observation sequences. However, their forward-prediction objectives do not explicitly enforce reliable identifiability of robot-centric physical state from individual latents or state changes from latent pairs, which can limit downstream planning and policy performance. We propose PSG-JEPA, a physically grounded JEPA world model that shapes its latent space with two complementary grounding objectives beyond forward prediction: grounding individual latents in robot proprioceptive state, and grounding latent pairs in multi-horizon joint-angle changes. Both objectives are applied only during training, leaving the inference architecture and computational cost unchanged. To comprehensively evaluate PSG-JEPA, we conduct experiments at three levels: (1) latent identifiability via probing, (2) goal-conditioned planning on frozen latents, and (3) policy learning in simulation and on a real robot. Experiments demonstrate that our PSG-JEPA consistently outperforms state-of-the-art latent world-model baselines at all three levels.

Haodong Yan, Jiaguan Zhu, Mingyuan Jia +12

1The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · 2COCO Matrix

Latent World Models World Models World

Ask PDF