Frozen in a Frame: The Velocity Blind Spot in JEPA World Models

Tinghe Zhang; Chunyu Liu; Yu Leon Liu; Zerui Zhao; Jiaheng Chen; Yucheng Xiao; Jiaxing Li; Yunlong Wang

cs.CVOct 3, 2026

Frozen in a Frame: The Velocity Blind Spot in JEPA World Models

Authors: Tinghe Zhang, Chunyu Liu, Yu Leon Liu, Zerui Zhao, Jiaheng Chen, Yucheng Xiao, Jiaxing Li, Yunlong Wang,

Organizations: Tsinghua University · Institute of Automation, Chinese Academy of Sciences · Northeastern University · Harvard University ·

Abstract

Joint-embedding predictive architectures (JEPAs) for world modeling train an encoder so a predictor maps a current embedding and action to the next frame's embedding, always from a single rendered frame. This has a structural blind spot: a renderer without motion blur draws a scene from configuration alone, so a single-frame embedding carries no velocity information, for any encoder, including the official released LeWM weights. We confirm this on official checkpoints across four real benchmarks (PushT, Reacher, Cube, TwoRoom): every linear velocity probe sits at or below chance while position probes reach R^2 about 0.95. We introduce RateIdent, a three-stage diagnostic protocol, and TI-JEPA, a lightweight fix splitting the latent into a pose code and an explicit finite-difference motion code, predicted jointly. Across three physically grounded environments, TI-JEPA gives a significant, seed-robust gain on a stop-at-goal planning task over a matched-memory baseline, e.g. 55% lower final distance on Pendulum (p=3.2x10^-10) and 64% on CartPole (p=5.1x10^-15). We reproduce this at official ViT-Tiny plus AdaLN-transformer scale, then push the same recipe onto real dm_control Reacher photographs trained from scratch, where TI-JEPA's branch separation exceeds the memory-having baseline's by roughly 38x, the paper's largest margin. Against a same-footprint recurrent RSSM-style predictor, TI-JEPA matches or beats its rollout accuracy on two of three environments, stays separately probeable for pose and motion, and wins outright on the most coupled one. A checkable formal argument and six evaluated environments show single-frame targets are the wrong object to predict when velocity matters, and a small, interpretable structural change fixes it with no privileged supervision. Code, checkpoints, and the project page are linked below the title.

Figures & tables

Figure 1: Overview of the evidence chain. A : probe $r$ for velocity on an 8-spoke radar, single-frame baseline versus TI-JEPA’s explicit motion code, across eight environment/scale combinations ( $\times$ marks official checkpoints with no $v$ -head trained, not a zero). B : kill-experiment branch-separation ratio; bar height is separation magnitude relative to ground truth (1.0 = exact match) and is independent of the sign/direction accuracy annotated above each bar. C : closed-loop planning improvement over the memory-matched baseline on the three environments where the task is well-posed (real PushT’s null result on this task is reported in the main text instead).

Figure 2: Left: the single-frame JEPA computational graph this paper argues is not identifiable for velocity — one embedding $z_{t}=E(o_{t})$ per frame, with no coordinate reserved for $\dot{q}$ (Proposition 1), feeding a predictor that still only outputs a single next-frame embedding $\hat{z}_{t+1}$ . Right: TI-JEPA. The pose encoder $E_{q}$ is a single shared network (one set of weights, drawn twice for the two things it is applied to): it turns each frame of a $k{=}3$ window $(o_{t-2},o_{t-1},o_{t})$ into a pose code, and $q_{t}$ is simply that window encoding’s last-frame output, with SIGReg applied to $q$ alone (annotated on the arrow, not a separate computational layer). Consecutive pose differences $\Delta q$ pass through a bias-free MLP to give the motion code $v_{t}$ , so a frozen input forces $\Delta q\equiv 0$ and hence $v_{t}\equiv 0$ by construction, the mechanism spelled out below. The memoryless predictor consumes only $(q_{t},v_{t},a_{t})$ — no extra history window — and outputs $(\hat{q}_{t+1},\hat{v}_{t+1})$ , both supervised by the loss.

env	target	single $z_{t}$	pair	window $k{=}3$	GT-pos confound	shuffled floor
PushT	agent vel.	$-0.0006$	$-0.0013$	$-0.0018$	$+0.0011$	$\approx-0.001$
PushT	block vel.	$-0.0017$	$-0.0023$	$-0.0026$	$\mathbf{+0.0774}$	$\approx-0.075$
Reacher	joint vel.	$-0.0016$	$-0.0026$	$-0.0038$	$+0.0009$	$\approx-0.002$
Reacher	finger vel.	$-0.0014$	$-0.0022$	$-0.0029$	$+0.0086$	$\approx-0.008$
Cube	joint vel.	$-0.0021$	$-0.0028$	$-0.0032$	$+0.0021$	$\approx-0.003$
Cube	block vel.	$-0.0026$	$-0.0033$	$-0.0040$	$\mathbf{+0.0301}$	$\approx-0.025$

Table 1: Protocol A on official LeWM checkpoints. Every embedding-based velocity probe sits at or below chance, including the three-frame window that matches the official predictor’s history size. Single-frame position decoding on the same embeddings reaches

R^{2}\approx 0.94

–

0.98

, showing the pipeline works and the null is specific to velocity.

Figure 3: The construction behind every kill experiment in this paper, on real rollout frames from all four environments we build or train on (InertiaBall, Pendulum, CartPole, and real dm_control Reacher). Each pair of rows shares the identical starting frame ( $t{=}0$ pixel difference $0.0000$ ) and only the sign of the initial velocity differs; the two branches are indistinguishable at $t{=}0$ and separate steadily over the rest of the rollout under the environment’s real dynamics. This is the visual fact every branch-separation number in the paper is measuring.

	Protocol A: probe $r$			Protocol B: branch-sep ratio
env	baseline $z_{t}$	TI-JEPA $q_{t}$	TI-JEPA $v_{t}$	baseline k=1	baseline k=3	TI-JEPA	sign acc.
InertiaBall	$-0.025$	$-0.038$	$\mathbf{+0.571}$	$\mathbf{0.000}$	$0.894$	$0.662$	$0.995$
Pendulum	$-0.009$	$-0.019$	$\mathbf{+0.879}$	$\mathbf{0.000}$	$1.117$	$1.071$	$0.625$
CartPole	$+0.111$	$+0.008$	$+0.302$	$\mathbf{0.000}$	$0.208$	$1.152$	$0.835$

Table 2: Protocol A (linear probe

r

for velocity) and Protocol B (branch-separation ratio to ground truth), three physically grounded environments. TI-JEPA’s explicit motion code is the only single-frame-adjacent channel with a consistently positive probe signal, and the memoryless baseline’s branch separation is exactly zero in every environment, as Corollary 1 guarantees.

Figure 4: Kill experiment on Pendulum and CartPole side by side. Top two rows of each panel: the real rollout under opposite-velocity branches, pixel-identical at $t{=}0$ and visibly diverging afterward. Bottom: decoded position for the blind rollout of each arm, one shared legend. The memoryless baseline (red, flat by construction) cannot separate the two branches on either environment; TI-JEPA (blue) tracks the true divergence, matching the history baseline (orange) without reading extra context at inference. Versions with the recurrent-aggregator arm added, and the InertiaBall panel, are in Appendix E .

env (mode)	oracle	baseline k=1	TI-JEPA	vs. baseline k=1
InertiaBall (open)	$0.009$	$0.321$	$\mathbf{0.271}$	16% closer, $p{=}5.3{\times}10^{-4}$
InertiaBall (closed)	$0.068$	$0.230$	$\mathbf{0.178}$	23% closer, $p{=}5.3{\times}10^{-7}$
Pendulum (open)	$0.112$	$1.390$	$\mathbf{0.434}$	69% closer, $p{=}2.7{\times}10^{-16}$
Pendulum (closed)	$0.211$	$0.647$	$\mathbf{0.292}$	55% closer, $p{=}3.2{\times}10^{-10}$
CartPole (open)	$0.046$	$0.904$	$\mathbf{0.624}$	31% closer, $p{=}8.9{\times}10^{-6}$
CartPole (closed)	$0.116$	$0.490$	$\mathbf{0.178}$	64% closer, $p{=}5.1{\times}10^{-15}$

Table 3: Protocol C, final distance to goal, lower is better. All

p

-values are two-sided paired Wilcoxon signed-rank tests on matched tasks and seeds.

env	metric	baseline k=1	baseline k=3	TI-JEPA
CartPole	explicit $v_{t}$ probe $r$	n/a	n/a	$\mathbf{0.910}$ (small-CNN: $0.302$ )
CartPole	branch-sep ratio	$\mathbf{0.000}$	$0.080$	$\mathbf{0.205}$
CartPole	rollout position MSE	$0.190$	$2.662$	$\mathbf{0.144}$
Pendulum	explicit $v_{t}$ probe $r$	n/a	n/a	$0.832$ (small-CNN: $0.879$ )
Pendulum	branch-sep ratio	$\mathbf{0.000}$	$1.601$	$1.172$
Pendulum	rollout position MSE	$0.598$	$\mathbf{0.391}$	$0.576$

Table 4: Official-scale results, CartPole and Pendulum. The memoryless baseline’s branch separation is exactly zero in both environments, a theory-mandated guarantee that reproduces at a completely different architecture scale.

metric	InertiaBall	Pendulum	CartPole
branch-sep: baseline k=3	$0.894$	$1.117$	$0.208$
branch-sep: baseline k=1	$0.000$	$0.000$	$0.000$
branch-sep: TI-JEPA	$0.662$	$1.071$	$\mathbf{1.152}$
branch-sep: recurrent aggregator	$0.624$	$\mathbf{1.071}$	$0.689$
closed-loop success: baseline	$0.055$	$0.410$	$0.110$
closed-loop success: TI-JEPA	$0.050$	$0.270$	$\mathbf{0.200}$

Table 5: Recurrent-aggregator ablation, four arms on each environment.

metric	baseline k=1	baseline k=3	TI-JEPA
branch-separation ratio	$\mathbf{0.000}$	$0.057$	$\mathbf{2.166}$
rollout position MSE	$0.404$	$\mathbf{0.041}$	$0.230$
velocity-sign accuracy (chance $0.5$ )	$0.500$	$0.494$	$0.525$

Table 6: Protocol B, live dm_control environment,

n{=}80

pairs, horizon 8 model steps.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Control	Result
$k=1$ forced, $v\equiv 0$ by construction	Representation collapses: prediction loss, motion loss, and SIGReg all freeze at exact fixed values from step $\approx 100$ onward, reproduced bit-for-bit at an independent seed.
SIGReg applied to the whole $z=(q,v)$ jointly, instead of $q$ alone plus a separate variance floor on $v$	Collapses to the same qualitative degenerate fixed point as above.
Wider and deeper CNN backbone, CartPole only, four channel-widths 32-64-64-64 widened to five channel-widths 64-128-128-128-128, roughly $13\times$ more backbone parameters, matched steps and data	$v_{t}$ probe $r$ : $0.302\to 0.298$ , flat if anything marginally lower. Kill-experiment sign accuracy drops from $0.835$ to $0.525$ , near chance level, despite healthy training (rising $v_{\mathrm{std}}$ , falling prediction loss).
Larger $q\_dim/v\_dim$ ( $4\to 8$ ), CartPole only	$v_{t}$ probe $r$ : $0.302\to 0.311$ , also flat. Two independent capacity axes both failing to move the number is stronger evidence than either alone that the ceiling is not simply a parameter-count issue.
Real, interactive PushT gym, default damping, real contact physics, expert-trajectory goals, closed-loop MPC, two seeds	Every arm, including the baseline, recovers only 1–4px of the expert’s $\approx$ 120px displacement, success rate $\approx 0\%$ , pairwise Wilcoxon tests not significant. A null result, not evidence against TI-JEPA: at this compute and data scale no arm can meaningfully move the block.
Shorter frameskip (1 instead of 3) on real Reacher, CartPole-scale recipe, 220 episodes	Tests whether shorter dt recovers the qvel probe (Appendix F ). It does not, and pose decoding degrades too, a confound rather than a clean answer.

Appendix

Table 7: Negative controls and ablations. Each row is a deliberate attempt to break TI-JEPA or to test an alternative explanation for its results.

combination	TI-JEPA $v_{t}$ $r$	baseline k=1 br.-sep	baseline k=3 br.-sep	TI-JEPA br.-sep
small CNN, CartPole	$0.302$	$0.000$	$0.208$	$1.152$
official, CartPole	$0.910$	$0.000$	$0.080$	$0.205$
small CNN, Pendulum	$0.879$	$0.000$	$1.117$	$1.071$
official, Pendulum	$0.832$	$0.000$	$1.601$	$1.172$

Appendix

Table 8: Every architecture-and-environment combination tested for the memoryless baseline’s branch separation. It is exactly zero in every case, never small but nonzero, matching Corollary 1’s guarantee exactly.

env	baseline single-frame	recurrent single-frame	recurrent pair $r$	recurrent window $r$
InertiaBall	$\approx 0$	$-0.037$	$0.72$ – $0.81$	$0.85$ – $0.90$
Pendulum	$\approx 0$	$-0.008$	$0.85$ – $0.91$	$0.93$ – $0.96$
CartPole	$\approx 0$	$+0.037$	$0.74$ – $0.79$	$0.83$ – $0.88$

Appendix

Table 9: Protocol A, single-frame probe

r

, four arms per environment. The recurrent aggregator’s single-frame code is statistically indistinguishable from chance, exactly like every other single-frame-target encoder in the paper.

	branch-sep ratio			closed-loop success
env	base k3	TI-J.	recur. (s0/s2)	base k3	TI-J.	recur. (s0/s2)
InertiaBall	$0.894$	$0.662$	$0.624$ / $0.826$	$0.055$	$0.050$	$0.070$ / $0.090$
Pendulum	$1.117$	$1.071$	$1.071$ / $1.176$	$0.410$	$0.270$	$0.520$ / $0.370$
CartPole	$0.208$	$\mathbf{1.152}$	$0.689$ / $\mathbf{0.091}$	$0.110$	$\mathbf{0.200}$	$0.120$ / $0.090$

Appendix

Table 10: Recurrent-aggregator branch-separation ratio and closed-loop success, two independent training seeds. Seed 0 is the run reported in Table 5 ; seed 2 is the replication. baseline k=3 and TI-JEPA columns are identical across seeds because those checkpoints are not retrained, confirming the evaluation itself is deterministic given fixed checkpoints.

env	oracle	base k3	TI-JEPA	recur. (seed 0)	recur. (seed 2)
InertiaBall	$0.275$	$0.234$	$0.276$	$\mathbf{0.209}$	$0.215$
Pendulum	$0.081$	$0.153$	$0.175$	$\mathbf{0.134}$	$0.188$
CartPole	$0.106$	$0.240$	$\mathbf{0.175}$	$0.237$	$0.235$

Appendix

Table 11: Closed-loop MPC mean final distance to goal (lower is better), same runs as Table 10 . oracle, baseline k=3 , and TI-JEPA are one fixed run each (the checkpoints are not retrained across seeds); only the recurrent-aggregator column changes between seed 0 and seed 2.

env	metric	seed 0	seed 2
CartPole	explicit $v_{t}$ probe $r$	$0.910$	$0.911$
CartPole	branch-sep: base k1 /base k3 /TI-J.	$0.000$ / $0.080$ / $0.205$	$0.000$ / $0.318$ / $0.443$
CartPole	rollout MSE: base k1 /base k3 /TI-J.	$0.190$ / $2.662$ / $0.144$	$0.232$ / $0.253$ / $0.220$
CartPole	sign accuracy (chance $0.5$ )	$0.213$	$0.227$
Pendulum	explicit $v_{t}$ probe $r$	$0.832$	$0.819$
Pendulum	branch-sep: base k1 /base k3 /TI-J.	$0.000$ / $1.601$ / $1.172$	$0.000$ / $3.608$ / $1.895$

Appendix

Table 12: Official-scale results, two independent training seeds. baseline k=1 ’s branch separation is exactly zero in all four seed/environment combinations, the theory-mandated guarantee holding at official scale regardless of training seed.

Figure 5: Kill experiment on InertiaBall, small-CNN recipe, all four arms (including the recurrent-aggregator baseline). Top two rows: the real rollout under the two opposite-velocity branches, pixel-identical at $t=0$ and visibly diverging afterward. Bottom: decoded position for every arm’s blind rollout. The memoryless baseline (red) is flat by construction; TI-JEPA (blue) and the recurrent baseline (purple) both track the true divergence, the history baseline (orange) also tracks it but only by reading extra context at inference.

Figure 6: Kill experiment on Pendulum, small-CNN recipe, with the recurrent-aggregator arm added to the simplified version shown in the main text (Figure 4 ). TI-JEPA, the history baseline, and the recurrent aggregator all track the ground-truth swing reasonably well here; only TI-JEPA does so without reading extra history or an opaque recurrent state at inference, the comparison that matters for the memory-matched planning result in Table 3 .

Figure 7: Kill experiment on CartPole, small-CNN recipe, all four arms. Same layout as Figure 5 . The recurrent baseline (purple) sits between the memoryless baseline and TI-JEPA, matching Table 5 ’s quantitative branch-separation gap.

Figure 8: Kill experiment on CartPole at official ViT-Tiny plus AdaLN scale. The memoryless baseline again fails to separate the two velocity branches. TI-JEPA’s branch-separation ratio is the highest of the three arms at this scale, and its rollout position error is the lowest, beating even the history baseline, a stronger qualitative result than the small-CNN CartPole finding.

Figure 9: Kill experiment on Pendulum at official ViT-Tiny plus AdaLN scale. The two branches shown here happen to have very different swing amplitudes; both non-memoryless arms overshoot the ground-truth separation curve, most plausibly reflecting compounding autoregressive rollout error on a genuinely non-trivial pendulum trajectory rather than better velocity use.

Figure 10: Full version of the real dm_control Reacher construction in Figure 3 (main text), official scale, with the decoded blind-rollout curve overlaid below the frame strip. TI-JEPA’s curve tracks and then exceeds the ground-truth branch-separation curve; both baselines stay close to zero for the whole horizon.

Figure 11: Protocol C across all three environments. TI-JEPA closes a substantial fraction of the gap between the memoryless baseline and an oracle that reads true velocity, in both the open-loop and closed-loop planning regimes.

Figure 12: Switching from a small CNN to the official ViT-Tiny plus AdaLN-transformer architecture substantially raises CartPole’s explicit velocity probe, while Pendulum’s, already strong, stays roughly flat.

Figure 13: The wider and deeper CNN backbone ablation on CartPole. Training is healthy (rising $v_{\mathrm{std}}$ , falling prediction loss) but the kill-experiment sign accuracy drops to chance, a real training pathology distinct from representation collapse.

Figure 14: InertiaBall: four independent same- $q_{0}$ , opposite-velocity examples, eight frames each. The free point mass visibly separates within two to three frames in every example.

Figure 15: Pendulum: four independent same- $q_{0}$ , opposite-velocity examples, eight frames each. Swing direction and amplitude both differ visibly across the two branches in every example, including cases (example 4) where one branch swings through a large arc while the other stays comparatively still.

Figure 16: CartPole: four independent same- $q_{0}$ , opposite-velocity examples, eight frames each. The coupled cart-pole dynamics produce a slower, more subtle divergence than InertiaBall or Pendulum, consistent with CartPole’s smaller branch-separation ratios in Table 2 .

Figure 17: Real dm_control Reacher: four independent same- $q_{0}$ , opposite-velocity examples, eight frames each, rendered by the actual MuJoCo simulator used to collect the official-scale training data. Every example shows the two-link arm swinging into a visibly different configuration by the last frame despite starting from an identical photograph.

World Model Learning

World model learning builds predictive models of how environments change under actions to support planning, simulation, and decision-making.

See all 399 papers

UniJEPA: A Unified Joint-Embedding Predictive Architecture for Task-Agnostic Visual World Modeling
Aug 7, 2026An Lanji, Dawei Liu, Jin Li +3World Model Learning Joint-Embedding Predictive Architecture
University of Electronic Science and Technology of China, Chengdu, China.
LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
Mar 13, 2026Lucas Maes, Quentin Le Lidec, Damien Scieur +2World Model Learning Joint-Embedding Predictive Architecture
Mila & Université de Montréal · New York University · Samsung SAIL +1
Temporal-Distance JEPA: Plan-Aware Representation Learning for Latent World Model Predictive Control
Jul 28, 2026Jiaxin Bai, Jiaxuan XiongWorld Model-Based Planning Joint-Embedding Predictive Architecture
Hong Kong Baptist University Hong Kong, China

Aug 7, 2026J/K move · Enter open · S save

UniJEPA: A Unified Joint-Embedding Predictive Architecture for Task-Agnostic Visual World Modeling

An Lanji, Dawei Liu, Jin Li, Haoran Xu+2

University of Electronic Science and Technology of China, Chengdu, China.

World Model Learning Joint-Embedding Predictive Architecture Model-Based Planning Predictive Representation Learning Self-Supervised Visual Representation Learning Video World Models

Open paper AskPDF arXiv

Joint-Embedding Predictive Architectures (JEPAs) have emerged as a principled framework for self-supervised learning of world models in compact latent spaces, yet existing methods are fragmented: some predict masked parts of a single image in latent space (I-JEPA), others learn to predict global photometric transformations (Image World Models), while video-scale JEPAs predict future temporal states and are post-trained for action-conditioned planning (V-JEPA~2, DINO-World, DINO-WM). These objectives are treated as distinct recipes with separate encoders, predictors, and anti-collapse regularizers, hindering a single model from unifying image-level and video-level world modeling. We present UniJEPA, a unified JEPA that jointly learns photometric prediction (image-level transformations) and temporal prediction (video-level next-state dynamics) in one shared latent space. A single end-to-end objective, composed of a next-embedding prediction loss and a Gaussian regularizer, yields a provably anti-collapse encoder-predictor pair trainable from raw pixels without EMA, stop-gradient, or pre-trained encoders. We show that the same latent space supports controllable abstraction: photometric prediction learns invariant structure while temporal prediction learns equivariant dynamics. After action-conditioned post-training on offline trajectories, UniJEPA enables zero-shot planning by treating goal features as prediction targets. On image, video, and control benchmarks, UniJEPA matches or surpasses task-specific JEPAs while requiring a single loss hyperparameter, and plans up to tens of times faster than generative world models at comparable accuracy.

Abstract

World Model Learning

Explore similar work

UniJEPA: A Unified Joint-Embedding Predictive Architecture for Task-Agnostic Visual World Modeling

LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels

Temporal-Distance JEPA: Plan-Aware Representation Learning for Latent World Model Predictive Control