Learning intrinsic dynamics---the relationship between actions and their consequences---is essential for world models. However, visual variations unrelated to the underlying control process can obscure this relationship in learned representations. We investigate the connection between action predictability, defined as the ability to reconstruct an action from consecutive latent states, and the preservation of intrinsic dynamics. Our central hypothesis is that encouraging action predictability discourages representations from discarding the distinctions between different actions and their consequences. We provide two complementary forms of evidence for this connection. Theoretically, we derive a positive lower bound on squared inverse prediction error when the encoder merges distinguishable outcomes of different actions from the same state and appearance, and show that forward prediction can still be exact on those transitions. Empirically, controlled appearance-diversity comparisons, supervision interventions, and latent probes support the role of action prediction in preserving control-relevant transition information. We instantiate action predictability with Concat IDM, which predicts actions from concatenated latent states and updates the shared encoder using action labels already available in offline trajectories, without changing the test-time planner. Across three independent training runs on Push-T with 24 appearances, Concat IDM increases planning success from 9.1% to 77.2%; its paired improvement is 52.9 percentage points larger than that observed with a single appearance. Together, these findings support action predictability as an inductive bias toward intrinsic dynamics and demonstrate the practical benefits of Concat IDM for latent planning under visual diversity.
Figures & tables
Figure 1: Visual diversity amplifies the benefit of action prediction. (a) Schematic of appearance variation with physical trajectories held fixed. (b) Under this controlled variation on Push-T, the gain of Concat IDM over Plain JEPA grows from 15.3 pp at K=1 to 68.1 pp at K=24 .
Figure 2: Action-supervised latent world modeling and planning. Eθ : visual encoder; Pϕ : latent predictor; Iψ : inverse dynamics head. (a) CEM ranks predicted rollouts by terminal latent goal cost. (b) Concat IDM predicts actions from concatenated observed latents, supervising the shared encoder during training.
Single appearance ( K=1 )
Multiple appearances
Task
Plain JEPA
Concat IDM
Gain
K
Plain JEPA
Concat IDM
Gain
Push-T
72.2±7.9
87.5±4.4
+15.3
24
9.1±2.7
77.2±0.2
+68.1
TwoRoom
88.5±1.3
91.1±1.0
+2.6
36
53.2±1.4
97.3±3.2
+44.1
Cube
60.9±1.2
71.1±2.2
+10.2
24
60.7±1.2
75.4±1.3
+14.7
Reacher
100.0±0.0
100.0±0.0
0.0
24
100.0±0.0
100.0±0.0
0.0
Table 1: Planning gains across appearance conditions. Gain is Concat IDM minus Plain JEPA, computed from the displayed success means in percentage points (pp). Bold marks the higher success mean, including ties, and positive gains.
Figure 4
Figure 4: Robustness to training appearance assignments on Push-T ( K=24 ). (a) Planning success. (b) Normalized final error. Bars show means with sample SDs; dots denote trained models and lines connect matched runs.
Figure 5: Action information and latent rollout consistency on Push-T. (a) Frozen-encoder action-probe test R2 ; Pair denotes [zt;zt+1] . (b) Spearman correlation between predicted and realized latent costs. Thin lines show individual models; thick lines show means.
λ
Success ↑
NFE ↓
0
9.1±2.7
1.353±0.108
0.1
71.0
0.713
1
77.2±0.2
0.652±0.032
10
85.8
0.610
Table 3: IDM weight and training progress on Push-T ( K=24 ). In (a), n=3 for λ∈{0,1} and n=1 otherwise. In (b), each epoch uses the same three training runs per method. Success is in percent; NFE is lower-is-better.
Learning visual world models for planning requires compact latent dynamics that remain sensitive to actions, yet reconstruction-free joint-embedding objectives can collapse to action-insensitive representations. We propose Delta-JEPA, an end-to-end reconstruction-free world model that augments latent forward prediction with a Latent Difference Action Decoder (LDAD). Unlike inverse decoders that infer actions from concatenated endpoint embeddings, LDAD reconstructs the executed action from the latent displacement between consecutive observations. This displacement-level supervision directly regularizes transition geometry: adjacent embeddings cannot collapse without losing action information, and different actions are encouraged to induce distinguishable latent changes for rollout-based planning. Delta-JEPA uses only latent prediction and action reconstruction, avoiding pixel reconstruction and distribution-matching regularizers. Across four visual continuous-control tasks, Delta-JEPA improves planning over JEPA-based and representation-learning world model baselines. Ablations show that displacement-based action decoding is consistently more effective than endpoint concatenation, and action-sensitivity analyses show clearer action-conditioned latent responses. These results indicate that supervising latent differences is a simple and effective mechanism for collapse-resistant and action-sensitive world model learning.
Zhenghao Zhang, Yuanxiang Wang, Zhenyu Guan +11
School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing · Institute of Information Engineering, Chinese Academy of Sciences, Beijing · School of Computer Science and Technology, Harbin Institute of Technology, Weihai +2
Latent world models are trained to predict future states in a learned representation and are then deployed inside a planner that selects actions by simulating them forward. Current practice adopts the prediction error, the single- or multi-step rollout loss on held-out data, as the training and model-selection objective, on the assumption that a lower prediction error yields better control. We show that this assumption is unreliable for a structural reason: a planner does not query the model on the training distribution but on the states that its candidate actions reach, which generally leave the data manifold, so an error averaged over the data cannot by itself govern control. We therefore reframe the objective as the discrepancy between the predicted and the true plan-cost at the plan the planner commits to, and prove that the planner's suboptimality is bounded by twice this discrepancy, whereas the data-averaged prediction error neither bounds nor tracks it. Under a linear-control premise the discrepancy separates into two terms. The first is a small on-manifold residual, on which the predicted and true dynamics agree and which a spectral tax prices through the non-normality of the latent transition operator. The second is an off-manifold divergence, on which an action carries the state off the manifold and the two dynamics diverge; this divergence is the binding term and is bounded by no data-averaged error. Synthetic operators confirm the pricing formulas, and latent model-predictive control experiments confirm the decoupling: across seeds, the single-step validation error is essentially uncorrelated with control success, whereas a fidelity score on the planner-reachable measure tracks it.
Hanzhe You, Yonggang Zhang, Maohao Ran +6
University of Science and Technology of China · The Hong Kong University of Science and Technology, HKGAI · Hong Kong Baptist University
Compact JEPA world models enable efficient latent-space planning, but low-dimensional representation trained under reward-free self-supervision must encode both action-conditioned dynamics and predictable visual context. This competition can entangle controllable state with high-rank nuisance appearance and degrade planning as scenes become more complex. We introduce LRC-JEPA, a lightweight end-to-end world model that routes information into a compact predictive latent z and learned-query residual-context embeddings u. Only z is propagated by the dynamics model and used for planning, while u captures temporally persistent information for cross-attention reconstruction; a differentiable residual connection encourages the latent to retain complementary dynamic content. Under explicit assumptions, we show that the resulting representation is sufficient, minimal, nuisance-invariant, and disentangled. Across four simulated control environments, LRC-JEPA improves average planning success over a parameter-matched JEPA baseline by 9 percentage points and matches or exceeds substantially larger pretrained models. On the real-world Bridge-v2 set, its 5.5M-parameter active encoder outperforms DINO-WM (22.1M) and V-JEPA2 (303.9M) encoders while also enabling faster planning. Physical-state probes, reconstruction interventions, and ablations confirm the effectiveness of LRC-JEPA's representation disentanglement.