LRC-JEPA: Disentangling Dynamics and Residual Context for Efficient World Models

Luzhe Huang; Lei Chu; Jingyi Liang; Yuhuan Zhao

cs.LGSep 28, 2026

LRC-JEPA: Disentangling Dynamics and Residual Context for Efficient World Models

Authors: Luzhe Huang, Lei Chu, Jingyi Liang, Yuhuan Zhao

Organizations: Independent researcher

Abstract

Compact JEPA world models enable efficient latent-space planning, but low-dimensional representation trained under reward-free self-supervision must encode both action-conditioned dynamics and predictable visual context. This competition can entangle controllable state with high-rank nuisance appearance and degrade planning as scenes become more complex. We introduce LRC-JEPA, a lightweight end-to-end world model that routes information into a compact predictive latent $\mathbf{z}$ and learned-query residual-context embeddings $\mathbf{u}$ . Only $\mathbf{z}$ is propagated by the dynamics model and used for planning, while $\mathbf{u}$ captures temporally persistent information for cross-attention reconstruction; a differentiable residual connection encourages the latent to retain complementary dynamic content. Under explicit assumptions, we show that the resulting representation is sufficient, minimal, nuisance-invariant, and disentangled. Across four simulated control environments, LRC-JEPA improves average planning success over a parameter-matched JEPA baseline by 9 percentage points and matches or exceeds substantially larger pretrained models. On the real-world Bridge-v2 set, its 5.5M-parameter active encoder outperforms DINO-WM (22.1M) and V-JEPA2 (303.9M) encoders while also enabling faster planning. Physical-state probes, reconstruction interventions, and ablations confirm the effectiveness of LRC-JEPA's representation disentanglement.

Figures & tables

Figure 1: Zero-shot capacity–complexity stress tests for compact JEPA planning. Left: Starting from the original $192$ -dimensional predictive latent, PCA is fitted on clean representations and latents are projected onto top- $k$ components during planning. Right: In TwoRoom, a trajectory-persistent grayscale pattern is blended into both current and goal observations with opacity $\alpha$ , where $\alpha=0$ denotes the clean environment. Details are provided in Appendix A.2 .

Figure 2: LRC-JEPA architecture. During training, functional information routing separates a compact, dynamics-predictive latent $z_{t}$ from an auxiliary representation $u_{t}$ that supports reconstruction through cross-attention. This separation is the key novelty of our method: it concentrates control-relevant dynamics in $z$ , enabling efficient $z$ -only planning with CEM and latent-distance objectives at test time.

Figure 3: CEM planning success rates on four simulated environments. Values are averaged over three repetitions of 50 evaluation scenes.

Figure 4: Expert action recall and ranking percentile of world models by CEM planning on Bridge-v2 offline dataset. Results averaged on 3 repetitions of 50 distinct scenes.

Model	LRC-JEPA	LWM	V-JEPA2-GAP-AC	DINO-WM	EB-JEPA
Num. of params.	5.5M	5.5M	303.9M	22.1M	1.1M
Full planning time (s)	0.45	0.48	0.83	57.56	0.20

Table 1: The number of active parameters of each world model encoder and the average full planning time over 50 different trajectories.

Quantity	Linear			MLP
	LWM	LRC-JEPA		LWM	LRC-JEPA
	$\mathbf{z}$	$\mathbf{z}$	$\mathbf{u}$	$\mathbf{z}$	$\mathbf{z}$	$\mathbf{u}$
Joint pos.	0.293 / 0.698	0.274 / 0.723	0.572 / 0.500	0.311 / 0.692	0.305 / 0.696	0.444 / 0.653
Joint vel.	0.969 / 0.111	0.734 / 0.392	0.990 / 0.039	0.999 / 0.068	0.893 / 0.312	1.013 / 0.026
EE pos.	0.019 / 0.990	0.011 / 0.995	0.380 / 0.685	0.009 / 0.995	0.007 / 0.997	0.144 / 0.925
EE yaw	0.989 / 0.073	0.930 / 0.250	0.876 / 0.336	1.009 / 0.100	1.027 / 0.128	0.897 / 0.321

Table 2: Physical quantity probes on Cube. Each entry reports MSE (

\downarrow

) / Pearson correlation

r

(

\uparrow

). Bold marks the better predictive latent between LWM and LRC-JEPA; underlining marks the better representation between LRC-JEPA’s

\mathbf{z}

and

\mathbf{u}

, separately for each metric and probe type. EE: end-effector; quat.: quaternion.

Figure 5: Reconstruction interventions on Cube environment. “Shuffled” denotes within-trajectory permutation, while “batch-shuffled” denotes replacement using context codes from other trajectories. Metrics calculated on the shown sample.

Figure 6: Effect of the number of residual-context queries $Q$ on Bridge-v2. We report expert-action recall, linear probes on $\mathbf{z}$ , and reconstruction MSE, PSNR, and SSIM.

Model	Push-T $\uparrow$	Bridge recall@1/5/10 $\uparrow$	Bridge rank pct. $\downarrow$
LRC-JEPA	96.0	0.38/0.64/0.72	0.061
w/ stop-gradient	92.0	0.10/0.28/0.32	0.286

Table 3: Effect of blocking reconstruction gradients at

\mathbf{z}

. Push-T reports planning success in percent. Bridge-v2 reports expert-action recall and rank percentile (pct.).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Quantity	Model	$z$ Linear		$z$ MLP
Quantity	Model	MSE $\downarrow$	$r\uparrow$	MSE $\downarrow$	$r\uparrow$
Agent position	LRC-JEPA	0.066	0.967	0.022	0.989
Agent position	LWM	0.044	0.978	0.018	0.991
Block position	LRC-JEPA	0.021	0.989	0.005	0.997
Block position	LWM	0.021	0.990	0.006	0.997
Block angle	LRC-JEPA	0.175	0.905	0.049	0.974

Appendix

Table 4: Physical quantity probes on the simulated Push-T environment comparing LRC-JEPA and LWM. Each entry reports the mean-squared error (MSE; lower is better) and the Pearson correlation coefficient (

r

; higher is better). Both probe heads are fitted using the latent representation

z

Quantity	Model	$z$ Linear		$z$ MLP
Quantity	Model	MSE $\downarrow$	$r\uparrow$	MSE $\downarrow$	$r\uparrow$
Agent position	LRC-JEPA	0.002	0.999	0.000	1.000
Agent position	LWM	0.008	0.996	0.001	1.000

Appendix

Table 5: Physical quantity probes on the simulated TwoRoom environment comparing LRC-JEPA and LWM. Each entry reports the mean-squared error (MSE; lower is better) and the Pearson correlation coefficient (

r

; higher is better). Both probe heads are fitted using the latent representation

z

Quantity	Linear	MLP
Joint position	0.272 / 0.731	0.306 / 0.711
Joint velocity	0.746 / 0.397	1.002 / 0.088
End-effector position	0.009 / 0.996	0.019 / 0.991
End-effector yaw	0.891 / 0.323	0.923 / 0.299
Gripper	0.058 / 0.970	0.060 / 0.969
Block position	0.004 / 0.998	0.011 / 0.995

Appendix

Table 6: Physical quantity probes using the concatenated

\bm{z}\oplus\bm{u}

representation of LRC-JEPA on the Cube environment. Each entry reports MSE / Pearson correlation

r

(lower / higher is better). Here,

\oplus

denotes concatenation.

	$z$		$u$
Probe target	Linear	MLP	Linear	MLP
Gripper centroid	0.724 / 0.690	0.604 / 0.735	1.193 / 0.321	0.858 / 0.402
Moving-object centroid	1.001 / 0.484	0.879 / 0.473	1.528 / 0.225	0.995 / 0.323
Moving-object RGB	2.440 / 0.131	1.141 / 0.269	1.914 / 0.208	1.179 / 0.265
Static-object centroid	1.841 / 0.252	1.277 / 0.259	1.508 / 0.246	1.239 / 0.267
Static-object RGB	2.324 / 0.036	1.208 / 0.326	2.035 / 0.137	0.999 / 0.413

Appendix

Table 7: Object-centric physical quantity probes using

\bm{z}

and mean-pooled (

Q=2

)

\bm{u}

on Bridge-v2. Each entry reports MSE / Pearson correlation

r

. One valid moving, static, and background object is selected per frame. Bold marks the better representation for each probe type and metric.

Probe target	Linear	MLP
Gripper centroid	10.829 / 0.299	0.703 / 0.690
Moving-object centroid	15.260 / 0.144	0.888 / 0.460
Moving-object RGB	13.374 / 0.165	1.277 / 0.258
Static-object centroid	13.174 / $-0.018$	1.219 / 0.289
Static-object RGB	11.721 / 0.204	1.200 / 0.347
Background centroid	4.750 / 0.156	0.598 / 0.571

Appendix

Table 8: Object-centric physical quantity probes using the concatenated

\bm{z}\oplus\bm{u}

representation on Bridge-v2. Each entry reports MSE / Pearson correlation

r

(lower / higher is better) for linear and MLP probes. Here,

\bm{u}

is mean-pooled (

Q=2

) before concatenation. One valid moving, static, and background object is selected per frame.

Figure 7: Push-T reconstructions. “Shuffled” permutes frames within a trajectory; “batch-shuffled” exchanges context across trajectories.

Figure 8: TwoRoom reconstructions under the same within-trajectory and across-trajectory interventions.

Explore similar work

Jun 30, 2026cs.AI

Delta-JEPA: Learning Action-Sensitive World Models via Latent Difference Decoding

Learning visual world models for planning requires compact latent dynamics that remain sensitive to actions, yet reconstruction-free joint-embedding objectives can collapse to action-insensitive representations. We propose Delta-JEPA, an end-to-end reconstruction-free world model that augments latent forward prediction with a Latent Difference Action Decoder (LDAD). Unlike inverse decoders that infer actions from concatenated endpoint embeddings, LDAD reconstructs the executed action from the latent displacement between consecutive observations. This displacement-level supervision directly regularizes transition geometry: adjacent embeddings cannot collapse without losing action information, and different actions are encouraged to induce distinguishable latent changes for rollout-based planning. Delta-JEPA uses only latent prediction and action reconstruction, avoiding pixel reconstruction and distribution-matching regularizers. Across four visual continuous-control tasks, Delta-JEPA improves planning over JEPA-based and representation-learning world model baselines. Ablations show that displacement-based action decoding is consistently more effective than endpoint concatenation, and action-sensitivity analyses show clearer action-conditioned latent responses. These results indicate that supervising latent differences is a simple and effective mechanism for collapse-resistant and action-sensitive world model learning.

Zhenghao Zhang, Yuanxiang Wang, Zhenyu Guan +11

School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing · Institute of Information Engineering, Chinese Academy of Sciences, Beijing · School of Computer Science and Technology, Harbin Institute of Technology, Weihai +2

Latent World Models Video Joint Embedding Predictive Architecture S-Jepa

Ask PDF

Aug 6, 2026cs.CV

PhyLatent: Learning Dynamics-Relevant Representations for JEPA World Models

We propose PhyLatent, a dynamics-relevant training objective for Joint-Embedding Predictive Architecture (JEPA) world models. Our key observation is that preventing global latent collapse does not necessarily ensure that the learned representation preserves physically meaningful state and action relationships. We identify three failure modes: sensitivity to appearance changes that leave the physical state unchanged, insufficient separation of distinct physical states, and insufficient separation of different action-conditioned futures. We refer to these as Physical Invariance Collapse, Physical Distinguishability Collapse, and Counterfactual Dynamics Collapse, respectively. PhyLatent targets these failures through three coordinated training pathways, implemented with static visual invariance, physical state grounding, future representation alignment, counterfactual branch separation, and latent denoising. On OGBench-Cube, PhyLatent reduces the three collapse rates by 43.9%, 27.9%, and 47.1%, respectively, while improving model predictive control (MPC) success by 12.0 percentage points (17.2% relative). Across four visual-control tasks, average planning success increases by 6.62 percentage points (8.3% relative). These results show that global non-collapse alone is insufficient for learning a reliable JEPA world-model state space, and that explicitly preserving dynamics-relevant structure can improve closed-loop planning.

Xi Zeng, Haojie Ren, Ziying Song +2

School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore · The University of Sheﬃeld, Sheﬃeld, United Kingdom · School of Artiﬁcial Intelligence (School of Software), Yanshan University, Qinhuangdao, China

Latent World Models Joint-Embedding Predictive Architectures World Models Inference Dynamics Latent Variable

Ask PDF

Jun 30, 2026cs.LG

AdaJEPA: An Adaptive Latent World Model

Latent world models enable planning from high-dimensional observations by predicting future states in a compact latent space. However, these models are typically kept frozen at test time: when their predictions become inaccurate, planning can fail, especially under test-time distribution shift. To address this, we propose AdaJEPA, an adaptive latent world model that performs test-time adaptation within the closed loop of model predictive control (MPC). After training, AdaJEPA plans and executes the first action chunk, uses the observed next-state transition as a self-supervised adaptation signal, and replans with the updated model. This closed-loop update continuously recalibrates the world model without additional expert demonstrations. Across a range of goal-reaching tasks, AdaJEPA substantially improves planning success with as few as one gradient step per MPC replanning step.

Ying Wang, Oumayma Bounou, Yann LeCun +1

New York University · AMI Labs

Latent World Models Model Predictive Control World Models World Test-Time Adaptation

Ask PDF