LRC-JEPA: Disentangling Dynamics and Residual Context for Efficient World Models

Luzhe Huang; Lei Chu; Jingyi Liang; Yuhuan Zhao

cs.LGSep 28, 2026

LRC-JEPA: Disentangling Dynamics and Residual Context for Efficient World Models

Authors: Luzhe Huang, Lei Chu, Jingyi Liang, Yuhuan Zhao

Organizations: Independent researcher

Abstract

Compact JEPA world models enable efficient latent-space planning, but low-dimensional representation trained under reward-free self-supervision must encode both action-conditioned dynamics and predictable visual context. This competition can entangle controllable state with high-rank nuisance appearance and degrade planning as scenes become more complex. We introduce LRC-JEPA, a lightweight end-to-end world model that routes information into a compact predictive latent $\mathbf{z}$ and learned-query residual-context embeddings $\mathbf{u}$ . Only $\mathbf{z}$ is propagated by the dynamics model and used for planning, while $\mathbf{u}$ captures temporally persistent information for cross-attention reconstruction; a differentiable residual connection encourages the latent to retain complementary dynamic content. Under explicit assumptions, we show that the resulting representation is sufficient, minimal, nuisance-invariant, and disentangled. Across four simulated control environments, LRC-JEPA improves average planning success over a parameter-matched JEPA baseline by 9 percentage points and matches or exceeds substantially larger pretrained models. On the real-world Bridge-v2 set, its 5.5M-parameter active encoder outperforms DINO-WM (22.1M) and V-JEPA2 (303.9M) encoders while also enabling faster planning. Physical-state probes, reconstruction interventions, and ablations confirm the effectiveness of LRC-JEPA's representation disentanglement.

Figures & tables

Figure 1: Zero-shot capacity–complexity stress tests for compact JEPA planning. Left: Starting from the original $192$ -dimensional predictive latent, PCA is fitted on clean representations and latents are projected onto top- $k$ components during planning. Right: In TwoRoom, a trajectory-persistent grayscale pattern is blended into both current and goal observations with opacity $\alpha$ , where $\alpha=0$ denotes the clean environment. Details are provided in Appendix A.2 .

Figure 2: LRC-JEPA architecture. During training, functional information routing separates a compact, dynamics-predictive latent $z_{t}$ from an auxiliary representation $u_{t}$ that supports reconstruction through cross-attention. This separation is the key novelty of our method: it concentrates control-relevant dynamics in $z$ , enabling efficient $z$ -only planning with CEM and latent-distance objectives at test time.

Figure 3: CEM planning success rates on four simulated environments. Values are averaged over three repetitions of 50 evaluation scenes.

Figure 4: Expert action recall and ranking percentile of world models by CEM planning on Bridge-v2 offline dataset. Results averaged on 3 repetitions of 50 distinct scenes.

Model	LRC-JEPA	LWM	V-JEPA2-GAP-AC	DINO-WM	EB-JEPA
Num. of params.	5.5M	5.5M	303.9M	22.1M	1.1M
Full planning time (s)	0.45	0.48	0.83	57.56	0.20

Table 1: The number of active parameters of each world model encoder and the average full planning time over 50 different trajectories.

Quantity	Linear			MLP
	LWM	LRC-JEPA		LWM	LRC-JEPA
	$\mathbf{z}$	$\mathbf{z}$	$\mathbf{u}$	$\mathbf{z}$	$\mathbf{z}$	$\mathbf{u}$
Joint pos.	0.293 / 0.698	0.274 / 0.723	0.572 / 0.500	0.311 / 0.692	0.305 / 0.696	0.444 / 0.653
Joint vel.	0.969 / 0.111	0.734 / 0.392	0.990 / 0.039	0.999 / 0.068	0.893 / 0.312	1.013 / 0.026
EE pos.	0.019 / 0.990	0.011 / 0.995	0.380 / 0.685	0.009 / 0.995	0.007 / 0.997	0.144 / 0.925
EE yaw	0.989 / 0.073	0.930 / 0.250	0.876 / 0.336	1.009 / 0.100	1.027 / 0.128	0.897 / 0.321

Table 2: Physical quantity probes on Cube. Each entry reports MSE (

\downarrow

) / Pearson correlation

r

(

\uparrow

). Bold marks the better predictive latent between LWM and LRC-JEPA; underlining marks the better representation between LRC-JEPA’s

\mathbf{z}

and

\mathbf{u}

, separately for each metric and probe type. EE: end-effector; quat.: quaternion.

Figure 5: Reconstruction interventions on Cube environment. “Shuffled” denotes within-trajectory permutation, while “batch-shuffled” denotes replacement using context codes from other trajectories. Metrics calculated on the shown sample.

Figure 6: Effect of the number of residual-context queries $Q$ on Bridge-v2. We report expert-action recall, linear probes on $\mathbf{z}$ , and reconstruction MSE, PSNR, and SSIM.

Model	Push-T $\uparrow$	Bridge recall@1/5/10 $\uparrow$	Bridge rank pct. $\downarrow$
LRC-JEPA	96.0	0.38/0.64/0.72	0.061
w/ stop-gradient	92.0	0.10/0.28/0.32	0.286

Table 3: Effect of blocking reconstruction gradients at

\mathbf{z}

. Push-T reports planning success in percent. Bridge-v2 reports expert-action recall and rank percentile (pct.).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Quantity	Model	$z$ Linear		$z$ MLP
Quantity	Model	MSE $\downarrow$	$r\uparrow$	MSE $\downarrow$	$r\uparrow$
Agent position	LRC-JEPA	0.066	0.967	0.022	0.989
Agent position	LWM	0.044	0.978	0.018	0.991
Block position	LRC-JEPA	0.021	0.989	0.005	0.997
Block position	LWM	0.021	0.990	0.006	0.997
Block angle	LRC-JEPA	0.175	0.905	0.049	0.974

Appendix

Table 4: Physical quantity probes on the simulated Push-T environment comparing LRC-JEPA and LWM. Each entry reports the mean-squared error (MSE; lower is better) and the Pearson correlation coefficient (

r

; higher is better). Both probe heads are fitted using the latent representation

z

Quantity	Model	$z$ Linear		$z$ MLP
Quantity	Model	MSE $\downarrow$	$r\uparrow$	MSE $\downarrow$	$r\uparrow$
Agent position	LRC-JEPA	0.002	0.999	0.000	1.000
Agent position	LWM	0.008	0.996	0.001	1.000

Appendix

Table 5: Physical quantity probes on the simulated TwoRoom environment comparing LRC-JEPA and LWM. Each entry reports the mean-squared error (MSE; lower is better) and the Pearson correlation coefficient (

r

; higher is better). Both probe heads are fitted using the latent representation

z

Quantity	Linear	MLP
Joint position	0.272 / 0.731	0.306 / 0.711
Joint velocity	0.746 / 0.397	1.002 / 0.088
End-effector position	0.009 / 0.996	0.019 / 0.991
End-effector yaw	0.891 / 0.323	0.923 / 0.299
Gripper	0.058 / 0.970	0.060 / 0.969
Block position	0.004 / 0.998	0.011 / 0.995

Appendix

Table 6: Physical quantity probes using the concatenated

\bm{z}\oplus\bm{u}

representation of LRC-JEPA on the Cube environment. Each entry reports MSE / Pearson correlation

r

(lower / higher is better). Here,

\oplus

denotes concatenation.

	$z$		$u$
Probe target	Linear	MLP	Linear	MLP
Gripper centroid	0.724 / 0.690	0.604 / 0.735	1.193 / 0.321	0.858 / 0.402
Moving-object centroid	1.001 / 0.484	0.879 / 0.473	1.528 / 0.225	0.995 / 0.323
Moving-object RGB	2.440 / 0.131	1.141 / 0.269	1.914 / 0.208	1.179 / 0.265
Static-object centroid	1.841 / 0.252	1.277 / 0.259	1.508 / 0.246	1.239 / 0.267
Static-object RGB	2.324 / 0.036	1.208 / 0.326	2.035 / 0.137	0.999 / 0.413

Appendix

Table 7: Object-centric physical quantity probes using

\bm{z}

and mean-pooled (

Q=2

)

\bm{u}

on Bridge-v2. Each entry reports MSE / Pearson correlation

r

. One valid moving, static, and background object is selected per frame. Bold marks the better representation for each probe type and metric.

Probe target	Linear	MLP
Gripper centroid	10.829 / 0.299	0.703 / 0.690
Moving-object centroid	15.260 / 0.144	0.888 / 0.460
Moving-object RGB	13.374 / 0.165	1.277 / 0.258
Static-object centroid	13.174 / $-0.018$	1.219 / 0.289
Static-object RGB	11.721 / 0.204	1.200 / 0.347
Background centroid	4.750 / 0.156	0.598 / 0.571

Appendix

Table 8: Object-centric physical quantity probes using the concatenated

\bm{z}\oplus\bm{u}

representation on Bridge-v2. Each entry reports MSE / Pearson correlation

r

(lower / higher is better) for linear and MLP probes. Here,

\bm{u}

is mean-pooled (

Q=2

) before concatenation. One valid moving, static, and background object is selected per frame.

Figure 7: Push-T reconstructions. “Shuffled” permutes frames within a trajectory; “batch-shuffled” exchanges context across trajectories.

Figure 8: TwoRoom reconstructions under the same within-trajectory and across-trajectory interventions.

Explore similar work

Delta-JEPA: Learning Action-Sensitive World Models via Latent Difference Decoding
Jun 30, 2026Zhenghao Zhang, Yuanxiang Wang, Zhenyu Guan +11Latent World Models Video Joint Embedding Predictive Architecture
School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing · Institute of Information Engineering, Chinese Academy of Sciences, Beijing · School of Computer Science and Technology, Harbin Institute of Technology, Weihai +2
PhyLatent: Learning Dynamics-Relevant Representations for JEPA World Models
Aug 6, 2026Xi Zeng, Haojie Ren, Ziying Song +2Latent World Models Joint-Embedding Predictive Architectures
School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore · The University of Sheﬃeld, Sheﬃeld, United Kingdom · School of Artiﬁcial Intelligence (School of Software), Yanshan University, Qinhuangdao, China
AdaJEPA: An Adaptive Latent World Model
Jun 30, 2026Ying Wang, Oumayma Bounou, Yann LeCun +1Latent World Models Model Predictive Control
New York University · AMI Labs

Jun 30, 2026, cs.AIJ/K move · Enter open · S save

Delta-JEPA: Learning Action-Sensitive World Models via Latent Difference Decoding

Zhenghao Zhang, Yuanxiang Wang, Zhenyu Guan, Yujia Yang+10

School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing · Institute of Information Engineering, Chinese Academy of Sciences, Beijing · School of Computer Science and Technology, Harbin Institute of Technology, Weihai+2

Latent World Models Video Joint Embedding Predictive Architecture S-Jepa

Open paper AskPDF arXiv

Learning visual world models for planning requires compact latent dynamics that remain sensitive to actions, yet reconstruction-free joint-embedding objectives can collapse to action-insensitive representations. We propose Delta-JEPA, an end-to-end reconstruction-free world model that augments latent forward prediction with a Latent Difference Action Decoder (LDAD). Unlike inverse decoders that infer actions from concatenated endpoint embeddings, LDAD reconstructs the executed action from the latent displacement between consecutive observations. This displacement-level supervision directly regularizes transition geometry: adjacent embeddings cannot collapse without losing action information, and different actions are encouraged to induce distinguishable latent changes for rollout-based planning. Delta-JEPA uses only latent prediction and action reconstruction, avoiding pixel reconstruction and distribution-matching regularizers. Across four visual continuous-control tasks, Delta-JEPA improves planning over JEPA-based and representation-learning world model baselines. Ablations show that displacement-based action decoding is consistently more effective than endpoint concatenation, and action-sensitivity analyses show clearer action-conditioned latent responses. These results indicate that supervising latent differences is a simple and effective mechanism for collapse-resistant and action-sensitive world model learning.

Abstract

Explore similar work

Delta-JEPA: Learning Action-Sensitive World Models via Latent Difference Decoding

PhyLatent: Learning Dynamics-Relevant Representations for JEPA World Models

AdaJEPA: An Adaptive Latent World Model