World Model Interpretability: From Decoding Representations to Understanding Latent Worlds

Jize Wang; Yongcheng Jing; Xinyi Le; Cailian Chen; Xinping Guan; Dacheng Tao

cs.LGOct 18, 2026

World Model Interpretability: From Decoding Representations to Understanding Latent Worlds

Authors: Jize Wang, Yongcheng Jing, Xinyi Le, Cailian Chen, Xinping Guan, Dacheng Tao

Organizations: School of Automation and Intelligent Sensing, Shanghai Jiao Tong University · College of Computing and Data Science, Nanyang Technological University

Abstract

Emerging Joint-Embedding Predictive Architectures (JEPAs) are moving beyond representation learning towards latent world models that support prediction, planning, and control. Yet their latent worlds naturally lack a human-interpretable interface. In this Perspective, we argue that as JEPA latents become internal states for prediction and action, interpretability must move beyond what they encode to how latent variables interact, evolve, and causally drive prediction. To this end, we envision a four-stage research agenda, involving Ground, Structure, Verify, and Act, to guide this shift toward actionable latent-world understanding. Within our framework, existing JEPA interpretability efforts provide early evidence for Ground, while systematic analysis of latent structure, causal verification, and intervention remains largely unexplored. Overall, this Perspective aims to call for a broader latent-world interpretability agenda that shifts attention from decoding representations toward understanding, validating, and acting on latent worlds.

Figures & tables

Figure 1: Evolution and positioning of interpretability for latent-space world models. Despite rapid advances in latent-space world modeling and a rich XAI toolbox, direct interpretation of learned world models remains sparse. Existing efforts largely focus on latent-content probing or isolated analysis and intervention techniques, leaving systematic understanding of latent worlds underexplored.

Figure 2: From decoding representations to understanding latent worlds. The proposed blueprint progresses through four stages: Ground connects latent factors to interpretable world variables; Structure recovers their relations and dynamics; Verify tests their causal roles through controlled interventions; and Act uses verified understanding to diagnose failures and improve model behavior.

Figure 3: Cross-domain instantiations of latent worlds. Representative JEPA-based models across robot manipulation, 3D localization, medical care, and scientific discovery illustrate how relevant world variables, relations, and dynamics depend on the application domain. Latent-world interpretation therefore requires grounding representations in domain-specific concepts and interactions rather than assuming a universal set of interpretable features.

Role	Model Scope	Approach	Ground	Structure	Verify	Act
Method	JEPA	V-JEPA 2 Probing	$\circ$	$\circ$	$\times$	$\times$
Method	JEPA	AIM Probing	$\circ$	$\times$	$\times$	$\times$
Infrastructure	JEPA + Non-JEPA	WorldModelLens	$\circ$	$\times$	$\circ$	$\times$
Transferable Method	Non-JEPA	WAM Steering	$\circ$	$\circ$	$\circ$	$\checkmark$

Table 1: Representative work related to latent-world interpretability, mapped onto the proposed Ground, Structure, Verify, Act blueprint. We focus on work that directly analyzes the representations, dynamics, or causal mechanisms of learned world models, with particular emphasis on JEPAs.

\checkmark

explicit coverage;

\circ

partial coverage;

\times

not addressed.

World Models

World models represent and predict how environments change, helping agents plan, act, and reason about possible futures.

See all 363 papers

PhyLatent: Learning Dynamics-Relevant Representations for JEPA World Models
Aug 6, 2026Xi Zeng, Haojie Ren, Ziying Song +2Representation Learning World Model Learning
School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore · The University of Sheﬃeld, Sheﬃeld, United Kingdom · School of Artiﬁcial Intelligence (School of Software), Yanshan University, Qinhuangdao, China
Do-JEPA: From Masking to Intervention in Latent World Models
Sep 29, 2026Hossein Resani, Javen Qinfeng ShiCausal Effect Estimation Causal World Models
Australian Institute for Machine Learning, Adelaide University · Responsible AI Research Centre
Orthogonal JEPA: Factorized Predictive States for Latent World Models
Aug 20, 2026Taoyong Cui, Pheng Ann Heng, Wanli OuyangJoint-Embedding Predictive Architecture Orthogonal Representation Learning
The Chinese University of Hong Kong (CUHK)

Aug 6, 2026J/K move · Enter open · S save

PhyLatent: Learning Dynamics-Relevant Representations for JEPA World Models

Xi Zeng, Haojie Ren, Ziying Song, Yuanbo Nie+1

School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore · The University of Sheﬃeld, Sheﬃeld, United Kingdom · School of Artiﬁcial Intelligence (School of Software), Yanshan University, Qinhuangdao, China

Representation Learning World Model Learning World Model-Based Planning Joint-Embedding Predictive Architecture State Representation Learning

Open paper AskPDF arXiv

We propose PhyLatent, a dynamics-relevant training objective for Joint-Embedding Predictive Architecture (JEPA) world models. Our key observation is that preventing global latent collapse does not necessarily ensure that the learned representation preserves physically meaningful state and action relationships. We identify three failure modes: sensitivity to appearance changes that leave the physical state unchanged, insufficient separation of distinct physical states, and insufficient separation of different action-conditioned futures. We refer to these as Physical Invariance Collapse, Physical Distinguishability Collapse, and Counterfactual Dynamics Collapse, respectively. PhyLatent targets these failures through three coordinated training pathways, implemented with static visual invariance, physical state grounding, future representation alignment, counterfactual branch separation, and latent denoising. On OGBench-Cube, PhyLatent reduces the three collapse rates by 43.9%, 27.9%, and 47.1%, respectively, while improving model predictive control (MPC) success by 12.0 percentage points (17.2% relative). Across four visual-control tasks, average planning success increases by 6.62 percentage points (8.3% relative). These results show that global non-collapse alone is insufficient for learning a reliable JEPA world-model state space, and that explicitly preserving dynamics-relevant structure can improve closed-loop planning.

Abstract

World Models

Explore similar work

PhyLatent: Learning Dynamics-Relevant Representations for JEPA World Models

Do-JEPA: From Masking to Intervention in Latent World Models

Orthogonal JEPA: Factorized Predictive States for Latent World Models