Probing the Latent World: Emergent Discrete Symbols and Physical Structure in Latent Representations

Liu hung ming

cs.LGMar 20, 2026

Probing the Latent World: Emergent Discrete Symbols and Physical Structure in Latent Representations

Authors: Liu hung ming

Organizations: PARRAWA AI

Abstract

Video world models trained with Joint Embedding Predictive Architectures (JEPA) acquire rich spatiotemporal representations by predicting masked regions in latent space rather than reconstructing pixels. This removes the visual verification pathway of generative models, creating a structural interpretability gap: the encoder has learned physical structure inaccessible in any inspectable form. Existing probing methods either operate in continuous space without a structured intermediate layer, or attach generative components whose parameters confound attribution of behavior to the encoder. We propose the AI Mother Tongue (AIM) framework as a passive quantization probe: a lightweight, vocabulary-free probe that converts V-JEPA 2 continuous latent vectors into discrete symbol sequences without task-specific supervision or modifying the encoder. Because the encoder is kept completely frozen, any symbolic structure in the AIM codebook is attributable entirely to V-JEPA 2 pre-trained representations -- not to the probe. We evaluate through category-contrast experiments on Kinetics-mini along three physical dimensions: grasp angle, object geometry, and motion temporal structure. AIM symbol distributions differ significantly across all three experiments (chi^2 p < 10^{-4}; MI 0.036--0.117 bits, NMI 1.2--3.9% of the 3-bit maximum; JSD up to 0.342; codebook active ratio 62.5%). The experiments reveal that V-JEPA 2 latent space is markedly compact: diverse action categories share a common representational core, with semantic differences encoded as graded distributional variations rather than categorical boundaries. These results establish Stage 1 of a four-stage roadmap toward an action-conditioned symbolic world model, demonstrating that structured symbolic manifolds are discoverable properties of frozen JEPA latent spaces.

Figures & tables

Panel of Figure 1 — Figure 1: Stage 1 pipeline overview. Top row (A): The input pipeline feeds 50 Kinetics-mini videos through the frozen V-JEPA 2 ViT-L encoder ( $\nabla\theta=0$ ), producing $B\times 1568\times 1024$ latent token vectors that are precomputed once and cached. Middle row (A): The Stage A quantizer trains for 3,000 steps on the precomputed vectors using a linear projection layer ( $1024\to 256$ dimensions, followed by LayerNorm and L2 normalization), an EMA-updated codebook ( $K=8$ , $\gamma=0.90$ ), and a dead-code reset mechanism ( $\beta=2.0$ commitment loss); the V-JEPA 2 encoder weights remain fully frozen throughout. Bottom row (B/C): After training, the pipeline is evaluated on two diagnostic criteria: H1 symbol stability (20 repeated forward passes per video; consistency $=100\%$ ) and H2 category-contrast experiments (three physical-variable pairs assessed via $\chi^{2}$ , MI, and JSD), with a Gaussian-noise random baseline confirming MI $\approx 0$ for unstructured inputs. All four pass criteria are satisfied, producing the Stage 1 report and AIM dictionary that certify readiness for Stage 2.

$\gamma$ (EMA decay)	$\beta$ (commitment)	Outcome
0.99	0.25	Codebook collapse; active ratio $<5\%$
0.95	1.0	Collapse; perplexity continuously decreasing
0.90	2.0	Stabilized; active ratio $=62.5\%$ ; all criteria passed

Table 1: Ablation of EMA decay rate and commitment loss coefficient. The ablation covers only three configurations due to computational constraints;

\beta=2.0

was selected as the only configuration achieving stable codebook utilization, and is treated as a diagnostic finding rather than a universally optimal hyperparameter.

Figure 2: Stage A quantizer training curves. Left : commitment loss as a function of training iteration, showing rapid convergence within the first 200 steps. Right : codebook perplexity trajectory; the red dashed line marks the health threshold ( $0.4\times K=3.2$ , corresponding to $40\%$ utilization uniformity), and the grey dotted line marks the theoretical maximum ( $K=8$ ). The converged perplexity of $4.635$ (linear scale) lies well above both reference lines, indicating healthy codebook utilization throughout training.

Intervention	Condition pair		MI (bits)	NMI (%)	$\chi^{2}$ $p$ -value	JSD	MI Ratio	Pass
grasp_angle	archery	bowling	$0.036$	$1.2$	$1.19\times 10^{-4}$	$0.190$	$3.6\times 10^{6}$	✓
object_geometry	flying_kite	high_jump	$0.036$	$1.2$	$1.19\times 10^{-4}$	$0.190$	$3.6\times 10^{6}$	✓
motion_speed	marching	archery	$0.117$	$3.9$	$<10^{-10}$	$0.343$	$1.17\times 10^{7}$	✓
Random baseline	—	—	$\approx 0$	$\approx 0$	—	—	—	—

Table 2: H2 category-contrast experiment results. All three experiments pass the significance threshold (

p<0.01

). Baseline MI

<10^{-7}\approx 0

(Gaussian noise input). Normalized MI is defined as

\mathrm{NMI}=\mathrm{MI}/\log_{2}K

, where

\log_{2}8=3

bits is the theoretical maximum symbol entropy. H1 is a pipeline integrity check under deterministic execution; it confirms the absence of residual stochasticity but does not constitute independent evidence of semantic symbol validity.

Figure 3: Intervention results for grasp_angle ( archery vs. bowling ). Left : symbol distribution over the top-20 codebook entries; archery concentrates entirely on entry #5 while bowling shows a secondary mass on entry #4. Centre : pairwise JSD heatmap ( $\mathrm{JSD}=0.19$ ). Right : symbol sensitivity plot; red bars indicate entries included in the condition–symbol mapping.

Figure 4: Intervention results for object_geometry ( flying_kite vs. high_jump ). Left : flying_kite exhibits a small secondary mass on entry #4 absent in high_jump . Centre : pairwise JSD heatmap ( $\mathrm{JSD}=0.19$ ). Right : symbol sensitivity plot; entries #4 and #5 are the active mapping symbols.

Figure 5: Intervention results for motion_speed ( marching vs. archery ). Left : marching distributes mass across entries #5, #4, and #3, while archery concentrates entirely on #5, reflecting the contrast between periodic gait and quasi-static release dynamics. Centre : pairwise JSD heatmap ( $\mathrm{JSD}=0.34$ ), the largest of the three interventions. Right : symbol sensitivity plot; entry #5 shows the highest sensitivity, with #4 and #3 also active (red/blue bars).

Criterion	Result	Threshold	Decision
H1 symbol stability ( $\bar{\rho}$ )	$1.000$	$>0.95$	✓
H2 $\chi^{2}$ $p$ — grasp_angle	$1.19\times 10^{-4}$	$<0.01$	✓
H2 $\chi^{2}$ $p$ — object_geometry	$1.19\times 10^{-4}$	$<0.01$	✓
H2 $\chi^{2}$ $p$ — motion_speed	$<10^{-10}$	$<0.01$	✓
H2 MI Ratio (minimum across interventions)	$3.6\times 10^{6}$	$>5.0$	✓
Codebook active ratio	$62.5\%$ ( $5/8$ )	$>30\%$	✓

Table 3: Stage 1 pass criteria and results. All six criteria are satisfied. Note that H1 symbol stability (

\bar{\rho}=1.000

) is a mathematical consequence of the deterministic pipeline configuration rather than an empirical finding; its role is to serve as a prerequisite integrity check confirming the absence of residual stochasticity, not as primary evidence of semantic symbol validity. The latter is provided by the H2 chi-squared tests and MI ratios.

Figure 6: Stage 1 diagnosis summary. Left : MI ratio (experiment / baseline) for each intervention on a linear scale ( $\times 10^{7}$ ); the red dashed line marks the threshold of $5.0$ . All three interventions exceed the threshold by factors of $10^{6}$ – $10^{7}$ . Centre : chi-squared significance expressed as $-\log_{10}(p)$ ; the red dashed line marks the $p=0.01$ threshold ( $-\log_{10}(0.01)=2$ ). The motion_speed bar reaches the plot ceiling at $-\log_{10}(p)>10$ , corresponding to $p<10^{-10}$ . Right : pass/fail summary for all four diagnostic criteria ( h1_consistency , h2_chi2_p , h2_mi_ratio , codebook_active ); all bars reach the PASS level. The figure footer confirms readiness for Stage 2.

Representation Learning

Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.

See all 799 papers

Mar 23, 2026

ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model

Recent progress in latent world models (e.g., V-JEPA2) has shown promising capability in forecasting future world states from video observations. Nevertheless, dense prediction from a short observation window limits temporal context and can bias predictors toward local, low-level extrapolation, making it difficult to capture long-horizon semantics and reducing downstream utility. Vision--language models (VLMs), in contrast, provide strong semantic grounding and general knowledge by reasoning over uniformly sampled frames, but they are not ideal as standalone dense predictors due to compute-driven sparse sampling, a language-output bottleneck that compresses fine-grained interaction states into text-oriented representations, and a data-regime mismatch when adapting to small action-conditioned datasets. We propose a VLM-guided JEPA-style latent world modeling framework that combines dense-frame dynamics modeling with long-horizon semantic guidance via a dual-temporal pathway: a dense JEPA branch for fine-grained motion and interaction cues, and a uniformly sampled VLM thinker branch with a larger temporal stride for knowledge-rich guidance. To transfer the VLM's progressive reasoning signals effectively, we introduce a hierarchical pyramid representation extraction module that aggregates multi-layer VLM representations into guidance features compatible with latent prediction. Experiments on hand-manipulation trajectory prediction show that our method outperforms both a strong VLM-only baseline and a JEPA-predictor baseline, and yields more robust long-horizon rollout behavior.

Haichao Zhang, Yijiang Li, Shwai He +5

Northeastern University · University of California San Diego · University of Maryland +2

Latent World Models Trajectory Prediction Joint-Embedding Predictive Architecture VLMs for Robotics

Ask PDF

Oct 18, 2026

World Model Interpretability: From Decoding Representations to Understanding Latent Worlds

Emerging Joint-Embedding Predictive Architectures (JEPAs) are moving beyond representation learning towards latent world models that support prediction, planning, and control. Yet their latent worlds naturally lack a human-interpretable interface. In this Perspective, we argue that as JEPA latents become internal states for prediction and action, interpretability must move beyond what they encode to how latent variables interact, evolve, and causally drive prediction. To this end, we envision a four-stage research agenda, involving Ground, Structure, Verify, and Act, to guide this shift toward actionable latent-world understanding. Within our framework, existing JEPA interpretability efforts provide early evidence for Ground, while systematic analysis of latent structure, causal verification, and intervention remains largely unexplored. Overall, this Perspective aims to call for a broader latent-world interpretability agenda that shifts attention from decoding representations toward understanding, validating, and acting on latent worlds.

Jize Wang, Yongcheng Jing, Xinyi Le +3

School of Automation and Intelligent Sensing, Shanghai Jiao Tong University · College of Computing and Data Science, Nanyang Technological University

World Models Latent World Models Joint-Embedding Predictive Architecture Causal Interpretability

Ask PDF

May 29, 2026

Subspace-Decomposed JEPAs: Disentangling Progression and Content in Latent World Models

World models used for planning learn to predict how a scene evolves, yet they rarely hold a global view of how far a task has progressed. We argue that such a view should be part of the representation, and that it can be compact: describing a scene takes many dimensions, but where the scene stands on its way to the goal can be summarised by only a few. We propose SD-JEPA, a joint-embedding predictive world model whose latent is split into a small progression subspace and a large content subspace. A temporal triplet loss keeps consecutive frames of a trajectory close in the progression coordinates and separates different trajectories, while a distributional regulariser keeps the content coordinates from collapsing. On four control benchmarks, the model plans as well as the same architecture without the split. In return, its latent becomes easier to read. A few percent of its dimensions are enough to predict progress within an episode, and the angle they form follows the task, moves backward when the agent backtracks and changes fastest around events such as grasps and releases. Surprise also becomes anchored in the trajectory: when the scene changes mid-episode, the angle spikes and then relocalizes on a new part of the progression, anchoring surprise in the agent's own map of the task.

Lucas Thil, Jesse Read, Rim Kaddah +1

LIX, École Polytechnique Palaiseau, France · IRT SystemX Palaiseau, France · Safran Tech Chateaufort, France

Disentangled Representation Learning Representation Learning Joint-Embedding Predictive Architecture Latent World Models

Ask PDF

Abstract

Representation Learning

Explore similar work

ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model

World Model Interpretability: From Decoding Representations to Understanding Latent Worlds

Subspace-Decomposed JEPAs: Disentangling Progression and Content in Latent World Models