Probing the Latent World: Emergent Discrete Symbols and Physical Structure in Latent Representations

Liu hung ming

cs.LGMar 20, 2026

Probing the Latent World: Emergent Discrete Symbols and Physical Structure in Latent Representations

Authors: Liu hung ming

Organizations: PARRAWA AI

Abstract

Video world models trained with Joint Embedding Predictive Architectures (JEPA) acquire rich spatiotemporal representations by predicting masked regions in latent space rather than reconstructing pixels. This removes the visual verification pathway of generative models, creating a structural interpretability gap: the encoder has learned physical structure inaccessible in any inspectable form. Existing probing methods either operate in continuous space without a structured intermediate layer, or attach generative components whose parameters confound attribution of behavior to the encoder. We propose the AI Mother Tongue (AIM) framework as a passive quantization probe: a lightweight, vocabulary-free probe that converts V-JEPA 2 continuous latent vectors into discrete symbol sequences without task-specific supervision or modifying the encoder. Because the encoder is kept completely frozen, any symbolic structure in the AIM codebook is attributable entirely to V-JEPA 2 pre-trained representations -- not to the probe. We evaluate through category-contrast experiments on Kinetics-mini along three physical dimensions: grasp angle, object geometry, and motion temporal structure. AIM symbol distributions differ significantly across all three experiments (chi^2 p < 10^{-4}; MI 0.036--0.117 bits, NMI 1.2--3.9% of the 3-bit maximum; JSD up to 0.342; codebook active ratio 62.5%). The experiments reveal that V-JEPA 2 latent space is markedly compact: diverse action categories share a common representational core, with semantic differences encoded as graded distributional variations rather than categorical boundaries. These results establish Stage 1 of a four-stage roadmap toward an action-conditioned symbolic world model, demonstrating that structured symbolic manifolds are discoverable properties of frozen JEPA latent spaces.

Figures & tables

Panel of Figure 1 — Figure 1: Stage 1 pipeline overview. Top row (A): The input pipeline feeds 50 Kinetics-mini videos through the frozen V-JEPA 2 ViT-L encoder ( $\nabla\theta=0$ ), producing $B\times 1568\times 1024$ latent token vectors that are precomputed once and cached. Middle row (A): The Stage A quantizer trains for 3,000 steps on the precomputed vectors using a linear projection layer ( $1024\to 256$ dimensions, followed by LayerNorm and L2 normalization), an EMA-updated codebook ( $K=8$ , $\gamma=0.90$ ), and a dead-code reset mechanism ( $\beta=2.0$ commitment loss); the V-JEPA 2 encoder weights remain fully frozen throughout. Bottom row (B/C): After training, the pipeline is evaluated on two diagnostic criteria: H1 symbol stability (20 repeated forward passes per video; consistency $=100\%$ ) and H2 category-contrast experiments (three physical-variable pairs assessed via $\chi^{2}$ , MI, and JSD), with a Gaussian-noise random baseline confirming MI $\approx 0$ for unstructured inputs. All four pass criteria are satisfied, producing the Stage 1 report and AIM dictionary that certify readiness for Stage 2.

$\gamma$ (EMA decay)	$\beta$ (commitment)	Outcome
0.99	0.25	Codebook collapse; active ratio $<5\%$
0.95	1.0	Collapse; perplexity continuously decreasing
0.90	2.0	Stabilized; active ratio $=62.5\%$ ; all criteria passed

Table 1: Ablation of EMA decay rate and commitment loss coefficient. The ablation covers only three configurations due to computational constraints;

\beta=2.0

was selected as the only configuration achieving stable codebook utilization, and is treated as a diagnostic finding rather than a universally optimal hyperparameter.

Figure 2: Stage A quantizer training curves. Left : commitment loss as a function of training iteration, showing rapid convergence within the first 200 steps. Right : codebook perplexity trajectory; the red dashed line marks the health threshold ( $0.4\times K=3.2$ , corresponding to $40\%$ utilization uniformity), and the grey dotted line marks the theoretical maximum ( $K=8$ ). The converged perplexity of $4.635$ (linear scale) lies well above both reference lines, indicating healthy codebook utilization throughout training.

Intervention	Condition pair		MI (bits)	NMI (%)	$\chi^{2}$ $p$ -value	JSD	MI Ratio	Pass
grasp_angle	archery	bowling	$0.036$	$1.2$	$1.19\times 10^{-4}$	$0.190$	$3.6\times 10^{6}$	✓
object_geometry	flying_kite	high_jump	$0.036$	$1.2$	$1.19\times 10^{-4}$	$0.190$	$3.6\times 10^{6}$	✓
motion_speed	marching	archery	$0.117$	$3.9$	$<10^{-10}$	$0.343$	$1.17\times 10^{7}$	✓
Random baseline	—	—	$\approx 0$	$\approx 0$	—	—	—	—

Table 2: H2 category-contrast experiment results. All three experiments pass the significance threshold (

p<0.01

). Baseline MI

<10^{-7}\approx 0

(Gaussian noise input). Normalized MI is defined as

\mathrm{NMI}=\mathrm{MI}/\log_{2}K

, where

\log_{2}8=3

bits is the theoretical maximum symbol entropy. H1 is a pipeline integrity check under deterministic execution; it confirms the absence of residual stochasticity but does not constitute independent evidence of semantic symbol validity.

Figure 3: Intervention results for grasp_angle ( archery vs. bowling ). Left : symbol distribution over the top-20 codebook entries; archery concentrates entirely on entry #5 while bowling shows a secondary mass on entry #4. Centre : pairwise JSD heatmap ( $\mathrm{JSD}=0.19$ ). Right : symbol sensitivity plot; red bars indicate entries included in the condition–symbol mapping.

Figure 4: Intervention results for object_geometry ( flying_kite vs. high_jump ). Left : flying_kite exhibits a small secondary mass on entry #4 absent in high_jump . Centre : pairwise JSD heatmap ( $\mathrm{JSD}=0.19$ ). Right : symbol sensitivity plot; entries #4 and #5 are the active mapping symbols.

Figure 5: Intervention results for motion_speed ( marching vs. archery ). Left : marching distributes mass across entries #5, #4, and #3, while archery concentrates entirely on #5, reflecting the contrast between periodic gait and quasi-static release dynamics. Centre : pairwise JSD heatmap ( $\mathrm{JSD}=0.34$ ), the largest of the three interventions. Right : symbol sensitivity plot; entry #5 shows the highest sensitivity, with #4 and #3 also active (red/blue bars).

Criterion	Result	Threshold	Decision
H1 symbol stability ( $\bar{\rho}$ )	$1.000$	$>0.95$	✓
H2 $\chi^{2}$ $p$ — grasp_angle	$1.19\times 10^{-4}$	$<0.01$	✓
H2 $\chi^{2}$ $p$ — object_geometry	$1.19\times 10^{-4}$	$<0.01$	✓
H2 $\chi^{2}$ $p$ — motion_speed	$<10^{-10}$	$<0.01$	✓
H2 MI Ratio (minimum across interventions)	$3.6\times 10^{6}$	$>5.0$	✓
Codebook active ratio	$62.5\%$ ( $5/8$ )	$>30\%$	✓

Table 3: Stage 1 pass criteria and results. All six criteria are satisfied. Note that H1 symbol stability (

\bar{\rho}=1.000

) is a mathematical consequence of the deterministic pipeline configuration rather than an empirical finding; its role is to serve as a prerequisite integrity check confirming the absence of residual stochasticity, not as primary evidence of semantic symbol validity. The latter is provided by the H2 chi-squared tests and MI ratios.

Figure 6: Stage 1 diagnosis summary. Left : MI ratio (experiment / baseline) for each intervention on a linear scale ( $\times 10^{7}$ ); the red dashed line marks the threshold of $5.0$ . All three interventions exceed the threshold by factors of $10^{6}$ – $10^{7}$ . Centre : chi-squared significance expressed as $-\log_{10}(p)$ ; the red dashed line marks the $p=0.01$ threshold ( $-\log_{10}(0.01)=2$ ). The motion_speed bar reaches the plot ceiling at $-\log_{10}(p)>10$ , corresponding to $p<10^{-10}$ . Right : pass/fail summary for all four diagnostic criteria ( h1_consistency , h2_chi2_p , h2_mi_ratio , codebook_active ); all bars reach the PASS level. The figure footer confirms readiness for Stage 2.

Representation Learning

Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.

See all 798 papers

ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model
Mar 23, 2026Haichao Zhang, Yijiang Li, Shwai He +5Latent World Models Trajectory Prediction
Northeastern University · University of California San Diego · University of Maryland +2
World Model Interpretability: From Decoding Representations to Understanding Latent Worlds
Oct 18, 2026Jize Wang, Yongcheng Jing, Xinyi Le +3World Models Latent World Models
School of Automation and Intelligent Sensing, Shanghai Jiao Tong University · College of Computing and Data Science, Nanyang Technological University
Subspace-Decomposed JEPAs: Disentangling Progression and Content in Latent World Models
May 29, 2026Lucas Thil, Jesse Read, Rim Kaddah +1Disentangled Representation Learning Representation Learning
LIX, École Polytechnique Palaiseau, France · IRT SystemX Palaiseau, France · Safran Tech Chateaufort, France

Mar 23, 2026J/K move · Enter open · S save

ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model

Haichao Zhang, Yijiang Li, Shwai He, Tushar Nagarajan+4

Northeastern University · University of California San Diego · University of Maryland+2

Latent World Models Trajectory Prediction Joint-Embedding Predictive Architecture VLMs for Robotics

Open paper AskPDF arXiv

Recent progress in latent world models (e.g., V-JEPA2) has shown promising capability in forecasting future world states from video observations. Nevertheless, dense prediction from a short observation window limits temporal context and can bias predictors toward local, low-level extrapolation, making it difficult to capture long-horizon semantics and reducing downstream utility. Vision--language models (VLMs), in contrast, provide strong semantic grounding and general knowledge by reasoning over uniformly sampled frames, but they are not ideal as standalone dense predictors due to compute-driven sparse sampling, a language-output bottleneck that compresses fine-grained interaction states into text-oriented representations, and a data-regime mismatch when adapting to small action-conditioned datasets. We propose a VLM-guided JEPA-style latent world modeling framework that combines dense-frame dynamics modeling with long-horizon semantic guidance via a dual-temporal pathway: a dense JEPA branch for fine-grained motion and interaction cues, and a uniformly sampled VLM thinker branch with a larger temporal stride for knowledge-rich guidance. To transfer the VLM's progressive reasoning signals effectively, we introduce a hierarchical pyramid representation extraction module that aggregates multi-layer VLM representations into guidance features compatible with latent prediction. Experiments on hand-manipulation trajectory prediction show that our method outperforms both a strong VLM-only baseline and a JEPA-predictor baseline, and yields more robust long-horizon rollout behavior.

Abstract

Representation Learning

Explore similar work

ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model

World Model Interpretability: From Decoding Representations to Understanding Latent Worlds

Subspace-Decomposed JEPAs: Disentangling Progression and Content in Latent World Models