Your Probabilistic JEPA Is Secretly a Hidden Markov Model: A State-Space Interpretation of Joint-Embedding Predictive Learning

Yongchao Huang

cs.AIAug 13, 2026

Your Probabilistic JEPA Is Secretly a Hidden Markov Model: A State-Space Interpretation of Joint-Embedding Predictive Learning

Authors: Yongchao Huang

Organizations: Department of Computing Science University of Aberdeen

Abstract

A hidden Markov model (HMM) combines three roles: inference of a hidden-state belief from observations, propagation through a Markov transition, and emission back to observation space. We show that full, time-indexed Predictive Information Bottleneck VJEPA (PIB-VJEPA) exposes the same computational structure: a stochastic context encoder plays the role of an amortized filtering distribution, a probabilistic predictor defines latent-state dynamics, and a decoder, inverse target encoder, or induced implicit conditional supplies the emission direction. We distinguish 4 progressively stronger levels of correspondence and give sufficient conditions for exact sequence-level HMM equivalence. To make the connection concrete, we introduce Markov-Chain JEPA (MCJEPA), which replaces the latent predictor by a learned transition matrix; in the finite time-homogeneous case, matrix powers guarantee exact multi-horizon Chapman--Kolmogorov consistency. Conditioned discrete-state transitions, continuous-state Markov kernels, and continuous-time dynamics extend this construction, while deterministic temporal JEPA appears as a degenerate Dirac-kernel special case. We further interpret predictive information-bottleneck learning as seeking a compact predictive state: compression promotes minimality, while residual predictability tests sufficiency. Controlled experiments support transition composition, the filtering interpretation, predictive Markovization in a known synthetic process, and the distinction between JEPA latent prediction and HMM-style sequence learning. Together, these results give temporal JEPA a principled state-space interpretation.

Figures & tables

Panel of Figure 1 — Figure 1: Unified graphical and computational comparison of an HMM and full PIB-VJEPA. Top: the classical HMM view, in which an unobserved Markov chain emits observations. Bottom: the corresponding PIB-VJEPA view, in which stochastic encoders infer predictive latent states from observation-level data and a probabilistic predictor propagates those states. Solid downward arrows in the PIB-VJEPA diagram denote observation-to-state encoding, while dashed upward arrows denote the optional state-to-observation map implemented by an explicit decoder or an inverse target encoder. The boxed rows reproduce the aligned computational pipelines in Eqs. 1 and 2 .

Component	Hidden Markov model	PIB-VJEPA
Observation	$X_{t}$	Observation-level input $X_{t}$
Hidden state	$S_{t}$	Predictive latent state $Z_{t}$
State inference	$p(S_{t}\mid X_{\leq t})$	Online encoder $q_{\theta}(Z_{t}\mid X_{\leq t})$
Future-state training target	Posterior or inferred belief over $S_{t+1}$	EMA target encoder $q_{\bar{\theta}}(Z_{t+1}\mid X_{t+1})$
State transition	$p(S_{t+1}\mid S_{t})$	Predictor $p_{\phi}(Z_{t+1}\mid Z_{t},\xi_{t})$
Emission or observation map	$p(X_{t}\mid S_{t})$	Decoder $p_{\psi}(X_{t}\mid Z_{t})$ , inverse target encoder $f_{\bar{\theta}}^{-1}$ , or induced implicit emission

Table 1: Component-wise analogy between an HMM and a full PIB-VJEPA.

Role	Hidden Markov model	Full PIB-VJEPA
Observed variable	Observation $X_{t}$ generated by an emission model	Observation-level input $X_{t}$ , such as a frame, pixel array, or time-series measurement
State inference	Filtering distribution $p(S_{t}\mid X_{\leq t})$	Stochastic context encoder $q_{\theta}(Z_{t}\mid X_{\leq t})$
Hidden state	Latent state $S_{t}$	Predictive latent representation $Z_{t}$
Future-state reference	Posterior or inferred belief over $S_{t+1}$	EMA target encoder $q_{\bar{\theta}}(Z_{t+1}\mid X_{t+1})$
State transition	$p(S_{t+1}\mid S_{t})$ or transition matrix $A$	Probabilistic predictor $p_{\phi}(Z_{t+1}\mid Z_{t},\xi_{t})$
Emission direction	$p(X_{t+1}\mid S_{t+1})$ or emission matrix $B$	Decoder $p_{\psi}(X_{t+1}\mid Z_{t+1})$ , inverse target encoder $f_{\bar{\theta}}^{-1}$ , or induced implicit emission

Table 2: Direct correspondence between an HMM and full PIB-VJEPA. Observation-level inputs correspond to HMM observations, the stochastic context encoder plays the hidden-state inference role, the predictor defines the latent transition, and an explicit decoder, inverse target encoder, or induced implicit conditional can supply the emission direction.

Panel of Figure 2 — Figure 2: Markov-Chain JEPA. The context encoder infers a predictive-state distribution, the transition matrix propagates it to future latent states, and training matches the prediction to an EMA target distribution. The dashed observation branch is optional: a decoder or inverse target encoder can map the predicted latent state back to observation space when observation forecasting or an explicit emission model is required.

Model	Time	State	Transition
MCJEPA	Discrete	Discrete	Fixed matrix $A$
Conditioned MCJEPA	Discrete	Discrete	$A_{\phi}(\xi_{t})$
Continuous-state Markov JEPA	Discrete	Continuous	Kernel $p_{\phi}(Z_{t+1}\mid Z_{t},\xi_{t})$
Deterministic temporal JEPA	Discrete	Typically continuous	Dirac kernel at $g_{\phi}(Z_{t},\xi_{t})$
Continuous-time MCJEPA	Continuous	Discrete	Generator $Q_{\phi}$ and $\exp(\Delta tQ_{\phi})$
Continuous-time stochastic JEPA	Continuous	Continuous	Latent SDE or continuous-time kernel

Table 3: Representative hierarchy of latent Markov transitions for temporal JEPA. The main development focuses on discrete-time models; continuous-time variants situate the transition-matrix formulation within the broader state-space family.

Exp.	Structural question	Main evidence
1	Does a shared Markov transition recover coherent finite-state dynamics?	State and transition recovery, multi-horizon prediction, path consistency, collapse ablations
2	When is history-based filtering necessary?	Local versus filtered state accuracy, NLL, and posterior trajectories under emission ambiguity
3	Can predictive compression construct a compact sufficient state?	Exact information trade-off, deterministic-partition frontier, learned compression path, residual predictability
4	Can the same architecture be trained by an HMM sequence objective?	Sequence NLL, transition recovery, latent-state recovery, and filtering agreement

Table 4: Overview of the experimental questions. All experiments use controlled synthetic processes for which the relevant latent structure is known.

Regime	Model	ARI $\uparrow$	$\mathcal{E}_{A}\downarrow$	$\mathcal{L}_{1}\downarrow$	$\mathcal{L}_{8}\downarrow$
Separated	Gaussian HMM	$\mathbf{\underline{0.9947\pm 0.0006}}$	$\mathbf{\underline{0.0082\pm 0.0020}}$	$\mathbf{\underline{0.5240\pm 0.0026}}$	$\mathbf{\underline{1.3027\pm 0.0050}}$
	Horizon-specific $A_{h}$	$0.9836\pm 0.0034$	$0.0231\pm 0.0018$	$0.5433\pm 0.0019$	$1.3036\pm 0.0058$
	MCJEPA shared $A$	$0.9796\pm 0.0055$	$0.0191\pm 0.0019$	$0.5438\pm 0.0018$	$1.3048\pm 0.0049$
Ambiguous	Gaussian HMM	$\mathbf{\underline{0.7543\pm 0.0115}}$	$\mathbf{\underline{0.0091\pm 0.0027}}$	$\mathbf{\underline{0.6676\pm 0.0103}}$	$\mathbf{\underline{1.3127\pm 0.0060}}$
	Horizon-specific $A_{h}$	$0.5581\pm 0.0476$	$0.2410\pm 0.0029$	$0.8488\pm 0.0306$	$1.3288\pm 0.0041$
	MCJEPA shared $A$	$0.5323\pm 0.0513$	$0.0901\pm 0.0091$	$0.8500\pm 0.0349$	$1.3491\pm 0.0093$

Table 5: Finite-HMM recovery. Values are mean

\pm

standard deviation over 5 seeds.

\mathcal{E}_{A}

is the permutation-aligned transition-matrix error;

\mathcal{L}_{1}

and

\mathcal{L}_{8}

are true-state prediction NLL at horizons

1

and

8

Regularization	ARI $\uparrow$	$K_{\mathrm{eff}}^{\mathrm{hard}}$	Assignment entropy
Both	$\mathbf{0.9748\pm 0.0134}$	$3.9551\pm 0.0084$	$0.0716\pm 0.0987$
Occupancy only	$0.6582\pm 0.0736$	$3.8092\pm 0.2249$	$1.1037\pm 0.1216$
Entropy only	$0.0776\pm 0.1734$	$1.1703\pm 0.3807$	$0.0015\pm 0.0034$
None	$0.5626\pm 0.1612$	$3.4007\pm 0.5026$	$1.1062\pm 0.0837$

Table 6: Collapse ablation in Experiment 1. Values are mean

\pm

standard deviation over 5 seeds.

K_{\mathrm{eff}}^{\mathrm{hard}}

is the effective number of states induced by hard assignments. Assignment entropy measures per-example confidence and should be interpreted jointly with state usage: very low entropy can also arise from single-state collapse. The occupancy and entropy terms are complementary: the former encourages global state use, while the latter encourages confident per-example assignments.

Panel of Figure 3 — Figure 3: Structural diagnostics for Experiment 1. Left: direct-versus-composed prediction disagreement $\mathcal{D}_{\mathrm{path}}$ from Eq. 29 in the ambiguous-emission regime. The labels $1+1$ , $2+2$ , and $4+4$ compare direct predictions at total horizons $2$ , $4$ , and $8$ with predictions obtained by composing two successive predictions of horizons $1$ , $2$ , and $4$ , respectively. MCJEPA is exactly path-consistent because all horizons are generated by powers of one shared transition matrix; independently learned $A_{h}$ are not. Right: visualization of the ARI column of Table 6 under the four state-regularization ablations. Occupancy and entropy regularization are complementary, and using both gives the strongest state recovery. Error bars denote mean $\pm$ one standard deviation over 5 seeds.

Panel of Figure 4 — Figure 4: Filtering under emission ambiguity. Left: comparison of the exact local-evidence and filtering distributions associated with the local and history-dependent encoder roles in MCJEPA. Filtering increasingly outperforms local evidence as emission overlap grows. Error bars denote mean $\pm$ one standard deviation over 5 seeds. Right: a representative ambiguous sequence around a true state transition. The local posterior reacts strongly to individual noisy observations, whereas the filtered belief integrates temporal evidence through the transition model. These are oracle posteriors computed from the known generating process, rather than separately trained HMM and MCJEPA models.

Representation	$\mathrm{I}(H_{t};Z_{t})$	$\mathrm{I}(Z_{t};X_{t+1})$	NLL	States
Full history $H_{t}$	1.7356	0.2807	0.3978	8
Minimal predictive pair	1.3378	0.2807	0.3978	4
Current observation $X_{t}$	0.6785	0.0192	0.6593	2

Table 7: Exact representation controls for Experiment 3. The minimal predictive pair removes redundant history while preserving all one-step predictive information. Compressing further to the current observation alone loses information required for prediction.

Panel of Figure 5 — Figure 5: Predictive compression and Markovization. Left: exhaustive deterministic compression provides a global reference for the prediction–compression trade-off, while the learned stochastic encoder traces a warm-started continuation path as the compression weight $\beta$ is increased. Moderate compression can remove redundant history while preserving predictive information, whereas excessive compression eventually sacrifices information required for prediction. Right: residual history gain $\Delta_{\mathrm{hist}}$ diagnoses predictive insufficiency by testing whether adding the previous representation state $Z_{t-1}$ improves held-out prediction beyond using $Z_{t}$ alone. This augmentation substantially helps the insufficient representation $Z_{t}=X_{t}$ , for which $Z_{t-1}=X_{t-1}$ restores omitted transition-relevant information, but provides essentially no additional predictive benefit for either the minimal sufficient or the overcomplete representation. Error bars denote mean $\pm$ one standard deviation over 5 seeds where applicable.

Training regime	$\mathcal{L}_{\mathrm{HMM}}$	$\mathcal{L}_{\mathrm{MC}}$	$\mathcal{L}_{\mathrm{filter}}$	ARI $\uparrow$	Seq. NLL $\downarrow$	$\mathcal{E}_{A}\downarrow$	Filter KL $\downarrow$
HMM sequence + filter distill	$\checkmark$	–	$\checkmark$	$\mathbf{0.9945\pm 0.0008}$	$\mathbf{1.2659\pm 0.0072}$	$0.0105\pm 0.0014$	$\mathbf{0.00006\pm 0.00001}$
Hybrid HMM + latent	$\checkmark$	$\checkmark$	$\checkmark$	$0.9905\pm 0.0008$	$1.2660\pm 0.0073$	$\mathbf{0.0103\pm 0.0017}$	$0.00436\pm 0.00102$
JEPA latent objective	–	$\checkmark$	–	$0.9802\pm 0.0053$	$1.2753\pm 0.0069$	$0.0191\pm 0.0020$	$0.02546\pm 0.00642$

Table 8: Objective-level comparison in Experiment 4. The three objective columns indicate which training signals are active: HMM sequence likelihood

\mathcal{L}_{\mathrm{HMM}}

, JEPA/Markov latent prediction

\mathcal{L}_{\mathrm{MC}}

, and filtering-posterior distillation

\mathcal{L}_{\mathrm{filter}}

. The state-use regularizer

\mathcal{L}_{\mathrm{state}}

is shared across all three regimes. Values are mean

\pm

standard deviation over five seeds. Boldface marks the numerically best mean and does not imply statistical significance. Filtering KL is training aligned for the HMM+filter and hybrid regimes because both explicitly optimize

\mathcal{L}_{\mathrm{filter}}

; for JEPA-only it is evaluated post hoc.

Panel of Figure 6 — Figure 6: Objective-level comparison in Experiment 4. Left: observation-sequence NLL per time step relative to the HMM sequence + filter-distillation regime. Hybrid training nearly closes the entire JEPA-to-HMM sequence-likelihood gap while retaining the genuine MCJEPA latent-prediction objective. Right: permutation-aligned transition-matrix recovery error. HMM-style and hybrid training achieve closely matched transition recovery, and both substantially outperform latent-only JEPA training. Error bars denote mean $\pm$ one standard deviation over five seeds.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Complete multi-horizon state-prediction results for Experiment 1. Left: separated emissions. Right: ambiguous emissions. Lower true-state NLL is better. Error bars denote mean $\pm$ one standard deviation over five seeds.

Figure 8: Direct-versus-composed prediction disagreement in the separated-emission regime. MCJEPA has zero path disagreement by construction because all horizons are powers of one shared transition matrix.

Panel of Figure 9 — Figure 9: Additional collapse diagnostics for Experiment 1. Left: effective number of hard states. Right: mean per-example assignment entropy. Occupancy regularization primarily promotes broad global state usage, whereas entropy regularization promotes confident assignments; neither diagnostic should be interpreted in isolation.

Figure 10: State NLL for exact local evidence and exact filtering under increasing emission ambiguity. Smaller $\mu/\sigma$ corresponds to stronger overlap between the two Gaussian emissions. Error bars denote mean $\pm$ one standard deviation over five independently generated datasets.

Representation	$I(H;Z)$	$I(Z;Y)$	$H(Y\mid Z)$	States
Three-bit history	1.7356	0.2807	0.3978	8
Predictive pair	1.3378	0.2807	0.3978	4
Current bit only	0.6785	0.0192	0.6593	2

Appendix

Table 9: Exact structural controls for Experiment 3. The four-state predictive pair removes redundant history while preserving all one-step predictive information.

Compression range	$I(H;Z)$	$I(Z;Y)$	Pred. NLL	States
$\beta=0$ (tie-broken)	1.3378	0.2807	0.3978	4
$0<\beta\leq 0.014$	1.3378	0.2807	0.3978	4
$0.015\leq\beta\leq 0.30$	0.6689	0.2711	0.4074	2
$\beta=1$	0	0	0.6785	1

Appendix

Table 10: Exact deterministic compression regimes in Experiment 3. At

\beta=0

, the four-state entry is the lower-information representative selected among predictively tied optima.

Figure 11: Exact deterministic compression frontier for Experiment 3. All $4140$ deterministic partitions of the eight histories are evaluated exactly. The nondominated frontier gives the best achievable deterministic trade-offs between retaining history information and preserving information about $X_{t+1}$ .

Figure 12: Number of occupied hard states as the compression weight $\beta$ increases. The dashed reference gives the exact deterministic optimum, while the learned stochastic continuation follows the warm-started gradient trajectory.

Loss	Context encoder	Transition $A$	Emission / initial-state parameters
$\mathcal{L}_{\mathrm{HMM}}$	–	$\checkmark$	$\checkmark$
$\mathcal{L}_{\mathrm{MC}}$	$\checkmark$	$\checkmark$	–
$\mathcal{L}_{\mathrm{filter}}$	$\checkmark$	–	–
$\mathcal{L}_{\mathrm{state}}$	$\checkmark$	–	–

Appendix

Table 11: Effective parameter updates in the revised Experiment 4 implementation. The target encoder receives no gradient and is updated only by EMA.

Training regime	$\Delta$ sequence NLL / step
HMM sequence + filter distill	$0$
Hybrid HMM + latent	$0.000134\pm 0.000092$
JEPA latent objective	$0.009409\pm 0.003164$

Appendix

Table 12: Paired observation-sequence NLL difference relative to HMM+filter training. Differences are computed within each seed before aggregation. Lower is better.

Training regime	$h=1$	$h=2$	$h=4$	$h=8$
HMM sequence + filter distill	$\mathbf{0.5271\pm 0.0027}$	$\mathbf{0.7983\pm 0.0045}$	$\mathbf{1.0939\pm 0.0054}$	$1.3035\pm 0.0047$
Hybrid HMM + latent	$0.5295\pm 0.0030$	$0.7996\pm 0.0048$	$1.0943\pm 0.0057$	$\mathbf{1.3029\pm 0.0051}$
JEPA latent objective	$0.5437\pm 0.0019$	$0.8106\pm 0.0040$	$1.1004\pm 0.0056$	$1.3049\pm 0.0049$

Appendix

Table 13: True-state multi-horizon prediction NLL in Experiment 4. Values are mean

\pm

standard deviation over five seeds. Lower is better.

Figure 13: Filtering agreement across the three Experiment 4 training regimes. The quantity is $\mathrm{KL}(q_{t}^{\mathrm{exact}}\|q_{t}^{\mathrm{enc}})$ averaged over test time points. HMM+filter and hybrid training explicitly optimize filtering alignment, so their values should be interpreted as training-aligned role diagnostics. JEPA-only is evaluated post hoc using the filtering distribution induced by its fitted observation model.

Joint-Embedding Predictive Architecture

Joint-Embedding Predictive Architecture (JEPA) learns representations by predicting latent features of hidden or future data from observed context.

See all 233 papers

Apr 10, 2026

A Predictive View on Streaming Hidden Markov Models

We develop a predictive-first optimisation framework for streaming hidden Markov models. Unlike classical approaches that prioritise full posterior recovery under a fully specified generative model, we assume access to regime-specific predictive models whose parameters are learned online while maintaining a fixed transition prior over regimes. Our objective is to sequentially identify latent regimes while maintaining accurate step-ahead predictive distributions. Because the number of possible regime paths grows exponentially, exact filtering is infeasible. We therefore formulate streaming inference as a constrained projection problem in predictive-distribution space: under a fixed hypothesis budget, we approximate the full posterior predictive by the forward-KL optimal mixture supported on

S

paths. The solution is the renormalised top-

S

posterior-weighted mixture, providing a principled derivation of beam search for HMMs. The resulting algorithm is fully recursive and deterministic, performing beam-style truncation with closed-form predictive updates and requiring neither EM nor sampling. Empirical comparisons against Online EM and Sequential Monte Carlo under matched computational budgets demonstrate competitive prequential performance.

Gerardo Duran-Martin

Oxford-Man Institute, University of Oxford

Hidden Markov Models Streaming Algorithms Conditional Distribution Estimation

Ask PDF

Aug 4, 2026

SJEPA: Learning Elegant Latent Dynamics with Hybrid Symbolic-Neural Predictors

Joint-embedding predictive architectures learn abstract states by predicting target embeddings from context embeddings, but their transition models are typically opaque neural maps. We introduce SJEPA, a reconstruction-free JEPA framework that learns predictive representations whose induced dynamics admit compact symbolic descriptions. Its hybrid transition combines a symbolic law with a regularised neural correction for dynamics outside the selected grammar. The central principle is to learn the simplest adequate dynamics: representation constraints preserve informative, non-collapsed predictive coordinates, while operator compression favours low-complexity symbolic-neural transitions that remain predictively adequate. We formalise this principle through induced-dynamics complexity, analyse predictive-coordinate non-identifiability, and show that unconstrained operator compression creates a direct shortcut to representation collapse. The framework supports both alternating representation-equation learning and symbolic dynamics fitted to fixed representations. In controlled pendulum experiments, joint learning discovers substantially simpler symbolic dynamics with lower long-horizon rollout error and divergence than post-hoc fitting, while an unconstrained one-step diagnostic realises the predicted collapse shortcut. Under grammar misspecification, correction regularisation preserves the representable symbolic mechanism and directs the neural component towards residual dynamics. The results expose a controllable trade-off among predictive fidelity, representation quality, symbolic parsimony, and symbolic-neural allocation.

Yongchao Huang

Latent Dynamics Modeling Joint-Embedding Predictive Architecture Predictive Representation Learning Neurosymbolic Learning

Ask PDF

Aug 20, 2026

Orthogonal JEPA: Factorized Predictive States for Latent World Models

World models construct latent states that support prediction, planning, and reasoning about an underlying system. Joint-embedding predictive architectures (JEPAs) offer a direct way to learn such states by predicting targets in representation space instead of reconstructing every detail of the observation. Standard JEPAs, however, organize all predictable content through one target embedding and one prediction pathway. In complex systems, this monolithic state can allocate redundant capacity to dominant signals while providing weak or conflicting gradients to less dominant predictive structure. We introduce \method, a latent world-modeling framework based on orthogonal predictive factorization. Learned basis matrices analyze each target state into multiple components, and a dedicated prediction branch estimates each component from a shared context representation. Predictive regression preserves the factor magnitudes required for state synthesis, an orthogonality objective discourages repeated directions, factor-activity regularization maintains variation in projected targets, and online variance regularization discourages coordinate-wise encoder collapse. Predicted components are synthesized into a complete latent state that can be used by a readout, decoder, planner, or autoregressive rollout. The same predictive-state mechanism applies when the target is temporally future, spatially hidden, or another partial observation of the same system. Experiments on controlled vision, single-cell transcriptomics, longitudinal health records, continuous control, and molecular dynamics evaluate representation quality, forecasting, planning, and long-horizon stability.

Taoyong Cui, Pheng Ann Heng, Wanli Ouyang

The Chinese University of Hong Kong (CUHK)

Joint-Embedding Predictive Architecture Orthogonal Representation Learning Predictive Representation Learning Latent World Models

Ask PDF

Abstract

Joint-Embedding Predictive Architecture

Explore similar work

A Predictive View on Streaming Hidden Markov Models

SJEPA: Learning Elegant Latent Dynamics with Hybrid Symbolic-Neural Predictors

Orthogonal JEPA: Factorized Predictive States for Latent World Models