cs.LG · 2603.14131 Copy arXiv ID · Mar 14, 2026 Save Is the reconstruction loss culprit? An attempt to outperform JEPA Authors: Alexey Potapov , Oleg Shcherbakov , Ivan Kravchenko
Organizations: SingularityNET
In Abstract We evaluate JEPA-style predictive representation learning versus reconstruction-based autoencoders on a controlled "TV-series" linear dynamical system with known latent state and a single noise parameter. While an initial comparison suggests JEPA is markedly more robust to noise, further diagnostics show that autoencoder failures are strongly influenced by asymmetries in objectives and by bottleneck/component-selection effects (confirmed by PCA baselines). Motivated by these findings, we introduce gated predictive autoencoders that learn to select predictable components, mimicking the beneficial feature-selection behavior observed in over-parameterized PCA. On this toy testbed, the proposed gated model is stable across noise levels and matches or outperforms JEPA.
More in this topic Predictive representation learning trains models to encode useful structure by predicting unseen or future representations from context.
See all 172 papers → Also in Oct 19, 2026 · Mosh Levy, Aly Lidayan, Yoav Goldberg Joint-Embedding Predictive Architecture Predictive Representation Learning
Bar-Ilan University · University of California, Berkeley · Allen Institute for AI
Oct 1, 2026 · José Lucas De Melo Costa, Seong Woo Ahn, Fabrice Popineau +2 Joint-Embedding Predictive Architecture Representation Collapse
Université Paris-Saclay, CNRS, CentraleSupélec, LISN, Orsay, France
Aug 4, 2026 J/K move · Enter open · S save
Yongchao Huang
Joint-embedding predictive architectures learn abstract states by predicting target embeddings from context embeddings, but their transition models are typically opaque neural maps. We introduce SJEPA, a reconstruction-free JEPA framework that learns predictive representations whose induced dynamics admit compact symbolic descriptions. Its hybrid transition combines a symbolic law with a regularised neural correction for dynamics outside the selected grammar. The central principle is to learn the simplest adequate dynamics: representation constraints preserve informative, non-collapsed predictive coordinates, while operator compression favours low-complexity symbolic-neural transitions that remain predictively adequate. We formalise this principle through induced-dynamics complexity, analyse predictive-coordinate non-identifiability, and show that unconstrained operator compression creates a direct shortcut to representation collapse. The framework supports both alternating representation-equation learning and symbolic dynamics fitted to fixed representations. In controlled pendulum experiments, joint learning discovers substantially simpler symbolic dynamics with lower long-horizon rollout error and divergence than post-hoc fitting, while an unconstrained one-step diagnostic realises the predicted collapse shortcut. Under grammar misspecification, correction regularisation preserves the representable symbolic mechanism and directs the neural component towards residual dynamics. The results expose a controllable trade-off among predictive fidelity, representation quality, symbolic parsimony, and symbolic-neural allocation.