From Alignment to Prediction: A Study of Self-Supervised Learning and Predictive Representation Learning
Authors: Mintu Dutta, Ritesh Vyas, Mohendra Roy
Organizations: Department of Information and Communication Technology, School of Technology, Pandit Deendayal Energy University, Gandhinagar, Gujarat 382007, India
Self-supervised learning has emerged as a major technique for the task of learning from unlabeled data, where the current methods mostly revolve around alignment of representations and input recon struction. Although such approaches have demonstrated excellent performance in practice, their scope remains mostly confined to learning from observed data and does not provide much help in terms of a learning structure that is predictive of the data distribution. In this paper, we study some of the recent developments in the realm of self-supervised learning. We define a new category called Predictive Representation Learning (PRL), which revolves around the latent prediction of unobserved components of data based on the observation. We propose a common taxonomy that classifies PRL along with alignment and reconstruction-based learning approaches. Furthermore, we argue that Joint-Embedding Predictive Architecture(JEPA) can be considered as an exemplary member of this new paradigm. We further discuss theoretical perspectives and open challenges, highlighting predictive representation learning as a promising direction for future self-supervised learning research. In this study, we implemented Bootstrap Your Own Latent (BYOL), Masked Autoencoders (MAE), and Image-JEPA (I-JEPA) for comparative analysis. The results indicate that MAE achieves perfect similarity of 1.00, but exhibits relatively weak robustness of 0.55. In contrast, BYOL and I-JEPA attain accuracies of 0.98 and 0.95, with robustness scores of 0.75 and 0.78, respectively.
Figures & tables
Figure 1: Architectural comparison of Contrastive, Non Contrastive, Reconstruction based and Predictive Representation
Dimension
Contrastive SSL
Non-Contrastive SSL
Reconstruction SSL
Predictive Representation Learning
Representative methods
SimCLR, MoCo
BYOL, SimSiam
MAE, BEiT
JEPA-style methods
Learning signal
Discrimination
View alignment
Input reconstruction
Latent prediction
Negative samples
Yes
No
No
No
Operates in input space
No
No
Yes
No
Operates in latent space
Yes
Yes
Partially
Yes
Objective symmetry
Symmetric
Symmetric
Directional
Directional
Table 1: Conceptual and architectural comparison
Method
Augmentation Similarity
Occlusion Robustness
BYOL (Alignment)
0.988±0.007
0.750±0.046
I-JEPA (PRL)
0.950±0.027
0.788±0.045
MAE (Reconstruction)
1.000±0.000
0.550±0.086
Table 2: Empirical Comparison of SSL Paradigms from our implementation and training results of BYOL, I-JEPA for their Allighment, PRL and MAE
Figure 2: Comparision of the results from our custom implimentation and training: Comparison of augmentation similarity and occlusion robustness across BYOL, I-JEPA, and MAE. Predictive learning (I-JEPA) achieves superior robustness despite lower similarity.
Model
Modality
Dataset
Evaluation Protocol
Performance
I-JEPA [ 9 ]
Vision
ImageNet-1K
Linear Probe (ViT-H/14)
72.8% Top-1
I-JEPA [ 9 ]
Vision
ImageNet-1K
k-NN
∼ 73%
V-JEPA [ 10 ]
Video
Kinetics-400
Linear Probe
Competitive with VideoMAE
VL-JEPA [ 11 ]
Vision-Language
Image-Text datasets
Retrieval / Alignment
Improved cross-modal retrieval
Graph-JEPA [ 12 ]
Graph
OGB datasets
Node Classification
SOTA / Competitive
Table 3: Published Benchmark Results for JEPA Variants
Figure 3: New Taxonomy for SSL categorization
Aspect
Contrastive
Non-Contrastive
Reconstruction
Predictive (PRL)
Representative loss
InfoNCE
Alignment loss
Reconstruction loss
Latent prediction loss
Learning signal
Discrimination
View matching
Input recovery
Context-to-target prediction
Negative samples
Required
Not required
Not required
Not required
Operates in input space
No
No
Yes
No
Operates in latent space
Yes
Yes
Partially
Yes
Objective symmetry
Symmetric
Symmetric
Directional
Directional
Table 4: Loss functions and optimization characteristics
Dimension
Alignment-Based SSL
Reconstruction-Based SSL
Predictive Representation Learning
Primary learning objective
Representation alignment
Input recovery
Latent prediction
Source of supervision
Observed views
Observed inputs
Unobserved components
Objective formulation
Symmetric
Directional
Directional
Learning space
Latent space
Input space
Latent space
Role of masking or augmentation
View generation
Exact reconstruction
Partial observability
Treatment of missing information
Implicitly ignored
Explicitly recovered
Predicted in representation space
Table 5: Conceptual differences between self-supervised learning categories
The Joint-Embedding Predictive Architecture (JEPA) is often seen as a non-generative alternative to likelihood-based self-supervised learning, emphasizing prediction in representation space rather than reconstruction in observation space. We argue that the resulting separation from probabilistic generative modeling is largely rhetorical rather than structural: the canonical JEPA design (coupled encoders with a context-to-target predictor) mirrors the variational posteriors and learned conditional priors obtained when variational inference is applied to a particular class of coupled latent-variable models, and standard JEPA can be viewed as a deterministic specialization in which regularization is imposed via architectural and training heuristics rather than an explicit likelihood. Building on this view, we derive the Variational JEPA (Var-JEPA), which makes the latent generative structure explicit by optimizing a single Evidence Lower Bound (ELBO). This yields meaningful representations without ad-hoc anti-collapse regularizers and allows principled uncertainty quantification in the latent space. We instantiate the framework for tabular data (Var-T-JEPA) and achieve strong representation learning and downstream performance, improving over T-JEPA across real-world tabular benchmarks while remaining competitive with strong raw-feature baselines.
Moritz Gögl, Christopher Yau
University of Oxford, UK · Health Data Research UK
Self-supervised learning (SSL) excels at finding general-purpose latent representations from complex data, yet lacks a unifying theoretical framework that explains the diverse existing methods and guides the design of new ones. We cast SSL as latent distribution matching (LDM): learning representations that maximize their log-probability under an assumed latent model (alignment), while maximizing latent entropy to prevent collapse (uniformity). This view unifies independent component analysis with contrastive, non-contrastive, and predictive SSL methods, including stop gradient approaches. Leveraging LDM, we derive a nonlinear, sampling-free Bayesian filtering model with a Kalman-based predictor for high-dimensional timeseries. We further prove that predictive LDM yields identifiable latent representations under mild assumptions, even with nonlinear predictors. Overall, LDM clarifies the assumptions behind established SSL methods and provides principled guidance for developing new approaches.
Fabian A Mikulasch, Friedemann Zenke
Friedrich Miescher Institute for Biomedical Research, 4056 Basel, Switzerland · Faculty of Science, University of Basel, 4033 Basel, Switzerland
Non-contrastive self-supervised learning (SSL) is an effective framework for predictive representation learning, but popular (and in practice effective) methods such as SimSiam, BYOL, I-JEPA or DINO, which rely on a form of self-distillation to train a teacher-student network, remain poorly understood as they typically do not minimize a well-defined objective. We analyze the dynamics of a variant of the Joint Embedding Predictive Architecture (JEPA) using a regularized linear regressor to predict the learned representations of two views of the data from one another, and fully characterize its stability: non-collapsed stable equilibria align with leading nonlinear canonical correlation subspaces, while collapsed equilibria may also be stable attractors. Motivated by this result, we introduce PEIRA, a non-contrastive SSL method with an explicit objective defined through the trace of the optimal linear regressor. We show that its only stable equilibria are nontrivial global minimizers and recover the same canonical correlation subspaces, with regularization selecting the effective dimension. Experiments on ImageNet-1K and CIFAR-10 show PEIRA is competitive with VICReg and LeJEPA baselines, and qualitative empirical results support the theory.