Self-supervised learning (SSL) by predicting in latent space, without generating the input data itself, learns highly abstract, useful representations. Intuitively, this success is often attributed to its ability to discard nuisance information that is irrelevant to prediction. However, this poses a conundrum: both stochastic variation in a prediction-relevant latent signal and true nuisance make observations partly unpredictable; how could they be distinguished? Surprisingly, we prove that common SSL methods can achieve exactly this, by implicitly instantiating a latent-variable model with stochastic dynamics and observation-private nuisance. We trace their ability to recover the stochastic signal to two complementary principles: Predictive mutual information maximization ensures that representations retain the information needed for prediction, while latent distribution matching constrains how this information is encoded, thereby making the retained signal identifiable. We confirm this identifiability result in simulations for Gaussian predictors, which recover the true signal up to an affine transformation even in dynamic, nuisance-laden environments.
Figures & tables
Figure 1: Graphical model for observation-private nuisance. The three layers show (from left to right) unobserved latent variables, observations, and learned representations. Square nodes denote deterministically defined variables.
Entropy estimator
ds=5
ds=10
ds=15
ds=20
KDE
0.92±0.16
0.81±0.04
0.60±0.08
0.48±0.08
kNN
0.98±0.00
0.95±0.04
0.76±0.01
0.64±0.02
logdet
0.99±0.00
0.95±0.04
0.78±0.02
0.68±0.01
Table 1: R2 of the learned representations with the true factors of variation for the numerical test, averaged over 5 runs plus minus standard deviations. We highlight well learned factors ( R2>0.9 ).
Scenario
Entr. est.
pobj-x
pobj-y
pobj-z
θobj-α
θobj-β
θobj-γ
hobj
pspot
hspot
hback
Env. signal
KDE
0.20
0.47
0.37
0.05
0.08
0.12
0.41
0.97
0.95
0.97
Env. signal
kNN
0.05
0.55
0.39
0.08
0.05
0.07
0.11
0.97
0.95
0.97
Env. signal
logdet
−0.01
−0.02
−0.00
−0.01
−0.00
−0.01
−0.01
0.98
0.76
0.99
Object signal
KDE
0.98
0.98
0.97
0.95
0.96
0.94
0.92
−0.01
0.01
0.13
Object signal
kNN
0.97
0.96
0.96
0.87
0.92
0.26
0.91
−0.01
−0.00
0.56
Object signal
logdet
0.98
0.97
0.97
0.92
0.95
0.91
0.87
0.15
−0.00
−0.00
Table 2: R2 of the learned representations with the true factors of variation for the stochastic Causal3DIdent dataset, averaged over 5 runs. Standard deviations are not displayed here but are mostly small ( <0.02 ) except for unidentified signal / identified nuisance variables, indicating intermittent optimization failures. We highlight well learned factors ( R2>0.9 ) in bold and signal factors with grey background.
Figure 2: Causal3DIdent image-pairs. The left diagrams show the true latent factors which are: object identity ( iobj ), position ( pobj-x , etc., here displayed grouped), rotation ( θobj-α , etc.) and hue ( hobj ), spotlight position ( pspot ), hue ( hspot ), and background hue ( hback ). Arrows mark the factors causally coupled across image pairs. Right panels show example rendered image pairs. Even though factors between images are causally connected, there still exists significant variation due to noisy coupling.
Figure 3: Learning a world model for the controlled MuJoCo hopper. (A) Dataset generation. (B) Example timeseries. (C) UMAP of true signal (hopper state) and learned representations. Colors denote variable values. (D) R2 of latent representations with hopper variables from linear readout.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Scenario
Entr. est.
pobj-x
pobj-y
pobj-z
θobj-α
θobj-β
θobj-γ
hobj
pspot
hspot
hback
Environment signal
kde
0.35
0.52
0.34
0.07
0.10
0.16
0.53
0.99
0.97
0.99
Environment signal
knn
0.21
0.47
0.34
0.08
0.06
0.08
0.16
0.98
0.96
0.98
Environment signal
logdet
−0.02
−0.02
−0.03
−0.02
−0.01
−0.00
−0.01
0.99
0.57
0.98
Object signal
kde
0.98
0.98
0.97
0.94
0.97
0.85
0.91
0.14
0.03
0.36
Object signal
knn
0.97
0.96
0.95
0.87
0.92
0.29
0.91
0.14
−0.04
0.69
Object signal
logdet
0.99
0.99
0.98
0.95
0.97
0.96
0.92
0.16
−0.04
−0.04
Appendix
Table 3: Nonlinear R2 for the Causal3DIdent experiment (Table 2 ), computed with trained MLP predictor on held-out dataset.
Figure 4: Linear (Top) and nonlinear (Bottom) R2 with physical (signal) and nuisance variables for the hopper experiment. Velocity is read out from LSTM state. Background pattern nuisance is not included.
Self-supervised learning (SSL) excels at finding general-purpose latent representations from complex data, yet lacks a unifying theoretical framework that explains the diverse existing methods and guides the design of new ones. We cast SSL as latent distribution matching (LDM): learning representations that maximize their log-probability under an assumed latent model (alignment), while maximizing latent entropy to prevent collapse (uniformity). This view unifies independent component analysis with contrastive, non-contrastive, and predictive SSL methods, including stop gradient approaches. Leveraging LDM, we derive a nonlinear, sampling-free Bayesian filtering model with a Kalman-based predictor for high-dimensional timeseries. We further prove that predictive LDM yields identifiable latent representations under mild assumptions, even with nonlinear predictors. Overall, LDM clarifies the assumptions behind established SSL methods and provides principled guidance for developing new approaches.
Fabian A Mikulasch, Friedemann Zenke
Friedrich Miescher Institute for Biomedical Research, 4056 Basel, Switzerland · Faculty of Science, University of Basel, 4033 Basel, Switzerland
Self-supervised learning (SSL) is often deployed under changing information, such as shorter histories, missing features, or partially observed images. In these settings, predictions from coarse and refined views should be coherent: before refinement, the coarse-view prediction should match the average prediction expected after refinement. Martingales formalize this coherence principle, but standard SSL objectives do not enforce it. Unlike invariance objectives that pull views together, martingale consistency constrains only the expected refined prediction, allowing predictions to update as information is revealed while preventing systematic drift. We introduce a martingale-consistent SSL framework that closes this gap, with practical prediction- and latent-space variants and an unbiased two-sample Monte Carlo estimator based on stochastic refinement. We evaluate the approach on synthetic and real time-series, tabular, and image benchmarks under partial-observation regimes, in both semi-self-supervised and fully label-free settings. Across these experiments, our framework improves robustness and calibration under partial observation, yielding more stable representations as information is revealed.
Joint-embedding self-supervised learning (SSL), the key paradigm for unsupervised representation learning from visual data, learns from invariances between semantically-related data pairs. We study the one-to-many mapping problem in SSL, where each datum may be mapped to multiple valid targets. This arises when data pairs come from naturally occurring generative processes, e.g., successive video frames. We show that existing methods struggle to flexibly capture this conditional uncertainty. As a remedy, we introduce a latent variable to account for this uncertainty and derive a variational lower bound on the mutual information between paired embeddings. Our derivation yields a simple regularization term for standard SSL objectives. The resulting method, which we call AdaSSL, applies to both contrastive and distillation-based SSL objectives, and we empirically show its versatility in causal representation learning, fine-grained image understanding, and world modeling on videos.
Yipeng Zhang, Hafez Ghaemi, Jungyoon Lee +3
Mila - Qu´ebec AI Institute · Universit´e de Montr´eal · CHU Sainte-Justine +2