Predictive Self-Supervised Learning Provably Identifies Stochastic Signals under Nuisance

Fabian A. Mikulasch; Friedemann Zenke

cs.LGSep 29, 2026

Predictive Self-Supervised Learning Provably Identifies Stochastic Signals under Nuisance

Authors: Fabian A. Mikulasch, Friedemann Zenke

Organizations: Friedrich Miescher Institute for Biomedical Research, Basel, Switzerland · Faculty of Science, University of Basel, Switzerland

Abstract

Self-supervised learning (SSL) by predicting in latent space, without generating the input data itself, learns highly abstract, useful representations. Intuitively, this success is often attributed to its ability to discard nuisance information that is irrelevant to prediction. However, this poses a conundrum: both stochastic variation in a prediction-relevant latent signal and true nuisance make observations partly unpredictable; how could they be distinguished? Surprisingly, we prove that common SSL methods can achieve exactly this, by implicitly instantiating a latent-variable model with stochastic dynamics and observation-private nuisance. We trace their ability to recover the stochastic signal to two complementary principles: Predictive mutual information maximization ensures that representations retain the information needed for prediction, while latent distribution matching constrains how this information is encoded, thereby making the retained signal identifiable. We confirm this identifiability result in simulations for Gaussian predictors, which recover the true signal up to an affine transformation even in dynamic, nuisance-laden environments.

Figures & tables

Panel of Figure 1 — Figure 1: Graphical model for observation-private nuisance. The three layers show (from left to right) unobserved latent variables, observations, and learned representations. Square nodes denote deterministically defined variables.

Entropy estimator	$d_{s}=5$	$d_{s}=10$	$d_{s}=15$	$d_{s}=20$
KDE	$\mathbf{0.92\pm 0.16}$	$0.81\pm 0.04$	$0.60\pm 0.08$	$0.48\pm 0.08$
kNN	$\mathbf{0.98\pm 0.00}$	$\mathbf{0.95\pm 0.04}$	$0.76\pm 0.01$	$0.64\pm 0.02$
logdet	$\mathbf{0.99\pm 0.00}$	$\mathbf{0.95\pm 0.04}$	$0.78\pm 0.02$	$0.68\pm 0.01$

Table 1:

R^{2}

of the learned representations with the true factors of variation for the numerical test, averaged over 5 runs plus minus standard deviations. We highlight well learned factors (

R^{2}>0.9

Scenario	Entr. est.	$p_{\text{obj-}x}$	$p_{\text{obj-}y}$	$p_{\text{obj-}z}$	$\theta_{\text{obj-}\alpha}$	$\theta_{\text{obj-}\beta}$	$\theta_{\text{obj-}\gamma}$	$h_{\text{obj}}$	$p_{\text{spot}}$	$h_{\text{spot}}$	$h_{\text{back}}$
Env. signal	KDE	$0.20$	$0.47$	$0.37$	$0.05$	$0.08$	$0.12$	$0.41$	$\mathbf{0.97}$	$\mathbf{0.95}$	$\mathbf{0.97}$
Env. signal	kNN	$0.05$	$0.55$	$0.39$	$0.08$	$0.05$	$0.07$	$0.11$	$\mathbf{0.97}$	$\mathbf{0.95}$	$\mathbf{0.97}$
Env. signal	logdet	$-0.01$	$-0.02$	$-0.00$	$-0.01$	$-0.00$	$-0.01$	$-0.01$	$\mathbf{0.98}$	$0.76$	$\mathbf{0.99}$
Object signal	KDE	$\mathbf{0.98}$	$\mathbf{0.98}$	$\mathbf{0.97}$	$\mathbf{0.95}$	$\mathbf{0.96}$	$\mathbf{0.94}$	$\mathbf{0.92}$	$-0.01$	$0.01$	$0.13$
Object signal	kNN	$\mathbf{0.97}$	$\mathbf{0.96}$	$\mathbf{0.96}$	$0.87$	$\mathbf{0.92}$	$0.26$	$\mathbf{0.91}$	$-0.01$	$-0.00$	$0.56$
Object signal	logdet	$\mathbf{0.98}$	$\mathbf{0.97}$	$\mathbf{0.97}$	$\mathbf{0.92}$	$\mathbf{0.95}$	$\mathbf{0.91}$	$0.87$	$0.15$	$-0.00$	$-0.00$

Table 2:

R^{2}

of the learned representations with the true factors of variation for the stochastic Causal3DIdent dataset, averaged over 5 runs. Standard deviations are not displayed here but are mostly small (

<0.02

) except for unidentified signal / identified nuisance variables, indicating intermittent optimization failures. We highlight well learned factors (

R^{2}>0.9

) in bold and signal factors with grey background.

Panel of Figure 2 — Figure 2: Causal3DIdent image-pairs. The left diagrams show the true latent factors which are: object identity ( $i_{\mathrm{obj}}$ ), position ( $p_{\text{obj-}x}$ , etc., here displayed grouped), rotation ( $\theta_{\text{obj-}\alpha}$ , etc.) and hue ( $h_{\text{obj}}$ ), spotlight position ( $p_{\text{spot}}$ ), hue ( $h_{\text{spot}}$ ), and background hue ( $h_{\text{back}}$ ). Arrows mark the factors causally coupled across image pairs. Right panels show example rendered image pairs. Even though factors between images are causally connected, there still exists significant variation due to noisy coupling.

Figure 3: Learning a world model for the controlled MuJoCo hopper. (A) Dataset generation. (B) Example timeseries. (C) UMAP of true signal (hopper state) and learned representations. Colors denote variable values. (D) $R^{2}$ of latent representations with hopper variables from linear readout.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Scenario	Entr. est.	$p_{\text{obj-}x}$	$p_{\text{obj-}y}$	$p_{\text{obj-}z}$	$\theta_{\text{obj-}\alpha}$	$\theta_{\text{obj-}\beta}$	$\theta_{\text{obj-}\gamma}$	$h_{\text{obj}}$	$p_{\text{spot}}$	$h_{\text{spot}}$	$h_{\text{back}}$
Environment signal	kde	$0.35$	$0.52$	$0.34$	$0.07$	$0.10$	$0.16$	$0.53$	$\mathbf{0.99}$	$\mathbf{0.97}$	$\mathbf{0.99}$
Environment signal	knn	$0.21$	$0.47$	$0.34$	$0.08$	$0.06$	$0.08$	$0.16$	$\mathbf{0.98}$	$\mathbf{0.96}$	$\mathbf{0.98}$
Environment signal	logdet	$-0.02$	$-0.02$	$-0.03$	$-0.02$	$-0.01$	$-0.00$	$-0.01$	$\mathbf{0.99}$	$0.57$	$\mathbf{0.98}$
Object signal	kde	$\mathbf{0.98}$	$\mathbf{0.98}$	$\mathbf{0.97}$	$\mathbf{0.94}$	$\mathbf{0.97}$	$0.85$	$\mathbf{0.91}$	$0.14$	$0.03$	$0.36$
Object signal	knn	$\mathbf{0.97}$	$\mathbf{0.96}$	$\mathbf{0.95}$	$0.87$	$\mathbf{0.92}$	$0.29$	$\mathbf{0.91}$	$0.14$	$-0.04$	$0.69$
Object signal	logdet	$\mathbf{0.99}$	$\mathbf{0.99}$	$\mathbf{0.98}$	$\mathbf{0.95}$	$\mathbf{0.97}$	$\mathbf{0.96}$	$\mathbf{0.92}$	$0.16$	$-0.04$	$-0.04$

Appendix

Table 3: Nonlinear

R^{2}

for the Causal3DIdent experiment (Table 2 ), computed with trained MLP predictor on held-out dataset.

Figure 4: Linear (Top) and nonlinear (Bottom)

R^{2}

with physical (signal) and nuisance variables for the hopper experiment. Velocity is read out from LSTM state. Background pattern nuisance is not included.

Explore similar work

Understanding Self-Supervised Learning via Latent Distribution Matching
May 5, 2026Fabian A Mikulasch, Friedemann ZenkeFuture Latent Representations Bayesian Filtering
Friedrich Miescher Institute for Biomedical Research, 4056 Basel, Switzerland · Faculty of Science, University of Basel, 4033 Basel, Switzerland
Martingale-Consistent Self-Supervised Learning
May 12, 2026Moritz Gögl, Hanwen Xing, Christopher YauSelf-Supervised Learning Conformal Test Martingales
University of Oxford · Health Data Research UK
Self-Supervised Learning from Structural Invariance
Feb 2, 2026Yipeng Zhang, Hafez Ghaemi, Jungyoon Lee +3Self-Supervised Learning Latent Representation Alignment
Mila - Qu´ebec AI Institute · Universit´e de Montr´eal · CHU Sainte-Justine +2

May 5, 2026, cs.LGJ/K move · Enter open · S save

Understanding Self-Supervised Learning via Latent Distribution Matching

Fabian A Mikulasch, Friedemann Zenke

Friedrich Miescher Institute for Biomedical Research, 4056 Basel, Switzerland · Faculty of Science, University of Basel, 4033 Basel, Switzerland

Future Latent Representations Bayesian Filtering Independent Component Analysis

Open paper AskPDF arXiv

Self-supervised learning (SSL) excels at finding general-purpose latent representations from complex data, yet lacks a unifying theoretical framework that explains the diverse existing methods and guides the design of new ones. We cast SSL as latent distribution matching (LDM): learning representations that maximize their log-probability under an assumed latent model (alignment), while maximizing latent entropy to prevent collapse (uniformity). This view unifies independent component analysis with contrastive, non-contrastive, and predictive SSL methods, including stop gradient approaches. Leveraging LDM, we derive a nonlinear, sampling-free Bayesian filtering model with a Kalman-based predictor for high-dimensional timeseries. We further prove that predictive LDM yields identifiable latent representations under mild assumptions, even with nonlinear predictors. Overall, LDM clarifies the assumptions behind established SSL methods and provides principled guidance for developing new approaches.

Abstract

Explore similar work

Understanding Self-Supervised Learning via Latent Distribution Matching

Martingale-Consistent Self-Supervised Learning

Self-Supervised Learning from Structural Invariance