Spectral Ghost in Representation Learning: A Unified View from Component Analysis to Self-Supervised Learning
Organizations: Google DeepMind · Georgia Tech · Harvard University · University of Alberta
Abstract
Self-supervised learning (SSL) extracts representations from massive unlabeled data that transfer to a variety of downstream tasks. Its empirical success has produced a dizzying array of objectives and procedures, yet a clear understanding of what representation is desired, and why these seemingly different methods work, is still missing. In this paper, we develop a unified spectral framework for representation learning. We first show that a representation is sufficient for arbitrary downstream prediction with a linear head if it spans the spectral decomposition of the pointwise mutual information, which can be extracted purely from unlabeled data. This view organizes representation learning along two axes, the parametrization of the spectral representation (linear, energy-based, denoising, nonlinear, latent-variable, and multi-modal) and the learning objective for fitting it, under which a broad range of existing methods, including SimCLR, MoCo, BYOL, Barlow Twins, VICReg, DINO, MAE, JEPA, BERT, GPT, and CLIP, as well as classic PCA, CCA, Laplacian eigenmaps, LLE, and SNE, are recovered as particular combinations. The framework also clarifies optimization behaviors: the batch-size requirement is determined by whether the objective is unbiased, a consistent surrogate, or biased, rather than by contrastive vs. non-contrastive losses. Finally, the framework inspires new scalable algorithms and extends spectral representations to causal inference and reinforcement learning.
Figures & tables
| Method | Parametrization | Learning objective | Section |
|---|---|---|---|
| Spectral contrastive | Linear | Square loss | 3.1 |
| Barlow-Twins, VICReg | Linear | Penalized orthogonality | 3.1 |
| BYOL, MINC | Linear | Variational power iteration | 3.1 |
| SimCLR, word2vec | Energy-based | Ranking / binary NCE | 4.1 |
| MoCo | Energy-based | Power iteration with ranking NCE | 4.1 |
| Diffusion SSL | Denoising (lifted energy-based) | Diffusion (square) loss | 5.2 |
| Objective | Mini-batch gradient | Role of batch size | Existing instances |
|---|---|---|---|
| Square loss ( 12 ) | Unbiased | Variance | Spectral contrastive |
| Denoising loss (Gaussian / masked) | Unbiased | Variance | Diffusion, DAE, BERT |
| Binary NCE ( 43 ) | Unbiased | Variance | word2vec, SigLIP |
| Density ratio ( 47 ) | Unbiased | Variance | Lu et al. (2024) |
| Projected square loss ( 51 ) | Unbiased | Variance | MAE |
| Variational power iteration | Unbiased | Variance | BYOL, MINC, SimSiam, JEPA, DINO |
| Parametrization | Linear sufficient representation | Predictor upon the learned representation | Section |
|---|---|---|---|
| Linear | linear head | 2.3 | |
| Energy-based | (infinite-dimensional) | attention; softmax with projector (linear softmax as first-order truncation) | 4 |
| Nonlinear (projector) | linear head upon ; nonlinear head upon | 6 | |
| Asymmetric projector across tokens (MAE) | decoder output | nonlinear head with attention upon | 6.1 |
| Latent-variable | posterior-weighted average ; softmax for classification | 7 |
| Method | Target | Parametrization | Learning objective |
|---|---|---|---|
| SNE | Symmetric | Energy-based, nonparametric | Ranking NCE |
| t-SNE | Symmetric | Nonlinear (Cauchy kernel), nonparametric | Jointly normalized NCE |
| NCA | Symmetric (label-defined) | Energy-based, linear | Marginal likelihood / Ranking NCE |
| Laplacian Embedding, Normalized cut | Symmetric | Linear, nonparametric | Variational SVD |
| LPP | Symmetric | Linear, linear map | Variational SVD |
| LLE | Symmetric | Linear, nonparametric | Fixed-point residual with shift |
Representation Learning
Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.
See all 860 papers