cs.LGJan 28, 2026

Spectral Ghost in Representation Learning: A Unified View from Component Analysis to Self-Supervised Learning

Authors: Bo Dai, Na Li, Dale Schuurmans

Organizations: Google DeepMind · Georgia Tech · Harvard University · University of Alberta

Abstract

Self-supervised learning (SSL) extracts representations from massive unlabeled data that transfer to a variety of downstream tasks. Its empirical success has produced a dizzying array of objectives and procedures, yet a clear understanding of what representation is desired, and why these seemingly different methods work, is still missing. In this paper, we develop a unified spectral framework for representation learning. We first show that a representation is sufficient for arbitrary downstream prediction with a linear head if it spans the spectral decomposition of the pointwise mutual information, which can be extracted purely from unlabeled data. This view organizes representation learning along two axes, the parametrization of the spectral representation (linear, energy-based, denoising, nonlinear, latent-variable, and multi-modal) and the learning objective for fitting it, under which a broad range of existing methods, including SimCLR, MoCo, BYOL, Barlow Twins, VICReg, DINO, MAE, JEPA, BERT, GPT, and CLIP, as well as classic PCA, CCA, Laplacian eigenmaps, LLE, and SNE, are recovered as particular combinations. The framework also clarifies optimization behaviors: the batch-size requirement is determined by whether the objective is unbiased, a consistent surrogate, or biased, rather than by contrastive vs. non-contrastive losses. Finally, the framework inspires new scalable algorithms and extends spectral representations to causal inference and reinforcement learning.

Figures & tables

Explore similar work

CardsList
  1. Sharp Regret Bounds and a Task-Covariance Correction for Spectral Representation Learning

    Sep 14, 2026Dier Tang, Jing Yee Tan, Guangyue HanRepresentation LearningSelf-Supervised Learning

  2. Understanding Self-Supervised Learning via Latent Distribution Matching

    May 5, 2026Fabian A Mikulasch, Friedemann ZenkeRepresentation LearningSelf-Supervised Learning

  3. Which Tasks Survive Self-Supervised Learning?

    Sep 29, 2026Achleshwar Luthra, Lucas Bryant, Tracy Zhu +1Representation LearningSelf-Supervised Learning