Centroid Spectral Coupling: Implicit Alignment and Representation Diversity in Self-Supervised Learning
Organizations: Department of Systems Design Engineering, University of Waterloo, Waterloo, ON, Canada · Department of Electrical and Computer Engineering, University of Waterloo, Waterloo, ON, Canada
Abstract
Self-Supervised Learning (SSL) aims to align augmented views while preserving representation diversity and preventing collapse. Existing non-contrastive methods often address these goals using multiple weighted loss terms. We introduce Centroid Spectral Coupling (CSC), a single-term SSL objective that implicitly couples view alignment, representation diversity, and collapse resistance. CSC constructs centroids from independently normalized augmented-view pairs and applies a ridge-based spectral objective to their uncentered second-moment matrix. Preserving centroid magnitudes encodes view agreement while encouraging balanced representation energy across spectral directions. We establish an exact decomposition into nonnegative alignment and spectral-imbalance components, characterize the geometry of global optima, and derive finite-loss guarantees for alignment and diversity. We also show that complete constant-output collapse is a strict saddle in normalized representation space. Experiments on CIFAR-10, CIFAR-100, and Imagenette show that CSC achieves the highest linear Top-1 accuracy among the evaluated non-contrastive methods on all three benchmarks, and the highest kNN Top-1 accuracy on both CIFAR datasets. A ridge-parameter sensitivity study reveals an empirical alignment--diversity trade-off: increasing strengthens view alignment but reduces representation effective rank. These results show that a unified spectral objective yields competitive visual representations without explicit contrastive negatives or separately weighted alignment and diversity penalties.
Figures & tables
| Linear Probing | kNN | ||||
|---|---|---|---|---|---|
| Dataset | Method | Top-1 | Top-5 | Top-1 | Top-5 |
| CIFAR-10 | CSC | 84.69 | 99.36 | 85.31 | 98.56 |
| MMCR (matched) | 82.13 | 99.06 | 82.73 | 98.28 | |
| VICReg | 81.97 | 99.09 | 79.62 | 98.05 | |
| Barlow Twins | 83.94 | 99.42 | 82.71 | 98.69 | |
| CIFAR-100 | CSC | 56.92 | 82.23 | 55.21 | 79.82 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Method | Linear Top-1 | kNN Top-1 | Alignment |
|---|---|---|---|---|
| CIFAR-10 | CSC | 78.07 0.20 | 75.91 0.37 | 0.8023 0.0009 |
| MMCR | 77.34 0.18 | 75.41 0.08 | 0.7453 0.0008 | |
| MNIST | CSC | 98.92 0.17 | 98.15 0.09 | 0.9933 0.0003 |
| MMCR | 98.52 0.06 | 97.21 0.25 | 0.9904 0.0004 | |
| CIFAR-100 | CSC | 47.67 0.42 | 41.33 0.12 | 0.7833 0.0012 |
| CSC-Cov | 47.92 0.54 | 41.36 0.47 | 0.7752 0.0017 |
| Linear Top-1 (%) | kNN Top-1 (%) | Alignment | Effective Rank | |
|---|---|---|---|---|
| 1 | 47.08 0.51 | 41.44 0.34 | 0.7103 0.0013 | 76.52 0.83 |
| 10 | 47.67 0.42 | 41.33 0.12 | 0.7833 0.0012 | 50.85 0.69 |
| 100 | 42.15 0.72 | 33.97 0.29 | 0.8680 0.0006 | 24.18 0.22 |
Self-Supervised Learning
Self-supervised learning uses structure within unlabeled data to train representations and models for downstream tasks.
See all 279 papers