Temporal Regularized Learning: Self-Supervised Learning Local in Space and Time
Organizations: Department of Computer Science, TU Darmstadt, Darmstadt, Germany
Abstract
Self-supervised objectives such as VICReg learn representations by making related inputs similar, while variance and covariance penalties prevent collapse. These penalties are statistics over a batch of samples, so layer-local variants, too, are typically local in space but not in time. We introduce Temporal Regularized Learning (TeReL), which optimizes a VICReg-style objective one sample at a time with purely local updates: each update uses only the current activity of its own and the preceding layer and a fixed-size running state. Temporally adjacent inputs serve as positive pairs, which makes the objective a soft form of Slow Feature Analysis (SFA). Running estimates of each neuron's mean and variance, together with one lateral matrix per layer, replace the batch statistics; held fixed during differentiation, they turn the gradient into a Hebbian feedforward update and an anti-Hebbian lateral update. We show that a two-layer network trained with batch size one on an MNIST stream of class-homogeneous chunks reaches 96.48% linear-probe accuracy, above random features (95.09%) and classical SFA, and that the temporal order and each loss term contribute to this result. With both locality constraints relaxed, the same objective matches supervised backpropagation. TeReL thus shows that the collapse-preventing statistics of modern self-supervised learning can be held as running state at each neuron, which brings these objectives closer to what brains and neuromorphic hardware can implement.
Figures & tables
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Samples | Batch or subsequence | Optimizer | Normalization |
|---|---|---|---|---|
| TeReL | SGD | none | ||
| TeReL -Offline | SGD | none | ||
| Layer-local SupCon | SGD | BatchNorm | ||
| Supervised offline | SGD | none | ||
| Batch SFA | complete fit | eigensolver | whitening | |
| Incremental SFA | incremental | whitening |
| Method / seed | 1101 | 1202 | 1303 | 1404 | 1505 |
|---|---|---|---|---|---|
| TeReL | .9637 | .9647 | .9671 | .9633 | .9651 |
| TeReL -Offline | .9847 | .9837 | .9841 | .9836 | .9842 |
| Random encoder | .9475 | .9529 | .9495 | .9520 | .9524 |
| Input linear probe | .9254 | .9262 | .9257 | .9250 | .9246 |
| Supervised offline | .9836 | .9858 | .9840 | .9846 | .9838 |
| Layer-local SupCon | .9698 | .9713 | .9677 | .9711 | .9678 |
| Category | Quantity | Size |
|---|---|---|
| Dynamic neuron state | ||
| Auxiliary lateral parameters | ||
| Feedforward parameters | ||
| Optimizer state | none |
Self-Supervised Learning
Self-supervised learning uses structure within unlabeled data to train representations and models for downstream tasks.
See all 276 papers