cs.LGNov 12, 2026

Temporal Regularized Learning: Self-Supervised Learning Local in Space and Time

Authors: Davide Wiest

Organizations: Department of Computer Science, TU Darmstadt, Darmstadt, Germany

Abstract

Self-supervised objectives such as VICReg learn representations by making related inputs similar, while variance and covariance penalties prevent collapse. These penalties are statistics over a batch of samples, so layer-local variants, too, are typically local in space but not in time. We introduce Temporal Regularized Learning (TeReL), which optimizes a VICReg-style objective one sample at a time with purely local updates: each update uses only the current activity of its own and the preceding layer and a fixed-size running state. Temporally adjacent inputs serve as positive pairs, which makes the objective a soft form of Slow Feature Analysis (SFA). Running estimates of each neuron's mean and variance, together with one lateral matrix per layer, replace the batch statistics; held fixed during differentiation, they turn the gradient into a Hebbian feedforward update and an anti-Hebbian lateral update. We show that a two-layer network trained with batch size one on an MNIST stream of class-homogeneous chunks reaches 96.48% linear-probe accuracy, above random features (95.09%) and classical SFA, and that the temporal order and each loss term contribute to this result. With both locality constraints relaxed, the same objective matches supervised backpropagation. TeReL thus shows that the collapse-preventing statistics of modern self-supervised learning can be held as running state at each neuron, which brings these objectives closer to what brains and neuromorphic hardware can implement.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. λλ-JEPA Spectral Anti-Collapse Regularization for Self-Supervised Learning

    Sep 28, 2026Berker Demirel, Clémentine Dominé, Valentino Maiorca +3Spectral RegularizationJoint-Embedding Predictive Architecture

  2. Increasing Width Allows Greedy Layer-wise Training to Rival End-to-End Backpropagation in Self-Supervised Learning

    Sep 30, 2026Syon Mansur, Joel ZylberbergDeep Learning OptimizationConvolutional Neural Networks

  3. Dataset-Level Regularization for Joint-Embedding Self-Supervised Learning

    Oct 14, 2026Konstantinos Almpanakis, Anna KreshukSelf-Supervised Visual Representation LearningRepresentation Learning