Self-Supervised Collapse Prevention with Score-Based Entropy Maximization
Organizations: University of Surrey · New York University · AMI Labs · Shenzhen University of Advanced Technology ·
Abstract
Diffusion models have demonstrated the effectiveness of score-based objectives for generative modeling, yet their potential for self-supervised representation learning remains underexplored. We introduce Entropy-Based Regularization (EBReg), a framework that uses latent score estimation to regularize the latent distribution during self-supervised training. EBReg jointly learns an encoder and a latent score model, and utilizes the learned score to provide a diversity-preserving signal that prevents representational collapse. Motivated by a mutual-information lower bound, we formulate the objective as view alignment with smoothed marginal entropy maximization and optimize the marginal entropy without explicit density evaluation. The same score model can additionally be used at test time to refine corrupted representations toward the training latent distribution. Empirically, EBReg learns competitive representations on ImageNet-1K, achieving 75.57% linear-probe accuracy with ViT-L. Across diverse architectures, score-guided test-time adaptation further improves robustness on ImageNet-C, yielding an average relative accuracy gain of approximately 5%.
Figures & tables
| Latent constraint | Typical realization | Max.-entropy target | SSL connection |
|---|---|---|---|
| , | Batch normalization without affine parameters | Isotropic Gaussian | SIGReg: Gaussian target; VICReg: corresponding first- and second-order moments |
| , RMS, or layer normalization | Uniform on the sphere | Contrastive learning | |
| Discrete prototype / code assignment | Uniform code usage | SwAV and MSN: balanced prototype usage; | |
| -th moment scaling with support constraint | Generalized Gaussian (truncated to ) | Rectified LpJEPA |
| Backbone | Method / Regularizer | Epochs | Linear probe (%) | Eff. rank |
|---|---|---|---|---|
| ResNet-50 | Alignment only ( ) | 100 | Collapse | – |
| VICReg | 100 | 68.02 | 1368 | |
| SIGReg | 100 | 66.42 | 1256 | |
| EBReg (ours) | 100 | 67.92 | 1431 | |
| ViT-L | SIGReg | 100 | 75.32 | 648 |
| -DAE | 1600 | 75.00 | – |
| ViT-L Setup from Balestriero & LeCun (2025) | ViT-H Setup from Assran et al. (2023) | ResNet-50 Setup from Bardes et al. (2022) | ||||
|---|---|---|---|---|---|---|
| Corruption | Before | After | Before | After | Before | After |
| Fog | 44.18 | 46.34 | 58.41 | 61.42 | 24.00 | 38.60 |
| Impulse noise | 43.41 | 46.86 | 50.77 | 56.33 | 15.52 | 22.71 |
| Motion blur | 41.12 | 46.29 | 58.01 | 61.52 | 18.86 | 25.37 |
| Glass blur | 21.72 | 25.39 | 44.75 | 49.93 | 7.78 | 13.59 |
| Zoom blur | 39.26 | 43.00 | 50.92 | 54.70 | 20.19 | 27.27 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| 1 | for do |
|---|---|
| 2 | Sample a minibatch and paired views using . |
| 3 | . |
| 4 | repeat times |
| 5 | Draw noise ; . |
| 6 | Update only on . |
| 7 | end repeat |
| 1 | for each test minibatch do |
|---|---|
| 2 | Reset to its pretrained source values. |
| 3 | repeat times |
| 4 | Compute encoder representation . |
| 5 | Compute score-space embedding . |
| 6 | . |
| 7 | . |
Self-Supervised Visual Representation Learning
Self-supervised visual representation learning trains image, video, and 3D encoders on unlabeled data to produce transferable visual features.
See all 254 papers