cs.LGNov 8, 2026

Self-Supervised Collapse Prevention with Score-Based Entropy Maximization

Authors: Adarsh Kappiyath, Silpa Vadakkeeveetil Sreelatha, Xiatian Zhu, Yann LeCun, Lu Yin, Amir Bar

Organizations: University of Surrey · New York University · AMI Labs · Shenzhen University of Advanced Technology · 

Abstract

Diffusion models have demonstrated the effectiveness of score-based objectives for generative modeling, yet their potential for self-supervised representation learning remains underexplored. We introduce Entropy-Based Regularization (EBReg), a framework that uses latent score estimation to regularize the latent distribution during self-supervised training. EBReg jointly learns an encoder and a latent score model, and utilizes the learned score to provide a diversity-preserving signal that prevents representational collapse. Motivated by a mutual-information lower bound, we formulate the objective as view alignment with smoothed marginal entropy maximization and optimize the marginal entropy without explicit density evaluation. The same score model can additionally be used at test time to refine corrupted representations toward the training latent distribution. Empirically, EBReg learns competitive representations on ImageNet-1K, achieving 75.57% linear-probe accuracy with ViT-L. Across diverse architectures, score-guided test-time adaptation further improves robustness on ImageNet-C, yielding an average relative accuracy gain of approximately 5%.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. λλ-JEPA Spectral Anti-Collapse Regularization for Self-Supervised Learning

    Sep 28, 2026Berker Demirel, Clémentine Dominé, Valentino Maiorca +3Spectral RegularizationJoint-Embedding Predictive Architecture

  2. Self-Distillation of Hidden Layers for Self-Supervised Representation Learning

    Mar 16, 2026Scott C. Lowe, Anthony Fuller, Sageev Oore +2Self-Supervised Visual Representation LearningPredictive Representation Learning