Diffusion models have demonstrated the effectiveness of score-based objectives for generative modeling, yet their potential for self-supervised representation learning remains underexplored. We introduce Entropy-Based Regularization (EBReg), a framework that uses latent score estimation to regularize the latent distribution during self-supervised training. EBReg jointly learns an encoder and a latent score model, and utilizes the learned score to provide a diversity-preserving signal that prevents representational collapse. Motivated by a mutual-information lower bound, we formulate the objective as view alignment with smoothed marginal entropy maximization and optimize the marginal entropy without explicit density evaluation. The same score model can additionally be used at test time to refine corrupted representations toward the training latent distribution. Empirically, EBReg learns competitive representations on ImageNet-1K, achieving 75.57% linear-probe accuracy with ViT-L. Across diverse architectures, score-guided test-time adaptation further improves robustness on ImageNet-C, yielding an average relative accuracy gain of approximately 5%.
Figures & tables
Figure 1: Illustration of the EBReg anti-collapse mechanism. Top: (a) Representations lie near a one-dimensional subspace of R2 . (b) EBReg alone spreads them across both dimensions. (c) Alignment alone contracts them, resulting in dimensional collapse. (d) Alignment + EBReg yields compact, separated classes while preventing dimensional collapse. Bottom: Evolution of the latent distribution under EBReg. Samples are moved away from the diagonal and populate both dimensions, ultimately forming an isotropic Gaussian distribution.
Figure 2: Overview of EBReg: training and test-time adaptation. (a) Two augmented views are encoded into normalized embeddings. The encoder minimizes a view-alignment loss and an entropy regularizer, LEBReg , using a detached estimate of the score of the noise-smoothed embedding distribution. (b) . The latent score is parameterized by a noise-prediction network fitted on detached embeddings, with Kscore updates per encoder step; it predicts the injected Gaussian noise, giving the score estimate sψ,σ(h~)=−ϵ^ψ(h~)/σ . (c) At test time, the learned score is reused as a test-time adaptation signal. The score model and remaining encoder–projector parameters are frozen, while only normalization parameters α are updated by minimizing LTTA , guiding corrupted embeddings toward higher-density regions of the learned source distribution.
Latent constraint
Typical realization
Max.-entropy target
SSL connection
E[h]=0 , trCov(h)=d
Batch normalization without affine parameters
Isotropic Gaussian
SIGReg: Gaussian target; VICReg: corresponding first- and second-order moments
∥h∥2=r
ℓ2 , RMS, or layer normalization
Uniform on the sphere
Contrastive learning
h∈{e1,…,eK}
Discrete prototype / code assignment
Uniform code usage
SwAV and MSN: balanced prototype usage;
E∥h∥pp=M,h∈S
p -th moment scaling with support constraint
Generalized Gaussian (truncated to S )
Rectified LpJEPA
Table 1: Maximum-entropy targets under different latent constraints. Each constraint induces a corresponding maximum-entropy target linked to existing SSL regularizers. First three targets maximize the smoothed entropy for any σ>0 . The last holds as σ→0 (Appendix E ).
Backbone
Method / Regularizer
Epochs
Linear probe (%) ↑
Eff. rank ↑
ResNet-50
Alignment only ( γ=0 )
100
Collapse
–
VICReg
100
68.02
1368
SIGReg
100
66.42
1256
EBReg (ours)
100
67.92
1431
ViT-L
SIGReg
100
75.32
648
ℓ -DAE
1600
75.00
–
Table 2: Linear-probe performance and effective rank on ImageNet-1K. Results are grouped by pretraining setup. ResNet-50, ViT-L, and ViT-H use the setups from Bardes et al. (2022) , Balestriero & LeCun (2025) , and Assran et al. (2023) , respectively. Within each setup, EBReg and the corresponding baselines use the same backbone, data augmentations, optimization settings, training schedule, and evaluation procedure. Effective rank ( Garrido et al., 2023 ) is computed on backbone representations after pretraining, with higher values indicating less dimensional collapse. Bold indicates higher accuracy within each setup, and underline indicates the second-best.
Table 5
ViT-L Setup from Balestriero & LeCun (2025)
ViT-H Setup from Assran et al. (2023)
ResNet-50 Setup from Bardes et al. (2022)
Corruption
Before
After
Before
After
Before
After
Fog
44.18
46.34
58.41
61.42
24.00
38.60
Impulse noise
43.41
46.86
50.77
56.33
15.52
22.71
Motion blur
41.12
46.29
58.01
61.52
18.86
25.37
Glass blur
21.72
25.39
44.75
49.93
7.78
13.59
Zoom blur
39.26
43.00
50.92
54.70
20.19
27.27
Table 5: ImageNet-C accuracy (%) before and after five adaptation steps. Seven corruptions with gains for every model are selected by mean gain across models; the final row averages all 15 corruptions. Bold indicates higher accuracy within each model.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
1
for r=1,…,T do
2
Sample a minibatch and paired views x(1),x(2) using T .