Contrastive latent representation learning with denoising score matching
Organizations: ENS Paris-Saclay · University of Amsterdam
Abstract
Contrastive representation learning methods operate by pulling different views of the same data sample together while pushing them away from views of different samples. A fundamental challenge is the estimation of the repulsive force, which typically requires large batch sizes, can exhibit high variance, and is susceptible to the curse of dimensionality. We introduce ScoreCLR, a contrastive representation learning method that offers a novel way to estimate this repulsion term. We consider the framework of representation learning by maximizing the mutual information between two latent views of a sample. By deriving the gradient of the Shannon entropy with respect to the encoder parameters, we obtain an equivalent objective that involves the score of the latent representation. We then use denoising score matching techniques to estimate this score with a lightweight network that is trained jointly with the encoder. We further propose regularization techniques using different noise levels and a Gaussian prior on the latent representation. ScoreCLR replaces the batch of explicit negative samples in contrastive methods with a learned repulsion whose cost is linear in the batch size rather than quadratic. On CIFAR-10 and ImageNet-100 it does so with minimal loss of linear-probe accuracy compared to SimCLR under a shared recipe, while producing projector outputs of higher effective rank, closer to the Gaussian prior.
Figures & tables
| CIFAR-10 | ImageNet-100 | |||||||||
| Linear | -NN | Linear | -NN | |||||||
| Method | Optim. | Projector | Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 |
| SimCLR | AdamW | 2048/256 | 99.70 | 98.90 | 92.18 | 87.32 | ||||
| LeJEPA | AdamW | 2048/256 | 99.65 | 98.65 | – | – | – | – | ||
| BYOL | AdamW | 2048/256 | 99.70 | 99.01 | – | – | – | – | ||
| Barlow Twins | AdamW | 2048/256 | 99.57 | 98.57 | – | – | – | – | ||
| CIFAR-10 | ImageNet-100 | ||||
|---|---|---|---|---|---|
| Method | Optim. | Align | Unif | Align | Unif |
| SimCLR | LARS | 0.349 | 0.240 | ||
| SimCLR | AdamW | ||||
| LeJEPA | AdamW | – | – | ||
| ScoreCLR (ours) | AdamW | ||||
| Pre-training | Method | Optim. | RankMe (/512) | RankMe (/256) | of |
| CIFAR-10 | SimCLR | LARS | 372 | 88 | – |
| ResNet-18 | SimCLR | AdamW | |||
| LeJEPA | AdamW | ||||
| BYOL | AdamW | – | |||
| Barlow Twins | AdamW | – | |||
| ScoreCLR (ours) | AdamW |
| Method | Optim. | CIFAR-10 | STL-10 | DTD |
|---|---|---|---|---|
| Pretrained on CIFAR-10 (ResNet-18, 32 px) | ||||
| SimCLR | AdamW | |||
| LeJEPA | AdamW | |||
| BYOL | AdamW | |||
| Barlow Twins | AdamW | |||
| ScoreCLR (ours) | AdamW | |||
| ScoreCLR | SimCLR | |||||
|---|---|---|---|---|---|---|
| Linear | -NN | RankMe | Linear | -NN | RankMe | |
| 128 | 88.80 | 88.51 | 229 | 89.45 | 88.56 | 96 |
| 512 | 88.51 | 88.39 | 231 | 89.32 | 89.07 | 108 |
| 2048 | 88.29 | 87.76 | 232 | 88.63 | 88.92 | 107 |
| 4096 | 87.50 | 86.88 | 230 | 88.88 | 88.78 | 107 |
| Precision | Linear | -NN | RankMe |
|---|---|---|---|
| fp32 | |||
| fp16 |
| Time (ms) | Peak memory (GiB) | |
|---|---|---|
| Encoder only (no objective) | 499.8 | 43.1 |
| ScoreCLR, | 508.2 | 43.1 |
| SimCLR | 530.9 | 48.7 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Shared AdamW | SimCLR (LARS) | ScoreCLR | |
| Backbone; epochs | ResNet-18; 1000 / 400 | ResNet-18; 1000 / 400 | ResNet-18; 1000 / 400 |
| Batch size | 128 / | 256 / | 128 / |
| Optimizer | AdamW | LARS | AdamW (encoder and score net) |
| Encoder LR | 0.4 / 0.3 | / | |
| Weight decay | (score net ) | ||
| LR schedule | constant, cosine over last 100 ep | 10-ep warm-up + cosine | constant, cosine over last 100 ep (encoder and score net) |
| Variant | Seeds | Linear | -NN | RankMe | Ref. |
| Reference ( , , , clip) | 1–3 | 229 | – | ||
| 1–3 | 233 | – | |||
| Conditional score network | 1 / 2 | 83.83 / 85.27 | 83.54 / 84.34 | 25 / 51 | 89.38 / 88.58 |
| , no clipping † | 5, 42 | 88.87 / 88.85 | 88.44 / 88.43 | 229 | – |
| , no clipping | 1–3 | – |
| Objective only | Full iteration, memory ratio | |||||
| TFLOPs | Memory (GB) | R18 | R18 | |||
| SimCLR | ScoreCLR | SimCLR | ScoreCLR | 32 px | 224 px | |
| 4096 | 0.10 | 0.08 | 1.1 | 0.16 | 0.96 | 0.99 |
| 8192 | 0.41 | 0.17 | 4.3 | 0.32 | 0.92 | 0.98 |
| 16384 | 1.65 | 0.33 | 17.2 | 0.65 | 0.85 | 0.96 |
| 32768 | 6.60 | 0.66 | 68.7 | 1.29 | 0.73 | 0.92 |
| Time per step (ms) | Extra memory | |||||
|---|---|---|---|---|---|---|
| ScoreCLR | ||||||
| InfoNCE | , | , | , | InfoNCE | ScoreCLR | |
| 128 | 0.62 | 2.3 | 8.2 | 15.1 | 2.0 MiB | 5.4 MiB |
| 512 | 0.78 | 2.4 | 8.5 | 16.0 | 16 MiB | 19 MiB |
| 2048 | 1.4 | 2.5 | 8.6 | 14.9 | 208 MiB | 71 MiB |
| 4096 | 4.4 | 2.5 | 8.9 | 15.2 | 800 MiB | 141 MiB |
| Setting | ScoreCLR | Epoch time (s) | Peak memory (GiB) |
|---|---|---|---|
| CIFAR-10, , fp32 | , | 8.96 / 8.28 | – |
| ImageNet-100, , fp32 | , | 71.0 / 70.3 | 5.63 / 5.62 |
| CIFAR-10, , fp16 | , | 6.81 / 6.64 | 11.0 / 11.1 |
| CIFAR-10, , fp16 | , | 8.27 / 8.28 | 21.7 / 22.2 |
| CIFAR-10, , fp16 | , | 10.00 / 9.98 | 43.1 / 45.8 |
Contrastive Learning
Contrastive learning trains representations by bringing related examples together and separating unrelated ones for recognition, retrieval, and transfer.
See all 348 papers