Scaling Channel-Adaptive Self-Supervised Learning
Organizations: University of São Paulo, Brasil (Work done during an internship at FAIR, Meta.) · Meta FAIR Paris, France · University of Wisconsin–Madison, USA · Columbia University, New York, USA
Abstract
Recent advances in self-supervised pre-training of foundation models for natural images have made them a popular choice for various visual systems and applications. Self-supervised strategies are also promising in non-RGB scientific imaging domains such as in biology, medical and satellite imagery, but their broader application is hampered by heterogeneity in channel composition and semantics between relevant datasets: two datasets may contain different numbers of channels, and these may reveal distinct aspects of an object or scene. Recent works on channel adaptive strategies report substantial advantages for those that account for variable channel compositions without sacrificing the ability to jointly encode channels; yet, how these strategies behave at scale remains unclear. We here show that, surprisingly, trained across large-scale datasets, independent-encoding of channels outperforms joint-encoding methods by a substantial margin. We validate this result along an extensive set of experiments on various datasets from cell microscopy to geospatial imagery. Our DINO BoC approach sets a new state-of-the-art across challenging benchmarks, including generalization to out-of-distribution tasks and unseen channel combinations. We open-source code and model weights for a new general-purpose feature extractor for fluorescent microscopy.
Figures & tables
| Model | SSL | Network | Channel | Training set | HPA-FOV F1 | HPA-FOV F1 | JUMP-CP | WTC F1 |
|---|---|---|---|---|---|---|---|---|
| method | size | adaptive | Protein loc. | Cell type | Accuracy | Cell cycle st. | ||
| CellDINOv2 | DINOv2 | ViT-L | ✗ | HPA-FOV | 65.0 | 89.3 | ✗ | ✗ |
| CellDINOv2 | DINOv2 | ViT-L | ✗ | JUMP-CP | ✗ | ✗ | 44.3 | ✗ |
| CellDINOv1 | DINOv1 | ViT-L | ✗ | WTC | ✗ | ✗ | ✗ | 82.3 |
| Channel-ViT | DINOv2 | ViT-L | ✓ | ExtendedCHAMMI | 57.4 -7.6 | 90.4 +1.1 | 39.4 -4.9 | 87.2 +4.9 |
| DINO HA | DINOv2 | ViT-L | ✓ | ExtendedCHAMMI | 61.4 -3.6 | 91.3 +2.0 | 45.2 +0.9 | 91.0 +8.7 |
| Class | Budtip | Cell periphery | Budneck | Actin | All |
|---|---|---|---|---|---|
| frequency | 1.5 % | 1.9% | 2.4% | 3.8% | |
| Microsnoop ( Xun et al., 2024 ) | 32.1 | 96.4 | 43.4 | 48.0 | 75.9 |
| DINO BoC | 62.1 | 97.5 | 72.6 | 63.7 | 83.1 |
| Model | Channel | Training set | HPA-FOV F1 | HPA-FOV F1 | JUMP-CP | WTC F1 |
|---|---|---|---|---|---|---|
| adaptive | Protein loc. | Cell type | Accuracy | Cell cycle st. | ||
| DINO Channel-ViT | ✓ | HPA-FOV | 65.5 | 90.9 | 35.3 | 80.0 |
| DINO HA (Ours) | ✓ | HPA-FOV | 66.7 | 91.3 | 37.3 | 88.9 |
| DINO BoC (Ours) | ✓ | HPA-FOV | 65.2 | 91.5 | 40.2 | 89.8 |
| DINO Channel-ViT | ✓ | JUMP-CP | 29.5 | 82.0 | 53.4 | 81.8 |
| DINO HA (Ours) | ✓ | JUMP-CP | 30.3 | 85.2 | 52.0 | 84.2 |
| Model | Average OOD | WTC | HPA | CP | ||||||||||
| Mean | WTC | HPA | CP | Task1 | Task2 | Task1 | Task2 | Task3 | Task1 | Task2 | Task3 | Task4 | ||
| FixedChannels | 50.0 | 64.8 | 59.2 | 25.9 | 64.9 | 64.8 | 80.7 | 76.3 | 42.1 | 66.0 | 48.1 | 23.0 | 6.6 | |
| Depthwise | 51.7 | 65.2 | 64.4 | 25.6 | 68.9 | 65.2 | 84.9 | 81.3 | 47.5 | 67.3 | 47.8 | 22.4 | 6.5 | |
| TargetParam | 49.6 | 59.0 | 62.3 | 27.3 | 69.5 | 59.0 | 83.7 | 79.4 | 45.2 | 71.7 | 50.8 | 23.4 | 7.7 | |
| SliceParam | 45.7 | 56.8 | 54.6 | 25.6 | 61.6 | 56.8 | 77.0 | 69.0 | 40.3 | 64.6 | 47.5 | 22.2 | 7.1 | |
| HyperNet | 53.7 | 66.1 | 67.1 | 27.8 | 72.6 | 66.1 | 88.7 | 85.8 | 48.3 | 72.0 | 51.7 | 24.7 | 6.9 | |
| Model | Average OOD | WTC | HPA | CP | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mean | WTC | HPA | CP | Task 1 | Task 2 | Task 1 | Task 2 | Task 3 | Task 1 | Task 2 | Task 3 | Task 4 | |
| HyperNet | 53.7 | 66.1 | 67.1 | 27.8 | 72.6 | 66.1 | 88.7 | 85.8 | 48.3 | 72.0 | 51.7 | 24.7 | 6.9 |
| DINO Channel-ViT | 43.6 | 46.2 | 55.6 | 28.9 | 64.5 | 46.2 | 92.1 | 65.3 | 45.9 | 89.0 | 53.5 | 21.8 | 11.3 |
| DINO HA (Ours) | 43.2 | 54.3 | 46.9 | 28.4 | 66.4 | 54.3 | 90.1 | 60.4 | 33.5 | 93.6 | 57.1 | 15.8 | 12.1 |
| DINO BoC (Ours) | 51.6 | 79.0 | 43.0 | 32.7 | 79.4 | 79.0 | 86.6 | 59.3 | 29.6 | 92.6 | 57.6 | 22.1 | 18.5 |
| Average OOD | WTC | HPA | CP | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Mean | WTC | HPA | CP | Task1 | Task2 | Task1 | Task2 | Task 3 | Task1 | Task2 | Task3 | Task4 |
| HyperNet | 65.4 | 85.3 | 82.9 | 27.9 | 87.9 | 85.3 | 94.4 | 92.5 | 73.2 | 93.5 | 51.5 | 17.3 | 15.0 |
| DINO Channel-ViT | 59.8 | 66.9 | 76.7 | 35.9 | 83.1 | 66.9 | 88.2 | 84.9 | 68.4 | 80.5 | 54.5 | 23.3 | 30.0 |
| DINO HA (Ours) | 62.7 | 76.2 | 72.4 | 39.5 | 88.0 | 76.2 | 88.5 | 82.4 | 62.4 | 91.7 | 61.6 | 27.5 | 29.3 |
| DINO BoC (Ours) | 67.9 | 89.2 | 74.9 | 39.7 | 90.5 | 89.2 | 88.3 | 84.7 | 65.0 | 90.5 | 60.5 | 25.8 | 32.7 |
| Approach | Architecture | Channel adaptive | Test dataset | Pre-training dataset | mAP |
| Meter-ML Zhu et al. (2022) | DenseNet-121 | ✗ | NAIP, S2, S1 | ImageNet | 51.7 |
| LHRS-bot Muhtar et al. (2024) | VLM | ✗ | NAIP, S2, S1 | LHRS-Align-Recap (1.1M images & text) | 71.8 |
| VHM Pang et al. (2024) | VLM | ✗ | NAIP, S2, S1 | VersaD (14M images & text) | 72.7 |
| DINO Channel ViT | ViT-L | ✓ | NAIP, S1, S2 | NAIP, S1, S2 train (85K) | 64.4 |
| DINO BoC (ours) | ViT-L | ✓ | NAIP, S1, S2 | NAIP, S1, S2 train (85K) | 78.1 |
| Approach | Architecture | Channel adaptive | Test dataset | Pre-training dataset | mAP |
| Model | Channel adaptive | Batch size Channels | Top-1 Accuracy | |
|---|---|---|---|---|
| fixed channels | ✗ | 1024 3 = 3072 | ImageNet1K | 81.9 |
| fixed channels | ✗ | 3072 3 = 9207 | ImageNet1K | 83.1 |
| DINO HA | ✓ | 1024 3 = 3072 | ImageNet1K | 78.6 |
| DINO BoC(Ours) | ✓ | 3072 1 = 3072 | ImageNet1K | 82.8 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Image type | Size | Channels | Discarded factors |
|---|---|---|---|---|
| HPA Single Cell | One cell | 4 | Cell line (HEK 293); protein localization (cytosol, endoplasmic reticulum, nucleoplasm) | |
| WTC | One cell | 3 | Tagged cellular components (centrioles, tight junctions, actin bundles) | |
| OpenCell | Cropped FOV | 2 | None | |
| Cell Painting | Cropped FOV | 5 | Plates (SQ00015125, SQ00015168, SQ00015221); data source (BBBC022); treatments (BRD-K11129031, BRD-K62310379, BRD-K77947974) | |
| HPA FOV | FOV | 4 | Cell line (HEK 293); protein localization (cytosol, endoplasmic reticulum, nucleoplasm) |
| Model | HCS | Average OOD | WTC | HPA | CP | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mean | WTC | HPA | CP | Task1 | Task2 | Task1 | Task2 | Task3 | Task1 | Task2 | Task3 | Task4 | ||
| DINO Channel-ViT | ✗ | 43.6 | 46.2 | 55.6 | 28.9 | 64.5 | 46.2 | 92.1 | 65.3 | 45.9 | 89.0 | 53.5 | 21.8 | 11.3 |
| DINO Channel-ViT | ✓ | 39.9 | 39.5 | 51.9 | 28.4 | 66.4 | 39.5 | 88.5 | 62.3 | 41.5 | 90.2 | 56.0 | 22.3 | 6.9 |
| Training | HPA-FOV F1 | HPA-FOV F1 | Accuracy | WTC F1 |
|---|---|---|---|---|
| set | Protein loc. | Cell type | on JUMP-CP | Cell cycle st. |
| ExtendedCHAMMI | 61.7 | 91.1 | 45.2 | 90.5 |
| minus WTC | 60.4 - 1.3 | 91.1 - 0.0 | 43.7 - 1.5 | 90.9 + 0.4 |
| minus Cell painting | 60.2 - 1.5 | 91.1 - 0.0 | 44.1 - 1.1 | 89.8 - 0.7 |
| minus HPA (FOV, single cell) | 41.7 -20.0 | 89.9 - 1.2 | 46.9 + 1.7 | 92.3 + 1.8 |
| minus OpenCell | 60.9 - 0.2 | 91.0 - 0.1 | 44.2 - 1.0 | 90.0 - 0.5 |
| JUMP-CP 5 channels | JUMP-CP 8 channels | |
|---|---|---|
| Channel ViT | 39.5 | ✗ |
| DINO HA | 45.2 | 51.4 |
| DINO BoC | 45.2 | 51.6 |
| Feature | Average OOD | WTC | HPA | CP | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | dimension | Mean | WTC | HPA | CP | Task1 | Task2 | Task1 | Task2 | Task 3 | Task1 | Task2 | Task3 | Task4 |
| DINO Channel-ViT | 59.8 | 66.9 | 76.7 | 35.9 | 83.1 | 66.9 | 88.2 | 84.9 | 68.4 | 80.5 | 54.5 | 23.3 | 30.0 | |
| DINO BoC | 65.4 | 86.7 | 67.9 | 41.5 | 89.4 | 86.7 | 82.9 | 79.1 | 56.7 | 83.8 | 61.2 | 26.6 | 36.8 | |
| DINO Channel-ViT | 63.2 | 74.2 | 77.9 | 37.4 | 86.4 | 74.2 | 90.2 | 86.2 | 69.7 | 83.3 | 56.5 | 24.4 | 31.3 | |
| DINO BoC | 67.9 | 89.2 | 74.9 | 39.7 | 90.5 | 89.2 | 88.3 | 84.7 | 65.0 | 90.5 | 60.5 | 25.8 | 32.7 | |
| Number of pretraining iterations | AUROC | |
|---|---|---|
| Channel ViT | 34K | 69.3 |
| DINO HA | 34K | 77.3 |
| DINO BoC | 34K | 78.7 |
| DINO BoC | 81.9 |
Self-Supervised Learning
Self-supervised learning uses structure within unlabeled data to train representations and models for downstream tasks.
See all 280 papers