Authors: Alice V. De Lorenci, Seungeun Yi, Theo Moutakanni, Piotr Bojanowski, Camille Couprie, Juan C. Caicedo, Wolfgang M. Pernice
Organizations: University of São Paulo, Brasil (Work done during an internship at FAIR, Meta.) · Meta FAIR Paris, France · University of Wisconsin–Madison, USA · Columbia University, New York, USA
Recent advances in self-supervised pre-training of foundation models for natural images have made them a popular choice for various visual systems and applications. Self-supervised strategies are also promising in non-RGB scientific imaging domains such as in biology, medical and satellite imagery, but their broader application is hampered by heterogeneity in channel composition and semantics between relevant datasets: two datasets may contain different numbers of channels, and these may reveal distinct aspects of an object or scene. Recent works on channel adaptive strategies report substantial advantages for those that account for variable channel compositions without sacrificing the ability to jointly encode channels; yet, how these strategies behave at scale remains unclear. We here show that, surprisingly, trained across large-scale datasets, independent-encoding of channels outperforms joint-encoding methods by a substantial margin. We validate this result along an extensive set of experiments on various datasets from cell microscopy to geospatial imagery. Our DINO BoC approach sets a new state-of-the-art across challenging benchmarks, including generalization to out-of-distribution tasks and unseen channel combinations. We open-source code and model weights for a new general-purpose feature extractor for fluorescent microscopy.
Figures & tables
Figure 1: DINO-BoC is a channel-agnostic approach for SSL in channel-heterogeneous data domains that outperforms channel-adaptive methods and advances the SOTA across a variety of benchmarks in cellular microscopy (results relative to specialized baselines).
Figure 2: Training across datasets with various numbers of channels. (A) We consolidate images from Human Protein Atlas (2019) (HPA-FOV), Le et al. (2022) (HPA-SC), Cho et al. (2022) (OpenCell), Viana et al. (2023) (WTC) and Doron et al. (2023) (Cell Painting), into a dataset (ExtendedCHAMMI) that reflects the diversity in channel number, order, and semantics, that characterizes fluorescent microscopy studies. Despite some conventions (e.g. the nucleus is usually imaged using blue fluorescing stains), there is no necessary correspondence between specific channels and/or wavelengths, and their biological semantics. Images are pseudo-colored according to emission wavelength. CellPainting visualizes Plasma Membrane (PM) and Actin in one channel. (B) In aero-spatial imagery, images acquired at different resolution and using different wavelength / acquisition protocols are often available, but it remains a challenge to fully exploit all image modalities. We benchmark models on the MeterML dataset ( Zhu et al., 2022 ) , one sample is shown here.
Figure 3: Strategies for datasets with (A) variable-channel compositions (solid, dashed, dotted borders). (B) Channel-Adaptive methods, once trained, can accommodate variable channel numbers up to a maximum (here Kmax =4), with full cross-channel computation (akin to early fusion). Inset: Our Hierarchical Attention approach limits cross-channel computation for channel-adaptive methods to the class token. (C) DINO-BoC is similar to late-fusion multi-modal strategies and trains on individual channels, randomly drawn (thick borders in A) from the datasets (no cross-channel computation). On inference, the encoder can be used to embed images with arbitrary channel combinations by concatenating channel-wise outputs.
Model
SSL
Network
Channel
Training set
HPA-FOV F1
HPA-FOV F1
JUMP-CP
WTC F1
method
size
adaptive
Protein loc.
Cell type
Accuracy
Cell cycle st.
CellDINOv2
DINOv2
ViT-L
✗
HPA-FOV
65.0
89.3
✗
✗
CellDINOv2
DINOv2
ViT-L
✗
JUMP-CP
✗
✗
44.3
✗
CellDINOv1
DINOv1
ViT-L
✗
WTC
✗
✗
✗
82.3
Channel-ViT
DINOv2
ViT-L
✓
ExtendedCHAMMI
57.4 -7.6
90.4 +1.1
39.4 -4.9
87.2 +4.9
DINO HA
DINOv2
ViT-L
✓
ExtendedCHAMMI
61.4 -3.6
91.3 +2.0
45.2 +0.9
91.0 +8.7
Table 1: Comparison of channel-adaptive models trained on the ExtendedCHAMMI dataset and fixed-channel models. The first three rows (CellDINO) are fixed-channel baseline models, separately pre-trained either on the HPA-FOV, JUMP-CP or WTC datasets. Best channel-adaptive results in bold; best results overall with frozen backbones are underlined. Note that the fixed-channel models can only be evaluated on data with matching number of channels.
Class
Budtip
Cell periphery
Budneck
Actin
All
frequency
1.5 %
1.9%
2.4%
3.8%
Microsnoop ( Xun et al., 2024 )
32.1
96.4
43.4
48.0
75.9
DINO BoC
62.1
97.5
72.6
63.7
83.1
Table 2: Comparison to Microsnoop on the Cyclops dataset. DINO BoC dramatically outperforms Microsnoop, especially on the 4 least frequent classes (out of 16).
Model
Channel
Training set
HPA-FOV F1
HPA-FOV F1
JUMP-CP
WTC F1
adaptive
Protein loc.
Cell type
Accuracy
Cell cycle st.
DINO Channel-ViT
✓
HPA-FOV
65.5
90.9
35.3
80.0
DINO HA (Ours)
✓
HPA-FOV
66.7
91.3
37.3
88.9
DINO BoC (Ours)
✓
HPA-FOV
65.2
91.5
40.2
89.8
DINO Channel-ViT
✓
JUMP-CP
29.5
82.0
53.4
81.8
DINO HA (Ours)
✓
JUMP-CP
30.3
85.2
52.0
84.2
Table 3: Cross-dataset generalization of channel-adaptive models . DINO BoC shows superior performance on unseen channel combinations. In-dataset results are shown in gray for reference.
Model
Average OOD
WTC
HPA
CP
Mean
WTC
HPA
CP
Task1
Task2
Task1
Task2
Task3
Task1
Task2
Task3
Task4
FixedChannels
50.0
64.8
59.2
25.9
64.9
64.8
80.7
76.3
42.1
66.0
48.1
23.0
6.6
Depthwise
51.7
65.2
64.4
25.6
68.9
65.2
84.9
81.3
47.5
67.3
47.8
22.4
6.5
TargetParam
49.6
59.0
62.3
27.3
69.5
59.0
83.7
79.4
45.2
71.7
50.8
23.4
7.7
SliceParam
45.7
56.8
54.6
25.6
61.6
56.8
77.0
69.0
40.3
64.6
47.5
22.2
7.1
HyperNet
53.7
66.1
67.1
27.8
72.6
66.1
88.7
85.8
48.3
72.0
51.7
24.7
6.9
Table 4: F1 scores for 1-NN search on the CHAMMI test set. The models were pre-trained on the CHAMMI train split. Lines 1-6 report the results of Chen et al. (2024) for CNN-based models trained from scratch in a supervised fashion. Line 1 reports the performance of FixedChannels, that consists on a separate model trained for each fixed channel combination. Lines 2-6 are channel-adaptive models. Lines 7-8 are self-supervised channel-adaptive ViTs. Best results between channel-adaptive self-supervised approaches are in bold.
Model
Average OOD
WTC
HPA
CP
Mean
WTC
HPA
CP
Task 1
Task 2
Task 1
Task 2
Task 3
Task 1
Task 2
Task 3
Task 4
HyperNet
53.7
66.1
67.1
27.8
72.6
66.1
88.7
85.8
48.3
72.0
51.7
24.7
6.9
DINO Channel-ViT
43.6
46.2
55.6
28.9
64.5
46.2
92.1
65.3
45.9
89.0
53.5
21.8
11.3
DINO HA (Ours)
43.2
54.3
46.9
28.4
66.4
54.3
90.1
60.4
33.5
93.6
57.1
15.8
12.1
DINO BoC (Ours)
51.6
79.0
43.0
32.7
79.4
79.0
86.6
59.3
29.6
92.6
57.6
22.1
18.5
Table 5: F1 scores for 1-NN search on the CHAMMI test set. The self-supervised models were pre-trained on the ExtendedCHAMMI dataset. Line 1 presents the best performing supervised baseline (HyperNet), which can only be trained on the annotated subset of CHAMMI.
Average OOD
WTC
HPA
CP
Model
Mean
WTC
HPA
CP
Task1
Task2
Task1
Task2
Task 3
Task1
Task2
Task3
Task4
HyperNet
65.4
85.3
82.9
27.9
87.9
85.3
94.4
92.5
73.2
93.5
51.5
17.3
15.0
DINO Channel-ViT
59.8
66.9
76.7
35.9
83.1
66.9
88.2
84.9
68.4
80.5
54.5
23.3
30.0
DINO HA (Ours)
62.7
76.2
72.4
39.5
88.0
76.2
88.5
82.4
62.4
91.7
61.6
27.5
29.3
DINO BoC (Ours)
67.9
89.2
74.9
39.7
90.5
89.2
88.3
84.7
65.0
90.5
60.5
25.8
32.7
Table 6: F1 scores for a linear probe on CHAMMI test set. Self-supervised models were pre-trained on the ExtendedCHAMMI dataset, while the supervised HyperNet was pre-trained on CHAMMI.
Approach
Architecture
Channel adaptive
Test dataset
Pre-training dataset
mAP
Meter-ML Zhu et al. (2022)
DenseNet-121
✗
NAIP, S2, S1
ImageNet
51.7
LHRS-bot Muhtar et al. (2024)
VLM
✗
NAIP, S2, S1
LHRS-Align-Recap (1.1M images & text)
71.8
VHM Pang et al. (2024)
VLM
✗
NAIP, S2, S1
VersaD (14M images & text)
72.7
DINO Channel ViT
ViT-L
✓
NAIP, S1, S2
NAIP, S1, S2 train (85K)
64.4
DINO BoC (ours)
ViT-L
✓
NAIP, S1, S2
NAIP, S1, S2 train (85K)
78.1
Approach
Architecture
Channel adaptive
Test dataset
Pre-training dataset
mAP
Table 7: DINO BoC outperforms the state-of-the-art models when using all channels on the Meter-ML dataset. Top: AUROC of models trained on NAIP and Sentinel data. Bottom: AUROC of models using only NAIP channels at inference.
Model
Channel adaptive
Batch size × Channels
Top-1 Accuracy
fixed channels
✗
1024 × 3 = 3072
ImageNet1K
81.9
fixed channels
✗
3072 × 3 = 9207
ImageNet1K
83.1
DINO HA
✓
1024 × 3 = 3072
ImageNet1K
78.6
DINO BoC(Ours)
✓
3072 × 1 = 3072
ImageNet1K
82.8
Table 8: Mean accuracy on the ImageNet 1k val set. ViT-L models, linear evaluation.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Overview of different channel-adaptive strategies. (A) Joint-channel-encoding strategy of Channel-ViT ( Bao et al., 2024 ; Bourriez et al., 2024 ) : the image is reshaped into a sequence of single-channel patches and channel embeddings are used to retain channel information. (B) Hierarchical Attention model: a specialized attention mask is used to enforce independent-channel-encoding via channel class tokens xCH , while the global class token xCLS supports inter-channel reasoning. (C) Independent-channel-encoding strategy, note that a common backbone is used. (D) Attention mask of the Hierarchical Attention model.
Dataset
Image type
Size
Channels
Discarded factors
HPA Single Cell
One cell
296670/839612
4
Cell line (HEK 293); protein localization (cytosol, endoplasmic reticulum, nucleoplasm)
Cell line (HEK 293); protein localization (cytosol, endoplasmic reticulum, nucleoplasm)
Appendix
Table 9: Data included on the ExtendedCHAMMI dataset. The third column lists the amount of data that was included on ExtendedCHAMMI over the total size of the data source. For each data source, the number of channels and image type is specified: field-of-view (FOV) images, cropped FOV images containing a smaller number of cells, or images of a single cell. The last column lists which biological or experimental factors were discarded from the original datasets to preserve the OOD characteristic of the CHAMMI tasks.
Model
HCS
Average OOD
WTC
HPA
CP
Mean
WTC
HPA
CP
Task1
Task2
Task1
Task2
Task3
Task1
Task2
Task3
Task4
DINO Channel-ViT
✗
43.6
46.2
55.6
28.9
64.5
46.2
92.1
65.3
45.9
89.0
53.5
21.8
11.3
DINO Channel-ViT
✓
39.9
39.5
51.9
28.4
66.4
39.5
88.5
62.3
41.5
90.2
56.0
22.3
6.9
Appendix
Table 10: Influence of HSC on DINO Channel-ViT models. The models were pre-trained on the ExtendedCHAMMI dataset and evaluated on CHAMMI.
Figure 5: UMAP of the HPA FOV dataset highlighting different factors of variation. UMAP space of the HPA FOV features obtained from the DINO BoC model pre-trained on the same dataset, colored according to (a) cell type and (b) protein localization. In (b) only samples with a single protein localization are displayed. The features are obtained separately for each channel, and then concatenated.
Figure 6: UMAP of the HPA FOV dataset to assess utility of unified feature space. a) UMAP space of the HPA FOV features obtained from the Channel-ViT model pre-trained on the same dataset, comparing features computed when the model sees all four channels of the dataset, in blue, and features computed with only two channels (microtobules and protein), in orange. b) Same UMAP as in (a) but filtered for images with only one protein localization and colored according to them.
Training
HPA-FOV F1
HPA-FOV F1
Accuracy
WTC F1
set
Protein loc.
Cell type
on JUMP-CP
Cell cycle st.
ExtendedCHAMMI
61.7
91.1
45.2
90.5
minus WTC
60.4 - 1.3
91.1 - 0.0
43.7 - 1.5
90.9 + 0.4
minus Cell painting
60.2 - 1.5
91.1 - 0.0
44.1 - 1.1
89.8 - 0.7
minus HPA (FOV, single cell)
41.7 -20.0
89.9 - 1.2
46.9 + 1.7
92.3 + 1.8
minus OpenCell
60.9 - 0.2
91.0 - 0.1
44.2 - 1.0
90.0 - 0.5
Appendix
Table 11: Ablation removing one dataset from the ExtendedCHAMMI dataset. We report the linear evaluation results for DINO BoC.
JUMP-CP 5 channels
JUMP-CP 8 channels
Channel ViT
39.5
✗
DINO HA
45.2
51.4
DINO BoC
45.2
51.6
Appendix
Table 12: Mean accuracy on the JUMP-CP dataset with models pre-trained on ExtendedCHAMMI with a maximum of five channels.
Feature
Average OOD
WTC
HPA
CP
Model
dimension
Mean
WTC
HPA
CP
Task1
Task2
Task1
Task2
Task 3
Task1
Task2
Task3
Task4
DINO Channel-ViT
D
59.8
66.9
76.7
35.9
83.1
66.9
88.2
84.9
68.4
80.5
54.5
23.3
30.0
DINO BoC
D
65.4
86.7
67.9
41.5
89.4
86.7
82.9
79.1
56.7
83.8
61.2
26.6
36.8
DINO Channel-ViT
KD
63.2
74.2
77.9
37.4
86.4
74.2
90.2
86.2
69.7
83.3
56.5
24.4
31.3
DINO BoC
KD
67.9
89.2
74.9
39.7
90.5
89.2
88.3
84.7
65.0
90.5
60.5
25.8
32.7
Appendix
Table 13: F1 scores for a linear probe on CHAMMI test set: Ablation with similar embedding size.
Number of pretraining iterations
AUROC
Channel ViT
34K
69.3
DINO HA
34K
77.3
DINO BoC
34K
78.7
DINO BoC
4×34K
81.9
Appendix
Table 14: AUROC on the Meter-ML dataset in function of number of pretraining iterations . Using NAIP images only.
High-content multi-channel imaging, from Cell Painting assays to remote sensing, is central to modern scientific pipelines. Because these channels are highly heterogeneous and experimental configurations frequently evolve, a practical vision backbone must be fundamentally channel-agnostic. Current channel-adaptive vision transformers rely on global self-attention with learnable channel embeddings, which can be computed only for previously known channels. Attempts to overcome this limitation of new channels either discard the embeddings, losing channel identity, or process each channel independently, thereby eliminating critical cross-channel interactions. To address this, we introduce ChanSFormer, an efficient Vision Transformer that replaces rigid embeddings with disentangled spatial-channel attention and individual channel CLS tokens. This architecture natively preserves both distinct channel identities and cross-channel information flow. Furthermore, the individual channel class token design unlocks more informative feature representations for self-supervised learning and enables representation-based channel sampling while remaining channel-agnostic. We evaluate ChanSFormer on two biology multi-channel datasets, CHAMMI, JUMP-CP and a satellite dataset, So2Sat. Experimental results show that ChanSFormer outperforms state-of-the-art methods by 3.93%-4.27% on JUMP-CP in classification accuracy. It also demonstrates exceptional cross-dataset transferability under a self-supervised setting, outperforming previous methods by up to 2.72% in accuracy in the same classification tasks. Furthermore, the disentangled attention reduces the quadratic complexity of channel-by-spatial sequence length of the global attention baseline, thus improving throughput by 55%-313%. Code is available at https://github.com/cvlab-stonybrook/ChanSFormer.
Background: Task-specific microscopy datasets are often small, making it difficult to train deep learning models that learn robust features. While self-supervised learning (SSL) has shown promise through pretraining on large, domain-specific datasets, generalizability across datasets with differing staining protocols and channel configurations remains underexplored. We investigated the generalizability of SSL models pretrained on ImageNet-1k and HPA FOV, evaluating their embeddings on OpenCell with and without fine-tuning, two channel-mismatch strategies, and varying fine-tuning data fractions. We additionally analyzed single-cell embeddings on a labeled OpenCell subset. Result: DINO-based ViT backbones pretrained on HPA FOV or ImageNet-1k transfer well to OpenCell even without fine-tuning. Fine-tuning further improved performance to 0.704 ± 0.027 (17 classes, multi-class labels). At the single-cell level, the HPA single-cell-pretrained model achieved the highest k-nearest neighbor performance across all neighborhood sizes (macro F1≥ 0.515). Conclusion: SSL methods like DINO, pretrained on large domain-relevant datasets, enable effective use of deep learning features for fine-tuning on small, task-specific microscopy datasets.
Ben Isselmann, Dilara Göksu, Heinz Neumann +1
Department of Mathematics and Natural Sciences, Hochschule Darmstadt, Schoefferstraße 3, Darmstadt, 64295, Hessen, Germany. · Department of Chemical Engineering and Biotechnology, Hochschule Darmstadt, Stephanstrasse 7, Darmstadt, 64295, Hessen, Germany. · European University of Technology, European Union. +1
Scientific image segmentation methods rely on extensive annotation and task-specific training, limiting adaptation across imaging modalities and experimental conditions. We present a training-free, one-shot framework that specializes vision foundation models using a single annotated reference image. The framework combines DINOv3 representations with background-adaptive feature orthogonalization to suppress artifact-related feature directions, after which cosine similarity localizes candidate regions for SAM segmentation. We evaluate the framework on red-blood-cell microscopy, structured-illumination pool boiling, and chest radiography. Relative to the strongest baseline, the proposed method improves mean IoU by 5.91% and 78.62% on the microscopy and pool-boiling datasets, respectively, while achieving comparable performance on chest radiographs. These results demonstrate that one-shot reference conditioning can adapt general-purpose vision models to specialized scientific segmentation tasks.
Tejaswi V. Panchagnula, Allison M. Davis, Fengqing Zhu
Elmore Family School of Electrical and Computer Engineering Purdue University West Lafayette, Indiana, USA 47907