cs.CVApr 12, 2026

Self-supervised Pretraining of Cell Segmentation Models

Authors: Kaden Stillwagon, Alexandra Dunnum VandeLoo, Benjamin Magondu, Craig R. Forest

Organizations: School of Computer Science, Georgia Institute of Technology · School of Materials Science and Engineering, Georgia Institute of Technology · Department of Biomedical Engineering, Georgia Institute of Technology · School of Mechanical Engineering, Georgia Institute of Technology

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Instance segmentation enables the analysis of spatial and temporal properties of cells in microscopy images by identifying the pixels belonging to each cell. However, progress is constrained by the scarcity of high-quality labeled microscopy datasets. Many recent approaches address this challenge by initializing models with segmentation-pretrained weights from large-scale natural-image models such as Segment Anything Model (SAM). However, representations learned from natural images often encode objectness and texture priors that are poorly aligned with microscopy data, leading to degraded performance under domain shift. We propose DINOCell, a self-supervised framework for cell instance segmentation that leverages representations from DINOv2 and adapts them to microscopy through continued self-supervised training on unlabeled cell images prior to supervised fine-tuning. On the LIVECell benchmark, DINOCell achieves a SEG score of 0.784, improving by 10.42% over leading SAM-based models, and demonstrates strong zero-shot performance on three out-of-distribution microscopy datasets. These results highlight the benefits of domain-adapted self-supervised pretraining for robust cell segmentation.

Figures & tables

Figure 1: DINOCell Model Architecture. Schematic of the DINOCell model architecture containing a DINOv2-pretrained Oquab et al. (2024) ViT-B Dosovitskiy et al. (2021) encoder that has been domain-adapted on microscopy images, a lightweight upsampling decoder, and convolutional objective prediction heads. The model outputs are combined and decoded using gradient-tracking to produce instance segmentation masks.
Figure 2: Three Stage Training Process of the DINOCell Model.(a) Stage 1: A ViT-B Dosovitskiy et al. (2021) image encoder is pretrained on 1.2 billion images using DINOv2 Oquab et al. (2024) , producing strong object- and part-aware visual embeddings. DINOCell is initialized with these pretrained weights. (b) Stage 2 (Domain Adaptation): The encoder is further trained with DINOv2 on a curated dataset of 130k unlabeled microscopy images to adapt the representation to cell morphology. (c) Stage 3 (Fine-tuning): The full DINOCell model is trained for cell segmentation with the encoder unfrozen. The model predicts flow fields (dx, dy) Stringer et al. (2021) and cell probability from an input image. A mean squared error loss is computed between predictions and ground truth, reweighted using an ignore mask, and backpropagated to update model parameters.
Figure 3: Evolution of Image Encoder Representations through DINOCell Training. We compute a PCA between image encoder patches embeddings and show their first three components where each component is mapped to a different color channel (red, green, blue). This is done for four images from different datasets (LIVECell Edlund et al. (2021) , Glioma C6 Malashin et al. (2025) , Cellpose cyto Stringer et al. (2021) , and cpg0024 Weisbart et al. (2024) ) and at each stage of DINOCell’s three stage training process. Clear, per-object structure emerges showing an evolutionary path from initialization to fine-tuning.
Figure 4: Effects of pretraining (PT) and domain adaptation (DA) on model performance and encoder representations. Across epochs, we find that there is a significant effect of self-supervised pretraining on both model performance and encoder representations, while effects of domain adaptation are less significant. (a) Encoder representations of a single image from the cpg0024 Weisbart et al. (2024) subset of the Cellpainting Gallery Weisbart et al. (2024) across epochs using different combinations of PT and DA show that PT yields highly structured representations after 5+ epochs, making segmentation ostensibly appear trivial. (b) Segmentation of cells from a single image of the LIVECell Edlund et al. (2021) test set across epochs using different combinations of PT and DA show that PT yields highly accurate segmentations and much faster model convergence compared to no-PT. (c) MMA scores across both test and zero-shot datasets show that PT, with and without DA, enhances both in-distribution performance and out-of-distribution generalization. (d) During training, validation loss curves reveal that PT converges much faster than no-PT.
Figure 5: DINOCell per-dataset performance on the test split of the Finetuning Dataset, shown for both (a) aggregate performance, and (b) representative images. (a) Maximum Matching Accuracy (MMA) on the test split of each dataset used for fine-tuning. MMA evaluates how well predicted instances correspond to ground-truth cells (maximum score = 1.0). DINOCell achieves high performance on most datasets, indicating strong detection and segmentation accuracy across a range of cell morphologies and imaging conditions. (b) Representative images from each dataset with predicted segmentations overlaid. These qualitative examples illustrate DINOCell’s ability to accurately delineate cells with diverse shapes, densities, and imaging modalities.
Figure 6: DINOCell outperforms Cellpose-SAM , SAMCell-LIVECell, and SAMCell-cyto both visually and quantitatively across all metrics. (a) Qualitative results are shown using segmentation outputs for representative images across the LIVECell Edlund et al. (2021) and zero-shot datasets. On all data sets, the cells appear visually to be thoroughly and accurately segmented as compared to models from prior publications on the same images. Specifically, cells are not missed or clumped and artifacts are absent, relatively speaking. (b) Intermediate Principal Component Analysis (PCA) of the encoder output. PCA of DINOCell encodings reveal per-object features with well-defined boundaries that do not emerge as clearly or completely in other models. (c) Maximum Matching Accuracy (MMA), Segmentation Accuracy Metric (SEG), and Detection Accuracy Metric (DET) shown for DINOCell, Cellpose-SAM, and SAMCell-LIVECell model outputs on the LIVECell testing. Zero-shot performance is also shown for all prior models with the addition of the SAMCell-cyto model. DINOCell displays dominance with the highest MMA, SEG, and DET scores across all datasets.
More in this topic

Instance Segmentation

Instance segmentation identifies each object and assigns it a separate pixel-level mask in images or video.

See all 159 papers

Explore similar work

CardsList

  1. Revisiting foundation models for cell instance segmentation

    Mar 18, 2026Anwai Archit, Constantin PapeInstance SegmentationMicroscopy Image Analysis

    Georg-August-University Göttingen, Institute of Computer Science · CAIMed - Lower Saxony Center for AI & Causal Methods in Medicine, Göttingen · Cluster of Excellence Multiscale Bioimaging (MBExC), Georg-August-University Göttingen

  2. Boundary-Aware Instance Segmentation in Microscopy Imaging

    Mar 22, 2026Thomas Mendelson, Joshua Francois, Galit Lahav +1Instance SegmentationBoundary-Aware Segmentation

    The School of Electrical and Computer Engineering, Ben-Gurion University of the Negev · Department of Systems Biology, Harvard Medical School

  3. Using Deep Learning Models Pretrained by Self-Supervised Learning for Protein Localization

    Apr 13, 2026Ben Isselmann, Dilara Göksu, Heinz Neumann +1Supervised Fine-TuningSelf-Supervised Pre-Training

    Department of Mathematics and Natural Sciences, Hochschule Darmstadt, Schoefferstraße 3, Darmstadt, 64295, Hessen, Germany. · Department of Chemical Engineering and Biotechnology, Hochschule Darmstadt, Stephanstrasse 7, Darmstadt, 64295, Hessen, Germany. · European University of Technology, European Union. +1