Unsupervised Learning of Cell Instances with Generative Routing Pyramids

Ziwen Liu; Martin Weigert

cs.CVAug 17, 2026

Unsupervised Learning of Cell Instances with Generative Routing Pyramids

Authors: Ziwen Liu, Martin Weigert

Organizations: Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI), Technische Universität Dresden, Germany

Abstract

Identifying and representing object instances such as cells or nuclei is a common task in microscopy image analysis. Established machine learning workflows typically use supervised detection or segmentation followed by feature extraction or classification, which requires manual annotations and treats instance segmentation and cell representation as separate stages. We describe a new unsupervised method for cell instance segmentation and phenotypic classification from unlabeled microscopy images. Our method is based on reconstructing each image using a coarse-to-fine routing pyramid that associates pixels with spatially sparse latent sources. The resulting pixel-to-latent associations yield instance masks, while the source latents encode cell morphology. We demonstrate competitive performance in instance segmentation across diverse cell morphologies and imaging modalities, as well as generative modeling of cellular phenotypes under perturbations. Source code and checkpoints are available at https://github.com/weigertlab/routing-pyramids.

Figures & tables

Figure 1: Overview of Generative Routing Pyramids. a) From an input image, the encoder predicts a latent field, which is then used to generate pyramidal (coarse-to-fine) flows to reconstruct the image. b) Per-layer flow and routed feature maps. The flows are shown as HSV encoding, where hue is the flow direction, saturation is the flow magnitude, and value is the routed probability mass. The features are shown as RGB values representing the first three principle components.

Detection	Reconstruction	Method
$O(KN)$ , sequential	$O(KN)$	MONet [ 5 ]
$O(TKN)$	$O(TKN)$	IODINE [ 9 ]
$O(TKN)$	$O(KN)$	Slot Attention [ 15 ]
$O(N^{2}+TKN)$	$O(N^{2}+KN)$	DINOSAUR [ 24 ]
$O(N)$	Up to $O(N^{2})$	SPACE [ 13 ]
$O(RN)$	$O(IN\log N+KN)$	Cellulus $\dagger$ [ 28 ]

Table 1: Inference scaling of related methods.

N

is the number of pixels, with feature-token count linear in

N

;

K

is the object, slot, or seed component count;

T

is the number of iterative inference steps;

R

is the number of noisy inference passes; and

I

is the number of mean-shift iterations. Fixed network-depth and local-neighborhood factors are omitted.

\dagger

Cellulus does not reconstruct the image, its instance-segmentation cost is shown instead.

Dataset	Cell type	Visible structure	Modality
Allen	Human induced pluripotent stem cells (hiPSCs)	Nuclear envelope (Lamin B1-mEGFP)	Fluorescence
Fluo-HeLa	HeLa cells	Nuclei (H2B-GFP)	Fluorescence
PhC-PSC	Rat pancreatic stem cells	Cell body	Phase contrast

Table 2: Biological and imaging properties of the instance-segmentation datasets.

Dataset	Method	Type	$F_{1}^{[0.5]}$	$F_{1}^{[0.7]}$	$F_{1}^{[0.9]}$	PQ
Allen	Cellpose-SAM	supervised	0.977	0.964	0.256	0.859
	Otsu (size 42)	non-parametric	0.906	0.861	0.149	0.773
	Cellulus (size 52)	unsupervised	0.908	0.854	0.325	0.787
	Routing Pyramids (ours)	unsupervised	0.960	0.943	0.649	0.867
Fluo-HeLa	Cellpose-SAM	supervised	0.915	0.892	0.752	0.843
	Otsu (size 26)	non-parametric	0.768	0.642	0.190	0.623

Table 3: Instance segmentation on Allen, Fluo-HeLa, and PhC-PSC datasets. We compare the unsupervised methods Routing Pyramids, Cellulus, and Otsu with the supervised Cellpose-SAM reference. We report instance-level

F_{1}

scores at IoU thresholds of 0.5, 0.7, and 0.9, and panoptic quality (PQ). Higher values are better. Bold and underlined values indicate the best and second-best result for each dataset and metric, respectively.

Panel of Figure 2 — Figure 2: Instance segmentation on Allen (top), Fluo-HeLa (middle), and PhC-PSC (bottom). From left to right, columns show the input image, ground-truth instances (GT), the supervised Cellpose-SAM reference, the unsupervised Otsu and Cellulus baselines, and our method. Instances are indicated by distinct colors. The lower row per dataset shows two inset regions marked by white boxes in the input image.

Figure 3: Object latents encode perturbation-induced phenotypes. a) UMAP of the object latent codes on the series dilution images. b) PCA and GMM fit of the training split applied to the control object latents. Stars denote the means, and elliptical contours show the 95% confidence intervals (CI) for the GMM. Legend shows the ratios for each component. c) Positive (GMM k=2) probability of objects in the series dilution images. Lines indicate mean over objects, and the bands indicate 95% CI.

Figure 4: Generated (a) and retrieved (b) $32\times 32$ crops from the series dilution images. Top two rows show GFP (green) and DNA (magenta) channels corresponding to the "negative control" GMM component, while the bottom two rows correspond to the "positive control" component. a) Generated crops by sampling from the two Gaussian distributions. b) Retrieved crops by nearest neighbor search with component means in the observed object latent set.

Instance Segmentation

Instance segmentation identifies each object and assigns it a separate pixel-level mask in images or video.

See all 159 papers

Revisiting foundation models for cell instance segmentation
Mar 18, 2026Anwai Archit, Constantin PapeInstance Segmentation Microscopy Image Analysis
Georg-August-University Göttingen, Institute of Computer Science · CAIMed - Lower Saxony Center for AI & Causal Methods in Medicine, Göttingen · Cluster of Excellence Multiscale Bioimaging (MBExC), Georg-August-University Göttingen
Cell Instance Segmentation via Multi-Task Image-to-Image Schrödinger Bridge
Apr 14, 2026Hayato Inoue, Shota Harada, Shumpei Takezaki +1Instance Segmentation Histopathology Image Segmentation
Dept. of Information Science and Technology, Kyushu University, Fukuoka, Japan
Boundary-Aware Instance Segmentation in Microscopy Imaging
Mar 22, 2026Thomas Mendelson, Joshua Francois, Galit Lahav +1Instance Segmentation Boundary-Aware Segmentation
The School of Electrical and Computer Engineering, Ben-Gurion University of the Negev · Department of Systems Biology, Harvard Medical School

Mar 18, 2026J/K move · Enter open · S save

Revisiting foundation models for cell instance segmentation

Anwai Archit, Constantin Pape

Georg-August-University Göttingen, Institute of Computer Science · CAIMed - Lower Saxony Center for AI & Causal Methods in Medicine, Göttingen · Cluster of Excellence Multiscale Bioimaging (MBExC), Georg-August-University Göttingen

Instance Segmentation Microscopy Image Analysis

Open paper AskPDF arXiv

Cell segmentation is a fundamental task in microscopy image analysis. Several foundation models for cell segmentation have been introduced, virtually all of them are extensions of Segment Anything Model (SAM), improving it for microscopy data. Recently, SAM2 and SAM3 have been published, further improving and extending the capabilities of general-purpose segmentation foundation models. Here, we comprehensively evaluate foundation models for cell segmentation (CellPoseSAM, CellSAM,

μ

SAM) and for general-purpose segmentation (SAM, SAM2, SAM3) on a diverse set of (light) microscopy datasets, for tasks including cell, nucleus and organoid segmentation. Furthermore, we introduce a new instance segmentation strategy called automatic prompt generation (APG) that can be used to further improve SAM-based microscopy foundation models. APG consistently improves segmentation results for

μ

SAM, which is used as the base model, and is competitive with the state-of-the-art model CellPoseSAM. Moreover, our work provides important lessons for adaptation strategies of SAM-style models to microscopy and provides a strategy for creating even more powerful microscopy foundation models. Our code is publicly available at https://github.com/computational-cell-analytics/micro-sam.

Abstract

Instance Segmentation

Explore similar work

Revisiting foundation models for cell instance segmentation

Cell Instance Segmentation via Multi-Task Image-to-Image Schrödinger Bridge

Boundary-Aware Instance Segmentation in Microscopy Imaging