Revisiting foundation models for cell instance segmentation

Anwai Archit; Constantin Pape

cs.CVMar 18, 2026

Revisiting foundation models for cell instance segmentation

Authors: Anwai Archit, Constantin Pape

Organizations: Georg-August-University Göttingen, Institute of Computer Science · CAIMed - Lower Saxony Center for AI & Causal Methods in Medicine, Göttingen · Cluster of Excellence Multiscale Bioimaging (MBExC), Georg-August-University Göttingen

Abstract

Cell segmentation is a fundamental task in microscopy image analysis. Several foundation models for cell segmentation have been introduced, virtually all of them are extensions of Segment Anything Model (SAM), improving it for microscopy data. Recently, SAM2 and SAM3 have been published, further improving and extending the capabilities of general-purpose segmentation foundation models. Here, we comprehensively evaluate foundation models for cell segmentation (CellPoseSAM, CellSAM, $μ$ SAM) and for general-purpose segmentation (SAM, SAM2, SAM3) on a diverse set of (light) microscopy datasets, for tasks including cell, nucleus and organoid segmentation. Furthermore, we introduce a new instance segmentation strategy called automatic prompt generation (APG) that can be used to further improve SAM-based microscopy foundation models. APG consistently improves segmentation results for $μ$ SAM, which is used as the base model, and is competitive with the state-of-the-art model CellPoseSAM. Moreover, our work provides important lessons for adaptation strategies of SAM-style models to microscopy and provides a strategy for creating even more powerful microscopy foundation models. Our code is publicly available at https://github.com/computational-cell-analytics/micro-sam.

Figures & tables

Dataset	LIVECell	CellPose	DSB	PanNuke
Modality	Label Free	Fluorescence	Fluorescence	Histopathology
Task	Cell	Cell	Nucleus	Nucleus

Text Prompts
cell(s)	✓ ( 0.331 ) $\mid$ ✓ (0.311)	✓ (0.299) $\mid$ ✓ (0.231)	✓ (0.366) $\mid$ ✓ (0.386)	✓ (0.341) $\mid$ ✓ (0.159)
nucleus (nuclei)	$-\mid-$	$-\mid-$	${\color[rgb]{1,0,0}\times}\mid$ ✓ (0.085)	${\color[rgb]{1,0,0}\times}\mid{\color[rgb]{1,0,0}\times}$

Table 1: SAM3 results for text prompt-based microscopy segmentation. Datasets marked in italic font are part of SAM3’s training data, datasets marked in bold font are not. The results for best performing text prompts per dataset are marked in bold . All reported scores are mean segmentation accuracy for the entire dataset. Prompts resulting in a score

<=

0.001 are marked as “

\times

”. Prompts that are not applicable to the given segmentation task are marked as “

-

”.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Imaging Modality	Input Dimensions
LIVECell Edlund et al. (2021)	Phase Contrast	2D
OmniPose Cutler et al. (2022)	Phase Contrast & Brightfield	2D
DeepBacs Spahn et al. (2022)	Brightfield & Fluorescence	2D
Usiigaci Tsai et al. (2019)	Phase Contrast	2D
Vicar Vicar et al. (2021)	Quantitative Phase	2D
TOIAM Seiffarth et al. (2024)	Phase Contrast	2D+T

Appendix

Table 2: Description of the different datasets used in our study. For the 3D datasets / 2D+T datasets, we evaluate over individual slices / frames. For the histopathology datasets, the imaging modality describes the staining protocol (in italics) for the images.

Method	Label-Free (Cell)	Fluorescence (Cell)	Fluorescence (Nucleus)	Histopathology (Nucleus)
AMG (SAM)	0.106	0.091	0.283	0.200
AIS	0.480	0.347	0.513	0.390
SAM3	0.269	0.143	0.255	0.331
CellPose 3	0.424	0.218	0.438	0.155
CellPoseSAM	0.544	0.363	0.483	0.418
CellSAM	0.380	0.272	0.386	0.277

Appendix

Table 3: Mean Segmentation Accuracy (mSA) averaged over all datasets for each modality. The best / second / third ranking methods are shown in bold / underline / italics . For histopathology, AIS / APG correspond to the PathoSAM, and

\mu

SAM for others.

Dataset	AMG (SAM)	AIS ( $\mu$ SAM)	SAM3	CellPose 3	CellPoseSAM	CellSAM	APG ( $\mu$ SAM)
LIVECell	0.075	0.415	0.331	0.414	0.444	0.098	0.437
OmniPose	0.137	0.599	0.356	0.468	0.644	0.531	0.651
DeepBacs	0.057	0.497	0.157	0.455	0.612	0.441	0.586
Usiigaci	0.051	0.353	0.362	0.291	0.445	0.167	0.383
Vicar	0.115	0.411	0.086	0.338	0.458	0.426	0.417
TOIAM	0.009	0.387	0.027	0.837	0.898	0.631	0.701

Appendix

Table 4: Quantitative results on label-free microscopy datasets for cell segmentation: mean segmentation accuracy (mSA) per dataset and method. For each dataset, the best / second / third ranking methods are shown in bold / underline / italics .

Dataset	AMG (SAM)	AIS ( $\mu$ SAM)	SAM3	CellPose 3	CellPoseSAM	CellSAM	APG ( $\mu$ SAM)
TissueNet	0.069	0.329	0.121	0.154	0.475	0.345	0.324
CellPose	0.147	0.383	0.299	0.431	0.566	0.413	0.416
PlantSeg (Root)	0.091	0.507	0.067	0.076	0.161	0.096	0.489
PlantSeg (Ovules)	0.135	0.341	0.184	0.266	0.331	0.333	0.325
PNAS Arabidopsis	0.145	0.461	0.241	0.411	0.471	0.459	0.462
Covid-IF	0.007	0.317	0.005	0.161	0.333	0.154	0.296

Appendix

Table 5: Quantitative results on fluorescence microscopy datasets for cell segmentation: mean segmentation accuracy per dataset and method. For each dataset, the best / second / third ranking methods are shown in bold / underline / italics .

Dataset	AMG (SAM)	AIS ( $\mu$ SAM)	SAM3	CellPose 3	CellPoseSAM	CellSAM	APG ( $\mu$ SAM)
DSB	0.331	0.654	0.367	0.484	0.656	0.634	0.665
U20S	0.258	0.786	0.674	0.787	0.787	0.673	0.787
Arvidsson	0.416	0.594	0.297	0.611	0.484	0.434	0.567
BitDepth NucSeg	0.224	0.323	0.182	0.302	0.377	0.168	0.346
IFNuclei	0.293	0.729	0.301	0.404	0.728	0.589	0.749
Dynamic- NuclearNet	0.298	0.592	0.346	0.512	0.379	0.455	0.584

Appendix

Table 6: Quantitative results on fluorescence microscopy datasets for nucleus segmentation: mean segmentation accuracy per dataset and method. For each dataset, the best / second / third ranking methods are shown in bold / underline / italics .

Dataset	AMG (SAM)	AIS (PathoSAM)	SAM3	CellPose 3	CellPoseSAM	CellSAM	APG (PathoSAM)
PanNuke	0.199	0.467	0.341	0.152	0.342	0.244	0.478
IHC TMA	0.236	0.264	0.435	0.297	0.452	0.333	0.272
LynSec	0.233	0.291	0.157	0.163	0.561	0.213	0.314
MoNuSeg	0.182	0.395	0.345	0.125	0.373	0.302	0.394
NuInsSeg	0.161	0.238	0.312	0.144	0.349	0.229	0.239
PUMA	0.232	0.707	0.415	0.101	0.501	0.294	0.712

Appendix

Table 7: Quantitative results on histopathology datasets for nucleus segmentation: mean segmentation accuracy per dataset and method. For each dataset, the best / second / third ranking methods are shown in bold / underline / italics .

Instance Segmentation

Instance segmentation identifies each object and assigns it a separate pixel-level mask in images or video.

See all 159 papers

Boundary-Aware Instance Segmentation in Microscopy Imaging
Mar 22, 2026Thomas Mendelson, Joshua Francois, Galit Lahav +1Instance Segmentation Boundary-Aware Segmentation
The School of Electrical and Computer Engineering, Ben-Gurion University of the Negev · Department of Systems Biology, Harvard Medical School
One Click per Cell Type Suffices: Training-free Group Interaction for Cell Instance Segmentation
May 28, 2026Sanghyun Jo, Seo Jin Lee, Seohyung Hong +4Interactive Medical Image Segmentation Cross-Modality Medical Image Segmentation
1OGQ, Korea · 2Seoul National University, Korea · 3LG CNS, Korea
Evaluating Vision Foundation Models for Pixel and Object Classification in Microscopy
Mar 20, 2026Carolin Teuber, Anwai Archit, Tobias Boothe +3Vision Foundation Models Semantic Segmentation
Georg-August-University Göttingen, Institute of Computer Science · Department of Tissue Dynamics and Regeneration, Max Planck Institute for Multidisciplinary Sciences, Göttingen · Georg-August-University Göttingen, Faculty of Biology and Psychology +3

Mar 22, 2026J/K move · Enter open · S save

Boundary-Aware Instance Segmentation in Microscopy Imaging

Thomas Mendelson, Joshua Francois, Galit Lahav, Tammy Riklin-Raviv

The School of Electrical and Computer Engineering, Ben-Gurion University of the Negev · Department of Systems Biology, Harvard Medical School

Instance Segmentation Boundary-Aware Segmentation Microscopy Image Analysis

Open paper AskPDF arXiv

Accurate delineation of individual cells in microscopy videos is essential for studying cellular dynamics, yet separating touching or overlapping instances remains a persistent challenge. Although foundation-model for segmentation such as SAM have broadened the accessibility of image segmentation, they still struggle to separate nearby cell instances in dense microscopy scenes without extensive prompting. We propose a prompt-free, boundary-aware instance segmentation framework that predicts signed distance functions (SDFs) instead of binary masks, enabling smooth and geometry-consistent modeling of cell contours. A learned sigmoid mapping converts SDFs into probability maps, yielding sharp boundary localization and robust separation of adjacent instances. Training is guided by a unified Modified Hausdorff Distance (MHD) loss that integrates region- and boundary-based terms. Evaluations on both public and private high-throughput microscopy datasets demonstrate improved boundary accuracy and instance-level performance compared to recent SAM-based and foundation-model approaches. Source code is available at: https://github.com/ThomasMendelson/BAISeg.git

Abstract

Instance Segmentation

Explore similar work

Boundary-Aware Instance Segmentation in Microscopy Imaging

One Click per Cell Type Suffices: Training-free Group Interaction for Cell Instance Segmentation

Evaluating Vision Foundation Models for Pixel and Object Classification in Microscopy