Boundary-Aware Instance Segmentation in Microscopy Imaging

Thomas Mendelson; Joshua Francois; Galit Lahav; Tammy Riklin-Raviv

cs.CVMar 22, 2026

Boundary-Aware Instance Segmentation in Microscopy Imaging

Authors: Thomas Mendelson, Joshua Francois, Galit Lahav, Tammy Riklin-Raviv

Organizations: The School of Electrical and Computer Engineering, Ben-Gurion University of the Negev · Department of Systems Biology, Harvard Medical School

Abstract

Accurate delineation of individual cells in microscopy videos is essential for studying cellular dynamics, yet separating touching or overlapping instances remains a persistent challenge. Although foundation-model for segmentation such as SAM have broadened the accessibility of image segmentation, they still struggle to separate nearby cell instances in dense microscopy scenes without extensive prompting. We propose a prompt-free, boundary-aware instance segmentation framework that predicts signed distance functions (SDFs) instead of binary masks, enabling smooth and geometry-consistent modeling of cell contours. A learned sigmoid mapping converts SDFs into probability maps, yielding sharp boundary localization and robust separation of adjacent instances. Training is guided by a unified Modified Hausdorff Distance (MHD) loss that integrates region- and boundary-based terms. Evaluations on both public and private high-throughput microscopy datasets demonstrate improved boundary accuracy and instance-level performance compared to recent SAM-based and foundation-model approaches. Source code is available at: https://github.com/ThomasMendelson/BAISeg.git

Figures & tables

Panel of Figure 1 — Figure 1: Examples illustrating the limitations of conventional pixel-wise loss functions for cell instance segmentation. (a) Original input image $I$ ; (b) Zoomed-in example from (a), showing an example edge prediction (in red) highlighting boundary ambiguity; (c) Binary ground-truth segmentation map $B_{\mathrm{GT}}$ ; (d) Ground-truth instance label map $\Gamma_{\mathrm{GT}}$ . Pixel-wise losses fail to capture subtle boundary discrepancies in densely packed cellular regions.

Panel of Figure 2 — Figure 2: Visualization of the boundary-based loss computation. The binary GT segmentation $S_{\mathrm{GT}}$ is derived from the GT label maps, where touching cells are separated in a preprocessing stage using known cell boundaries. The network predicts the signed distance function (SDF) of the cell segmentation map, $\phi_{\mathrm{PRED}}$ ; the corresponding binary mask $S_{\mathrm{PRED}}$ is shown for reference. For comparison, the GT SDF, $\phi_{\mathrm{GT}}$ , is also displayed. Soft boundary maps, $B_{\mathrm{GT}}$ and $B_{\mathrm{PRED}}$ , are obtained as $\sigma(\phi)\,\sigma(-\phi)$ . The rightmost images illustrate Eq. 8 (top) and Eq. 9 (bottom). Boundary-based loss values are computed by summing over the respective image domains.

$\mathcal{L}_{\mathrm{CE}}$	$\mathcal{L}_{\mathrm{LSE}}$	$\mathcal{L}_{\mathrm{LMHD}}$	$\mathcal{L}_{\mathrm{RMHD}}$	GOWT1	MSC
✓	✓			.901(.180)	.759(.123)
✓	✓		✓	.882(.045)	.634(.147)
✓	✓	✓		.899(.020)	.689(.123)
✓	✓	✓	✓	.944(.018)	.775(.102)

Table 1: SEG scores for different loss combinations.

Method	MCF7	SIM+	HeLa	GOWT1	MSC
UNet	.492(.120)	.629(.254)	.774(.026)	.904(.025)	.420(.194)
SAM ( $p{=}32$ )	.411(.177)	.508(.256)	.795(.042)	.943(.019)	.229(.140)
SAM ( $p{=}64$ )	.431(.182)	.518(.243)	.917(.012)	.938(.015)	.253(.141)
$\mu$ SAM	.517(.184)	.789(.104) ∗	.867(.022) ∗	.843(.030) ∗	.713(.107) ∗
Cellpose	.521(.106)	.597(.206)	.849(.044)	.933(.026)	.554(.175)
Ours	.798(.093)	.788(.077)	.892(.030)	.944(.018)	.775(.102)

Table 2: SEG (

\uparrow

) performance across all datasets. Mean(std) over test images. A dashed line separates the 3D spheroid dataset from the four CSB datasets. For SAM (ViT_B),

p{=}32

and

p{=}64

indicate prompt-grid densities of

32{\times}32

and

64{\times}64

\mu

SAM results ( ∗ ) are computed on 20 test images due to high inference cost.

Panel of Figure 3 — Figure 3: Qualitative segmentation results Top four images - CSB datasets, bottom two rows - representative 2D slices from the private spheroid data.

Instance Segmentation

Instance segmentation identifies each object and assigns it a separate pixel-level mask in images or video.

See all 159 papers

Revisiting foundation models for cell instance segmentation
Mar 18, 2026Anwai Archit, Constantin PapeInstance Segmentation Microscopy Image Analysis
Georg-August-University Göttingen, Institute of Computer Science · CAIMed - Lower Saxony Center for AI & Causal Methods in Medicine, Göttingen · Cluster of Excellence Multiscale Bioimaging (MBExC), Georg-August-University Göttingen
QCell: Recombining and Aligning Cell Queries for Overlapping Instance Segmentation
Aug 29, 2026Yaroslav Prytula, Anton Popov, Dmytro FishmanInstance Segmentation Medical Image Segmentation
Institute of Computer Science University of Tartu, Tartu, Estonia · Faculty of Applied Sciences Ukrainian Catholic University, Lviv, Ukraine · Department of Electronic Engineering, Micro- and Biomedical Electronics Igor Sikorsky Kyiv Polytechnic Institute, Kyiv, Ukraine +2
Self-supervised Pretraining of Cell Segmentation Models
Apr 12, 2026Kaden Stillwagon, Alexandra Dunnum VandeLoo, Benjamin Magondu +1Instance Segmentation Microscopy Image Analysis
School of Computer Science, Georgia Institute of Technology · School of Materials Science and Engineering, Georgia Institute of Technology · Department of Biomedical Engineering, Georgia Institute of Technology +1

Mar 18, 2026J/K move · Enter open · S save

Revisiting foundation models for cell instance segmentation

Anwai Archit, Constantin Pape

Georg-August-University Göttingen, Institute of Computer Science · CAIMed - Lower Saxony Center for AI & Causal Methods in Medicine, Göttingen · Cluster of Excellence Multiscale Bioimaging (MBExC), Georg-August-University Göttingen

Instance Segmentation Microscopy Image Analysis

Open paper AskPDF arXiv

Cell segmentation is a fundamental task in microscopy image analysis. Several foundation models for cell segmentation have been introduced, virtually all of them are extensions of Segment Anything Model (SAM), improving it for microscopy data. Recently, SAM2 and SAM3 have been published, further improving and extending the capabilities of general-purpose segmentation foundation models. Here, we comprehensively evaluate foundation models for cell segmentation (CellPoseSAM, CellSAM,

μ

SAM) and for general-purpose segmentation (SAM, SAM2, SAM3) on a diverse set of (light) microscopy datasets, for tasks including cell, nucleus and organoid segmentation. Furthermore, we introduce a new instance segmentation strategy called automatic prompt generation (APG) that can be used to further improve SAM-based microscopy foundation models. APG consistently improves segmentation results for

μ

SAM, which is used as the base model, and is competitive with the state-of-the-art model CellPoseSAM. Moreover, our work provides important lessons for adaptation strategies of SAM-style models to microscopy and provides a strategy for creating even more powerful microscopy foundation models. Our code is publicly available at https://github.com/computational-cell-analytics/micro-sam.

Abstract

Instance Segmentation

Explore similar work

Revisiting foundation models for cell instance segmentation

QCell: Recombining and Aligning Cell Queries for Overlapping Instance Segmentation

Self-supervised Pretraining of Cell Segmentation Models