QCell: Recombining and Aligning Cell Queries for Overlapping Instance Segmentation

Yaroslav Prytula; Anton Popov; Dmytro Fishman

cs.CVAug 29, 2026

QCell: Recombining and Aligning Cell Queries for Overlapping Instance Segmentation

Authors: Yaroslav Prytula, Anton Popov, Dmytro Fishman

Organizations: Institute of Computer Science University of Tartu, Tartu, Estonia · Faculty of Applied Sciences Ukrainian Catholic University, Lviv, Ukraine · Department of Electronic Engineering, Micro- and Biomedical Electronics Igor Sikorsky Kyiv Polytechnic Institute, Kyiv, Ukraine · STACC OÜ, Tartu, Estonia ·

Abstract

Instance segmentation of overlapping cells in microscopy remains challenging due to semi-transparent structures that produce weak boundaries and mixed visual evidence in overlap regions. Existing methods address this through local regions of interest or shape priors but lack global reasoning across overlapping objects. We present QCell, a novel query-based model that de-overlaps cell instances in microscopy scenes. Our approach combines (i) an instance recombination module that decomposes and recombines query representations in latent space, enabling the model to reason about complete object structure under overlap, and (ii) a contrastive query alignment objective that combines distinctive instance feature learning and separation of overlapping cell queries. We additionally introduce a new Organoid dataset benchmark for overlapping cell segmentation. We show that QCell outperforms state-of-the-art methods across multiple benchmarks, achieving +2.2 AP and +2.7 AJI on ISBI2014. Code is available at https://github.com/SlavkoPrytula/QCell

Figures & tables

Figure 2: Overview of QCell. QCell builds on a MaskDINO-style query-based segmentation pipeline, where multi-scale image features and positional embeddings are processed by the encoder and refined by transformer decoder layers with content and DN queries. The proposed modules are shown above: (a) instance recombination decomposes each query into amodal, visible, and occluded sub-representations and recombines them into a refined full-instance query; (b) contrastive query learning uses matched instance queries $\hat{q}_{i}$ by Hungarian matching as anchors, corresponding DN queries $\hat{d}^{+}_{i}$ across all groups as positives and other DN queries as negatives to align queries of the same cell and separate queries of different cells in latent space.

	ISBI2014						Revvity-25

Models	AP	AP 50	AP 75	DICE	F1	AJI	AP	AP 50	AP 75	DICE	F1	AJI	#params.	FLOPs
Mask R-CNN [ 11 ]	55.6	84.0	58.9	91.2	88.1	72.4	39.7	78.4	36.5	86.5	82.9	63.7	44M	128G
ORCNN [ 42 ]	60.6	86.3	65.3	92.0	90.0	75.4	40.4	78.2	38.6	86.9	83.7	64.6	46M	132G
VRSP-Net [ 39 ]	51.1	81.6	53.4	91.6	88.5	73.1	39.1	77.7	34.7	87.7	85.4	67.2	99M	118G
BCNet [ 15 ]	47.6	77.8	48.3	56.6	53.6	41.1	37.7	75.7	33.7	88.3	40.7	24.9	38M	308G

Table 1: Comparison on ISBI2014 and Revvity-25. QCell achieves the best overall performance on both benchmarks, improving AP and AJI over prior amodal and cell instance segmentation methods. Rankings are determined from unrounded values. † denotes our adapted implementation. Best results are shown in bold , and second-best results are underlined .

Organoids

Models	AP	AP 50	AP 75	DICE	F1	AJI	#params.	FLOPs
Mask R-CNN [ 11 ]	50.6	68.7	55.9	93.9	68.6	61.7	44M	128G
ORCNN [ 42 ]	48.7	67.1	54.3	93.4	70.6	62.4	46M	132G
VRSP-Net [ 39 ]	50.3	67.6	55.7	93.0	72.3	62.6	99M	118G
BCNet [ 15 ]	50.3	67.7	55.3	94.1	62.7	58.6	38M	308G

Table 2: Comparison on Organoids. † denotes our adapted implementation of the method. ‡ denotes a model trained with

N

=100 object queries. The best result is shown in bold , and the second-best result is underlined .

Figure 3: Qualitative comparison across datasets. QCell produces more object-consistent instance masks across ISBI2014, Revvity-25, and Organoids. On ISBI2014, QCell better separates cells under heavy occlusion. On Revvity-25, it preserves fine cell details and boundaries. On Organoids, it maintains improved object consistency in dense overlapping scenes. Best viewed in color and zoomed in.


$\mathcal{L}_{\mathrm{base}}$	$\mathcal{L}_{\mathrm{decomp}}$	$\mathcal{L}_{\mathrm{recomb}}$	$\mathcal{L}_{\mathrm{disc}}$	$\mathcal{L}_{\mathrm{align}}$	AP	AP 50	AP 75	DICE	F1	AJI
✓					63.7	89.0	65.4	92.3	90.0	75.9
✓	✓				65.1	89.8	68.1	92.6	90.0	76.2
✓	✓	✓			66.6	90.1	69.2	92.8	90.7	77.1
✓			✓		65.5	89.8	67.5	92.6	90.6	76.7
✓				✓	66.2	89.7	68.6	92.6	90.5	76.4

Table 3: Ablation study on ISBI2014.

\mathcal{L}_{\mathrm{base}}

denotes the standard MaskDINO objective,

\mathcal{L}_{\mathrm{decomp}}

the decomposition loss,

\mathcal{L}_{\mathrm{recomb}}=\mathcal{L}_{\mathrm{refined}}+\mathcal{L}_{\mathrm{CR}}

the recombination loss,

\mathcal{L}_{\mathrm{disc}}

the instance-discriminative loss, and

\mathcal{L}_{\mathrm{align}}

the cosine alignment loss.


$\mathcal{L}_{\mathrm{base}}$	$\mathcal{L}_{\mathrm{decomp}}$	$\mathcal{L}_{\mathrm{recomb}}$	$\mathcal{L}_{\mathrm{disc}}$	$\mathcal{L}_{\mathrm{align}}$	AP	AP 50	AP 75
✓					11.67	29.09	7.64
✓	✓				12.12	28.71	8.93
✓	✓	✓			12.29	29.52	8.92
✓			✓	✓	12.63	30.21	8.50
\rowcolor our_results_color ✓	✓	✓	✓	✓	13.68	30.17	10.97

Table 4: Ablation results on highly overlapping ISBI2014 instances. The subset includes instances whose ground-truth mask has pairwise IoU

\geq 0.5

with another instance. Combining Instance Recombination and contrastive query learning gives the strongest performance under severe overlap.


Models	AP	AP 50	AP 75	DICE	F1	AJI
Baseline	63.7	89.0	65.4	92.3	90.0	75.9
DN oracle	$\mathbf{68.6}$ $(\pm 1.2)$	$\mathbf{93.5}$ $(\pm 0.6)$	$\mathbf{71.9}$ $(\pm 1.7)$	$\mathbf{92.4}$ $(\pm 0.2)$	$\mathbf{94.1}$ $(\pm 0.3)$	$\mathbf{80.0}$ $(\pm 0.6)$

Table 5: DN oracle analysis on ISBI2014. Replacing main-query predictions matched to ground-truth instances using the Hungarian matcher with their corresponding DN-query predictions improves all metrics, showing that DN queries often provide cleaner instance representations and motivating their use for contrastive query learning.

	ISBI2014		Organoids		Revvity-25

Models	FNo $\downarrow$	TPp $\uparrow$	FNo $\downarrow$	TPp $\uparrow$	FNo $\downarrow$	TPp $\uparrow$	#params.	FLOPs
Mask R-CNN [ 11 ]	$17.3$ $(\pm 0.7)$	$95.6$ $(\pm 0.1)$	$43.1$ $(\pm 0.7)$	$93.4$ $(\pm 0.1)$	$17.7$ $(\pm 0.2)$	$85.0$ $(\pm 0.9)$	44M	128G
ORCNN [ 42 ]	$14.8$ $(\pm 0.9)$	$95.1$ $(\pm 0.3)$	$40.7$ $(\pm 0.9)$	$93.3$ $(\pm 0.0)$	$17.5$ $(\pm 0.0)$	$83.4$ $(\pm 0.0)$	46M	132G
VRSP-Net [ 39 ]	$15.3$ $(\pm 1.0)$	$93.8$ $(\pm 0.1)$	$32.2$ $(\pm 2.2)$	$93.6$ $(\pm 0.3)$	$16.1$ $(\pm 0.2)$	$86.0$ $(\pm 0.4)$	99M	118G
BCNet [ 15 ]	$64.5$ $(\pm 7.6)$	$45.7$ $(\pm 19.0)$	$50.3$ $(\pm 0.0)$	$94.2$ $(\pm 0.0)$	$73.4$ $(\pm 0.1)$	$86.9$ $(\pm 0.3)$	38M	308G

Table 6: Segmentation quality and model efficiency. Object-based false-negative rate (FNo) and pixel-based true-positive rate (TPp) across the three datasets, with parameter counts and FLOPs. FLOPs are averaged over the ISBI2014 test set.

Instance Segmentation

Instance segmentation identifies each object and assigns it a separate pixel-level mask in images or video.

See all 159 papers

Boundary-Aware Instance Segmentation in Microscopy Imaging
Mar 22, 2026Thomas Mendelson, Joshua Francois, Galit Lahav +1Instance Segmentation Boundary-Aware Segmentation
The School of Electrical and Computer Engineering, Ben-Gurion University of the Negev · Department of Systems Biology, Harvard Medical School
Revisiting foundation models for cell instance segmentation
Mar 18, 2026Anwai Archit, Constantin PapeInstance Segmentation Microscopy Image Analysis
Georg-August-University Göttingen, Institute of Computer Science · CAIMed - Lower Saxony Center for AI & Causal Methods in Medicine, Göttingen · Cluster of Excellence Multiscale Bioimaging (MBExC), Georg-August-University Göttingen
Self-supervised Pretraining of Cell Segmentation Models
Apr 12, 2026Kaden Stillwagon, Alexandra Dunnum VandeLoo, Benjamin Magondu +1Instance Segmentation Microscopy Image Analysis
School of Computer Science, Georgia Institute of Technology · School of Materials Science and Engineering, Georgia Institute of Technology · Department of Biomedical Engineering, Georgia Institute of Technology +1

Mar 22, 2026J/K move · Enter open · S save

Boundary-Aware Instance Segmentation in Microscopy Imaging

Thomas Mendelson, Joshua Francois, Galit Lahav, Tammy Riklin-Raviv

The School of Electrical and Computer Engineering, Ben-Gurion University of the Negev · Department of Systems Biology, Harvard Medical School

Instance Segmentation Boundary-Aware Segmentation Microscopy Image Analysis

Open paper AskPDF arXiv

Accurate delineation of individual cells in microscopy videos is essential for studying cellular dynamics, yet separating touching or overlapping instances remains a persistent challenge. Although foundation-model for segmentation such as SAM have broadened the accessibility of image segmentation, they still struggle to separate nearby cell instances in dense microscopy scenes without extensive prompting. We propose a prompt-free, boundary-aware instance segmentation framework that predicts signed distance functions (SDFs) instead of binary masks, enabling smooth and geometry-consistent modeling of cell contours. A learned sigmoid mapping converts SDFs into probability maps, yielding sharp boundary localization and robust separation of adjacent instances. Training is guided by a unified Modified Hausdorff Distance (MHD) loss that integrates region- and boundary-based terms. Evaluations on both public and private high-throughput microscopy datasets demonstrate improved boundary accuracy and instance-level performance compared to recent SAM-based and foundation-model approaches. Source code is available at: https://github.com/ThomasMendelson/BAISeg.git

Abstract

Instance Segmentation

Explore similar work

Boundary-Aware Instance Segmentation in Microscopy Imaging

Revisiting foundation models for cell instance segmentation

Self-supervised Pretraining of Cell Segmentation Models