CytoSPM: Open-Vocabulary Cytopathology Detection with Structured Prompt Bank

Wenjie Li; Zishan Xu; Jinyang Huang; Zhengxin Nie; Shichao Kan; Yixiong Liang

cs.CVSep 25, 2026

CytoSPM: Open-Vocabulary Cytopathology Detection with Structured Prompt Bank

Authors: Wenjie Li, Zishan Xu, Jinyang Huang, Zhengxin Nie, Shichao Kan, Yixiong Liang

Organizations: Central South University · Shanghai Jiao Tong University

Abstract

Cytopathology detection requires open-vocabulary recognition because cellular categories are fine-grained, long-tailed, and continuously evolving across different organ systems. However, existing cytology detectors are mostly single-domain and closed-set, and there is still no unified benchmark for evaluating open-vocabulary cytopathology detection. We present PentaCyto, a multi-domain benchmark covering cervical, urinary, respiratory, serous fluid, and thyroid cytology, with 24 base categories and 9 held-out novel categories. Each category is associated with structured cytomorphology prompts that describe diagnostic morphological attributes and provide clinically grounded textual knowledge. We further propose CytoSPM, an efficient detector based on a decoupled two-stage design. It first extracts reusable class-agnostic visual representations, and then performs class-aware structural prompt matching with class names and cytomorphology prompts. On PentaCyto, CytoSPM outperforms existing methods in novel-category detection and open-vocabulary detection while maintaining efficient inference.

Figures & tables

Fig. 1: Comparison of traditional closed-set detection and CytoSPM. Traditional object detection methods rely on a fixed label set and can only recognize cell categories predefined during training, making it difficult to generalize to unseen, rare, or newly defined cytopathology classes. In contrast, CytoSPM addresses open-vocabulary cytopathology detection with a decoupled two-stage design. It first extracts reusable class-agnostic visual representations from cytology images, and then performs class-aware structural prompt matching with cytomorphology prompts. By combining class-name, structural prototype, and fine-grained attribute matching, CytoSPM enables text-prompt-driven detection of both seen and unseen cytology cell categories.

		Base				Novel
Domain	Standard	# Cat.	# WSI	# Images	# Boxes	# Cat.	# WSI	# Images	# Boxes
Cervical	TBS	9	1,588	34,305	123,474	–	–	–	–
Urinary	Paris	3	220	25,720	8,410	–	–	–	–
Respiratory	PSC	6	162	50,102	611,320	3	20	2,593	3,517
Serous fluid	TIS	1	10	4,187	19,705	3	16	6,213	17,445
Thyroid	Bethesda	5	261	40,719	149,172	3	15	2,656	8,250

TABLE I: Statistics of the PentaCyto dataset. PentaCyto covers five cytopathology domains with domain-specific reporting standards. For each domain, we report the number of categories, whole-slide images (WSIs), images, and bounding-box annotations under the base and novel splits. Dashes indicate that no novel categories are defined for the corresponding domain.

Fig. 2: The bar chart presents the bounding-box counts of each category across five cytology domains.

Fig. 3: Overview of CytoSPM. CytoSPM is an efficient open-vocabulary cytopathology detection framework based on a decoupled two-stage design. In Stage I, the visual branch extracts reusable class-agnostic region-level representations from the input cytology image without introducing class names, textual prompts, or cross-modal fusion. In Stage II, class names and class-specific cytomorphology prompt banks are encoded by a frozen text encoder, and class-aware structural prompt matching is performed over the visual regions. Through mutual selection, CytoSPM dynamically selects category-relevant visual evidence and visually supported morphology prompts, which are aggregated into an image-conditioned structural prototype. The final classification score combines class-name matching, structural prototype matching, and fine-grained attribute matching, enabling efficient recognition of both seen and unseen cytopathology categories.

Name	Multi	Domain	Task
MedROV [ 22 ]	✓	CT, MRI, ultrasound, X-ray, histopathology, colonoscopy, microscopy, dermoscopy, and fundoscopy	Det.
CytoFM [ 23 ]	✓	Breast, cervical, and thyroid	Cls.
CytoCrowd [ 24 ]	✓	Endocervical, metaplastic cells, etc.	Det. / Cls.
CytoSAE [ 25 ]	✗	Hematology cytology	Det.
HERO [ 21 ]	✗	Cervical cytology	Det.

TABLE II: Comparison with related cytology and medical open-vocabulary detection works. Multi indicates whether multiple medical or cytology domains are covered; Cls. denotes classification, and Det. denotes detection, which requires both cell localization and category recognition.

				Base			Novel
Method	Backbone	#Params	FPS	AP	AP 50	AP 75	AP	AP 50	AP 75	AP AVG
GLIP [ 15 ]	Swin-T	231.8M	0 7.1	28.90	43.50	32.70	12.10	16.90	14.10	20.50
Grounding DINO [ 16 ]	Swin-T	173.5M	0 6.2	30.40	47.50	33.80	0 2.80	0 4.30	0 3.30	16.60
OV-DEIM [ 26 ]	DINOv3-ViT-T	11.4M	23.2	31.60	45.50	37.20	12.70	17.00	14.80	22.15
YOLOE (text) [ 27 ]	YOLOv8-L	51.2M	75.3	33.60	48.30	39.10	0 6.40	0 8.80	0 7.40	20.00
YOLOE (visual) [ 27 ]	YOLOv8-L	51.2M	75.3	29.70	43.10	34.30	12.50	16.90	14.90	21.10

TABLE III: Comparison on the PentaCyto. YOLOE (text), (visual), and (mixed) use textual prompts, visual prompts, and both prompts, respectively. The best and second-best results are shown in bold and underlined .

Fig. 4: Qualitative comparison on PentaCyto. Green, red, and blue boxes denote correct predictions, false positives, and ground truth, respectively.

Fig. 5: Ablation of Top-

Q

and Top-

P

. Top-

Q

selects morphology prompts, while Top-

P

selects visual evidence for structural matching.

Panel of Figure 9 — Fig. 6: Effect of the structural matching warm-up in CytoSPM.

Explore similar work

Beyond Classification: Pathology Foundation Models as Detection Encoders for Mitotic Figures
Jul 30, 2026Sweta Banerjee, Alireza Teimoury, Nils Porsche +11Computational Pathology Pathology Foundation Models
Flensburg University of Applied Sciences, Flensburg, Germany · Julius-Maximilians-Universität Würzburg, Würzburg, Germany · Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany +4
Refining Cytology Predictions with Conditional Random Fields
Sep 25, 2026Manon Dausort, Tiffanie Godelaine, Karim El Khoury +3Medical Image Classification Conditional Random Fields
ICTEAM, Universit´e Catholique de Louvain, Belgium
A Multi-Stage Optimization Pipeline for Bethesda Cell Detection in Pap Smear Cytology
Apr 15, 2026Martin Amster, Camila María PolottoObject Detection Computational Pathology
Instituto de Ciencias de la Computación (ICC), CONICET-UBA · Facultad de Ciencias Exactas y Naturales, Universidad de Buenos Aires · Instituto de Física Interdisciplinaria y Aplicada, Departamento de Física, Universidad de Buenos Aires

Jul 30, 2026J/K move · Enter open · S save

Beyond Classification: Pathology Foundation Models as Detection Encoders for Mitotic Figures

Sweta Banerjee, Alireza Teimoury, Nils Porsche, Alexandra K. Stoll+10

Flensburg University of Applied Sciences, Flensburg, Germany · Julius-Maximilians-Universität Würzburg, Würzburg, Germany · Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany+4

Computational Pathology Pathology Foundation Models Object Detection Whole-Slide Image Analysis

Open paper AskPDF arXiv

Pathology foundation models (FMs) are models trained on vast amounts of typically unlabeled data and have been shown to yield regularized latent spaces that can be used effectively in downstream classification tasks. This is also true for the classification of mitotic figures vs. other cells. However, it is so far unclear if the latent space of current FMs provides features that are discriminant and spatially suitably resolved to also serve as a backbone for dense object detection paradigms. In this work, we investigate this question for common current pathology FMs (UNI, UNI2-h, Virchow, Virchow2, H-optimus-0, H-optimus-1) and compare their performance against a fully end-to-end trained baseline based on a ResNet50 architecture. We combine FM backbones with representatives of single stage, dual stage and self-attention-based detectors (RetinaNet, Faster R-CNN, Deformable DETR respectively) on the multi-domain MIDOG++ dataset, and on the TUPAC16 dataset as an out-of-domain case. We show that the H-optimus-0 and Virchow models yielded competitive performance, indicating that the latent spaces of current FMs, all trained on image-level self-supervision, are suitable for direct mitotic figure detection and may be slightly more robust on our out-of-domain test case. All code is made available publicly at https://github.com/DeepMicroscopy/FM4MFdet.

Abstract

Explore similar work

Beyond Classification: Pathology Foundation Models as Detection Encoders for Mitotic Figures

Refining Cytology Predictions with Conditional Random Fields

A Multi-Stage Optimization Pipeline for Bethesda Cell Detection in Pap Smear Cytology