A Multi-Stage Optimization Pipeline for Bethesda Cell Detection in Pap Smear Cytology

Martin Amster; Camila María Polotto

cs.CVApr 15, 2026

A Multi-Stage Optimization Pipeline for Bethesda Cell Detection in Pap Smear Cytology

Authors: Martin Amster, Camila María Polotto

Organizations: Instituto de Ciencias de la Computación (ICC), CONICET-UBA · Facultad de Ciencias Exactas y Naturales, Universidad de Buenos Aires · Instituto de Física Interdisciplinaria y Aplicada, Departamento de Física, Universidad de Buenos Aires

Abstract

Computer vision techniques have advanced significantly in recent years, finding diverse and impactful applications within the medical field. In this paper, we introduce a new framework for the detection of Bethesda cells in Pap smear images, developed for Track B of the Riva Cytology Challenge held in association with the International Symposium on Biomedical Imaging (ISBI). This work focuses on enhancing computer vision models for cell detection, with performance evaluated using the mAP50-95 metric. We propose a solution based on an ensemble of YOLO and U-Net architectures, followed by a refinement stage utilizing overlap removal techniques and a binary classifier. Our framework achieved second place with a mAP50-95 score of 0.5909 in the competition. The implementation and source code are available at the following repository: github.com/martinamster/riva-trackb

Figures & tables

Figure 1: Example of an annotated image with a 100×100 bounding box from the RIVA dataset.

Figure 2: Overview of the architecture and optimization pipeline. Models are trained with preprocessed images to detect Bethesda cells. The YOLO ensemble is created using the output from both YOLO models. The final ensemble is created combining the YOLO ensemble with the output of the U-Net model. Final predictions are obtained from the three-step postprocessing pipeline.

Model	TP	FP	Recall	Precision	F1	mAP50-95
YOLOv8n 20×20	2654	30472	0.9896	0.0801	0.1482	0.5966
YOLOv8n 50×50	2665	37398	0.9937	0.0665	0.1247	0.6059
YOLO ensemble	2647	19403	0.987	0.12	0.2141	0.6153
U-Net	2141	944	0.7983	0.694	0.7425	0.5494
Final prediction	2544	11410	0.9485	0.1823	0.3058	0.6232

Table 1: Quantitative evaluation on the RIVA validation set. The metrics are calculated using the models’ output and ground-truths.

Figure 3: Comprehensive performance metrics vs. confidence threshold for the proposed ensemble model. The graph highlights the peaks of each metric.

Explore similar work

Detection and Classification of (Pre)Cancerous Cells in Pap Smears: An Ensemble Strategy for the RIVA Cervical Cytology Challenge
Mar 24, 2026Lautaro Kogan, María Victoria RíosComputational Pathology Class-Imbalanced Learning
Universidad de San Andrés, Buenos Aires, Argentina
Center-Aware Detection with Swin-based Co-DETR Framework for Cervical Cytology
Apr 2, 2026Yan Kong, Yuan Yin, Hongan Chen +2Object Detection Medical Image Localization
School of Intelligence Science and Technology, Nanjing University, Nanjing, China. · School of Biomedical Engineering, ShanghaiTech University, Shanghai, China. · State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China.
CytoSPM: Open-Vocabulary Cytopathology Detection with Structured Prompt Bank
Sep 25, 2026Wenjie Li, Zishan Xu, Jinyang Huang +3Computational Pathology Open-Vocabulary Object Detection
Central South University · Shanghai Jiao Tong University

Mar 24, 2026J/K move · Enter open · S save

Detection and Classification of (Pre)Cancerous Cells in Pap Smears: An Ensemble Strategy for the RIVA Cervical Cytology Challenge

Lautaro Kogan, María Victoria Ríos

Universidad de San Andrés, Buenos Aires, Argentina

Computational Pathology Class-Imbalanced Learning Ensemble Learning

Open paper AskPDF arXiv

Automated detection and classification of cervical cells in conventional Pap smear images can strengthen cervical cancer screening at scale by reducing manual workload, improving triage, and increasing consistency across readers. However, it is challenged by severe class imbalance and frequent nuclear overlap. We present our approach to the RIVA Cervical Cytology Challenge (ISBI 2026), which requires multi-class detection of eight Bethesda cell categories under these conditions. Using YOLOv11m as the base architecture, we systematically evaluate three strategies to improve detection performance: loss reweighting, data resampling and transfer learning. We build an ensemble by combining models trained under each strategy, promoting complementary detection behavior and combining them through Weighted Boxes Fusion (WBF). The ensemble achieves a mAP50-95 of 0.201 on the preliminary test set and 0.147 on the final test set, representing a 29% improvement over the best individual model on the final test set and demonstrating the effectiveness of combining complementary imbalance mitigation strategies.

Abstract

Explore similar work

Detection and Classification of (Pre)Cancerous Cells in Pap Smears: An Ensemble Strategy for the RIVA Cervical Cytology Challenge

Center-Aware Detection with Swin-based Co-DETR Framework for Cervical Cytology

CytoSPM: Open-Vocabulary Cytopathology Detection with Structured Prompt Bank