cs.CVOct 9, 2026

Detection of extremely rare dead cells in fluorescence microscopy images

Authors: Prateek Verma, Benjamin Butter, Jian Zhang, Xintao Wu

Organizations: Department of Electrical Engineering and Computer Science University of Arkansas, Fayetteville, Arkansas, USA · Department of Biomedical Engineering University of Arkansas, Fayetteville, Arkansas, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Dead cell detection in fluorescence microscopy is essential for phenotypic screening and drug efficacy assessment. However, automated identification remains challenging due to signal variability, optical artifacts, and colocalization ambiguities across multiple channels. Markers that define nuclear boundaries do not neatly coincide spatially or temporally with apoptotic markers. Furthermore, marker accumulation kinetics vary across individual cells and apoptosis stages, making universal thresholds unreliable. We present a systematic comparison of several computational approaches for detecting apoptotic cells in nucleus-stained specimens: (1) heuristic methods using intensity thresholds, distributions, correlations, and domain guidance; (2) classification using numerical features (Logistic Regression, Random Forest, XGBoost and LightGBM) and image features (ResNet-18 and EfficientNet-B0); and (3) anomaly detection, also using numerical features (Isolation Forest, one-class SVM and PCA reconstruction) and image features (convolutional autoencoders and Deep SVDD). Using a dataset of 258 images containing 65,527 cells/nuclei across multiple scenes with 243 manually annotated ground truth dead cells, we evaluate each method on metrics such as PR-AUC, Precision, Recall, F1-score, and macro-F1. Our results indicate that classification using image features (deep learning) and numerical features (traditional) achieved the best performance, with ResNet-18 and XGBoost significantly outperforming manually designed heuristics and anomaly detection methods. Our findings demonstrate that such biomedical problems remain highly challenging to solve even with the state-of-the-art computational and learning techniques. We aim to provide an effective decision-support tool to researchers while tackling such problems either fully-automatically or in a human-in-the-loop setting as an aide to domain scientists.

Figures & tables

Fig. 1: An illustration of the computational approaches used in the automation pipeline.
  counts
(a)images766
(b)total detected cell/nuclear masks112,011
(c)images with ground truth258
(d)masks in images with ground truth65,527
(e)dead cell labels330
(f)masks mapped to dead labels243
TABLE I: Data Characteristics
Fig. 2: Two examples each of TP, TN, FP, and FN detections by the best performing heuristic model (percentile intensity thresholding). The cells of interest are centered in the tile. The three rows correspond to the blue, green and red channels respectively. The white outlines correspond to the Cellpose masks. White dots/crosses indicate the exact marked position of alive/dead cells as annotated by the domain expert. The red crosses, always placed at centroids of the Cellpose mask, indicate a predicted dead cell.
Fig. 3: Two examples each of TP, TN, FP, and FN detections by the best performing classification model (Resnet-18). The cells of interest are centered in the tile and indicated by white dots. Unlike the heuristic visualization in Figure 2 , where predictions were made on the entire image, the deep learning models classified isolated image patches; therefore, only the centrally located cell (white dot) represents the classification target, while surrounding cells (not classified) provide spatial context. The three rows correspond to the blue, green and red channels. The white outlines correspond to the Cellpose masks.
ScopeIntensity FeatureCounts
Cellmean3
median3
maximum3
 minimum3
 coefficient of variation3
Image Patchmean3
TABLE II: Scope and Count of Numerical Features
FamilyMethodVariantPR-AUCPrecisionRecallF1-ScoreMacro-F1
Progressive HeuristicsIntensity Thresholdingpercentile (99%)n/a0.3540.5720.4370.717
 distribution-based (bimodal)n/a0.1110.4550.1790.586
 distribution-based (z-score)n/a0.1560.7820.2600.626
 Adaptive Gap Splitsize 10%, position 75%n/a0.4820.3830.4270.712
 Colocalization-Aware Preprocessingnuclear normalizedn/a0.3170.6010.4150.706
 Pearson Correlationn/a0.4780.2220.3030.651
TABLE III: Comparative Performance of the Four Computational Approaches. The best performing metric is shown in bold. The best performing metric in each family is underlined.
More in this topic

Microscopy Image Analysis

Microscopy image analysis uses computational methods to detect, segment, track, reconstruct, and quantify structures in scientific images.

See all 195 papers

Explore similar work

CardsList
Jun 10, 2026

CellNet -- Localizing Cells using Sparse and Noisy Point Annotations

Counting living cells is an important step in many biological research workflows. Our collaborators at the Wellcome Sanger Institute study vital genes in humans via large scale saturation genome editing screening, which requires repeatedly counting cells a great number of times. Computer Vision based automation is crucial for high throughput and resource efficiency. In this work, we develop a regression-based deep learning computer vision algorithm to detect and count cells in phase-contrast microscopy images. To reduce annotation effort, which in practice often becomes a bottleneck, we focus on counting cells only using sparse point annotations, which are fast and easy to acquire. By comparison to state-of-the-art 0-shot methods, we show that regression-based counting is a promising alternative in low data regimes. Through developing methods to automatically count living cells in microscopy images, we contribute to valuable research on the human genome. The code is available at https://github.com/beijn/cellnet.

Benjamin Eckhardt, Dmytro Fishman, Stuart Fawke +3

University of Göttingen · University of Tartu · Wellcome Sanger Institute

Object CountingMicroscopy Image Analysis
AskPDF
Oct 7, 2026

Label-free cell counting and viability prediction with brightfield imaging and deep learning

Cell viability assessment is a core requirement in cell culture systems, with critical applications in biopharmaceutical manufacturing and drug development. Conventionally, it is measured by adding membrane-impermeable dyes to a sample (a process called staining), which allows compromised cell membranes to be distinguished from intact ones. However, staining has several limitations: (a) chemical agents can perturb normal cellular processes of the cells being measured, (b) it is often ambiguous to assign viability to individual cells whose membrane integrity is only partially compromised. (c) photobleaching can undermine measurement accuracy over time when using fluorescent stains, and (d) staining cannot be performed in situ or in real time. Here, we show that (1) stained cells captured under brightfield imaging contain sufficient information to distinguish live and dead cells, and (2) cells captured under unstained brightfield imaging exhibit similar image features to their stained counterparts, enabling models trained on stained cells to generalize to unstained ones. We then report the development and validation of ViabiLens, an AI-assisted software for label-free cell viability analysis. The ViabiLens combines a cell detection model for localizing individual cells with a convolutional neural network (CNN) classifier for live/dead prediction, paired with an interactive UMAP-based viewer for visualizing and exploring individual cells across the sample. Evaluated on Chinese Hamster Ovary (CHO) cells spanning a wide range of viability conditions, ViabiLens achieves a mean absolute error of 2.68% on unstained samples against fluorescence-based reference measurements. We also release a benchmark dataset for label-free cell viability analysis to facilitate future research, available at https://amirrezavazifeh.github.io/ViabiLens-Project-Page/.

Amir Reza Vazifeh, Christian Zeigler, Sornanathan Meyyappan +2

Department of Electrical and Computer Engineering, Princeton University, Princeton, NJ 08544, USA · Waters Corporation, Immerse Cambridge, 301 Binney Street, Suite 102, Cambridge, MA 02142, USA · Waters Corporation, 34 Maple St, Milford, MA 01757, USA +1

Microscopy Image Analysis
AskPDF
Apr 9, 2026

Needle in a Haystack: One-Class Representation Learning for Detecting Rare Malignant Cells in Computational Cytology

In computational cytology, detecting malignancy on whole-slide images is difficult because malignant cells are morphologically diverse yet vanishingly rare amid a vast background of normal cells. Accurate detection of these extremely rare malignant cells remains challenging due to large class imbalance and limited annotations. Conventional weakly supervised approaches, such as multiple instance learning (MIL), often fail to generalize at the instance level, especially when the fraction of malignant cells (witness rate) is exceedingly low. In this study, we explore the use of one-class representation learning techniques for detecting malignant cells in low-witness-rate scenarios. These methods are trained exclusively on slide-negative patches, without requiring any instance-level supervision. Specifically, we evaluate two OCC approaches, DSVDD and DROC, and compare them with FS-SIL, WS-SIL, and the recent ItS2CLR method. The one-class methods learn compact representations of normality and detect deviations at test time. Experiments on a publicly available bone marrow cytomorphology dataset (TCIA) and an in-house oral cancer cytology dataset show that DSVDD achieves state-of-the-art performance in instance-level abnormality ranking, particularly in ultra-low witness-rate regimes (≤1%\leq 1\%) and, in some cases, even outperforming fully supervised learning, which is typically not a practical option in whole-slide cytology due to the infeasibility of exhaustive instance-level annotations. DROC is also competitive under extreme rarity, benefiting from distribution-augmented contrastive learning. These findings highlight one-class representation learning as a robust and interpretable superior choice to MIL for malignant cell detection under extreme rarity.

Swarnadip Chatterjee, Vladimir Basic, Arrigo Capitanio +2

Department of Information Technology, Uppsala University, Sweden · Department of Clinical Diagnostics, School of Health Sciences, Jönköping University, Sweden · Department of Pathology, Linköping University Hospital, Sweden

Whole-Slide Image AnalysisComputational PathologyRepresentation LearningOne-Class ClassificationClass-Imbalanced Learning
AskPDF