Organizations: Department of Information Technology, Uppsala University, Sweden · Department of Clinical Diagnostics, School of Health Sciences, Jönköping University, Sweden · Department of Pathology, Linköping University Hospital, Sweden
In computational cytology, detecting malignancy on whole-slide images is difficult because malignant cells are morphologically diverse yet vanishingly rare amid a vast background of normal cells. Accurate detection of these extremely rare malignant cells remains challenging due to large class imbalance and limited annotations. Conventional weakly supervised approaches, such as multiple instance learning (MIL), often fail to generalize at the instance level, especially when the fraction of malignant cells (witness rate) is exceedingly low. In this study, we explore the use of one-class representation learning techniques for detecting malignant cells in low-witness-rate scenarios. These methods are trained exclusively on slide-negative patches, without requiring any instance-level supervision. Specifically, we evaluate two OCC approaches, DSVDD and DROC, and compare them with FS-SIL, WS-SIL, and the recent ItS2CLR method. The one-class methods learn compact representations of normality and detect deviations at test time. Experiments on a publicly available bone marrow cytomorphology dataset (TCIA) and an in-house oral cancer cytology dataset show that DSVDD achieves state-of-the-art performance in instance-level abnormality ranking, particularly in ultra-low witness-rate regimes (≤1%) and, in some cases, even outperforming fully supervised learning, which is typically not a practical option in whole-slide cytology due to the infeasibility of exhaustive instance-level annotations. DROC is also competitive under extreme rarity, benefiting from distribution-augmented contrastive learning. These findings highlight one-class representation learning as a robust and interpretable superior choice to MIL for malignant cell detection under extreme rarity.
Figures & tables
Figure 1: Overview of the proposed pipeline for rare abnormal cell detection in whole-slide cytology images. For training the one-class classifier, only slide-negative patches are used, which are known to be negative without requiring any instance-level annotations. At inference, patches from test slides are assigned anomaly scores, enabling the ranking and retrieval of rare malignant cells.
Cell type
Total
Train
Test
LYT (normal)
26,242
18,369
7,873
BAS (abnormal)
441
308
133
HAC (abnormal)
409
286
123
OTH (abnormal)
294
205
89
LYI (abnormal)
65
45
20
FGC (abnormal)
47
32
15
Table 1: Cell-level statistics for the bone marrow dataset [ 20 ] . All abnormal classes are merged into a single abnormal group for anomaly detection.
Table 2: Number of nuclei per slide in each partition (sorted in decreasing order). Bold indicates the two slides shown later in Fig. 7(a) and Fig. 7(b) .
WR (%)
Train abnormal cells
Test abnormal cells
9
910
396
5
455
198
1
90
40
0.5
45
20
0.1
10
4
0.05
5
2
Table 3: Number of abnormal cells used for training and evaluation at each witness rate (WR).
Figure 2: Mean Recall@400 on the bone marrow dataset across witness rates (log-log scale).
WR
FS-SIL
WS-SIL
DSVDD
DROC
ItS2CLR
9%
223.0 ± 0.0
121.0 ± 0.0
49.0 ± 0.0
11.0 ± 0.0
27.0 ± 0.0
5%
84.2 ± 3.6
38.6 ± 2.6
24.1 ± 2.5
5.8 ± 1.3
7.7 ± 4.2
1%
10.0 ± 2.3
2.5 ± 2.1
5.3 ± 1.6
2.1 ± 1.3
1.0 ± 0.9
0.5%
4.4 ± 2.2
1.3 ± 1.0
2.7 ± 1.5
0.8 ± 1.0
0.5 ± 0.5
0.1%
0.5 ± 0.7
0.4 ± 0.5
0.7 ± 0.6
0.2 ± 0.4
0.6 ± 0.7
0.05%
0.1 ± 0.3
0.2 ± 0.4
0.3 ± 0.5
0.2 ± 0.6
0.0 ± 0.0
Table 4: Mean true positives in the top-400 (TP@400) on the bone marrow dataset (mean ± std). Bold values indicate the top values across witness rates. The fully supervised FS-SIL is not considered a realistic option (and is therefore grayed out) since it relies on instance level annotations, and is included only to indicate an expected upper bound of performance.
Figure 3: Mean nDCG@400 on the bone marrow dataset across witness rates (log-log scale).
Figure 4: Mean AUTK@400 on the bone marrow dataset across witness rates (log-log scale).
Figure 5: Mean normalized AUFROC on the bone marrow dataset across witness rates (log-log scale).
Figure 6: Example mosaic for the bone marrow dataset at 10% WR. Shown are the top 100 patches (unordered) using DSVDD anomaly scores and arranged in a 10×10 grid. A small white square in the top-right corner of a patch indicates that the underlying instance is annotated as non-LYT (i.e. abnormal or suspicious); within this box, ‘ × ’ marks such non-LYT cells. Patches without a white box correspond to LYT (normal) cells. For this mosaic, 25 of the top 100 predictions were true positives (non-LYT).
Type
WS-SIL
DSVDD
DROC
ItS2CLR
Malignant 1
(0,0)
(0,0)
(0,2)
(1,2)
Malignant 2
(25,93)
(18,63)
(2,65)
(2,47)
Malignant 3
(7,46)
(9,52)
(18,19)
(4,19)
Malignant 4
(8,11)
(4,22)
(9,16)
(2,10)
Table 5: Expert assessment of top-100 patches on the four malignant oral-cancer test slides. Tuples report the number of patches marked as suspicious by the cytologist and cytotechnologist (in this order). Bold indicates the slides visualized in Fig. 7 .
Figure 7: DSVDD-selected top-100 expert-review grids for two malignant test slides (shown in randomized order). For each slide, we show the unordered top-100 patches (arranged as a 10×10 grid). A white square in the top-right corner of a patch indicates that at least one expert marked it as suspicious. Within the square, a red backslash ‘ \ ’ denotes a mark by one expert, an blue forward slash ‘ / ’ denotes a mark by the other, and a full cross indicates agreement (both experts).