From UNI2-h to ConvNeXt-T: Lightweight Nuclei Instance Segmentation via Knowledge Distillation

Background and Objective: Precise and scalable instance segmentation of cell nuclei is a fundamental prerequisite for computational pathology, yet gigapixel whole-slide images (WSIs) pose significant computational challenges. While patch-based processing is standard during training, existing methods are often limited to small tile sizes during inference due to architectural bottlenecks or reliance on computationally expensive post-processing for instance separation. We introduce a faster, scalable, and end-to-end framework capable of processing large-scale image tiles while accurately modeling biologically realistic overlapping nuclei. Methods: We propose LSP-DETR (Local Star Polygon DEtection TRansformer). The model represents nuclei as star-convex polygons and employs a lightweight transformer with linear complexity, enabling the processing of high-resolution images in a single forward pass. A novel radial distance loss accommodates annotation uncertainty, allowing the segmentation of overlapping nuclei to emerge naturally without explicit overlap labels. Results: LSP-DETR achieves state-of-the-art efficiency, with an inference time of 0.45 s/mm^2, a 3.2x speedup over StarDist, the next-fastest method. On PanNuke, the model achieves competitive accuracy (67.5 bPQ), while yielding an F$_1$-score of 0.964 in polygon overlap when evaluated against consensus annotations from two expert pathologists. Furthermore, it outperforms larger models such as LKCell in generalization robustness, reaching an F1-score of 85.0 on MoNuSeg. Conclusions: LSP-DETR bridges the gap between high-fidelity segmentation and practical clinical requirements by eliminating heuristic post-processing. By providing a scalable, linear-complexity solution that naturally handles overlaps between nuclei, this framework sets a new direction for efficient high-throughput WSI analysis in digital pathology.

In computational pathology, nuclear instance segmentation is a fundamental task with many downstream clinical applications. With the advent of deep learning, many approaches, including convolutional neural networks (CNNs) and vision transformers (ViTs), have been proposed for this task, along with both machine learning-based and non-machine learning-based pre- and post-processing techniques to further boost performance. However, one fundamental aspect that has received less attention is the evaluation pipeline. In this study, we identify four key issues associated with nuclear instance segmentation evaluation and propose corresponding solutions. Our proposed modifications, namely handling vague regions, score normalization, overlapping instances, and border uncertainty, are integrated into a unified framework called NucEval, which enables robust evaluation of nuclear instance segmentation. We evaluate this pipeline using the NuInsSeg dataset, which provides unique characteristics that make it particularly suitable for this study, as well as two additional external datasets, with three CNN- and ViT-based nuclear instance segmentation models, to demonstrate the impact of these modifications on instance segmentation metrics. The code, along with complete guidelines and illustrative examples, is publicly available at: https://github.com/masih4/nuc_eval.

Multiplexed fluorescence microscopy improves tissue segmentation by providing complementary channels including nuclear (DAPI) and membrane (E-cadherin), that together encode richer spatial context than single-channel imaging alone. However, multiplexed models require all channels at inference, limiting deployment where only a subset is available. This work proposes a cross-modal knowledge distillation framework that transfers semantic information from a frozen foundation model teacher processing multiplexed input to a lightweight student operating on the nuclear channel only. The distillation objective combines MSE-based probability matching, boundary-aware supervision, and learnable uncertainty weighting. SAM ViT-H and CellSAM are evaluated as teachers across four U-Net students: Swin-Tiny (27M), ResNet18 (11M), EfficientNet-B0 (5.3M), and MobileNetV3 (1.5M), on TissueNet and BBBC038. On TissueNet, the SAM-distilled Swin-Tiny student achieves Dice 78.36 (plus or minus 1.44), a 13.05-point improvement over the no-KD baseline (65.31 plus or minus 1.35) and 87.9% recovery of teacher oracle performance (89.12 plus or minus 1.21) at a 23x parameter reduction. KD consistently improves all four students by approximately 12 Dice points, confirming architecture-agnostic distillation. SAM ViT-H outperforms CellSAM as teacher across all settings. Cross-dataset evaluation on BBBC038 shows consistent gains without teacher retraining.