Automated Nucleus Segmentation in Histology Images: An Attention U-Net Approach with Comprehensive Preprocessing and Inference Pipelines

Akshat G; Divyansh Gupta; Tusar Kanti Mishra

cs.CVOct 22, 2026

Automated Nucleus Segmentation in Histology Images: An Attention U-Net Approach with Comprehensive Preprocessing and Inference Pipelines

Authors: Akshat G, Divyansh Gupta, Tusar Kanti Mishra

Organizations: Manipal Institute of Technology Bengaluru, Manipal Academy of Higher Education, Manipal, India

Abstract

Objectives: In the area of digital pathology, nucleus segmentation is an essential step for quantitative analysis; however, high nuclear density and staining heterogeneity in histology images pose a great challenge. Methods: This study implements a lightweight Attention U-Net model on the MoNuSeg 2018 dataset to overcome these challenges. The proposed pipeline integrates comprehensive image and mask preprocessing, including stain deconvolution and multi-pass mask smoothing, with a high-fidelity overlapping patch-stitching strategy during inference on high-resolution images. Results: The model achieved an Intersection over Union (IoU) score of 0.8745 on full-resolution, unseen test images. This result demonstrates the effectiveness of attention mechanisms in enhancing feature localization and validates the stitching methodology for generating seamless and accurate segmentation maps. Conclusion: Attention U-Net is an effective and competitive method for automated, scalable nucleus delineation in challenging clinical contexts.

Figures & tables

Figure 1: High-level representation of the preprocessing pipeline.

Figure 2: Modular representation of image preprocessing pipeline.

Figure 3: Modular representation of mask preprocessing pipeline.

Figure 4: Attention U-Net Architecture with Attention Gate.

Metric	Mean	Variance
IoU	0.8745	0.0024
Dice Coefficient	0.9323	0.0009
Accuracy	0.9721	0.0002
Precision	0.9067	0.0017
Sensitivity	0.9602	0.0005
Specificity	0.9733	0.0002

Table 1: Performance metrics for the proposed method

Model	Dice Coefficient	Jaccard Index	Reference
Baseline U-Net	0.8892	0.8031	[ 14 ]
Mask R-CNN	0.8876	0.8005	[ 14 ]
U-Net	0.7907	0.6826
CUMedVision (Challenge Winner)	0.9079	0.8354	[ 14 ]
BiU-net	0.8027	0.6722	[ 9 ]
Proposed Attention U-Net	0.9323	0.8745	This study

Table 2: Comparative Analysis of Segmentation Performance on the MoNuSeg 2018 Test Set.

Figure 7: Visual representation of information loss in histopathological images during resizing.

Figure 8: Receiver Operating Characteristic (ROC) curve for the Attention U-Net model

Histopathology Image Segmentation

Histopathology image segmentation identifies tissue regions, cells, and their boundaries in microscopic pathology images.

See all 53 papers

AMN: An Adaptive Multi-Scale Fusion Network with Boundary and Uncertainty Modeling for Nuclei Segmentation
May 31, 2026Spoorthi M, Suja PalaniswamyHistopathology Image Segmentation Hybrid CNN-Transformer Architectures
Department of Computer Science & Engineering, Amrita School of Computing, Bengaluru, Amrita Vishwa Vidyapeetham, India
NuNext: Reframing Nucleus Detection as Next-Point Detection
Mar 7, 2026Zhongyi Shui, Honglin Li, Xiaozhong Ji +7Computational Pathology Object Detection
Zhejiang University · Westlake University · Shanghai Artificial Intelligence Laboratory +3
LSP-DETR: Efficient and Scalable Nuclei Segmentation in Whole-Slide Images
Jan 6, 2026Matěj Pekár, Vít Musil, Rudolf Nenutil +2Computational Pathology Image Segmentation
Masaryk University, Faculty of Informatics, Botanická 68a, Brno, 602 00, Czech Republic · Masaryk Memorial Cancer Institute, Žlutý kopec 7, Brno, 656 53, Czech Republic · BBMRI-ERIC, Neue Stiftingtalstraße 2/B/6, Graz, 8010, Austria

May 31, 2026J/K move · Enter open · S save

AMN: An Adaptive Multi-Scale Fusion Network with Boundary and Uncertainty Modeling for Nuclei Segmentation

Spoorthi M, Suja Palaniswamy

Department of Computer Science & Engineering, Amrita School of Computing, Bengaluru, Amrita Vishwa Vidyapeetham, India

Histopathology Image Segmentation Hybrid CNN-Transformer Architectures Multi-Scale Feature Fusion Medical Image Segmentation

Open paper AskPDF arXiv

Accurate classification of nuclei subtypes in histopathology images is critical for downstream tasks including tumor grading, immune infiltrate quantification, and prognosis prediction. Existing approaches rely on either convolutional or transformer-based encoders in isolation, limiting their ability to simultaneously capture fine-grained local texture and long-range spatial context. We present AMN (Adaptive Multi-Scale Nuclei Network), a dual-encoder segmentation framework that jointly leverages a Swin Transformer and a ResNet-50 feature pyramid, fused via a learned per-channel gating mechanism that dynamically weighs each encoder's contribution at every scale. AMN is trained with a multi-objective loss combining class-weighted focal loss, boundary-aware loss with positive-pixel emphasis, and a novel uncertainty-modulated classification term that suppresses overconfident erroneous predictions. Evaluated on the CoNIC benchmark across seven nuclei classes, AMN achieves a mean Dice of 0.82 and mean F1 of 0.68, with an F1 of 0.67 on the diagnostically challenging lymphocyte class. AMN outperforms eight baseline models spanning pure-CNN, pure-transformer, and recent hybrid architectures: U-Net, ResU-Net, DeepLabV3+, SegNet, ViT-Small, HmsU-Net, ConvFormer-UNet, and BEFUnet. Cross-dataset evaluation on MoNuSeg demonstrates strong generalization without retraining and validating the domain robustness of the learned representations.

Abstract

Histopathology Image Segmentation

Explore similar work

AMN: An Adaptive Multi-Scale Fusion Network with Boundary and Uncertainty Modeling for Nuclei Segmentation

NuNext: Reframing Nucleus Detection as Next-Point Detection

LSP-DETR: Efficient and Scalable Nuclei Segmentation in Whole-Slide Images