Adapting SAM to Nuclei Instance Segmentation and Classification via Cooperative Fine-Grained Refinement
Authors: Jingze Su, Tianle Zhu, Jiaxin Cai, Zhiyi Wang, Qi Li, Xiao Zhang, Tong Tong, Shu Wang, Wenxi Liu
Organizations: College of Computer and Data Science, Fuzhou University, Fuzhou, China · College of Physics and Information Engineering, Fuzhou University, Fuzhou, China · School of Mechanical Engineering and Automation, Fuzhou University, Fuzhou, China
Nuclei instance segmentation is critical in computational pathology for cancer diagnosis and prognosis. Recently, the Segment Anything Model has demonstrated exceptional performance in various segmentation tasks, leveraging its rich priors and powerful global context modeling capabilities derived from large-scale pre-training on natural images. However, directly applying SAM to the medical imaging domain faces significant limitations: it lacks sufficient perception of the local structural features that are crucial for nuclei segmentation, and full fine-tuning for downstream tasks requires substantial computational costs. To efficiently transfer SAM's robust prior knowledge to nuclei instance segmentation while supplementing its task-aware local perception, we propose a parameter-efficient fine-tuning framework, named Cooperative Fine-Grained Refinement of SAM, consisting of three core components: 1) a Multi-scale Adaptive Local-aware Adapter, which enables effective capability transfer by augmenting the frozen SAM backbone with minimal parameters and instilling a powerful perception of local structures through dynamically generated, multi-scale convolutional kernels; 2) a Hierarchical Modulated Fusion Module, which dynamically aggregates multi-level encoder features to preserve fine-grained spatial details; and 3) a Boundary-Guided Mask Refinement, which integrates multi-context boundary cues with semantic features through explicit supervision, producing a boundary-focused signal to refine initial mask predictions for sharper delineation. These three components work cooperatively to enhance local perception, preserve spatial details, and refine boundaries, enabling SAM to perform accurate nuclei instance segmentation directly.
Figures & tables
Figure 1: This figure illustrates our motivation. Compared to the segmentation results of a full fine-tuned SAM (second row), our CFR-SAM achieves superior precision, particularly in capturing faint boundaries and distinguishing between densely clustered objects.
Figure 2: The pipeline illustrates the two-stage procedure of CFR-SAM. First, a Prompt Learner predicts nuclei centroids to generate point prompts. Second, a PEFT strategy adapts the SAM for nuclear instance segmentation. The final instance segmentation results are displayed with contours overlaid on each detected nucleus for clarity.
Figure 3: The structure of the proposed MALA is detailed, showcasing its role in enhancing the frozen Transformer backbone.
Figure 4: The design of the HMFM is presented, highlighting its mechanism for aggregating multi-level encoder features.
Figure 5: The workflow of the BGMR is depicted, demonstrating the process of integrating boundary cues to refine the initial mask prediction.
Model
Mask R-CNN
HoVer-Net
CPP-Net
PointNu-Net
CellViT-H
CFR-SAM-B
CFR-SAM-H
( He et al., 2017 )
( Graham et al., 2019 )
( Chen et al., 2023b )
( Yao et al., 2023 )
( Hörst et al., 2024 )
Ours
Ours
Params* (M)
41.9
122.8
37.6
122.8
699.7
63.4
74.2
Tissue
bPQ
mPQ
bPQ
mPQ
bPQ
mPQ
bPQ
mPQ
bPQ
mPQ
bPQ
mPQ
bPQ
mPQ
Adrenal
0.5546
0.3470
0.6962
0.4812
0.7031
0.4922
0.7134
0.5115
0.7086
0.5134
0.7161
0.5111
0.7195
0.5140
Bile Duct
0.5567
0.3536
0.6696
0.4714
0.6739
0.4650
0.6814
0.4868
0.6784
0.4887
0.6931
0.4952
0.6935
0.4953
Bladder
0.6049
0.5065
0.7031
0.5792
0.7057
0.5932
0.7226
0.6065
0.7068
0.5844
0.7234
0.5902
0.7252
0.5920
Table 1: Performance comparison on the PanNuke dataset for nuclei instance segmentation. Following ( Chen et al., 2023b ; Hörst et al., 2024 ) , both binary PQ (bPQ) and multi-class PQ (mPQ) are computed for evaluation. Red / Blue represents the best/second performance. Params* denotes trainable parameters of model.
Model
Trainable
mPQ
bPQ
AJI
Params (M)
CellViT-H
699.7
0.4980
0.6793
0.6519
Ours-B
63.4
0.5091 ± 0.0013
0.6923 ± 0.0020
0.6865 ± 0.0001
Ours-H
74.2
0.5115 ± 0.0003
0.6956 ± 0.0004
0.6887 ± 0.0004
Table 2: Performance of our CFR-SAM on PanNuke over 5 independent runs.
Model
Detection
Classification
Neoplastic
Epithelial
Inflammatory
Connective
Dead
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
DIST ( Naylor et al., 2018 )
0.74
0.71
0.73
0.49
0.55
0.50
0.38
0.33
0.35
0.42
0.45
0.42
0.42
0.37
0.39
0.00
0.00
0.00
Mask-RCNN ( He et al., 2017 )
0.76
0.68
0.72
0.55
0.63
0.59
0.52
0.52
0.52
0.46
0.54
0.50
0.42
0.43
0.42
0.17
0.30
0.22
Micro-Net ( Raza et al., 2019 )
0.78
0.82
0.80
0.59
0.66
0.62
0.63
0.54
0.58
0.59
0.46
0.52
0.50
0.45
0.47
0.23
0.17
0.19
HoVer-Net ( Graham et al., 2019 )
0.82
0.79
0.80
0.58
0.67
0.62
0.54
0.60
0.56
0.56
0.51
0.54
0.52
0.47
0.49
0.28
0.35
0.31
Table 3: Comparison of Precision (P), Recall (R), and F1 -score ( F1 ) for nuclei detection and classification across all categories. Results are averaged over the three PanNuke splits.
Model
N
E
I
C
D
DIST
0.439
0.290
0.343
0.275
0.000
Mask-RCNN
0.472
0.403
0.290
0.300
0.069
Micro-Net
0.504
0.442
0.333
0.334
0.051
HoVer-Net
0.551
0.491
0.417
0.388
0.139
STARDIST
0.547
0.532
0.424
0.380
0.123
CPP-Net
0.571
0.565
0.405
0.395
0.131
Table 4: Average PQ for each nuclear category across the three PanNuke splits. N, E, I, C, and D represent Neoplastic, Epithelial, Inflammatory, Connective, and Dead, respectively.
Figure 6: Visual comparison of our method and CellViT on the PanNuke dataset for nuclei instance segmentation. Each predicted and GT instance mask is assigned a distinct color at random for visual separation. The red boxes highlight the better results achieved by our model.
Figure 7: Visual comparison of our method and CellViT on the PanNuke dataset for nuclei classification. Nuclear types are represented by distinct boundary colors.
Model
AJI
PQ
U-Net ( Ronneberger et al., 2015 )
0.666
0.625
DCAN ( Chen et al., 2017 )
0.561
0.545
Mask-RCNN ( He et al., 2017 )
0.684
0.674
DIST ( Naylor et al., 2018 )
0.616
0.504
Micro-Net ( Raza et al., 2019 )
0.668
0.661
Hover-Net ( Graham et al., 2019 )
0.705
0.697
Table 5: Performance comparison on the CPM-17 dataset for nuclei instance segmentation.
Model
AJI
PQ
U-Net ( Ronneberger et al., 2015 )
0.556
0.478
Mask-RCNN ( He et al., 2017 )
0.546
0.365
DCAN ( Chen et al., 2017 )
0.477
0.488
DIST ( Naylor et al., 2018 )
0.559
0.443
Micro-Net ( Raza et al., 2019 )
0.560
0.519
CIA-Net ( Zhou et al., 2019 )
0.620
0.577
Table 6: Performance comparison on the MoNuSeg dataset for nuclei instance segmentation. † indicates our reproduced results.
Model
Trainable
mPQ
bPQ
Params (M)
Different Fine-Tuning Method
FFT
93.7
0.4940
0.6754
Head Only
4.1
0.4848
0.6645
Ours
8.3
0.5046
0.6912
Multi-Scale Adaptive Local-Aware Adapter
Table 7: Ablation study of different modules. All trainable parameters are from the second-stage fine-tuning of SAM. The symbol ∗ indicates that we increase the low-rank dimension to match the trainable parameters.
Figure 8: Compared to the segmentation results of a full fine-tuned SAM (second row), our CFR-SAM achieves superior precision, particularly in capturing faint boundaries and distinguishing between densely clustered objects.
Figure 9: Visual comparison of the BGMR ablation study for nuclei instance segmentation. Nuclear types are represented by distinct boundary colors.
Figure 10: Visual comparison of different adapting strategies for nuclei instance segmentation.
Model
Trainable
mPQ
bPQ
Params (M)
Adapter ( Houlsby et al., 2019 )
6.4
0.4969
0.6799
Adapter ∗ ( Houlsby et al., 2019 )
8.0
0.4955
0.6782
Lora ( Hu et al., 2022 )
12.8
0.4923
0.6741
MALA
7.8
0.4983
0.6821
Table 8: Ablation study of different PEFT methods. It is important to note that these results are obtained using SAM adapted solely by the respective PEFT methods, without any other auxiliary modules. The symbol ∗ indicates that we increase the low-rank dimension of Adapter to approximately match the trainable parameters of MALA for a fair comparison.
Model
Trainable
mPQ
bPQ
Params (M)
MALA (16)
5.6
0.5033
0.6892
MALA (32)
6.5
0.5031
0.6889
MALA (64)
8.3
0.5046
0.6912
MALA (128)
12.1
0.5043
0.6911
Table 9: Ablation study of different low-rank dimension.
Model
mPQ
bPQ
Mask (20)
0.5043
0.6908
Mask (25)
0.5046
0.6912
Mask (30)
0.5046
0.6910
Mask (35)
0.5045
0.6906
Table 10: Ablation study on the number of sampled nuclei per image during training.
Model
mPQ
bPQ
Point (Ours-B)
0.5046
0.6912
Box (Ours-B)
0.4804
0.6612
Table 11: Comparison of point prompts and bounding box prompts for SAM adaptation.
Model
Trainable
Total
MACs (G)
FPS
Params (M)
Params (M)
STARDIST ( Schmidt et al., 2018 )
122.8
122.8
263.6
17
HoVer-Net ( Graham et al., 2019 )
37.6
37.6
150.0
7
CPP-Net ( Chen et al., 2023b )
122.8
122.8
264.4
14
PointNu-Net ( Yao et al., 2023 )
158.1
158.1
335.1
11
CellViT-B ( Hörst et al., 2024 )
142.9
142.9
232.0
20
Table 12: Comparison of model parameters, computational cost and inference speed on the PanNuke dataset. The FPS of all methods is tested in a single NVIDIA RTX 3090 GPU.
School of Information and Communication Technology, Griffith University, QLD, Australia · School of MDP – Clinical Medicine, Griffith University, QLD, Australia
Department of Electrical Engineering and Computer Science, University of Missouri, Columbia, MO 65211, USA · Bond Life Sciences Center, University of Missouri, Columbia, MO 65211, USA · Department of Applied Computing, Lander University, Greenwood, SC 29649, USA +2