Prompt-Free Lightweight SAM Adaptation for Histopathology Nuclei Segmentation with Strong Cross-Dataset Generalization

Muhammad Hassan Maqsood; Yanming Zhu; Alfred Lam; Getamesay Dagnaw; Xuefei Yin; Alan Wee-Chung Liew

cs.CVMar 20, 2026

Prompt-Free Lightweight SAM Adaptation for Histopathology Nuclei Segmentation with Strong Cross-Dataset Generalization

Authors: Muhammad Hassan Maqsood, Yanming Zhu, Alfred Lam, Getamesay Dagnaw, Xuefei Yin, Alan Wee-Chung Liew

Organizations: School of Information and Communication Technology, Griffith University, QLD, Australia · School of MDP – Clinical Medicine, Griffith University, QLD, Australia

Abstract

Histopathology nuclei segmentation is crucial for quantitative tissue analysis and cancer diagnosis. Although existing segmentation methods have achieved strong performance, they are often computationally heavy and show limited generalization across datasets, which constrains their practical deployment. Recent SAM-based approaches have shown great potential in general and medical imaging, but typically rely on prompt guidance or complex decoders, making them less suitable for histopathology images with dense nuclei and heterogeneous appearances. We propose a prompt-free and lightweight SAM adaptation that leverages multi-level encoder features and residual decoding for accurate and efficient nuclei segmentation. The framework fine-tunes only LoRA modules within the frozen SAM encoder, requiring just 4.1M trainable parameters. Experiments on three benchmark datasets TNBC, MoNuSeg, and PanNuke demonstrate state-of-the-art performance and strong cross-dataset generalization, highlighting the effectiveness and practicality of the proposed framework for histopathology applications.

Figures & tables

Figure 1: Overview of the proposed framework. A frozen SAM image encoder is adapted with LoRA modules inserted into its attention layers for parameter-efficient fine-tuning. Multi-level features from selected Transformer blocks are processed by lightweight residual decoder blocks and then fused to produce the final segmentation mask.

Model	TNBC		MoNuSeg		PanNuke		Params
Model	Dice (%)	IoU (%)	Dice (%)	IoU (%)	Dice (%)	IoU (%)	(M)
UNet [ 16 ]	67.84	52.46	68.99	53.20	71.41	58.40	31
Swin-Unet [ 2 ]	75.89	61.67	78.25	64.61	77.63	65.09	49
UCTransNet [ 19 ]	79.13	65.83	79.08	65.50	79.71	69.40	66
TSCA-Net [ 3 ]	81.90	69.75	80.23	67.13	–	–	$\sim 25$
SAM [ 9 ]	62.56	49.87	76.60	62.30	66.03	52.76	93

Table 1: Quantitative comparison of segmentation performance on the TNBC, MoNuSeg, and PanNuke datasets. Bold indicates the best results. Underline denotes the second best.

Model	TNBC $\rightarrow$		MoNuSeg $\rightarrow$		PanNuke $\rightarrow$
Model	MoNuSeg	PanNuke	TNBC	PanNuke	TNBC	MoNuSeg
UNet [ 16 ]	66.0/49.91	68.03/54.36	16.20/10.31	66.49/53.17	12.39/7.28	51.43/34.90
Swin-Unet [ 2 ]	68.81/52.88	63.63/49.15	41.92/30.52	56.63/41.22	39.40/27.84	74.20/59.14
UCTransNet [ 19 ]	61.70/45.42	58.73/44.16	68.80/53.58	51.70/36.73	37.20/28.53	36.38/24.36
TSCA-Net [ 3 ]	89.01 / 65.97	– / –	46.76/41.36	– / –	– / –	– / –
SAM [ 9 ]	78.95/65.43	26.84/16.51	51.28/39.51	54.35/40.34	60.45/44.89	76.16/61.81

Table 2: Cross-dataset generalization of nuclei segmentation models, reported as Dice/IoU percentages. Bold indicates the best result and underline the second best.

Figure 2: Qualitative segmentation results on TNBC (Col 1-3), PanNuke (Col 4-5), and MoNuSeg (Col 6-8) datasets. Input images (top), ground truth annotations (middle), and predicted masks (bottom) are shown.

Variant	TNBC		MoNuSeg		PanNuke
Variant	Dice (%)	IoU (%)	Dice (%)	IoU (%)	Dice (%)	IoU (%)
No LoRA	81.60	69.22	82.74	70.65	82.11	71.10
No bias prior	82.14	70.10	82.74	70.68	83.36	73.03
Single-Level (12)	79.45	66.10	83.28	71.46	82.97	72.40
Dual-Level (10,12)	82.42	70.58	82.78	70.73	83.14	72.69
Ours (full)	82.90	70.90	82.85	70.85	83.60	73.26

Table 3: Ablation studies on TNBC, MoNuSeg, and PanNuke datasets. Bold indicates the best result and underline the second best.

Histopathology Image Segmentation

Histopathology image segmentation identifies tissue regions, cells, and their boundaries in microscopic pathology images.

See all 53 papers

GDBA-Net: A Gated Dual-Branch Adaptation of SAM2 for Robust Cross-Domain Nuclei Segmentation
Oct 14, 2026Biplab Poudel, Betserai Albert Magarire, Anish Kumar Shah +2Histopathology Image Segmentation Medical Image Segmentation
Department of Electrical Engineering and Computer Science, University of Missouri, Columbia, MO 65211, USA · Bond Life Sciences Center, University of Missouri, Columbia, MO 65211, USA · Department of Applied Computing, Lander University, Greenwood, SC 29649, USA +2
Adapting SAM to Nuclei Instance Segmentation and Classification via Cooperative Fine-Grained Refinement
Mar 30, 2026Jingze Su, Tianle Zhu, Jiaxin Cai +6Instance Segmentation Histopathology Image Segmentation
College of Computer and Data Science, Fuzhou University, Fuzhou, China · College of Physics and Information Engineering, Fuzhou University, Fuzhou, China · School of Mechanical Engineering and Automation, Fuzhou University, Fuzhou, China
AMN: An Adaptive Multi-Scale Fusion Network with Boundary and Uncertainty Modeling for Nuclei Segmentation
May 31, 2026Spoorthi M, Suja PalaniswamyHistopathology Image Segmentation Hybrid CNN-Transformer Architectures
Department of Computer Science & Engineering, Amrita School of Computing, Bengaluru, Amrita Vishwa Vidyapeetham, India

Oct 14, 2026J/K move · Enter open · S save

GDBA-Net: A Gated Dual-Branch Adaptation of SAM2 for Robust Cross-Domain Nuclei Segmentation

Biplab Poudel, Betserai Albert Magarire, Anish Kumar Shah, Jiacheng Xie+1

Department of Electrical Engineering and Computer Science, University of Missouri, Columbia, MO 65211, USA · Bond Life Sciences Center, University of Missouri, Columbia, MO 65211, USA · Department of Applied Computing, Lander University, Greenwood, SC 29649, USA+2

Histopathology Image Segmentation Medical Image Segmentation Vision Foundation Model Adaptation Few-Shot Domain Adaptation

Open paper AskPDF arXiv

Accurate nuclei segmentation underpins quantitative computational pathology, yet remains challenging due to morphological variability, weak boundaries, nuclear crowding, and pronounced domain shifts arising from differences in tissue preparation, staining, and acquisition. While deep learning and recent foundation models have improved segmentation performance under matched distributions with the original training data, their generalization across heterogeneous histopathology domains remains limited. We introduce GDBA-Net, a parameter-efficient nuclei segmentation framework built around the proposed Gated Dual-Branch Adapter (GDBA). GDBA-Net adapts a frozen SAM2 encoder by integrating pretrained semantic representations with explicit frequency-derived structural priors. The complementary representations are independently transformed through Grid MLP branches and dynamically integrated using sample-adaptive channel-wise gating, selectively emphasizing structural cues relevant to nuclear delineation. Despite using only 4.3 million trainable parameters, GDBA-Net achieves state-of-the-art in-dataset performance and consistently superior cross-dataset generalization across six diverse nuclei segmentation benchmarks under fully supervised, cross-dataset, and five-shot adaptation protocols, outperforming competing CNN-, transformer-, and foundation model based methods. Comprehensive ablations confirm that these gains arise from the coordinated design of semantic-frequency complementarity and adaptive gating rather than increased model capacity. By uniting transferable semantic representations with explicit structural priors, GDBA-Net provides a robust, annotation-efficient solution to domain variability in nuclei segmentation. https://github.com/biplabpoudel25/GDBA_Net

Abstract

Histopathology Image Segmentation

Explore similar work

GDBA-Net: A Gated Dual-Branch Adaptation of SAM2 for Robust Cross-Domain Nuclei Segmentation

Adapting SAM to Nuclei Instance Segmentation and Classification via Cooperative Fine-Grained Refinement

AMN: An Adaptive Multi-Scale Fusion Network with Boundary and Uncertainty Modeling for Nuclei Segmentation