GDBA-Net: A Gated Dual-Branch Adaptation of SAM2 for Robust Cross-Domain Nuclei Segmentation

Biplab Poudel; Betserai Albert Magarire; Anish Kumar Shah; Jiacheng Xie; Dong Xu

cs.CVOct 14, 2026

GDBA-Net: A Gated Dual-Branch Adaptation of SAM2 for Robust Cross-Domain Nuclei Segmentation

Authors: Biplab Poudel, Betserai Albert Magarire, Anish Kumar Shah, Jiacheng Xie, Dong Xu

Organizations: Department of Electrical Engineering and Computer Science, University of Missouri, Columbia, MO 65211, USA · Bond Life Sciences Center, University of Missouri, Columbia, MO 65211, USA · Department of Applied Computing, Lander University, Greenwood, SC 29649, USA · Department of Electronics and Computer Engineering, Sagarmatha Engineering College, Tribhuvan University, Nepal ·

Abstract

Accurate nuclei segmentation underpins quantitative computational pathology, yet remains challenging due to morphological variability, weak boundaries, nuclear crowding, and pronounced domain shifts arising from differences in tissue preparation, staining, and acquisition. While deep learning and recent foundation models have improved segmentation performance under matched distributions with the original training data, their generalization across heterogeneous histopathology domains remains limited. We introduce GDBA-Net, a parameter-efficient nuclei segmentation framework built around the proposed Gated Dual-Branch Adapter (GDBA). GDBA-Net adapts a frozen SAM2 encoder by integrating pretrained semantic representations with explicit frequency-derived structural priors. The complementary representations are independently transformed through Grid MLP branches and dynamically integrated using sample-adaptive channel-wise gating, selectively emphasizing structural cues relevant to nuclear delineation. Despite using only 4.3 million trainable parameters, GDBA-Net achieves state-of-the-art in-dataset performance and consistently superior cross-dataset generalization across six diverse nuclei segmentation benchmarks under fully supervised, cross-dataset, and five-shot adaptation protocols, outperforming competing CNN-, transformer-, and foundation model based methods. Comprehensive ablations confirm that these gains arise from the coordinated design of semantic-frequency complementarity and adaptive gating rather than increased model capacity. By uniting transferable semantic representations with explicit structural priors, GDBA-Net provides a robust, annotation-efficient solution to domain variability in nuclei segmentation. https://github.com/biplabpoudel25/GDBA_Net

Figures & tables

Fig. 1: Overall architecture of the proposed Gated Dual-Branch Adapter Network (GDBA-Net) framework for nuclei segmentation. The input H&E image is processed through multiple encoder stages, while the extracted frequency-derived structural and semantic features are integrated by the GDBA modules at the corresponding stages. The resulting representations are subsequently used to generate the predicted nuclei mask. The frequency and semantic feature extraction modules are detailed in Figure 2 .

Fig. 2: Frequency and semantic feature extraction process used in GDBA framework. The upper panel illustrates the frequency branch, whereas the lower panel shows the semantic feature extraction process. These complementary features are subsequently integrated by the GDBA modules.

Fig. 3: Overview of the proposed Gated Dual-Branch Adapter (GDBA). The upper light-gray shaded region illustrates the GDBA module, which integrates independently adapted handcrafted and semantic features to generate prompt features. The lower-left inset presents the branch adapter architecture, which is applied independently to the handcrafted and semantic feature streams. The generated prompts are combined with the current SAM2 tokens and passed to the corresponding transformer block.

Dataset	Tissue / Condition	Images	Resolution	Nuclei	Split (Train/Val/Test)
Kumar [ 25 ]	Multi-organ (7 organs)	30	1000 $\times$ 1000	21,623	12/4/14
MoNuSeg [ 26 ]	Multi-organ (9 organs)	51	1000 $\times$ 1000	$\sim$ 28,500	33/7/11
CryoNuSeg [ 27 ]	Multi-organ (10 organs)	30	512 $\times$ 512	$\sim$ 7,596	15/5/10
Combined CPM [ 28 ]	Brain, Head & Neck, Lung	79	Variable	$\sim$ 10,000	30/10/39
TNBC [ 29 ]	Breast (TNBC subtype)	50	512 $\times$ 512	4,022	32/8/10
CoNSeP [ 5 ]	Colorectal adenocarcinoma	41	1000 $\times$ 1000	24,319	22/5/14

TABLE I: Summary of benchmark datasets used for nuclei segmentation evaluation.

Models	MoNuSeg				Kumar				CryoNuSeg
Models	Dice	Jaccard	Precision	Specificity	Dice	Jaccard	Precision	Specificity	Dice	Jaccard	Precision	Specificity
U-Net [ 8 ]	0.8021	0.6712	0.7929	0.9411	0.7278	0.5775	0.6332	0.8149	0.7737	0.6349	0.7400	0.8531
Attn. U-Net [ 30 ]	0.7903	0.6551	0.7615	0.9264	0.7609	0.6212	0.7451	0.8833	0.7596	0.6176	0.7164	0.8410
nnU-Net [ 31 ]	0.8218	0.7091	0.8015	0.9396	0.8199	0.6966	0.8177	0.9424	0.8108	0.6845	0.8144	0.9186
PSPNet [ 32 ]	0.5922	0.4311	0.4347	0.6061	0.6014	0.4348	0.4402	0.6274	0.6805	0.5204	0.5361	0.6042
HoVer-Net [ 5 ]	0.8208	0.7086	0.7996	0.9399	0.7981	0.6749	0.7741	0.9223	0.8284	0.7125	0.8452	0.9231

TABLE II: In-dataset nuclei segmentation performance comparison of GDBA-Net and competing methods on the MoNuSeg, CryoNuSeg, and Kumar datasets. Best and second-best results are highlighted in bold and underline, respectively.

	Methods	Test dataset
	Methods	Combined CPM				TNBC				Combined CoNSeP
		Dice	Jaccard	Precision	Specificity	Dice	Jaccard	Precision	Specificity	Dice	Jaccard	Precision	Specificity
Trained on MoNuSeg	U-Net	0.7329	0.5878	0.8281	0.9640	0.6732	0.5187	0.7145	0.9642	0.6236	0.4678	0.5698	0.9000
	Attn U-Net	0.7262	0.5799	0.8170	0.9589	0.7027	0.5529	0.7438	0.9629	0.6003	0.4477	0.5147	0.8568
	nnU-Net	0.7801	0.6604	0.8004	0.9440	0.7025	0.5712	0.6132	0.9065	0.6322	0.4854	0.5444	0.8407
	PSPNet	0.6062	0.4565	0.4674	0.6863	0.2606	0.1634	0.1639	0.1928	0.4142	0.2844	0.2900	0.4596

TABLE III: Cross-dataset generalization performance of GDBA-Net and competing methods trained on MoNuSeg, CryoNuSeg, or CoNSeP and directly evaluated on the unseen Combined CPM, TNBC, and Combined CoNSeP test datasets. For each source–target dataset pair and metric, the best and second-best results are highlighted in bold and underline, respectively.

Fig. 4: Boxplot-based comparison of nuclei segmentation performance under in-dataset and cross-dataset evaluation. The top panel presents results when models are trained and tested on CryoNuSeg, whereas the bottom panel shows cross-dataset performance when models are trained on CryoNuSeg and evaluated on CPM dataset. Boxes represent the interquartile range, horizontal lines indicate the median, whiskers show the non-outlier range, and circular markers denote outlier samples.

Fig. 5: Comparison of dice scores for GDBA across cross-dataset transfer, five-shot adaptation, and full target-domain fine-tuning on the CPM, TNBC, and CoNSeP datasets. Cross-dataset models are trained on MoNuSeg or CryoNuSeg and evaluated directly on each unseen target dataset.

Fig. 6: Qualitative comparison of nuclei segmentation results on representative samples. Ground-truth masks and overlap maps produced by the competing methods are shown for visual comparison. In the overlay, white denotes correctly segmented nuclei pixels, red denotes false positives predictions, and blue indicates false negatives predictions.

Fig. 7: Comparison of trainable parameter counts for GDBA and competing segmentation methods.

Models	ISIC 2018 [ 44 , 45 ]		DRIVE [ 43 ]
Models	Dice	Jaccard	Dice	Jaccard
U-Net	0.858	0.776	0.816	0.701
nnU-Net	0.855	0.766	0.780	0.640
Attn U-Net	0.867	0.785	0.808	0.681
NucleiSegNet	0.881	0.808	0.650	0.482
MedSAM	0.863	0.759	0.806	0.676

TABLE IV: Performance comparison of GDBA-Net and competing methods on ISIC 2018 and DRIVE for skin lesion and retinal vessel segmentation.

Frequency branch	Semantic branch	Adaptive gate	Shared projections	Params (M)	MoNuSeg				CryoNuSeg
Frequency branch	Semantic branch	Adaptive gate	Shared projections	Params (M)	Dice	Jacc.	Prec.	Spec.	Dice	Jacc.	Prec.	Spec.
$\times$	$\checkmark$	$\checkmark$	$\checkmark$	4.11	0.8145	0.6893	0.8028	0.9332	0.8331	0.7156	0.8618	0.9369
$\checkmark$	$\times$	$\checkmark$	$\checkmark$	4.07	0.8087	0.6813	0.7954	0.9292	0.8360	0.7196	0.8248	0.9123
$\checkmark$	$\checkmark$	$\times$	$\checkmark$	4.28	0.8108	0.6841	0.7982	0.9306	0.8340	0.7173	0.8686	0.9396
$\checkmark$	$\checkmark$	$\checkmark$	$\times^{*}$	4.88	0.8048	0.6758	0.7907	0.9268	0.8402	0.7260	0.8568	0.9311
$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	4.30	0.8408	0.7259	0.8388	0.9516	0.8481	0.7381	0.8519	0.9286

TABLE V: Component-wise ablation analysis of the proposed GDBA framework on the MoNuSeg and CryoNuSeg datasets.

$\alpha$	Params (M)	MoNuSeg				CryoNuSeg
$\alpha$	Params (M)	Dice	Jacc.	Prec.	Spec.	Dice	Jacc.	Prec.	Spec.
8	10.23	0.8176	0.6941	0.8068	0.9347	0.8368	0.7213	0.8591	0.9332
16	5.54	0.8189	0.6958	0.8087	0.9359	0.8352	0.7189	0.8618	0.9250
32	4.30	0.8408	0.7259	0.8388	0.9516	0.8481	0.7381	0.8519	0.9286
64	3.97	0.8194	0.6963	0.8096	0.9361	0.8372	0.7213	0.8475	0.9276

TABLE VI: Effect on bottleneck scaling factor

\alpha

on model performance and parameter efficiency. Larger

\alpha

corresponds to stronger dimensionality reduction.

Histopathology Image Segmentation

Histopathology image segmentation identifies tissue regions, cells, and their boundaries in microscopic pathology images.

See all 53 papers

Prompt-Free Lightweight SAM Adaptation for Histopathology Nuclei Segmentation with Strong Cross-Dataset Generalization
Mar 20, 2026Muhammad Hassan Maqsood, Yanming Zhu, Alfred Lam +3Histopathology Image Segmentation Vision Foundation Model Adaptation
School of Information and Communication Technology, Griffith University, QLD, Australia · School of MDP – Clinical Medicine, Griffith University, QLD, Australia
AMN: An Adaptive Multi-Scale Fusion Network with Boundary and Uncertainty Modeling for Nuclei Segmentation
May 31, 2026Spoorthi M, Suja PalaniswamyHistopathology Image Segmentation Hybrid CNN-Transformer Architectures
Department of Computer Science & Engineering, Amrita School of Computing, Bengaluru, Amrita Vishwa Vidyapeetham, India
Adapting SAM to Nuclei Instance Segmentation and Classification via Cooperative Fine-Grained Refinement
Mar 30, 2026Jingze Su, Tianle Zhu, Jiaxin Cai +6Instance Segmentation Histopathology Image Segmentation
College of Computer and Data Science, Fuzhou University, Fuzhou, China · College of Physics and Information Engineering, Fuzhou University, Fuzhou, China · School of Mechanical Engineering and Automation, Fuzhou University, Fuzhou, China

Mar 20, 2026J/K move · Enter open · S save

Prompt-Free Lightweight SAM Adaptation for Histopathology Nuclei Segmentation with Strong Cross-Dataset Generalization

Muhammad Hassan Maqsood, Yanming Zhu, Alfred Lam, Getamesay Dagnaw+2

School of Information and Communication Technology, Griffith University, QLD, Australia · School of MDP – Clinical Medicine, Griffith University, QLD, Australia

Histopathology Image Segmentation Vision Foundation Model Adaptation Cross-Dataset Generalization

Open paper AskPDF arXiv

Histopathology nuclei segmentation is crucial for quantitative tissue analysis and cancer diagnosis. Although existing segmentation methods have achieved strong performance, they are often computationally heavy and show limited generalization across datasets, which constrains their practical deployment. Recent SAM-based approaches have shown great potential in general and medical imaging, but typically rely on prompt guidance or complex decoders, making them less suitable for histopathology images with dense nuclei and heterogeneous appearances. We propose a prompt-free and lightweight SAM adaptation that leverages multi-level encoder features and residual decoding for accurate and efficient nuclei segmentation. The framework fine-tunes only LoRA modules within the frozen SAM encoder, requiring just 4.1M trainable parameters. Experiments on three benchmark datasets TNBC, MoNuSeg, and PanNuke demonstrate state-of-the-art performance and strong cross-dataset generalization, highlighting the effectiveness and practicality of the proposed framework for histopathology applications.

Abstract

Histopathology Image Segmentation

Explore similar work

Prompt-Free Lightweight SAM Adaptation for Histopathology Nuclei Segmentation with Strong Cross-Dataset Generalization

AMN: An Adaptive Multi-Scale Fusion Network with Boundary and Uncertainty Modeling for Nuclei Segmentation

Adapting SAM to Nuclei Instance Segmentation and Classification via Cooperative Fine-Grained Refinement