Hierarchical Prototype-Memory Adaptation of SAM for Surgical Instrument Segmentation

Xinning Yao; Jingjing Wang; Jinghua Yue; Xiaoyan Luo; Fugen Zhou; Bo Liu

cs.CVAug 25, 2026

Hierarchical Prototype-Memory Adaptation of SAM for Surgical Instrument Segmentation

Authors: Xinning Yao, Jingjing Wang, Jinghua Yue, Xiaoyan Luo, Fugen Zhou, Bo Liu

Organizations: Image Processing Center, Beihang University, Beijing, 100191, China.

Abstract

Surgical instrument segmentation (SIS) is fundamental for computer-assisted surgery, where reliable instrument masks enable precise scene understanding and clinical assistance. Recently, adapting foundation models like the Segment Anything Model (SAM) to the surgical domain via prompt-learning has shown encouraging results. However, the performance of these adapted models under challenging surgical conditions is constrained by suboptimal adaptation mechanisms. Specifically, optimizing prompts or prototypes purely via downstream segmentation loss tends to cause them to degenerate into task-specific parameters rather than serving as persistent, stable category memory, thereby degrading their robustness against complex intraoperative variations. Moreover, routing multi-scale visual cues through a single prompt pathway creates a bottleneck that hinders effective scale-matched coupling. To address these limitations, we propose HPMA, a Hierarchical Prototype-Memory Adaptation framework for SAM. Specifically, HPMA constructs a frozen, multi-scale visual prototype memory bank from annotated surgical scenes and integrates it into SAM's feature space using lightweight adapters to preserve stable category evidence. To maximize the utility of multi-scale cues, we introduce a scale-matched coupling mechanism where global prototypes calibrate class-level prompt features, structural prototypes guide decoder object queries, and local prototypes align high-resolution feature maps through a local alignment objective. Extensive experiments on the public EndoVis2017 and EndoVis2018 datasets demonstrate that our approach achieves state-of-the-art performance, outperforming existing foundation model adaptation methods.

Figures & tables

Fig. 1: The conceptual comparison of (a) previous prompt-learning method and our (b) hierarchical prototype-memory adaptation method.

Fig. 2: Overview of the proposed HPMA framework. (a) The Multi-scale Prototype-Memory Construction extracts multi-scale FPN features from labeled surgical images using a frozen SAM3 image encoder and constructs global-, structural-, and local-level prototype banks. (b) The Hierarchical Representation Level-to-Module Coupling mechanism injects global prototypes into text features before multimodal fusion, uses structural prototypes to refine decoder object queries, and aligns high-resolution image features with local prototypes during training. (c) The Adaptation Detail of the modules at each level.

Method	Ref.	Challenge IoU	IoU	mc IoU	mDice	Instrument Categories
Method	Ref.	Challenge IoU	IoU	mc IoU	mDice	BF	PF	LND	VS	GR	MCS	UP
ISINet	MICCAI’ 20	55.62	52.20	28.96	52.00	38.70	38.50	50.09	27.43	2.01	28.72	12.56
MATIS Frame	ISBI’ 23	68.79	62.74	37.30	53.06	66.18	50.99	52.23	32.84	15.71	19.27	23.90
S3Net	WACV’ 23	72.54	71.99	46.55	66.44	75.08	54.32	61.84	35.50	27.47	43.23	28.38
SCI-Net	JBHI’ 26	72.26	69.08	-	-	-	-	-	-	-	-	-
TrackAnything	-	67.41	64.50	62.97	-	55.42	44.46	62.43	67.03	65.17	83.68	62.59

TABLE I: Comparative Results on the EndoVis2017 Dataset. The best results are presented in bold.

Method	Ref.	Challenge IoU	IoU	mc IoU	mDice	Instrument Categories
Method	Ref.	Challenge IoU	IoU	mc IoU	mDice	BF	PF	LND	MCS	UP	SI	CA
ISINet	MICCAI’ 20	73.03	70.97	40.21	71.10	73.83	48.61	30.98	88.16	2.16	37.68	0.00
MATIS Frame	ISBI’ 23	82.37	77.01	48.65	63.66	83.35	38.82	40.19	93.18	16.17	64.49	4.32
S3Net	WACV’ 23	75.81	74.02	42.58	68.02	77.22	50.87	19.83	92.12	7.44	50.59	0.00
SCI-Net	JBHI’ 26	83.59	81.10	-	-	-	-	-	-	-	-	-
TrackAnything	-	65.72	60.88	38.60	-	72.90	31.07	64.73	61.05	17.93	10.24	12.28

TABLE II: Comparative Results on the EndoVis2018 Dataset. The best results are presented in bold.

Fig. 3: The visual results of the comparative experiments. The green boxes highlight the comparison details.

Fig. 4: The attention map visualization of SAM3 and our HPMA method.

No.	Global	Structural	Local	Challenge IoU	IoU	mc IoU
1	$\times$	$\times$	$\times$	81.11	81.11	70.22
2	$\checkmark$	$\times$	$\times$	81.90	81.90	75.38
3	$\times$	$\checkmark$	$\times$	82.66	82.66	75.03
4	$\times$	$\times$	$\checkmark$	83.16	83.16	74.05
5	$\checkmark$	$\checkmark$	$\times$	83.44	83.44	76.89
6	$\checkmark$	$\checkmark$	$\checkmark$	84.12	84.12	77.20

TABLE III: Ablation Study on the EndoVis2018 Dataset.

Injection Strategy	Challenge IoU	IoU	mc IoU
Multi-scale mean fusion	82.20	82.20	74.08
Multi-scale learned fusion	82.45	82.45	71.74
G-Text, S-Dec	83.44	83.44	76.89
G-Dec, S-Text	82.60	82.60	72.50
G-Text, L-SegHead	82.98	82.98	76.44
G-Text, S-Dec, L-SegHead	82.51	82.51	76.59

TABLE IV: Ablation Study on the Injection Strategies.

Fig. 5: Ablation Study on the value of K.

Vision Foundation Model Adaptation

Vision foundation model adaptation tailors pretrained image and video models to new tasks and domains while balancing accuracy, efficiency, and generalization.

See all 323 papers

Jul 26, 2026

Parameter-Efficient Adaptation of SAM3 for Prompt-Driven Surgical Concept Segmentation

Efficient surgical segmentation empowers clinical diagnosis, intraoperative monitoring, and downstream robotic pipelines for reconstruction and simulation. Although prompt-driven foundation models like Segment Anything Model 3 (SAM3) achieve strong segmentation performance on natural images, surgical data exhibits domain gaps against its pre-training data, resulting in degraded segmentation accuracy. Furthermore, existing medical SAM methods require full-parameter fine-tuning, incurring heavy computational consumption and low efficiency. To address these limitations, this work proposes a parameter-efficient Low-Rank Adaptation (LoRA) adaptation of SAM3 for surgical concept segmentation. We inject low-rank adapters into the prompt encoder, detector and tracker while fully freezing the vision backbone, which only optimizes 0.98% of the total model parameters and supports training on a single consumer GPU. Comprehensive experiments demonstrate that our method consistently outperforms zero-shot SAM3 and other mainstream baselines, and the generated segmentation results can be directly deployed to support downstream robotic surgical scene reconstruction and physical simulation pipelines.

Changjing Liu, Yiming Huang, Beilei Cui +5

Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China · XGEN Labs

Promptable Image Segmentation Vision Foundation Model Adaptation Medical Image Segmentation Parameter-Efficient Fine-Tuning

Ask PDF

Aug 9, 2026

Toward Mask Annotation-Free Surgical Instrument Segmentation from Endoscopic Images Using Text-Prompted Segment Anything Model 3 (SAM3)

Surgical instrument segmentation is a fundamental task for computer-assisted interventions, yet most existing methods rely on pixel-level annotations or manual spatial prompts, which limit scalability and automation. The recently introduced Segment Anything Model 3 (SAM3) offers a pathway to annotation-free, automatic segmentation via text-based prompting; however, the instrument name as a text prompt could not be directly used due to a large domain gap. To overcome these limitations, we propose a two-stage framework that achieves instance-level segmentation without requiring ground truth masks or manual interaction. In the first stage, we leverage a natural-language-aligned generic prompt - "tool" - to produce binary masks using SAM3's zero-shot capability. In the second stage, these masks are extended to instance-level by integrating a vision-language model (Qwen) that is fine-tuned on SAM3-generated masked regions for instrument classification. We evaluate our approach on the EndoVis 2017 and 2018 datasets. Results show that, while our two-stage approach does not reach the performance of current fully supervised methods, it significantly outperforms the direct use of SAM3 for instance-level instrument segmentation with text prompts. Overall, our findings highlight both the limitations and potential of SAM3, suggesting a promising direction toward annotation-free surgical instrument segmentation.

Nakul Poudel, Richard Simon, Cristian A. Linte

Center for Imaging Science, Rochester Institute of Technology, Rochester, NY 14623, USA · Biomedical Engineering, Rochester Institute of Technology, Rochester, NY 14623, USA

Image Segmentation Text-Guided Image Segmentation Instance Segmentation Medical Image Segmentation

Ask PDF

Jul 8, 2026

HPR-SAM: Hierarchical Probabilistic Representation Learning for Prompt-free SAM-based Medical Image Segmentation

Prompt-free adaptation of the Segment Anything Model (SAM) has emerged as a promising paradigm for automatic medical image segmentation. Existing methods mainly focus on prompt generation, while overlooking that prompt quality is fundamentally constrained by the expressiveness of anatomical representations. However, deterministic prototypes or semantic tokens are insufficient to jointly capture global anatomical priors, intra-structure diversity, and local structural reliability. To address this limitation, we propose the Hierarchical Probabilistic Representation (HPR) framework, which learns complementary anatomical representations through Distributional Anatomical Representation (DAR), Multi-component Anatomical Representation (MAR), and Local Reliability Representation (LRR), and integrates their predictions via Hierarchical Prediction Fusion (HPF) while remaining compatible with the original SAM decoder. Experiments on the Synapse, LA, and PROMISE12 datasets demonstrate that HPR-SAM achieves state-of-the-art performance on Synapse and the best performance under few-shot settings on LA and PROMISE12, validating the effectiveness of the proposed hierarchical probabilistic representation learning framework for prompt-free medical image segmentation. Code is available at https://anonymous.4open.science/r/HPR-SAM-E4AF.

Yingzhen Hu, Yiheng Zhong, Keying Zhu +5

Yale University, United States

Medical Imaging Foundation Models Representation Learning Hierarchical Representation Learning Medical Image Segmentation

Ask PDF

Abstract

Vision Foundation Model Adaptation

Explore similar work

Parameter-Efficient Adaptation of SAM3 for Prompt-Driven Surgical Concept Segmentation

Toward Mask Annotation-Free Surgical Instrument Segmentation from Endoscopic Images Using Text-Prompted Segment Anything Model 3 (SAM3)

HPR-SAM: Hierarchical Probabilistic Representation Learning for Prompt-free SAM-based Medical Image Segmentation