Unlocking Positive Transfer in Incrementally Learning Surgical Instruments: A Self-reflection Hierarchical Prompt Framework

Yu Zhu; Kang Li; Zheng Li; Pheng-Ann Heng

cs.CVApr 3, 2026

Unlocking Positive Transfer in Incrementally Learning Surgical Instruments: A Self-reflection Hierarchical Prompt Framework

Authors: Yu Zhu, Kang Li, Zheng Li, Pheng-Ann Heng

Organizations: The Chinese University of Hong Kong · University of Electronic Science and Technology of China

Abstract

To continuously enhance model adaptability in surgical video scene parsing, recent studies incrementally update it to progressively learn to segment an increasing number of surgical instruments over time. However, prior works constantly overlooked the potential of positive forward knowledge transfer, i.e., how past knowledge could help learn new classes, and positive backward knowledge transfer, i.e., how learning new classes could help refine past knowledge. In this paper, we propose a self-reflection hierarchical prompt framework that unlocks the power of positive forward and backward knowledge transfer in class incremental segmentation, aiming to proficiently learn new instruments, improve existing skills of regular instruments, and avoid catastrophic forgetting of old instruments. Our framework is built on a frozen, pre-trained model that adaptively appends instrument-aware prompts for new classes throughout training episodes. To enable positive forward knowledge transfer, we organize instrument prompts into a hierarchical prompt parsing tree with the instrument-shared prompt partition as the root node, n-part-shared prompt partitions as intermediate nodes and instrument-distinct prompt partitions as leaf nodes, to expose the reusable historical knowledge for new classes to simplify their learning. Conversely, to encourage positive backward knowledge transfer, we conduct self-reflection refining on existing knowledge by directed-weighted graph propagation, examining the knowledge associations recorded in the tree to improve its representativeness without causing catastrophic forgetting. Our framework is applicable to both CNN-based models and advanced transformer-based foundation models, yielding more than 5% and 11% improvements over the competing methods on two public benchmarks respectively.

Figures & tables

Figure 1: (a) Prior works learn each instrument prompt independently. (b,c) They overlook the power of positive forward and backward transfer between early-acquired and newly-acquired knowledge. (d) In contrast, our work organizes instrument prompts from general to specific as a hierarchical prompt parsing tree. (e) It supports positive forward transfer by helping new instrument prompts inherit profitable knowledge from the past to assist new-class learning. (f) It further enables positive backward transfer by letting the newly-acquired knowledge wisely refine early-acquired knowledge to improve old-class segmentation.

Figure 2: Overview of our self-reflection hierarchical prompt framework. We progressively append instrument-aware prompts into a pre-trained model during the training lifespan. For each new class, before diving into learning its specific prompts, we first build a hierarchical prompt parsing tree to discover the profitable knowledge it could inherit from the past, i.e. , the instrument-shared (IS) and $n$ -part-shared knowledge, promoting new-class learning by positive forward knowledge transfer. Simply learning one instrument-distinct (ID) prompt partition would be sufficient to delineate its contour. Based on the newly acquired ID partition, we trigger self-reflection on existing knowledge to support positive backward knowledge transfer, wisely refining their representativeness within the context of all seen classes without causing catastrophic forgetting.

Figure 3: Visual comparisons of our approaches and highly competitive approaches. More comparison results are in the supplementary.

Method	Old Class		Regular Class					New Class		Old Class	Regular Class	New Class	All	BWT	FWT
Method	VS	GR	PF	BF	UP	LND	MCS	SI	CA	Old Class	Regular Class	New Class	All	BWT	FWT
Ind-T ( $t=1$ )	75.12	62.50	68.27	66.22	64.65	71.89	70.91	-	-	68.81	68.39	-	68.51	-	-
Ind-T ( $t=2$ )	-	-	69.67	67.67	66.44	70.63	70.80	32.98	9.18	-	69.04	21.08	55.34	-	-
Joint-T	83.46	70.91	79.79	71.45	71.88	71.08	75.04	35.94	8.90	77.18	73.85	22.42	63.16	-	-
Seq-F	4.90	2.97	48.73	52.23	44.81	46.73	37.47	9.45	1.29	3.93	45.99	5.37	27.62	-8.92	0.14
SI	6.51	2.65	50.26	58.91	29.58	51.29	42.02	11.48	1.82	4.58	46.41	6.65	28.28	-8.44	1.42

Table 1: Porcine robot-assisted nephrectomy instrument segmentation results for the last episode evaluated by IoU and all episodes assessed by BWT and FWT. For all metrics, the higher the better. We highlight the best and second-best results in bold and underline , respectively.

Method	Old Class	Regular Class	New Class					New Class	All	BWT	FWT
Method	LH	Gr	Cl	Sc	Ir	SB	Bi	New Class	All	BWT	FWT
Ind-T ( $t=1$ )	76.04	74.11	-	-	-	-	-	-	75.08	-	-
Ind-T ( $t=2$ )	-	79.40	69.34	47.02	39.05	63.91	59.56	55.78	59.71	-	-
Joint-T	79.54	80.88	71.07	49.03	39.64	67.65	57.87	57.05	63.67	-	-
Seq-F	39.07	53.67	62.59	49.61	32.18	47.14	43.50	47.01	46.82	-12.52	-1.00
SI	41.65	54.30	63.15	49.70	31.86	50.75	47.60	48.61	48.43	-10.92	0.60

Table 2: Laparoscopic cholecystectomy instrument segmentation results for the last episode evaluated by IoU and all episodes assessed by BWT and FWT. For all metrics, the higher the better. We highlight the best and second-best results in bold and underline , respectively.

DLv3+	SAM	HPPT	SRS	Old Classes	Regular Classes	New Classes	All
	✓			21.42	69.11	22.24	48.15
	✓	✓		65.29	69.23	22.16	57.89
	✓	✓	✓	68.52	70.67	22.30	59.44
✓				3.93	45.99	5.37	27.62
✓		✓	✓	22.46	59.79	12.19	40.92

Table 3: Ablation study of our key components.

Figure 4: Effects of the decay factor $\gamma$ and the teleport probability $\alpha$ on model performance.

Metric	SAM	Ours	$\Delta$ vs. SAM
Parameters (M)	636	639	+0.47%
FLOPs (G)	2733	2751	+0.65%
Train. time (s per iteration)	0.901	0.915	+1.55%
Infer. time (s per slice)	0.464	0.466	+0.43%

Table 4: Computational cost analysis.

Figure 1: Illustration of HPPT construction.

Figure 2: Visualization comparison of segmentation results across all methods. Rows 1–2, 4-5 : previous datasets. Row 3, 6 : current dataset.

Figure 3: Automatic estimation of instrument part count.

Instrument	Total Cutout	Part Count Inferred by MLLM				GT
		1	2	3	Result
Large Needle Driver	1197	156	213	828	3	3
Prograsp Forceps	1051	55	232	764	3	3
Bipolar Forceps	657	189	132	336	3	3
Ultrasound Probe	449	294	103	52	1	1
Curved Scissors	351	54	205	92	2	2

Table 1: Part Count Inference Results on EndoVis2017 Dataset

Instrument	Total Cutout	Part Count Inferred by MLLM				GT
		1	2	3	Result
Large Needle Driver	278	58	96	124	3	3
Prograsp Forceps	972	167	232	573	3	3
Bipolar Forceps	1757	395	132	1230	3	3
Ultrasound Probe	156	103	34	52	1	1
Curved Scissors	1554	205	1257	92	2	2

Table 2: Part Count Inference Results on EndoVis2018 Dataset

Class-Incremental Learning

Class-incremental learning studies how models can add new categories over time while retaining the ability to recognize previously learned classes.

See all 159 papers

Oct 26, 2026

Rethinking Data Imbalance in Class Incremental Surgical Instrument Segmentation

In surgical instrument segmentation, the increasing variety of instruments over time poses a significant challenge for existing neural networks, as they are unable to effectively learn such incremental tasks and suffer from catastrophic forgetting. When learning new data, the model experiences a sharp performance drop on previously learned data. Although several continual learning methods have been proposed for incremental understanding tasks in surgical scenarios, the issue of data imbalance often leads to a strong bias in the segmentation head, resulting in poor performance. Data imbalance can occur in two forms: (i) class imbalance between new and old data, and (ii) class imbalance within the same time point of data. Such imbalances often cause the dominant classes to take over the training process of continual semantic segmentation (CSS). To address this issue, we propose SurgCSS, a novel plug-and-play CSS framework for surgical instrument segmentation under data imbalance. Specifically, we generate realistic surgical backgrounds through inpainting and blend instrument foregrounds with the generated backgrounds in a class-aware manner to balance the data distribution in various scenarios. We further propose the Class Desensitization Loss by employing contrastive learning to correct edge biases caused by data imbalance. Moreover, we dynamically fuse the weight parameters of the old and new models to achieve a better trade-off between the biased and unbiased model weights. To investigate the data imbalance problem in surgical scenarios, we construct a new benchmark for surgical instrument CSS by integrating datasets. Extensive experiments demonstrate the effectiveness of the proposed framework, achieving significant performance improvement against existing baselines.

Shifang Zhao, Long Bai, Kun Yuan +8

Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong, Hong Kong · Chair for Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany · University of Strasbourg, CNRS, INSERM, ICube, Strasbourg, France +2

Semantic Segmentation Medical Image Segmentation Continual Learning Class-Imbalanced Learning

Ask PDF

Sep 15, 2026

TEDi: Temporal Memory-Enhanced and Denoising Transformer for Surgical Instrument Segmentation

Query-based segmentation methods have shown promising potential for surgical instrument segmentation and recognition, which is essential for scene understanding and downstream tasks in computer assisted surgery. However, most existing approaches predominantly rely on per-frame predictions and overlook cross-frame temporal priors as well as temporal-consistency constraints. This limitation often leads to unstable query representations and suboptimal category recognition. In this paper, we propose TEDi, a Temporal memory-Enhanced and Denoising transformer for surgical instrument segmentation that addresses these is sues through Memory Search Enhancement and Temporal Consistency Denoising. The former introduces a query-level memory bank and a memory search enhancement encoder to retrieve discriminative representations from historical frames, enriching current-frame features. The latter constructs a temporally consistent reference as a cross-frame semantic anchor to suppress temporally unstable predictions and promote semantic coherence across frames. Extensive experiments on two benchmark datasets, EndoVis 2017 and EndoVis 2018, demonstrate that TEDi consistently outperforms state-of-the-art methods, highlighting its potential to further advance computer-assisted surgery. Our code is available at github.com/argon-xixi/TEDi.

Jiahong Yuan, Weiming Mi, Tao Zhang +1

Department of Automation, Tsinghua University, Beijing, China · Surgical Planning Laboratory, Brigham and Women’s Hospital, Harvard Medical School, United States

Video Object Segmentation Medical Image Segmentation Semantic Segmentation

Ask PDF

Aug 25, 2026

Hierarchical Prototype-Memory Adaptation of SAM for Surgical Instrument Segmentation

Surgical instrument segmentation (SIS) is fundamental for computer-assisted surgery, where reliable instrument masks enable precise scene understanding and clinical assistance. Recently, adapting foundation models like the Segment Anything Model (SAM) to the surgical domain via prompt-learning has shown encouraging results. However, the performance of these adapted models under challenging surgical conditions is constrained by suboptimal adaptation mechanisms. Specifically, optimizing prompts or prototypes purely via downstream segmentation loss tends to cause them to degenerate into task-specific parameters rather than serving as persistent, stable category memory, thereby degrading their robustness against complex intraoperative variations. Moreover, routing multi-scale visual cues through a single prompt pathway creates a bottleneck that hinders effective scale-matched coupling. To address these limitations, we propose HPMA, a Hierarchical Prototype-Memory Adaptation framework for SAM. Specifically, HPMA constructs a frozen, multi-scale visual prototype memory bank from annotated surgical scenes and integrates it into SAM's feature space using lightweight adapters to preserve stable category evidence. To maximize the utility of multi-scale cues, we introduce a scale-matched coupling mechanism where global prototypes calibrate class-level prompt features, structural prototypes guide decoder object queries, and local prototypes align high-resolution feature maps through a local alignment objective. Extensive experiments on the public EndoVis2017 and EndoVis2018 datasets demonstrate that our approach achieves state-of-the-art performance, outperforming existing foundation model adaptation methods.

Xinning Yao, Jingjing Wang, Jinghua Yue +3

Image Processing Center, Beihang University, Beijing, 100191, China.

Vision Foundation Model Adaptation Medical Image Segmentation Prototype-Based Learning

Ask PDF

Abstract

Class-Incremental Learning

Explore similar work

Rethinking Data Imbalance in Class Incremental Surgical Instrument Segmentation

TEDi: Temporal Memory-Enhanced and Denoising Transformer for Surgical Instrument Segmentation

Hierarchical Prototype-Memory Adaptation of SAM for Surgical Instrument Segmentation