Unlocking Positive Transfer in Incrementally Learning Surgical Instruments: A Self-reflection Hierarchical Prompt Framework

Yu Zhu; Kang Li; Zheng Li; Pheng-Ann Heng

cs.CVApr 3, 2026

Unlocking Positive Transfer in Incrementally Learning Surgical Instruments: A Self-reflection Hierarchical Prompt Framework

Authors: Yu Zhu, Kang Li, Zheng Li, Pheng-Ann Heng

Organizations: The Chinese University of Hong Kong · University of Electronic Science and Technology of China

Abstract

To continuously enhance model adaptability in surgical video scene parsing, recent studies incrementally update it to progressively learn to segment an increasing number of surgical instruments over time. However, prior works constantly overlooked the potential of positive forward knowledge transfer, i.e., how past knowledge could help learn new classes, and positive backward knowledge transfer, i.e., how learning new classes could help refine past knowledge. In this paper, we propose a self-reflection hierarchical prompt framework that unlocks the power of positive forward and backward knowledge transfer in class incremental segmentation, aiming to proficiently learn new instruments, improve existing skills of regular instruments, and avoid catastrophic forgetting of old instruments. Our framework is built on a frozen, pre-trained model that adaptively appends instrument-aware prompts for new classes throughout training episodes. To enable positive forward knowledge transfer, we organize instrument prompts into a hierarchical prompt parsing tree with the instrument-shared prompt partition as the root node, n-part-shared prompt partitions as intermediate nodes and instrument-distinct prompt partitions as leaf nodes, to expose the reusable historical knowledge for new classes to simplify their learning. Conversely, to encourage positive backward knowledge transfer, we conduct self-reflection refining on existing knowledge by directed-weighted graph propagation, examining the knowledge associations recorded in the tree to improve its representativeness without causing catastrophic forgetting. Our framework is applicable to both CNN-based models and advanced transformer-based foundation models, yielding more than 5% and 11% improvements over the competing methods on two public benchmarks respectively.

Figures & tables

Figure 1: (a) Prior works learn each instrument prompt independently. (b,c) They overlook the power of positive forward and backward transfer between early-acquired and newly-acquired knowledge. (d) In contrast, our work organizes instrument prompts from general to specific as a hierarchical prompt parsing tree. (e) It supports positive forward transfer by helping new instrument prompts inherit profitable knowledge from the past to assist new-class learning. (f) It further enables positive backward transfer by letting the newly-acquired knowledge wisely refine early-acquired knowledge to improve old-class segmentation.

Figure 2: Overview of our self-reflection hierarchical prompt framework. We progressively append instrument-aware prompts into a pre-trained model during the training lifespan. For each new class, before diving into learning its specific prompts, we first build a hierarchical prompt parsing tree to discover the profitable knowledge it could inherit from the past, i.e. , the instrument-shared (IS) and $n$ -part-shared knowledge, promoting new-class learning by positive forward knowledge transfer. Simply learning one instrument-distinct (ID) prompt partition would be sufficient to delineate its contour. Based on the newly acquired ID partition, we trigger self-reflection on existing knowledge to support positive backward knowledge transfer, wisely refining their representativeness within the context of all seen classes without causing catastrophic forgetting.

Figure 3: Visual comparisons of our approaches and highly competitive approaches. More comparison results are in the supplementary.

Method	Old Class		Regular Class					New Class		Old Class	Regular Class	New Class	All	BWT	FWT
Method	VS	GR	PF	BF	UP	LND	MCS	SI	CA	Old Class	Regular Class	New Class	All	BWT	FWT
Ind-T ( $t=1$ )	75.12	62.50	68.27	66.22	64.65	71.89	70.91	-	-	68.81	68.39	-	68.51	-	-
Ind-T ( $t=2$ )	-	-	69.67	67.67	66.44	70.63	70.80	32.98	9.18	-	69.04	21.08	55.34	-	-
Joint-T	83.46	70.91	79.79	71.45	71.88	71.08	75.04	35.94	8.90	77.18	73.85	22.42	63.16	-	-
Seq-F	4.90	2.97	48.73	52.23	44.81	46.73	37.47	9.45	1.29	3.93	45.99	5.37	27.62	-8.92	0.14
SI	6.51	2.65	50.26	58.91	29.58	51.29	42.02	11.48	1.82	4.58	46.41	6.65	28.28	-8.44	1.42

Table 1: Porcine robot-assisted nephrectomy instrument segmentation results for the last episode evaluated by IoU and all episodes assessed by BWT and FWT. For all metrics, the higher the better. We highlight the best and second-best results in bold and underline , respectively.

Method	Old Class	Regular Class	New Class					New Class	All	BWT	FWT
Method	LH	Gr	Cl	Sc	Ir	SB	Bi	New Class	All	BWT	FWT
Ind-T ( $t=1$ )	76.04	74.11	-	-	-	-	-	-	75.08	-	-
Ind-T ( $t=2$ )	-	79.40	69.34	47.02	39.05	63.91	59.56	55.78	59.71	-	-
Joint-T	79.54	80.88	71.07	49.03	39.64	67.65	57.87	57.05	63.67	-	-
Seq-F	39.07	53.67	62.59	49.61	32.18	47.14	43.50	47.01	46.82	-12.52	-1.00
SI	41.65	54.30	63.15	49.70	31.86	50.75	47.60	48.61	48.43	-10.92	0.60

Table 2: Laparoscopic cholecystectomy instrument segmentation results for the last episode evaluated by IoU and all episodes assessed by BWT and FWT. For all metrics, the higher the better. We highlight the best and second-best results in bold and underline , respectively.

DLv3+	SAM	HPPT	SRS	Old Classes	Regular Classes	New Classes	All
	✓			21.42	69.11	22.24	48.15
	✓	✓		65.29	69.23	22.16	57.89
	✓	✓	✓	68.52	70.67	22.30	59.44
✓				3.93	45.99	5.37	27.62
✓		✓	✓	22.46	59.79	12.19	40.92

Table 3: Ablation study of our key components.

Figure 4: Effects of the decay factor $\gamma$ and the teleport probability $\alpha$ on model performance.

Metric	SAM	Ours	$\Delta$ vs. SAM
Parameters (M)	636	639	+0.47%
FLOPs (G)	2733	2751	+0.65%
Train. time (s per iteration)	0.901	0.915	+1.55%
Infer. time (s per slice)	0.464	0.466	+0.43%

Table 4: Computational cost analysis.

Figure 1: Illustration of HPPT construction.

Figure 2: Visualization comparison of segmentation results across all methods. Rows 1–2, 4-5 : previous datasets. Row 3, 6 : current dataset.

Figure 3: Automatic estimation of instrument part count.

Instrument	Total Cutout	Part Count Inferred by MLLM				GT
		1	2	3	Result
Large Needle Driver	1197	156	213	828	3	3
Prograsp Forceps	1051	55	232	764	3	3
Bipolar Forceps	657	189	132	336	3	3
Ultrasound Probe	449	294	103	52	1	1
Curved Scissors	351	54	205	92	2	2

Table 1: Part Count Inference Results on EndoVis2017 Dataset

Instrument	Total Cutout	Part Count Inferred by MLLM				GT
		1	2	3	Result
Large Needle Driver	278	58	96	124	3	3
Prograsp Forceps	972	167	232	573	3	3
Bipolar Forceps	1757	395	132	1230	3	3
Ultrasound Probe	156	103	34	52	1	1
Curved Scissors	1554	205	1257	92	2	2

Table 2: Part Count Inference Results on EndoVis2018 Dataset

Class-Incremental Learning

Class-incremental learning studies how models can add new categories over time while retaining the ability to recognize previously learned classes.

See all 160 papers

Rethinking Data Imbalance in Class Incremental Surgical Instrument Segmentation
Oct 26, 2026Shifang Zhao, Long Bai, Kun Yuan +8Semantic Segmentation Medical Image Segmentation
Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong, Hong Kong · Chair for Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany · University of Strasbourg, CNRS, INSERM, ICube, Strasbourg, France +2
TEDi: Temporal Memory-Enhanced and Denoising Transformer for Surgical Instrument Segmentation
Sep 15, 2026Jiahong Yuan, Weiming Mi, Tao Zhang +1Video Object Segmentation Medical Image Segmentation
Department of Automation, Tsinghua University, Beijing, China · Surgical Planning Laboratory, Brigham and Women’s Hospital, Harvard Medical School, United States
Hierarchical Prototype-Memory Adaptation of SAM for Surgical Instrument Segmentation
Aug 25, 2026Xinning Yao, Jingjing Wang, Jinghua Yue +3Vision Foundation Model Adaptation Medical Image Segmentation
Image Processing Center, Beihang University, Beijing, 100191, China.

Oct 26, 2026J/K move · Enter open · S save

Rethinking Data Imbalance in Class Incremental Surgical Instrument Segmentation

Shifang Zhao, Long Bai, Kun Yuan, Feng Li+7

Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong, Hong Kong · Chair for Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany · University of Strasbourg, CNRS, INSERM, ICube, Strasbourg, France+2

Semantic Segmentation Medical Image Segmentation Continual Learning Class-Imbalanced Learning

Open paper AskPDF arXiv

In surgical instrument segmentation, the increasing variety of instruments over time poses a significant challenge for existing neural networks, as they are unable to effectively learn such incremental tasks and suffer from catastrophic forgetting. When learning new data, the model experiences a sharp performance drop on previously learned data. Although several continual learning methods have been proposed for incremental understanding tasks in surgical scenarios, the issue of data imbalance often leads to a strong bias in the segmentation head, resulting in poor performance. Data imbalance can occur in two forms: (i) class imbalance between new and old data, and (ii) class imbalance within the same time point of data. Such imbalances often cause the dominant classes to take over the training process of continual semantic segmentation (CSS). To address this issue, we propose SurgCSS, a novel plug-and-play CSS framework for surgical instrument segmentation under data imbalance. Specifically, we generate realistic surgical backgrounds through inpainting and blend instrument foregrounds with the generated backgrounds in a class-aware manner to balance the data distribution in various scenarios. We further propose the Class Desensitization Loss by employing contrastive learning to correct edge biases caused by data imbalance. Moreover, we dynamically fuse the weight parameters of the old and new models to achieve a better trade-off between the biased and unbiased model weights. To investigate the data imbalance problem in surgical scenarios, we construct a new benchmark for surgical instrument CSS by integrating datasets. Extensive experiments demonstrate the effectiveness of the proposed framework, achieving significant performance improvement against existing baselines.

Abstract

Class-Incremental Learning

Explore similar work

Rethinking Data Imbalance in Class Incremental Surgical Instrument Segmentation

TEDi: Temporal Memory-Enhanced and Denoising Transformer for Surgical Instrument Segmentation

Hierarchical Prototype-Memory Adaptation of SAM for Surgical Instrument Segmentation