Rethinking Data Imbalance in Class Incremental Surgical Instrument Segmentation
Authors: Shifang Zhao, Long Bai, Kun Yuan, Feng Li, Jieming Yu, Wenzhen Dong, Guankun Wang, Mobarakol Islam, Nicolas Padoy, Nassir Navab, Hongliang Ren
Organizations: Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong, Hong Kong · Chair for Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany · University of Strasbourg, CNRS, INSERM, ICube, Strasbourg, France · IHU Strasbourg, Strasbourg, France · UCL Hawkes Institute, University College London, London, United Kingdom
In surgical instrument segmentation, the increasing variety of instruments over time poses a significant challenge for existing neural networks, as they are unable to effectively learn such incremental tasks and suffer from catastrophic forgetting. When learning new data, the model experiences a sharp performance drop on previously learned data. Although several continual learning methods have been proposed for incremental understanding tasks in surgical scenarios, the issue of data imbalance often leads to a strong bias in the segmentation head, resulting in poor performance. Data imbalance can occur in two forms: (i) class imbalance between new and old data, and (ii) class imbalance within the same time point of data. Such imbalances often cause the dominant classes to take over the training process of continual semantic segmentation (CSS). To address this issue, we propose SurgCSS, a novel plug-and-play CSS framework for surgical instrument segmentation under data imbalance. Specifically, we generate realistic surgical backgrounds through inpainting and blend instrument foregrounds with the generated backgrounds in a class-aware manner to balance the data distribution in various scenarios. We further propose the Class Desensitization Loss by employing contrastive learning to correct edge biases caused by data imbalance. Moreover, we dynamically fuse the weight parameters of the old and new models to achieve a better trade-off between the biased and unbiased model weights. To investigate the data imbalance problem in surgical scenarios, we construct a new benchmark for surgical instrument CSS by integrating datasets. Extensive experiments demonstrate the effectiveness of the proposed framework, achieving significant performance improvement against existing baselines.
Figures & tables
Figure 1: Experimental results on a surgical dataset confirm the presence of bias in the final layer, as observed through weight visualization of the segmentation head in task “13-1". Heads 1-13 correspond to the old class, while heads 14-18 represent incremental classes, one by one (Best viewed in color). We visualize the bias within each class-specific output channel of the final convolutional layer using two methods: a summation average of activations (left) and a t-SNE embedding of the feature space (right).
Figure 2: Framework of Our Two-Stage Class-Incremental Continual Segmentation Method. Following the incremental training stage, similar to previous approaches, we introduce (a) an additional rebalance stage using a synthetic balanced dataset. During this stage, (b) a Class Desensitization Loss is applied to reduce class bias introduced in the incremental stage. A Dynamic Weight Fusion, shown in the purple dotted box, is performed between the biased model and the unbiased model to integrate knowledge derived from more diverse real-world data.
Figure 3: Construction of a rebalance dataset through inpainting: During training, the model performs a reconstruction task on background-masked regions. At inference time, it is used to remove existing foreground instruments to isolate the background.
Figure 4: Illustration of the class distribution in the collected dataset, ordered from the most frequent to the least frequent class.
Figure 5: Demonstration of visually similar samples across different classes in our dataset. We present several surgical instruments with similar appearances, including Bipolar Forceps, Prograsp Forceps, Grasping Retractor, and Grasper.
Method
13-1 (5 steps)
13-5 (1 step)
9-3 (3 steps)
17-1 (1 step)
Base
New
All
Base
New
All
Base
New
All
Base
New
All
Old Model (t=0)
61.29
0.00
45.16
61.29
0.00
45.16
60.62
0.00
31.90
56.93
0.00
53.93
FT
0.00
17.67
8.53
0.00
32.39
13.71
0.00
10.99
9.52
0.00
5.49
8.77
EWC ( Kirkpatrick et al., 2017 )
0.00
10.98
5.30
0.00
52.60
22.25
0.00
7.65
8.75
0.00
5.47
8.73
LwF ( Li and Hoiem, 2017 )
1.48
21.82
11.30
14.04
14.23
20.11
3.56
27.42
18.35
23.14
4.64
23.31
LwF-MC ( Rebuffi et al., 2017 )
0.32
17.48
9.05
10.07
13.49
14.35
5.63
19.31
16.02
10.62
15.18
15.98
Table 1: Performance comparison across different continual learning tasks (13-1, 13-5, 9-3, and 17-1, “x-y" denotes that x is the number of old classes and y is the number of incremental classes introduced in each step). Results are reported for methods without continual learning (WO CL), including the old model at time point t=0 and naive fine-tuning (FT), and those with continual learning (W CL). The metrics presented are for base, new, and all classes, with the offline training serving as an upper bound.
Method
CholecSeg8k
EndoVis 2017
EndoVis 2018
SAR-RAPR50
Base
New
All
Base
New
All
Base
New
All
Base
New
All
Old Model (t=0)
62.53
0.00
44.66
83.82
0.00
55.88
73.33
0.00
61.10
61.69
0.00
52.97
FT
0.00
0.00
31.60
0.00
0.00
11.00
0.00
0.00
12.02
0.00
75.05
24.49
EWC ( Kirkpatrick et al., 2017 )
0.00
0.00
41.26
0.00
0.00
16.63
0.21
0.97
12.91
0.00
74.82
23.45
LwF ( Li and Hoiem, 2017 )
0.00
78.39
58.30
0.00
0.44
11.78
0.00
0.05
12.81
0.00
66.23
22.69
LwF-MC ( Rebuffi et al., 2017 )
0.00
18.40
37.86
0.00
0.00
11.03
0.00
0.00
12.14
0.00
19.56
16.55
Table 2: Performance comparison across different datasets for the 13-1 continual learning task. Results are reported for methods without continual learning (WO CL), including the old model at time point t=0 and naive fine-tuning (FT), and those with continual learning (W CL). The metrics include performance on base, new, and all classes, with offline training serving as the upper bound.
Figure 6: Visual comparison with different state-of-the-art methods (Task 13-1 on our surgical dataset).
Setting
LCD
WF
13-1
13-5
9-3
17-1
Incremental Stage
-
-
43.98
43.22
23.77
53.93
Rebalance Stage
✗
✗
45.88
45.29
41.38
50.09
✓
✗
49.83
43.06
45.75
54.64
✓
✓
50.13
45.62
46.68
55.33
Table 3: Ablation study on the Class Desensitization Loss and dynamic weight fusion in our Rebalance Stage.
Figure 7: Ablation study on the Class Desensitization Loss via the t-SNE visualization on the feature space.
LCD
DB Index ↓
CH Index ↑
WCSS ↓
✗
3.56
1.13
31445.56
✓
2.13
3.02
11366.84
Table 4: Ablation study on the Class Desensitization Loss using cluster metrics on the t-SNE embedding.
Method
Number
Base classes
New classes
All classes
Random
100
44.20
2.64
36.02
300
48.39
5.43
39.66
500
50.83
7.69
41.93
1000
54.08
7.33
44.05
Class-aware
100
46.65
7.02
38.78
300
49.38
4.50
40.09
Table 5: Comparisons of our ablation study on different blending methods and sample numbers.
Method
Base classes
New classes
All classes
Crop
56.10
6.47
45.21
Generate (DDPM, Ho et al., 2020 )
44.02
5.79
36.49
Inpaint (LDM, Rombach et al., 2022 )
62.37
8.70
50.13
Table 6: Comparisons of different background synthetic methods. Crop refers to directly cropping the background from samples.
Method
FID ↓
IS ↑
PSNR ↑
DDPM ( Ho et al., 2020 )
266.77
0.99
11.25
StyleGAN-XL ( Sauer et al., 2022 )
138.46
1.09
12.16
FLUX.1-Fill-dev ( Labs, 2024 )
158.60
1.86
11.06
LDM ( Ours , Rombach et al., 2022 )
48.26
1.30
12.40
Table 7: Synthesis quality comparison of different generation methods.
Figure 8: Visualization samples of our synthetic background inpainting.
Figure 9: Qualitative comparison of different generation methods.
α
β
Base classes
New classes
All classes
1.0
0.0
56.54
7.76
45.92
0.0
1.0
61.33
9.90
49.65
1.0
1.0
61.98
8.74
49.84
1.0
2.0
59.98
5.79
47.74
2.0
1.0
61.65
7.82
49.36
2.0
2.0
57.22
4.64
45.48
Table 8: Comparisons of our ablation study on the weight of the Class Desensitization Loss (Task 13-1 on our surgical dataset).
Method
top- k
λ
Base classes
New classes
All classes
None
-
-
62.02
8.49
49.83
Replace
1%
-
55.98
9.31
45.88
10%
-
56.38
5.87
45.25
30%
-
54.76
5.32
44.00
50%
-
54.69
5.05
43.80
Weighted
10%
0.1
62.41
8.13
49.97
Table 9: Comparisons of our ablation study on the fusion parameter selection top- k and merged weight λ (Task 13-1 on our surgical dataset).
Department of Automation, Tsinghua University, Beijing, China · Surgical Planning Laboratory, Brigham and Women’s Hospital, Harvard Medical School, United States