Organizations: Sun Yat-sen University · Shanghai Jiao Tong University · Southwest University · The Chinese University of Hong Kong · South China University of Technology · South China Normal University
Continual semantic segmentation (CSS) is a cornerstone task in computer vision that enables a large number of downstream applications, but faces the catastrophic forgetting challenge. In conventional class-incremental semantic segmentation (CISS) frameworks using Softmax-based classification heads, catastrophic forgetting originates from Catastrophic forgetting and task affiliation probability. We formulate these problems and provide a theoretical analysis to more deeply understand the limitations in existing CISS methods, particularly Strict Parameter Isolation (SPI). To address these challenges, we follow a dual-phase intuition from human annotators, and introduce Cognitive Cascade Segmentation (CogCaS), a novel dual-phase cascade formulation for CSS tasks in the CISS setting. By decoupling the task into class-existence detection and class-specific segmentation, CogCaS enables more effective continual learning, preserving previously learned knowledge while incorporating new classes. Using two benchmark datasets PASCAL VOC 2012 and ADE20K, we have shown significant improvements in a variety of challenging scenarios, particularly those with long sequence of incremental tasks, when compared to exsiting state-of-the-art methods. Our code will be made publicly available upon paper acceptance.
Figures & tables
Figure 1: mIoU of classes learned at each task in PASCAL VOC continual semantic segmentation under 2-2 (left) and 4-2 (right) settings. While existing methods exhibit declining learning capability as tasks progress, our CogCaS consistently achieves strong performance around 70% mloU,demonstrating stable and robust learning ability across all incremental stages.
Figure 2: Demonstration of existing typical CISS framework and the proposed CogCaS framework. (A) Traditional CISS framework consists of a set of segmentation heads which are trained sequentially and activated for inference simultaneously. (B) our proposed CogCaS restructures the traditional CISS formulation into a dual-phase cascade using multi-label classifier and class-specific segmentation head. The multi-label classifier determine whether each learned class exists in the image, and only segmentation heads corresponding to existing classes are activated to produce a foreground-background mask. These masks are then fused to obtain the final segmentation mask using a mask fusion strategy.
Figure 3: Representative segmentation results from different methods after the model learns all tasks in the Pascal VOC 2012 15-1 setting.
Method
10-1 (11 tasks)
15-5 (2 tasks)
15-1 (6 tasks)
0-10
11-20
all
0-15
16-20
all
0-15
16-20
all
Ours ∘
73.9
70.2
72.1
75.5
70.3
74.1
75.5
71.4
74.4
Joint Deeplab-v3 ∘ ( Chen et al., 2017 )
78.4
76.4
77.4
79.8
72.4
77.4
79.8
72.4
77.4
Joint Ours ∘
79.3
77.6
78.4
79.2
76.2
78.4
79.2
76.2
78.4
LwF-MC ∘ ( Rebuffi et al., 2016 )
4.7
5.9
5.0
58.1
35.0
52.3
64.0
8.4
6.9
ILT ∘ ( Michieli and Zanuttigh, 2021 )
7.2
3.7
5.5
67.1
39.2
60.5
8.8
8.0
8.6
Table 1: Comparison with exsiting CISS methods on PASCAL VOC 2012 in mIoU (%). The best results are marked in bold . ∘ : ResNet101 backbone. ⋄ : Swin-L backbone. † : unlike other methods, this one is based on Mask2Former ( Cheng et al., 2021 ) .
Method
100-50 (2 tasks)
100-10 (6 tasks)
100-5 (11 tasks)
0-100
101-150
all
0-100
101-150
all
0-100
101-150
all
Ours
41.2
29.4
37.3
42.3
25.6
36.8
40.1
24.7
35.0
Joint Deeplab-v3 ( Chen et al., 2017 )
47.2
31.8
42.1
47.2
21.8
42.1
47.2
21.8
42.1
Joint Ours
47.8
38.7
44.7
47.8
38.7
44.7
47.8
38.7
44.7
MiB ( Cermelli et al., 2020 )
39.0
16.7
31.2
36.6
9.8
27.7
34.7
4.8
24.7
MiB+NeST ( Xie et al., 2024 )
38.8
23.1
33.5
38.8
19.1
32.2
35.2
13.6
28.1
Table 2: Comparison with existing CISS methods on ADE20K using Swin-L backbone. † : unlike other methods, this one is based on Mask2Former ( Cheng et al., 2021 ) .
Method
VOC 1-1 (20 tasks)
VOC 2-1 (19 tasks)
VOC 2-2 (10 tasks)
0-1
2-20
all
0-2
3-20
all
0-2
3-20
all
Ours
79.4
70.1
70.9
76.3
71.2
71.9
75.9
70.8
71.5
MiB ( Cermelli et al., 2020 )
27.3
6.4
8.3
23.6
7.9
10.14
41.1
23.4
25.9
PLOP ( Douillard et al., 2020 )
25.4
4.2
6.2
19.4
6.2
8.1
39.7
22.8
25.2
MiB+NeST ( Xie et al., 2024 )
28.1
6.8
8.7
24.5
8.1
10.4
40.4
25.8
27.8
PLOP+NeST ( Xie et al., 2024 )
32.5
4.6
7.3
20.1
7.9
10.5
38.1
23.5
25.5
Table 3: Comparison with existing CISS methods under more challenging continual learning settings.
Dataset
mAP ↑
Prec. ↑
Rec. ↑
PASCAL VOC 2012
85.12
93.47
90.25
ADE20K
79.84
73.86
78.03
Table 4: Phase I class-existence detection on the evaluation split (%). Results are averaged across all settings in the datasets.
Method
VOC 1-1 (20 tasks)
VOC 2-1 (19 tasks)
VOC 2-2 (10 tasks)
ADE 100-5 (11 tasks)
0-1
2-20
all
0-2
3-20
all
0-2
3-20
all
0-100
101-150
all
Segmentation Only
18.2
13.1
13.5
14.3
17.2
16.7
13.8
16.5
16.1
6.7
9.5
7.6
Full Model
79.4
70.1
70.9
76.3
71.2
71.9
75.9
70.8
71.5
40.1
24.7
35.0
Oracle
80.2
70.3
71.2
77.4
71.5
72.3
77.4
71.8
72.4
48.6
31.8
43.0
Table 5: Ablation experiments with respect to the classification head using parameters trained under different task settings
Fusion Strategy
VOC 10-1
VOC 15-5
VOC 15-1
Logits-based
71.4
73.3
73.4
Random
71.9
73.2
73.9
Strict
70.9
72.8
73.0
Distributed (Ours)
72.1
74.1
74.4
Loose
72.8
74.9
75.2
Table 6: Sensitivity analysis of mask fusion strategies on VOC across three incremental settings. Results are reported in mIoU (%).
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: The figure illustrate near-ood samples and process.