Class-incremental learning (CIL) for outdoor 3D object detection remains largely underexplored. Existing outdoor studies are limited to single-benchmark evaluations, with SDCoT++ considering KITTI and CL2OD considering nuScenes under different incremental settings. This makes it difficult to assess whether observed continual-learning behavior generalizes across autonomous-driving benchmarks. To the best of our knowledge, we provide the first unified cross-benchmark analysis of outdoor class-incremental 3D object detection across multiple autonomous-driving benchmarks, covering both two-task and multi-stage settings. We investigate: (1) how severely existing 3D detectors forget previously learned classes, (2) how incremental training exposes the stability-plasticity dilemma between knowledge retention and new-class acquisition, and (3) which forms of supervision are most effective for balancing these objectives. We introduce CL3DOD, a class-incremental 3D detection framework that restores missing supervision for previous classes using predictions from the previous-stage detector, preserves old-class responses through query-wise knowledge distillation, and employs EMA-based temporal consistency regularization. Experiments on KITTI and nuScenes reveal a consistent stability-plasticity dilemma across both benchmarks. CL3DOD generally achieves a more favorable stability-plasticity balance than controlled incremental baselines and improves upon published prior methods in the reported benchmark comparisons. This advantage persists across multiple incremental stages, demonstrating strong retention of previously learned knowledge while maintaining effective new-class acquisition. Our code, evaluation setup, and incremental task splits will be publicly released.
Figures & tables
Figure 1: Overview of CL3DOD at incremental task Tt . A frozen previous-stage teacher provides Old-Class Supervision and Query-Wise Knowledge Distillation through shared queries, while an EMA teacher provides Temporal Consistency Regularization.
Task
Method
mAPT1↑
mAPT2↑
mAPT3↑
mAPAll↑
NDS ↑
F↓
IM↑
T1
Task-1 Training
46.28
–
–
46.28
55.88
–
–
T2
Fine-tuning
8.18
58.48
–
33.33
45.42
38.10
-8.72
Freeze-and-Add
46.09
22.01
–
34.05
42.15
0.19
-26.95
CL3DOD
45.88
68.79
–
57.33
64.10
0.40
-3.56
Joint Training
50.07
69.81
–
59.94
66.47
–
–
T3
Fine-tuning
0.00
0.00
42.44
16.97
20.03
52.38
-13.11
Table 1: Quantitative results on the nuScenes val. set under the three-task class-incremental setting. Task-wise mAP is reported together with cumulative mAP and NDS. F measures Average Forgetting ( ↓ ), while IM measures learning plasticity ( ↑ ) following the RICO formulation and uses the final all-task Joint Training model as reference. CL3DOD rows are highlighted for emphasis, while Joint Training is reported separately as the full-supervision upper-bound reference. “–” denotes not applicable.
Task
Method
mAPT1↑
mAPT2↑
mAPAll↑
F↓
IM↑
Easy
Mod.
Hard
Easy
Mod.
Hard
Easy
Mod.
Hard
T1
Task-1 Training
79.04
68.03
63.13
–
–
–
79.04
68.03
63.13
–
–
Fine-tuning
0.06
0.08
0.10
26.68
19.22
16.69
10.70
7.73
6.74
78.98
1.04
T2
Freeze-and-Add
80.32
68.28
63.60
11.99
5.45
4.19
52.99
43.15
39.84
-1.28
-6.31
CL3DOD
66.21
56.81
51.62
28.11
18.43
15.47
50.97
41.46
37.16
12.83
1.75
Joint Training
76.52
67.55
63.20
27.13
19.46
15.90
56.76
48.31
44.28
–
–
Table 2: Quantitative results on the KITTI val. set under the two-task class-incremental setting using AP 40 . F and IM are computed using Easy AP 40 .
Task
Method
T1
T2
T3
Car
Trailer
Constr.
Truck
Bus
Ped.
Motor.
Bicycle
Cone
Barrier
T1
Task-1 Training
81.6
35.2
22.0
–
–
–
–
–
–
–
T2
Fine-tuning
24.5
0.0
0.0
49.9
64.0
61.6
–
–
–
–
Freeze-and-Add
81.5
34.7
22.1
38.3
26.9
0.8
–
–
–
–
CL3DOD
80.7
34.7
22.5
56.0
68.2
82.2
–
–
–
–
Joint Training
85.2
41.8
23.2
57.9
69.2
82.3
–
–
–
–
Table 3: Class-wise AP (%) on the nuScenes val. set across the three-task incremental sequence. CL3DOD rows are highlighted for emphasis, while Joint Training is shown separately as the upper-bound reference. “–” denotes not applicable.
Task
Method
Car
Ped.
Cyc.
Van
Truck
T1
Task-1 Training
82.62
66.48
88.03
–
–
T2
Fine-tuning
0.17
0.00
0.00
50.37
2.98
Freeze-and-Add
87.06
66.12
87.77
23.98
0.00
CL3DOD
68.45
53.57
76.61
53.69
2.54
Joint Training
80.96
63.58
85.03
54.00
0.26
Table 4: Class-wise Easy AP 40 (%) on the KITTI val. set after incremental learning. The CL3DOD row is highlighted for emphasis, while Joint Training is shown separately as the upper-bound reference. “–” denotes not applicable.
Method
After T1
After T2
T1
T1
T2
All
F↓
IM↑
SDCoT [ 25 ]
81.49
61.27
22.50
45.77
20.22
-4.65
SDCoT++ [ 26 ]
81.49
62.56
22.52
46.55
18.93
-4.64
CL3DOD
79.04
66.21
28.11
50.97
12.83
1.75
Table 5: Comparison with prior class-incremental 3D detection methods on the KITTI val split using the same two-task class partition. Results are reported using Easy mAP. SDCoT and SDCoT++ results are taken from SDCoT++ [ 26 ] .
Task
Method
mAPCur.
mAPAll
NDS
F↓
T1
CL2OD
40.90
40.90
52.10
–
CL3DOD
46.28
46.28
55.88
–
T2
CL2OD
53.00
41.80
53.60
10.20
CL3DOD
68.79
57.33
64.10
0.40
T3
CL2OD
30.00
34.30
49.80
9.70 †
CL3DOD
55.71
55.45
63.99
2.27
Table 6: Contextual comparison with published CL2OD [ 22 ] on the nuScenes val set. mAPCur. and mAPAll denote current-task and cumulative performance, respectively. † Computed from published class-wise AP values.
Dataset
Sup.
KD
Cons.
mAPT1
mAPT2
mAPAll
nuScenes
✓
46.24
68.96
57.60
✓
✓
46.12
68.57
57.35
✓
✓
45.33
68.93
57.13
✓
✓
✓
45.88
68.79
57.33
KITTI
✓
62.94
24.58
47.60
✓
✓
61.40
30.47
49.03
Table 7: Component ablation of CL3DOD after task T2 . KITTI results use Easy AP 40 . Sup. , KD , and Cons. denote Old-Class Supervision, Query-Wise Knowledge Distillation, and Temporal Consistency Regularization, respectively. Best results within each dataset are shown in bold.
Figure 2: Task-level annotation distribution of the KITTI and nuScenes incremental splits for training and validation. KITTI is strongly dominated by T1 , whereas the nuScenes tasks exhibit a substantially more balanced distribution.
Dataset
Task
Train %
Val. %
KITTI
T1
90.65
88.76
T2
9.35
11.24
nuScenes
T1
47.20
46.31
T2
28.77
28.37
T3
24.03
25.32
Table 8: Task-level annotation distribution of the KITTI and nuScenes incremental benchmark partitions.
Task
Class
Train
Validation
Count
%
Count
%
T1
Car
14,357
75.23
14,385
72.72
Pedestrian
2,207
11.57
2,280
11.53
Cyclist
734
3.85
893
4.51
Task total
17,298
90.65
17,558
88.76
T2
Van
1,297
6.80
1,617
8.17
Table 9: Class-wise annotation statistics for the incremental splits used in our experiments: (a) KITTI and (b) nuScenes . Percentages denote the fraction of all annotated instances within the corresponding train or validation split.
Task
Method
Car
Pedestrian
Cyclist
Van
Truck
Easy
Moderate
Hard
Easy
Moderate
Hard
Easy
Moderate
Hard
Easy
Moderate
Hard
Easy
Moderate
Hard
T1
Task-1 Training
82.62
74.95
70.83
66.48
58.95
52.64
88.03
70.19
65.90
–
–
–
–
–
–
T2
Fine-tuning
0.17
0.23
0.31
0.00
0.00
0.00
0.00
0.00
0.00
50.37
37.76
32.57
2.98
0.68
0.82
Freeze-and-Add
87.06
76.76
72.28
66.12
58.75
52.53
87.77
69.33
65.99
23.98
10.90
8.39
0.00
0.00
0.00
CL3DOD
68.45
60.87
55.19
53.57
47.00
41.39
76.61
62.56
58.28
53.69
36.62
30.72
2.54
0.25
0.22
Joint Training
80.96
73.97
70.04
63.58
57.29
52.02
85.03
71.38
67.53
54.00
38.49
31.38
0.26
0.43
0.42
Table 10: Class-wise 3D AP 40 (%) on the KITTI validation set at the Easy, Moderate, and Hard difficulty levels. CL3DOD is highlighted for emphasis, while Joint Training is shown separately as the full-supervision upper-bound reference. “–” denotes not applicable.
γconf
mAPT1↑
mAPT2↑
mAPAll↑
0.45
65.57
26.36
49.89
0.55
66.21
28.11
50.97
0.65
70.53
19.84
50.25
Table 11: Ablation of the pseudo-label confidence threshold γconf on the KITTI validation set. Results are reported using AP40 at the Easy difficulty level.
Figure 3: Qualitative comparison on representative KITTI validation scenes after incremental learning on T2 . From left to right: Ground truth, Fine-tuning, Freeze-and-Add, and CL3DOD. Ground-truth boxes are color-coded by class: Car (blue), Pedestrian (yellow), Cyclist (magenta), Van (cyan), and Truck (green), while predicted boxes are shown in red. Fine-tuning exhibits severe forgetting of previously learned classes, whereas Freeze-and-Add preserves previous-task detections but shows limited acquisition of the newly introduced classes. CL3DOD achieves a more balanced retention and acquisition behavior.
Figure 4: Qualitative comparison on representative nuScenes validation scenes after incremental learning on T3 . From left to right: ground truth, Fine-tuning, Freeze-and-Add, and CL3DOD. Ground-truth boxes are color-coded by class: Car (blue), Truck (pink), Bus (orange), Trailer (magenta), Construction Vehicle (cyan), Pedestrian (yellow), Motorcycle (red), Bicycle (purple), Traffic Cone (light blue), and Barrier (gray), while predicted boxes are shown in red. Fine-tuning exhibits severe forgetting of previously learned classes, whereas Freeze-and-Add retains some previous-task detections but shows limited adaptation across the incremental sequence. CL3DOD produces a more complete set of detections, illustrating a more favorable balance between retention and acquisition after the three-task sequence.
Figure 5: Representative failure cases of CL3DOD on the KITTI validation set after incremental learning on Task 2. Ground-truth boxes (left) are color-coded by class: Car (blue), Pedestrian (yellow), Cyclist (magenta), Van (cyan), and Truck (green), while CL3DOD predictions (right) are shown in red. Although several objects are correctly detected, some annotated instances remain missed, illustrating remaining limitations under challenging scene conditions.
Incremental 3D object detection requires a detector to learn novel object classes while remembering previously learned ones over sequentially arriving data. Previous methods, primarily based on pseudo-labeling, perform reasonably in short-incremental stages but still suffer from severe model forgetting when dealing with long-incremental sequences. We investigate this failure and reveal a detrimental self-reinforcing cycle: data distribution shift of novel classes causes model forgetting on old classes, which further produces accumulated error in pseudo-labeling that exacerbates model degradation. To address this issue, we draw inspiration from the human learning process and propose the Learning-Dynamics-driven Memory and Review (LDMR) framework. LDMR monitors per-class detection quality at periodic training checkpoints and uses these learning-dynamics signals to drive two innovative mechanisms, namely (i) human-like intra-stage review that divides each incremental stage into multiple sub-stages' training and concentrates on remembering the most-forgotten objects, and (ii) scene-aware cross-stage memory evolution that evolves a memory bank to transfer knowledge between two consecutive stages by jointly considering scene learnability and diversity. Extensive experiments across multiple long-incremental protocols on indoor benchmarks SUN RGB-D and ScanNetV2 show that LDMR substantially mitigates the model forgetting and outperforms all baselines by a clear margin. Code is available at https://github.com/qianpeisheng/LDMR.
Peisheng Qian, Jie Xu, Xulei Yang +1
Singapore University of Technology and Design · Institute for Infocomm Research, A*STAR, Singapore
3D perception plays a crucial role in real-world applications such as autonomous driving, robotics, and AR/VR. In practical scenarios, 3D perception models need to continually adapt to newly emerging 3D object categories, making class-incremental learning (CIL) particularly important. However, unlike 2D images, 3D point clouds are inherently heterogeneous: objects from the same class may not only come from the clean CAD domain, but also from RGB-D camera scans of varying quality, video reconstructions, or even corrupted observations. We discover that such heterogeneity introduces a new challenge beyond catastrophic forgetting: the degree of performance degradation can vary substantially across domains, a phenomenon we term performance discrepancy. To investigate this problem, we establish the Domain3D-CIL training and evaluation protocol, which contains point cloud categories from heterogeneous domains. We further adapt a wide range of mainstream CIL methods to the 3D modality. The results demonstrate that this performance discrepancy consistently appears across these baselines. To mitigate this issue, we introduce PolyMem, an exemplar-free approach that implicitly models rich high-order statistics of the feature distribution to enhance cross-domain robustness. Experiments demonstrate that our method effectively alleviates the performance discrepancy while improving the model's performance across domains. Code will be made publicly available upon acceptance.
Incremental 3D object perception is a critical step toward embodied intelligence in dynamic indoor environments. However, existing incremental 3D detection methods rely on extensive annotations of novel classes for satisfactory performance. To address this limitation, we propose FI3Det, a Few-shot Incremental 3D Detection framework that enables efficient 3D perception with only a few novel samples by leveraging vision-language models (VLMs) to learn knowledge of unseen categories. FI3Det introduces a VLM-guided unknown object learning module in the base stage to enhance perception of unseen categories. Specifically, it employs VLMs to mine unknown objects and extract comprehensive representations, including 2D semantic features and class-agnostic 3D bounding boxes. To mitigate noise in these representations, a weighting mechanism is further designed to re-weight the contributions of point- and box-level features based on their spatial locations and feature consistency within each box. Moreover, FI3Det proposes a gated multimodal prototype imprinting module, where category prototypes are constructed from aligned 2D semantic and 3D geometric features to compute classification scores, which are then fused via a multimodal gating mechanism for novel object detection. As the first framework for few-shot incremental 3D object detection, we establish both batch and sequential evaluation settings on two datasets, ScanNet V2 and SUN RGB-D, where FI3Det achieves strong and consistent improvements over baseline methods. Code is available at https://github.com/zyrant/FI3Det.
Yun Zhu, Jianjun Qian, Jian Yang +2
Nanjing University of Science and Technology · Nanjing University · Singapore University of Technology and Design