Leveraging 2D Foundation Priors for Generalized Category Discovery under Test-Time Adaptation in 3D LiDAR Segmentation
Authors: Hanqi Chen, Chaoqi Chen, Tian Ji, Hui Huang
Organizations: CSSE, Shenzhen University Shenzhen, Guangdong, China · The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, Guangdong, China
In autonomous driving, 3D LiDAR semantic segmentation is expected to adapt to evolving environments and recognize previously unseen categories, making generalized category discovery under test-time adaptation (GCD-TTA) an important problem. However, existing works treat GCD and TTA separately, while the limited semantic information in 3D point clouds makes GCD-TTA under domain shift more prone to feature collapse when combining GCD with TTA. To address this issue, we propose a two-stage framework for GCD-TTA in 3D LiDAR semantic segmentation. Specifically, in the first stage, we adaptively aggregate local 3D features according to geometric density variations and obtain reliable pseudo labels under a global class-wise prototype consistency constraint. In the second stage, we cast category discovery as an optimal transport assignment process for globally consistent grouping of novel categories. Meanwhile, semantic features from 2D vision foundation models are fused with 3D point cloud features to enrich 3D semantic information and promote a more separable feature space. Extensive experiments on Synth4D to SemanticKITTI, SynLiDAR to SemanticKITTI, and Synth4D to nuScenes demonstrate the effectiveness of our framework. Under a unified evaluation protocol, our method achieves the best overall mIoU on 9 out of 10 benchmark splits and improves the overall mIoU by up to 12.21 points over the strongest re-implemented baseline.
Figures & tables
Figure 1. Illustration of the TTA and GCD pipelines.
Figure 2. Overview of the source pretraining, test-time adaptation, and generalized category discovery pipeline.
mIoU
Split
Method
vehicle
person
road
sidewalk
terrain
manmade
vegetation
Novel
Known
All
Full
77.03
0.00
20.29
19.35
21.07
77.61
81.28
-
-
42.37
Source
75.50
14.35
-
-
-
39.91
71.42
-
50.29
-
NOPS ∗
45.58
7.68
44.70
10.76
14.75
38.84
64.92
24.40
39.26
32.46
SNOPS ∗
68.50
14.94
58.85
30.12
34.36
52.16
67.53
41.11
50.78
46.64
Synth4d 0
Ours
69.54
13.14
21.25
18.15
21.76
38.53
66.96
20.40
47.10
35.65
Table 2. The novel class discovery results on the Synth4D → SemanticKITTI setting with 7 classes.
mIoU
Split
Method
vehicle
person
road
sidewalk
terrain
manmade
vegetation
Novel
Known
All
Full
63.59
24.78
40.37
12.16
5.09
74.83
61.24
-
-
40.29
Source
66.80
24.21
-
-
-
76.91
60.89
-
57.20
-
NOPS ∗
58.37
18.15
43.35
6.72
8.41
68.29
49.87
19.49
48.67
36.17
SNOPS ∗
51.11
2.15
45.65
12.75
3.12
58.48
56.07
20.51
41.95
32.76
Synth4d 0
Ours
64.72
20.48
37.15
8.77
3.90
72.94
62.91
16.61
55.26
38.70
Table 3. The novel class discovery results on the Synth4D → nuScenes setting with 7 classes.
Method
SemanticKITTI → nuScenes
nuScenes → SemanticKITTI
Novel
Known
All
Novel
Known
All
NOPS
30.15
28.47
29.19
26.67
29.08
28.04
SNOPS
24.04
19.79
21.61
39.21
41.42
40.47
Ours
46.72
38.41
41.97
42.07
42.38
42.25
Table 4. Real-to-real setting (Split 0).
Figure 3. Comparison between H^(A) and effective rank eH^(A) . The count of the largest eigenvalues necessary to account for 99% of the total eigenvalue energy serves as a surrogate for the rank.
Figure 4. Comparison of UMAP visualizations between SNOPS and our method on the nuScenes dataset (Split 1).
Figure 5. Pseudo label visualization (Split 0).
F
FL FR
B
BL BR
mIoU all
Split 0
Split 1
Split 2
34.78
38.83
41.22
✓
32.03 ( -2.75 )
38.01 ( -0.82 )
42.99 ( +1.77 )
✓
✓
33.75 ( -1.03 )
39.88 ( +1.05 )
46.71 ( +5.49 )
✓
✓
✓
✓
38.70 ( +3.92 )
44.27 ( +5.44 )
48.90 ( +7.68 )
F
FL FR
B
BL BR
mIoU novel
Table 5. Ablation on different numbers of 2D image views on Synth4D → nuScenes setting. F, FL, FR, B, BL, and BR denote FRONT, FRONT_LEFT, FRONT_RIGHT, BACK, BACK_LEFT, and BACK_RIGHT, respectively.
mIoU
Split 0
Split 1
Split 2
Split 3
w/o adapt kNN
21.8
24.1
24.8
23.6
w/ adapt kNN
27.4 ( +5.6 )
23.8 ( -0.3 )
25.1 ( +0.3 )
24.8 ( +1.2 )
Table 6. Ablation study on whether to adopt adaptive kNN in the SynLiDAR → SemanticKITTI setting.
Figure 6. Impact of different hyper-parameters on the Synth4D → nuScenes setting (Split 0).