Leveraging 2D Foundation Priors for Generalized Category Discovery under Test-Time Adaptation in 3D LiDAR Segmentation
Authors: Hanqi Chen, Chaoqi Chen, Tian Ji, Hui Huang
Organizations: CSSE, Shenzhen University Shenzhen, Guangdong, China · The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, Guangdong, China
In autonomous driving, 3D LiDAR semantic segmentation is expected to adapt to evolving environments and recognize previously unseen categories, making generalized category discovery under test-time adaptation (GCD-TTA) an important problem. However, existing works treat GCD and TTA separately, while the limited semantic information in 3D point clouds makes GCD-TTA under domain shift more prone to feature collapse when combining GCD with TTA. To address this issue, we propose a two-stage framework for GCD-TTA in 3D LiDAR semantic segmentation. Specifically, in the first stage, we adaptively aggregate local 3D features according to geometric density variations and obtain reliable pseudo labels under a global class-wise prototype consistency constraint. In the second stage, we cast category discovery as an optimal transport assignment process for globally consistent grouping of novel categories. Meanwhile, semantic features from 2D vision foundation models are fused with 3D point cloud features to enrich 3D semantic information and promote a more separable feature space. Extensive experiments on Synth4D to SemanticKITTI, SynLiDAR to SemanticKITTI, and Synth4D to nuScenes demonstrate the effectiveness of our framework. Under a unified evaluation protocol, our method achieves the best overall mIoU on 9 out of 10 benchmark splits and improves the overall mIoU by up to 12.21 points over the strongest re-implemented baseline.
Figures & tables
Figure 1. Illustration of the TTA and GCD pipelines.
Figure 2. Overview of the source pretraining, test-time adaptation, and generalized category discovery pipeline.
mIoU
Split
Method
vehicle
person
road
sidewalk
terrain
manmade
vegetation
Novel
Known
All
Full
77.03
0.00
20.29
19.35
21.07
77.61
81.28
-
-
42.37
Source
75.50
14.35
-
-
-
39.91
71.42
-
50.29
-
NOPS ∗
45.58
7.68
44.70
10.76
14.75
38.84
64.92
24.40
39.26
32.46
SNOPS ∗
68.50
14.94
58.85
30.12
34.36
52.16
67.53
41.11
50.78
46.64
Synth4d 0
Ours
69.54
13.14
21.25
18.15
21.76
38.53
66.96
20.40
47.10
35.65
Table 2. The novel class discovery results on the Synth4D → SemanticKITTI setting with 7 classes.
mIoU
Split
Method
vehicle
person
road
sidewalk
terrain
manmade
vegetation
Novel
Known
All
Full
63.59
24.78
40.37
12.16
5.09
74.83
61.24
-
-
40.29
Source
66.80
24.21
-
-
-
76.91
60.89
-
57.20
-
NOPS ∗
58.37
18.15
43.35
6.72
8.41
68.29
49.87
19.49
48.67
36.17
SNOPS ∗
51.11
2.15
45.65
12.75
3.12
58.48
56.07
20.51
41.95
32.76
Synth4d 0
Ours
64.72
20.48
37.15
8.77
3.90
72.94
62.91
16.61
55.26
38.70
Table 3. The novel class discovery results on the Synth4D → nuScenes setting with 7 classes.
Method
SemanticKITTI → nuScenes
nuScenes → SemanticKITTI
Novel
Known
All
Novel
Known
All
NOPS
30.15
28.47
29.19
26.67
29.08
28.04
SNOPS
24.04
19.79
21.61
39.21
41.42
40.47
Ours
46.72
38.41
41.97
42.07
42.38
42.25
Table 4. Real-to-real setting (Split 0).
Figure 3. Comparison between H^(A) and effective rank eH^(A) . The count of the largest eigenvalues necessary to account for 99% of the total eigenvalue energy serves as a surrogate for the rank.
Figure 4. Comparison of UMAP visualizations between SNOPS and our method on the nuScenes dataset (Split 1).
Figure 5. Pseudo label visualization (Split 0).
F
FL FR
B
BL BR
mIoU all
Split 0
Split 1
Split 2
34.78
38.83
41.22
✓
32.03 ( -2.75 )
38.01 ( -0.82 )
42.99 ( +1.77 )
✓
✓
33.75 ( -1.03 )
39.88 ( +1.05 )
46.71 ( +5.49 )
✓
✓
✓
✓
38.70 ( +3.92 )
44.27 ( +5.44 )
48.90 ( +7.68 )
F
FL FR
B
BL BR
mIoU novel
Table 5. Ablation on different numbers of 2D image views on Synth4D → nuScenes setting. F, FL, FR, B, BL, and BR denote FRONT, FRONT_LEFT, FRONT_RIGHT, BACK, BACK_LEFT, and BACK_RIGHT, respectively.
mIoU
Split 0
Split 1
Split 2
Split 3
w/o adapt kNN
21.8
24.1
24.8
23.6
w/ adapt kNN
27.4 ( +5.6 )
23.8 ( -0.3 )
25.1 ( +0.3 )
24.8 ( +1.2 )
Table 6. Ablation study on whether to adopt adaptive kNN in the SynLiDAR → SemanticKITTI setting.
Figure 6. Impact of different hyper-parameters on the Synth4D → nuScenes setting (Split 0).
LiDAR-based 3D Object Detection (L3OD) is a fundamental 3D perception task, yet its performance is often compromised by domain shifts arising from diverse sensor configurations and environmental conditions. Existing Test-Time Adaptation (TTA) methods primarily leverage self-supervision in the spatial domain, yet they frequently confound semantic content with domain style due to the sparse, non-Euclidean nature of point clouds. To overcome this limitation, we propose FIND (Frequency INvariance Disentanglement), a novel TTA framework that precisely disentangles domain-invariant content from domain-specific interference within the frequency domain. Our methodology utilizes B-Spline fitting to construct locally adaptive filters integrated into a dual-stream architecture: an Invariant Stream extracts robust features to guide pseudo-labeling, while a Specific Stream enforces consistency against domain perturbations. By combining invariance-driven alignment with stability-guided regularization, our approach dynamically extracts robust domain-invariant features while suppressing domain-specific interference. Extensive experiments on cross-dataset adaptation and robust corruption benchmarks demonstrate that FIND significantly outperforms state-of-the-art methods. The source code is publicly available at https://github.com/llldx03/FIND.
Dongxiao Li, Zixuan Hu, Ling-Yu Duan
School of Computer Science, Peking University, Beijing 100871, China
Accurate 3D semantic perception is critical for safe autonomous navigation. However, supervised LiDAR segmentation remains tied to closed taxonomies and to the cost of point-wise manual annotation. Open-vocabulary methods avoid that cost by projecting the output of 2D vision-language models onto LiDAR and distilling it into a 3D network. These methods rely almost exclusively on voxel-based sparse convolutions, and point transformers have so far been limited to indoor environments, where 3D data is dense and bounded. We present Lang3DSeg, which establishes a point transformer as the backbone for annotation-free open-vocabulary segmentation of outdoor 3D LiDAR, and is trained from scratch without geometric pre-training. This training paradigm necessitates addressing the inherent noise in 2D-to-3D label projections; specifically, naive projection often suffers from depth ambiguity, where points behind an object are erroneously assigned its semantic label. We therefore composite masks using an explicit class-priority rule and truncate each projected instance at the first gap in its depth distribution, correcting the projection error directly rather than averaging it over registered sequences. Lang3DSeg achieves 52.8% mIoU on nuScenes validation and 41.4% on SemanticKITTI, the highest among published annotation-free methods on both benchmarks. Every 3D semantic segmentation is on a single LiDAR sweep, and inference operates in real-time without running vision-language models.
Cigdem Kokenoz, Amir Salarpour, Alkim Domeke +5
Department of Automotive Engineering, Clemson University, Greenville, SC, USA. · School of Computing, Clemson University, Clemson, SC, USA.
LiDAR semantic segmentation often degrades under real-world deployment due to evolving sensing conditions, while collecting new annotations for retraining is impractical. Test-time adaptation (TTA) updates model parameters online using pseudo-label supervision, but directly applying standard TTA strategies to LiDAR data is challenging. Because pseudo-label reliability is spatially heteroscedastic under range-dependent sparsity and occlusion, uniform updates on globally shared parameters can inject unstable gradients and destabilize adaptation. We propose a geometry-constrained test-time prompt tuning framework for LiDAR semantic segmentation. Our method estimates per-location sensing reliability from depth-consistent beam terminations and neighborhood support, and uses it to reweight spatial supervision. Adaptation is confined to lightweight prompt adapters inserted into a frozen backbone, with spatial gating to prevent unreliable regions from perturbing globally shared representations. A temporally smoothed prototype alignment strategy further stabilizes online updates by accumulating reliable semantic evidence over time. Experiments on standard LiDAR benchmarks demonstrate improved adaptation stability and segmentation performance under deployment variations without additional annotations.
Linlian Jiang, Wentao Ju, Sadman Rakib Pinon +4
Concordia University · Mila – Quebec AI Institute · University of Toronto