COSTA: A Cluster-Centric Paradigm for Annotation-Free Open-Set Semantic Segmentation of Aerial Point Clouds with Domain Shifts
Authors: Yanghong Lin, Li Fang, Tianyu Li, Shudong Zhou, Wei Yao
Organizations: Fujian Institute of Research on the Structure of Matter, Chinese Academy of Sciences, Fuzhou, 350108, Fujian, China. · Spatial Intelligence and Urban Computing, Institute of Urban Environment, Chinese Academy of Sciences, Xiamen, 361021, Fujian, China. · State Key Lab for Ecological Security of Regions and Cities, Institute of Urban Environment, Chinese Academy of Sciences, Xiamen, 361021, Fujian, China. · University of Chinese Academy of Sciences, Beijing, 101408, Beijing, China. · School of Resource and Environmental Sciences, Wuhan University, Wuhan, 430072, Hubei, China.
Semantic segmentation of aerial point cloud is trapped in a generalization crisis under distinct domain shifts. While test-time adaptation offers a privacy-preserving and computationally efficient way to adapt pre-trained models to unlabeled target-domain data during inference, existing methods, bound to closed-set label assumptions and non-scalable point-wise segmentation pipelines, still struggle with semantic shifts. We ask: can we adapt any given pre-trained aerial point cloud segmentation model to a shifted target domain at the inference phase alone, without additional training, while segmenting target-specific categories beyond the source label space on demand? This paper introduces COSTA, which breaks this limitation by shifting from closed-set point-wise adaptation to cluster-centric open-set semantic propagation. Our core discovery is that, once effectively adapted at test time, the rich feature distribution of aerial point clouds can be distilled into a compact set of well-separated semantic centroids that are transferable across label spaces. COSTA leverages this to reformulate open-set semantic segmentation as a cluster-level propagating process: it first bridges the domain gap through proven test-time adaptation, then groups each batch of target-domain points into a small set of semantic clusters based on the similarity distribution in the adapted feature space, and finally propagates high-confidence pseudo labels obtained from an open-vocabulary vision-language model to all points through cluster-level voting. This cluster-centric paradigm enables test-time adaptation of aerial point clouds under significant domain gaps with mixed semantic shifts. With DALES as the source domain, COSTA enables on-demand segmentation across three aerial point cloud benchmarks with distinct domains and heterogeneous category spaces, achieving up to 70.09% mIoU under this new setting.
Figures & tables
Figure 1: With DALES as the source domain and SUM as the target domain, we partition the target-domain data with K-Means in TTA feature space. The results show that most clusters exhibit high semantic purity: over 80% of the clusters have a proportion of same-class points ranging from 0.9 to 1.0. Notably, this observation extends beyond the source label space.
Figure 2: Overview of the proposed COSTA for OSTTA. We first perform hybrid multi-view rendering on target point cloud by exploiting complementary BEV and oblique virtual camera projection. This yields hybrid view set with pixel-to-point correspondences for seamless VLMs semantic querying without well-aligned images. Given user-specified target classes, we prompt VLMs to generate open-vocabulary 2D probability masks, which are filtered adaptively and fused through reliability-constrained multi-view evidence accumulation. This provides reliability-aware pseudo labels for the next step. Given the pseudo labels, we adapt source-pretrained 3D model to the target data through differentiated self-supervised learning during inference. Finally, we perform reliability-weighted cluster-level semantic propagation by clustering the adapted point features. The framework produces on-demand open-set semantic segmentation on the target domain without source-data replay, target annotations, or offline retraining.
Figure 3: Illustration of BN TTA with differentiated self-supervised learning. For each target batch, only the affine parameters {γl,βl}l∈LBN of the BN layers are optimized, while all remaining network parameters are frozen. Reliable pseudo-ID points are encouraged to produce confident source-space predictions through entropy minimization, whereas pseudo-OOD points are prevented from being over-confidently assigned to source classes through entropy maximization. A diversity regularization term further prevents class collapse and preserves batch-level discriminability.
Table 1: Semantic relationship of the constructed OSTTA benchmarks between the target domain and the source domain. 0 0 footnotetext: Known classes denote target classes that directly overlap with the source classes. Cross-granularity classes denote target classes that are finer or coarser semantic variants of the source classes. Unseen classes denote target classes without reliable source-domain counterparts.
Setting
Pretrained model
Target data
Offline training
Fully-supervised method
×
xt,yt
✓
Open-vocabulary distillation
✓
xt
✓
Multi-dataset pretraining
✓
xt,yt
✓
Open-set test-time adaptation
✓
xt
×
Table 2: Comparison of different segmentation and adaptation settings.
Method
mIoU
OA
Terrain
High Veg.
Building
Water
Vehicle
Boat
Fully-supervised
PointNet ( Qi et al., 2017a )
23.70
80.70
67.90
89.50
80.00
0.00
0.00
3.90
PointNet++ ( Qi et al., 2017a )
32.90
84.30
72.40
94.20
84.70
2.70
2.00
25.70
SPGraph ( Landrieu and Simonovsky, 2018 )
37.20
76.90
69.90
94.50
88.80
32.80
12.50
15.70
SparseConv ( Graham et al., 2018 )
42.60
85.20
74.10
97.90
94.20
63.30
7.50
24.20
MinkUnet ( Choy et al., 2019 )
70.00
93.30
84.40
92.00
92.90
76.80
61.60
12.60
Table 3: Quantitative comparison on the DALES-SUM benchmark. Target classes are divided into known, cross-granularity, and unseen classes according to their semantic relationship with the source.
Figure 4: Classification result of SUM dataset with a model adapted from DALES dataset.
Figure 5: Classification result of H3D dataset with a model adapted from DALES dataset.
Figure 6: Classification result of CUS3D dataset with a model adapted from DALES dataset.
Dataset
Mode
OA (%)
mAcc (%)
mIoU (%)
Unknown ratio (%)
SUM
BEV-only
48.82
51.20
43.06
38.83
oblique-only
78.72
67.83
58.92
8.26
Fused
81.07
73.08
62.89
5.90
H3D
BEV-only
48.94
33.18
24.17
20.61
oblique-only
60.84
38.41
27.22
14.80
Fused
60.16
42.20
28.38
7.27
Table 6: Overall pseudo-label evaluation results under different projection modes. The best OA, mAcc, and mIoU within each dataset are highlighted in bold, and the lowest unknown ratio is also highlighted.
Mode
Terrain
High Vegetation
Building
Water
Vehicle
Boat
BEV-only
55.63
43.29
35.95
69.01
27.67
26.83
Oblique-only
63.06
54.09
76.58
88.73
11.95
59.10
Fused
63.17
63.10
78.19
85.83
25.75
61.32
Table 7: Per-class IoU comparison of different projection modes on SUM. The best result for each class is highlighted in bold.
Mode
Low Veg.
Imp. Surf.
Vehicle
Urb. Furn.
Roof
Facade
Shrub
Tree
Soil/Gravel
Vert. Surf.
Chimney
BEV-only
41.82
47.69
42.51
6.63
73.63
0.48
4.44
16.34
29.88
2.30
0.16
Oblique-only
58.39
65.15
17.02
5.48
60.75
10.76
2.09
31.34
39.31
8.74
0.37
Fused
54.04
56.55
35.24
8.00
71.49
12.04
5.01
31.82
29.93
7.77
0.34
Table 8: Per-class IoU comparison of different projection modes on H3D. The best result for each class is highlighted in bold.
Mode
Building
Road
Car
Grass
High Veg.
Playground
Water
Farmland
Bldg. sites
Ground
BEV-only
36.89
54.92
38.39
29.46
40.48
14.19
67.25
31.90
14.39
3.29
Oblique-only
46.84
51.97
22.96
25.14
39.77
3.37
79.84
57.94
3.35
3.87
Fused
58.74
57.58
39.13
33.71
53.70
6.24
78.06
53.92
3.68
8.02
Table 9: Per-class IoU comparison of different projection modes on CUS3D. The best result for each class is highlighted in bold.
Figure 7: Overall pseudo-label quality changes w.r.t. to the reliability weight thresholds across SUM, H3D, and CUS3D. Bars show pseudo-label recall, and plots show pseudo-label precision with different reliability thresholds.
Setting
H+S
Ent.
Div.
OOD
RW-CP
mIoU (%)
OA (%)
Base
✓
–
–
–
–
62.89
81.07
CP
✓
✓
✓
–
–
65.96
86.38
OOD+CP
✓
✓
✓
✓
–
67.80
87.42
Full
✓
✓
✓
✓
✓
70.09
88.52
Table 10: Ablation study of the proposed pipeline on the DALES-SUM benchmark. 0 0 footnotetext: H+S denotes hybrid projection with Sa2VA pseudo-label generation. Ent. and Div. denote entropy minimization and diversity regularization used in BN-only test-time adaptation. OOD denotes the out-of-distribution entropy. CP denotes cluster-centric propagation, and RW-CP denotes reliability-weighted cluster-centric propagation.
Figure 8: Feature space visualization for target classes on SUM. Our method produces a well-structured feature space for target-specific classes.
Adverse weather conditions significantly degrade the performance of LiDAR point cloud semantic segmentation networks by introducing large distribution shifts. Existing augmentation-based methods attempt to enhance robustness by simulating weather interference during training. However, they struggle to fully exploit the potential of augmentations due to the trade-off between minor and aggressive augmentations. To address this, we propose A3Point, an adaptive augmentation-aware latent learning framework that effectively utilizes a diverse range of augmentations while mitigating the semantic shift, which refers to the change in the semantic meaning caused by augmentations. A3Point consists of two key components: semantic confusion prior (SCP) latent learning, which captures the model's inherent semantic confusion information, and semantic shift region (SSR) localization, which decouples semantic confusion and semantic shift, enabling adaptive optimization strategies for different disturbance levels. Extensive experiments on multiple standard generalized LiDAR segmentation benchmarks under adverse weather demonstrate the effectiveness of our method, setting new state-of-the-art results.
Wangkai Li, Zhaoyang Li, Yuwen Pan +3
University of Science and Technology of China · National Key Laboratory of Deep Space Exploration, Deep Space Exploration Laboratory
Point cloud data face serious semantic privacy risks during acquisition, transmission, and cross-institutional sharing. Existing methods mostly rely on geometric perturbation or destructive encryption, which can reduce the recognizability of the original class but often impair downstream usability. This paper proposes a class-transfer-based semantic encryption framework for point clouds, aiming to conceal original class information while preserving task utility and supporting authorized recovery. Specifically, we construct a unified latent space with a shared-backbone Normalizing Flow, and combine LoRA and FiLM to achieve parameter-efficient class-conditional adaptation. We further introduce diffusion-guided flow alignment to regularize the latent distribution, construct an energy-based category transition graph, and obtain an optimal class-transfer table through global matching. Then, a latent-space Neural ODE continuously evolves source-class latents into target-class latents, which are decoded into target-class point clouds through the inverse flow. We adopt attacker-oriented metrics, including New-Class Recognition Rate (NCRR), Original-Class Leakage Rate (OCLR), and Original Label Recovery Rate (OLRR), to evaluate privacy and utility. Experiments on classification and segmentation benchmarks show that the proposed method achieves controllable semantic transformation, effectively reduces original-class semantic leakage, preserves downstream learnability in the protected domain, and supports reliable authorized reconstruction.
Jinchang zhang, Jiakai Lin, David Crandall +1
Department of Computer Science, Indiana University, Bloomington, IN, 47405, USA.
Recent vision-language models (VLMs) such as CLIP demonstrate impressive cross-modal reasoning, extending beyond images to 3D perception. Yet, these models remain fragile under domain shifts, especially when adapting from synthetic to real-world point clouds. Conventional 3D domain adaptation approaches rely on heavy trainable encoders, yielding strong accuracy but at the cost of efficiency. We introduce CLIPoint3D, the first framework for few-shot unsupervised 3D point cloud domain adaptation built upon CLIP. Our approach projects 3D samples into multiple depth maps and exploits the frozen CLIP backbone, refined through a knowledge-driven prompt tuning scheme that integrates high-level language priors with geometric cues from a lightweight 3D encoder. To adapt task-specific features effectively, we apply parameter-efficient fine-tuning to CLIP's encoders and design an entropy-guided view sampling strategy for selecting confident projections. Furthermore, an optimal transport-based alignment loss and an uncertainty-aware prototype alignment loss collaboratively bridge source-target distribution gaps while maintaining class separability. Extensive experiments on PointDA-10 and GraspNetPC-10 benchmarks show that CLIPoint3D achieves consistent 3-16% accuracy gains over both CLIP-based and conventional encoder-based baselines. Project page: https://sarthakm320.github.io/CLIPoint3D.
Mainak Singha, Sarthak Mehrotra, Paolo Casari +3
University of Trento, Italy · MDSR Labs Adobe, India · IIT Bombay, India +1