COSTA: A Cluster-Centric Paradigm for Annotation-Free Open-Set Semantic Segmentation of Aerial Point Clouds with Domain Shifts
Authors: Yanghong Lin, Li Fang, Tianyu Li, Shudong Zhou, Wei Yao
Organizations: Fujian Institute of Research on the Structure of Matter, Chinese Academy of Sciences, Fuzhou, 350108, Fujian, China. · Spatial Intelligence and Urban Computing, Institute of Urban Environment, Chinese Academy of Sciences, Xiamen, 361021, Fujian, China. · State Key Lab for Ecological Security of Regions and Cities, Institute of Urban Environment, Chinese Academy of Sciences, Xiamen, 361021, Fujian, China. · University of Chinese Academy of Sciences, Beijing, 101408, Beijing, China. · School of Resource and Environmental Sciences, Wuhan University, Wuhan, 430072, Hubei, China.
Semantic segmentation of aerial point cloud is trapped in a generalization crisis under distinct domain shifts. While test-time adaptation offers a privacy-preserving and computationally efficient way to adapt pre-trained models to unlabeled target-domain data during inference, existing methods, bound to closed-set label assumptions and non-scalable point-wise segmentation pipelines, still struggle with semantic shifts. We ask: can we adapt any given pre-trained aerial point cloud segmentation model to a shifted target domain at the inference phase alone, without additional training, while segmenting target-specific categories beyond the source label space on demand? This paper introduces COSTA, which breaks this limitation by shifting from closed-set point-wise adaptation to cluster-centric open-set semantic propagation. Our core discovery is that, once effectively adapted at test time, the rich feature distribution of aerial point clouds can be distilled into a compact set of well-separated semantic centroids that are transferable across label spaces. COSTA leverages this to reformulate open-set semantic segmentation as a cluster-level propagating process: it first bridges the domain gap through proven test-time adaptation, then groups each batch of target-domain points into a small set of semantic clusters based on the similarity distribution in the adapted feature space, and finally propagates high-confidence pseudo labels obtained from an open-vocabulary vision-language model to all points through cluster-level voting. This cluster-centric paradigm enables test-time adaptation of aerial point clouds under significant domain gaps with mixed semantic shifts. With DALES as the source domain, COSTA enables on-demand segmentation across three aerial point cloud benchmarks with distinct domains and heterogeneous category spaces, achieving up to 70.09% mIoU under this new setting.
Figures & tables
Figure 1: With DALES as the source domain and SUM as the target domain, we partition the target-domain data with K-Means in TTA feature space. The results show that most clusters exhibit high semantic purity: over 80% of the clusters have a proportion of same-class points ranging from 0.9 to 1.0. Notably, this observation extends beyond the source label space.
Figure 2: Overview of the proposed COSTA for OSTTA. We first perform hybrid multi-view rendering on target point cloud by exploiting complementary BEV and oblique virtual camera projection. This yields hybrid view set with pixel-to-point correspondences for seamless VLMs semantic querying without well-aligned images. Given user-specified target classes, we prompt VLMs to generate open-vocabulary 2D probability masks, which are filtered adaptively and fused through reliability-constrained multi-view evidence accumulation. This provides reliability-aware pseudo labels for the next step. Given the pseudo labels, we adapt source-pretrained 3D model to the target data through differentiated self-supervised learning during inference. Finally, we perform reliability-weighted cluster-level semantic propagation by clustering the adapted point features. The framework produces on-demand open-set semantic segmentation on the target domain without source-data replay, target annotations, or offline retraining.
Figure 3: Illustration of BN TTA with differentiated self-supervised learning. For each target batch, only the affine parameters {γl,βl}l∈LBN of the BN layers are optimized, while all remaining network parameters are frozen. Reliable pseudo-ID points are encouraged to produce confident source-space predictions through entropy minimization, whereas pseudo-OOD points are prevented from being over-confidently assigned to source classes through entropy maximization. A diversity regularization term further prevents class collapse and preserves batch-level discriminability.
Table 1: Semantic relationship of the constructed OSTTA benchmarks between the target domain and the source domain. 0 0 footnotetext: Known classes denote target classes that directly overlap with the source classes. Cross-granularity classes denote target classes that are finer or coarser semantic variants of the source classes. Unseen classes denote target classes without reliable source-domain counterparts.
Setting
Pretrained model
Target data
Offline training
Fully-supervised method
×
xt,yt
✓
Open-vocabulary distillation
✓
xt
✓
Multi-dataset pretraining
✓
xt,yt
✓
Open-set test-time adaptation
✓
xt
×
Table 2: Comparison of different segmentation and adaptation settings.
Method
mIoU
OA
Terrain
High Veg.
Building
Water
Vehicle
Boat
Fully-supervised
PointNet ( Qi et al., 2017a )
23.70
80.70
67.90
89.50
80.00
0.00
0.00
3.90
PointNet++ ( Qi et al., 2017a )
32.90
84.30
72.40
94.20
84.70
2.70
2.00
25.70
SPGraph ( Landrieu and Simonovsky, 2018 )
37.20
76.90
69.90
94.50
88.80
32.80
12.50
15.70
SparseConv ( Graham et al., 2018 )
42.60
85.20
74.10
97.90
94.20
63.30
7.50
24.20
MinkUnet ( Choy et al., 2019 )
70.00
93.30
84.40
92.00
92.90
76.80
61.60
12.60
Table 3: Quantitative comparison on the DALES-SUM benchmark. Target classes are divided into known, cross-granularity, and unseen classes according to their semantic relationship with the source.
Figure 4: Classification result of SUM dataset with a model adapted from DALES dataset.
Figure 5: Classification result of H3D dataset with a model adapted from DALES dataset.
Figure 6: Classification result of CUS3D dataset with a model adapted from DALES dataset.
Dataset
Mode
OA (%)
mAcc (%)
mIoU (%)
Unknown ratio (%)
SUM
BEV-only
48.82
51.20
43.06
38.83
oblique-only
78.72
67.83
58.92
8.26
Fused
81.07
73.08
62.89
5.90
H3D
BEV-only
48.94
33.18
24.17
20.61
oblique-only
60.84
38.41
27.22
14.80
Fused
60.16
42.20
28.38
7.27
Table 6: Overall pseudo-label evaluation results under different projection modes. The best OA, mAcc, and mIoU within each dataset are highlighted in bold, and the lowest unknown ratio is also highlighted.
Mode
Terrain
High Vegetation
Building
Water
Vehicle
Boat
BEV-only
55.63
43.29
35.95
69.01
27.67
26.83
Oblique-only
63.06
54.09
76.58
88.73
11.95
59.10
Fused
63.17
63.10
78.19
85.83
25.75
61.32
Table 7: Per-class IoU comparison of different projection modes on SUM. The best result for each class is highlighted in bold.
Mode
Low Veg.
Imp. Surf.
Vehicle
Urb. Furn.
Roof
Facade
Shrub
Tree
Soil/Gravel
Vert. Surf.
Chimney
BEV-only
41.82
47.69
42.51
6.63
73.63
0.48
4.44
16.34
29.88
2.30
0.16
Oblique-only
58.39
65.15
17.02
5.48
60.75
10.76
2.09
31.34
39.31
8.74
0.37
Fused
54.04
56.55
35.24
8.00
71.49
12.04
5.01
31.82
29.93
7.77
0.34
Table 8: Per-class IoU comparison of different projection modes on H3D. The best result for each class is highlighted in bold.
Mode
Building
Road
Car
Grass
High Veg.
Playground
Water
Farmland
Bldg. sites
Ground
BEV-only
36.89
54.92
38.39
29.46
40.48
14.19
67.25
31.90
14.39
3.29
Oblique-only
46.84
51.97
22.96
25.14
39.77
3.37
79.84
57.94
3.35
3.87
Fused
58.74
57.58
39.13
33.71
53.70
6.24
78.06
53.92
3.68
8.02
Table 9: Per-class IoU comparison of different projection modes on CUS3D. The best result for each class is highlighted in bold.
Figure 7: Overall pseudo-label quality changes w.r.t. to the reliability weight thresholds across SUM, H3D, and CUS3D. Bars show pseudo-label recall, and plots show pseudo-label precision with different reliability thresholds.
Setting
H+S
Ent.
Div.
OOD
RW-CP
mIoU (%)
OA (%)
Base
✓
–
–
–
–
62.89
81.07
CP
✓
✓
✓
–
–
65.96
86.38
OOD+CP
✓
✓
✓
✓
–
67.80
87.42
Full
✓
✓
✓
✓
✓
70.09
88.52
Table 10: Ablation study of the proposed pipeline on the DALES-SUM benchmark. 0 0 footnotetext: H+S denotes hybrid projection with Sa2VA pseudo-label generation. Ent. and Div. denote entropy minimization and diversity regularization used in BN-only test-time adaptation. OOD denotes the out-of-distribution entropy. CP denotes cluster-centric propagation, and RW-CP denotes reliability-weighted cluster-centric propagation.
Figure 8: Feature space visualization for target classes on SUM. Our method produces a well-structured feature space for target-specific classes.