Open-vocabulary 3D semantic segmentation aims to segment arbitrary categories beyond the training set. Existing methods predominantly rely on distilling knowledge from 2D open-vocabulary models. However, aligning 3D features to the 2D representation space restricts intrinsic 3D geometric learning and inherits errors from 2D predictions. To address these limitations, we propose GeoGuide, a novel framework that leverages pretrained 3D models to integrate hierarchical geometry-semantic consistency for open-vocabulary 3D segmentation. Specifically, we introduce an Uncertainty-based Superpoint Distillation module to fuse geometric and semantic features for estimating per-point uncertainty, adaptively weighting 2D features within superpoints to suppress noise while preserving discriminative information to enhance local semantic consistency. Furthermore, our Instance-level Mask Reconstruction module leverages geometric priors to enforce semantic consistency within instances by reconstructing complete instance masks. Additionally, our Inter-Instance Relation Consistency module aligns geometric and semantic similarity matrices to calibrate cross-instance consistency for same-category objects, mitigating viewpoint-induced semantic drift. Extensive experiments on ScanNet v2, Matterport3D, and nuScenes demonstrate the superior performance of GeoGuide.
Figures & tables
Figure 1: (a) Occlusions and ambiguous boundaries in 2D predictions lead to inaccurate 3D segmentation, whereas our geometry-guided modeling effectively rectifies these errors. (b) The baseline suffers from the lack of geometric relation modeling, whereas our method exploits inter-instance geometric constraints to produce accurate predictions.
Figure 2: Overview of our proposed GeoGuide framework . GeoGuide integrates hierarchical geometric priors to achieve geometry–semantic consistent open-vocabulary 3D segmentation
Figure 3: Preliminary Experiments. Comparison between OpenScene and our preliminary attempt under different 2D models. LS: LSeg, OS: OpenSeg.
Figure 4: Overview of the USD . Uncertainty prediction from 3D geometry and 2D semantics guides the distillation of semantic knowledge.
Method
ScanNetv2
nuScenes
Matterport3D
mIoU
mAcc
mIoU
mAcc
mIoU
mAcc
Fully-supervised
TangentConv [ 64 ]
40.9
-
-
-
-
46.8
TextureNet [ 28 ]
54.8
-
-
-
-
63.0
ScanComplete [ 17 ]
56.6
-
-
-
-
44.9
Mix3D [ 53 ]
73.6
-
-
-
-
-
Table 1: Open-vocabulary 3D semantic segmentation results. Comparison between GeoGuide and existing fully-supervised and zero-shot methods. SAS∗ denotes using the same 2D features as in SAS [ 42 ] , ‡ denotes the use of the 2D-3D ensemble strategy, LS: LSeg, OS: OpenSeg . Best results are shown in bold .
Method
K = 21
K = 40
K = 80
K = 160
mIoU
mAcc
mIoU
mAcc
mIoU
mAcc
mIoU
mAcc
OpenScene(LS) ‡ † [ 57 ]
43.4
53.5
25.6
31.3
12.5
15.8
6.4
8.2
OpenScene(LS)† [ 57 ]
41.9
51.1
25.4
30.7
12.0
15.1
5.9
7.6
OpenScene(OS) ‡ † [ 57 ]
42.6
58.7
34.4
49.9
21.0
33.2
10.9
18.3
OpenScene(OS)† [ 57 ]
41.1
55.2
33.4
46.7
18.1
27.2
8.9
14.0
DMA(OS) [ 38 ]
45.1
57.6
37.9
47.7
19.7
26.7
9.4
14.1
Table 2: Long-tail scenario evaluation on Matterport3D with varying numbers of categories. † denotes results reproduced by us using officially provided weights, ‡ denotes the use of the 2D-3D ensemble strategy. SAS∗ denotes using the same 2D features as in SAS [ 42 ] .
Method
K = 21
K = 40
K = 80
K = 160
mIoU
mAcc
mIoU
mAcc
mIoU
mAcc
mIoU
mAcc
OpenScene(OS)† [ 57 ]
34.4
50.8
23.8
37.6
12.9
21.0
6.3
11.1
OpenScene(LS)† [ 57 ]
36.4
48.1
21.3
27.7
10.8
14.1
5.4
7.1
GGSD [ 69 ]
40.1
54.4
22.8
31.6
11.9
16.2
6.3
9.6
CUA-O3D ‡ [ 37 ]
37.4
49.2
23.3
30.2
12.2
16.3
6.1
8.4
SAS† [ 42 ]
43.0
56.5
18.0
26.2
9.8
14.7
4.9
7.3
Table 3: Cross-domain generalization evaluation. Models are trained on ScanNet v2 and evaluated on Matterport3D (21/40/80/160 categories) , † denotes results reproduced by us using officially provided weights, ‡ denotes the use of the 2D-3D ensemble strategy. SAS∗ denotes using the same 2D features as in SAS [ 42 ] .
Method
mIoU
mAcc
OpenScene(LS)† [ 57 ]
37.2
44.8
OpenScene(OS)† [ 57 ]
38.8
52.2
CUA-O3D ‡ [ 37 ]
38.6
46.6
SAS† [ 42 ]
44.3
52.9
Ours (LS)
52.2
64.5
Ours (OS)
51.1
71.3
Table 4: Cross-domain generalization evaluation . Models are trained on Matterport3D and evaluated on ScanNet v2 . † denotes results reproduced by us using officially provided weights, ‡ denotes the use of the 2D-3D ensemble strategy. SAS∗ denotes using the same 2D features as in SAS [ 42 ] .
Figure 5: Visualization of semantic segmentation results on ScanNet v2 and Matterport3D.
Group
SD
USD
IMR
IIRC
mIoU
mAcc
A
✗
✗
✗
✗
54.2
66.6
B
✗
✗
✗
✗
53.1
63.9
C
✗
✗
✗
✗
53.9
65.8
D
✓
✗
✗
✗
55.8
67.2
E
✗
✓
✗
✗
56.4
68.3
F
✗
✗
✓
✗
56.8
68.9
Table 5: Ablation study of the model with different designs .
3D instance segmentation requires representations that are both geometrically structured and semantically discriminative. Pure 3D encoders model spatial structure well but often lack strong appearance semantics, while directly lifting 2D priors into 3D by concatenation injects rich semantics without ensuring that they align well with the underlying 3D structure. In this paper, we address this problem by formulating 2D-to-3D semantic transfer as a masked semantic reconstruction task. Specifically, we mask a subset of background lifted 2D priors and train the 3D encoder to recover the missing semantic priors, providing explicit supervision for distilling rich 2D semantics into the 3D backbone. We further propose geometry-modulated semantic injection, which uses raw 3D geometry to modulate lifted 2D priors before sparse encoding, so that geometry explicitly controls semantic injection and alleviates the distribution gap between 2D semantic features and 3D geometric features. Experiments on two baseline frameworks demonstrate that the proposed method achieves state-of-the-art results in both full-data and few-shot settings on ScanNetV2 and ScanNet200. Code is publicly available at https://github.com/yharoldsu0627/Geometry_Guided_Semantic_Reconstruction
Yuanhao Su, Shaofeng Zhang
University of Science and Technology of China · Fuzhou University
This paper introduces EPS3D, a new end-to-end feed-forward framework for open-vocabulary 3D panoptic segmentation. Unlike existing methods relying on additional preprocessing, we design an end-to-end architecture, with a distillation-based training strategy on diverse 3D scenes to predict 3D-aware semantic and instance features from multi-view images, improving 3D consistency and avoiding error accumulation. We further propose a mutual enhancement module to enforce inherent semantic-instance consistency. By aligning semantics within instances (Ins2Sem) and refining instance features with semantic guidance (Sem2Ins), we achieve more coherent 3D scene understanding. Ultimately, EPS3D outperforms SOTA baselines on two benchmarks (e.g., +13% mIoU for semantics on Replica) with high efficiency (e.g., 1s per scene), supporting tasks like robotic manipulation and 3D scene editing.
Runsong Zhu, Jiaxin Guo, Xiaoyang Guo +9
The Chinese University of Hong Kong, HK SAR, China · Hong Kong Centre for Logistics Robotics, HK SAR, China · Horizon Robotics, China +2
Understanding open-vocabulary 3D scenes with Gaussian-based representations remains challenging due to fragmented and spatially inconsistent semantic predictions across multi-view observations. In this paper, we present OpenGaFF, a novel framework for open-vocabulary 3D scene understanding built upon 3D Gaussian Splatting. At the core of our method is a Gaussian Feature Field that models semantics as a continuous function of Gaussian geometry and appearance. By explicitly conditioning semantic predictions on geometric structure, this formulation strengthens the coupling between geometry and semantics, leading to improved spatial coherence across similar structures in 3D space. To further enforce object-level semantic consistency, we introduce a structured codebook that serves as a set of shared semantic primitives. Furthermore, a codebook-guided attention mechanism is proposed to retrieve language features via similarity matching between query embeddings and learned codebook entries, enabling robust open-vocabulary reasoning while reducing intra-object feature variance. Extensive experiments on standard 2D and 3D open-vocabulary benchmarks demonstrate that our method consistently outperforms prior approaches, achieving improved segmentation quality, stronger 3D semantic consistency and a semantically interpretable codebook that provides insight into the learned representation.
Kunyi Li, Michael Niemeyer, Sen Wang +3
1Technical University of Munich · 3Munich Center for Machine Learning · 2Google +1