GeoGuide: Hierarchical Geometric Guidance for Open-Vocabulary 3D Semantic Segmentation

Xujing Tao; Chuxin Wang; Yubo Ai; Zhixin Cheng; Zhuoyuan Li; Liangsheng Liu; Yujia Chen; Xinjun Li

cs.CVMar 27, 2026

GeoGuide: Hierarchical Geometric Guidance for Open-Vocabulary 3D Semantic Segmentation

Authors: Xujing Tao, Chuxin Wang, Yubo Ai, Zhixin Cheng, Zhuoyuan Li, Liangsheng Liu, Yujia Chen, Xinjun Li, +3 more

Organizations: University of Science and Technology of China · National Key Laboratory of Deep Space Exploration, Deep Space Exploration Laboratory

Abstract

Open-vocabulary 3D semantic segmentation aims to segment arbitrary categories beyond the training set. Existing methods predominantly rely on distilling knowledge from 2D open-vocabulary models. However, aligning 3D features to the 2D representation space restricts intrinsic 3D geometric learning and inherits errors from 2D predictions. To address these limitations, we propose GeoGuide, a novel framework that leverages pretrained 3D models to integrate hierarchical geometry-semantic consistency for open-vocabulary 3D segmentation. Specifically, we introduce an Uncertainty-based Superpoint Distillation module to fuse geometric and semantic features for estimating per-point uncertainty, adaptively weighting 2D features within superpoints to suppress noise while preserving discriminative information to enhance local semantic consistency. Furthermore, our Instance-level Mask Reconstruction module leverages geometric priors to enforce semantic consistency within instances by reconstructing complete instance masks. Additionally, our Inter-Instance Relation Consistency module aligns geometric and semantic similarity matrices to calibrate cross-instance consistency for same-category objects, mitigating viewpoint-induced semantic drift. Extensive experiments on ScanNet v2, Matterport3D, and nuScenes demonstrate the superior performance of GeoGuide.

Figures & tables

Figure 1: (a) Occlusions and ambiguous boundaries in 2D predictions lead to inaccurate 3D segmentation, whereas our geometry-guided modeling effectively rectifies these errors. (b) The baseline suffers from the lack of geometric relation modeling, whereas our method exploits inter-instance geometric constraints to produce accurate predictions.

Figure 2: Overview of our proposed GeoGuide framework . GeoGuide integrates hierarchical geometric priors to achieve geometry–semantic consistent open-vocabulary 3D segmentation

Panel of Figure 3 — Figure 3: Preliminary Experiments. Comparison between OpenScene and our preliminary attempt under different 2D models. LS: LSeg, OS: OpenSeg.

Figure 4: Overview of the USD . Uncertainty prediction from 3D geometry and 2D semantics guides the distillation of semantic knowledge.

Method	ScanNetv2		nuScenes		Matterport3D
Method	mIoU	mAcc	mIoU	mAcc	mIoU	mAcc
Fully-supervised
TangentConv [ 64 ]	40.9	-	-	-	-	46.8
TextureNet [ 28 ]	54.8	-	-	-	-	63.0
ScanComplete [ 17 ]	56.6	-	-	-	-	44.9
Mix3D [ 53 ]	73.6	-	-	-	-	-

Table 1: Open-vocabulary 3D semantic segmentation results. Comparison between GeoGuide and existing fully-supervised and zero-shot methods.

\text{SAS}^{*}

denotes using the same 2D features as in SAS [ 42 ] ,

{\ddagger}

denotes the use of the 2D-3D ensemble strategy, LS: LSeg, OS: OpenSeg . Best results are shown in bold .

Method	K = 21		K = 40		K = 80		K = 160
Method	mIoU	mAcc	mIoU	mAcc	mIoU	mAcc	mIoU	mAcc
OpenScene(LS) ${\ddagger}$ † [ 57 ]	43.4	53.5	25.6	31.3	12.5	15.8	6.4	8.2
OpenScene(LS)† [ 57 ]	41.9	51.1	25.4	30.7	12.0	15.1	5.9	7.6
OpenScene(OS) ${\ddagger}$ † [ 57 ]	42.6	58.7	34.4	49.9	21.0	33.2	10.9	18.3
OpenScene(OS)† [ 57 ]	41.1	55.2	33.4	46.7	18.1	27.2	8.9	14.0
DMA(OS) [ 38 ]	45.1	57.6	37.9	47.7	19.7	26.7	9.4	14.1

Table 2: Long-tail scenario evaluation on Matterport3D with varying numbers of categories.

{\dagger}

denotes results reproduced by us using officially provided weights,

{\ddagger}

denotes the use of the 2D-3D ensemble strategy.

\text{SAS}^{*}

denotes using the same 2D features as in SAS [ 42 ] .

Method	K = 21		K = 40		K = 80		K = 160
Method	mIoU	mAcc	mIoU	mAcc	mIoU	mAcc	mIoU	mAcc
OpenScene(OS)† [ 57 ]	34.4	50.8	23.8	37.6	12.9	21.0	6.3	11.1
OpenScene(LS)† [ 57 ]	36.4	48.1	21.3	27.7	10.8	14.1	5.4	7.1
GGSD [ 69 ]	40.1	54.4	22.8	31.6	11.9	16.2	6.3	9.6
CUA-O3D ${\ddagger}$ [ 37 ]	37.4	49.2	23.3	30.2	12.2	16.3	6.1	8.4
SAS† [ 42 ]	43.0	56.5	18.0	26.2	9.8	14.7	4.9	7.3

Table 3: Cross-domain generalization evaluation. Models are trained on ScanNet v2 and evaluated on Matterport3D (21/40/80/160 categories) ,

{\dagger}

denotes results reproduced by us using officially provided weights,

{\ddagger}

denotes the use of the 2D-3D ensemble strategy.

\text{SAS}^{*}

denotes using the same 2D features as in SAS [ 42 ] .

Method	mIoU	mAcc
OpenScene(LS)† [ 57 ]	37.2	44.8
OpenScene(OS)† [ 57 ]	38.8	52.2
CUA-O3D ${\ddagger}$ [ 37 ]	38.6	46.6
SAS† [ 42 ]	44.3	52.9
Ours (LS)	52.2	64.5
Ours (OS)	51.1	71.3

Table 4: Cross-domain generalization evaluation . Models are trained on Matterport3D and evaluated on ScanNet v2 .

{\dagger}

denotes results reproduced by us using officially provided weights,

{\ddagger}

denotes the use of the 2D-3D ensemble strategy.

\text{SAS}^{*}

denotes using the same 2D features as in SAS [ 42 ] .

Figure 5: Visualization of semantic segmentation results on ScanNet v2 and Matterport3D.

Group	SD	USD	IMR	IIRC	mIoU	mAcc
A	✗	✗	✗	✗	54.2	66.6
B	✗	✗	✗	✗	53.1	63.9
C	✗	✗	✗	✗	53.9	65.8
D	✓	✗	✗	✗	55.8	67.2
E	✗	✓	✗	✗	56.4	68.3
F	✗	✗	✓	✗	56.8	68.9

Table 5: Ablation study of the model with different designs .

Explore similar work

Geometry-Guided Semantic Reconstruction for 3D Instance Segmentation
Oct 12, 2026Yuanhao Su, Shaofeng Zhang3D Instance Segmentation Feature Fusion
University of Science and Technology of China · Fuzhou University
EPS3D: End-to-End Feed-Forward 3D Panoptic Segmentation
Jun 8, 2026Runsong Zhu, Jiaxin Guo, Xiaoyang Guo +9Panoptic Segmentation Open-Vocabulary 3D Segmentation
The Chinese University of Hong Kong, HK SAR, China · Hong Kong Centre for Logistics Robotics, HK SAR, China · Horizon Robotics, China +2
OpenGaFF: Open-Vocabulary Gaussian Feature Field with Codebook Attention
May 7, 2026Kunyi Li, Michael Niemeyer, Sen Wang +33D Gaussian Splatting Open-Vocabulary 3D Segmentation
1Technical University of Munich · 3Munich Center for Machine Learning · 2Google +1

Oct 12, 2026J/K move · Enter open · S save

Geometry-Guided Semantic Reconstruction for 3D Instance Segmentation

Yuanhao Su, Shaofeng Zhang

University of Science and Technology of China · Fuzhou University

3D Instance Segmentation Feature Fusion

Open paper AskPDF arXiv

3D instance segmentation requires representations that are both geometrically structured and semantically discriminative. Pure 3D encoders model spatial structure well but often lack strong appearance semantics, while directly lifting 2D priors into 3D by concatenation injects rich semantics without ensuring that they align well with the underlying 3D structure. In this paper, we address this problem by formulating 2D-to-3D semantic transfer as a masked semantic reconstruction task. Specifically, we mask a subset of background lifted 2D priors and train the 3D encoder to recover the missing semantic priors, providing explicit supervision for distilling rich 2D semantics into the 3D backbone. We further propose geometry-modulated semantic injection, which uses raw 3D geometry to modulate lifted 2D priors before sparse encoding, so that geometry explicitly controls semantic injection and alleviates the distribution gap between 2D semantic features and 3D geometric features. Experiments on two baseline frameworks demonstrate that the proposed method achieves state-of-the-art results in both full-data and few-shot settings on ScanNetV2 and ScanNet200. Code is publicly available at https://github.com/yharoldsu0627/Geometry_Guided_Semantic_Reconstruction

Abstract

Explore similar work

Geometry-Guided Semantic Reconstruction for 3D Instance Segmentation

EPS3D: End-to-End Feed-Forward 3D Panoptic Segmentation

OpenGaFF: Open-Vocabulary Gaussian Feature Field with Codebook Attention