Geometry-Guided Semantic Reconstruction for 3D Instance Segmentation

Yuanhao Su; Shaofeng Zhang

cs.CVOct 12, 2026

Geometry-Guided Semantic Reconstruction for 3D Instance Segmentation

Authors: Yuanhao Su, Shaofeng Zhang

Organizations: University of Science and Technology of China · Fuzhou University

Abstract

3D instance segmentation requires representations that are both geometrically structured and semantically discriminative. Pure 3D encoders model spatial structure well but often lack strong appearance semantics, while directly lifting 2D priors into 3D by concatenation injects rich semantics without ensuring that they align well with the underlying 3D structure. In this paper, we address this problem by formulating 2D-to-3D semantic transfer as a masked semantic reconstruction task. Specifically, we mask a subset of background lifted 2D priors and train the 3D encoder to recover the missing semantic priors, providing explicit supervision for distilling rich 2D semantics into the 3D backbone. We further propose geometry-modulated semantic injection, which uses raw 3D geometry to modulate lifted 2D priors before sparse encoding, so that geometry explicitly controls semantic injection and alleviates the distribution gap between 2D semantic features and 3D geometric features. Experiments on two baseline frameworks demonstrate that the proposed method achieves state-of-the-art results in both full-data and few-shot settings on ScanNetV2 and ScanNet200. Code is publicly available at https://github.com/yharoldsu0627/Geometry_Guided_Semantic_Reconstruction

Figures & tables

Figure 1: Comparison of three paradigms for 3D instance segmentation with 2D priors. (a) Pure 3D encoding preserves geometry but lacks rich semantics. (b) Direct concatenation injects lifted 2D priors, yet the transfer remains passive. (c) Our method combines geometry-modulated semantic injection and masked semantic reconstruction to enforce explicit semantic transfer.

Figure 2: Overview of the proposed framework. Multi-view DINO-X features are lifted to 3D points and processed by a sparse 3D encoder. During training, a subset of background superpoints is masked in the lifted 2D stream and reconstructed from geometry-modulated encoder features. The aggregated superpoint features are finally decoded into instance masks.

Figure 3: Efficiency of superpoint-level representation. Superpoint-level 2D priors reduce loading time from 263.3 ms to 0.4 ms and memory from 475.5 MB to 0.8 MB per scene, a 600 $\times$ reduction compared to point-level priors.

Method	Year	Paradigm	Cue	mAP $\uparrow$	AP@50 $\uparrow$	AP@25 $\uparrow$
DyCo3D	CVPR21	NQ	3D	35.4	57.6	72.9
SSTNet	ICCV21	NQ	3D	49.4	64.3	74.0
MaskGroup	ICME22	NQ	3D	42.0	63.3	73.3
SoftGroup	CVPR22	NQ	3D	46.0	67.6	78.9
ISBNet	CVPR23	NQ	3D	56.8	73.3	81.3
Mask3D	ICRA23	Q	3D	55.2	73.7	82.9

Table 1: Comparison on ScanNetV2 validation set. Paradigm: Q = Query-based, NQ = Non-query. Cue: 3D = 3D-only, 2D+3D = 2D priors + 3D geometry. ∗ SegDINO3D rows use identical 2D feature loading with superpoint-level aggregation (not original point-level features), enabling efficient caching. Best and second-best are marked in bold and underline .

Method	Year	Paradigm	Cue	mAP $\uparrow$	AP@50 $\uparrow$	AP@25 $\uparrow$
SPFormer	AAAI23	Q	3D	25.2	33.8	39.6
Mask3D	ICRA23	Q	3D	27.4	37.0	42.3
QueryFormer	ICCV23	Q	3D	28.1	37.1	43.4
Maft	ICCV23	Q	3D	29.2	38.2	43.3
IKNE	CVPR25	NQ	3D	29.9	38.9	44.9
Relation3D	CVPR25	Q	3D	31.6	41.2	45.6

Table 2: Comparison on ScanNet200 validation set. Paradigm and Cue definitions same as Table 1. ∗ SegDINO3D rows use identical 2D feature loading with superpoint-level aggregation (not original point-level features), enabling efficient caching.

Figure 4: Qualitative comparison on the ScanNetV2 validation set. Our method yields cleaner instance boundaries and more complete instance separation in the highlighted regions, especially under occlusion and around thin or ambiguous structures.

Figure 5: Efficiency comparison on ScanNetV2. Reconstruction adds no inference cost; modulation increases parameters and FLOPs, leading to modest latency increase.

Ratio	Framework	mAP $\uparrow$	AP@50 $\uparrow$	AP@25 $\uparrow$
5%	Relation3D	10.29	15.40	18.60
	Relation3D(Ours)	12.28	17.39	20.52
	SegDINO3D ∗	10.54	16.38	20.43
	SegDINO3D(Ours) ∗	16.81	24.98	29.79
10%	Relation3D	14.80	20.80	24.40
	Relation3D(Ours)	17.89	24.51	28.60

Table 3: Limited-data training results on ScanNet200. Each framework is trained with 5% or 10% of the training set and evaluated on the full validation set. Our method consistently improves the corresponding baseline under both low-data regimes on Relation3D and SegDINO3D. ∗ SegDINO3D rows use identical 2D feature loading with superpoint-level aggregation (not original point-level features), enabling efficient caching.

Table 9

Figure 6: Sensitivity analysis on ScanNetV2. Left: reconstruction weight

\lambda_{\text{rec}}

with

r=0.1

. Right: masking ratio

r

with

\lambda_{\text{rec}}=1.0

Explore similar work

GeoGuide: Hierarchical Geometric Guidance for Open-Vocabulary 3D Semantic Segmentation
Mar 27, 2026Xujing Tao, Chuxin Wang, Yubo Ai +8Open-Vocabulary 3D Segmentation 3D Semantic Segmentation
University of Science and Technology of China · National Key Laboratory of Deep Space Exploration, Deep Space Exploration Laboratory
SegVGGT: Joint 3D Reconstruction and Instance Segmentation from Multi-View Images
Mar 20, 2026Jinyuan Qu, Hongyang Li, Lei Zhang3D Instance Segmentation Multi-View 3D Reconstruction
Tsinghua University · International Digital Economy Academy (IDEA) · South China University of Technology
MV3DIS: Multi-View Mask Matching via 3D Guides for Zero-Shot 3D Instance Segmentation
Apr 10, 2026Yibo Zhao, Yigong Zhang, Jin Xie3D Instance Segmentation
College of Computer Science, Nankai University · School of Intelligence Science and Technology, Nanjing University

Mar 27, 2026J/K move · Enter open · S save

GeoGuide: Hierarchical Geometric Guidance for Open-Vocabulary 3D Semantic Segmentation

Xujing Tao, Chuxin Wang, Yubo Ai, Zhixin Cheng+7

University of Science and Technology of China · National Key Laboratory of Deep Space Exploration, Deep Space Exploration Laboratory

Open-Vocabulary 3D Segmentation 3D Semantic Segmentation 3D Point Cloud Segmentation Open-Vocabulary Semantic Segmentation Knowledge Distillation

Open paper AskPDF arXiv

Open-vocabulary 3D semantic segmentation aims to segment arbitrary categories beyond the training set. Existing methods predominantly rely on distilling knowledge from 2D open-vocabulary models. However, aligning 3D features to the 2D representation space restricts intrinsic 3D geometric learning and inherits errors from 2D predictions. To address these limitations, we propose GeoGuide, a novel framework that leverages pretrained 3D models to integrate hierarchical geometry-semantic consistency for open-vocabulary 3D segmentation. Specifically, we introduce an Uncertainty-based Superpoint Distillation module to fuse geometric and semantic features for estimating per-point uncertainty, adaptively weighting 2D features within superpoints to suppress noise while preserving discriminative information to enhance local semantic consistency. Furthermore, our Instance-level Mask Reconstruction module leverages geometric priors to enforce semantic consistency within instances by reconstructing complete instance masks. Additionally, our Inter-Instance Relation Consistency module aligns geometric and semantic similarity matrices to calibrate cross-instance consistency for same-category objects, mitigating viewpoint-induced semantic drift. Extensive experiments on ScanNet v2, Matterport3D, and nuScenes demonstrate the superior performance of GeoGuide.

Abstract

Explore similar work

GeoGuide: Hierarchical Geometric Guidance for Open-Vocabulary 3D Semantic Segmentation

SegVGGT: Joint 3D Reconstruction and Instance Segmentation from Multi-View Images

MV3DIS: Multi-View Mask Matching via 3D Guides for Zero-Shot 3D Instance Segmentation