Stream3Dv2: Geometric-Semantic Fusion Enhanced Streaming Zero-Shot 3D Scene Understanding

Jie Xu; Na Zhao

cs.CVAug 21, 2026

Stream3Dv2: Geometric-Semantic Fusion Enhanced Streaming Zero-Shot 3D Scene Understanding

Authors: Jie Xu, Na Zhao

Organizations: ISTD Pillar, Singapore University of Technology and Design, Singapore 487372

Abstract

Recently, open-vocabulary zero-shot 3D scene understanding using vision foundation models has emerged as a promising alternative to data-intensive supervised methods. However, deploying these models in real-world scenarios is severely hindered by their inability to efficiently handle streaming RGB-D inputs and their inherent vulnerability to noise 2D segmentation masks. To address these critical limitations, we propose Stream3Dv2, a novel training-free framework designed for robust streaming 3D perception. Stream3Dv2 processes sequential data through an original nested local-to-historical architecture, capturing multi-view consistency while circumventing the high computational overhead so as to support timely responses. At its core, we introduce a comprehensive geometric-semantic fusion mechanism that resolves geometric noise and semantic ambiguity by explicitly utilizing semantic guidance and formulating 3D segmentation as solving point-and-set merging and partitioning problems. Furthermore, we present an innovative manifold-distance-based point cloud refinement strategy. This approach leverages local manifold graphs for point-to-manifold optimization that mitigates the boundary delineation failures caused by Euclidean-distance metrics, and employs geometric bounding boxes to dynamically activate and update historical instances for achieving rapid manifold-to-manifold refinement. Extensive experiments on public datasets demonstrate that Stream3Dv2 consistently outperforms existing baselines in foundational open-vocabulary streaming 3D segmentation and detection. Finally, we show that integrating our framework with an LLM-based agent enables advanced language-driven 3D scene understanding, underscoring its potential for open-world embodied intelligence. Code will be updated at https://github.com/SubmissionsIn/Stream3D.

Figures & tables

Fig. 1 : Overview of Stream3Dv2 : (a) Streaming 3D scene understanding for embodied AI applications. (b) We leverage semantic&grid-prompted 2D segmentation and integrate local multi-view and local-to-historical modules to enable streaming zero-shot 3D instance segmentation. Specifically, (c) local multi-view module conducts class-agnostic noise mask filtering and semantic-driven mask merging&partitioning with set covering problem (SCP) and set partitioning problem (SPP) based strategies. Local geometric-semantic fusion produces robust 3D semantic segmentation. Then, (d) local-to-historical module refines 3D masks’ manifolds by assigning point-to-manifold on Riemannian graph metrics, followed by the detection-based efficient 3D instance localization&division to achieve manifold-to-manifold 3D mask updates.

Fig. 2 : Local multi-view coarse-grained 3D segmentation.

Fig. 3 : 3D mask merging and partitioning. Left: threshold-based methods suffer from dynamical overlapping or wrongly-merged masks. Right: semantic-driven method (our work) is able to make more feasible mask merging and partitioning with semantic information.

Method	ST	ZS	ScanNet200						ScanNet++						MatterPort3D
			Class-agnostic			Semantic			Class-agnostic			Semantic			Class-agnostic			Semantic
			AP	AP 50	AP 25	AP	AP 50	AP 25	AP	AP 50	AP 25	AP	AP 50	AP 25	AP	AP 50	AP 25	AP	AP 50	AP 25
Mask3D [ 2 ]	✗	✗	39.7	53.6	62.5	26.9	36.2	41.4	22.8	33.3	45.7	3.6	5.1	6.7	4.4	9.8	20.6	2.5	4.5	6.7
OpenMask3D [ 16 ]	✗	✗	39.7	53.6	62.5	15.1	19.6	22.6	22.8	33.3	45.7	2.0	2.7	3.4	4.4	9.8	20.6	4.6	8.5	13.0
OVIR-3D [ 23 ]	✗	✓	14.4	27.5	38.8	9.3	18.7	25.0	19.4	34.1	46.5	3.6	5.7	7.3	5.9	13.9	24.6	6.3	16.4	24.4
MaskClustering [ 18 ]	✗	✓	19.7	36.4	51.4	12.0	23.3	30.1	24.6	40.3	51.5	7.8	11.9	13.2	8.3	18.9	33.4	9.2	19.7	26.5

TABLE I : Open-vocabulary 3D instance segmentation on ScanNet200, ScanNet++, and MatterPort3D benchmarks, evaluated by Class-agnostic and Semantic segmentation performance. ST ✓: streaming method. ZS ✓: zero-shot method.

	chair	table	door	couch	cabinet	shelf	desk	offi.cha.	bed	pillow	sink	picture	window	toilet	booksh.	monitor	curtain	book	armcha.	coff.tab.	Avg.
MaskClustering	36.1	35.2	0.5	20.5	2.1	8.2	4.8	19.3	26.8	21.7	37.6	2.6	6.3	32.0	2.0	44.3	23.1	2.8	16.1	31.2	18.7
Stream3D	38.8	36.9	0.3	21.9	4.3	9.1	4.1	22.6	25.9	15.3	29.5	1.4	8.4	44.6	2.0	41.9	30.1	3.9	19.0	44.4	20.2
Stream3Dv2	40.9	49.7	50.0	22.5	24.2	9.1	6.2	56.6	60.0	23.7	41.5	35.4	33.7	74.6	17.5	45.7	58.9	24.7	46.3	45.6	38.3
Improve. $\uparrow$	2.1	12.8	49.5	0.6	19.9	0.0	1.4	34.0	33.2	2.0	3.9	32.8	25.3	30.0	15.5	1.4	28.8	20.8	27.3	1.2	18.1

TABLE II : Per-class segmentation performance comparison (semantic AP 50 across the first 20 classes) on ScanNet200 benchmark.

Fig. 4 : Visualization of 3D instance segmentation comparison across RGB-D scene, Stream3D, Stream3Dv2, and Ground truth.

Fig. 5 : Visualization of 3D object detection comparison between our Stream3Dv2 and Ground truth on ScanNet200 benchmark.

Method	ST	ZS	ScanNet200
Method	ST	ZS	AP	AP 50	AP 25	FPS
EmbodiedSAM [ 37 ]	✓	✗	28.8	42.7	54.2	10
SAM3D [ 22 ]	✓	✓	9.6	24.8	49.6	8
OnlineAnySeg [ 36 ]	✓	✓	18.6	36.1	53.5	15
MoonSeg3R [ 54 ]	✓	✓	16.7	33.3	50.0	15
Stream3D (ours)	✓	✓	22.4	37.6	55.1	13

TABLE III : Class-agnostic streaming 3D instance segmentation on ScanNet200 benchmark. ST ✓: streaming method. ZS ✓: zero-shot method. FPS : frames per second.

Method	ST	ZS	ScanNet200
Method	ST	ZS	AP 50	AP 25
OpenM3D [ 19 ]	✗	✗	–	4.2
Zoo3D 1 [ 21 ]	✗	✗	15.2	23.5
Zoo3D 0 [ 21 ]	✗	✓	14.1	21.1
Stream3D (ours)	✓	✓	10.6	18.9
Stream3Dv2 (ours)	✓	✓	14.2	24.8

TABLE IV : Open-vocabulary 3D object detection on ScanNet200 benchmark. ST ✓: streaming method. ZS ✓: zero-shot method.

Fig. 6 : Examples of downstream tasks by Stream3Dv2 equipped with LLM agent. Top : 3D instance grounding, reasoning, and navigation. Bottom : open-vocabulary queries with indirect semantic descriptions.

Fig. 7 : Our method obtains robust 3D segmentation by mitigating 2D noise masks from VFMs, in cases such as dynamically inconsistent masks, class-agnostic fine-grained masks, semantically error masks.

Fig. 8 : The process of our streaming zero-shot 3D instance segmentation on Top : room-level cases and Bottom : large building-level cases.

	MVF	SDS	PMR	ScanNet200						ScanNet++						MatterPort3D
				Class-agnostic			Semantic			Class-agnostic			Semantic			Class-agnostic			Semantic
				AP	AP 50	AP 25	AP	AP 50	AP 25	AP	AP 50	AP 25	AP	AP 50	AP 25	AP	AP 50	AP 25	AP	AP 50	AP 25
(I)	✗	✗	✗	16.6	28.7	45.7	8.0	14.3	21.5	5.0	9.6	19.4	1.7	3.5	5.3	3.0	7.0	18.4	1.6	3.6	10.7
(II)	✓			19.0	32.4	49.7	9.6	16.1	22.6	11.3	22.1	39.8	3.4	6.2	9.8	5.6	13.1	33.3	4.3	11.7	23.2
(III)		✓		20.1	33.9	48.9	17.1	28.9	42.0	18.3	30.8	43.3	12.8	20.6	28.4	10.6	21.3	39.9	10.4	22.5	38.7
(IV)			✓	18.6	32.2	48.3	8.8	16.0	28.5	14.1	23.4	35.1	4.5	6.4	8.0	8.0	17.1	33.9	2.5	4.8	12.6

TABLE V : Ablation study on the main methodologies in Stream3Dv2 paradigm. MVF denotes the multi-view noise mask filtering. SDS is the semantic-driven fine-grained 3D segmentation. PMR is the point cloud manifold refinement.

	$t=400$	$t=1200$	$t=2000$	$t=2800$
Loc 3D seg. (ms)	52	67	43	73
Loc2his 3D seg. (ms)	91	241	237	377
Memory (MB)	10.4	15.6	19.0	21.3

TABLE VI : Time latency during long sequence segmentation on MatterPort3D benchmark (scene id: gxdoqLR6rwA).

Panel of Figure 15 — Fig. 9 : Parameter analysis on (a) window size $k$ , (b) key point rate $\gamma$ , (c) geometry distance $\delta$ , and (d) overlap threshold $\alpha$ .

Prompt	ScanNet200						ScanNet++						MatterPort3D
	Class-agnostic			Semantic			Class-agnostic			Semantic			Class-agnostic			Semantic
	AP	AP 50	AP 25	AP	AP 50	AP 25	AP	AP 50	AP 25	AP	AP 50	AP 25	AP	AP 50	AP 25	AP	AP 50	AP 25
semantic	14.0	26.5	44.3	16.0	29.0	42.6	13.7	26.1	43.2	9.1	15.8	25.5	9.1	18.8	32.4	11.5	24.4	38.6
grid	22.4	37.6	55.1	13.0	21.2	28.9	24.7	39.1	52.0	6.6	10.0	12.5	8.7	19.1	39.7	6.4	14.4	27.0
semantic + grid	27.1	42.0	55.3	20.6	33.7	44.6	27.4	41.1	53.2	13.5	21.3	29.1	12.5	25.4	42.7	12.3	25.8	40.0

TABLE VII : Ablation Study on two kinds of 2D image segmentation prompts in Local Multi-View Coarse-Grained 3D Segmentation.

MLS	ScanNet200	ScanNet++	MatterPort3D
Normal	0.041	0.234	0.826
Detection-based	0.038 ( $\downarrow 7.3\%$ )	0.196 ( $\downarrow 16.2\%$ )	0.423 ( $\downarrow 48.8\%$ )

TABLE VIII : Ablation study on mask localization strategies (MLS) in local-to-historical streaming segmentation. The reported results are average time cost for each RGB-D frame (ms).

	door	table	cabinet	ceiling lamp	curtain	chair	blinds	storage cabinet	bookshelf	office chair	window	whiteboard	monitor	shelf	window frame	pipe	box	kitchen cabinet	Avg.
MaskClustering	0.0	6.5	0.0	0.8	0.0	22.2	0.0	0.0	16.0	1.3	0.0	22.2	0.0	0.0	0.0	0.0	0.0	0.0	3.5
Stream3D	1.4	3.1	0.0	0.0	11.1	27.6	0.0	0.0	22.2	4.3	1.7	22.2	0.0	0.0	0.0	0.0	0.0	0.0	4.7
Stream3Dv2	56.2	26.4	17.0	48.8	30.3	52.0	32.2	8.9	53.5	38.9	10.5	45.3	39.8	23.4	8.3	15.3	18.1	13.9	26.9
Improve. $\uparrow$	54.8	19.9	17.0	48.0	19.2	24.4	32.2	8.9	31.3	34.6	8.8	23.1	39.8	23.4	8.3	15.3	18.1	13.9	22.2

TABLE IX : Per-class segmentation performance comparison (semantic AP 50 across the first 20 classes) on ScanNet++ benchmark.

	door	picture	window	chair	pillow	lamp	cabinet	curtain	table	plant	mirror	towel	sink	shelves	sofa	bed	night stand	toilet	column	banister	Avg.
MaskClustering	9.3	2.9	5.2	24.9	16.7	4.3	0.0	14.1	3.3	14.8	3.1	33.3	15.6	0.0	5.4	22.5	8.9	11.2	29.5	12.8	11.9
Stream3D	10.1	3.2	7.7	26.0	15.8	6.7	8.9	9.7	3.2	19.4	5.7	24.7	8.6	0.0	11.5	31.7	15.3	38.6	27.4	0.0	13.7
Stream3Dv2	19.6	15.3	27.0	26.3	17.2	53.4	18.9	56.5	7.3	50.1	5.8	36.1	28.1	2.4	30.9	56.6	27.3	62.2	51.1	50.8	32.1
Improve. $\uparrow$	9.5	12.1	19.3	0.3	0.5	46.7	10.0	42.4	4.0	30.7	0.1	2.8	12.5	2.4	19.4	24.9	12.0	23.6	21.6	38.0	18.4

TABLE X : Per-class segmentation performance comparison (semantic AP 50 across the first 20 classes) on MatterPort3D benchmark.

Fig. 10 : More 3D object detection cases on ScanNet200 benchmark.

Panel of Figure 21 — Fig. 11 : Parameter analysis of the number of local frames $k$ .

Panel of Figure 22 — Fig. 12 : Parameter analysis of the manifold distance $\delta$ .

Panel of Figure 23 — Fig. 13 : Parameter analysis of the overlap threshold $\alpha$ .

3D Scene Understanding

3D scene understanding builds geometric and semantic representations of environments for spatial reasoning, perception, and interaction.

See all 293 papers

EPS3D: End-to-End Feed-Forward 3D Panoptic Segmentation
Jun 8, 2026Runsong Zhu, Jiaxin Guo, Xiaoyang Guo +9Panoptic Segmentation Open-Vocabulary 3D Segmentation
The Chinese University of Hong Kong, HK SAR, China · Hong Kong Centre for Logistics Robotics, HK SAR, China · Horizon Robotics, China +2
Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors
Jun 5, 2026Hanxun Yu, Xuan Qu, Lei Ke +43D Spatial Reasoning 3D VQA
1Zhejiang University · 2Tencent Hunyuan · 3HKUST +1
EmbodiedSplat: Online Feed-Forward Semantic 3DGS for Open-Vocabulary 3D Scene Understanding
Mar 4, 2026Seungjun Lee, Zihan Wang, Yunsong Wang +13D Scene Understanding Language-Embedded 3DGS
National University of Singapore

Jun 8, 2026J/K move · Enter open · S save

EPS3D: End-to-End Feed-Forward 3D Panoptic Segmentation

Runsong Zhu, Jiaxin Guo, Xiaoyang Guo, Zhengzhe Liu+8

The Chinese University of Hong Kong, HK SAR, China · Hong Kong Centre for Logistics Robotics, HK SAR, China · Horizon Robotics, China+2

Panoptic Segmentation Open-Vocabulary 3D Segmentation 3D Semantic Segmentation 3D Instance Segmentation

Open paper AskPDF arXiv

This paper introduces EPS3D, a new end-to-end feed-forward framework for open-vocabulary 3D panoptic segmentation. Unlike existing methods relying on additional preprocessing, we design an end-to-end architecture, with a distillation-based training strategy on diverse 3D scenes to predict 3D-aware semantic and instance features from multi-view images, improving 3D consistency and avoiding error accumulation. We further propose a mutual enhancement module to enforce inherent semantic-instance consistency. By aligning semantics within instances (Ins2Sem) and refining instance features with semantic guidance (Sem2Ins), we achieve more coherent 3D scene understanding. Ultimately, EPS3D outperforms SOTA baselines on two benchmarks (e.g., +13% mIoU for semantics on Replica) with high efficiency (e.g., 1s per scene), supporting tasks like robotic manipulation and 3D scene editing.

Abstract

3D Scene Understanding

Explore similar work

EPS3D: End-to-End Feed-Forward 3D Panoptic Segmentation

Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors

EmbodiedSplat: Online Feed-Forward Semantic 3DGS for Open-Vocabulary 3D Scene Understanding