Recently, open-vocabulary zero-shot 3D scene understanding using vision foundation models has emerged as a promising alternative to data-intensive supervised methods. However, deploying these models in real-world scenarios is severely hindered by their inability to efficiently handle streaming RGB-D inputs and their inherent vulnerability to noise 2D segmentation masks. To address these critical limitations, we propose Stream3Dv2, a novel training-free framework designed for robust streaming 3D perception. Stream3Dv2 processes sequential data through an original nested local-to-historical architecture, capturing multi-view consistency while circumventing the high computational overhead so as to support timely responses. At its core, we introduce a comprehensive geometric-semantic fusion mechanism that resolves geometric noise and semantic ambiguity by explicitly utilizing semantic guidance and formulating 3D segmentation as solving point-and-set merging and partitioning problems. Furthermore, we present an innovative manifold-distance-based point cloud refinement strategy. This approach leverages local manifold graphs for point-to-manifold optimization that mitigates the boundary delineation failures caused by Euclidean-distance metrics, and employs geometric bounding boxes to dynamically activate and update historical instances for achieving rapid manifold-to-manifold refinement. Extensive experiments on public datasets demonstrate that Stream3Dv2 consistently outperforms existing baselines in foundational open-vocabulary streaming 3D segmentation and detection. Finally, we show that integrating our framework with an LLM-based agent enables advanced language-driven 3D scene understanding, underscoring its potential for open-world embodied intelligence. Code will be updated at https://github.com/SubmissionsIn/Stream3D.
Figures & tables
Fig. 1 : Overview of Stream3Dv2 : (a) Streaming 3D scene understanding for embodied AI applications. (b) We leverage semantic&grid-prompted 2D segmentation and integrate local multi-view and local-to-historical modules to enable streaming zero-shot 3D instance segmentation. Specifically, (c) local multi-view module conducts class-agnostic noise mask filtering and semantic-driven mask merging&partitioning with set covering problem (SCP) and set partitioning problem (SPP) based strategies. Local geometric-semantic fusion produces robust 3D semantic segmentation. Then, (d) local-to-historical module refines 3D masks’ manifolds by assigning point-to-manifold on Riemannian graph metrics, followed by the detection-based efficient 3D instance localization&division to achieve manifold-to-manifold 3D mask updates.
Fig. 2 : Local multi-view coarse-grained 3D segmentation.
Fig. 3 : 3D mask merging and partitioning. Left: threshold-based methods suffer from dynamical overlapping or wrongly-merged masks. Right: semantic-driven method (our work) is able to make more feasible mask merging and partitioning with semantic information.
Method
ST
ZS
ScanNet200
ScanNet++
MatterPort3D
Class-agnostic
Semantic
Class-agnostic
Semantic
Class-agnostic
Semantic
AP
AP 50
AP 25
AP
AP 50
AP 25
AP
AP 50
AP 25
AP
AP 50
AP 25
AP
AP 50
AP 25
AP
AP 50
AP 25
Mask3D [ 2 ]
✗
✗
39.7
53.6
62.5
26.9
36.2
41.4
22.8
33.3
45.7
3.6
5.1
6.7
4.4
9.8
20.6
2.5
4.5
6.7
OpenMask3D [ 16 ]
✗
✗
39.7
53.6
62.5
15.1
19.6
22.6
22.8
33.3
45.7
2.0
2.7
3.4
4.4
9.8
20.6
4.6
8.5
13.0
OVIR-3D [ 23 ]
✗
✓
14.4
27.5
38.8
9.3
18.7
25.0
19.4
34.1
46.5
3.6
5.7
7.3
5.9
13.9
24.6
6.3
16.4
24.4
MaskClustering [ 18 ]
✗
✓
19.7
36.4
51.4
12.0
23.3
30.1
24.6
40.3
51.5
7.8
11.9
13.2
8.3
18.9
33.4
9.2
19.7
26.5
TABLE I : Open-vocabulary 3D instance segmentation on ScanNet200, ScanNet++, and MatterPort3D benchmarks, evaluated by Class-agnostic and Semantic segmentation performance. ST ✓: streaming method. ZS ✓: zero-shot method.
chair
table
door
couch
cabinet
shelf
desk
offi.cha.
bed
pillow
sink
picture
window
toilet
booksh.
monitor
curtain
book
armcha.
coff.tab.
Avg.
MaskClustering
36.1
35.2
0.5
20.5
2.1
8.2
4.8
19.3
26.8
21.7
37.6
2.6
6.3
32.0
2.0
44.3
23.1
2.8
16.1
31.2
18.7
Stream3D
38.8
36.9
0.3
21.9
4.3
9.1
4.1
22.6
25.9
15.3
29.5
1.4
8.4
44.6
2.0
41.9
30.1
3.9
19.0
44.4
20.2
Stream3Dv2
40.9
49.7
50.0
22.5
24.2
9.1
6.2
56.6
60.0
23.7
41.5
35.4
33.7
74.6
17.5
45.7
58.9
24.7
46.3
45.6
38.3
Improve. ↑
2.1
12.8
49.5
0.6
19.9
0.0
1.4
34.0
33.2
2.0
3.9
32.8
25.3
30.0
15.5
1.4
28.8
20.8
27.3
1.2
18.1
TABLE II : Per-class segmentation performance comparison (semantic AP 50 across the first 20 classes) on ScanNet200 benchmark.
Fig. 4 : Visualization of 3D instance segmentation comparison across RGB-D scene, Stream3D, Stream3Dv2, and Ground truth.
Fig. 5 : Visualization of 3D object detection comparison between our Stream3Dv2 and Ground truth on ScanNet200 benchmark.
Method
ST
ZS
ScanNet200
AP
AP 50
AP 25
FPS
EmbodiedSAM [ 37 ]
✓
✗
28.8
42.7
54.2
10
SAM3D [ 22 ]
✓
✓
9.6
24.8
49.6
8
OnlineAnySeg [ 36 ]
✓
✓
18.6
36.1
53.5
15
MoonSeg3R [ 54 ]
✓
✓
16.7
33.3
50.0
15
Stream3D (ours)
✓
✓
22.4
37.6
55.1
13
TABLE III : Class-agnostic streaming 3D instance segmentation on ScanNet200 benchmark. ST ✓: streaming method. ZS ✓: zero-shot method. FPS : frames per second.
Method
ST
ZS
ScanNet200
AP 50
AP 25
OpenM3D [ 19 ]
✗
✗
–
4.2
Zoo3D 1 [ 21 ]
✗
✗
15.2
23.5
Zoo3D 0 [ 21 ]
✗
✓
14.1
21.1
Stream3D (ours)
✓
✓
10.6
18.9
Stream3Dv2 (ours)
✓
✓
14.2
24.8
TABLE IV : Open-vocabulary 3D object detection on ScanNet200 benchmark. ST ✓: streaming method. ZS ✓: zero-shot method.
Fig. 6 : Examples of downstream tasks by Stream3Dv2 equipped with LLM agent. Top : 3D instance grounding, reasoning, and navigation. Bottom : open-vocabulary queries with indirect semantic descriptions.
Fig. 7 : Our method obtains robust 3D segmentation by mitigating 2D noise masks from VFMs, in cases such as dynamically inconsistent masks, class-agnostic fine-grained masks, semantically error masks.
Fig. 8 : The process of our streaming zero-shot 3D instance segmentation on Top : room-level cases and Bottom : large building-level cases.
MVF
SDS
PMR
ScanNet200
ScanNet++
MatterPort3D
Class-agnostic
Semantic
Class-agnostic
Semantic
Class-agnostic
Semantic
AP
AP 50
AP 25
AP
AP 50
AP 25
AP
AP 50
AP 25
AP
AP 50
AP 25
AP
AP 50
AP 25
AP
AP 50
AP 25
(I)
✗
✗
✗
16.6
28.7
45.7
8.0
14.3
21.5
5.0
9.6
19.4
1.7
3.5
5.3
3.0
7.0
18.4
1.6
3.6
10.7
(II)
✓
19.0
32.4
49.7
9.6
16.1
22.6
11.3
22.1
39.8
3.4
6.2
9.8
5.6
13.1
33.3
4.3
11.7
23.2
(III)
✓
20.1
33.9
48.9
17.1
28.9
42.0
18.3
30.8
43.3
12.8
20.6
28.4
10.6
21.3
39.9
10.4
22.5
38.7
(IV)
✓
18.6
32.2
48.3
8.8
16.0
28.5
14.1
23.4
35.1
4.5
6.4
8.0
8.0
17.1
33.9
2.5
4.8
12.6
TABLE V : Ablation study on the main methodologies in Stream3Dv2 paradigm. MVF denotes the multi-view noise mask filtering. SDS is the semantic-driven fine-grained 3D segmentation. PMR is the point cloud manifold refinement.
t=400
t=1200
t=2000
t=2800
Loc 3D seg. (ms)
52
67
43
73
Loc2his 3D seg. (ms)
91
241
237
377
Memory (MB)
10.4
15.6
19.0
21.3
TABLE VI : Time latency during long sequence segmentation on MatterPort3D benchmark (scene id: gxdoqLR6rwA).
Fig. 9 : Parameter analysis on (a) window size k , (b) key point rate γ , (c) geometry distance δ , and (d) overlap threshold α .
Prompt
ScanNet200
ScanNet++
MatterPort3D
Class-agnostic
Semantic
Class-agnostic
Semantic
Class-agnostic
Semantic
AP
AP 50
AP 25
AP
AP 50
AP 25
AP
AP 50
AP 25
AP
AP 50
AP 25
AP
AP 50
AP 25
AP
AP 50
AP 25
semantic
14.0
26.5
44.3
16.0
29.0
42.6
13.7
26.1
43.2
9.1
15.8
25.5
9.1
18.8
32.4
11.5
24.4
38.6
grid
22.4
37.6
55.1
13.0
21.2
28.9
24.7
39.1
52.0
6.6
10.0
12.5
8.7
19.1
39.7
6.4
14.4
27.0
semantic + grid
27.1
42.0
55.3
20.6
33.7
44.6
27.4
41.1
53.2
13.5
21.3
29.1
12.5
25.4
42.7
12.3
25.8
40.0
TABLE VII : Ablation Study on two kinds of 2D image segmentation prompts in Local Multi-View Coarse-Grained 3D Segmentation.
MLS
ScanNet200
ScanNet++
MatterPort3D
Normal
0.041
0.234
0.826
Detection-based
0.038 ( ↓7.3% )
0.196 ( ↓16.2% )
0.423 ( ↓48.8% )
TABLE VIII : Ablation study on mask localization strategies (MLS) in local-to-historical streaming segmentation. The reported results are average time cost for each RGB-D frame (ms).
door
table
cabinet
ceiling lamp
curtain
chair
blinds
storage cabinet
bookshelf
office chair
window
whiteboard
monitor
shelf
window frame
pipe
box
heater
kitchen cabinet
storage rack
Avg.
MaskClustering
0.0
6.5
0.0
0.8
0.0
22.2
0.0
0.0
16.0
1.3
0.0
22.2
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
3.5
Stream3D
1.4
3.1
0.0
0.0
11.1
27.6
0.0
0.0
22.2
4.3
1.7
22.2
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
4.7
Stream3Dv2
56.2
26.4
17.0
48.8
30.3
52.0
32.2
8.9
53.5
38.9
10.5
45.3
39.8
23.4
8.3
15.3
18.1
0.0
13.9
0.0
26.9
Improve. ↑
54.8
19.9
17.0
48.0
19.2
24.4
32.2
8.9
31.3
34.6
8.8
23.1
39.8
23.4
8.3
15.3
18.1
0.0
13.9
0.0
22.2
TABLE IX : Per-class segmentation performance comparison (semantic AP 50 across the first 20 classes) on ScanNet++ benchmark.
door
picture
window
chair
pillow
lamp
cabinet
curtain
table
plant
mirror
towel
sink
shelves
sofa
bed
night stand
toilet
column
banister
Avg.
MaskClustering
9.3
2.9
5.2
24.9
16.7
4.3
0.0
14.1
3.3
14.8
3.1
33.3
15.6
0.0
5.4
22.5
8.9
11.2
29.5
12.8
11.9
Stream3D
10.1
3.2
7.7
26.0
15.8
6.7
8.9
9.7
3.2
19.4
5.7
24.7
8.6
0.0
11.5
31.7
15.3
38.6
27.4
0.0
13.7
Stream3Dv2
19.6
15.3
27.0
26.3
17.2
53.4
18.9
56.5
7.3
50.1
5.8
36.1
28.1
2.4
30.9
56.6
27.3
62.2
51.1
50.8
32.1
Improve. ↑
9.5
12.1
19.3
0.3
0.5
46.7
10.0
42.4
4.0
30.7
0.1
2.8
12.5
2.4
19.4
24.9
12.0
23.6
21.6
38.0
18.4
TABLE X : Per-class segmentation performance comparison (semantic AP 50 across the first 20 classes) on MatterPort3D benchmark.
Fig. 10 : More 3D object detection cases on ScanNet200 benchmark.
Fig. 11 : Parameter analysis of the number of local frames k .
Fig. 12 : Parameter analysis of the manifold distance δ .
Fig. 13 : Parameter analysis of the overlap threshold α .
This paper introduces EPS3D, a new end-to-end feed-forward framework for open-vocabulary 3D panoptic segmentation. Unlike existing methods relying on additional preprocessing, we design an end-to-end architecture, with a distillation-based training strategy on diverse 3D scenes to predict 3D-aware semantic and instance features from multi-view images, improving 3D consistency and avoiding error accumulation. We further propose a mutual enhancement module to enforce inherent semantic-instance consistency. By aligning semantics within instances (Ins2Sem) and refining instance features with semantic guidance (Sem2Ins), we achieve more coherent 3D scene understanding. Ultimately, EPS3D outperforms SOTA baselines on two benchmarks (e.g., +13% mIoU for semantics on Replica) with high efficiency (e.g., 1s per scene), supporting tasks like robotic manipulation and 3D scene editing.
Runsong Zhu, Jiaxin Guo, Xiaoyang Guo +9
The Chinese University of Hong Kong, HK SAR, China · Hong Kong Centre for Logistics Robotics, HK SAR, China · Horizon Robotics, China +2
Despite advances in 3D scene understanding, existing 3D Large Multimodal Models operate in offline settings, requiring complete scene observations or predefined video clips. In this paper, we present an online 3D vision-language model that enables real-time spatial understanding from streaming video. Our approach adopts an autoregressive streaming control modeling based on the LLM's next-token prediction objective to learn when to respond, and employs a lightweight Visual-Spatial Feature Integration (VSFI) module to incrementally inject temporally aligned geometry priors into the visual stream. To alleviate long-context decoding overhead, we propose a plug-and-play Geometry-Adaptive Voxel Compression (GAVC) module for efficient visual token compression. To address the scarcity of streaming 3D-language data, we further develop a scalable data generation pipeline that curates over 1M online spatio-temporal 3D QA pairs and establishes a comprehensive benchmark spanning 29 tasks. Extensive experiments show that our approach significantly outperforms both proprietary and open-source models across online and offline 3D spatial understanding, reasoning, and grounding tasks. The project page is available at https://stream3d-vlm.github.io/
Hanxun Yu, Xuan Qu, Lei Ke +4
1Zhejiang University · 2Tencent Hunyuan · 3HKUST +1
Understanding a 3D scene immediately with its exploration is essential for embodied tasks, where an agent must construct and comprehend the 3D scene in an online and nearly real-time manner. In this study, we propose EmbodiedSplat, an online feed-forward 3DGS for open-vocabulary scene understanding that enables simultaneous online 3D reconstruction and 3D semantic understanding from the streaming images. Unlike existing open-vocabulary 3DGS methods which are typically restricted to either offline or per-scene optimization setting, our objectives are two-fold: 1) Reconstructs the semantic-embedded 3DGS of the entire scene from over 300 streaming images in an online manner. 2) Highly generalizable to novel scenes with feed-forward design and supports nearly real-time 3D semantic reconstruction when combined with real-time 2D models. To achieve these objectives, we propose an Online Sparse Coefficients Field with a CLIP Global Codebook where it binds the 2D CLIP embeddings to each 3D Gaussian while minimizing memory consumption and preserving the full semantic generalizability of CLIP. Furthermore, we generate 3D geometric-aware CLIP features by aggregating the partial point cloud of 3DGS through 3D U-Net to compensate the 3D geometric prior to 2D-oriented language embeddings. Extensive experiments on diverse indoor datasets, including ScanNet, ScanNet++, and Replica, demonstrate both the effectiveness and efficiency of our method. Check out our project page in https://0nandon.github.io/EmbodiedSplat/.