Organizations: Shanghai Jiao Tong University, China · University of Cambridge, England · University of Technology Nuremberg, Germany · Nanyang Technological University, Singapore · Department of Automation, Key Laboratory of System Control and Information Processing of Ministry of Education, State Key Laboratory of Avionics Integration and Aviation System-of-Systems Synthesis, Shanghai Key Laboratory of Navigation and Location Based Services, Shanghai Jiao Tong University, Shanghai, China
Open-vocabulary scene understanding is crucial for robotic applications, enabling robots to comprehend complex 3D environmental contexts and supporting various downstream tasks such as navigation and manipulation. However, existing methods require pre-built complete 3D semantic maps to construct scene graphs for scene understanding, which limits their applicability in robotic scenarios where environments are explored incrementally. To address this challenge, we propose OGScene3D, an open-vocabulary scene understanding system that achieves accurate 3D semantic mapping and scene graph construction incrementally. Our system employs a confidence-based Gaussian semantic representation that jointly models semantic predictions and their reliability, enabling robust scene modeling. Building on this representation, we introduce a hierarchical 3D semantic optimization strategy that achieves semantic consistency through local correspondence establishment and global refinement, thereby constructing globally consistent semantic maps. Moreover, we design a long-term global optimization method that leverages temporal memory of historical observations to enhance semantic predictions. By integrating 2D-3D semantic consistency with Gaussian rendering contribution, this method continuously refines the semantic understanding of the entire scene. Furthermore, we develop a progressive graph construction approach that dynamically creates and updates both nodes and semantic relationships, allowing continuous updating of the 3D scene graphs. Extensive experiments on widely used datasets and real-world scenes demonstrate the effectiveness of our OGScene3D on open-vocabulary scene understanding.
Figures & tables
Fig. 1: Our OGScene3D achieves incremental open-vocabulary scene understanding by continuously updating scene graphs and semantic Gaussian maps.
Fig. 2: An overview of OGScene3D. Given sequential RGB-D images, our method achieves open-vocabulary semantic mapping through hierarchical 3D semantic optimization and long-term global optimization. At the same time, our method enables incremental scene understanding through progressive scene construction.
Fig. 3: Local 3D Optimization. When a new keyframe Ij arrives, new 3D Gaussians are initialized from its segmented regions using depth information. To maintain semantic consistency with the existing scene representation, KNN correspondence is established between new Gaussian segments and existing scene Gaussians. Each new Gaussian segment is then evaluated using semantic ambiguity ratio r(m) and spatial-semantic coherence p(m) to determine label assignment. Finally, Gaussian semantic confidence is updated using spatial consistency measure and semantic confidence.
Fig. 4: Designed prompts for progressive graph construction, including caption prompt Pcaption , tag prompt Ptag , and edge prompt Pedge .
Fig. 5: Qualitative comparison of semantic segmentation on Replica [ 15 ] . Details are highlighted with colorful boxes. Our method achieves more precise semantic representation than baselines.
Fig. 6: Qualitative comparison of semantic segmentation on real-world dataset ScanNet [ 16 ] . Details are highlighted with colorful boxes. Our method achieves more precise semantic mapping than baselines.
Methods
Metrics
room0
room1
room2
office0
office1
office2
office3
office4
Avg.
Feature 3DGS [ 57 ]
mIoU
49.62
47.71
53.00
51.37
44.94
49.49
48.93
46.06
48.89
mAcc
59.46
55.37
62.21
59.42
52.19
58.25
57.84
55.36
57.51
GS-Grouping [ 58 ]
mIoU
60.55
57.68
62.82
63.42
55.41
59.33
57.91
56.05
59.15
mAcc
69.02
67.83
71.41
73.61
66.78
71.51
70.29
69.06
69.94
OpenGS-SLAM [ 10 ]
mIoU
64.06
61.18
64.38
61.53
58.11
63.05
62.71
60.25
61.91
mAcc
73.53
72.13
74.17
73.05
69.03
75.36
73.62
74.01
73.11
TABLE I: Comparison of zero-shot novel-view Semantic Segmentation.
Methods
Metric
room0
room1
room2
office0
office1
office2
office3
office4
Avg.
ConceptGraphs [ 1 ]
mIoU
12.35
14.04
10.58
20.23
11.61
19.08
9.71
22.04
14.95
F-mIoU
21.20
15.75
27.16
32.20
11.54
25.85
11.64
42.44
23.47
mAcc
27.08
34.68
27.34
31.73
31.49
29.27
26.73
41.22
31.19
HOV-SG [ 2 ]
mIoU
33.74
27.37
18.27
24.81
11.93
20.98
19.95
27.37
23.05
F-mIoU
62.77
40.84
43.78
33.78
17.78
40.51
26.00
43.58
38.63
mAcc
43.57
39.22
26.18
28.17
23.88
24.99
28.46
28.72
30.40
TABLE II: Quantitative comparison of open-vocabulary 3D semantic segmentation on Replica dataset.
Methods
Metric
scene0011_00
scene0050_00
scene0231_00
scene0378_00
scene0518_00
Avg.
ConceptGraphs [ 1 ]
mIoU
–
–
–
–
–
16.00
F-mIoU
–
–
–
–
–
20.00
mAcc
–
–
–
–
–
28.00
HOV-SG [ 2 ]
mIoU
27.80
17.23
22.72
26.67
16.58
22.20
F-mIoU
42.19
36.50
26.39
29.15
17.27
30.30
mAcc
44.59
41.62
40.50
51.28
37.51
43.10
TABLE III: Quantitative comparison of open-vocabulary 3D semantic segmentation on ScanNet dataset.
Fig. 7: Qualitative comparison on 3D semantic mapping. Our method achieves predictive completion, enabling more accurate and comprehensive 3D semantic reconstruction results. Only our method can correctly segment objects such as rug , table and background wall.
Methods
137a8158
10b17940
bf9a3dd3
0988ea72
ConceptGraphs [ 1 ]
2.5
fail
3.5
0.9
OGScene3D (Ours)
28.7
25.6
22.8
18.3
TABLE IV: Quantitative comparison of scene graph for Recall(%) metric on 4 scenes of the 3RScan dataset.
Fig. 8: Visualization of real-world experiments. Using RGB-D data from the Gemini 336 camera as input, our method performs accurate dense RGB mapping and open-set semantic mapping.
Fig. 9: The hardware setup used for real-world semantic mapping and scene understanding experiments.
Methods
Time/scene
mIoU
F-mIoU
mAcc
ConceptGraphs [ 1 ]
122m5s
16.00
20.00
28.00
HOV-SG [ 2 ]
31m56s
22.20
30.30
43.10
OGScene3D (Ours)
10m13s
29.44
36.34
46.41
TABLE V: Runtime comparison on ScanNet dataset. (m) minutes (s) seconds.
Fig. 10: Visualization of ablation study. Semantic rendering results and ground truth labels are shown on Replica dataset [ 15 ] . Blue boxes show degraded results without global 3D refinement. Red boxes demonstrate successful segmentation of sparsely observed objects through long-term global optimization.
Methods
mIoU
F-mIoU
mAcc
(a)
Ours (only Local 3D optimization)
22.98
35.15
42.85
(b)
Ours ( + Global 3D refinement to (a))
25.35
38.16
49.02
(c)
Ours (full, + Long-term global optimization to (b))
36.07
42.22
55.16
TABLE VII: Ablation study of our innovations on Replica dataset.
School of Electrical Engineering and Robotics, Queensland University of Technology (QUT), Brisbane, QLD 4000, Australia · CSIRO Robotics, CSIRO, Brisbane, QLD 4069, Australia