Organizations: German Research Center for Artificial Intelligence (DFKI) · RPTU University Kaiserslautern-Landau · University of Modena and Reggio Emilia
Achieving unified 3D perception and reasoning across tasks such as segmentation, retrieval, and relation understanding remains challenging, as existing methods are either object-centric or rely on costly training for inter-object reasoning. We present a novel framework that constructs a hierarchical language-distilled Gaussian scene and its 3D semantic scene graph without scene-specific training. A Gaussian pruning mechanism refines scene geometry, while a robust multi-view language alignment strategy aggregates noisy 2D features into accurate 3D object embeddings. On top of this hierarchy, we build an open-vocabulary 3D scene graph with Vision Language derived annotations and Graph Neural Network-based relational reasoning. Our approach enables efficient and scalable open-vocabulary 3D reasoning by jointly modeling hierarchical semantics and inter/intra-object relationships, validated across tasks including open-vocabulary segmentation, scene graph generation, and relation-guided retrieval. Project page: https://dfki-av.github.io/ReLaGS/
Figures & tables
Figure 2: ReLaGS Overview . Given a reconstructed Gaussian scene, redundant primitives are first pruned to improve geometric accuracy. Heuristic clustering under multi-level SAM supervision then forms a hierarchical scene structure, where each cluster is assigned a CLIP-based language feature with outlier rejection. Finally, open-vocabulary inter- and intra-object scene graphs are obtained either by lifting LLM-derived relations for semantic diversity or by using a pretrained graph network for efficient offline inference.
Figure 3: Illustration of proposed two improvement methods for hierarchical scene construction and two example scene graphs. (a): Low contribution Gaussian points (red) are removed to improve scene geometry. (b): Outlier features (e.g., due to occluded or inconsistent masks) are filtered before aggregation, producing a more coherent and consistent embedding for the target. (c): The spatial relationships are predicted by our GNN. (d): The more semantic-enriched relationship lifted with LLM, the root object is marked as -1.
Method
Object
Predicate
Scene agno.
R@5
R@10
R@3
R@5
ConceptGraphs [ 10 ]
0.37
0.46
0.74
0.79
✗
RelationField [ 19 ]
0.69
0.80
0.76
0.82
✗
Ours (VLM)
0.68
0.79
0.10
0.35
✗
Open3DSG [ 18 ]
0.56
0.61
0.58
0.65
✓
Ours (pred.)
0.68
0.79
0.79
0.87
✓
Table 1: Results of 3D scene graph prediction on 3DSSG [ 37 ] . “Scene agnostic” denotes methods without per-scene training.
Figure 4: Qualitative results of open vocabulary object segmentation. We show results on LERF dataset for segmentation mask on 2D view. With multi-hierarchy querying search and 3D scene graph for relation guidance, our method shows strong improvement against THGS.
Method
mIoU
Scene agno.
Lerf [ 16 ]
0.25
✗
OpenNeRF [ 8 ]
0.45
✗
LangSplat [ 30 ]
0.49
✗
RelationField [ 19 ]
0.53
✗
THGS [ 7 ]
0.29
✓
Ours w/o MH
0.54
✓
Table 2: Relationship-Guided 3D Instance Segmentation on ScanNet++ [ 50 ] . Our method achieves the highest mean IoU, even operating only on single hierarchy (w/o MH).
Method
LERF-OVS mIoU (%)
Mean
T-F
Fig.
Ramen
Teatime
Waldo
LangSplatV2 [ 23 ]
56.4
51.4
72.2
59.1
59.9
✗
LAGA [ 3 ]
64.1
55.6
70.9
65.6
64.0
✗
THGS [ 7 ]
57.3
43.5
68.3
50.7
54.9
✓
Occam’s [ 5 ]
58.6
51.0
70.2
65.3
61.3
✓
VALA [ 39 ]
59.9
51.5
70.2
65.1
61.7
✓
Table 3: Open-vocabulary segmentation on LERF-OVS [ 16 ] . Our method achieves the highest mean IoU among both training and training-free approaches.
Methods
19 cls
15 cls
10 cls
T-F
mIoU
mAcc
mIoU
mAcc
mIoU
mAcc
LangSplat [ 30 ]
3.78
9.11
5.35
13.20
8.40
22.06
✗
OpenGau. [ 46 ]
24.73
41.54
30.13
48.25
38.29
55.19
✗
LAGA [ 3 ]
32.50
49.10
35.50
53.50
42.60
63.20
✗
THGS [ 7 ]
34.39
50.74
39.61
57.07
46.38
64.74
✓
Occam’s [ 5 ]
31.93
48.93
34.25
53.71
45.16
64.39
✓
Table 4: Open-vocabulary semantic segmentation on ScanNet [ 6 ] . Our method achieves the highest mean performance among both training and training-free approaches on 15 and 10 classes subset.
Variant
Figurines
Ramen
Teatime
Kitchen
Mean
Base
52.05
47.19
76.77
47.5
55.88
Base+ M.
59.16
47.41
80.98
60.59
62.04
Base+ M. + R. (Full)
64.69
51.15
80.98
60.6
64.36
Table 5: Ablation on LeRF-OVS dataset for open-vocabulary segmentation. M. is Maximum Weight Pruning and R. is Robust Outlier-Aware Feature Aggregation .
τlang
Figurines
Ramen
Teatime
Waldo Kitchen
Mean
1
61.85
43.3
76.02
63.37
60.78
2
63.39
51.07
73.8
60.62
62.22
3
64.69
51.15
80.98
60.60
64.36
4
63.3
47.4
80.98
60.60
63.07
Table 6: Ablation on different values of τlang on LeRF-OVS.
Method
Time (min) / Storage (MB) / GPU Memory (GB)
Scene Rec.
Language Distill.
Scene Graph
Overall
RF [ 19 ]
60/500/32 (inseparable)
60/500/32
Ours
11/52.2/3.2
1.5/10.6/7.5
0.1/2.2/3.0
12.6/65/7.5
Table 7: Runtime and resource comparison against RelationField [ 19 ] on 3DSSG scenes. Our method is 4.7 × faster and 7.6 × more memory efficient and a maximum of 7.5GB of GPU memory.
Method
Object mIoU
Part mIoU
THGS
63.2
45
Ours
65.3
60.3
Table 8: Hierarchical query evaluation on LERF. We manually categorize LERF queries into object-level and part-level queries and report mIoU for each category. Our method improves performance across both groups, with particularly large gains on part-level queries, demonstrating the ability of the proposed representation to support fine-grained hierarchical language queries.
Method
19 Classes
15 Classes
10 Classes
mIoU
mAcc
mIoU
mAcc
mIoU
mAcc
THGS
39.33
54.23
43.61
61.55
52.52
70.50
Ours
41.26
57.29
45.24
62.75
54.04
71.48
Table 9: ScanNet Results with Revised Evaluation Protocol. Quantitative results on ScanNet using densified Gaussian reconstruction and nearest-point assignment (L2). THGS* refers to the THGS method re-run by us on all scenes under this protocol. Our method consistently outperforms THGS* across all class subsets.
Variants
Object
Predicate
R@5
R@10
R@3
R@5
Base
0.57
0.69
0.75
0.85
Base+ M.
0.65
0.75
0.77
0.87
Base+ M. + R.
0.68
0.79
0.79
0.87
Table 10: Ablation on 3DSSG dataset for 3D scene graph prediction. M. is Maximum Weight Pruning and R. is Robust Outlier-Aware Feature Aggregation .
Variants
Object
Predicate
R@5
R@10
R@3
R@5
L=1, n↦e
0.68
0.79
0.79
0.87
L=2, n↦e
0.68
0.79
0.77
0.84
L=1, 0↦e
0.68
0.79
0.75
0.88
Table 11: Ablation on 3DSSG dataset for 3D scene graph prediction. L is the number of transformer layer, and 0↦e means using zero vector instead of using node feature pair to initialize edge feature n↦e .
τcontrib
mIoU ↑
SSIM ↑
PSNR ↑
LPIPS ↓
-
57.84
0.8478
23.576
0.2492
5×10−4
64.36
0.8478
23.576
0.2492
5×10−3
59.27
0.8478
23.576
0.2492
5×10−2
57.53
0.8462
23.526
0.2516
5×10−1
39.18
0.4967
10.905
0.5185
Table 12: Ablation Study on weight pruning.
Method
3D-OVS mIoU (%)
Mean
Bed
Bench
Lawn
Room
Sofa
LangSplatV2 [ 23 ]
93.0
94.9
96.1
92.3
96.6
94.6
LAGA [ 3 ]
96.8
92.8
97.0
93.0
96.9
95.3
Occam’s [ 5 ]
96.8
94.8
97.0
96.5*
88.8
95.0
THGS [ 7 ] †
68.9
67.6
73.9
68.4*
54.6
66.7
Ours
95.4
94.4
91.2
94.8*
86.9
92.5
Table 13: Open-vocabulary segmentation on 3D-OVS [ 25 ] . Methods with ∗ are evaluated on testing-data that has an annotation error in the 3D-OVS testset corrected. † means we run the open-source code under our environment.
Figure 5: Qualitative results on LERF, ScanNet++, ScanNet and 3D-OVS.
Figure 6: Visualization of multi-hierarchy scene reconstruction on LERF and 3D-OVS datasets.
Figure 7: Examples of our SoM+LLM scene graph annotation on 2D images from 3DSSG dataset.
While 3D Gaussian Splatting (3DGS) has advanced open vocabulary scene understanding, existing methods remain confined to explicit queries. They struggle to interpret implicit intents, complex spatial constraints, and commonsense reasoning required for practical embodied interactions. To address this gap, we introduce the task of reasoning 3D Gaussian segmentation and construct two benchmarks, Causal-LERF and Causal-ScanNet. These benchmarks systematically evaluate commonsense, spatial, affordance, and counterfactual reasoning. Evaluations reveal that current state of the art methods perform poorly on these reasoning challenges. Therefore, we propose CausalSplat, a framework that integrates vision-language models with 3D scene graphs to disentangle explicit structural perception from implicit logical inference. Extensive experiments demonstrate that CausalSplat achieves state of the art performance on our reasoning benchmarks while showing strong generalizability on standard referring and open vocabulary 3D segmentation tasks. Project Page: https://jiayuding031020.github.io/CausalSplat
Jiayu Ding, Meilu Song, Yun Chen +2
Hunan University College of Computer Science and Electronic Engineering Changsha, China
Open-vocabulary 3D scene understanding aims to segment 3D scenes beyond predefined categories by transferring semantic knowledge from vision-language models. Existing methods have advanced this task by lifting language-aligned 2D features into 3D, yet they often rely on context-independent semantic representations, leaving object relationships underexplored for contextual refinement. We propose RelGraphOV, a relationship-aware framework that uses 3D scene graphs to enhance open-vocabulary 3D understanding. Our method constructs relational scene graphs from multi-view observations by leveraging vision-language reasoning to infer object relationships and prune geometrically implausible connections, without manual relationship annotations. To aggregate relational context while avoiding feature interference, we introduce an Adaptive Gated Dual-Stream Contextual GAT that separates dense geometric features and semantic CLIP embeddings, performs edge-guided message passing, and adaptively fuses complementary semantics. A hierarchical contrastive objective further promotes instance-level consistency and category-level discrimination. Experiments on ScanNetV2, ScanNet200, ScanNet++, and Replica demonstrate strong performance and generalization ability. Project Page: https://cxavireh.github.io/relgraphov-projectpage
Xianhao Chen, Jiarui Hu, Yuanbo Yang +5
State Key Lab of CAD&CG, Zhejiang University, China · Zhejiang University, China
Recent advances in 3D reconstruction have progressed from per-scene optimization to feed-forward inference, and semantic scene understanding has followed suit -- yet existing methods remain confined to object-centric perception, neglecting spatial relations between objects. We formulate 3D spatial relation segmentation in a feed-forward, pose-free multi-view setting: given a visually specified subject and a relational text query, the model segments the target across views without receiving its category name. To this end, we propose RelationVGGT, a novel feed-forward framework that integrates semantic features from a visual foundation model with geometry-aware representations from a 3D geometry foundation model and leverages a relation transformer for subject-conditioned, cross-view relation prediction -- requiring neither per-scene optimization nor known camera poses. We additionally provide a fully automated annotation pipeline built on ScanNet++ with VLMs and LLMs, enabling scalable training data generation for this new task.