3D scene graph generation requires understanding both object semantics and metric spatial relations. However, existing open-vocabulary methods often struggle with spatial predicates because language similarity reflects semantic co-occurrence rather than explicit geometry. We propose GHOST, a Geometry-constrained Hybrid diffusion framework for Open-vocabulary 3D Scene graph generaTion. GHOST jointly models continuous node geometry and discrete edge predicates, following a geometry-to-constraint-to-predicate denoising process to ensure geometrically consistent relation prediction. It introduces language-grounded constraint induction, which maps predicate embeddings to differentiable geometric constraints that guide both layout refinement and predicate reweighting. A Geometry-Topology Collaborative Denoiser further couples node and edge reasoning through noise-adaptive interaction and orientation-aware modeling. Experiments on 3DSSG demonstrate state-of-the-art performance in both closed-set and open-vocabulary settings, including over 17-point gains over RelationField on spatial predicates.
Figures & tables
Figure 1. VLM-based open-vocabulary 3DSGG methods perform about 30 points worse on spatial predicates than on non-spatial ones (a), because they rely on language co-occurrence priors rather than true 3D layout, e.g., a model may predict “pillow on sofa” simply because the relation is common, while ignoring the actual scene layout (b). GHOST replaces such priors with explicit geometric constraints, substantially narrowing this gap (c).
Figure 2. Overview of GHOST. GHOST jointly denoises object geometry and relation predicates through a hybrid reverse diffusion process. At each step, the Geometry-Topology Collaborative Denoiser (GTCD) first refines object geometry, then applies language-grounded constraint guidance to enforce geometric consistency, and finally decodes predicate tokens conditioned on the corrected layout. This denoise → project → decode ordering ensures that the recovered scene graph is both spatially coherent and open-vocabulary capable.
Method
Pred. Cls.
Triplets
R/mR@3
R/mR@5
R/mR@50
R/mR@100
Closed-set Setting
3DSSG ( Wald et al., 2020 )
0.89/0.54
0.93/0.56
0.40/0.26
0.66/0.42
SGFN ( Wu et al., 2021 )
0.97/0.68
0.99/0.69
0.85/0.51
0.87/0.55
Lang3DSG ( Koch et al., 2024a )
0.96/0.62
0.99/0.65
0.87/0.52
0.89/0.53
SGRec3D ( Koch et al., 2024b )
0.96/0.58
0.99/0.59
0.89/0.54
0.91/0.56
Table 1. Predicate classification and relationship triplet detection on 3DSSG under closed-set and open-vocabulary (OV) protocols. † Results reproduced from released code. “–” Not reported / no public implementation. Bold = best; underline = second best per section.
Method
Spatial
Non-Spatial
R/mR@1
R/mR@3
R/mR@5
R/mR@1
R/mR@3
R/mR@5
CG ( Gu et al., 2024 )
0.08/0.04
0.31/0.17
0.35/0.19
0.26/0.14
0.70/0.43
0.75/0.45
O3DSG ( Koch et al., 2024c )
0.11/0.06
0.35/0.20
0.41/0.24
0.29/0.17
0.72/0.45
0.77/0.49
SRGPT ( Cheng et al., 2024 )
0.22/0.14
0.50/0.32
0.55/0.36
0.39/0.26
0.81/0.54
0.86/0.60
RF ( Koch et al., 2025 )
0.26/0.17
0.53/0.36
0.60/0.40
0.42 / 0.29
0.83/0.57
0.90/0.61
GHOST
0.43 / 0.30
0.84 / 0.60
0.93 / 0.65
0.41/0.28
0.83 / 0.58
0.92 / 0.63
Table 2. Spatial vs. non-spatial predicate recall (OV setting).
Method
Seen Predicate
Unseen Predicate
R/mR@3
R/mR@5
R/mR@3
R/mR@5
ConceptGraphs ( Gu et al., 2024 )
0.65/0.36
0.69/0.38
0.50/0.27
0.55/0.31
Open3DSG ( Koch et al., 2024c )
0.67/0.40
0.72/0.43
0.56/0.33
0.61/0.36
SpatialRGPT ( Cheng et al., 2024 )
0.77/0.50
0.80/0.54
0.67/0.44
0.74/0.48
RelationField ( Koch et al., 2025 )
0.79 / 0.54
0.85 / 0.56
0.73 / 0.46
0.78 / 0.50
GHOST (Ours)
0.85/0.60
0.94/0.67
0.81/0.55
0.87/0.63
Table 3. Seen vs. unseen predicate recall (OV setting).
Figure 3. Accuracy–efficiency trade-off (predicate R@5). All GHOST variants share the same weights and environment and differ only in reverse-step count T . ReLaGS is excluded due to the unavailable source code.
Variants
Predicate
Spatial Pred.
R@3
R@5
R@1
R@5
Input configuration
w/ vision tokens
0.84
0.93
0.41
0.90
w/o geometry tokens
0.61
0.64
0.12
0.32
Annealing schedule
Pred. → Geo. order
0.60
0.63
0.13
0.33
Table 4. Ablation of GHOST’s components at the OV setting. Each variant removes or replaces a single component while keeping all others fixed.
Figure 4. Qualitative OV scene graph predictions on 3DSSG ( Wald et al., 2020 ) . Top: comparison between RelationField and GHOST. Bottom: a GHOST failure case caused by an object recognition error from the upstream detector. Green = correct; red = error.
Figure 5. Geometric Oracle Experiments. Even a perfect geometric oracle applied post-hoc yields only modest spatial gains over the one-shot baseline.