cs.ROAug 28, 2026

Generation of High-Level Concepts in 3D Scene Graphs via Autoregressive Diffusion

Authors: Jose Andres Millan-Romera, Samuel Cognolato, Holger Voos, Jose Luis Sanchez-Lopez, Luciano Serafini

Organizations: Automation and Robotics Research Group, SnT University of Luxembourg, Luxembourg · University of Padova, Padova, Italy · Fondazione Bruno Kessler, Trento, Italy · Automation and Robotics Research Group, SnT, and Faculty of Science, Technology and Medicine, University of Luxembourg, Luxembourg

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Indoor 3D Scene Graphs (3DSGs) represent environments as multi-layer hierarchies that connect observed geometric primitives (e.g., planes) to higher-level metric-semantic concepts (e.g., rooms, floors, buildings), enabling incremental spatial reasoning for robotic perception and SLAM. However, classical high-level concept generation approaches rely on hand-crafted rules for specific concept classes, while learning-based methods require separate models for graph structure and spatial node features (e.g., centroids), which limits scalability to novel classes and more complex hierarchies. We propose a unified autoregressive diffusion-based graph generative model that jointly learns structure and features, constructing complete 3DSGs bottom-up from observed vertical planes across arbitrary hierarchy depths. Our method consistently surpasses all learning-based and random baselines across 3DSG datasets spanning synthetic scenes, real architectural floor plans, and robotic sensor data, with varying layout complexity and hierarchy depth, and surpasses a one-shot model with oracle access to the target graph size on the largest hierarchy and on real single-floor data. Finally, we propose an adaptation of the Fused Gromov--Wasserstein distance for principled graph-level evaluation of generated 3DSGs against ground truth.

Figures & tables

Figure 1: Hierarchical 3D Scene Graph Generation. Starting from wall planes observed by the robot (blue lines), levels are generated bottom-up, with nodes colored by class and placed at their centroids. Grey edges denote part-of relations.
Figure 2: Plane nodes from robot mapping are preprocessed into a proximity graph GPG_{\mathcal{P}} (left), from which the 3DSG is grown iteratively (right). At generation step tt , Insert selects the size mtm_{t} of the next block added to Gt−1G_{t-1} ; Fill jointly denoises its classes, edges, and 3D positions from Zts=SZ^{s=S}_{t} to Zts=0Z^{s=0}_{t} ; and Halt predicts HtH_{t} , ending generation at Gt=TG_{t=T} .
SourceSyntheticMSDReal
Highest levelFloorFloorCityFloorBuildingCityFloorBuildingCity
DatasetS-F- □\square ( □\square rooms)S-FS-CM-FM-BM-CR-FR-BR-C
# graphs2,4002,9992,9993,4662,4161,084411
∣V∣|{\mathcal{V}}|23301109515922345135172
∣E∣|{\mathcal{E}}|26.234.8130.5112.3186.8258.646.0145.0185.0
Degree2.282.342.382.362.352.322.042.132.11
Table 1: Datasets. Each level includes all the levels below it: floor-level graphs contain walls, rooms and floors, building-level graphs add buildings, and city-level graphs add cities. Per-graph averages.
Source: Synthetic
MethodS-F- □\squareS-FS-C
FGW ↓\downarrowGW ↓\downarrowW ↓\downarrowDeg. ↓\downarrowSpec. ↓\downarrowClust. ↓\downarrowGIN ↓\downarrowFGW ↓\downarrowGW ↓\downarrowW ↓\downarrowDeg. ↓\downarrowSpec. ↓\downarrowClust. ↓\downarrowGIN ↓\downarrowFGW ↓\downarrowGW ↓\downarrowW ↓\downarrowDeg. ↓\downarrowSpec. ↓\downarrowClust. ↓\downarrowGIN ↓\downarrow
EC- □\square12.0 ±\pm 1.017.0 ±\pm 1.08.0 ±\pm 0.018.0 ±\pm 1.012.0 ±\pm 1.00.0 ±\pm 0.040.0 ±\pm 10.013.0 ±\pm 1.017.0 ±\pm 1.08.0 ±\pm 1.019.0 ±\pm 2.012.0 ±\pm 1.00.0 ±\pm 0.036.0 ±\pm 9.0-------
EC-L8.0 ±\pm 0.07.0 ±\pm 0.09.0 ±\pm 0.025.0 ±\pm 2.08.0 ±\pm 1.00.0 ±\pm 0.016.0 ±\pm 1.08.0 ±\pm 0.07.0 ±\pm 0.09.0 ±\pm 0.026.0 ±\pm 2.08.0 ±\pm 1.00.0 ±\pm 0.020.0 ±\pm 2.0-------
MiDi+Or3.4 ±\pm 0.10.1 ±\pm 0.26.7 ±\pm 0.10.2 ±\pm 0.10.5 ±\pm 0.10.0 ±\pm 0.00.2 ±\pm 0.03.3 ±\pm 0.00.0 ±\pm 0.06.5 ±\pm 0.00.4 ±\pm 0.10.5 ±\pm 0.10.0 ±\pm 0.00.4 ±\pm 0.06.0 ±\pm 0.37.3 ±\pm 1.34.8 ±\pm 0.833.3 ±\pm 25.86.8 ±\pm 1.81.4 ±\pm 1.012.7 ±\pm 10.5
Ours0.8 ±\pm 0.10.7 ±\pm 0.11.0 ±\pm 0.10.3 ±\pm 0.11.0 ±\pm 0.10.0 ±\pm 0.00.3 ±\pm 0.01.1 ±\pm 0.11.0 ±\pm 0.11.3 ±\pm 0.10.4 ±\pm 0.00.6 ±\pm 0.00.0 ±\pm 0.00.4 ±\pm 0.03.0 ±\pm 0.53.2 ±\pm 0.82.8 ±\pm 0.32.8 ±\pm 0.81.6 ±\pm 0.10.1 ±\pm 0.22.0 ±\pm 1.1
Table 2: Evaluation results for synthetic and MSD datasets. Values are multiplied by 10210^{2} and reported as mean ±\pm standard deviation over three seeds. Missing values (-) occur when the dataset contains hierarchy levels out of the scope of the baseline. Best results are highlighted in light gray and bold; second-best results in lighter gray.
MethodS-C [ ×10−2\times 10^{-2} ]
FGW ↓\downarrowGW ↓\downarrowW ↓\downarrowDeg. ↓\downarrowSpec. ↓\downarrowClust. ↓\downarrowGIN ↓\downarrow
w/o noisy blocks6.897.686.0925.525.710.004.35
w/o noise aug.12.2811.2813.2844.3316.111.243.75
Ours-FD4 filler5.785.765.8027.847.180.4111.64
Ours6.106.865.3418.734.890.053.11
Table 3: Results in real environments.
Figure 3: Qualitative results. Selection of the most complex datasets at the three hierarchy levels explored, comparing our model with the best-performing baselines. Full comparison in the Appendix F .
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

HyperparameterValue
Ours — Insertion and Halting models
ArchitectureGINE, 5 layers, 128 hidden
Ours — node removal process
Node ordering σ\sigmascene-graph proximity permutation
Node schedulecategorical removals, class order (0,4,5,3,2,1)(0,4,5,3,2,1)
Block sizes D\mathcal{D}{1,4}\{1,4\}
Appendix
Table 5: Hyperparameters of Ours (FLAGG with a MiDi-based Fill module) and of the MiDi+Or baseline, identical for all datasets. Presets: preset/final/midi2/base.yaml (Ours) and preset/final/midi/base.yaml (MiDi+Or), relative to config/ .
Figure 4: FD4 variant in the Fill module
 GenerationArchitecture
MethodRWF,B,CCentroidsGraph sizeECClustering1-shotAutoreg.Diff.
EC- □\square✓ □\square✓✗✗✓✓✓✓✗✗
EC-L✓✓✗✓✓✓✓✓✗✗
MiDi+Or✓✓✓✓✗✗✗✓✗✓
Ours✓✓✓✓✓✗✗✗✓✓
Appendix
Table 6: Method comparison. The EC baselines are limited to wall/room generation, whereas MiDi+Or , and Ours generate the full hierarchy (floor, building, city).
Source: Synthetic
MethodS-F- □\squareS-FS-C
FGW ↓\downarrowGW ↓\downarrowW ↓\downarrowDeg. ↓\downarrowSpec. ↓\downarrowClust. ↓\downarrowGIN ↓\downarrowFGW ↓\downarrowGW ↓\downarrowW ↓\downarrowDeg. ↓\downarrowSpec. ↓\downarrowClust. ↓\downarrowGIN ↓\downarrowFGW ↓\downarrowGW ↓\downarrowW ↓\downarrowDeg. ↓\downarrowSpec. ↓\downarrowClust. ↓\downarrowGIN ↓\downarrow
Rand65.0 ±\pm 0.050.0 ±\pm 0.080.0 ±\pm 0.084.0 ±\pm 3.055.0 ±\pm 2.0141.0 ±\pm 4.069.0 ±\pm 3.069.0 ±\pm 2.054.0 ±\pm 2.083.0 ±\pm 3.0104.0 ±\pm 1.073.0 ±\pm 1.0149.0 ±\pm 3.074.0 ±\pm 4.065.0 ±\pm 1.055.0 ±\pm 1.075.0 ±\pm 1.0143.0 ±\pm 0.0123.0 ±\pm 2.0160.0 ±\pm 1.095.0 ±\pm 2.0
Source: MSD
MethodM-FM-BM-C
FGW ↓\downarrowGW ↓\downarrowW ↓\downarrowDeg. ↓\downarrowSpec. ↓\downarrowClust. ↓\downarrowGIN ↓\downarrowFGW ↓\downarrowGW ↓\downarrowW ↓\downarrowDeg. ↓\downarrowSpec. ↓\downarrowClust. ↓\downarrowGIN ↓\downarrowFGW ↓\downarrowGW ↓\downarrowW ↓\downarrowDeg. ↓\downarrowSpec. ↓\downarrowClust. ↓\downarrowGIN ↓\downarrow
Appendix
Table 7: Evaluation results for Rand algorithm in synthetic and MSD datasets. Values are reported ×102\times 10^{2} as mean ±\pm standard deviation over three seeds.
Figure 5: Qualitative results across all training datasets and methods.
Figure 6: Qualitative results across all real datasets.
 M-FM-BM-C
MethodTime/graph (s) ↓\downarrowMem. (GiB) ↓\downarrowTime/graph (s) ↓\downarrowMem. (GiB) ↓\downarrowTime/graph (s) ↓\downarrowMem. (GiB) ↓\downarrow
MiDi+Or20.7 ±\pm 0.81.43 ±\pm 0.0038.7 ±\pm 0.54.10 ±\pm 0.0061.4 ±\pm 0.53.49 ±\pm 0.00
Ours27.8 ±\pm 1.12.63 ±\pm 0.0266.8 ±\pm 0.96.92 ±\pm 0.04118.8 ±\pm 0.45.97 ±\pm 0.03
Appendix
Table 9: Sampling cost on the M- datasets. Wall-clock time per generated graph and peak GPU memory while generating from the test split (128 / 200 / 128 graphs), on one NVIDIA V100-32GB, mean ±\pm std over 3 seeds. Each method uses its own sampling batch size: Ours generates the whole split in one batch, MiDi+Or in adaptive batches of a few graphs, so peak memory is per configuration, not per graph.

Explore similar work

CardsList

  1. Generation of Uncertainty-Aware High-Level Spatial Concepts in Factorized 3D Scene Graphs via Graph Neural Networks

    Sep 18, 2024Jose Andres Millan-Romera, Muhammad Shaheer, Miguel Fernandez-Cortizas +3Simultaneous Localization and Mapping3D SGG

    Automation and Robotics Research Group, Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg · University of Amsterdam

  2. OP3DSG: Open-Vocabulary Part-Aware 3D Scene Graph Generation for Real-World Environments

    Jun 29, 2026Yirum Kim, Ue-Hwan Kim3D SGGRobotic Perception

    Gwangju Institute of Science and Technology (GIST), Gwangju, Republic of Korea

  3. DeWorldSG: Depth-Aware 3D Semantic Scene Graph Generation via World-Model Priors

    Jul 1, 2026Seok-Young Kim, Abdelrahman Elskhawy, Taewook Ha +4Scene Graph Generation3D SGG

    Korea Advanced Institute of Science and Technology · Technical University of Munich · Munich Center for Machine Learning (MCML) +1