Probabilistic Scene Graphs: Hierarchical Representation and Real-time System

Waqas Ali; Michele Antonazzi; Timon Homberger; Thien-Minh Nguyen; Lukas Rosenberger Schmid; Patric Jensfelt; Yixi Cai

cs.ROSep 19, 2026

Probabilistic Scene Graphs: Hierarchical Representation and Real-time System

Authors: Waqas Ali, Michele Antonazzi, Timon Homberger, Thien-Minh Nguyen, Lukas Rosenberger Schmid, Patric Jensfelt, Yixi Cai

Organizations: Division of Robotics, Perception, and Learning (RPL), KTH Royal Institute of Technology, Stockholm 114 28, Sweden · School of Mechanical and Mining Engineering (SMME), The University of Queensland, Brisbane, QLD 4072, Australia · Department of Computer Science and Artificial Intelligence, University of Technology Nuremberg, 90461 Nürnberg, Germany

Abstract

3D scene graphs provide semantically rich and hierarchical representations for robot perception. However, existing systems do not maintain uncertainty as an explicit belief or propagate it through the operations that construct and refine the graph. We introduce Probabilistic Scene Graph (PSG), a generalization of the conventional scene graph that represents a posterior over possible graphs, factorized into a discrete graph structure of entities, relations, and semantic attributes, and continuous states that ground them spatially, with uncertainty maintained over both components. Geometry is carried directly by the nodes rather than selected from a separately constructed metric map, so a metric map, where needed, follows from the graph rather than preceding it. We instantiate PSG's probabilistic spatial grounding with hierarchical graphs of Gaussians (HGG): each object primitive is represented by a full-covariance Gaussian under a Normal-Inverse-Wishart belief, and the same parametrization applied recursively within a node yields a geometry graph that resolves its surface at finer resolution. We then build a mapping pipeline that preserves these beliefs throughout graph construction and refinement: a purely graph-based coarse-to-fine alignment registers observations by comparing node beliefs, while a nested Expectation-Maximization and factor-graph optimization jointly refines poses, object parameters, and internal geometry. Across six datasets spanning indoor RGB-D, outdoor LiDAR, and cross-modality deployment, HGG operates at sensor rate with near-constant memory and achieves state-of-the-art object accuracy and zero-shot graph alignment.

Figures & tables

Fig. 1 : Our hierarchical probabilistic scene graph (HGG). Top: a symbolic object graph $G^{o}$ , with a belief over centroid and full covariance, connected by relational edges. Middle: the geometry graph $G^{g}_{i}$ held inside each object, the same definition applied one level down, whose combined means and covariances provide a belief over continuous surface geometry. Bottom: detail of the geometry graph (left). Realizations of dense semantic point clouds of the scene can be sampled from our PSG (right), comparable to the original measurements. Color denotes the semantic label.

Fig. 2 : System overview. Given a semantic point cloud frame and odometry, we create a hierarchical local graph $G_{t}$ , align it with the global graph $\mathcal{G}_{t-1}$ through coarse-to-fine graph alignment, and update the global graph $\mathcal{G}_{t}$ through map registration. The resulting representation is refined through nested optimization within a sliding window, followed by global optimization.

Notations	Description
$t$	Frame index
$G_{t}$	Local scene graph created from frame $t$
$\mathcal{G}_{t}$	Global scene graph registered up to frame $t$
$G^{o}$ , $\mathcal{G}^{o}$	Superscript $o$ denotes the object graph
$G^{g}_{i}$ , $\mathcal{G}^{g}_{i}$	Superscript $g$ denotes the geometry graph belonging to
	$i^{\text{th}}$ -node of the object graph $G^{o}$ , $\mathcal{G}^{o}$

TABLE I : Notation for the Probabilistic Scene Graph

Fig. 3 : Coarse-to-fine graph alignment between the local graph $G_{t}^{o}$ and the global graph $\mathcal{G}_{t-1}^{o}$ . Top: object-node-level correspondences combine geometric and structural costs, solved by the Hungarian algorithm. Bottom: the resulting correspondences initialize geometry-level matching, which combines spatial and Weisfeiler–Lehman structural costs.

Dataset	Sensor	Environment	Semantic Labels
uHumans2	RGB-D	indoor	Simulator [ 4 ]
KITTI	LiDAR	outdoor	RangeNet++ [ 76 ]
KITTI-Carla	LiDAR	outdoor	Simulator [ 77 ]
MCD	LiDAR	outdoor	2DPASS [ 78 ]
ScanNet-SG	RGB-D	indoor	ScanNet-SG [ 79 ]
In-house	LiDAR+RGB	indoor+outdoor	Mask2Former [ 80 ] +SAM2 [ 81 ]

TABLE II : Evaluation Datasets

Fig. 4 : Qualitative object accuracy on uHumans2-Subway. All panels share Ground-truth rgb point cloud map and objects colored by classes in (a), and matching accuracy in (b-d). Hydra (b) is dominated by missed detection (blue), since a graph abstracted from a dense reconstruction can contain only what that reconstruction builds. ConceptGraph (c) reaches most instances but re-instantiates them, surrounding recovered objects with ghosts (red). HGG (d) cover almost all object nodes with high precision.

Method	$\Delta_{p}$	Recall	Prec.	F1	t (ms)	$\Delta_{p}$	Recall	Prec.	F1	t (ms)	$\Delta_{p}$	Recall	Prec.	F1	t (ms)
	Apartment					Office					Subway
Kimera	0.13	0.30	0.51	0.38	2018.5	0.26	0.29	0.50	0.36	1956.7	0.73	0.34	0.78	0.47	3742.8
Hydra	0.14	0.26	0.58	0.35	62.8	0.18	0.18	0.49	0.26	58.4	1.46	0.34	0.93	0.50	41.6
ConceptGraph	0.21	0.23	0.77	0.36	1033.8	0.26	0.24	0.47	0.32	3169.1	0.82	0.65	0.23	0.34	4672.7
HGG	0.06	0.79	0.94	0.86	29.3	0.11	0.79	0.86	0.83	28.5	0.55	0.97	0.80	0.88	30.6

TABLE III : Object Accuracy Evaluation

	Apartment				Office				Subway
Method	$\Delta_{p}$	Recall	Prec.	F1	$\Delta_{p}$	Recall	Prec.	F1	$\Delta_{p}$	Recall	Prec.	F1
Hydra+GT	0.12	0.36	0.76	0.50	0.14	0.33	0.89	0.48	0.60	0.37	1.00	0.54
ConceptGraph+GT	0.06	0.15	0.62	0.24	0.12	0.20	0.57	0.30	0.40	0.62	0.37	0.46
HGG+GT	0.04	0.87	0.98	0.92	0.05	0.88	0.90	0.89	0.10	0.98	0.97	0.97

TABLE IV : Object Accuracy Evaluation with Ground-Truth Poses and Different Odometry.

	Apartment				Office				Subway
	$\Delta_{p}$	Recall	Precision	F1-score	$\Delta_{p}$	Recall	Precision	F1-score	$\Delta_{p}$	Recall	Precision	F1-score
OA-SLAM	0.23	0.17	0.33	0.23	0.46	0.33	0.50	0.40	1.53	0.10	1.00	0.19
VOOM	0.41	0.18	0.27	0.21	0.60	0.30	0.38	0.33	0.76	0.05	0.13	0.08
HGG	0.06	0.79	0.94	0.86	0.11	0.79	0.86	0.83	0.55	0.97	0.80	0.88

TABLE V : Object Accuracy Evaluation Against Object SLAM on Indoor Datasets

Sequence	Dataset	Method	$\Delta_{p}$	R	Prec.	F1
HGG	Subway	w/o NIW	1.21	0.97	0.58	0.73
	Subway	with NIW	0.55	0.97	0.80	0.88
	In-house- 02	w/o NIW	0.21	0.62	0.65	0.63
	In-house- 02	with NIW	0.05	0.92	0.85	0.88
	MCDNTU	w/o NIW	0.87	0.83	0.69	0.74
	MCDNTU	with NIW	0.75	0.84	0.82	0.82

TABLE VI : Evaluation on the effect of NIW posterior belief in our scene graph representation.

Method	Accuracy	Precision	Recall	F1-score
SG-Reg [ 52 ]	0.056	0.171	0.078	0.107
ROMAN [ 51 ]	0.430	0.505	0.268	0.336
VLM CosSim [ 53 ]	0.461	0.370	0.906	0.502
VLM+Bert CosSim [ 53 ]	0.518	0.400	0.898	0.530
OpenSGA-L [ 53 ]	0.642	0.502	0.713	0.568
OpenSGA-H [ 53 ]	0.681	0.547	0.684	0.589

TABLE VII : Graph Alignment Evaluation (following [ 53 ] ).

uHumans2
Method	Apartment	Office	Subway
OA-SLAM [29]	0.14	0.39	0.45
VOOM [30]	0.14	0.42	0.47
Kimera-RPGO [3]	0.07	0.46	1.68
Kimera-PGMO [3]	0.08	0.21	1.47
HGG	0.11	0.17	0.52

TABLE VIII : ATE Evaluation (m).

EM	LC	Chamfer (m) $\downarrow$	Map Acc. (m) $\downarrow$	F1 $\uparrow$	ATE (m) $\downarrow$
$\checkmark$		0.72	0.49	0.63	2.16
	$\checkmark$	1.11	0.97	0.82	0.68
$\checkmark$	$\checkmark$	0.72	0.49	0.82	0.68

TABLE IX : Effect of EM optimization and loop closure (LC)

Sequence	Method	R@1	R@2	R@3
Apartment	Conceptgraphs	0.35	0.41	0.59
Apartment	HGG (centroid edges)	0.47	0.82	0.94
	HGG	0.65	0.88	0.94
Office	Conceptgraphs	0.20	0.47	0.47
Office	HGG (centroid edges)	0.47	0.67	0.93
	HGG	0.47	0.87	0.93

TABLE X : Language Grounded Object Retrieval Results. HGG (centroid edges) replaces the distributional edge weight of ( 8 ) with the Euclidean distance between centroids.

3D Scene Representation

3D scene representation studies how to encode geometry, appearance, semantics, and motion for reconstruction, rendering, generation, and interaction.

See all 242 papers

Hierarchical Aggregation of Semantic Uncertainty in 3D Scene Graphs
Sep 17, 2026Carlos Cueto Zumaya, Iacopo Catalano, Wallace Moreira Bessa +13D Scene Understanding 3D Scene Representation
University of Turku, Finland · Aragón Institute of Technology (ITA) and the University of Zaragoza, Spain
DGSG-Mind: Dynamic 3D Gaussian Scene Graphs for Long-Term Scene Understanding and Grounding
May 28, 2026Luzhou Ge, Xiangyu Zhu, Jinyan Liu +13D Gaussian Splatting Open-Vocabulary 3D Segmentation
School of Computer Science, Beijing Institute of Technology, Beijing 100081, China.
DeWorldSG: Depth-Aware 3D Semantic Scene Graph Generation via World-Model Priors
Jul 1, 2026Seok-Young Kim, Abdelrahman Elskhawy, Taewook Ha +4Scene Graph Generation 3D SGG
Korea Advanced Institute of Science and Technology · Technical University of Munich · Munich Center for Machine Learning (MCML) +1

Sep 17, 2026J/K move · Enter open · S save

Hierarchical Aggregation of Semantic Uncertainty in 3D Scene Graphs

Carlos Cueto Zumaya, Iacopo Catalano, Wallace Moreira Bessa, Julio A. Placed

University of Turku, Finland · Aragón Institute of Technology (ITA) and the University of Zaragoza, Spain

3D Scene Understanding 3D Scene Representation

Open paper AskPDF arXiv

Open-vocabulary 3D Scene Graphs (3DSGs) ground each object node in a vision-language embedding, yet they record every entry as equally certain, so a robot querying the map cannot tell which of its entries are unreliable. Estimators of semantic uncertainty could supply that distinction, but they require repeated sampling of a model, training, or held-out labels, none of which are available to a deployed system at query time. We present a framework that exploits the detector confidence and the embeddings a 3DSG already stores, converts them into a probability that an entry is correct, and propagates that probability through the containment hierarchy into a belief that a room contains a queried class. Four signals, each paired with the object-level error it indicates, are converted to probabilities at the logit scale learned by the vision-language model and combined in closed form with no additional perception or training. Objects sharing a detector and a vocabulary fail together, so the framework aggregates them in the fully correlated limit, where an aggregation under independence would treat one repeated error as repeated evidence. Evaluated on HM3DSem against a state-of-the-art 3DSG system, the framework improves object retrieval and lowers the error of the room-level assertions of the graph it reads.

Abstract

3D Scene Representation

Explore similar work

Hierarchical Aggregation of Semantic Uncertainty in 3D Scene Graphs

DGSG-Mind: Dynamic 3D Gaussian Scene Graphs for Long-Term Scene Understanding and Grounding

DeWorldSG: Depth-Aware 3D Semantic Scene Graph Generation via World-Model Priors