Open-Set 3D Scene Graphs for Field Robotics: An Outdoor Case Study

Chad R. Samuelson; Gabriel R. Slade; Joshua G. Mangelson

cs.ROSep 4, 2026

Open-Set 3D Scene Graphs for Field Robotics: An Outdoor Case Study

Authors: Chad R. Samuelson, Gabriel R. Slade, Joshua G. Mangelson

Organizations: Brigham Young University

Abstract

Three-dimensional scene graphs (3DSGs) have emerged as a promising approach for building geometrically grounded, semantically informed, hierarchical general-purpose maps to support high-level robotic reasoning. However, the behavior of 3DSGs in real-world outdoor deployments remains poorly understood, particularly when combined with open-set vision-language models (VLMs). In this field report, we analyze the components common to most 3DSG representations across five outdoor robotic datasets to characterize challenges that arise in complex outdoor environments. Using the recently proposed Terra 3DSG as a case study, we investigate semantic point embeddings, place-node graph navigation, region-level understanding, and memory size across the five diverse datasets. We additionally introduce novel consistency metrics to evaluate whether semantic and structural graph properties remain stable across repeated traversals of the same environment. Our analysis reveals that outliers and multiple modes are common in VLM point embeddings across all tested datasets with outlier ratios above $0.1$ for around $30\%$ of points. We demonstrate the feasibility of outdoor 3DSGs for navigation-based object retrieval, achieving success rates near $70\%$ , though performance is limited by traversability failures and inefficient routing, with trajectories averaging approximately $66\%$ suboptimal path efficiency. Region-level understanding remains challenging in complex natural environments, with low average F1 scores around $0.359$ . Overall, our results show that outdoor 3DSGs can maintain compact (less than $600$ MB for multi-kilometer trajectories) and relatively consistent large-scale environment representations, while highlighting open challenges in handling multiple semantic modes, incorporating traversability into graph structures, and improving higher-level region understanding.

Figures & tables

Panel of Figure 1 — Figure 1: Example qualitative Terra results showing the sequential phases of Terra in building the full 3DSG on the River Park dataset. (c) Place nodes are colored by terrain type with the following terrains: 1) leaves, 2) road, 3) sidewalk, 4) gravel, 5) dirt, 6) grass, and 7) water.

Dataset	Trajectory Length $[m]$	Environment Type	Vegetation	Terrain Types	Perceptual Challenges
River Park	951.56	Recreational park	Dense	Leaves, road, sidewalk, dirt, grass, water	Strong lighting contrast, canopy occlusions, large and small objects, trajectory on a variety of terrains
Nunns Park	1238.27	Riverside recreational area	Sparse	Leaves, road, sidewalk, gravel, dirt, grass, water	Leaf-covered surfaces, underpass shadowing
Marina Part 1	3004.28	Marina / RV park	Sparse	Leaves, road, sidewalk, gravel, dirt, grass, water	Open geometry, dynamic vehicles, large range of object sizes
Marina Part 2	1269.63	Marina	Sparse	Leaves, road, gravel, dirt, grass, water	Road-gravel transition, limited vertical structure
Rock Canyon Campground	1151.43	Forested trail	Dense	Leaves, gravel, dirt, grass	Repetitive natural geometry, mixed-surface trails, ambiguous objects
Feature comparisons of the five datasets collected for evaluating Terra. The datasets cover a multitude of diverse environments, terrain complexities, lengths, and lighting conditions to provide a reliable evaluation test bed for Terra. There were a total of seven terrain types, and the overall combined trajectory length was $7615.17~m$ .

Table 1: Dataset Summaries

Panel of Figure 2 — Figure 2: Overview of the five field-test datasets. For each dataset, the satellite image (left) shows the spatial extent of the trajectory, while the three camera images (right) illustrate representative visual conditions encountered during data collection.

Panel of Figure 3 — Figure 3: (a) Image of our robotic wheelchair platform. (b) Depicts the rigid sensor platform used for data collection. Only the 3 OAK-D LR cameras positioned at 90 degree angles from each other, the Ouster OS1 128-beam LiDAR, and 6-axis IMU were used.

Figure 4: Outlier ratio computed for every point across all five datasets.

Figure 5: Histogram of the number of embedding clusters associated with map points in the River Park dataset. Figure (b) shows a zoomed-in view of (a). While most points contain fewer than 10 clusters, some points exhibit up to 100 clusters, demonstrating the significant presence of multiple semantic modes within the dataset.

Figure 6: Representative masked image from each of 21 embedding clusters associated with a single point in the River Park dataset. This example highlights the diversity of visual observations for the same map point.

Figure 7: Masked images associated with two embedding clusters shown in Figure 6 . These examples qualitatively demonstrate the visual and semantic consistency present within individual clusters.

Method	RS-1 $\uparrow$	RS-3 $\uparrow$
Object Only	0.647	0.765
Full Sentence	0.529	0.706
Object+Context	0.765	0.824
Relaxed success rates (RS) for object detection on the 3DSG given $17$ contextualized object prompts. We see that combining separate object and then context information for unique contextualized object based prompts obtains the highest success rate.

Table 2: 3DSG Object Detection Results

Panel of Figure 8 — Figure 8: Qualitative results on three demonstrative object navigation tasks. Three camera images in each figure shows the robot’s camera view at the trajectory end from the three different starting locations. The trajectories taken by the robot are colored in red, green, and either blue/orange (switched for visual clarity). (a) Demonstrates a successful navigation to the stop sign from three diverse starting locations. (b) Demonstrates a partial success (i.e., RS) where the garbage can is not in the camera view, but it is within 10 meters of the robot’s GPS coordinates. (c) Demonstrates a success for the blue trajectory starting location, but a navigation failure for the other two starting locations. This navigation failure is caused by planned path GPS waypoints veering off the side of the road down a steep drop-off.

Dataset	# of Trials	SS-1 $\uparrow$	RS-1 $\uparrow$	Path Efficiency
River Park	15	0.400	0.800	0.68 ( $\pm$ 0.09)
Nunns Park	12	0.417	0.417	0.64 ( $\pm$ 0.26)
Marina Part 1	6	1.000	1.000	0.70 ( $\pm$ 0.16)
Marina Part 2	9	0.667	0.667	0.60 ( $\pm$ 0.17)
Summary	42	0.548	0.691	0.66 ( $\pm$ 0.17)
Relaxed success rates (RS) for object detection on the 3DSG. We see that combining separate object and then context information for unique contextualized object based prompts obtains the highest success rate.

Table 3: Real-World 3DSG-Based Object Navigation Results

Method	Micro $\uparrow$			Macro $\uparrow$
	Avg Prec.	Avg Rec.	Avg F1	Avg Prec.	Avg Rec.	Avg F1
River Park - 4 regions
Spectral	0.242 $\pm$ 0.059	0.843 $\pm$ 0.132	0.364 $\pm$ 0.032	0.261 $\pm$ 0.040	0.722 $\pm$ 0.155	0.359 $\pm$ 0.025
Agglomerative	0.284 $\pm$ 0.136	0.848 $\pm$ 0.182	0.384 $\pm$ 0.095	0.278 $\pm$ 0.059	0.751 $\pm$ 0.167	0.371 $\pm$ 0.055
Nunns Park - 4 regions
Spectral	0.210 $\pm$ 0.042	0.657 $\pm$ 0.290	0.305 $\pm$ 0.086	0.209 $\pm$ 0.073	0.634 $\pm$ 0.164	0.284 $\pm$ 0.047

Table 4: Region Querying Results

Figure 9: Labeled regions for the Marina Part 1 dataset as used in the region querying experiments. The given descriptions of the regions by color are as follows: Red: “Pavilions and picnic tables with parking stalls”; Green: “Long road with water on both sides”; Blue: “Boat ramps and docks with parking lot”; and Purple: “Buildings and sidewalk path.”

Figure 10: Region querying results. Micro-F1 scores (left) and Macro-F1 scores (right) reported across a range of

k

values for each dataset.

Figure 11: Region nodes and their cosine similarity scores by level for query “Long dirt road with water and dirt rocks” from the Marina Part 2 dataset.

Region Prompt	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$
Marina Part 1 - Only 1 of the 4 Regions
Original	0.713	0.521	0.602
Clarification	0.840	0.610	0.707
Negation	0.152	0.152	0.152
Scenery	0.688	0.345	0.460
Combination	0.970	0.313	0.474

Table 5: Region Querying with Prompt Tuning Results

Figure 12: Memory experiment comparing mapped area size to Terra memory size and number of nodes. The largest 3DSG remains under 600 MB supporting the claim that 3DSGs can be memory efficient in large-scale outdoor scenes.

Panel of Figure 13 — Figure 13: Four datasets collected across the same geographical area. Top row: LIO-SAM metric map. Bottom row: Place node graphs colored by terrain.

Panel of Figure 14 — Figure 14: Consistency results across RV Park datasets. (a) Shows a birds-eye-view (BEV) satelite image of the area mapped. (b) Evaluates graph structure consistency with the Node Count Ratio (NCR). (c) Evaluates graph structure consistency with the Node Clustering Consistency (NCC). (d) Evaluates the semantic consistency with the Node Cosine Similarities (NCS). (e) Evaluates the semantic consistency with the Terrain Matching Consistency (TMC). (f) Evaluates the geometric consistency with the Mean distance between nodes (Dist). All metrics are calculated from the row graph to the column graph. The results indicate the Terra 3DSG structure is reliably consistent across graph structure, semantics, and geometry.

Parameter	Description	Symbol	Value
Metric Map
Surface Voxel Size	Surface voxel resolution for LIO-SAM	$v_{s}$	$0.4$ m
Corner Voxel Size	Corner voxel resolution for LIO-SAM	$v_{c}$	$0.2$ m
Phase 1: Metric-Semantic Map
Time Alignment Threshold	Max LiDAR-image timestamp difference	$\tau_{align}$	$0.05$ s
Image Resolution	Input RGB resolution	$H\times W$	$480\times 768$

Table 6: Terra System Parameters

Dataset	$\alpha=0.25$	$\alpha=0.26$	$\alpha=0.27$
River Park	0.308	0.478	0.333
Nunns Park	0.195	0.420	0.499
Marina P1	0.162	0.188	0.213
Marina P2	0.260	0.368	0.197
Rock Canyon	0.143	0.143	0.000
Average F1-Score	0.214	0.319	0.248

Table 7:

\alpha

Ablation Study

Object	Context
River Park
“bridge”	“large brown over the river”
“bridge”	“small white in the woods”
“playground”	“in sand”
“garbage”	“in the parking lot”
“garbage”	“by the red building”

Table 8: Object Prompt Descriptions

Object Weight	Context Weight	SS-1
$w_{o}=0.0$	$w_{c}=1.0$	0.412
$w_{o}=0.1$	$w_{c}=0.9$	0.471
$w_{o}=0.2$	$w_{c}=0.8$	0.471
$w_{o}=0.3$	$w_{c}=0.7$	0.647
$w_{o}=0.4$	$w_{c}=0.6$	0.588
$w_{o}=0.5$	$w_{c}=0.5$	0.588

Table 9: Contextualized Object Querying Ablation Study

Region Prompt	GT Node Count
River Park - 4 regions
“Playground area with slides and swings”	30
“Dense trees with picnic tables”	214
“Duck Pond”	51
“Open field with red pavilions”	49
Nunns Park - 4 regions

Table 10: Region Prompt Descriptions

3D Scene Representation

3D scene representation studies how to encode geometry, appearance, semantics, and motion for reconstruction, rendering, generation, and interaction.

See all 240 papers

3D Scene Graphs: Open Challenges and Future Directions
Jun 15, 2026Dennis Rotondi, Francesco Argenziano, Sebastian Koch +103D SGG 3D Scene Representation
University of Stuttgart · IMPRS-IS · Sapienza University of Rome +7
OP3DSG: Open-Vocabulary Part-Aware 3D Scene Graph Generation for Real-World Environments
Jun 29, 2026Yirum Kim, Ue-Hwan Kim3D SGG Robotic Perception
Gwangju Institute of Science and Technology (GIST), Gwangju, Republic of Korea
OpenSplatGraph: From Dense Semantic Maps to Structured Scene Graphs for Open-Vocabulary Robot Perception
Oct 6, 2026Binh Long Nguyen, Kien Nguyen, Clinton Fookes +1Language-Embedded 3DGS Open-Vocabulary 3D Segmentation
School of Electrical Engineering and Robotics, Queensland University of Technology (QUT), Brisbane, QLD 4000, Australia · CSIRO Robotics, CSIRO, Brisbane, QLD 4069, Australia

Jun 15, 2026J/K move · Enter open · S save

3D Scene Graphs: Open Challenges and Future Directions

Dennis Rotondi, Francesco Argenziano, Sebastian Koch, Nathan Hughes+9

University of Stuttgart · IMPRS-IS · Sapienza University of Rome+7

3D SGG 3D Scene Representation 3D Scene Understanding

Open paper AskPDF arXiv

3D Scene Graphs (3DSGs) have emerged as a powerful representation for spatial AI by combining geometric grounding with semantic and relational abstractions of the environment. Their expressiveness has made them relevant to a broad range of problems in robotics and computer vision, including manipulation, navigation, task planning, scene understanding, and many others. However, the field remains fragmented: different communities adopt distinct formulations, construction pipelines, and evaluation protocols, making it difficult to compare methods, identify common assumptions, and assess remaining challenges for robust real-world deployment. This survey provides a unified and critical review of 3DSGs, with particular emphasis on open challenges and future directions. We first formalize 3DSGs under a common definition and analyze the principal modeling choices that characterize existing formulations, including node and edge attributes, hierarchical structure, dynamic scene representations, and affordance-aware extensions. We then review how 3DSGs are built from raw sensory observations, discussing the most common terminologies, conventions, and techniques. Finally, we examine downstream applications and evaluation strategies, from intrinsic graph quality to task-level performance. To support the community, we also provide a dedicated website that organizes and extends the surveyed content, accessible at https://3dscenegraphs.com/.

Abstract

3D Scene Representation

Explore similar work

3D Scene Graphs: Open Challenges and Future Directions

OP3DSG: Open-Vocabulary Part-Aware 3D Scene Graph Generation for Real-World Environments

OpenSplatGraph: From Dense Semantic Maps to Structured Scene Graphs for Open-Vocabulary Robot Perception