Relational Semantic Reasoning on 3D Scene Graphs for Open World Interactive Object Search

Imen Mahdi; Matteo Cassinelli; Fabien Despinoy; Tim Welschehold; Abhinav Valada

cs.ROMar 5, 2026

Relational Semantic Reasoning on 3D Scene Graphs for Open World Interactive Object Search

Authors: Imen Mahdi, Matteo Cassinelli, Fabien Despinoy, Tim Welschehold, Abhinav Valada

Organizations: University of Freiburg · Toyota Motor Europe

Abstract

Open-world interactive object search in household environments requires understanding semantic relationships between objects and their surrounding context to guide exploration efficiently. Prior methods either rely on vision-language embeddings similarity, which does not reliably capture task-relevant relational semantics, or large language models (LLMs), which are too slow and costly for real-time deployment. We introduce SCOUT: Scene Graph-Based Exploration with Learned Utility for Open-World Interactive Object Search, a novel method that searches directly over 3D scene graphs by assigning utility scores to rooms, frontiers, and objects using relational exploration heuristics such as room-object containment and object-object co-occurrence. To make this practical without sacrificing open-vocabulary generalization, we propose an offline procedural distillation framework that extracts structured relational knowledge from LLMs into lightweight models for on-robot inference. Furthermore, we present SymSearch, a scalable symbolic benchmark for evaluating semantic reasoning in interactive object search tasks. Extensive evaluations across symbolic and simulation environments show that SCOUT outperforms embedding similarity-based methods and matches LLM-level performance while remaining computationally efficient. Finally, real-world experiments demonstrate effective transfer to physical environments, enabling open-world interactive object search under realistic sensing and navigation constraints.

Figures & tables

Fig. 1: Overview of SCOUT: Scene Graph-Based Exploration with Learned Utility for Open-World Interactive Object Search. SCOUT procedurally distills structured, relational semantic knowledge between scene elements from large language models into lightweight models (1–2). During exploration, the agent assigns utility scores to scene graph nodes based on exploration heuristics previously learned (3) and grounds high-level actions through low-level navigation and manipulation policies (4). SymSearch, our symbolic benchmark (5), enables scalable evaluation of relational semantic reasoning over scene graphs with no simulation overhead.

Fig. 2: Full pipeline of SCOUT illustrated . From left to right: 3DSG is constructed online from raw observations. Scene graph nodes are scored based on their utility in finding the query. Once the node to explore is selected, its affordances determine which low-level policies to run.

Fig. 3: Overview of the procedural knowledge distillation pipeline. An LLM is queried procedurally to generate a diverse open-vocabulary set of household objects and relational annotations. The resulting object–object co-occurrence and room–object containment datasets are then used to supervise lightweight MLP models for utility estimation.

Fig. 4: Illustration of SymSearch’s roll-out process. At each step, the agent receives a scene-graph observation and a target object query. (a) At initialization, the agent spawns in a random region and room. Unexplored regions appear as frontier nodes, and rooms connected via doors are seen but unexplored. (b) Exploring a new room reveals the closest region to the agent in that room, along with its objects. The remaining regions are considered unexplored frontiers. (c) Exploring a frontier reveals the objects within that region. (d) Exploring an object reveals its nested objects (e.g., objects inside or on top of it). The episode terminates when the target object becomes visible.

Benchmark	Scenes	Room Cat.	Obj. Cat.	Rooms/Sc.	Room Cat./Sc.	Obj./Sc.	Obj. Inst./Sc.	Contained Obj./Sc.
AI2-THOR	120	4	117	1.0	1.0	46.3	30.6	17.9
OmniGibson	45	39	237	7.1	5.3	245.0	31.4	6.3
InteriorGS	1000	687	697	6.6	5.1	408.3	52.4	157.5 *

TABLE I: Statistics on interactive benchmarks. We report dataset scale, semantic diversity, and scene complexity.

Panel of Figure 6 — Fig. 5: 3DSG construction pipeline from InteriorGS scenes. (a) Each scene includes a top-down occupancy map, annotated object bounding boxes, room polygons, and door/opening bounding boxes, which we use to construct the 3DSG. (b) Each object’s position is defined as the center of its bounding box. Objects are assigned to rooms if their centers lie within the corresponding room polygon. Rooms are annotated using an LLM based on their contents, and are then manually verified for consistency. (c) To determine room connectivity, door polygons are dilated, and rooms sharing an intersecting door are marked as connected. (d) To define spatially coherent regions within rooms, we cluster object $(x,y)$ coordinates using a Gaussian Mixture Model. Each cluster represents a distinct region. (e) Nested object relationships (e.g., on-top-of , inside ) are manually annotated by experts by inspecting the Gaussian splats. (f) Traveled distances are computed by first identifying the closest unoccupied pixel to the goal object within the same room on a dilated occupancy map, then applying A* to compute realistic geodesic paths from the agent to the goal.

Fig. 6: Prompts used for procedural relational semantic knowledge distillation. Left: Prompts used to generate a large and diverse set of household objects; (top) household room categories, (middle) object categories per room, and (bottom) object set associated with each room–object category pair. Right: Prompts used to generate relational semantic datasets for object–object co-occurrence and room–object containment.

Hyperparameter	Value
Default room exploration score	0.7
Default frontier exploration score	0.6
$\Delta$ (utility margin)	0.1
Room influence weight $w$	0.3

TABLE II: Hyperparameters used to instantiate our agent.

Fig. 7: Comparison of embedding similarity distributions with our learned relational scoring models. Synonym pairs are separable across all models, whereas co-occurrence and containment relationships are not. Our learned models produce substantially stronger separation.

Relational Semantic	Positive Pairs		Negative Pairs
Synonyms and Functionally Equivalent	sponge	scrubber	tv	plant
Synonyms and Functionally Equivalent	laptop	notebook	bottle	car
	bookshelf	bookcase	broom	clock
	game console	video game system	fridge	chair
	ironing board	iron stand	fan	watch
Co-occur	shower	towel	kettle	bed

TABLE III: Examples of positive and negative pairs used to evaluate relational semantics.

Query	Scene	Interactive	Location Description
teapot	0900	✓	on top of a teatable in a combined living room and kitchen
refrigerator	0900	$\times$	in a combined living room and kitchen
vegetable	0901	✓	on top of a chopping board in a kitchen
eggbeater	0902	✓	inside a cabinet in a kitchen
tv	0902	$\times$	in a bedroom
multi person sofa	0902	$\times$	in a living room

TABLE IV: Examples of evaluation episodes from SymSearch

Query	Scene	Interactive	Location Description
chopstick	Pomaria_1_int	$\times$	on top of a breakfast table in a kitchen
wall_mounted_tv	Wainscott_0_int	$\times$	on top of a console table in a living room
lipstick	Beechwood_0_int	$\times$	on top of a sink in a bathroom
vacuum	Beechwood_0_int	$\times$	in a utility room
keyboard	Beechwood_0_int	$\times$	on top of a desk in a private office
dental_floss	Merom_0_int	✓	inside of a bottom cabinet in a bathroom

TABLE V: Examples of evaluation episodes from OmniGibson

Embedder	Hyperparameter	Value
SBERT	Default room exploration score	0.3
	Default frontier exploration score	0.25
	$\Delta$ (utility margin)	0.05
	Room influence weight $w$	0.3
CLIP	Default room exploration score	0.7
	Default frontier exploration score	0.4

TABLE VI: Hyperparameters used for Embeddings-based agents.

		SymSearch				OmniGibson
Method	Agent	SR $\uparrow$	SPL $\uparrow$	N Steps $\downarrow$	Inference	SR $\uparrow$	SPL $\uparrow$	N Steps $\downarrow$	Inference
					Time (s) $\downarrow$				Time (s) $\downarrow$
Random	Random Agent	$0.381\pm 0.043$	$0.074\pm 0.012$	$45\pm 2$	$0.00\pm 0.00$	-	-	-	-
Embedding Similarity	CLIP Similarity (w/o RI)	$0.564\pm 0.010$	$0.172\pm 0.018$	$37\pm 1$	$0.58\pm 0.01$	-	-	-	-
Embedding Similarity	CLIP Similarity	$0.650\pm 0.006$	$0.177\pm 0.019$	$33\pm 1$	$0.75\pm 0.01$	$0.565$	$0.322$	$23$	$1.5$
	SBERT Similarity (w/o RI)	$0.632\pm 0.029$	$0.180\pm 0.017$	$36\pm 1$	$0.08\pm 0.00$	-	-	-	-

TABLE VII: Evaluation results on SymSearch and Omnigibson. Table reports the Success Rate (SR), Success weighted by Path Length (SPL), average number of steps (high-level actions) per episode, and average inference time in seconds. Results on SymSearch are averaged across 5 different seeds. We report the best , second-best , and third-best results for each metric, respectively.

Fig. 8: Success rate over an increasing number of steps on SymSearch. Our method closely follows the performance of LLM-based baselines throughout the roll-out.

Query	Room Type	Room Score	Object	Object Score	Updated Score
plate	kitchen	0.78	counter	0.99	0.78
			cabinet	0.94	0.75
			sink	0.94	0.75
			table	0.92	0.74
			fridge	0.45	0.48
			chair	0.12	0.30

TABLE VIII: Example utility scores for room and objects categories across multiple queries. In-distribution queries are highlighted in green , out-of-distribution queries in yellow , and descriptive queries in blue .

	Seen		Unseen
Benchmark	SR	SPL	SR	SPL
SymSearch	0.859	0.277	0.945	0.300
OmniGibson	0.850	0.431	0.809	0.398

TABLE IX: Performance Breakdown to seen vs. unseen queries.

Agent	SR $\uparrow$	SPL $\uparrow$	N Steps $\downarrow$
Design Choices and Hyperparameters
SCOUT ( $\Delta$ =0.3)	0.792	0.270	24
SCOUT (Room Weigth: 0.7)	0.814	0.277	23
SCOUT (Room Weigth: 0.5)	0.825	0.273	22
SCOUT (Room Weigth: 0.1)	0.865	0.275	21
SCOUT (Unknown Room Score: 0.3)	0.796	0.233	28

TABLE X: Ablation study on SymSearch.

Model Card						Performance
Model	Version	Pricing (I/O)	Size	Local	Structured	SR $\uparrow$	SPL $\uparrow$	Steps $\downarrow$	Time (s) $\downarrow$
Mistral	Mistral-7B-Instruct-v0.1	Open-source	7B	✓	$\times$	0.24	0.149	42	146
LLaMA	Meta-Llama-3.1-8B-Instruct	Open-source	8B	✓	$\times$	0.24	0.139	39	423
DeepSeek (chat)	deepseek-chat	$0.028 /$ 0.42	671B	$\times$	$\times$	0.78	0.222	24	44
DeepSeek (reasoner)	deepseek-reasoner	$0.028 /$ 0.42	671B	$\times$	$\times$	0.96	0.356	15	721
Claude	claude-sonnet-4-5	$3 /$ 15	Undisclosed	$\times$	✓	0.82	0.216	23	171

TABLE XI: Comparison of LLM variants for interactive object search. We report model specifications and performance metrics on SymSearch. Prices are reported per 1 million tokens (Input / Output). We report the best , second-best , and third-best results for each metric, respectively.

Panel of Figure 20 — Fig. 9: Real-world robot experiments. The robot’s high-level-actions are grounded to the scene through object affordances. The robot can open containers to locate potential target objects (e.g., fridge, drawer, cabinet) or explore on top of furniture (e.g., table, desk, counter).

Query	Interactive	Location	Room	Outcome
milk bottle	✓	inside fridge	kitchen	S
				S
				P
remote-controller	$\times$	on top of table	living room	S
				S
				P

TABLE XII: Results on real world experiments. We report outcomes across interactive and non-interactive object search tasks. Letters denote result types: S = Success, R = Reasoning Failure, P = Perception Failure, M = Manipulation Failure.

Fig. 10: Example of a real-world experiment. The first column shows top-view maps of the environment, which consists of a kitchen, office space, and living room. The second column presents the robot’s front-camera observations with overlaid detected object labels. The third column displays the occupancy map together with the computed Voronoi graph, detected doors, and frontiers. The fourth column shows examples of the 3D semantic voxel maps. Top row: the robot starts in the kitchen and searches for a remote controller located on the living room table. Initially, the agent can observe only the kitchen and part of the office space, which are separated by a frontier. Bottom row: the agent explores the frontier, discovers the living room, and successfully observes the target object.

Scene Graph	Inference	Execution	Total
Construction	(Node Selection)	(Nav. + Manip.)
5.036 $\pm$ 0.016	0.211 $\pm$ 0.095	34.348 $\pm$ 24.810	39.596 $\pm$ 24.807

TABLE XIII: Runtime breakdown per timestep. Average execution time (in seconds) for each component of the pipeline.

Fig. 11: Distribution of real-robot experiments. We conduct a total of 36 experiments, of which 50% require interactivity with the environment (i.e., the target object is hidden inside containers). Failure modes are primarily due to errors in the perception system.

Robotic Object Search

See all 22 papers

Jun 4, 2026

SCOUT: Semantic scene COverage via Uncertainty-guided Traversal

Robots that operate over extended periods should not merely visit space; they should progressively understand it. Yet most 3D scene graph pipelines treat perception as a post-processing stage over a fixed dataset, decoupling scene representation from the decisions that determine what is observed in the first place. We present SCOUT, an online semantic exploration framework that closes this loop by coupling active traversal with probabilistic scene graph construction. Given a prior 2D occupancy map and posed RGB-D observations, SCOUT incrementally builds an uncertainty-aware 3D scene graph whose nodes maintain fused geometry and posterior beliefs over open-vocabulary object labels, while edges encode structural relations such as on, inside, belong, and next to. These beliefs are fed back to an uncertainty-guided traversal planner, which selects viewpoints by balancing expected semantic certainty gain, geometric coverage gain, and travel cost. In this way, the robot revisits ambiguous objects when additional evidence matters and expands into unseen free space when the scene remains incomplete. The resulting system treats semantic scene completeness as an operational objective rather than a passive by-product of semantic mapping, moving toward autonomous agents that can patrol, update, and reason about evolving indoor environments with minimal human intervention.

Junyu Mao, Sara Ayoubi, Vishnu D. Sharma +2

Nokia Bell Labs, France. · Nokia Bell Labs, Murray Hill, NJ, USA.

Active Perception Robot Navigation 3D SGG Semantic Mapping

Ask PDF

Mar 18, 2026

ReLaGS: Relational Language Gaussian Splatting

Achieving unified 3D perception and reasoning across tasks such as segmentation, retrieval, and relation understanding remains challenging, as existing methods are either object-centric or rely on costly training for inter-object reasoning. We present a novel framework that constructs a hierarchical language-distilled Gaussian scene and its 3D semantic scene graph without scene-specific training. A Gaussian pruning mechanism refines scene geometry, while a robust multi-view language alignment strategy aggregates noisy 2D features into accurate 3D object embeddings. On top of this hierarchy, we build an open-vocabulary 3D scene graph with Vision Language derived annotations and Graph Neural Network-based relational reasoning. Our approach enables efficient and scalable open-vocabulary 3D reasoning by jointly modeling hierarchical semantics and inter/intra-object relationships, validated across tasks including open-vocabulary segmentation, scene graph generation, and relation-guided retrieval. Project page: https://dfki-av.github.io/ReLaGS/

Yaxu Xie, Abdalla Arafa, Alireza Javanmardi +5

German Research Center for Artificial Intelligence (DFKI) · RPTU University Kaiserslautern-Landau · University of Modena and Reggio Emilia

Language-Embedded 3DGS 3D SGG Open-Vocabulary 3D Segmentation

Ask PDF

Jul 6, 2026

Beyond Isolated Objects: Relationship-aware Open Vocabulary Scene Understanding via 3D Scene Graph Analysis

Open-vocabulary 3D scene understanding aims to segment 3D scenes beyond predefined categories by transferring semantic knowledge from vision-language models. Existing methods have advanced this task by lifting language-aligned 2D features into 3D, yet they often rely on context-independent semantic representations, leaving object relationships underexplored for contextual refinement. We propose RelGraphOV, a relationship-aware framework that uses 3D scene graphs to enhance open-vocabulary 3D understanding. Our method constructs relational scene graphs from multi-view observations by leveraging vision-language reasoning to infer object relationships and prune geometrically implausible connections, without manual relationship annotations. To aggregate relational context while avoiding feature interference, we introduce an Adaptive Gated Dual-Stream Contextual GAT that separates dense geometric features and semantic CLIP embeddings, performs edge-guided message passing, and adaptively fuses complementary semantics. A hierarchical contrastive objective further promotes instance-level consistency and category-level discrimination. Experiments on ScanNetV2, ScanNet200, ScanNet

++

, and Replica demonstrate strong performance and generalization ability. Project Page: https://cxavireh.github.io/relgraphov-projectpage

Xianhao Chen, Jiarui Hu, Yuanbo Yang +5

State Key Lab of CAD&CG, Zhejiang University, China · Zhejiang University, China

Graph Attention Networks Open-Vocabulary 3D Segmentation 3D Scene Understanding

Ask PDF

Abstract

Robotic Object Search

Explore similar work

SCOUT: Semantic scene COverage via Uncertainty-guided Traversal

ReLaGS: Relational Language Gaussian Splatting

Beyond Isolated Objects: Relationship-aware Open Vocabulary Scene Understanding via 3D Scene Graph Analysis