LOCUS: Landmark-Oriented Container Discrimination Using Spatial Graphs

Taylor Bergeron; Shibani Senthilbabu; Kevin Leahy

cs.ROOct 2, 2026

LOCUS: Landmark-Oriented Container Discrimination Using Spatial Graphs

Authors: Taylor Bergeron, Shibani Senthilbabu, Kevin Leahy

Organizations: Department of Robotics Engineering, Worcester Polytechnic Institute, Worcester, MA, USA

Abstract

As robots are increasingly deployed in unstructured, real-world environments, the ability to reason about complex spatial and semantic relationships among objects remains a fundamental challenge in enabling robust and generalizable manipulation and navigation. For example, deciding where to search for an object that is not in plain sight depends on where it is physically plausible as well as semantically likely. Popular methods such as cosine similarity with CLIP struggle to disambiguate between spatially and semantically similar objects that could contain a target object. We propose Landmark-Oriented Container Discrimination Using Spatial Graphs (LOCUS). To jointly reason about spatial information and semantics, we train a GNN to update CLIP embeddings on a full environment scene graph by passing embedding information between nodes based on proximity. CLIP embeddings, augmented by semantic knowledge from a fusion of household ontologies provide a robust semantic signal. We evaluate in simulation on a noiseless scene graph from simulator metadata, making the approach detection-agnostic. Our approach outperforms random, CLIP, Tidybot, and an LLM planner in the majority of room classes and configurations in simulation. To show our approach is robust to scene graph node placement and label noise, we demonstrate a physical mobile manipulator running in an exploration pipeline start-to-finish including scene graph labels from Detic.

Figures & tables

Panel of Figure 1 — Fig. 1 : Feature extraction and graph construction.

Panel of Figure 2 — Fig. 2 : GNN architecture and scoring. A shared residual encoder maps container and landmark features; APPNP propagates over the graph.

Method	IRA $\uparrow$	ECO / $N$ $\downarrow$	S-ECO / $N$ $\downarrow$	TypeAcc $\uparrow$
Random (5 seeds)	.241	.455	.347	63.3%
Visual CLIP	.188	.466	.375	60.2%
Text CLIP	.182	.436	.385	52.2%
$P(q{\mid}\text{room}){\times}P(q{\mid}\text{type})$	.152	.398	.356	61.7%
$P(q{\mid}\text{room}){\times}P(q{\mid}\text{type}){\times}f(\text{dist})$	.455	.335	.288	60.7%
TidyBot commonsense (zero-shot)	.225 ${\pm}$ .019	.338 $\pm$ .008	.277 $\pm$ .007	92.0%

TABLE I : iTHOR test set. Best result per column in bold . Informed=our training placements.

Room	$\mathbf{n}$	Method	IRA $\uparrow$	S-IRA $\uparrow$	ECO/ $N$ $\downarrow$	S-ECO/ $N$ $\downarrow$
Bath.	32	Random	.383 $\pm$ .074	.554 $\pm$ .068	.498 $\pm$ .044	.423 $\pm$ .040
		Vis. CLIP	.414 $\pm$ .000	.586 $\pm$ .000	.478 $\pm$ .000	.403 $\pm$ .000
		Text CLIP	.382 $\pm$ .071	.555 $\pm$ .068	.547 $\pm$ .023	.493 $\pm$ .018
		LOCUS	.653 $\pm$ .063	.678 $\pm$ .048	.306 $\pm$ .011	.302 $\pm$ .009
Kit.	108	Random	.171 $\pm$ .034	.312 $\pm$ .038	.434 $\pm$ .022	.292 $\pm$ .016
		Vis. CLIP	.065 $\pm$ .006	.108 $\pm$ .009	.458 $\pm$ .001	.336 $\pm$ .001

TABLE II: Per-room breakdown IRA and ECO.

n

= episodes per room (201 total). mean

\pm

std over 30 seeds, LOCUS: training seeds, baseline: tiebreak seeds.

		Vis. CLIP		LOCUS
Query	$n$	IRA $\uparrow$	ECO $\downarrow$	IRA $\uparrow$	ECO $\downarrow$
fork	5	.00	16.8	.43	4.5
knife	5	.00	16.4	.25	3.4
spoon	5	.00	14.2	.47	3.0
butterknife	5	.00	17.4	.48	4.0
spatula	5	.00	13.6	.41	5.1

TABLE III : Per-query IRA and ECO.

n

= episodes per query.

Method	Class (%) $\uparrow$	Inst. (%) $\uparrow$	Inst./Class $\uparrow$
Constant: cabinets first	74.6 ∗	21.9 $\pm$ 2.4 ∗∗∗	29%
Random	63.3 $\pm$ 2.6 ∗∗∗	20.1 $\pm$ 2.6 ∗∗∗	32%
Visual CLIP	60.2 ∗∗∗	18.4 ∗∗∗	31%
Class Text CLIP	52.2 ∗∗∗	20.9 ∗∗∗	40%
TidyBot commonsense	92.0 ${}^{\text{ns}}$	33.4 $\pm$ 2.6 ∗∗∗	36%
TidyBot summarization	89.6 ${}^{\text{ns}}$	34.0 $\pm$ 2.4 ∗∗∗	38%

TABLE IV: Class-level against instance-level accuracy, iTHOR test set. Superscripts are two-sided paired

t

-tests on per-query values against LOCUS (

K{=}2

) (

{}^{***}p{<}.001

{}^{**}p{<}.01

{}^{*}p{<}.05

{}^{\text{ns}}

otherwise)

Room	Correct	Right type, wrong inst.	Wrong type
Overall	48.6 $\pm$ 2.5	44.6 $\pm$ 3.9	6.8 $\pm$ 4.8
Kitchen	47.3 $\pm$ 4.2	43.0 $\pm$ 7.1	9.7 $\pm$ 8.8
Living room	40.7 $\pm$ 4.6	50.7 $\pm$ 4.5	8.6 $\pm$ 0.8
Bedroom	43.3 $\pm$ 4.8	56.7 $\pm$ 4.8	0.0 $\pm$ 0.0
Bathroom	65.0 $\pm$ 6.3	31.4 $\pm$ 5.6	3.6 $\pm$ 1.7

TABLE V: Error taxonomy across 201 test episodes (mean

\pm

std, 30 seeds).

	ECO/ $N$ $\downarrow$		TypeAcc $\uparrow$
Method	Ours	HouseKeep	Ours	HouseKeep
Constant: cabinets first	.484	.248 $\pm$ .007	74.6%	98.9%
Random (5 seeds)	.455	.321 $\pm$ .015	63.3%	70.3%
Visual CLIP	.466	.322 $\pm$ .025	60.2%	58.3%
Text CLIP	.436	.301 $\pm$ .009	52.2%	43.4%
TidyBot commonsense (zero-shot)	.338	.257 $\pm$ .012	92.0%	87.2%

TABLE VI: ECO/

N

and TypeAcc from both label generators (HouseKeep: 119 eps.; ours: 201 eps.)

Condition	IRA $\uparrow$	S-IRA $\uparrow$	ECO $\downarrow$	S-ECO $\downarrow$
$-$ sldist, $K{=}0$	.267	.412	3.259	2.587
$-$ sldist, $K{=}2$ , uniform	.333	.442	2.706	2.259
$-$ sldist, $K{=}2$ , $e^{-d}$	.321	.412	2.642	2.189
$-$ height, $K{=}0$	.339	.448	3.502	2.612
$-$ height, $K{=}2$	.358	.455	3.393	2.527
Full, $K{=}0$	.406	.491	2.776	2.259

TABLE VII: Message passing vs. proximity and height features, best checkpoint. Best per column in bold .

	IRA $\uparrow$		ECO/ $N$ $\downarrow$
Method	iTHOR	RoboCasa	iTHOR	RoboCasa
Random	.241	.037	.455	.507
Visual CLIP	.188	.048	.465	.519
Class Text CLIP	.182	.075	.436	.373
LOCUS ( $K\!=\!2$ )	.367	.134 $\pm$ .031	.257	.314 $\pm$ .025

TABLE VIII: Sim-to-sim transfer: iTHOR test set (201 episodes) against RoboCasa (10 layouts

\times

3 styles

\times

5 seeds, 1,188 episodes). LOCUS is trained on iTHOR only, with no finetuning.

Panel of Figure 11 — Fig. 3 : ROS pipeline. The scene graph is built during exploration then fed to the GNN.

Method	Split	IRA $\uparrow$	ECO/ $N$ $\downarrow$
Random	In-dist	0.023	0.515
	Novel	0.032	0.542
Visual CLIP	In-dist	0.049	0.492
	Novel	0.035	0.545
Class Text CLIP	In-dist	0.147	0.273
	Novel	0.000	0.485

TABLE IX: In-distribution vs. novel (OOD) query split in RoboCasa.

Panel of Figure 13 — Fig. 4 : The physical kitchen used for the hardware trials, and the scene graph built by Stretch-3 after exploration. Ranked scene graph nodes (green:best) and node reachability (green:reachable, red:unreachable).

	ECO $\downarrow$
Query	Text CLIP	Vis. CLIP	LOCUS
Fork	11.0	1.0	1.0
Mug	2.0	3.0	1.0
Sponge	8.5	4.0	6.5

TABLE X: Per-query Stretch 3 in a real kitchen.

Robotic Object Search

See all 22 papers

EmbodimentSemantic: A Spatial Scene-Graph Dataset and Benchmark for Vision-Language Models on Embodied Manipulation Trajectories
Jun 6, 2026Hassan Jaber, Refinath S N, Luca Cagliero +2Visually Grounded Reasoning Visual Spatial Reasoning
Politecnico di Torino, Italy · Huawei, Noah’s Ark Lab, United Kingdom · University College London, United Kingdom
SpatialCLI: Learning to Reason With Spatial Tools, Then Without Them
Jul 30, 2026Yang Zhou, Zixuan Huang, Sunzhu Li +10Tool Use in VLMs Visual Spatial Reasoning
Zhejiang University · Zhuoyu Technology, Shenzhen, China · Beihang University +3
Relational Semantic Reasoning on 3D Scene Graphs for Open World Interactive Object Search
Mar 5, 2026Imen Mahdi, Matteo Cassinelli, Fabien Despinoy +2Robotic Object Search Robot Navigation
University of Freiburg · Toyota Motor Europe

Jun 6, 2026J/K move · Enter open · S save

EmbodimentSemantic: A Spatial Scene-Graph Dataset and Benchmark for Vision-Language Models on Embodied Manipulation Trajectories

Hassan Jaber, Refinath S N, Luca Cagliero, Christopher E. Mower+1

Politecnico di Torino, Italy · Huawei, Noah’s Ark Lab, United Kingdom · University College London, United Kingdom

Visually Grounded Reasoning Visual Spatial Reasoning Robot Manipulation Benchmarks Scene Graph Generation VLMs for Robotics 3D Visual Grounding+1

Open paper AskPDF arXiv

Spatial grounding remains a key limitation of vision-language-action (VLA) systems for robotic manipulation. While current models can recognize objects and follow language instructions, they often lack an explicit representation of how objects are arranged in space, including support, containment, ordering, occlusion, and depth-sensitive relations. We introduce EmbodimentSemantic, a spatial scene-graph dataset and benchmark for evaluating relational grounding in embodied manipulation. EmbodimentSemantic represents scenes as directed object-relation-object triplets, where each triplet specifies a spatial relation between an ordered pair of objects using a fixed set of relations. This representation enables direct evaluation of object binding, relation prediction, and spatial consistency. The dataset includes real-world manipulation observations collected with the low-cost SO101 robot arm, together with generated scene graphs for studying spatial grounding in practical robotic settings. To provide controlled validation, we also introduce a simulator-grounded LIBERO benchmark with over 60K manipulation frames and more than 120K camera-specific scene graphs across paired third-person and wrist views, where ground-truth relations are derived automatically from MuJoCo geometry, world coordinates, camera projections, and visibility constraints. We further test whether scene graphs improve downstream control by injecting them into existing VLA policy prompts. Experiments across open-source and commercial VLMs show that current models often predict plausible relations but struggle with exact depth-aware and viewpoint-dependent spatial structure. EmbodimentSemantic provides a unified framework for diagnosing spatial grounding in VLM perception and testing its utility for VLA manipulation.

Abstract

Robotic Object Search

Explore similar work

EmbodimentSemantic: A Spatial Scene-Graph Dataset and Benchmark for Vision-Language Models on Embodied Manipulation Trajectories

SpatialCLI: Learning to Reason With Spatial Tools, Then Without Them

Relational Semantic Reasoning on 3D Scene Graphs for Open World Interactive Object Search