When searching for an object, humans naturally combine semantic intuition - reasoning about where the object is likely to be stored - with geometric awareness of what parts of the space are visible, occluded, or unexplored. Effective robot object search requires the same joint reasoning, yet existing methods rely on only one: geometric methods maintain principled spatial beliefs but lack deep semantic understanding, while language-model based methods leverage commonsense knowledge but maintain no explicit belief over space. We argue that jointly reasoning over semantic and geometric evidence in an explicit 3D belief enables human-like interactive object search that outperforms either modality alone. To test this claim, we build a system that accumulates per-object VLM plausibility scores and geometric occupancy evidence via Bayesian log-odds filtering into a unified 3D voxel belief field. High-belief regions in the combined belief field drive action selection at each step: taking a closer look from a different viewpoint, navigating toward an unexplored region of the workspace, or physically interacting with a closed cabinet. We deploy this system end-to-end on a bimanual mobile humanoid and find that it outperforms geometry-only search and LLM-based planning methods. Project website - https://skymanaditya1.github.io/objsearch-website/
Figures & tables
Fig. 2: System overview. Given a natural-language query q (e.g., “find the stapler” ) and an RGB-D observation It , ssf maintains two persistent structures: the object-based memory O , which tracks object hypotheses and their per-object semantic log-odds, and the Belief Map B , which fuses geometric evidence G from occupancy raycasting with semantic evidence S derived from the obm ’s VLM scores. Action Module spatially clusters high-belief cells in B , generates one candidate action per selected cluster, and selects the action a∗ with the highest expected revealed belief mass Mrev . The corresponding skill is dispatched from the Skill Repository (here, look_at ). The observation It+1 returned by the action updates O and then B .
Fig. 3: Skill library. open_cabinet and open_drawer are learned via behavior cloning; explore navigates to frontier viewpoints; look_at repositions the base, torso, and head to uncover occluded voxels. The four pick_* skills (bottom row) retrieve objects from cabinets, drawers, shelves, and tables.
Fig. 4: Experimental setup and task settings. Top: Easy and hard initial viewpoints. The easy viewpoint exposes most of the workspace; hard viewpoints initially face away from the target object’s office or snack area. Bottom: Examples of visible, cluttered, and container tasks in both areas.
Fig. 5: Belief progression and action selection. Starting from the hard viewpoint, ssf searches for duct tape inside a closed drawer, executing two exploration actions before opening it. The top row shows robot states and actions; the belief maps below guide the selection of subsequent actions.
Fig. 6: Outcomes and failure modes. Results for ssf (left), the geometry-only baseline (middle), and the LLM planner (right), with 16 trials per method. Flows connect trial outcomes to task settings and, for failed trials, to failure categories. Flow widths represent trial counts; percentages are computed over the 16 trials shown for each method.
Ours
Geometry-only
LLM-planner
Setting
SR ↑
NA ↓
SR ↑
NA ↓
SR ↑
NA ↓
Visible
4/4
2.5
4/4
2.5
2/4
2.0
Clutter
4/4
4.0
2/4
6.0
3/4
3.0
Container
4/4
3.5
2/4
4.5
1/4
5.0
Floor
2/4
5.0
0/4
–
1/4
4.0
Overall
14/16
3.6
8/16
3.9
7/16
3.1
Table I: Quantitative results with a 15-minute search budget. SR is reported as successful trials over total trials; each setting comprises 2 object categories × 2 viewpoints. NA is averaged over successful trials only.
Open-world interactive object search in household environments requires understanding semantic relationships between objects and their surrounding context to guide exploration efficiently. Prior methods either rely on vision-language embeddings similarity, which does not reliably capture task-relevant relational semantics, or large language models (LLMs), which are too slow and costly for real-time deployment. We introduce SCOUT: Scene Graph-Based Exploration with Learned Utility for Open-World Interactive Object Search, a novel method that searches directly over 3D scene graphs by assigning utility scores to rooms, frontiers, and objects using relational exploration heuristics such as room-object containment and object-object co-occurrence. To make this practical without sacrificing open-vocabulary generalization, we propose an offline procedural distillation framework that extracts structured relational knowledge from LLMs into lightweight models for on-robot inference. Furthermore, we present SymSearch, a scalable symbolic benchmark for evaluating semantic reasoning in interactive object search tasks. Extensive evaluations across symbolic and simulation environments show that SCOUT outperforms embedding similarity-based methods and matches LLM-level performance while remaining computationally efficient. Finally, real-world experiments demonstrate effective transfer to physical environments, enabling open-world interactive object search under realistic sensing and navigation constraints.
Dynamic interactive object search in large-scale human environments presents substantial challenges for existing methods. Current scene representations like 3D Scene Graphs (3DSG) only provide coarse-grained spatial segmentation and cannot identify functional areas such as storage or leisure areas. Without functional area understanding, existing methods are constrained to exhaustive sequential exploration at large scales, resulting in inefficient search behaviors -- particularly in open-layout environments with numerous interactive objects such as drawers and cabinets. Moreover, these methods lack adaptability to environmental dynamics such as object relocations. To address these limitations, this paper proposes CMAR-search, a novel framework built upon Commonsense and Memory Augmented Reasoning (CMAR). Our approach leverages commonsense about area functionalities and aggregates environmental memory to construct a Functional 3D Scene Graph (F3DSG), which organizes the environment into functional areas with their associated containers. Through this structured representation, CMAR enables hierarchical action planning at both macro-area and micro-container levels, empowering the system to efficiently identify and inspect semantically relevant areas for effective object search. Notably, CMAR continuously integrates real-time perception, accumulated memory, and commonsense to dynamically relocalize objects in changing environments. Extensive experiments in simulation and real-world settings demonstrate that CMAR-search significantly surpasses state-of-the-art baselines in both success rate and search efficiency for object search in dynamic interactive environments.
Kaiyao Liao, Qingfeng Li, Xinlei Zhang +3
School of Computer Science and Engineering, Beihang University · Zhejiang Key Laboratory of Industrial Big Data and Robot Intelligent Systems, Hangzhou Innovation Institute of Beihang University
Robots searching for a target from a natural-language description must determine not only where to search, but also which observed candidate is the desired target. These decisions reflect two distinct sources of uncertainty - spatial uncertainty over candidate locations and semantic uncertainty over target identity - that are often conflated in VLM-based search systems. We introduce a spatial-semantic uncertainty formulation that maintains separate beliefs over each component and integrates probabilistic VLM evidence into a global target-identity posterior, including probability mass for undiscovered targets. This decomposition allows an information-theoretic planner to independently value candidate discovery and target disambiguation through spatial and semantic expected information gain (EIG), providing an explicit mechanism for trading broader exploration against earlier identification. We evaluate six VLM uncertainty-elicitation interfaces on 500 synthetic targets and show that similar recognition accuracy can conceal substantial differences in calibration and false confidence. In degraded-observation search-and-identify experiments, EIG-based planners reach confident decisions in 75.0%-92.5% of trials, compared with 20.0% for Random search, while different spatial-semantic weightings achieve comparable identification accuracy once confidence is attained. Increasing semantic emphasis reduces unnecessary exploration and VLM queries, demonstrating that explicitly planning over semantic uncertainty can accelerate target resolution without sacrificing decision quality. These results highlight the distinct roles of uncertainty representation and uncertainty-driven planning in embodied VLM systems.
Alkesh K. Srivastava, Jonathan Diller, Vijay Kumar +1
Temple University, Philadelphia, PA, 19122 USA · University of Pennsylvania, Philadelphia, PA, 19104 USA