When searching for an object, humans naturally combine semantic intuition - reasoning about where the object is likely to be stored - with geometric awareness of what parts of the space are visible, occluded, or unexplored. Effective robot object search requires the same joint reasoning, yet existing methods rely on only one: geometric methods maintain principled spatial beliefs but lack deep semantic understanding, while language-model based methods leverage commonsense knowledge but maintain no explicit belief over space. We argue that jointly reasoning over semantic and geometric evidence in an explicit 3D belief enables human-like interactive object search that outperforms either modality alone. To test this claim, we build a system that accumulates per-object VLM plausibility scores and geometric occupancy evidence via Bayesian log-odds filtering into a unified 3D voxel belief field. High-belief regions in the combined belief field drive action selection at each step: taking a closer look from a different viewpoint, navigating toward an unexplored region of the workspace, or physically interacting with a closed cabinet. We deploy this system end-to-end on a bimanual mobile humanoid and find that it outperforms geometry-only search and LLM-based planning methods. Project website - https://skymanaditya1.github.io/objsearch-website/
Figures & tables
Fig. 2: System overview. Given a natural-language query q (e.g., “find the stapler” ) and an RGB-D observation It , ssf maintains two persistent structures: the object-based memory O , which tracks object hypotheses and their per-object semantic log-odds, and the Belief Map B , which fuses geometric evidence G from occupancy raycasting with semantic evidence S derived from the obm ’s VLM scores. Action Module spatially clusters high-belief cells in B , generates one candidate action per selected cluster, and selects the action a∗ with the highest expected revealed belief mass Mrev . The corresponding skill is dispatched from the Skill Repository (here, look_at ). The observation It+1 returned by the action updates O and then B .
Fig. 3: Skill library. open_cabinet and open_drawer are learned via behavior cloning; explore navigates to frontier viewpoints; look_at repositions the base, torso, and head to uncover occluded voxels. The four pick_* skills (bottom row) retrieve objects from cabinets, drawers, shelves, and tables.
Fig. 4: Experimental setup and task settings. Top: Easy and hard initial viewpoints. The easy viewpoint exposes most of the workspace; hard viewpoints initially face away from the target object’s office or snack area. Bottom: Examples of visible, cluttered, and container tasks in both areas.
Fig. 5: Belief progression and action selection. Starting from the hard viewpoint, ssf searches for duct tape inside a closed drawer, executing two exploration actions before opening it. The top row shows robot states and actions; the belief maps below guide the selection of subsequent actions.
Fig. 6: Outcomes and failure modes. Results for ssf (left), the geometry-only baseline (middle), and the LLM planner (right), with 16 trials per method. Flows connect trial outcomes to task settings and, for failed trials, to failure categories. Flow widths represent trial counts; percentages are computed over the 16 trials shown for each method.
Ours
Geometry-only
LLM-planner
Setting
SR ↑
NA ↓
SR ↑
NA ↓
SR ↑
NA ↓
Visible
4/4
2.5
4/4
2.5
2/4
2.0
Clutter
4/4
4.0
2/4
6.0
3/4
3.0
Container
4/4
3.5
2/4
4.5
1/4
5.0
Floor
2/4
5.0
0/4
–
1/4
4.0
Overall
14/16
3.6
8/16
3.9
7/16
3.1
Table I: Quantitative results with a 15-minute search budget. SR is reported as successful trials over total trials; each setting comprises 2 object categories × 2 viewpoints. NA is averaged over successful trials only.
School of Computer Science and Engineering, Beihang University · Zhejiang Key Laboratory of Industrial Big Data and Robot Intelligent Systems, Hangzhou Innovation Institute of Beihang University