Fast and Reliable Open-Vocabulary Object Search with Absence Detection from Robot Memory

Nicholas Paul; Devson Butani; Eric Martinson

cs.RONov 5, 2026

Fast and Reliable Open-Vocabulary Object Search with Absence Detection from Robot Memory

Authors: Nicholas Paul, Devson Butani, Eric Martinson

Organizations: Lawrence Technological University, Southfield, MI, USA.

Abstract

Mobile manipulators deployed in everyday human environments, such as homes for assistive care or flexible automation, often need to locate specific objects on demand from accumulated observations. Equally important, they must reliably determine when a target is absent before committing to navigation or interaction. Existing approaches demand either slow exhaustive re-processing or expensive 3D representations, and are evaluated on recall and accuracy rather than specificity, leaving their ability to abstain on missing targets unmeasured. We find existing systems that search using text or visual embedding almost never abstain on absent objects achieving at most 9% specificity. To support accurate search at speed, we present a two-stage architecture that decouples fast candidate retrieval from verified selection, enabling both high recall and reliable absence detection from sequential RGB observations with pose. We evaluate eight retrieval-selection configurations plus an exhaustive search baseline across ScanNet apartment scenes, including significant numbers of absent objects. Adding a visual verification step to caption-based retrieval matches or exceeds exhaustive search reliability while requiring an order of magnitude fewer detector calls. We reproduce ScanNet scores on a physical robot operating entirely on-board with a single consumer GPU running in real time.

Figures & tables

Fig. 1: System overview. Observations are indexed in a vector DB. Stage 1 retrieves the top- $K$ candidates; Stage 2 selects a final answer or returns "not found."

Method	Retrieval	Reject?	Visual?
CLIP/Top-1	CLIP	No	No
CLIP/LLM	CLIP	Yes	No
CLIP/SAM3	CLIP	Yes	Yes
CLIP/LLM+SAM3	CLIP	Yes	Yes
Caption/Top-1	Caption	No	No
Caption/LLM	Caption	Yes	No

TABLE I: All evaluated method configurations with key properties.

Fig. 2: Retrieval Hit@K for CLIP, LLM-orchestrated caption search, and the SAM3 exhaustive baseline (ranked by detection confidence). CLIP leads at $K{=}1$ but caption search overtakes at $K{\geq}3$ , suggesting broader coverage of relevant observations. All three curves exhibit an elbow at $K{=}5$ . SAM3 has no gain from $K{=}5$ to $K{=}10$ and the retrieval methods add only $3$ to $6$ points beyond it, motivating our choice of $K{=}5$ as the operating point.

Method	$F_{1s}$	$F_{1}$	Specificity	Precision	Recall
CLIP/Top-1	0.0%	66.3%	0.0%	56.0%	81.2%
CLIP/SAM3	68.4%	69.0%	83.3%	85.1%	58.0%
CLIP/LLM	15.6%	64.6%	8.7%	55.8%	76.8%
CLIP/LLM+SAM3	76.2%	79.1%	78.6%	85.0%	73.9%
Caption/Top-1	0.0%	54.4%	0.0%	46.0%	66.7%
Caption/SAM3	75.5%	75.9%	92.5%	93.6%	63.8%

TABLE II: Aggregate results across all 8 ScanNet scenes (averaged over 100 queries per scene). Best score in each column is bold .

Fig. 3: False positive analysis. Methods without visual verification (Top-1, LLM) produce false positives on nearly all negative queries. SAM3 verification (SAM3, LLM+SAM3) dramatically reduces false positives; the no-SAM3 methods have no visual verification step and therefore cannot reject semantically plausible but absent objects.

Fig. 4: Recall on generic (short noun phrases) vs. non-generic (descriptive) queries by method. LLM-based selection methods maintain recall across query types; vision-only (SAM3) selection degrades significantly on non-generic queries.

Fig. 5: $F_{1s}$ scores on the real-robot scene (36 queries, 21 positive, 15 negative). Caption/SAM3 achieves the highest $F_{1s}$ of 89.7%, exceeding its ScanNet counterpart and the exhaustive SAM3-all baseline.

Method	$F_{1s}$	Specificity	Recall
CLIP/Top-1	0.0%	0.0%	47.6%
CLIP/SAM3	75.0%	95.2%	61.9%
CLIP/LLM	7.4%	4.0%	52.4%
CLIP/LLM+SAM3	68.8%	86.4%	57.1%
Caption/Top-1	0.0%	0.0%	66.7%
Caption/SAM3	89.7%	94.1%	85.7%

TABLE III: Results on the real-robot scene (36 queries: 21 positive, 15 negative).

Fig. 6: An American-flag-themed chair requiring joint color, pattern, and object-class reasoning. Retrieved by CLIP as the 3rd highest scoring result after 2 false positives (left). SAM3 verified the 3rd result and the correct target location was selected (right).

Fig. 7: A "17 Equations" poster on the wall in a department office and the robot’s selected target pose based on the query "17 Equations Poster" .

Stage	Time
Retrieval (CLIP or caption)	$<0.1$ s
SAM3 inference ( $K{=}5$ )	${\approx}2$ s
LLM reasoning (Gemma 3 12B)	${\approx}8$ s
End-to-end: SAM3-exhaustive	${\approx}3$ min
End-to-end: CLIP+SAM3	${\approx}2$ s
End-to-end: LLM+SAM3	${\approx}10$ s

TABLE IV: Per-query latency on on-board hardware (GTX 1080 Ti).

Robotic Object Search

See all 29 papers

Retrospective Open-Vocabulary Memory for Long-Term Object Search
Sep 29, 2026Jiaming Wang, Zhiwei Xue, Chen Jizhuo +2Robotic Object Search Robotic Spatial Memory
National University of Singapore
RoboFind: Multi-Agent Personalized Object Search for People Who Are Blind or Have Low Vision
Sep 17, 2026Ruiping Liu, Shaofang Quan, Qian Yin +10Assistive Robotics Multi-Agent Systems
The authors are with the Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology, Karlsruhe 76131, Germany. · The author is with the School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, Changsha 410012, China. · The author is with Mercedes-Benz.
Selective Commitment for Language-Guided Object Retrieval under Partial Observability
Sep 19, 2026Wonhee Koh, Sushil Samuel Dinesh, Hansol Ko +2Vision-Language Retrieval Active Vision for Robot Manipulation
School of Electrical, Computing, and Software Engineering, University of Arizona, Tucson, AZ 85721, USA · Department of Electrical and Computer Engineering, King Abdullah University of Science and Technology (KAUST), Thuwal 23955, Saudi Arabia

Sep 29, 2026J/K move · Enter open · S save

Retrospective Open-Vocabulary Memory for Long-Term Object Search

Jiaming Wang, Zhiwei Xue, Chen Jizhuo, Peng Shiqi+1

National University of Singapore

Robotic Object Search Robotic Spatial Memory Memory-Augmented Agents Learning with Missing Data

Open paper AskPDF arXiv

Long-term object search requires learning where objects usually appear from repeated but uneven observations of a changing environment. We formulate retrospective open-vocabulary memory as probabilistic inference from censored observations, where the key idea is to reason with evidence per opportunity: a detection or non-detection should influence belief only in proportion to the robot's opportunity to observe the corresponding location. We introduce ECROM, which uses this principle to estimate long-term prevalence for concepts specified only at query time and converts the resulting belief directly into an active-search prior. To evaluate this problem, we introduce a controlled long-term benchmark in ten HM3D homes that independently varies object placement and observation opportunity across repeated traversals. ECROM improves support-level AP on held-out queries by 4.5 points and search SPL by 4.2 points over the strongest competing memory in each metric. The benchmark, dataset, and code will be open-sourced. Project page: https://jiaming.im/ecrom/

Abstract

Robotic Object Search

Explore similar work

Retrospective Open-Vocabulary Memory for Long-Term Object Search

RoboFind: Multi-Agent Personalized Object Search for People Who Are Blind or Have Low Vision

Selective Commitment for Language-Guided Object Retrieval under Partial Observability