Retrospective Open-Vocabulary Memory for Long-Term Object Search

Jiaming Wang; Zhiwei Xue; Chen Jizhuo; Peng Shiqi; Harold Soh

cs.ROSep 29, 2026

Retrospective Open-Vocabulary Memory for Long-Term Object Search

Authors: Jiaming Wang, Zhiwei Xue, Chen Jizhuo, Peng Shiqi, Harold Soh

Organizations: National University of Singapore

Abstract

Long-term object search requires learning where objects usually appear from repeated but uneven observations of a changing environment. We formulate retrospective open-vocabulary memory as probabilistic inference from censored observations, where the key idea is to reason with evidence per opportunity: a detection or non-detection should influence belief only in proportion to the robot's opportunity to observe the corresponding location. We introduce ECROM, which uses this principle to estimate long-term prevalence for concepts specified only at query time and converts the resulting belief directly into an active-search prior. To evaluate this problem, we introduce a controlled long-term benchmark in ten HM3D homes that independently varies object placement and observation opportunity across repeated traversals. ECROM improves support-level AP on held-out queries by 4.5 points and search SPL by 4.2 points over the strongest competing memory in each metric. The benchmark, dataset, and code will be open-sourced. Project page: https://jiaming.im/ecrom/

Figures & tables

Figure 1: Searching for “a straw hat” in an HM3D home after ten mapping traversals. Two candidate surfaces, the dining table and the kitchen island, have the same number of detector responses but different observation opportunity. Top left : mapping routes (grey), mean observation opportunity of each surface patch (blue), and one ring per candidate surface with one segment per traversal, colored by detector response and exposure; a label $k/n$ gives $k$ responses in the $n$ traversals that exposed the surface (table 2/5, island 2/10), and $\times$ marks surfaces that never held the hat. Other panels : each memory’s belief and the route induced from the same start under a 120 m budget. Ecrom uses censored observations to exploit informative doorway views and verifies the table after only 3.2 m. Ecrom without exposure term and visit frequency ignore how much of each surface was seen and favor the island, reaching the hat after 17.8 m and 21.6 m. The PredictiveGraphs-style persistence filter also favors the island (14.2 m), while RAVEN-style retrieval ranks stored views and revisits the island from two sides before reaching the table (19.2 m). Coverage (no memory) requires 52.8 m.

Panel of Figure 2 — Figure 2: System overview. (1) During query-independent mapping, RGB-D traversals are organized into a place–view atlas with surface supports and per-episode observation opportunities $c_{je}$ . (2) At query time, open-vocabulary detector responses are lifted to supports and combined with opportunity to infer the long-term prevalence posterior $p(\theta_{j}\mid\mathcal{H}_{j},q)$ (Section 4.3 ). (3) The resulting belief guides budgeted active search and is updated after each executed view (Eq. 8 ).

Panel of Figure 3 — Figure 3: Latent model for one support and episode: presence $X$ , observability $V$ , visible presence $Z=XV$ , detector evidence $r$ .

Figure 4: Benchmark examples from five houses, rendered in the benchmark simulator. For every traversal each object is placed on a receptacle drawn from a hidden distribution and settled by physics; routes and gaze are sampled independently of the placements, so an object’s presence cannot cause the camera to look at it.

Method	AP $\uparrow$	Top-5 $\uparrow$	AUROC $\uparrow$	$\mathrm{NLL}_{\mathrm{Ber}}\downarrow$	SR $\uparrow$	SPL $\uparrow$
Ecrom (ours)	.216	.368	.760	.0287	.587	.381
Detection count	.148	.318	.659	–	.440	.330
Visit frequency	.116	.261	.619	.0811	.433	.299
Opportunity-normalized count	.171	.310	.655	.0773	.467	.310
PredictiveGraphs-style (Perpetua ⋆ )	.160	.263	.659	.0302	.533	.339
RAVEN-style retrieval	.150	.340	.600	–	.420	.310

Table 1: Retrospective distribution estimation and active search on LOOP-Bench (50 queries in 10 houses, 20 mapping traversals per house). AP, tie-aware Top-5 mass and AUROC rank the supports against the support-level ground truth; Bernoulli NLL scores each support’s unnormalized prevalence against the probability that the object rests within 45 cm of it (–: the score is not a probability). Search uses three test placements per query (150 trials) with a 120 m budget; SR and SPL average over all 150 trials, and the 22 trials without the object count as failures. Bold marks the best value in each column across all rows; the lower block ablates Ecrom . Paired 95% intervals are in Appendix F .

Panel of Figure 5 — Figure 5: Real-robot experiment. Left : Hello Robot Stretch 3 (image: Hello Robot). Right : the $\approx$ 500 m 2 office floor with the ten mapping traversals (grey) and, for one query, the search routes of Ecrom (green), detection count (blue) and PredictiveGraphs-style (magenta) from the start (yellow dot) to the target (red star). Scale bar 5 m.

Method	SR $\uparrow$	Travel (m) $\downarrow$	Views $\downarrow$
Ecrom (ours)	.93	19.4	4.7
Detection count	.67	29.1	9.3
PredictiveGraphs-style	.73	29.5	8.9

Table 2: Real-robot search on the Stretch 3 over 15 trials per method (five queried objects, three placements each; 20-view budget): success rate, mean travel and mean number of candidate views inspected. Per-query results are in Table 11 .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Original PredictiveGraphs (Perpetua ⋆ )	Our implementation on LOOP-Bench
Relation state	binary object–receptacle relation	binary object–support relation on the shared $0.5$ m supports
Edge instantiation	first accepted detection	first localized accepted positive ( $r_{je}\geq.25$ ) within the mapping traversals; supports without an edge carry the base rate of never-detected supports
Temporal models	persistence $M_{P}$ and emergence $M_{E}$ , log-normal duration mixtures, EM-fitted	unchanged; one to three components selected by AIC, fitted by the reference EM on the calibration histories with gaps preserved
Detector emissions	$P_{M}$ , $P_{F}$	unchanged; pooled rates of the calibration objects
Model selection	Bayesian selection with annealing $\eta$ and forgetting $\gamma$	unchanged; clock is the traversal index, so $\eta$ is per traversal and $\gamma$ per update
Switching prior $f(t)$	FreMEn / LLM schedules over weeks of hourly data	order zero $f_{0}$ from the edge’s own corrected detection fraction, shrunk towards the pooled rate (placements are independent across the 20 traversals)

Appendix

Table 3: PredictiveGraphs-style baseline: what is inherited from the original model, what our setting requires, and what is fitted. All fitting uses the calibration objects only.

Setting	Source of $\Lambda$	Labels	Used for
Simulation	class histograms of all episode–support cells of the calibration objects, pooled over houses	simulator $Z_{je}$	every Ecrom row of Table 1
Robot	score-stratified sample of about 120 stored keyframes per labeled prompt, pooled over prompts; histograms shrunk towards the simulation histograms ( $n_{0}=20$ )	VLM, whole keyframe	Table 2

Appendix

Table 4: Calibration regimes for the score likelihood ratio. Simulator labels exist only for the designated calibration objects; evaluation queries never receive them.

		Views			Opportunity $c_{je}$
Score bin	All	$\leq 11$	12–19	$\geq 20$	$<\frac{1}{3}$	$\frac{1}{3}$ – $\frac{2}{3}$	$\geq\frac{2}{3}$
$[0,.05)$	$-2.17$	$-1.92$	$-2.30$	$-2.57$	$-1.43$	$-2.55$	$-2.26$
$[.05,.15)$	$-0.26$	$+0.09$	$-0.34$	$-1.13$	$+0.26$	$-0.39$	$-0.37$
$[.15,.25)$	$+2.01$	$+2.10$	$+2.02$	$+1.83$	$+1.79$	$+1.95$	$+2.03$
$[.25,.35)$	$+3.13$	$+3.29$	$+2.99$	$+2.85$	$+4.50$	$+3.28$	$+2.94$
$[.35,1]$	$+5.03$	$+4.76$	$+4.94$	$+5.41$	$+5.07$	$+4.70$	$+5.09$

Appendix

Table 5: Sensor-model audit on the calibration objects:

\log\Lambda

per score bin, estimated separately for three ranges of the number of stored keyframes that observed the support in the episode (views) and three ranges of the opportunity

c_{je}

, with half-count smoothing. The last row gives the number of visible cells (

Z_{je}=1

) in each stratum.

Method	AP	Top-5	AUROC
Detection count	+.068 [+.034, +.105]	+.050 [+.016, +.089]	+.101 [+.053, +.151]
Visit frequency	+.101 [+.058, +.147]	+.106 [+.044, +.182]	+.141 [+.090, +.193]
Opportunity-normalized count	+.045 [+.005, +.092]	+.057 [+.001, +.123]	+.105 [+.059, +.154]
PredictiveGraphs-style	+.056 [+.017, +.095]	+.105 [+.012, +.196]	+.101 [+.052, +.151]
RAVEN-style retrieval	+.067 [+.031, +.105]	+.027 [ $-$ .054, +.109]	+.160 [+.113, +.205]
DynaMem-style latest state	+.115 [+.060, +.174]	+.171 [+.085, +.270]	+.132 [+.067, +.199]

Appendix

Table 6: Paired differences, Ecrom minus each method, with 95% house-then-query bootstrap intervals. Positive values favor Ecrom except for

\mathrm{NLL}_{\mathrm{Ber}}

, where negative values do.

Table 12

Queries	$n$	$\Delta$ AP	$\Delta$ Top-5	$\Delta$ AUROC	$\Delta\mathrm{NLL}_{\mathrm{Ber}}$
Least observed third	17	+.057	+.052	+.112	$-$ .0032
Middle third	16	+.020	+.013	+.038	$-$ .0031
Best observed third	17	$-$ .001	+.011	$-$ .024	$-$ .0005

Appendix

Table 9: Gain of the exposure term by how well the true supports were observed: Ecrom minus the variant without exposure term (

c_{je}{=}1

), averaged over the queries of each third.

Component	Simulation	Robot
Mapping traversals	20 per house	10
Likelihood ratio $\Lambda$	calibration objects, simulator labels $Z_{je}$	VLM labels of stored keyframes (whole-image question), shrunk towards the simulation histograms
Live belief update	verified discovery, Eq. 8 , on every support weighted by its exposure $c_{j}(u)$	detector event $\mathbf{1}[r\geq\tau]$ with $p_{\mathrm{det}}$ and $p_{\mathrm{false}}$ , on the one support each frame is assigned to
Success	detector fires and at least 500 target pixels are in the executed view	detector fires and the operator confirms
Budget	120 m of travel	20 inspected views

Appendix

Table 10: Differences between the simulated system of Table 1 and the robot system of Table 2 .

	Ecrom (ours)		Detection count		PredictiveGraphs-style
Query	Success	Travel (m)	Success	Travel (m)	Success	Travel (m)
red cup	3/3	25.4	2/3	49.9	2/3	42.2
luncheon meat	2/3	17.7	0/3	17.1	1/3	29.7
headphones	3/3	24.7	3/3	26.8	3/3	26.9
cupcakes	3/3	20.0	3/3	21.5	3/3	23.1
scissors	3/3	9.0	2/3	30.4	2/3	25.6

Appendix

Table 11: Real-robot search on the Stretch 3 per query: successful trials out of three and mean travel in meters for each memory (mean over the three trials, failed trials included at the distance traveled before stopping). The last two rows aggregate over all 15 trials. Bold marks the best method.

Robotic Object Search

See all 22 papers

FARM: Find Anything using Relational Spatial Memory
Jun 13, 2026Siming He, Leo Huang, Adam Lilja +6Visual Spatial Reasoning Relational Reasoning
1UC Berkeley · 2Stanford University
ECoMEM: Explicit Concept Memory for Memory-Dependent Robot Control
Sep 30, 2026Yize Liu, Ke Wang, Mac Schwager +2Robotic Control Long-Horizon Robotic Manipulation
Stanford University
OVAL: Open-Vocabulary Augmented Memory Model for Lifelong Object Goal Navigation
Apr 14, 2026Jiahua Pei, Yi Liu, Guoping Pan +3Object-Goal Navigation Open-Vocabulary Object Navigation
Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China

Jun 13, 2026J/K move · Enter open · S save

FARM: Find Anything using Relational Spatial Memory

Siming He, Leo Huang, Adam Lilja, Fabio Hubel+5

1UC Berkeley · 2Stanford University

Visual Spatial Reasoning Relational Reasoning Visually Grounded Reasoning Vision-Language Retrieval Robotic Perception Memory-Augmented Agents+2

Open paper AskPDF arXiv

Robots operating in homes, warehouses, and other object-rich environments need memory systems that can find specific object instances on demand. Object-level memory alone is often insufficient: scenes contain many plausibly matching objects, and users refer to the target through relations to landmarks and surrounding objects (e.g. ``the tall lamp below the dartboard and to the left of the poster''), demanding a relational spatial memory that supports retrieval through semantic, appearance, and spatial predicates over objects. To achieve this, we present FARM (Find Anything using Relational Spatial Memory), which builds, in real time at 5-10 Hz, a compact, open-vocabulary, object-level memory with geometry, visual-language descriptors, and viewpoint evidence. At query time, FARM uses VLMs to parse the query and score visual evidence, while grounding spatial constraints explicitly through object symbols and relational predicates. This structured use of VLMs enables more accurate and robust retrieval than end-to-end reasoning over frame histories or scene-graph context. In experiments on 44k language queries spanning 67 indoor and outdoor scenes, ranging from 15 to 15,000 m^2, FARM improves Recall@5 and Recall@10 over prior methods by 164% and 224%, and a final VLM reranking stage improves Accuracy@1 by 35%, while running in real time. We further demonstrate closed-loop deployment on a quadrupedal robot using onboard sensors and compute.

Abstract

Robotic Object Search

Explore similar work

FARM: Find Anything using Relational Spatial Memory

ECoMEM: Explicit Concept Memory for Memory-Dependent Robot Control

OVAL: Open-Vocabulary Augmented Memory Model for Lifelong Object Goal Navigation