Object co-occurrences provide a key cue for finding objects successfully and efficiently in unfamiliar environments. Typically, one looks for cups in kitchens and views fridges as evidence of being in a kitchen. Such priors have also been exploited in artificial agents, but they are typically learned from explicitly labeled data or queried from language models. It is still unclear whether these relations can be learned implicitly from unlabeled observations alone. In this work, we address this problem and propose ProReFF, a feature field model trained to predict relative distributions of features obtained from pre-trained vision language models. In addition, we introduce a learning-based strategy that enables training from unlabeled and potentially contradictory data by aligning inconsistent observations into a coherent relative distribution. For the downstream object search task, we propose an agent that leverages predicted feature distributions as a semantic prior to guide exploration toward regions with a high likelihood of containing the object. We present extensive evaluations demonstrating that ProReFF captures meaningful relative feature distributions in natural scenes and provides insight into the impact of our proposed alignment step. We further evaluate the performance of our search agent in 100 challenges in the Matterport3D simulator, comparing with feature-based baselines and human participants. The proposed agent is 20% more efficient than the strongest baseline and achieves up to 80% of human performance.
Figures & tables
Fig. 1 : Overview of ProReFF. We train a relative feature field, which predicts a mean embedding and a variance, given a query embedding q and a relative offset v . This network is trained in an unsupervised manner from feature point cloud observations. To enable this training, we introduce a learned data alignment model. We demonstrate the utility of ProReFF for object search by using it to infer distributions of features around a target object. These distributions are compared with the agent’s current observations, and a choice is made between following the current observations or inferring further features.
Fig. 2 : Depiction of the ambiguity problem in the data. Observing the same query feature (red dot), from two different locations can lead to contradictory target features (pink dot) given the same offset vector. In the schematic, we illustrate the effect of the learned alignment on this problem. In two separate observation instances, the alignment R rotates the observations such that the query contradiction is resolved.
Fig. 3 : UMAP visualization of ground truth clusters embeddings (left), base model predictions (center), and aligned model predictions (right). Cluster centroids are marked as circled points. The base model exhibits severe mode collapse, whereas the aligned model retains substantially more of the semantic diversity around the query embedding. This demonstrates the effectiveness of the Alignment Network at resolving conflicts in the training data.
Fig. 4 : Distribution of pointwise cosine similarities between ground truth and predicted embeddings. base : predictor trained without Alignment Network. aligned : predictor trained with Alignment Network, tested with positions rotated into canonical frame. aligned original : same aligned model tested with unrotated scene positions, showing frame-dependency of the learned predictions.
Fig. 5 : Cluster-based distance between ground truth and predicted embedding distributions. Lower values indicate better preservation of semantic structure. base : predictor without alignment. Aligned : predictor with Alignment Network using rotated input positions. Aligned original : same model using unrotated scene positions. Aligned sphere : same model with uniformly sampled spherical positions. Baselines : random (random embeddings) and mean (all embeddings set to scene mean).
Fig. 6 : Qualitative impression of similarity difference of predicted features given base model (a) and aligned model (b). Taking a query embedding from a validation scene and querying the model for embeddings of all other points, we observe a stark difference in the semantic accuracy of the predictions. We visualize this here as a heat map (blue = similar, red = dissimilar). In (c), we visualize the impact the rotation predicted by the Alignment Network has on the scene.
Fig. 7 : We compare Query Follower against the ProReFF agent and show the navigation path of both (pink arrows). Agents start at the blue circle, in the first picture the Query Follower travels to the goal (pink circle) by following cosine similarities to the query which we draw on the pointcloud as a heatmap where blue = similar, red = dissimilar. The second and third pictures show the ProReFF Wasserstein distance scores for the feature field at the second level of context expansion (2, more coarse) and the first level of expansion (1, near the query, so most of the points are very dissimilar).
Overall
Single-Floor
Multi-Floor
Method
SR
SPL
SR
SPL
SR
SPL
Random
0.45
0.05
0.6
0.08
0.3
0.03
BFS
0.95
0.25
0.98
0.27
0.92
0.23
DFS
0.90
0.35
0.92
0.41
0.88
0.29
CoW [ 11 ]
0.78
0.30
0.80
0.35
0.76
0.24
Query Follower (CLIP)
0.84
0.42
0.94
0.55
0.74
0.30
TABLE I : Object search results on MatterSim environments ( N=100 episodes). SR = Success Rate; SPL = Success weighted by Path Length (higher is better for both).
Overall
Single-Floor
Multi-Floor
τ
SR
SPL
SR
SPL
SR
SPL
0.20
0.91
0.46
0.96
0.59
0.88
0.34
0.25
0.91
0.48
0.96
0.60
0.86
0.35
0.30
0.94
0.53
0.96
0.63
0.92
0.42
0.35
0.92
0.47
0.96
0.49
0.88
0.45
TABLE II : Ablation on the threshold parameter τ for ProReFF.
Finding the object referred to by language in a partially observed 3D scene is a core capability for embodied agents. Existing approaches either couple object search with online exploration, which can be costly when relevant observations have already been captured, or query pre-built open-vocabulary maps and scene graphs in a static, one-shot fashion. We present Finder, an agentic closed-loop object-finding primitive for embodied grounding. Instead of treating grounding as passive retrieval from a fixed scene representation, Finder maintains a typed loop state that links query-conditioned planning, scoped evidence gathering, candidate verification, and accept/continue/abort control. When evidence is incomplete or ambiguous, the loop can redirect subsequent perception and comparison rather than simply returning the top retrieved object. On open-vocabulary embodied Object Retrieval in Habitat/HM3D and real-world RGB-D scenes, Finder improves the averaged 1m success rate by 15.75 points over strong baselines. The same primitive also transfers to sequential object grounding and embodied object-centric question answering, improving spatial and temporal localization without changing the inner grounding protocol. Project page: https://finder-vln.github.io.
Long-term object search requires learning where objects usually appear from repeated but uneven observations of a changing environment. We formulate retrospective open-vocabulary memory as probabilistic inference from censored observations, where the key idea is to reason with evidence per opportunity: a detection or non-detection should influence belief only in proportion to the robot's opportunity to observe the corresponding location. We introduce ECROM, which uses this principle to estimate long-term prevalence for concepts specified only at query time and converts the resulting belief directly into an active-search prior. To evaluate this problem, we introduce a controlled long-term benchmark in ten HM3D homes that independently varies object placement and observation opportunity across repeated traversals. ECROM improves support-level AP on held-out queries by 4.5 points and search SPL by 4.2 points over the strongest competing memory in each metric. The benchmark, dataset, and code will be open-sourced. Project page: https://jiaming.im/ecrom/
Open-world interactive object search in household environments requires understanding semantic relationships between objects and their surrounding context to guide exploration efficiently. Prior methods either rely on vision-language embeddings similarity, which does not reliably capture task-relevant relational semantics, or large language models (LLMs), which are too slow and costly for real-time deployment. We introduce SCOUT: Scene Graph-Based Exploration with Learned Utility for Open-World Interactive Object Search, a novel method that searches directly over 3D scene graphs by assigning utility scores to rooms, frontiers, and objects using relational exploration heuristics such as room-object containment and object-object co-occurrence. To make this practical without sacrificing open-vocabulary generalization, we propose an offline procedural distillation framework that extracts structured relational knowledge from LLMs into lightweight models for on-robot inference. Furthermore, we present SymSearch, a scalable symbolic benchmark for evaluating semantic reasoning in interactive object search tasks. Extensive evaluations across symbolic and simulation environments show that SCOUT outperforms embedding similarity-based methods and matches LLM-level performance while remaining computationally efficient. Finally, real-world experiments demonstrate effective transfer to physical environments, enabling open-world interactive object search under realistic sensing and navigation constraints.