Finder: Agentic Closed-Loop Object Finding for Embodied Grounding

Shixiong Xu; Zhiyuan Chen; Song Ding; Rui Luo; Xiaowei Liang; Dongxu Miao; Zhiying Du

cs.CVSep 16, 2026

Finder: Agentic Closed-Loop Object Finding for Embodied Grounding

Authors: Shixiong Xu, Zhiyuan Chen, Song Ding, Rui Luo, Xiaowei Liang, Dongxu Miao, Zhiying Du

Organizations: Xiaomi Robotics

Abstract

Finding the object referred to by language in a partially observed 3D scene is a core capability for embodied agents. Existing approaches either couple object search with online exploration, which can be costly when relevant observations have already been captured, or query pre-built open-vocabulary maps and scene graphs in a static, one-shot fashion. We present Finder, an agentic closed-loop object-finding primitive for embodied grounding. Instead of treating grounding as passive retrieval from a fixed scene representation, Finder maintains a typed loop state that links query-conditioned planning, scoped evidence gathering, candidate verification, and accept/continue/abort control. When evidence is incomplete or ambiguous, the loop can redirect subsequent perception and comparison rather than simply returning the top retrieved object. On open-vocabulary embodied Object Retrieval in Habitat/HM3D and real-world RGB-D scenes, Finder improves the averaged 1m success rate by 15.75 points over strong baselines. The same primitive also transfers to sequential object grounding and embodied object-centric question answering, improving spatial and temporal localization without changing the inner grounding protocol. Project page: https://finder-vln.github.io.

Figures & tables

Figure 1: Overview of Finder . Object grounding is a core embodied primitive requiring multi-source context. Finder realizes it as a reusable closed-loop inner protocol, wrapped by an outer harness for task- and session-level context injection.

Figure 2: Core execution protocol of Finder . The reusable unit is a typed closed loop: staged execution produces explicit request, plan, detection, context, selection, decision, hint, and snapshot objects, while only the judge can advance the episode to accept, abort, or another planning round.

Setting	Metric	Definition
Object Retrieval	[email protected] $\uparrow$	category-correct retrieval rate with $\\|\hat{\mathbf{x}}-\mathbf{x}^{\star}\\|_{2}<0.5$ m
	S@1 $\uparrow$	category-correct retrieval rate with $\\|\hat{\mathbf{x}}-\mathbf{x}^{\star}\\|_{2}<1.0$ m
	LocErr $\downarrow$	mean $\\|\hat{\mathbf{x}}-\mathbf{x}^{\star}\\|_{2}$ over successful S@1 queries
Sequential Grounding	s-acc $\uparrow$	fraction of subtasks with predicted pose inside the GT bbox
Sequential Grounding	t-acc $\uparrow$	fraction of tasks with all subtask poses inside the GT bboxes
Spatio-Temporal QA	QA Acc. $\uparrow$	answer match rate, i.e., $\hat{a}=a^{\star}$

Table 1: Evaluation metrics grouped by task setting.

	Simulated			Real-world			Average
Method	[email protected] $\uparrow$	S@1 $\uparrow$	Err $\downarrow$	[email protected] $\uparrow$	S@1 $\uparrow$	Err $\downarrow$	[email protected] $\uparrow$	S@1 $\uparrow$	Err $\downarrow$
HOV-SG [ 1 ]	33.03	50.90	0.409	31.71	42.68	0.391	32.96	50.49	0.408
DualMap [ 9 ]	38.26	51.94	0.340	9.76	19.51	0.494	36.83	50.31	0.348
FSR-VLN [ 35 ]	32.65	48.90	0.419	12.20	24.39	0.532	31.62	47.67	0.422
Finder (Ours)	61.23	67.29	0.187	35.37	46.34	0.339	59.93	66.24	0.193

Table 2: Object Retrieval results on simulated and real-world scenes. S@

r

denotes category-correct Success@

r

meters. Err denotes LocErr. Best in bold , second underlined .

Table 5

	[email protected] $\uparrow$			S@1 $\uparrow$			LocErr $\downarrow$
Variant	Simple	Hard	Avg	Simple	Hard	Avg	Simple	Hard	Avg
Single-shot	60.91	55.35	58.26	66.95	59.81	63.55	0.187	0.168	0.178
No Feedback	59.80	55.07	57.55	66.58	59.95	63.42	0.196	0.173	0.186
LLM Sel. Planner	60.79	54.67	57.87	67.32	60.22	63.94	0.192	0.178	0.186
No Support Targets	62.76	53.18	58.19	69.30	59.40	64.58	0.187	0.189	0.188
Rule Selector	58.20	49.80	54.19	65.72	55.89	61.03	0.204	0.195	0.200

Table 5: Main ablation on the simulated split. Metrics are reported for simple, hard, and averaged queries; variant definitions are listed in Appendix Table 8 .

Figure 3: Design-space studies for Finder’s default budgets. Left: perception budget. Right: phrase budget. The selected defaults balance performance and efficiency.

Figure 4: Qualitative examples. (a) Support-aware disambiguation. (b) Judge-guided cross-loop refinement. (c) Failure cases: missed target recall and anchor-attribute mismatch.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Default	Role
Loop budget	3 loops	Allows retry after zero detections, weak support evidence, or unstable candidates.
Frame cap	200 frames	Caps perception cost after scope-based routing and resampling.
Frame stride	10	Reduces redundant adjacent views before budgeted detection.
Primary phrases	2	Provides compact phrase diversity for the target object.
Support phrases	2	Adds relational/context evidence without dominating the target search.
Support execution	Adaptive	Lets the planner choose global, candidate-local, conditional, or no support search.

Appendix

Table 6: Default control settings used by Finder in the Object Retrieval benchmark.

	35B-A3B		122B-A10B		397B-A17B
Variant	S@1 $\uparrow$	LocErr $\downarrow$	S@1 $\uparrow$	LocErr $\downarrow$	S@1 $\uparrow$	LocErr $\downarrow$
MiMo-v2-Pro / Qwen	64.71	0.194	66.65	0.188	67.29	0.187
MiMo-v2.5 / Qwen	66.06	0.190	66.90	0.184	67.16	0.193
MiMo-v2.5-Pro / Qwen	68.69	0.191	67.81	0.190	66.97	0.186
Matched Qwen / Qwen	65.10	0.189	63.10	0.189	68.00	0.193

Appendix

Table 7: API-family ablation on the simulated retrieval split, pairing MiMo planner families with Qwen VLM choices. Cells report S@1 / LocErr; best in bold , second underlined .

Variant	Change
Single-shot	Set max_loops=1 .
No Feedback	Clear carried notes and next-loop hints between loops.
LLM Sel. Planner	Replace the default structured selection planner with an LLM-based planner.
No Support Targets	Remove supporting targets and candidate-local support follow-up.
Rule Selector	Replace VLM visual verification with a simple heuristic selector.
Rule Judge	Replace the VLM loop judge with a lightweight accept/continue heuristic.

Appendix

Table 8: Variant-to-change mapping for the main ablation in Table 5 .

Figure 5: Object Retrieval dataset overview. Left: query distribution across object categories, showing a long-tailed benchmark with 1,632 queries over 197 categories. Right: per-scene query counts and the simple/hard split, with 811 simple and 821 hard queries across 9 simulated and 4 real-world scenes.

Statistic	Simulated	Real-world
Number of scenes	9	4
Total rooms	122	4
Avg. rooms per scene	13.6	1.0
Total target objects	1,890	82
Avg. target objects per scene	210.0	20.5
Indexed RGB-D views	31,520	67

Appendix

Table 9: Dataset statistics for the Object Retrieval splits. Object counts refer to retrievable target objects after filtering. Category counts follow the query-target category canonicalization used throughout the benchmark.

Field	Prompt Constraint
Target binding	Each query must refer to one annotated target object with a canonical category and 3D centroid.
Simple query	Include available room/floor scope when it is part of the intended prior.
Hard query	Do not reveal room/floor metadata; use visible appearance, relation, or context to identify the target.
Ambiguity control	If multiple same-category objects exist, wording should preserve the intended ambiguity level without becoming ungrounded.
Quality control	Reject queries whose target is not visible in the evidence views or whose wording cannot be checked from visual context.

Appendix

Table 10: Query-writing prompt constraints used during dataset construction.

Panel of Figure 6 — Figure 6: Annotation and verification interface for real-world data.

Panel of Figure 7 — Figure 7: Annotation and verification interface for simulated data.

			HOV-SG			DualMap			FSR-VLN			Finder
Scene	Split	Num	[email protected]	S@1	Err	[email protected]	S@1	Err	[email protected]	S@1	Err	[email protected]	S@1	Err
00824-Dd4bFSTQ8gi	Hard	91	20.88	39.56	0.504	25.27	43.96	0.424	36.26	51.65	0.420	61.54	71.43	0.222
00824-Dd4bFSTQ8gi	Simple	101	32.67	53.47	0.397	31.68	49.50	0.382	37.62	62.38	0.424	58.42	69.31	0.255
00824-Dd4bFSTQ8gi	Overall	192	27.08	46.88	0.440	28.64	46.87	0.402	36.98	57.29	0.422	59.90	70.31	0.239
00829-QaLdnwvtxbs	Hard	55	18.18	32.73	0.404	23.64	34.55	0.322	30.91	45.45	0.449	49.09	54.55	0.187
00829-QaLdnwvtxbs	Simple	67	25.37	41.79	0.466	38.81	50.75	0.289	50.75	68.66	0.366	53.73	59.70	0.215

Appendix

Table 11: Per-scene Object Retrieval results on simulated scenes, split by hard, simple, and combined protocols. Values are percentages for [email protected] and S@1; Err is mean distance in meters over S@1 successes.

		HOV-SG			DualMap			FSR-VLN			Finder
Scene	Num	[email protected]	S@1	Err	[email protected]	S@1	Err	[email protected]	S@1	Err	[email protected]	S@1	Err
icra_ic3f	18	33.33	38.89	0.330	16.67	27.78	0.345	11.11	16.67	0.497	44.44	50.00	0.326
icra_ic4f	16	31.25	43.75	0.453	25.00	37.50	0.445	18.75	43.75	0.633	50.00	62.50	0.306
icra_ic7f	30	26.67	36.67	0.434	3.33	16.67	0.610	10.00	23.33	0.492	20.00	36.67	0.409
icra_sh3f	18	38.89	55.56	0.343	0.00	0.00	–	11.11	16.67	0.425	38.89	44.44	0.300
Query Weighted	82	31.71	42.68	0.391	9.76	19.51	0.494	12.20	24.39	0.532	35.37	46.34	0.339

Appendix

Table 12: Per-scene Object Retrieval results on real-world scenes. The real benchmark currently contains only hard queries, so no simple split is reported. Values are percentages for [email protected] and S@1; Err is mean distance in meters over S@1 successes.

Type	Examples	Queries	[email protected] $\uparrow$	S@1 $\uparrow$	S@3 $\uparrow$	$T_{q}\downarrow$
Large fixed	bed, sofa, cabinet, table	836	59.45	65.91	81.10	80.5s
Small clutter	cup, bottle, vase, toy	368	55.43	62.23	70.11	96.1s
Thin / flat	book, plate, tray, wall art	201	64.68	67.16	74.63	86.0s
Soft items	pillow, towel, bag, clothing	227	64.76	73.13	82.38	92.8s

Appendix

Table 13: Object-type response analysis for Finder on the aggregated Object Retrieval benchmark. Queries are grouped by target-object profile rather than dataset split.

Explore similar work

May 25, 2026cs.CV

AgentGrounder: Zero-Shot 3D Visual Pointcloud Grounding using Multimodal Language Models

3D Visual Grounding (3DVG) is an essential capability for embodied AI, requiring agents to localize objects in 3D scenes based on natural language descriptions. Recent zero-shot methods leverage 2D vision-language models (LVLMs). However, they often rely on existing sets of multi-view images and struggle with the limited semantic and spatial details provided by standard 3D segmentation tools. We present

\textbf{AgentGrounder}

, a zero-shot 3D visual grounding framework that operates directly on colored point clouds without task-specific 3D training. Our approach follows a two-stage design: (1) an offline stage that applies 3D model to build an Object Lookup Table (OLT) with instance IDs, semantic labels, 3D bounding boxes; and (2) an online tool-driven agent that decomposes each query, retrieves only relevant candidates from the OLT, performs geometric scoring, and triggers image rendering on demand when additional visual evidence (e.g., color, material, or viewpoint-sensitive cues) is required. Compared with fixed anchor-target matching pipelines, this design reduces cascading matching errors and improves context-window efficiency by avoiding prompts overloaded with irrelevant objects. We evaluate on ScanRefer and Nr3D under a zero-shot setting and observe consistent improvements over SeeGround in our setup, including +2.5% [email protected] on ScanRefer and +6.3% on Nr3D, with a notable +6.3% gain on Nr3D view-independent queries. These results show that combining selective retrieval, geometric reasoning, and adaptive visual inspection yields a practical and robust foundation for open-vocabulary 3D grounding. Our code is available at https://github.com/be2rlab/AgentGrounder.

Cuong Huynh, Maxim Popov, Denis Gridusov +1

Biomechatronics and Energy-Efficient Robotics (BE2R) Lab, ITMO University, Saint Petersburg, Russia

3D Visual Grounding

Ask PDF

Dec 28, 2025cs.CV

OpenGround: Planning-based Online Perception for Open-World 3D Visual Grounding

3D visual grounding aims to locate objects based on natural language descriptions in 3D scenes. Existing supervised methods are limited by generalization and recent zero-shot methods typically rely on a predefined Object Lookup Table (OLT) to query Visual Language Models (VLMs) for reasoning about object locations via a single step grounding, which limits the applications in scenarios with undefined targets and complex queries. To address these problems, we present OpenGround, a novel zero-shot framework for open-world 3D visual grounding that remains compatible with recent zero-shot methods. OpenGround integrates Task-Chain Planning to decompose a query into a plan of context-to-target sub-goals for progressive grounding, and Context-Guided Perception to perceive novel objects online under context guidance from the task chain. We also propose a new dataset named OpenTarget, which contains over 7000 object-description pairs to mimic open-world evaluation. Extensive experiments demonstrate that OpenGround achieves competitive performance on Nr3D, state-of-the-art on ScanRefer, and delivers a substantial 17.6% improvement on OpenTarget. Project Page at https://why-102.github.io/openground.io/.

Wenyuan Huang, Zhenyu Zhang, Zhao Wang +4

Nanjing University, School of Intelligent Science and Technology, Nanjing, China · China Mobile Zijin Innovation Institute, Nanjing, China

3D VLMs 3D Visual Grounding

Ask PDF

Aug 13, 2026cs.RO

FUSE: Active Functional Affordance Grounding through Adaptive Semantic-Geometric Evidence Acquisition

Embodied agents must often identify and interact with objects based on their function rather than their identity, requiring them to actively acquire observations that reveal discriminative functional evidence. Existing affordance grounding methods operate from fixed viewpoints and lack mechanisms for deciding where to look when functional cues are occluded or incomplete. We introduce Active Functional Affordance Grounding, a new task in which an agent sequentially explores a scene to identify and spatially ground an object satisfying a functional query. To address this problem, we propose FUSE, an adaptive semantic-geometric evidence acquisition framework that combines explicit uncertainty-driven exploration with a learned amortized planner to efficiently select informative viewpoints. We further introduce a Habitat-based benchmark for evaluating active functional grounding. Experiments show that FUSE achieves the highest observed non-oracle grounding performance while reducing computation by 1.33x relative to fully explicit exploration, and remains effective across multiple affordance knowledge sources.

Zhou Chen, Sathyanarayanan N. Aakur

CSSE Department Auburn University, Auburn, AL, 36849

Next-Best-View Planning Active Sensing Visual Grounding Active View Selection

Ask PDF