Finder: Agentic Closed-Loop Object Finding for Embodied Grounding

Shixiong Xu; Zhiyuan Chen; Song Ding; Rui Luo; Xiaowei Liang; Dongxu Miao; Zhiying Du

cs.CVSep 16, 2026

Finder: Agentic Closed-Loop Object Finding for Embodied Grounding

Authors: Shixiong Xu, Zhiyuan Chen, Song Ding, Rui Luo, Xiaowei Liang, Dongxu Miao, Zhiying Du

Organizations: Xiaomi Robotics

Abstract

Finding the object referred to by language in a partially observed 3D scene is a core capability for embodied agents. Existing approaches either couple object search with online exploration, which can be costly when relevant observations have already been captured, or query pre-built open-vocabulary maps and scene graphs in a static, one-shot fashion. We present Finder, an agentic closed-loop object-finding primitive for embodied grounding. Instead of treating grounding as passive retrieval from a fixed scene representation, Finder maintains a typed loop state that links query-conditioned planning, scoped evidence gathering, candidate verification, and accept/continue/abort control. When evidence is incomplete or ambiguous, the loop can redirect subsequent perception and comparison rather than simply returning the top retrieved object. On open-vocabulary embodied Object Retrieval in Habitat/HM3D and real-world RGB-D scenes, Finder improves the averaged 1m success rate by 15.75 points over strong baselines. The same primitive also transfers to sequential object grounding and embodied object-centric question answering, improving spatial and temporal localization without changing the inner grounding protocol. Project page: https://finder-vln.github.io.

Figures & tables

Figure 1: Overview of Finder . Object grounding is a core embodied primitive requiring multi-source context. Finder realizes it as a reusable closed-loop inner protocol, wrapped by an outer harness for task- and session-level context injection.

Figure 2: Core execution protocol of Finder . The reusable unit is a typed closed loop: staged execution produces explicit request, plan, detection, context, selection, decision, hint, and snapshot objects, while only the judge can advance the episode to accept, abort, or another planning round.

Setting	Metric	Definition
Object Retrieval	[email protected] $\uparrow$	category-correct retrieval rate with $\\|\hat{\mathbf{x}}-\mathbf{x}^{\star}\\|_{2}<0.5$ m
	S@1 $\uparrow$	category-correct retrieval rate with $\\|\hat{\mathbf{x}}-\mathbf{x}^{\star}\\|_{2}<1.0$ m
	LocErr $\downarrow$	mean $\\|\hat{\mathbf{x}}-\mathbf{x}^{\star}\\|_{2}$ over successful S@1 queries
Sequential Grounding	s-acc $\uparrow$	fraction of subtasks with predicted pose inside the GT bbox
Sequential Grounding	t-acc $\uparrow$	fraction of tasks with all subtask poses inside the GT bboxes
Spatio-Temporal QA	QA Acc. $\uparrow$	answer match rate, i.e., $\hat{a}=a^{\star}$

Table 1: Evaluation metrics grouped by task setting.

	Simulated			Real-world			Average
Method	[email protected] $\uparrow$	S@1 $\uparrow$	Err $\downarrow$	[email protected] $\uparrow$	S@1 $\uparrow$	Err $\downarrow$	[email protected] $\uparrow$	S@1 $\uparrow$	Err $\downarrow$
HOV-SG [ 1 ]	33.03	50.90	0.409	31.71	42.68	0.391	32.96	50.49	0.408
DualMap [ 9 ]	38.26	51.94	0.340	9.76	19.51	0.494	36.83	50.31	0.348
FSR-VLN [ 35 ]	32.65	48.90	0.419	12.20	24.39	0.532	31.62	47.67	0.422
Finder (Ours)	61.23	67.29	0.187	35.37	46.34	0.339	59.93	66.24	0.193

Table 2: Object Retrieval results on simulated and real-world scenes. S@

r

denotes category-correct Success@

r

meters. Err denotes LocErr. Best in bold , second underlined .

Table 5

	[email protected] $\uparrow$			S@1 $\uparrow$			LocErr $\downarrow$
Variant	Simple	Hard	Avg	Simple	Hard	Avg	Simple	Hard	Avg
Single-shot	60.91	55.35	58.26	66.95	59.81	63.55	0.187	0.168	0.178
No Feedback	59.80	55.07	57.55	66.58	59.95	63.42	0.196	0.173	0.186
LLM Sel. Planner	60.79	54.67	57.87	67.32	60.22	63.94	0.192	0.178	0.186
No Support Targets	62.76	53.18	58.19	69.30	59.40	64.58	0.187	0.189	0.188
Rule Selector	58.20	49.80	54.19	65.72	55.89	61.03	0.204	0.195	0.200

Table 5: Main ablation on the simulated split. Metrics are reported for simple, hard, and averaged queries; variant definitions are listed in Appendix Table 8 .

Figure 3: Design-space studies for Finder’s default budgets. Left: perception budget. Right: phrase budget. The selected defaults balance performance and efficiency.

Figure 4: Qualitative examples. (a) Support-aware disambiguation. (b) Judge-guided cross-loop refinement. (c) Failure cases: missed target recall and anchor-attribute mismatch.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Default	Role
Loop budget	3 loops	Allows retry after zero detections, weak support evidence, or unstable candidates.
Frame cap	200 frames	Caps perception cost after scope-based routing and resampling.
Frame stride	10	Reduces redundant adjacent views before budgeted detection.
Primary phrases	2	Provides compact phrase diversity for the target object.
Support phrases	2	Adds relational/context evidence without dominating the target search.
Support execution	Adaptive	Lets the planner choose global, candidate-local, conditional, or no support search.

Appendix

Table 6: Default control settings used by Finder in the Object Retrieval benchmark.

	35B-A3B		122B-A10B		397B-A17B
Variant	S@1 $\uparrow$	LocErr $\downarrow$	S@1 $\uparrow$	LocErr $\downarrow$	S@1 $\uparrow$	LocErr $\downarrow$
MiMo-v2-Pro / Qwen	64.71	0.194	66.65	0.188	67.29	0.187
MiMo-v2.5 / Qwen	66.06	0.190	66.90	0.184	67.16	0.193
MiMo-v2.5-Pro / Qwen	68.69	0.191	67.81	0.190	66.97	0.186
Matched Qwen / Qwen	65.10	0.189	63.10	0.189	68.00	0.193

Appendix

Table 7: API-family ablation on the simulated retrieval split, pairing MiMo planner families with Qwen VLM choices. Cells report S@1 / LocErr; best in bold , second underlined .

Variant	Change
Single-shot	Set max_loops=1 .
No Feedback	Clear carried notes and next-loop hints between loops.
LLM Sel. Planner	Replace the default structured selection planner with an LLM-based planner.
No Support Targets	Remove supporting targets and candidate-local support follow-up.
Rule Selector	Replace VLM visual verification with a simple heuristic selector.
Rule Judge	Replace the VLM loop judge with a lightweight accept/continue heuristic.

Appendix

Table 8: Variant-to-change mapping for the main ablation in Table 5 .

Figure 5: Object Retrieval dataset overview. Left: query distribution across object categories, showing a long-tailed benchmark with 1,632 queries over 197 categories. Right: per-scene query counts and the simple/hard split, with 811 simple and 821 hard queries across 9 simulated and 4 real-world scenes.

Statistic	Simulated	Real-world
Number of scenes	9	4
Total rooms	122	4
Avg. rooms per scene	13.6	1.0
Total target objects	1,890	82
Avg. target objects per scene	210.0	20.5
Indexed RGB-D views	31,520	67

Appendix

Table 9: Dataset statistics for the Object Retrieval splits. Object counts refer to retrievable target objects after filtering. Category counts follow the query-target category canonicalization used throughout the benchmark.

Field	Prompt Constraint
Target binding	Each query must refer to one annotated target object with a canonical category and 3D centroid.
Simple query	Include available room/floor scope when it is part of the intended prior.
Hard query	Do not reveal room/floor metadata; use visible appearance, relation, or context to identify the target.
Ambiguity control	If multiple same-category objects exist, wording should preserve the intended ambiguity level without becoming ungrounded.
Quality control	Reject queries whose target is not visible in the evidence views or whose wording cannot be checked from visual context.

Appendix

Table 10: Query-writing prompt constraints used during dataset construction.

Panel of Figure 6 — Figure 6: Annotation and verification interface for real-world data.

Panel of Figure 7 — Figure 7: Annotation and verification interface for simulated data.

			HOV-SG			DualMap			FSR-VLN			Finder
Scene	Split	Num	[email protected]	S@1	Err	[email protected]	S@1	Err	[email protected]	S@1	Err	[email protected]	S@1	Err
00824-Dd4bFSTQ8gi	Hard	91	20.88	39.56	0.504	25.27	43.96	0.424	36.26	51.65	0.420	61.54	71.43	0.222
00824-Dd4bFSTQ8gi	Simple	101	32.67	53.47	0.397	31.68	49.50	0.382	37.62	62.38	0.424	58.42	69.31	0.255
00824-Dd4bFSTQ8gi	Overall	192	27.08	46.88	0.440	28.64	46.87	0.402	36.98	57.29	0.422	59.90	70.31	0.239
00829-QaLdnwvtxbs	Hard	55	18.18	32.73	0.404	23.64	34.55	0.322	30.91	45.45	0.449	49.09	54.55	0.187
00829-QaLdnwvtxbs	Simple	67	25.37	41.79	0.466	38.81	50.75	0.289	50.75	68.66	0.366	53.73	59.70	0.215

Appendix

Table 11: Per-scene Object Retrieval results on simulated scenes, split by hard, simple, and combined protocols. Values are percentages for [email protected] and S@1; Err is mean distance in meters over S@1 successes.

		HOV-SG			DualMap			FSR-VLN			Finder
Scene	Num	[email protected]	S@1	Err	[email protected]	S@1	Err	[email protected]	S@1	Err	[email protected]	S@1	Err
icra_ic3f	18	33.33	38.89	0.330	16.67	27.78	0.345	11.11	16.67	0.497	44.44	50.00	0.326
icra_ic4f	16	31.25	43.75	0.453	25.00	37.50	0.445	18.75	43.75	0.633	50.00	62.50	0.306
icra_ic7f	30	26.67	36.67	0.434	3.33	16.67	0.610	10.00	23.33	0.492	20.00	36.67	0.409
icra_sh3f	18	38.89	55.56	0.343	0.00	0.00	–	11.11	16.67	0.425	38.89	44.44	0.300
Query Weighted	82	31.71	42.68	0.391	9.76	19.51	0.494	12.20	24.39	0.532	35.37	46.34	0.339

Appendix

Table 12: Per-scene Object Retrieval results on real-world scenes. The real benchmark currently contains only hard queries, so no simple split is reported. Values are percentages for [email protected] and S@1; Err is mean distance in meters over S@1 successes.

Type	Examples	Queries	[email protected] $\uparrow$	S@1 $\uparrow$	S@3 $\uparrow$	$T_{q}\downarrow$
Large fixed	bed, sofa, cabinet, table	836	59.45	65.91	81.10	80.5s
Small clutter	cup, bottle, vase, toy	368	55.43	62.23	70.11	96.1s
Thin / flat	book, plate, tray, wall art	201	64.68	67.16	74.63	86.0s
Soft items	pillow, towel, bag, clothing	227	64.76	73.13	82.38	92.8s

Appendix

Table 13: Object-type response analysis for Finder on the aggregated Object Retrieval benchmark. Queries are grouped by target-object profile rather than dataset split.

Explore similar work

AgentGrounder: Zero-Shot 3D Visual Pointcloud Grounding using Multimodal Language Models
May 25, 2026Cuong Huynh, Maxim Popov, Denis Gridusov +13D Visual Grounding
Biomechatronics and Energy-Efficient Robotics (BE2R) Lab, ITMO University, Saint Petersburg, Russia
OpenGround: Planning-based Online Perception for Open-World 3D Visual Grounding
Dec 28, 2025Wenyuan Huang, Zhenyu Zhang, Zhao Wang +43D VLMs 3D Visual Grounding
Nanjing University, School of Intelligent Science and Technology, Nanjing, China · China Mobile Zijin Innovation Institute, Nanjing, China
FUSE: Active Functional Affordance Grounding through Adaptive Semantic-Geometric Evidence Acquisition
Aug 13, 2026Zhou Chen, Sathyanarayanan N. AakurNext-Best-View Planning Active Sensing
CSSE Department Auburn University, Auburn, AL, 36849

May 25, 2026, cs.CVJ/K move · Enter open · S save

AgentGrounder: Zero-Shot 3D Visual Pointcloud Grounding using Multimodal Language Models

Cuong Huynh, Maxim Popov, Denis Gridusov, Sergey Kolyubin

Biomechatronics and Energy-Efficient Robotics (BE2R) Lab, ITMO University, Saint Petersburg, Russia

3D Visual Grounding

Open paper AskPDF arXiv

3D Visual Grounding (3DVG) is an essential capability for embodied AI, requiring agents to localize objects in 3D scenes based on natural language descriptions. Recent zero-shot methods leverage 2D vision-language models (LVLMs). However, they often rely on existing sets of multi-view images and struggle with the limited semantic and spatial details provided by standard 3D segmentation tools. We present

\textbf{AgentGrounder}

, a zero-shot 3D visual grounding framework that operates directly on colored point clouds without task-specific 3D training. Our approach follows a two-stage design: (1) an offline stage that applies 3D model to build an Object Lookup Table (OLT) with instance IDs, semantic labels, 3D bounding boxes; and (2) an online tool-driven agent that decomposes each query, retrieves only relevant candidates from the OLT, performs geometric scoring, and triggers image rendering on demand when additional visual evidence (e.g., color, material, or viewpoint-sensitive cues) is required. Compared with fixed anchor-target matching pipelines, this design reduces cascading matching errors and improves context-window efficiency by avoiding prompts overloaded with irrelevant objects. We evaluate on ScanRefer and Nr3D under a zero-shot setting and observe consistent improvements over SeeGround in our setup, including +2.5% [email protected] on ScanRefer and +6.3% on Nr3D, with a notable +6.3% gain on Nr3D view-independent queries. These results show that combining selective retrieval, geometric reasoning, and adaptive visual inspection yields a practical and robust foundation for open-vocabulary 3D grounding. Our code is available at https://github.com/be2rlab/AgentGrounder.

Abstract

Explore similar work

AgentGrounder: Zero-Shot 3D Visual Pointcloud Grounding using Multimodal Language Models

OpenGround: Planning-based Online Perception for Open-World 3D Visual Grounding

FUSE: Active Functional Affordance Grounding through Adaptive Semantic-Geometric Evidence Acquisition