Think, Act, Build: An Agentic Framework with Vision Language Models for Zero-Shot 3D Visual Grounding

Haibo Wang; Zihao Lin; Zhiyang Xu; Lifu Huang

cs.CVApr 1, 2026

Think, Act, Build: An Agentic Framework with Vision Language Models for Zero-Shot 3D Visual Grounding

Authors: Haibo Wang, Zihao Lin, Zhiyang Xu, Lifu Huang

Organizations: University of California, Davis · Virginia Tech

Abstract

3D Visual Grounding (3D-VG) aims to localize objects in 3D scenes via natural language descriptions. While recent advancements leveraging Vision-Language Models (VLMs) have explored zero-shot possibilities, they typically suffer from a static workflow relying on preprocessed 3D point clouds, essentially degrading grounding into proposal matching. To bypass this reliance, our core motivation is to decouple the task: leveraging 2D VLMs to resolve complex spatial semantics, while relying on deterministic multi-view geometry to instantiate the 3D structure. Driven by this insight, we propose "Think, Act, Build (TAB)", a dynamic agentic framework that reformulates 3D-VG tasks as a generative 2D-to-3D reconstruction paradigm operating directly on raw RGB-D streams. Specifically, guided by a specialized 3D-VG skill, our VLM agent dynamically invokes visual tools to track and reconstruct the target across 2D frames. Crucially, to overcome the multi-view coverage deficit caused by strict VLM semantic tracking, we introduce the Semantic-Anchored Geometric Expansion, a mechanism that first anchors the target in a reference video clip and then leverages multi-view geometry to propagate its spatial location across unobserved frames. This enables the agent to "Build" the target's 3D representation by aggregating these multi-view features via camera parameters, directly mapping 2D visual cues to 3D coordinates. Furthermore, to ensure rigorous assessment, we identify flaws such as reference ambiguity and category errors in existing benchmarks and manually refine the incorrect queries. Extensive experiments on ScanRefer and Nr3D demonstrate that our framework, relying entirely on open-source models, significantly outperforms previous zero-shot methods and even surpasses fully supervised baselines.

Figures & tables

Figure 1: (1) Top: Previous methods rely on preprocessed 3D point clouds, degrading the task into proposal matching. (2) Bottom: Our TAB operates directly on RGB-D streams. Through an iterative Think-Act-Build process, the agent reconstructs the target object.

Figure 2: The " Think, Act, Build (TAB) " framework. Guided by an expert 3D-VG Skill, the VLM agent reasons about the task, invokes visual tools, and reconstructs the target.

Figure 3: Examples of annotation noise in benchmarks.

Method	LLM/VLM	w/o	Unique		Multiple		Overall
Method	LLM/VLM	PC.	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]	[email protected]
Fully-Supervised Methods (One/Two-Stage Based)
ScanRefer ( Chen et al., 2020 )	-	✗	67.6	46.2	32.1	21.3	39.0	26.1
3DVG-T ( Zhao et al., 2021 )	-	✗	77.2	58.5	38.4	28.7	45.9	34.5
BUTD-DETR ( Jain et al., 2022 )	-	✗	84.2	66.3	46.6	35.1	52.2	39.8
EDA ( Wu et al., 2023 )	-	✗	85.8	68.6	49.1	37.6	54.6	42.3

Table 1: 3D Visual Grounding results on ScanRefer. "w/o PC" denotes methods that do not rely on 3D point clouds as input. † means results with two stage training.

Method	LLM/VLM	w/o PC.	Easy	Hard	Dep.	Indep.	Overall
Fully-Supervised Methods
ReferIt3DNet ( Achlioptas et al., 2020 )	-	✗	43.6	27.9	32.5	37.1	35.6
InstanceRefer ( Yuan et al., 2021 )	-	✗	46.0	31.8	34.5	41.9	38.8
3DVG-T ( Zhao et al., 2021 )	-	✗	48.5	34.8	34.8	43.7	40.8
EDA ( Wu et al., 2023 )	-	✗	58.2	46.1	50.2	53.1	52.1
BUTD-DETR ( Jain et al., 2022 )	-	✗	54.6	60.7	48.4	46.0	58.0

Table 2: 3D Visual Grounding results on Nr3D (w/o GT object class). "w/o PC" denotes methods that do not rely on 3D point clouds as input.

#	Modules		ScanRefer						Nr3D
	STE	MGE	Unique		Multiple		Overall		Easy	Hard	Dep.	Indep.	Overall
	STE	MGE	@0.25	@0.5	@0.25	@0.5	@0.25	@0.5	Easy	Hard	Dep.	Indep.	Overall
(a)	✗	✗	57.6	29.3	32.3	19.0	41.6	22.8	58.8	43.8	51.0	52.6	52.0
(b)	✗	✓	65.2	41.3	41.1	22.2	50.0	29.2	62.0	47.1	48.9	59.1	55.1
(c)	✓	✗	69.6	45.7	51.3	30.4	58.0	36.0	67.6	49.1	61.5	57.8	59.2
(d)	✓	✓	90.2	57.6	60.1	39.9	71.2	46.4	72.1	63.2	62.5	71.4	68.0

Table 3: Ablation study on the Semantic-Anchored Geometric Expansion. STE : Semantic Temporal Expansion. MGE : Multi-View Geometric Expansion.

Figure 4: Qualitative comparison of different components in TAB .

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Tool Name	Description
query_parse()	Parses the natural language query into a structured JSON format containing target_class , attributes , conditions , and scene_feature .
read_image_files()	Scans the local directory for a specific scene and indexes all image file paths into a structured list.
object_filter()	Filters candidate images using GroundingDINO to retain only frames containing the requested target_class .
vlm_filter()	Utilizes a Vision-Language Model to verify if an image strictly satisfies the global scene constraints.
vlm_score()	Scores and ranks candidate images based on how well their visual contents match the query’s attributes and spatial conditions.
argmax_image_and_seg_id()	Select the best candidate image and identify the specific target object ID within it. It iterates through the images (from highest score), generates segmentation masks using the SAM, and uses the VLM to pinpoint the specific object ID matching the query.

Appendix

Table 4: The complete registry of specialized tools available to the agent.

3D Visual Grounding

Three-dimensional visual grounding links language or visual queries to specific objects, parts, regions, and spatial relations in 3D scenes.

See all 134 papers

Apr 29, 2026

Multiple Consistent 2D-3D Mappings for Robust Zero-Shot 3D Visual Grounding

Zero-shot 3D Visual Grounding (3DVG) is a critical capability for open-world embodied AI. However, existing methods are fundamentally bottlenecked by the poor quality of open-vocabulary 3D proposals, suffering from inaccurate categories and imprecise geometries, as well as the spatial redundancy of exhaustive multi-view reasoning. To address these challenges, we propose MCM-VG, a novel framework that achieves robust zero-shot 3DVG by explicitly establishing Multiple Consistent 2D-3D Mappings. Instead of passively relying on noisy 3D segments, MCM-VG enforces 2D-3D consistency across three fundamental dimensions to achieve precise target localization and reliable reasoning. First, a Semantic Alignment module corrects category mismatches via LLM-driven query parsing and coarse-to-fine 2D-3D matching. Second, an Instance Rectification module leverages VLM-guided 2D segmentations to reconstruct missing targets, back-projecting these reliable visual priors to establish accurate 3D geometries. Finally, to eliminate spatial redundancy, a Viewpoint Distillation module clusters 3D camera directions to extract optimal frames. By pairing these optimal RGB frames with Bird's Eye View maps into concise visual prompt sets, we formulate the final target disambiguation as a multiple-choice reasoning task for Vision-Language Models. Extensive evaluations on ScanRefer and Nr3D benchmarks demonstrate that MCM-VG sets a new state-of-the-art for zero-shot 3D visual grounding. Remarkably, it achieves 62.0% and 53.6% in [email protected] and [email protected] on ScanRefer, outperforming previous baselines by substantial margins of 6.4% and 4.0%.

Yufei Yin, Jie Zheng, Qianke Meng +7

Hangzhou Dianzi University Hangzhou, Zhejiang, China · Zhejiang University Hangzhou, Zhejiang, China · Alibaba Group Hangzhou, Zhejiang, China

3D VLMs 3D Visual Grounding

Ask PDF

May 25, 2026

AgentGrounder: Zero-Shot 3D Visual Pointcloud Grounding using Multimodal Language Models

3D Visual Grounding (3DVG) is an essential capability for embodied AI, requiring agents to localize objects in 3D scenes based on natural language descriptions. Recent zero-shot methods leverage 2D vision-language models (LVLMs). However, they often rely on existing sets of multi-view images and struggle with the limited semantic and spatial details provided by standard 3D segmentation tools. We present

\textbf{AgentGrounder}

, a zero-shot 3D visual grounding framework that operates directly on colored point clouds without task-specific 3D training. Our approach follows a two-stage design: (1) an offline stage that applies 3D model to build an Object Lookup Table (OLT) with instance IDs, semantic labels, 3D bounding boxes; and (2) an online tool-driven agent that decomposes each query, retrieves only relevant candidates from the OLT, performs geometric scoring, and triggers image rendering on demand when additional visual evidence (e.g., color, material, or viewpoint-sensitive cues) is required. Compared with fixed anchor-target matching pipelines, this design reduces cascading matching errors and improves context-window efficiency by avoiding prompts overloaded with irrelevant objects. We evaluate on ScanRefer and Nr3D under a zero-shot setting and observe consistent improvements over SeeGround in our setup, including +2.5% [email protected] on ScanRefer and +6.3% on Nr3D, with a notable +6.3% gain on Nr3D view-independent queries. These results show that combining selective retrieval, geometric reasoning, and adaptive visual inspection yields a practical and robust foundation for open-vocabulary 3D grounding. Our code is available at https://github.com/be2rlab/AgentGrounder.

Cuong Huynh, Maxim Popov, Denis Gridusov +1

Biomechatronics and Energy-Efficient Robotics (BE2R) Lab, ITMO University, Saint Petersburg, Russia

3D Visual Grounding

Ask PDF

Sep 4, 2026

Where to Look Matters: Learning Influential Views for VLM-based 3D Visual Grounding

Recent zero-shot 3D visual grounding methods leverage vision-language models (VLMs) to localize objects in 3D scenes from natural language queries. However, these methods typically rely on heuristic rules to select which camera views are provided to the VLM, often prioritizing object visibility rather than grounding relevance. We present IVSGround, a framework that learns Influential View Selection for VLM-based 3D visual grounding. Instead of using fixed heuristics, a lightweight view selector is trained to identify views that provide discriminative evidence for grounding. To obtain supervision signals, we generate training signals using feedback from a reasoning VLM through a two-stage rejection sampling process. During inference, the learned selector predicts query-conditioned influential views for each candidate object, which are then evaluated by a frozen reasoning VLM through comparative grounding. Experiments on ScanRefer and NR3D show that IVSGround consistently improves grounding accuracy over existing zero-shot pipelines, demonstrating that selecting where to look is crucial for effective 3D visual grounding. Project page: https://ivsground.github.io/

Tsung-Chih Chiang, Hsuan-Kung Yang, Jou-Min Liu +4

National Tsing Hua University, Hsinchu, Taiwan · Woven by Toyota, Inc., Tokyo, Japan · National Taiwan University, Taipei, Taiwan

3D Visual Grounding 3D VLMs

Ask PDF

Abstract

3D Visual Grounding

Explore similar work

Multiple Consistent 2D-3D Mappings for Robust Zero-Shot 3D Visual Grounding

AgentGrounder: Zero-Shot 3D Visual Pointcloud Grounding using Multimodal Language Models

Where to Look Matters: Learning Influential Views for VLM-based 3D Visual Grounding