UniGround: Universal 3D Visual Grounding via Training-Free Scene Parsing

Jiaxi Zhang; Yunheng Wang; Wei Lu; Taowen Wang; Shuning Zhang; Yixiao Feng; Junzhe Xu; Shunwang Sun

cs.ROMar 9, 2026

UniGround: Universal 3D Visual Grounding via Training-Free Scene Parsing

Authors: Jiaxi Zhang, Yunheng Wang, Wei Lu, Taowen Wang, Shuning Zhang, Yixiao Feng, Junzhe Xu, Shunwang Sun,

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · JD Explore Academy · Zhejiang University

Abstract

3D Visual Grounding (3DVG) localizes objects from natural-language descriptions in 3D scenes and is fundamental to embodied AI applications. Although foundation models enable open-vocabulary reasoning, they typically rely on pre-generated candidates, creating two sequential bottlenecks. The candidate bottleneck occurs when dataset-specific 3D proposal models miss, fragment, or incorrectly group targets under distribution shifts, excluding them from VLM reasoning. The evidence bottleneck stems from incomplete visual evidence: global renderings preserve spatial context but obscure object details, whereas candidate-centric views capture local appearance but lack global context. To address these bottlenecks, we propose UniGround, a zero-shot 3DVG framework that addresses both bottlenecks through Global Candidate Filtering and Contextual Precision Grounding. Global Candidate Filtering constructs topology-consistent, class-agnostic candidates from 3D topology and multi-view semantic cues, without dataset-trained 3D detectors, task-specific proposal supervision, or predefined box and category priors. Contextual Precision Grounding jointly reasons over global spatial context and candidate-centric visual evidence, followed by closed-loop consistency verification for reliable target identification. UniGround achieves 46.1%/34.1% [email protected]/0.5 on ScanRefer and 28.7% [email protected] on the evaluated ARKitScenes subset of EmbodiedScan. Further experiments demonstrate competitive grounding without dataset-specific 3D priors, cross-dataset generalization to unseen indoor scenes, and robustness to real-world reconstruction noise and practical domain shifts.

Figures & tables

Figure 1: Overview of UniGround. Top left: Prior methods depend on dataset-specific proposals and limited visual evidence. Bottom left: UniGround combines training-free candidate filtering with contextual precision grounding. Right: Real-world deployment.

Figure 2: Overview of UniGround. Global Candidate Filtering constructs topology-consistent, query-relevant candidates without supervised 3D proposals. Contextual Precision Grounding identifies the referent by integrating global context, candidate-centric evidence, and closed-loop consistency verification.

Method	Agent		Prior		Unique		Multiple		ScanRefer		ES
	Rep.	VLM	Box	Cls.	.25	.50	.25	.50	.25	.50	.25
Fully Supervised Learning-Based Methods
ScanRefer Chen et al. [2020]	✗	✗	–	–	67.6	46.2	32.1	21.3	39.0	26.1	12.9
InstanceRefer Yuan et al. [2021]	✗	✗	–	–	77.5	66.8	31.3	24.8	40.2	32.9	–
3DVG-T Zhao et al. [2021]	✗	✗	–	–	77.2	58.5	38.4	28.7	45.9	34.5	–
BUTD-DETR Jain et al. [2022]	✗	✗	–	–	84.2	66.3	46.6	35.1	52.2	39.8	22.1

Table 1: Evaluations of 3DVG on the ScanRefer validation set and a subset of the EmbodiedScan set within ARKitScenes. “ES” denotes EmbodiedScan; .25 and .50 denote [email protected] and [email protected], respectively.

Figure 4: Qualitative comparison on ARKitScenes under domain shift. Purple/green boxes denote incorrect/correct predictions. UniGround improves candidate construction and disambiguation through topology-aware proposals and global–local evidence.

Figure 5: Qualitative results from the real-world indoor case study. UniGround localizes referred targets in previously unseen Conference, Lounge, Office, and Corridor environments under real-capture noise, imperfect reconstruction, and layout shifts.

Figure 6: Sensitivity to the number of candidates retained by Global Candidate Filtering.

Method	Conference	Lounge	Office	Corridor
SeeGround Li et al. [2025b]	2/25	0/25	5/25	0/25
SeqVLM Lin et al. [2025a]	0/25	0/25	3/25	0/25
Ours	10/25	13/25	15/25	5/25

Table 2: Evaluations of 3D visual grounding in four real-world environments.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Setting	Role
Frame sampling	Stride $\max(1,\lfloor N_{f}/1000\rfloor)$	Adapts image processing density to the available RGB sequence.
Geometric initialization	VCCS resolution $0.05$ ,m	Forms topology-preserving primitives before instance aggregation.
Progressive aggregation	Five stages; similarity $0.9\!\rightarrow\!0.5$	Merges confident fragments early and recovers more complete instances progressively.
Multi-view encoding	10 views; visibility threshold $0.2$	Accumulates semantic evidence across observations of an object hypothesis.
Mask prompting	10 rounds; 5 points; 5 crop levels; expansion $0.1$	Improves coverage across object scales and partial observations.
Exported visual evidence	3 diverse views per candidate	Supplies compact, object-centric evidence to Stage 2.

Appendix

Table 4: Principal Stage 1 implementation settings. They are fixed for the evaluated scenes and do not require task-specific training.

Module	Prompt Content	Purpose
System Role	You are a helpful assistant designed to identify objects based on images and descriptions.	Defines the VLM as an object-grounding agent.
Visual Inputs	A stitched global render provides scene-level spatial context (IDs and boxes), while stitched candidate images grouped by object ID provide object-centric visual evidence.	Combines global layout cues and local semantic evidence for robust grounding.
Grounding Context	Candidate IDs, target name, anchor name, optional anchor bbox, and each candidate’s 3D center and size are provided.	Defines the grounding search space and explicit spatial metadata.
Decision Protocol	First identify candidates from candidate views, then verify the final target using global spatial relations and 3D coordinates; if ambiguous, resolve via anchor-first spatial reasoning.	Separates semantic identification from spatial verification and improves ambiguity handling.
Response Format	Predicted ID, Explanation, Candidate Names, Target name matches?, Anchor ID, Relation used.	Ensures deterministic parsing and interpretable reasoning traces.

Appendix

Table 5: Agent definition prompt used in UniGround. The prompt explains visual inputs, grounding context, decision rules, and response constraints for the VLM.

Field	Description
Predicted ID	The final object ID selected as the grounding result.
Explanation	A concise explanation describing the appearance and spatial cues used in the decision.
Candidate Names	Semantic names assigned to all candidate objects, formatted as ID -> short phrase .
Target name matches?	Indicates whether the selected candidate semantically matches the queried target category.
Anchor ID	The anchor object used during spatial reasoning, or unknown if no anchor is used.
Relation used	The spatial relation applied in the final decision (e.g., left of, near, behind).

Appendix

Table 6: Structured output fields used for deterministic parsing of VLM reasoning results.

Example VLM Output (Real Inference Trace)

Query: a window that sits above the dresser cabinet. there is an air conditioner in the window.

Ground-truth ID: 22 Predicted ID: 22

Predicted 3D Box (center + size): [0.41773347556591034, 2.9167184829711914, 1.1108443140983582, 0.4044213593006134, 1.1485633850097656, 0.8206886053085327]

Ground-truth 3D Box (center + size): [0.39585648477077484, 3.0089060068130493, 1.1411845982074738, 0.3606673777103424, 0.9361202716827393, 0.7455374598503113]

unique: null

Explanation: ID 22 visually shows a window with a window air conditioner installed. In the 3D data it is positioned higher (Z 1.11) and directly above the dresser/radiator cabinet (ID 20, Z 0.49), matching “a window that sits above the dresser cabinet with an air conditioner in it.”

Appendix

Table 7: Example VLM reasoning output in our structured format. This case correctly grounds the query to ID 22.

Figure 7: Examples of global multi-view scene renderings used as scene-level spatial context for the VLM. Each row shows a real grounding case with three candidate-set-centered rendered views from different azimuths. Candidate IDs are overlaid on the scene, the queried anchor is marked in red, and coordinate axes are visualized to support relation-aware reasoning.

Method	Tasks	[email protected]	[email protected]
VLM-Grounder	250	51.6	32.8
UniGround	250	66.4	48.1

Appendix

Table 8: Comparison on the same 250-task subset used by VLM-Grounder. Both accuracy columns are percentages.

Configuration	Global	Candidate	Structured
Full (Spa.+Sem.+Vis.)	Yes	Yes	Yes
No Global	No	Yes	Yes
No Candidate	Yes	No	Yes
Simple Prompt	Yes	Yes	No

Appendix

Table 9: Implementation mapping of the Stage 2 component ablations. Global denotes the scene render, Candidate denotes candidate-centric RGB, and Structured denotes the full reasoning prompt.

Figure 8: Qualitative comparison of Stage 1 candidate construction across four previously unseen indoor scenes. From top to bottom, each column shows the language instruction, the instance partition and selected proposal from a reference trained 3D-mask front end (purple), and the training-free UniGround partition and retained proposal (green). The examples highlight same-category confusion and small targets that are absorbed into larger supporting objects by the reference proposals, while UniGround exposes target-aligned candidates for downstream grounding.

Candidates	Latency (s)	Tokens
3	30.61	1927.67
5	38.95	5737.66
7	49.09	7200.53
10	59.82	9597.21

Appendix

Table 10: Measured Stage 2 inference cost as the number of retained candidates increases.

Figure 9: Representative real-world grounding outputs from comparison predictions, UniGround, and the annotated targets.

Target Object	Instruction
cardboard box	Locate the cardboard box positioned to the right of the black box .
	Identify the cardboard box situated on the right side of the black box .
	Look for the cardboard box that is placed immediately to the right of the black box .
	Spot the cardboard box sitting adjacent to the right of the black box .
	Find the cardboard box located near the right-hand side of the black box .
black office chair	Find the black office chair closest to the door .

Appendix

Table 11: Examples of task instructions for the Office scene in real-world environments. For each target, five diverse natural language queries are provided to evaluate the model’s grounding consistency. Bold text denotes the target object, while underlined text indicates the spatial anchor used for relational reasoning.

3D Visual Grounding

Three-dimensional visual grounding links language or visual queries to specific objects, parts, regions, and spatial relations in 3D scenes.

See all 141 papers

AgentGrounder: Zero-Shot 3D Visual Pointcloud Grounding using Multimodal Language Models
May 25, 2026Cuong Huynh, Maxim Popov, Denis Gridusov +13D Visual Grounding
Biomechatronics and Energy-Efficient Robotics (BE2R) Lab, ITMO University, Saint Petersburg, Russia
Multiple Consistent 2D-3D Mappings for Robust Zero-Shot 3D Visual Grounding
Apr 29, 2026Yufei Yin, Jie Zheng, Qianke Meng +73D VLMs 3D Visual Grounding
Hangzhou Dianzi University Hangzhou, Zhejiang, China · Zhejiang University Hangzhou, Zhejiang, China · Alibaba Group Hangzhou, Zhejiang, China
PruneGround: Plug-and-play Spatial Pruning for 3D Visual Grounding
Jun 30, 2026Duc Cao Dinh, Khai Le-Duc, Florent Draye +43D Spatial Reasoning Visually Grounded Reasoning
Knovel Engineering Lab · University of Toronto · Vector Institute +3

May 25, 2026J/K move · Enter open · S save

AgentGrounder: Zero-Shot 3D Visual Pointcloud Grounding using Multimodal Language Models

Cuong Huynh, Maxim Popov, Denis Gridusov, Sergey Kolyubin

Biomechatronics and Energy-Efficient Robotics (BE2R) Lab, ITMO University, Saint Petersburg, Russia

3D Visual Grounding

Open paper AskPDF arXiv

3D Visual Grounding (3DVG) is an essential capability for embodied AI, requiring agents to localize objects in 3D scenes based on natural language descriptions. Recent zero-shot methods leverage 2D vision-language models (LVLMs). However, they often rely on existing sets of multi-view images and struggle with the limited semantic and spatial details provided by standard 3D segmentation tools. We present

\textbf{AgentGrounder}

, a zero-shot 3D visual grounding framework that operates directly on colored point clouds without task-specific 3D training. Our approach follows a two-stage design: (1) an offline stage that applies 3D model to build an Object Lookup Table (OLT) with instance IDs, semantic labels, 3D bounding boxes; and (2) an online tool-driven agent that decomposes each query, retrieves only relevant candidates from the OLT, performs geometric scoring, and triggers image rendering on demand when additional visual evidence (e.g., color, material, or viewpoint-sensitive cues) is required. Compared with fixed anchor-target matching pipelines, this design reduces cascading matching errors and improves context-window efficiency by avoiding prompts overloaded with irrelevant objects. We evaluate on ScanRefer and Nr3D under a zero-shot setting and observe consistent improvements over SeeGround in our setup, including +2.5% [email protected] on ScanRefer and +6.3% on Nr3D, with a notable +6.3% gain on Nr3D view-independent queries. These results show that combining selective retrieval, geometric reasoning, and adaptive visual inspection yields a practical and robust foundation for open-vocabulary 3D grounding. Our code is available at https://github.com/be2rlab/AgentGrounder.

Abstract

3D Visual Grounding

Explore similar work

AgentGrounder: Zero-Shot 3D Visual Pointcloud Grounding using Multimodal Language Models

Multiple Consistent 2D-3D Mappings for Robust Zero-Shot 3D Visual Grounding

PruneGround: Plug-and-play Spatial Pruning for 3D Visual Grounding