Urbex: Agentic Spatial Grounding for City-Scale 3D Scenes

Yu Wang; Rui Dai; Wei Chen; Yi Wang; Bo Dang; Kaikui Liu; Xiangxiang Chu; Yansheng Li

cs.CVOct 13, 2026

Urbex: Agentic Spatial Grounding for City-Scale 3D Scenes

Authors: Yu Wang, Rui Dai, Wei Chen, Yi Wang, Bo Dang, Kaikui Liu, Xiangxiang Chu, Yansheng Li

Organizations: Wuhan University · DreamX Team, Alibaba Group

Abstract

City-scale 3D grounding is commonly formulated as candidate ranking, which requires precomputed instance proposals and becomes inefficient in large outdoor scenes. We propose Urbex, an agentic vision-language framework that instead casts grounding as active spatial search. Urbex represents a city-scale 3D scene as an interactive multi-view environment, allowing a VLM agent to query landmarks, zoom into local regions, render oblique views, and commit a grounded bounding box. We optimize this tool-use policy with reinforcement learning on CityRefer, using smooth localization rewards and lightweight shaping rewards for efficient exploration. Experiments show that Urbex achieves the best Hit@1 and IoU on CityRefer, supports candidate-free grounding without ground-truth boxes, and is substantially faster than the strongest city-scale ranking baseline. Zero-shot results on STPLS3D-Refer further indicate improved cross-scene robustness.

Figures & tables

Figure 1: Overview of Urbex. Unlike regular grounding methods that downsample city-scale point clouds or ranking-based methods that score many precomputed instances, Urbex formulates grounding as agentic spatial search. A VLM agent actively interacts with the scene through tool calls, moving between global, zoomed, and oblique views before committing the final target location.

Figure 2: Architecture of Urbex: (left) a multi-view environment exposing global BEV, zoomed, and oblique views; (middle) the policy’s reasoning trace and tool calls over observation $o_{t}$ and interaction history $h_{t}$ ; (right) RL under a terminal reward (IoU and center-distance) plus tool-level shaping.

Figure 3: Qualitative examples on CityRefer. The visualization shows how Urbex moves from global BEV search to local zoom-in and final grounding in city-scale scenes with dense repeated objects and landmark-based references.

Method	ND With GT			NO With GT			Time (s)
Method	Hit@1	Hit@3	Hit@5	Hit@1	Hit@3	Hit@5	Time (s)
Closed-source VLMs w/ Urbex tools
GPT-5.4	9.3	12.4	15.7	13.1	18.6	20.1	New A
Qwen3.6-plus	6.4	9.6	11.9	11.5	16.2	16.8	New A
Indoor 3D grounding methods
InstanceRefer ( Yuan et al., 2021 )	1.6	5.0	10.5	4.9	15.9	24.5	0.6

Table 1: Main results on CityRefer. Urbex supports candidate-free grounding and efficient inference. Five-run Hit@1 statistics are reported in Table 8 .

Figure 4: Qualitative examples on STPLS3D-Refer. The visualization highlights zero-shot transfer cases where the same active-search policy anchors, zooms, verifies, and grounds targets in scenes with different geometry and object distributions from CityRefer.

Method	With GT			Without GT			Time (s)
Method	Hit@1	Hit@3	Hit@5	2D IoU	3D IoU	Point F1	Time (s)
Closed-source VLMs w/ Urbex tools
GPT-5.4	16.3	17.8	19.1	8.0	5.3	12.1	New A
Qwen3.6-plus	12.0	13.8	14.3	8.1	6.4	12.2	New A
Indoor 3D grounding methods
InstanceRefer ( Yuan et al., 2021 )	3.8	7.8	12.3	New A	New A	New A	0.6

Table 2: Zero-shot generalization on STPLS3D-Refer. Table 8 reports five-run Hit@1 statistics.

Table 7

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Tool	Definition	Inputs	Outputs
Landmark Query	Resolves a named urban landmark, such as a building, road, or facility, and uses it as a global spatial anchor.	landmark_name : text span or proper noun; source_view_id : view in which the returned projection should be expressed; optional landmark_type when the name is ambiguous.	Match status; matched landmark name and type; world-space center or footprint when available; projection of the landmark into the requested view.
Zoom-in	Maps a region from an existing BEV view to the original high-resolution image and crops it, exposing detail absent from the global observation.	source_view_id : global or local BEV view; crop_box : normalized 2D box; optional scale or requested output resolution.	Local BEV image; new view_id ; updated view-to-world transform; crop extent in the source and world coordinate frames.
Oblique Rendering	Renders an agent-selected candidate region from a cardinal direction to provide 3D-aware verification of shape, height, facade appearance, and spatial placement.	source_view_id : BEV view containing the candidate; candidate_box : normalized 2D region; direction : one of north, south, east, or west; optional margin around the region.	Oblique image observation; rendering metadata including direction and covered world region; supporting evidence for verification. The oblique view is not used as the coordinate frame for final box submission.

Appendix

Table 5: Definitions, inputs, and outputs of the city-aware tools.

Figure 5: Prompt template for tool-using VLM agents, specifying the task, tools, coordinates, action format, and final box format shared by prompted VLM baselines and Urbex.

Statistic	Total across 12 scenes	Per-scene range
Area (km 2 )	1.594	0.029–0.165
Points (millions)	877.4	12.1–116.3
GT candidate objects	2,234	31–367
OSM landmarks	299	9–51
Test queries	2,494	14–556

Appendix

Table 6: Scale of the 12 CityRefer scenes used in our experiments. Test queries combine ND and NO.

Scene	Inst. (Bldg./Veh.)	Building Expr.	Vehicle Expr.	Total Expr.	Lndmk.
USC	$59/62$	$177$	$184$	$361$	$272$
WMSC	$11/17$	$33$	$51$	$84$	$6$
Total	$70/79$	$210$	$235$	$445$	$278$

Appendix

Table 7: Statistics of STPLS3D-Refer. “Inst.” counts unique candidate instances, and “Lndmk.” counts named OSM landmarks.

Figure 6: Additional qualitative examples on the CityRefer NO split.

Figure 7: Training dynamics on CityRefer. (a) Total task reward. (b) Train/validation 2D grounding IoU with periodic validation markers and final-value annotations. (c) Per-episode tool-call frequency, showing convergence from over-calling zoom_in and under-using oblique_render to one call per tool.

Figure 8: Example agent reasoning processes from the initial query and global scene observation to landmark anchoring, local zoom-in, oblique verification, and final box commitment.

Dataset	Urbex Hit@1	Comparison Hit@1	Comparison method
CityRefer-ND	$25.62\pm 0.33$	23.4	CityAnchor
CityRefer-NO	$23.53\pm 0.73$	14.2	CityAnchor
STPLS3D-Refer	$26.34\pm 0.58$	16.3	GPT-5.4

Appendix

Table 8: Hit@1 over five decoding runs of one checkpoint (mean

\pm

sample standard deviation). Baselines are the strongest comparison results in the original main tables.

Model	Policy	Hit@1	Hit@3	Hit@5	Tokens/query
Qwen3-VL-4B base	Fixed	2.7	5.0	6.0	6,576
Urbex RL	Free	23.5	37.0	40.9	7,187
Urbex RL	Fixed	10.0	16.0	22.0	6,786
GPT-5.4	Free	13.1	18.6	20.1	16,321
GPT-5.4	Fixed	13.5	23.0	26.5	7,734

Appendix

Table 9: Free and fixed L-Z-O tool policies. The fixed workflow is extracted from the learned policy; tokens are reported per query.

Model / expression subset	$N$	Clean	Missing	Noisy 10 m	Wrong
CityRefer-NO
Urbex, with landmarks	1,302	23.5	12.7	22.3	11.3
Urbex, without landmarks	94	31.2	–	–	–
STPLS3D-Refer
GPT-5.4, with landmarks	445	16.3	5.7	15.3	7.3
Urbex, with landmarks	445	26.3	10.7	25.8	13.9

Appendix

Table 10: Landmark robustness (Hit@1, %). Dashes denote inapplicable landmark perturbations. The 30-example landmark-free STPLS3D set is a separate diagnostic.

Method	Proposals	Hit@1	Hit@3	Hit@5	BEV IoU	3D IoU	Point F1
CityAnchor	Ground truth	14.2	22.4	31.0	–	–	–
CityAnchor	SoftGroup	6.5	13.0	15.0	4.2	2.4	4.7
Urbex	None	23.5	37.0	40.9	14.6	7.7	17.8

Appendix

Table 11: CityRefer-NO proposal comparison. All metrics, including overlap and point F1, are percentages. Hit@

k

uses method-dependent candidate matching; direct IoU/F1 assess localization without a GT candidate pool.

Matching rule	Hit@1	Hit@3	Hit@5
Original IoU	23.53	36.97	40.86
IoU with positive-overlap filtering	23.53	36.87	40.49
CIoU	23.24	36.75	43.18

Appendix

Table 12: Matching-rule sensitivity on CityRefer-NO (five-run mean, %).

Split	No IoU threshold	Target IoU $\geq 0.25$	Target IoU $\geq 0.5$
CityRefer-NO	23.53	18.71	11.46
CityRefer-ND	25.62	20.38	12.61

Appendix

Table 13: Target-IoU-thresholded Hit@1 (%). The target must rank first under the original IoU matching rule.

Table 21

Target–landmark distance	ND	NO	STPLS3D
Regional-search success
$<50$ m	40.0	34.3	42.2
$50$ – $100$ m	31.1	32.6	26.6
$100$ – $200$ m	27.3	29.1	21.7
$\geq 200$ m	14.3	22.5	10.0
Same-class distractors within 50 m	ND	NO	STPLS3D

Appendix

Table 16: Reported trajectory diagnostics (success, %). The two panels measure different stages of grounding.

Figure 9: Aggregate trajectory diagnostics from Table 16 . Left: success versus target–landmark distance. Right: success versus same-class distractor count within 50 m. These curves visualize the reported bin rates, not individual failure trajectories; per-bin sample counts and error bars are unavailable.

Figure 10: Qualitative failure trajectories on CityRefer-NO. (a) The first zoom excludes the annotated target. (b) The zoom includes the target, but the prediction selects vehicles in a different parking row. Scene context in (a) is an offline display crop of the high-resolution BEV. Zoom and oblique panels preserve the complete recorded tool images; GT and prediction overlays on BEV views are added for diagnosis. Green denotes GT, red denotes the prediction, and dashed blue denotes the first zoom extent. Both runs retain their proposed box after oblique inspection and obtain zero BEV IoU.

Component	Latency / preparation time
Landmark Query	$<1$ ms
Zoom-in from high-resolution BEV	70 ms
Oblique Rendering from cache	292 ms
All three tools	$\approx 0.36$ s
Urbex, full model and cached tools	3.2 s/query
CityAnchor, warm online inference	32.1 s/query

Appendix

Table 17: CityRefer latency and optional scene-cache preparation. The on-demand total is an estimate; cache construction covers all 12 scenes.

3D Visual Grounding

Three-dimensional visual grounding links language or visual queries to specific objects, parts, regions, and spatial relations in 3D scenes.

See all 142 papers

UniGround: Universal 3D Visual Grounding via Training-Free Scene Parsing
Mar 9, 2026Jiaxi Zhang, Yunheng Wang, Wei Lu +83D Visual Grounding 3D VLMs
The Hong Kong University of Science and Technology (Guangzhou) · JD Explore Academy · Zhejiang University
AgentGrounder: Zero-Shot 3D Visual Pointcloud Grounding using Multimodal Language Models
May 25, 2026Cuong Huynh, Maxim Popov, Denis Gridusov +13D Visual Grounding
Biomechatronics and Energy-Efficient Robotics (BE2R) Lab, ITMO University, Saint Petersburg, Russia
PruneGround: Plug-and-play Spatial Pruning for 3D Visual Grounding
Jun 30, 2026Duc Cao Dinh, Khai Le-Duc, Florent Draye +43D Spatial Reasoning Visually Grounded Reasoning
Knovel Engineering Lab · University of Toronto · Vector Institute +3

Mar 9, 2026J/K move · Enter open · S save

UniGround: Universal 3D Visual Grounding via Training-Free Scene Parsing

Jiaxi Zhang, Yunheng Wang, Wei Lu, Taowen Wang+7

The Hong Kong University of Science and Technology (Guangzhou) · JD Explore Academy · Zhejiang University

3D Visual Grounding 3D VLMs 3D Scene Understanding

Open paper AskPDF arXiv

3D Visual Grounding (3DVG) localizes objects from natural-language descriptions in 3D scenes and is fundamental to embodied AI applications. Although foundation models enable open-vocabulary reasoning, they typically rely on pre-generated candidates, creating two sequential bottlenecks. The candidate bottleneck occurs when dataset-specific 3D proposal models miss, fragment, or incorrectly group targets under distribution shifts, excluding them from VLM reasoning. The evidence bottleneck stems from incomplete visual evidence: global renderings preserve spatial context but obscure object details, whereas candidate-centric views capture local appearance but lack global context. To address these bottlenecks, we propose UniGround, a zero-shot 3DVG framework that addresses both bottlenecks through Global Candidate Filtering and Contextual Precision Grounding. Global Candidate Filtering constructs topology-consistent, class-agnostic candidates from 3D topology and multi-view semantic cues, without dataset-trained 3D detectors, task-specific proposal supervision, or predefined box and category priors. Contextual Precision Grounding jointly reasons over global spatial context and candidate-centric visual evidence, followed by closed-loop consistency verification for reliable target identification. UniGround achieves 46.1%/34.1% [email protected]/0.5 on ScanRefer and 28.7% [email protected] on the evaluated ARKitScenes subset of EmbodiedScan. Further experiments demonstrate competitive grounding without dataset-specific 3D priors, cross-dataset generalization to unseen indoor scenes, and robustness to real-world reconstruction noise and practical domain shifts.

Abstract

3D Visual Grounding

Explore similar work

UniGround: Universal 3D Visual Grounding via Training-Free Scene Parsing

AgentGrounder: Zero-Shot 3D Visual Pointcloud Grounding using Multimodal Language Models

PruneGround: Plug-and-play Spatial Pruning for 3D Visual Grounding