SpatialPoint: Spatial-aware Point Prediction for Embodied Localization

Qiming Zhu; Zhirui Fang; Tianming Zhang; Chuanxiu Liu; Xiaoke Jiang; Lei Zhang

cs.ROMar 16, 2026

SpatialPoint: Spatial-aware Point Prediction for Embodied Localization

Authors: Qiming Zhu, Zhirui Fang, Tianming Zhang, Chuanxiu Liu, Xiaoke Jiang, Lei Zhang

Organizations: Visincept Research · Tsinghua University · International Digital Economy Academy (IDEA)

Abstract

Embodied intelligence fundamentally requires a capability to determine where to act in 3D space. We formalize this requirement as embodied localization -- the problem of predicting executable 3D points conditioned on visual observations and language instructions. We instantiate embodied localization with two complementary target types: touchable points, surface-grounded 3D points enabling direct physical interaction, and air points, free-space 3D points specifying placement and navigation goals, directional constraints, or geometric relations. Embodied localization is inherently a problem of embodied 3D spatial reasoning -- yet most existing vision-language systems rely predominantly on RGB inputs, necessitating implicit geometric reconstruction that limits cross-scene generalization, despite the widespread adoption of RGB-D sensors in robotics. To address this gap, we propose SpatialPoint, a spatial-aware vision-language framework with careful design that integrates structured depth into a vision-language model (VLM) and generates camera-frame 3D coordinates. We construct a 2.6M-sample RGB-D dataset covering both touchable and air points QA pairs for training and evaluation. Extensive experiments demonstrate that incorporating depth into VLMs significantly improves embodied localization performance. We further validate SpatialPoint through real-robot deployment across three representative tasks: language-guided robotic arm grasping at specified locations, object placement to target destinations, and mobile robot navigation to goal positions.

Figures & tables

Figure 1: Embodied localization as executable 3D target prediction. We reduce embodied execution to predicting camera-frame 3D points of two complementary types: touchable points grounded on observed surfaces, and air points located in free space and specified by spatial language.

Category	Subtype	#QA	Ratio (%)
Touchable-Data	Object detection	513,395	26.99
	Object pointing	161,681	8.50
	Object affordance	560,836	29.48
	Object reference	346,617	18.22
	Region reference	319,961	16.82
Touchable-Bench	Object affordance	124	36.69

Table 1: Composition of SpatialPoint-Data and SpatialPoint-Bench. Ratios are computed within each category.

Figure 2: Our data engine. Touchable points are converted from RoboAfford [ 33 ] 2D annotations using monocular depth estimation [ 37 ] . Air points are generated from objects’ 3D relations, which are computed by lifting DINO-X [ 10 ] detections (caption/bbox/mask) into the camera frame using the estimated depth map and intrinsics, and then applying geometric computations.

Figure 3: Model overview. We add a dedicated depth encoder by duplicating the original visual backbone and feeding it a three-channel depth map to obtain depth tokens, wrapped by <dpt_start> and <dpt_end> . RGB/depth/text tokens form one causal sequence, and the LM head decodes structured $(u,v,Z)$ point lists.

Method	Overall $\uparrow$	OAP $\uparrow$	OAR $\uparrow$	SAL $\uparrow$	MAE Z (in) $\downarrow$	MAE Z (out) $\downarrow$	MAE Z (all) $\downarrow$
RoboPoint [ 6 ]	0.447	0.350	0.557	0.442	$-$	$-$	$-$
RoboAfford-Qwen [ 33 ]	0.593	0.453	0.661	0.579	$-$	$-$	$-$
RoboAfford-Qwen $++$ [ 44 ]	0.634	0.631	0.705	0.558	$-$	$-$	$-$
RoboBrain 2.5 [ 34 ]	0.741	0.673	0.876	0.671	205.1	255.9	217.9
Qwen2.5-VL-7B [ 45 ]	0.161	0.083	0.195	0.219	$-$	$-$	$-$
Qwen3-VL-Inst-2B [ 11 ]	0.319	0.307	0.445	0.190	$-$	$-$	$-$

Table 2: Touchable points results on RoboAfford-Eval. We report overall accuracy and per-category accuracy for object affordance prediction (OAP), object affordance recognition (OAR), and spatial affordance localization (SAL). For methods that output

(u,v,Z)

targets, we additionally report depth MAE (mm) against the monocular depth reference used in our lifted-3D evaluation, with an inside/outside breakdown; otherwise depth metrics are marked as “

-

”.

Method	DirPt	MetPt@5cm	MeanErr (cm)
RoboBrain 2.5 [ 34 ]	0.0804	0.0637	30.3412
Qwen3-VL-Inst-4B [ 11 ]	0.0532	0.0896	54.7086
Ours (epoch 1)	0.4886	0.2587	8.5008
Ours (epoch 2)	0.5088	0.2907	7.3034
Ours (epoch 3)	0.5071	0.3347	6.8084

Table 3: Overall air points evaluation on SpatialPoint-Bench (2445 queries).

	Direction $\uparrow$			Between $\uparrow$			Body-length $\uparrow$
Method	DirPt	MetPt	FullPt	DirPt	MetPt	FullPt	DirPt	MetPt	FullPt
Qwen3-VL-Inst-4B [ 11 ]	0.0573	0.1111	0.0050	0.0108	0.2222	0.0024	0.0627	0.0175	0.0011
RoboBrain 2.5 [ 34 ]	0.0817	0.0827	0.0070	0.0543	0.0333	0.0018	0.0904	0.0455	0.0041
Ours (epoch 1)	0.4856	0.3096	0.1571	0.3825	0.1833	0.0701	0.5637	0.1959	0.1104
Ours (epoch 2)	0.5264	0.2864	0.1523	0.4031	0.2711	0.1093	0.4964	0.3093	0.1535
Ours (epoch 3)	0.5161	0.3764	0.1867	0.4371	0.2123	0.0928	0.5097	0.3143	0.1602

Table 4: Point-level micro results by category. MetPt is computed on direction-correct points in metric-offset queries (denominator: dir-passed points in has_metric samples).

Figure 4: Surface-target qualitative comparison on RoboAfford-Eval(touchable-point).

Figure 5: Free-space qualitative comparison on our benchmark. Under the same queries, our model better satisfies air points relation constraints, compared to Qwen3-VL, demonstrating more reliable air points target prediction.

Model	Extra input	Dual-branch	SpatialBot enc.	Special tokens	Overall Acc $\uparrow$	Depth MAE (mm) $\downarrow$
Ours e1	Depth map	✓	✓	✓	0.790	17.2
T1	–	×	×	×	0.676	227.1
T2	–	×	×	✓	0.698	191.9
T3	Depth map	×	✓	×	0.708	49.2
T4	Depth map	×	✓	✓	0.707	22.1
T5	Depth map	✓	×	✓	0.736	38.1

Table 5: Touchable points ablations on RoboAfford-Eval. We vary the extra 3D input modality (depth map vs. point cloud), whether using a dual-branch backbone, whether depth is encoded in the SpatialBot-style (Depth

\rightarrow

3ch) when applicable, and whether special depth tokens <dpt_start> / <dpt_end> are used. Depth MAE is reported only when valid depth predictions are available under our lifted-3D evaluation; otherwise it is marked as “–”.

Variant	Epoch	Input	DirPt $\uparrow$	MetPt@5cm $\uparrow$	FullPt $\uparrow$	MeanErr (cm) $\downarrow$
Ours1	1	SpatialBot-encoded depth	0.4886	0.2587	0.1300	8.5008
Ours2	2	SpatialBot-encoded depth	0.5088	0.2907	0.1456	7.3034
Ours3	3	SpatialBot-encoded depth	0.5071	0.3347	0.1641	6.8084
A1	1	rgb-only	0.2007	0.0810	0.0181	18.6502
A2	2	rgb-only	0.3050	0.1050	0.0344	22.5311
A3	3	rgb-only	0.2556	0.0617	0.0174	26.3364

Table 6: Air points ablations on SpatialPoint-Bench (overall). We fix the dual-branch backbone and special depth tokens (used only when an extra input is provided), and only vary the input representation.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Zero-shot real-world results on robot pick-and-place.

Figure 7: Zero-shot real-world robot results on mobile navigation and picking.

Figure 8: Zero-shot real-world robot results on long-horizon task.

Variant	Epoch	Dual Backbone	Special Tokens	DirPt $\uparrow$	MetPt@5cm $\uparrow$	FullPt $\uparrow$	MeanErr (cm) $\downarrow$
Ours1	1	$\checkmark$	$\checkmark$	0.4886	0.2587	0.1300	8.5008
Ours2	2	$\checkmark$	$\checkmark$	0.5088	0.2907	0.1456	7.3034
Ours3	3	$\checkmark$	$\checkmark$	0.5071	0.3347	0.1641	6.8084
A10	1	$\times$	$\times$	0.4088	0.1885	0.0795	11.2143
A11	2	$\times$	$\times$	0.4413	0.1916	0.0846	11.2934
A12	3	$\times$	$\times$	0.4403	0.2130	0.0937	10.2762

Appendix

Table 7: Supplementary air-point ablations under SpatialBot-style depth encoding.

Figure 9: Additional qualitative comparisons on touchable points prediction. We compare 3 models under the same instructions.

Figure 10: Additional qualitative comparisons on air points prediction. We compare 3 models under the same instructions.

Figure 11: Failure cases on touchable-point prediction. (a) and (b) Fine-grained localization errors. (c) Fine-grained constraint error. (d) Relational grounding error.

Figure 12: Failure cases on air-point prediction. (a) Relational direction misinterpretation. (b) Weak image support for free-space targets. (c) Occupancy-awareness failure.

VLMs for Robotics

Vision-language models for robotics connect visual and linguistic understanding to grounded navigation, manipulation, and robot control.

See all 657 papers

Sep 7, 2026

Contextual Observer Grounding: Evaluating Situated Spatial Reasoning in Vision-Language Models

Reasoning over language instructions in embodied tasks such as robotics often requires understanding spatial relations from a speaker's situated perspective. Humans infer such perspectives from shared environmental knowledge, activity context, and commonsense. Recent vision-language models (VLMs) appear capable of spatial reasoning, but their ability to infer a speaker's viewpoint from contextual cues and interpret situated spatial relations from that viewpoint remains unclear. We call this capability contextual observer grounding. To study this capability, we construct the Point-of-View Benchmark (POVBench), a dataset of 3D scenes and queries that disentangles Inferred, Stated, and Given forms of observer grounding in natural embodied communication. Given multi-view observations and a natural-language sentence, models must localize unseen or underspecified targets from situated spatial and contextual cues. Across state-of-the-art VLMs, localizing targets from directional language remains challenging, even when observer grounding is made explicit. We find that explicit breakdowns of observer-relative spatial reasoning improve target localization. Our project page is available at https://mimo-owl.github.io/POVBench/.

Mimo Shirasaka, Haochen Zhang, Yonatan Bisk

The University of Tokyo Tokyo, Japan · Carnegie Mellon University Pittsburgh, PA, USA

Vision-Language Grounding Visually Grounded Reasoning 3D Visual Grounding

Ask PDF

Mar 10, 2026

VLM-Loc: Localization in Point Cloud Maps via Vision-Language Models

Text-to-point-cloud (T2P) localization aims to infer precise spatial positions within 3D point cloud maps from natural language descriptions, reflecting how humans perceive and communicate spatial layouts through language. However, existing methods largely rely on shallow text-point cloud correspondence without effective spatial reasoning, limiting their accuracy in complex environments. To address this limitation, we propose VLM-Loc, a framework that leverages the spatial reasoning capability of large vision-language models (VLMs) for T2P localization. Specifically, we transform point clouds into bird's-eye-view (BEV) images and scene graphs that jointly encode geometric and semantic context, providing structured inputs for the VLM to learn cross-modal representations bridging linguistic and spatial semantics. On top of these representations, we introduce a partial node assignment mechanism that explicitly associates textual cues with scene graph nodes, enabling interpretable spatial reasoning for accurate localization. To facilitate systematic evaluation across diverse scenes, we present CityLoc, a benchmark built from multi-source point clouds for fine-grained T2P localization. Experiments on CityLoc demonstrate VLM-Loc achieves superior accuracy and robustness compared to state-of-the-art methods. Our code, model, and dataset are available at repository.

Shuhao Kang, Youqi Liao, Peijie Wang +5

VCIP, CS, Nankai University · Wuhan University · CASIA +6

Map-Based Localization 3D VLMs

Ask PDF

May 27, 2026

Embodied3DBench: Benchmarking Low-Level Embodied Spatial Intelligence of Vision Language Models

Are current Vision Language Models (VLMs) ready to comprehend and reason about complex embodied interactions in 3D environments? We introduce Embodied3DBench, a robot-centric benchmark targeting low-level spatial intelligence in embodied 3D environments. To systematically evaluate these foundational perceptual capabilities, the benchmark includes 6 task categories divided into two core groups: Spatial Structural Understanding (Grounding, Spatial Relation Prediction, and Multi-view Correspondence) and Interaction-Oriented Perception (Affordance Prediction, Grasp Point Prediction, and Trajectory Prediction). The benchmark spans 12 subcategories and contains over 21k high-quality question-answer pairs. We evaluate 13 state-of-the-art models, and the results show that while current models exhibit relatively strong high-level spatial reasoning, such as understanding object-to-object positional relations, they remain fragile in interaction-oriented perception, highlighting a significant lack of robust 3D-aware interaction priors. To actively bridge this capability gap revealed by our benchmark, we further synthesize a large-scale training dataset comprising 1.3M QA pairs. Notably, fine-tuning on this dataset yields significant improvements in low-level spatial intelligence. Ultimately, Embodied3DBench fills a critical gap by providing both a systematic evaluation framework and a scalable data solution, setting a clear target for the development of interaction-aware multimodal systems.

Jiyao Zhang, Mingxu Zhang, Yitong Peng +8

CFCS, School of CS, PKU Beijing, China · BUPT Beijing, China · PKU Beijing, China +2

VLM Evaluation Spatial Reasoning Benchmarks Robotic Grasping 3D Spatial Reasoning Robot Manipulation Benchmarks 3D Visual Grounding

Ask PDF

Abstract

VLMs for Robotics

Explore similar work

Contextual Observer Grounding: Evaluating Situated Spatial Reasoning in Vision-Language Models

VLM-Loc: Localization in Point Cloud Maps via Vision-Language Models

Embodied3DBench: Benchmarking Low-Level Embodied Spatial Intelligence of Vision Language Models