SpatialPoint: Spatial-aware Point Prediction for Embodied Localization

Qiming Zhu; Zhirui Fang; Tianming Zhang; Chuanxiu Liu; Xiaoke Jiang; Lei Zhang

cs.ROMar 16, 2026

SpatialPoint: Spatial-aware Point Prediction for Embodied Localization

Authors: Qiming Zhu, Zhirui Fang, Tianming Zhang, Chuanxiu Liu, Xiaoke Jiang, Lei Zhang

Organizations: Visincept Research · Tsinghua University · International Digital Economy Academy (IDEA)

Abstract

Embodied intelligence fundamentally requires a capability to determine where to act in 3D space. We formalize this requirement as embodied localization -- the problem of predicting executable 3D points conditioned on visual observations and language instructions. We instantiate embodied localization with two complementary target types: touchable points, surface-grounded 3D points enabling direct physical interaction, and air points, free-space 3D points specifying placement and navigation goals, directional constraints, or geometric relations. Embodied localization is inherently a problem of embodied 3D spatial reasoning -- yet most existing vision-language systems rely predominantly on RGB inputs, necessitating implicit geometric reconstruction that limits cross-scene generalization, despite the widespread adoption of RGB-D sensors in robotics. To address this gap, we propose SpatialPoint, a spatial-aware vision-language framework with careful design that integrates structured depth into a vision-language model (VLM) and generates camera-frame 3D coordinates. We construct a 2.6M-sample RGB-D dataset covering both touchable and air points QA pairs for training and evaluation. Extensive experiments demonstrate that incorporating depth into VLMs significantly improves embodied localization performance. We further validate SpatialPoint through real-robot deployment across three representative tasks: language-guided robotic arm grasping at specified locations, object placement to target destinations, and mobile robot navigation to goal positions.

Figures & tables

Figure 1: Embodied localization as executable 3D target prediction. We reduce embodied execution to predicting camera-frame 3D points of two complementary types: touchable points grounded on observed surfaces, and air points located in free space and specified by spatial language.

Category	Subtype	#QA	Ratio (%)
Touchable-Data	Object detection	513,395	26.99
	Object pointing	161,681	8.50
	Object affordance	560,836	29.48
	Object reference	346,617	18.22
	Region reference	319,961	16.82
Touchable-Bench	Object affordance	124	36.69

Table 1: Composition of SpatialPoint-Data and SpatialPoint-Bench. Ratios are computed within each category.

Figure 2: Our data engine. Touchable points are converted from RoboAfford [ 33 ] 2D annotations using monocular depth estimation [ 37 ] . Air points are generated from objects’ 3D relations, which are computed by lifting DINO-X [ 10 ] detections (caption/bbox/mask) into the camera frame using the estimated depth map and intrinsics, and then applying geometric computations.

Figure 3: Model overview. We add a dedicated depth encoder by duplicating the original visual backbone and feeding it a three-channel depth map to obtain depth tokens, wrapped by <dpt_start> and <dpt_end> . RGB/depth/text tokens form one causal sequence, and the LM head decodes structured $(u,v,Z)$ point lists.

Method	Overall $\uparrow$	OAP $\uparrow$	OAR $\uparrow$	SAL $\uparrow$	MAE Z (in) $\downarrow$	MAE Z (out) $\downarrow$	MAE Z (all) $\downarrow$
RoboPoint [ 6 ]	0.447	0.350	0.557	0.442	$-$	$-$	$-$
RoboAfford-Qwen [ 33 ]	0.593	0.453	0.661	0.579	$-$	$-$	$-$
RoboAfford-Qwen $++$ [ 44 ]	0.634	0.631	0.705	0.558	$-$	$-$	$-$
RoboBrain 2.5 [ 34 ]	0.741	0.673	0.876	0.671	205.1	255.9	217.9
Qwen2.5-VL-7B [ 45 ]	0.161	0.083	0.195	0.219	$-$	$-$	$-$
Qwen3-VL-Inst-2B [ 11 ]	0.319	0.307	0.445	0.190	$-$	$-$	$-$

Table 2: Touchable points results on RoboAfford-Eval. We report overall accuracy and per-category accuracy for object affordance prediction (OAP), object affordance recognition (OAR), and spatial affordance localization (SAL). For methods that output

(u,v,Z)

targets, we additionally report depth MAE (mm) against the monocular depth reference used in our lifted-3D evaluation, with an inside/outside breakdown; otherwise depth metrics are marked as “

-

”.

Method	DirPt	MetPt@5cm	MeanErr (cm)
RoboBrain 2.5 [ 34 ]	0.0804	0.0637	30.3412
Qwen3-VL-Inst-4B [ 11 ]	0.0532	0.0896	54.7086
Ours (epoch 1)	0.4886	0.2587	8.5008
Ours (epoch 2)	0.5088	0.2907	7.3034
Ours (epoch 3)	0.5071	0.3347	6.8084

Table 3: Overall air points evaluation on SpatialPoint-Bench (2445 queries).

	Direction $\uparrow$			Between $\uparrow$			Body-length $\uparrow$
Method	DirPt	MetPt	FullPt	DirPt	MetPt	FullPt	DirPt	MetPt	FullPt
Qwen3-VL-Inst-4B [ 11 ]	0.0573	0.1111	0.0050	0.0108	0.2222	0.0024	0.0627	0.0175	0.0011
RoboBrain 2.5 [ 34 ]	0.0817	0.0827	0.0070	0.0543	0.0333	0.0018	0.0904	0.0455	0.0041
Ours (epoch 1)	0.4856	0.3096	0.1571	0.3825	0.1833	0.0701	0.5637	0.1959	0.1104
Ours (epoch 2)	0.5264	0.2864	0.1523	0.4031	0.2711	0.1093	0.4964	0.3093	0.1535
Ours (epoch 3)	0.5161	0.3764	0.1867	0.4371	0.2123	0.0928	0.5097	0.3143	0.1602

Table 4: Point-level micro results by category. MetPt is computed on direction-correct points in metric-offset queries (denominator: dir-passed points in has_metric samples).

Figure 4: Surface-target qualitative comparison on RoboAfford-Eval(touchable-point).

Figure 5: Free-space qualitative comparison on our benchmark. Under the same queries, our model better satisfies air points relation constraints, compared to Qwen3-VL, demonstrating more reliable air points target prediction.

Model	Extra input	Dual-branch	SpatialBot enc.	Special tokens	Overall Acc $\uparrow$	Depth MAE (mm) $\downarrow$
Ours e1	Depth map	✓	✓	✓	0.790	17.2
T1	–	×	×	×	0.676	227.1
T2	–	×	×	✓	0.698	191.9
T3	Depth map	×	✓	×	0.708	49.2
T4	Depth map	×	✓	✓	0.707	22.1
T5	Depth map	✓	×	✓	0.736	38.1

Table 5: Touchable points ablations on RoboAfford-Eval. We vary the extra 3D input modality (depth map vs. point cloud), whether using a dual-branch backbone, whether depth is encoded in the SpatialBot-style (Depth

\rightarrow

3ch) when applicable, and whether special depth tokens <dpt_start> / <dpt_end> are used. Depth MAE is reported only when valid depth predictions are available under our lifted-3D evaluation; otherwise it is marked as “–”.

Variant	Epoch	Input	DirPt $\uparrow$	MetPt@5cm $\uparrow$	FullPt $\uparrow$	MeanErr (cm) $\downarrow$
Ours1	1	SpatialBot-encoded depth	0.4886	0.2587	0.1300	8.5008
Ours2	2	SpatialBot-encoded depth	0.5088	0.2907	0.1456	7.3034
Ours3	3	SpatialBot-encoded depth	0.5071	0.3347	0.1641	6.8084
A1	1	rgb-only	0.2007	0.0810	0.0181	18.6502
A2	2	rgb-only	0.3050	0.1050	0.0344	22.5311
A3	3	rgb-only	0.2556	0.0617	0.0174	26.3364

Table 6: Air points ablations on SpatialPoint-Bench (overall). We fix the dual-branch backbone and special depth tokens (used only when an extra input is provided), and only vary the input representation.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Zero-shot real-world results on robot pick-and-place.

Figure 7: Zero-shot real-world robot results on mobile navigation and picking.

Figure 8: Zero-shot real-world robot results on long-horizon task.

Variant	Epoch	Dual Backbone	Special Tokens	DirPt $\uparrow$	MetPt@5cm $\uparrow$	FullPt $\uparrow$	MeanErr (cm) $\downarrow$
Ours1	1	$\checkmark$	$\checkmark$	0.4886	0.2587	0.1300	8.5008
Ours2	2	$\checkmark$	$\checkmark$	0.5088	0.2907	0.1456	7.3034
Ours3	3	$\checkmark$	$\checkmark$	0.5071	0.3347	0.1641	6.8084
A10	1	$\times$	$\times$	0.4088	0.1885	0.0795	11.2143
A11	2	$\times$	$\times$	0.4413	0.1916	0.0846	11.2934
A12	3	$\times$	$\times$	0.4403	0.2130	0.0937	10.2762

Appendix

Table 7: Supplementary air-point ablations under SpatialBot-style depth encoding.

Figure 9: Additional qualitative comparisons on touchable points prediction. We compare 3 models under the same instructions.

Figure 10: Additional qualitative comparisons on air points prediction. We compare 3 models under the same instructions.

Figure 11: Failure cases on touchable-point prediction. (a) and (b) Fine-grained localization errors. (c) Fine-grained constraint error. (d) Relational grounding error.

Figure 12: Failure cases on air-point prediction. (a) Relational direction misinterpretation. (b) Weak image support for free-space targets. (c) Occupancy-awareness failure.

VLMs for Robotics

Vision-language models for robotics connect visual and linguistic understanding to grounded navigation, manipulation, and robot control.

See all 654 papers

Contextual Observer Grounding: Evaluating Situated Spatial Reasoning in Vision-Language Models
Sep 7, 2026Mimo Shirasaka, Haochen Zhang, Yonatan BiskVision-Language Grounding Visually Grounded Reasoning
The University of Tokyo Tokyo, Japan · Carnegie Mellon University Pittsburgh, PA, USA
VLM-Loc: Localization in Point Cloud Maps via Vision-Language Models
Mar 10, 2026Shuhao Kang, Youqi Liao, Peijie Wang +5Map-Based Localization 3D VLMs
VCIP, CS, Nankai University · Wuhan University · CASIA +6
Embodied3DBench: Benchmarking Low-Level Embodied Spatial Intelligence of Vision Language Models
May 27, 2026Jiyao Zhang, Mingxu Zhang, Yitong Peng +8VLM Evaluation Spatial Reasoning Benchmarks
CFCS, School of CS, PKU Beijing, China · BUPT Beijing, China · PKU Beijing, China +2

Sep 7, 2026J/K move · Enter open · S save

Contextual Observer Grounding: Evaluating Situated Spatial Reasoning in Vision-Language Models

Mimo Shirasaka, Haochen Zhang, Yonatan Bisk

The University of Tokyo Tokyo, Japan · Carnegie Mellon University Pittsburgh, PA, USA

Vision-Language Grounding Visually Grounded Reasoning 3D Visual Grounding

Open paper AskPDF arXiv

Reasoning over language instructions in embodied tasks such as robotics often requires understanding spatial relations from a speaker's situated perspective. Humans infer such perspectives from shared environmental knowledge, activity context, and commonsense. Recent vision-language models (VLMs) appear capable of spatial reasoning, but their ability to infer a speaker's viewpoint from contextual cues and interpret situated spatial relations from that viewpoint remains unclear. We call this capability contextual observer grounding. To study this capability, we construct the Point-of-View Benchmark (POVBench), a dataset of 3D scenes and queries that disentangles Inferred, Stated, and Given forms of observer grounding in natural embodied communication. Given multi-view observations and a natural-language sentence, models must localize unseen or underspecified targets from situated spatial and contextual cues. Across state-of-the-art VLMs, localizing targets from directional language remains challenging, even when observer grounding is made explicit. We find that explicit breakdowns of observer-relative spatial reasoning improve target localization. Our project page is available at https://mimo-owl.github.io/POVBench/.

Abstract

VLMs for Robotics

Explore similar work

Contextual Observer Grounding: Evaluating Situated Spatial Reasoning in Vision-Language Models

VLM-Loc: Localization in Point Cloud Maps via Vision-Language Models

Embodied3DBench: Benchmarking Low-Level Embodied Spatial Intelligence of Vision Language Models