Organizations: SIGS, Tsinghua University, China · Shanghai Jiao Tong University, China · Wuhan University of Technology, China · City University of Hong Kong, Hong Kong (China SAR) · AiDlab, Hong Kong (China SAR)
Vision-language-action (VLA) models often expose spatial grounding through autoregressive text coordinates or opaque action tokens, creating brittle interfaces between multimodal reasoning and robot execution. We present Pointing-VLA, a typed hidden-state spatial readout built on Embodied-R1. Geometry-specific heads predict normalized points, object-functional grounding (OFG) heatmaps, and visual trajectories without serializing geometry as text. For the evaluated Bridge/WidowX and physical pick-place deployments, an explicit execution contract assigns PICK to source-conditioned OFG and PLACE to Pointing, providing direct stage-aligned spatial targets. Pointing-VLA achieves SOTA performance on Bridge/WidowX, averaging 72.9% across the evaluated four-task set without Bridge-specific finetuning under collision-enabled CuRobo execution. Pointing and OFG show complementary strengths across native and cross-dataset evaluations. The OFG/contact readout transfers to NORA-1.5, preserving or improving success while reducing recorded controller time by more than 20×; typed heads are also 6.68--6.90× faster than Embodied-R1 text decoding on a shared external suite. When integrated as spatial guidance for a π0.5 action policy, Pointing-VLA raises autonomous real-robot success from 52.7% to 80.7% across three visual contexts. These results establish typed spatial readouts as an efficient, inspectable interface between embodied reasoning and robot execution.
Figures & tables
Figure 1: Representative text-coordinate failures. Left: generation yields no parseable coordinate. Right: [222, 172] is parseable but lies outside the target region.
Figure 2: Pointing-VLA architecture and typed interfaces. Embodied-R1 hidden states feed point, OFG/contact-heatmap, and VTG-waypoint heads; fixed PICK/PLACE slots use source-conditioned OFG/Pointing, and a wrapper maps image geometry to robot coordinates.
Figure 3: Stage-wise training. Geometry warm-up trains the adapter, eight queries, and geometry head; LoRA specialization updates only low-rank backbone parameters; task continuation updates only the required subset. Snowflakes and flames mark frozen and trainable modules.
Model
VABench-P acc. ↑
Part-Afford.2K acc. ↑
GPT-4o
9.3
10.2
ASMv2
10.1
13.8
RoboBrain
7.0
25.3
Qwen2.5-VL
9.9
23.4
RoboPoint ( 16 )
19.1
27.6
FSD ( 17 )
61.8
9.6
Table 1: Native point-in-region grounding accuracy (%). Pointing-VLA uses its Pointing head for VABench-P and its OFG head for the full Part-Affordance-2K run.
Figure 4: Representative successful real-robot rollouts across three visual contexts, each ordered from initialization to stable release.
Faculty of Computing, Harbin Institute of Technology · International Digital Economy Academy (IDEA) · National Key Laboratory of Smart Farm Technologies and Systems +4