cs.ROOct 29, 2026

Decoding Geometry, Not Tokens: Efficient VLMs for Robot Affordance and Trajectory Prediction

Authors: Akila Ayanthi, Darshana Priyasad, Tharindu Fernando, Dimity Miller, Sridha Sridharan, Clinton Fookes, Peyman Moghadam

Organizations: SAIVT, School of Electrical Engineering and Robotics, Queensland University of Technology, Australia · CSIRO Robotics, CSIRO, Australia

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Predicting where to grasp and how to move is fundamentally a geometry problem. Vision-language models treat it as language generation instead. They tokenize spatial coordinates into text and decode them autoregressively. This mechanism ignores spatial structure and truncates continuous precision. Errors compound across coordinate tokens during decoding. In this paper, we show that compact off-the-shelf VLMs with 0.5-2B parameters already encode rich spatial information in their intermediate hidden states. Autoregressive decoding is the dominant bottleneck for trajectory prediction, while affordance prediction depends additionally on feature depth and decoder architecture. We expose the latent spatial information with a lightweight, geometry-compatible prediction head that attaches to a frozen backbone. It reads features from an intermediate hidden layer, conditions a learned query on the task instruction and refines it via cross-attention over the visual tokens to regress affordance boxes and trajectory waypoints directly as continuous values, without modifying the backbone. We evaluate on ShareRobot-Bench and AGD20K. Our 2B variant reduces trajectory error by over 60% and improves ShareRobot-Bench affordance mAP by 85% relative to the autoregressive baseline. It approaches RoboBrain on trajectory prediction (0.251 vs 0.191 DFD) at 3.8x higher throughput. These results demonstrate a favorable accuracy-efficiency trade-off for instruction-conditioned 2D geometric prediction.

Figures & tables

Figure 1: Accuracy, efficiency, and model size. Each bubble represents a model; bubble size encodes the number of active parameters. Color denotes model family: LLaVA-OV , Qwen2-VL , RoboBrain . Dashed arrows indicate improvement over the corresponding base VLM. (Top) Trajectory prediction DFD ( ↓\downarrow ) vs. inference speed (FPS on H100) on ShareRobot-Bench-Trajectory. (Bottom) Affordance mAP ( ↑\uparrow ) vs. inference speed (FPS on H100) on AGD20K. Our method outperforms both base VLMs, approaches larger-model accuracy, and retains favorable parameter and inference efficiency.
Figure 2: Architecture overview. A frozen VLM processes an RGB image and task instruction; hidden states from an intermediate layer are passed to a lightweight geometric prediction head. A DETR-style transformer decoder, conditioned on the instruction via the final text token hidden state, refines a learned query against the visual memory tokens. Two independent FFNs decode the query into continuous outputs: affordance boxes (L1+GIoU loss) and trajectory waypoints (L1+startpoint+endpoint loss). No gradient flows into the VLM.
    TrajectoryAffordance
CategoryModelEmb.Head#P(B)SRBFPS ↑\uparrowTFLOPs ↓\downarrow#P(B)AGD20KSRBFPS ↑\uparrowTFLOPs ↓\downarrow
DFD ↓\downarrowHD ↓\downarrowRMSE ↓\downarrowH100RTXmAP ↑\uparrowmAP ↑\uparrowH100RTX
Large VLMsRoboBrain † [ 16 ]✓token8.20.1910.1710.1335.51.231.08.221.7820.574.10.880.0
Qwen2-VL-7B [ 38 ]×token8.30.6020.5910.49415.26.84.48.318.7918.656.21.722.0
LLaVA-OV-7B [ 19 ]×token8.00.6410.5910.4948.71.630.38.011.877.454.10.880.0
 Qwen2-VL-2B [ 38 ]×token2.20.7980.7770.59716.611.11.632.27.306.276.92.68.9
Table 1: Accuracy and inference efficiency comparison. Our Geometric Head variants on frozen Qwen2-VL-2B and LLaVA-OV-0.5B backbones ( Emb. = ✓ indicates embodied training; Head ∈{token,geo}\in\{\text{token},\text{geo}\} ). We report active backbone parameters (#P(B)), accuracy metrics (DFD, HD, RMSE, mAP), throughput on H100 and RTX 3090 (FPS ↑\uparrow ), and compute cost (TFLOPs ↓\downarrow ). † Fully trained on embodied data; all other large VLMs are evaluated zero-shot. SRB=ShareRobot-Bench. Geo-Head substantially outperforms zero-shot and LoRA baselines and approaches 7B/RoboBrain accuracy at a fraction of the parameters and compute.
VariantRepresentationBackbone AdaptationDecoderOutputQwen2-VL-2BLLaVA-OV-0.5B
Traj. DFD ↓\downarrowAff. mAP ↑\uparrowTraj. DFD ↓\downarrowAff. mAP ↑\uparrow
Base VLMFull LLMNoneARTokens0.7987.300.7771.09
LoRA-ARFull LLMLoRAARTokens0.752±0.0610.752\pm 0.0612.25±0.602.25\pm 0.600.695±0.0010.695\pm 0.0012.39±0.112.39\pm 0.11
l∗l^{*} -LoRA-ARFull LLMLoRA after l∗l^{*}ARTokens0.680±0.0100.680\pm 0.0102.39±0.202.39\pm 0.200.680±0.0300.680\pm 0.0302.04±0.212.04\pm 0.21
LoRA-RegFinal tokenLoRALT+MLPContinuous0.34±0.0080.34\pm 0.0083.24±0.063.24\pm 0.060.43±0.20.43\pm 0.25.78±1.045.78\pm 1.04
l0l_{0} -MLPl0l_{0}FrozenMLPContinuous0.291±0.0020.291\pm 0.0028.74±0.978.74\pm 0.970.311±0.0020.311\pm 0.0028.68±0.108.68\pm 0.10
Table 2: Disentangling the source of performance gains. Continuous regression from the final VLM representation already substantially outperforms autoregressive token decoding, while intermediate features and DETR provide additional gains. Pre-LLM controls test performance before language-model contextualization. AR denotes autoregressive decoding and LT+MLP denotes direct regression from a learned token using an MLP. Results are reported as mean ±\pm standard deviation where applicable.
Figure 3: Layer-wise representational analysis. Trajectory DFD on ShareRobot-Bench-Trajectory and affordance mAP on AGD20K across all language-model layers for Qwen2-VL-2B and LLaVA-OV-0.5B, compared against the base VLM and LoRA fine-tuned autoregressive baselines. Colored markers indicate the selected layer used in the final model.
Figure 4: Qualitative trajectory predictions. Red: predicted, green: ground truth. Top row: vanilla VLM baseline. Bottom row: with our geometry head.
Figure 5: Qualitative affordance predictions. Red: predicted box, green: ground truth. Top row: vanilla VLM baseline. Bottom row: with our geometry head.
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Efficiency–performance trade-off across layers on Qwen2-VL-2B for the trajectory prediction task.
  TrajectoryAffordance
BackboneQueryDFD ↓\downarrowHD ↓\downarrowRMSE ↓\downarrowmAP ↑\uparrow
Qwen2-VL-2BLast token0.251±0.0100.251\pm 0.0100.219±0.0080.219\pm 0.0080.177±0.0070.177\pm 0.00712.37±1.1612.37\pm 1.16
Attn. pool0.215±0.0020.215\pm 0.0020.194±0.0010.194\pm 0.0010.155±0.0020.155\pm 0.0028.67±1.018.67\pm 1.01
LLaVA-OV-0.5BLast token0.225±0.0390.225\pm 0.0390.225±0.0030.225\pm 0.0030.179±0.0030.179\pm 0.0039.53±1.449.53\pm 1.44
Attn. pool0.256±0.0040.256\pm 0.0040.24±0.010.24\pm 0.010.189±0.0020.189\pm 0.0027.27±1.27.27\pm 1.2
Appendix
Table 3: Ablation of query token selection for trajectory prediction on ShareRobot-Bench-Trajectory and affordance prediction on AGD20K.
ModelDFD↓HD↓RMSE↓
w/ start & end point loss0.251±0.0100.251\pm 0.0100.219±0.0080.219\pm 0.0080.177±0.0070.177\pm 0.007
w/ only overall point loss0.322±0.020.322\pm 0.020.26±0.010.26\pm 0.010.26±0.030.26\pm 0.03
Appendix
Table 4: Ablation on start and end point supervision for trajectory prediction head (Qwen2-VL-2B, ShareRobot-Bench).
Figure 7: Predicted waypoint heatmaps over the test set, visualizing how endpoint supervision shapes the spatial distribution of trajectory predictions. (a) Ground-truth distribution: waypoints are broadly spread, reflecting the spatial diversity of manipulation trajectories in the dataset. (b) Zero-shot Qwen2-VL-2B baseline: predictions collapse to the image boundaries, confirming that vanilla autoregressive generation fails to produce spatially meaningful trajectories. (c) Geometry head without endpoint supervision: predictions collapse to a single central region, as the uniform L1 objective is minimised by regressing toward the dataset mean. (d) Geometry head with endpoint supervision: the distribution recovers spatial diversity matching the ground truth, demonstrating that boundary supervision forces input-specific localization rather than dataset-level averaging.
Figure 8: Additional visualizations of diverse trajectories predicted by Qwen2-VL-2B + Geo-Head. Green curve represents the ground-truth and the red curve represents the prediction. The visualizations in the first three rows demonstrate that our geometry head effectively generates end-effector manipulation trajectories based on the observation and task instruction. The fourth row presents some failure cases arising from object-localization errors or insufficient spatial understanding.
Figure 9: Additional visualizations of diverse trajectories predicted by LLaVA-OV-0.5B + Geo-Head. Green curve represents the ground-truth and the red curve represents the prediction. The visualizations in the first three rows demonstrate that our geometry head effectively generates end-effector manipulation trajectories based on the observation and task instruction. The fourth row presents some failure cases.
Figure 10: Additional visualizations of diverse affordance regions predicted by Qwen2-VL-2B + Geo-Head. Green box represents the ground-truth and the red box represents the prediction. The visualizations in the first three rows demonstrate that our geometry head effectively generates affordance boxes based on the observation and task instruction. The fourth row presents some failure cases.
Figure 11: Additional visualizations of diverse affordance regions predicted by LLaVA-OV-0.5B + Geo-Head. Green box represents the ground-truth and the red box represents the prediction. The visualizations in the first three rows demonstrate that our geometry head effectively generates affordance boxes based on the observation and task instruction. The fourth row presents some failure cases.
More in this topic

VLMs for Robotics

Vision-language models for robotics connect visual and linguistic understanding to grounded navigation, manipulation, and robot control.

See all 664 papers

Explore similar work

CardsList
Aug 1, 2026

SpatialAfford: Teaching Compact VLMs Where to Look and Where to Ground for Affordance

Affordance grounding aims to localize the functional region for interaction, such as the handle to grasp or the button to press, rather than the whole object. This makes it more challenging than generic visual grounding because the target region is smaller, more ambiguous, and more dependent on task context, especially for compact vision-language models (VLMs) used in embodied settings. Recent sequence-level supervision and reinforcement learning improve coordinate prediction quality, yet compact autoregressive VLMs still lack reliable affordance-aware visual focus before coordinate generation: the model can produce better coordinate tokens while its cross-modal attention remains diffuse and weakly anchored to the true affordance evidence. To address it, we propose SpatialAfford, a two-stage framework that first aligns attention to the ground-truth affordance region through Spatial Attention Alignment (SAA), then refines coordinate prediction with Spatial-Aware GRPO. By explicitly teaching the model where to look before optimizing where to ground, SpatialAfford turns affordance grounding from a purely output-constrained objective into attention-grounded spatial reasoning. Across ShareRobot-Bench, ReasonAff, and PartAfford, SpatialAfford consistently improves affordance grounding, with a compact 4B model outperforming stronger 7B+ baselines.

Yufei Zhang, Chenlu Zhan, Donghui Sun +2

Zhejiang University · Vivo Mobile Communication Co., Ltd

Vision-Language GroundingVLMs for Robotics
AskPDF
Mar 13, 2026

Evaluating VLMs' Spatial Reasoning Over Robot Motion: A Step Towards Robot Planning with Motion Preferences

Understanding user instructions and object spatial relations in surrounding environments is crucial for intelligent robot systems to assist humans in various tasks. The natural language and spatial reasoning capabilities of Vision-Language Models (VLMs) have the potential to enhance the generalization of robot planners on new tasks, objects, and motion specifications. While foundation models have been applied to task planning, it is still unclear the degree to which they have the capability of spatial reasoning required to enforce user preferences or constraints on motion, such as desired distances from objects, topological properties, or motion style preferences. In this paper, we evaluate the capability of four state-of-the-art VLMs at spatial reasoning over robot motion, using four different querying methods. Our results show that, with the highest-performing querying method, Qwen2.5-VL achieves 71.4% accuracy zero-shot and 75% on a smaller model after fine-tuning, and GPT-4o leads to lower performance. We evaluate two types of motion preferences (object-proximity and path-style), and we also analyze the trade-off between accuracy and computation cost in number of tokens. This work shows some promise in the potential of VLM integration with robot motion planning pipelines.

Wenxi Wu, Jingjing Zhang, Martim Brandão

King’s College London · University College London

VLM ReasoningVLMs for RoboticsVisual Spatial ReasoningRobot Motion Planning
AskPDF
Sep 28, 2026

Geometric Encoding for Spatial Reasoning in Vision-Language Models

Vision-Language Models (VLMs) are far more reliable at recognizing what appears in a video than at reasoning about its spatial and temporal properties, such as metric distances, object dimensions, and consistent object identities across frames. We present Geometric Code, a perception-to-geometry pipeline that computes explicit spatial structure from video and supplies it to VLMs as context to augment reasoning. A perception layer segments and classifies objects and recovers depth, camera pose, and intrinsics from monocular RGB video. A deterministic geometric engine then back-projects, merges, and cleans these outputs into a spatial code, including per-object positions, dimensions, counts, inter-object distances, appearance order, and room geometry. The code is serialized into VLMs' prompts, either alongside the video or replacing it entirely. Specifically, there is no component trained or fine-tuned in our approach. On VSI-Bench, augmenting 2B and 4B open models with the spatial code improves average accuracy by +4.1 points over the frames-only baseline, with the largest gains on numeric estimation tasks such as absolute distance (+24.1 points). The results suggest that explicitly computed geometry, delivered through the language channel, recovers spatial competence that small VLMs cannot extract from pixels alone.

Antonio Jun, Haoshui Yu, Zhengyi Lu +2

School of Arts and Sciences New York University New York City, United States · Dept. of Engineering and Computer Science Oakland University Rochester, United States

Visual Spatial ReasoningRepresentation Geometry in VLMsVLM Reasoning
AskPDF