Decoding Geometry, Not Tokens: Efficient VLMs for Robot Affordance and Trajectory Prediction

Akila Ayanthi; Darshana Priyasad; Tharindu Fernando; Dimity Miller; Sridha Sridharan; Clinton Fookes; Peyman Moghadam

cs.ROOct 29, 2026

Decoding Geometry, Not Tokens: Efficient VLMs for Robot Affordance and Trajectory Prediction

Authors: Akila Ayanthi, Darshana Priyasad, Tharindu Fernando, Dimity Miller, Sridha Sridharan, Clinton Fookes, Peyman Moghadam

Organizations: SAIVT, School of Electrical Engineering and Robotics, Queensland University of Technology, Australia · CSIRO Robotics, CSIRO, Australia

Abstract

Predicting where to grasp and how to move is fundamentally a geometry problem. Vision-language models treat it as language generation instead. They tokenize spatial coordinates into text and decode them autoregressively. This mechanism ignores spatial structure and truncates continuous precision. Errors compound across coordinate tokens during decoding. In this paper, we show that compact off-the-shelf VLMs with 0.5-2B parameters already encode rich spatial information in their intermediate hidden states. Autoregressive decoding is the dominant bottleneck for trajectory prediction, while affordance prediction depends additionally on feature depth and decoder architecture. We expose the latent spatial information with a lightweight, geometry-compatible prediction head that attaches to a frozen backbone. It reads features from an intermediate hidden layer, conditions a learned query on the task instruction and refines it via cross-attention over the visual tokens to regress affordance boxes and trajectory waypoints directly as continuous values, without modifying the backbone. We evaluate on ShareRobot-Bench and AGD20K. Our 2B variant reduces trajectory error by over 60% and improves ShareRobot-Bench affordance mAP by 85% relative to the autoregressive baseline. It approaches RoboBrain on trajectory prediction (0.251 vs 0.191 DFD) at 3.8x higher throughput. These results demonstrate a favorable accuracy-efficiency trade-off for instruction-conditioned 2D geometric prediction.

Figures & tables

Figure 1: Accuracy, efficiency, and model size. Each bubble represents a model; bubble size encodes the number of active parameters. Color denotes model family: LLaVA-OV , Qwen2-VL , RoboBrain . Dashed arrows indicate improvement over the corresponding base VLM. (Top) Trajectory prediction DFD (

\downarrow

) vs. inference speed (FPS on H100) on ShareRobot-Bench-Trajectory. (Bottom) Affordance mAP (

\uparrow

) vs. inference speed (FPS on H100) on AGD20K. Our method outperforms both base VLMs, approaches larger-model accuracy, and retains favorable parameter and inference efficiency.

Figure 2: Architecture overview. A frozen VLM processes an RGB image and task instruction; hidden states from an intermediate layer are passed to a lightweight geometric prediction head. A DETR-style transformer decoder, conditioned on the instruction via the final text token hidden state, refines a learned query against the visual memory tokens. Two independent FFNs decode the query into continuous outputs: affordance boxes (L1+GIoU loss) and trajectory waypoints (L1+startpoint+endpoint loss). No gradient flows into the VLM.

				Trajectory							Affordance
Category	Model	Emb.	Head	#P(B)	SRB			FPS $\uparrow$		TFLOPs $\downarrow$	#P(B)	AGD20K	SRB	FPS $\uparrow$		TFLOPs $\downarrow$
Category	Model	Emb.	Head	#P(B)	DFD $\downarrow$	HD $\downarrow$	RMSE $\downarrow$	H100	RTX	TFLOPs $\downarrow$	#P(B)	mAP $\uparrow$	mAP $\uparrow$	H100	RTX	TFLOPs $\downarrow$
Large VLMs	RoboBrain † [ 16 ]	✓	token	8.2	0.191	0.171	0.133	5.5	1.2	31.0	8.2	21.78	20.57	4.1	0.8	80.0
	Qwen2-VL-7B [ 38 ]	×	token	8.3	0.602	0.591	0.494	15.2	6.8	4.4	8.3	18.79	18.65	6.2	1.7	22.0
	LLaVA-OV-7B [ 19 ]	×	token	8.0	0.641	0.591	0.494	8.7	1.6	30.3	8.0	11.87	7.45	4.1	0.8	80.0
	Qwen2-VL-2B [ 38 ]	×	token	2.2	0.798	0.777	0.597	16.6	11.1	1.63	2.2	7.30	6.27	6.9	2.6	8.9

Table 1: Accuracy and inference efficiency comparison. Our Geometric Head variants on frozen Qwen2-VL-2B and LLaVA-OV-0.5B backbones ( Emb. = ✓ indicates embodied training; Head

\in\{\text{token},\text{geo}\}

). We report active backbone parameters (#P(B)), accuracy metrics (DFD, HD, RMSE, mAP), throughput on H100 and RTX 3090 (FPS

\uparrow

), and compute cost (TFLOPs

\downarrow

). † Fully trained on embodied data; all other large VLMs are evaluated zero-shot. SRB=ShareRobot-Bench. Geo-Head substantially outperforms zero-shot and LoRA baselines and approaches 7B/RoboBrain accuracy at a fraction of the parameters and compute.

Variant	Representation	Backbone Adaptation	Decoder	Output	Qwen2-VL-2B		LLaVA-OV-0.5B
Variant	Representation	Backbone Adaptation	Decoder	Output	Traj. DFD $\downarrow$	Aff. mAP $\uparrow$	Traj. DFD $\downarrow$	Aff. mAP $\uparrow$
Base VLM	Full LLM	None	AR	Tokens	0.798	7.30	0.777	1.09
LoRA-AR	Full LLM	LoRA	AR	Tokens	$0.752\pm 0.061$	$2.25\pm 0.60$	$0.695\pm 0.001$	$2.39\pm 0.11$
$l^{*}$ -LoRA-AR	Full LLM	LoRA after $l^{*}$	AR	Tokens	$0.680\pm 0.010$	$2.39\pm 0.20$	$0.680\pm 0.030$	$2.04\pm 0.21$
LoRA-Reg	Final token	LoRA	LT+MLP	Continuous	$0.34\pm 0.008$	$3.24\pm 0.06$	$0.43\pm 0.2$	$5.78\pm 1.04$
$l_{0}$ -MLP	$l_{0}$	Frozen	MLP	Continuous	$0.291\pm 0.002$	$8.74\pm 0.97$	$0.311\pm 0.002$	$8.68\pm 0.10$

Table 2: Disentangling the source of performance gains. Continuous regression from the final VLM representation already substantially outperforms autoregressive token decoding, while intermediate features and DETR provide additional gains. Pre-LLM controls test performance before language-model contextualization. AR denotes autoregressive decoding and LT+MLP denotes direct regression from a learned token using an MLP. Results are reported as mean

\pm

standard deviation where applicable.

Figure 3: Layer-wise representational analysis. Trajectory DFD on ShareRobot-Bench-Trajectory and affordance mAP on AGD20K across all language-model layers for Qwen2-VL-2B and LLaVA-OV-0.5B, compared against the base VLM and LoRA fine-tuned autoregressive baselines. Colored markers indicate the selected layer used in the final model.

Panel of Figure 4 — Figure 4: Qualitative trajectory predictions. Red: predicted, green: ground truth. Top row: vanilla VLM baseline. Bottom row: with our geometry head.

Panel of Figure 5 — Figure 5: Qualitative affordance predictions. Red: predicted box, green: ground truth. Top row: vanilla VLM baseline. Bottom row: with our geometry head.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Efficiency–performance trade-off across layers on Qwen2-VL-2B for the trajectory prediction task.

		Trajectory			Affordance
Backbone	Query	DFD $\downarrow$	HD $\downarrow$	RMSE $\downarrow$	mAP $\uparrow$
Qwen2-VL-2B	Last token	$0.251\pm 0.010$	$0.219\pm 0.008$	$0.177\pm 0.007$	$12.37\pm 1.16$
Qwen2-VL-2B	Attn. pool	$0.215\pm 0.002$	$0.194\pm 0.001$	$0.155\pm 0.002$	$8.67\pm 1.01$
LLaVA-OV-0.5B	Last token	$0.225\pm 0.039$	$0.225\pm 0.003$	$0.179\pm 0.003$	$9.53\pm 1.44$
LLaVA-OV-0.5B	Attn. pool	$0.256\pm 0.004$	$0.24\pm 0.01$	$0.189\pm 0.002$	$7.27\pm 1.2$

Appendix

Table 3: Ablation of query token selection for trajectory prediction on ShareRobot-Bench-Trajectory and affordance prediction on AGD20K.

Model	DFD↓	HD↓	RMSE↓
w/ start & end point loss	$0.251\pm 0.010$	$0.219\pm 0.008$	$0.177\pm 0.007$
w/ only overall point loss	$0.322\pm 0.02$	$0.26\pm 0.01$	$0.26\pm 0.03$

Appendix

Table 4: Ablation on start and end point supervision for trajectory prediction head (Qwen2-VL-2B, ShareRobot-Bench).

Panel of Figure 7 — Figure 7: Predicted waypoint heatmaps over the test set, visualizing how endpoint supervision shapes the spatial distribution of trajectory predictions. (a) Ground-truth distribution: waypoints are broadly spread, reflecting the spatial diversity of manipulation trajectories in the dataset. (b) Zero-shot Qwen2-VL-2B baseline: predictions collapse to the image boundaries, confirming that vanilla autoregressive generation fails to produce spatially meaningful trajectories. (c) Geometry head without endpoint supervision: predictions collapse to a single central region, as the uniform L1 objective is minimised by regressing toward the dataset mean. (d) Geometry head with endpoint supervision: the distribution recovers spatial diversity matching the ground truth, demonstrating that boundary supervision forces input-specific localization rather than dataset-level averaging.

Panel of Figure 8 — Figure 8: Additional visualizations of diverse trajectories predicted by Qwen2-VL-2B + Geo-Head. Green curve represents the ground-truth and the red curve represents the prediction. The visualizations in the first three rows demonstrate that our geometry head effectively generates end-effector manipulation trajectories based on the observation and task instruction. The fourth row presents some failure cases arising from object-localization errors or insufficient spatial understanding.

Panel of Figure 9 — Figure 9: Additional visualizations of diverse trajectories predicted by LLaVA-OV-0.5B + Geo-Head. Green curve represents the ground-truth and the red curve represents the prediction. The visualizations in the first three rows demonstrate that our geometry head effectively generates end-effector manipulation trajectories based on the observation and task instruction. The fourth row presents some failure cases.

Panel of Figure 10 — Figure 10: Additional visualizations of diverse affordance regions predicted by Qwen2-VL-2B + Geo-Head. Green box represents the ground-truth and the red box represents the prediction. The visualizations in the first three rows demonstrate that our geometry head effectively generates affordance boxes based on the observation and task instruction. The fourth row presents some failure cases.

Panel of Figure 11 — Figure 11: Additional visualizations of diverse affordance regions predicted by LLaVA-OV-0.5B + Geo-Head. Green box represents the ground-truth and the red box represents the prediction. The visualizations in the first three rows demonstrate that our geometry head effectively generates affordance boxes based on the observation and task instruction. The fourth row presents some failure cases.

VLMs for Robotics

Vision-language models for robotics connect visual and linguistic understanding to grounded navigation, manipulation, and robot control.

See all 664 papers

SpatialAfford: Teaching Compact VLMs Where to Look and Where to Ground for Affordance
Aug 1, 2026Yufei Zhang, Chenlu Zhan, Donghui Sun +2Vision-Language Grounding VLMs for Robotics
Zhejiang University · Vivo Mobile Communication Co., Ltd
Evaluating VLMs' Spatial Reasoning Over Robot Motion: A Step Towards Robot Planning with Motion Preferences
Mar 13, 2026Wenxi Wu, Jingjing Zhang, Martim BrandãoVLM Reasoning VLMs for Robotics
King’s College London · University College London
Geometric Encoding for Spatial Reasoning in Vision-Language Models
Sep 28, 2026Antonio Jun, Haoshui Yu, Zhengyi Lu +2Visual Spatial Reasoning Representation Geometry in VLMs
School of Arts and Sciences New York University New York City, United States · Dept. of Engineering and Computer Science Oakland University Rochester, United States

Aug 1, 2026J/K move · Enter open · S save

SpatialAfford: Teaching Compact VLMs Where to Look and Where to Ground for Affordance

Yufei Zhang, Chenlu Zhan, Donghui Sun, Xiaoxin Chen+1

Zhejiang University · Vivo Mobile Communication Co., Ltd

Vision-Language Grounding VLMs for Robotics

Open paper AskPDF arXiv

Affordance grounding aims to localize the functional region for interaction, such as the handle to grasp or the button to press, rather than the whole object. This makes it more challenging than generic visual grounding because the target region is smaller, more ambiguous, and more dependent on task context, especially for compact vision-language models (VLMs) used in embodied settings. Recent sequence-level supervision and reinforcement learning improve coordinate prediction quality, yet compact autoregressive VLMs still lack reliable affordance-aware visual focus before coordinate generation: the model can produce better coordinate tokens while its cross-modal attention remains diffuse and weakly anchored to the true affordance evidence. To address it, we propose SpatialAfford, a two-stage framework that first aligns attention to the ground-truth affordance region through Spatial Attention Alignment (SAA), then refines coordinate prediction with Spatial-Aware GRPO. By explicitly teaching the model where to look before optimizing where to ground, SpatialAfford turns affordance grounding from a purely output-constrained objective into attention-grounded spatial reasoning. Across ShareRobot-Bench, ReasonAff, and PartAfford, SpatialAfford consistently improves affordance grounding, with a compact 4B model outperforming stronger 7B+ baselines.

Abstract

VLMs for Robotics

Explore similar work

SpatialAfford: Teaching Compact VLMs Where to Look and Where to Ground for Affordance

Evaluating VLMs' Spatial Reasoning Over Robot Motion: A Step Towards Robot Planning with Motion Preferences

Geometric Encoding for Spatial Reasoning in Vision-Language Models