Beyond Token Importance: Preserving Spatial Scaffolds for Efficient Vision-Language-Action Inference

Jiayu Chen; Shuyong Gao; Jingkai Jia; Xiaosheng Bu; Jiyuan Fu; Lingyi Hong; Kaixun Jiang; Yipan Xu

cs.ROSep 29, 2026

Beyond Token Importance: Preserving Spatial Scaffolds for Efficient Vision-Language-Action Inference

Authors: Jiayu Chen, Shuyong Gao, Jingkai Jia, Xiaosheng Bu, Jiyuan Fu, Lingyi Hong, Kaixun Jiang, Yipan Xu, +1 more

Organizations: Fudan University · The Hong Kong Polytechnic University

Abstract

Existing VLA pruning strategies primarily select individual visual tokens according to task-level semantic relevance, while overlooking the spatial information required for robotic manipulation. To examine this limitation, we construct a simple Stride baseline that uniformly samples tokens along the flattened one-dimensional visual sequence, representing a purely geometric pruning strategy. Surprisingly, Stride outperforms semantic pruning and random pruning at certain pruning ratios, but collapses when the token budget is only slightly reduced. We characterize this phenomenon through the spatial coverage radius, defined as the largest spatial blind spot induced by the retained token set after pruning. Our analysis reveals a strong correlation between the spatial structure of retained tokens and task success, suggesting that reliable VLA pruning requires preserving not only task-relevant tokens but also the spatial scaffold of the scene. Motivated by this diagnosis, we propose GeoScaffold, a training-free visual token pruning method that partitions each image into spatial regions, allocates inter-region token budgets using task-relevance weights, and selects intra-region scaffold tokens via farthest point sampling to reduce the local coverage radius. On pi 0.5 and LIBERO, GeoScaffold retains only 20% of visual tokens while preserving a 93.2% average success rate, and achieves a 1.78 times prefill speedup over the unpruned baseline.

Figures & tables

Figure 1: (a) Structured stride pruning can outperform random and attention-based pruning at moderate rates, yet fails abruptly at a nearby pruning rate. (b) Different pruning policies induce distinct spatial layouts of retained tokens, exposing large uncovered regions under aggressive pruning. (c) The coverage radius, measuring the largest spatial blind spot, shows a strong positive correlation with pruning-induced error across pruning policies and rates.

Figure 2: Overview of GeoScaffold. Three steps implement the two-level decomposition: Step 1: Region Scoring , aggregate token attention into region weights; Step 2: Budget Allocation , assign budgets with a one-token spatial floor per region; Step 3: Scaffold Selection , use center-seeded farthest point sampling to minimize local coverage radius.

Method	Spatial	Object	Goal	Long	Avg.	prefill (ms) $\downarrow$	total (ms) $\downarrow$
keep $256/256$ tokens per image, pruning rates = 0%
$\pi_{0.5}$ baseline	98.6	98.6	98.6	92.4	$\mathbf{97.1}$	27.68	61.03
keep $51/256$ tokens per image, pruning rates = 80%
FastV [ 6 ]	75.4	86.6	74.0	64.2	75.0	13.51	45.52
SparseVLM [ 26 ]	39.4	80.6	66.2	52.8	59.8	13.53	45.51
GeoScaffold (ours)	97.6	96.6	90.6	88.2	93.2	15.58 ( $\times$ 1.78)	47.56 ( $\times$ 1.28)

Table 1: Comparison of different VLA acceleration methods on the LIBERO benchmark.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

keep $512/512$ tokens per image, pruning rates = 0%
Method	Spatial	Object	Goal	Long	Avg.
OpenVLA-OFT baseline	95.8	98.7	96.3	90.7	95.4
keep $128/512$ tokens per image, pruning rates = 75%
VLA-Cache [ 21 ]	85.6	85.3	84.1	80.3	83.8
FastV [ 6 ]	92.6	89.8	96.2	83.2	90.5
GeoScaffold (ours)	90.4	86.0	96.8	90.0	90.8

Appendix

Table 3: Performance on LIBERO at OpenVLA-OFT [ 9 ] across two aggressive pruning rates. Bold = column best within each block.

Method	image_encode	prefill	denoise	total	prefill $\uparrow$	total $\uparrow$
Baseline (no pruning)	11.28	27.68	22.08	61.03	$1.00\times$	$1.00\times$
GeoScaffold $R{=}0.80$	11.19	15.58	20.80	47.56	$1.78\times$	$1.28\times$
GeoScaffold $R{=}0.875$	11.20	14.22	20.68	46.10	$1.95\times$	$1.32\times$
GeoScaffold $R{=}0.90$	11.20	14.11	20.63	45.92	$1.96\times$	$1.33\times$

Appendix

Table 4: Per-stage single-step latency (ms) of

\pi_{0.5}

on RTX 4090 under different pruning rates.

Asset	License or Terms	Usage in This Work
$\pi_{0.5}$ / OpenPI	Apache License 2.0	Primary VLA model for inference-time pruning evaluation; no redistribution of model weights.
OpenVLA-OFT	MIT License	Cross-architecture evaluation of the proposed pruning principle; no redistribution of model weights.
LIBERO	MIT License	Simulation benchmark and evaluation task suites.
FastV	License not specified	Algorithmic baseline for visual token pruning; no FastV code is redistributed.
SparseVLM	Apache License 2.0	Visual-token pruning baseline.
VLA-Cache	Apache License 2.0	Training-free VLA acceleration baseline.

Appendix

Table 5: Existing assets used in this work and their licenses or terms of use.

Explore similar work

SAFE-Pruner: Semantic Attention-Guided Future-Aware Token Pruning for Efficient Vision-Language-Action Manipulation
May 28, 2026Shilin Ma, Chubin Zhang, Changyuan Wang +6Diffusion-Based Vision-Language-Actions Visual Token Pruning
Tsinghua Shenzhen International Graduate School, Tsinghua University · GigaAI
See What Matters: Differentiable Grid Sample Pruning for Generalizable Vision-Language-Action Model
May 12, 2026Yixu Feng, Zinan Zhao, Yanxiang Ma +4Diffusion-Based Vision-Language-Actions Visual Token Pruning
The University of Sydney · City University of Hong Kong · StellarEdge Robotics
History-Conditioned Spatio-Temporal Visual Token Pruning for Efficient Vision-Language Navigation
Mar 6, 2026Qitong Wang, Yijun Liang, Ming Li +2Vision-Language Navigation Visual Token Pruning
Department of Computer and Information Sciences at the University of Delaware, Newark, DE 19711, USA · University of Maryland’s Department of Computer Science, College Park, MD 20742, USA · Mohamed bin Zayed University of Artificial Intelligence, Masdar City, Abu Dhabi 7909, UAE

May 28, 2026, cs.CVJ/K move · Enter open · S save

SAFE-Pruner: Semantic Attention-Guided Future-Aware Token Pruning for Efficient Vision-Language-Action Manipulation

Shilin Ma, Chubin Zhang, Changyuan Wang, Yuji Wang+5

Tsinghua Shenzhen International Graduate School, Tsinghua University · GigaAI

Diffusion-Based Vision-Language-Actions Visual Token Pruning Structured Pruning Saliency

Open paper AskPDF arXiv

Real-time inference of vision-language-action (VLA) models is essential for robotic control. While visual token pruning has shown strong potential for accelerating inference, most existing methods mainly base pruning decisions on shallow-layer cues and risk discarding visual information required by deep layers. To address this issue, we propose SAFE-Pruner, a plug-and-play pruning framework that incorporates attention cues of future layers into pruning decisions. Specifically, we identify semantic attention consistency, the tendency that VLA models concentrate their attention probability mass on the same semantic entity across execution steps. Based on this observation, we design a forward-looking strategy to forecast the token saliency in deep layers, which prevents the premature removal of critical tokens and leads to more stable acceleration. We further introduce an adaptive subtask division strategy to detect abrupt attention shifts, thereby improving forecasting accuracy and pruning reliability. Extensive experiments in simulation and real-world settings demonstrate that our method achieves up to 1.89x speedup with a minimal degradation in success rate of less than 1.7%, while outperforming state-of-the-art methods by up to 1.9%.

Abstract

Explore similar work

SAFE-Pruner: Semantic Attention-Guided Future-Aware Token Pruning for Efficient Vision-Language-Action Manipulation

See What Matters: Differentiable Grid Sample Pruning for Generalizable Vision-Language-Action Model

History-Conditioned Spatio-Temporal Visual Token Pruning for Efficient Vision-Language Navigation