SpecPrune-VLA: Accelerating Vision-Language-Action Models via Action-Aware Self-Speculative Pruning

Hanzhen Wang; Jiaming Xu; Yushun Xiang; Jiayi Pan; Yongkang Zhou; Yong-Lu Li; Guohao Dai

cs.CVSep 6, 2025

SpecPrune-VLA: Accelerating Vision-Language-Action Models via Action-Aware Self-Speculative Pruning

Authors: Hanzhen Wang, Jiaming Xu, Yushun Xiang, Jiayi Pan, Yongkang Zhou, Yong-Lu Li, Guohao Dai

Organizations: Shanghai Jiao Tong University · SII · Infinigence-AI

Abstract

Pruning is a typical acceleration technique for compute-bound models by removing computation on unimportant values. Recently, it has been applied to accelerate Vision-Language-Action (VLA) model inference. However, existing acceleration methods focus on local information from the current action step and ignore the global context, leading to >20% success rate drop and limited speedup in some scenarios. In this paper, we point out spatial-temporal consistency in VLA tasks: input images in consecutive steps exhibit high similarity, and propose the key insight that token selection should combine local information with global context of the model. Based on this, we propose SpecPrune-VLA, a training-free, two-level pruning method with heuristic control. (1) Action-level static pruning. We leverage global history and local attention to statically reduce visual tokens per action. (2) Layer-level dynamic pruning. We prune tokens adaptively per layer based on layer-wise importance. (3) Lightweight action-aware controller: We classify actions as coarse- or fine-grained by the speed of the end effector and adjust pruning aggressiveness accordingly. Extensive experiments show that SpecPrune-VLA achieves up to 1.57 $\times$ speedup in LIBERO simulation and 1.70 $\times$ on real-world tasks, with negligible success rate degradation.

Figures & tables

Figure 1: (a) The mainstream inference dataflow of VLA models. (b) Latency breakdown in four typical VLA models during each action generation. * In DB-OFT, we report the time of LLM backbone and Action Expert per denoising step, with the tokenizer’s single forward pass time averaged evenly across the 10 steps. (c) The practical arithmetic intensity of four models in different cases in the roofline model of NVIDIA A800 GPU.

Figure 2: Overview of SpecPrune-VLA. We prune the visual tokens with global and local information with a lightweight action-aware controller.

Figure 3: Insight 1: (a) Layers of different depth focus on different information. (b)(c)(d) In pick and place task, random pruning causes important tokens missed out and pruning based on global attention retain important tokens and ensure success rate. (e) The performance of different guiding layer selection. Insight 2: (a) The recall score of Top-30 globally important token sets of consecutive inference remains high in different camera view. (b) The visualization of globally important tokens between consecutive steps.

Panel of Figure 4 — Figure 4: (I) Attention entropy differs across layers, but are similar throughout the task. (II) (a) Comparison of the hitrate between leveraging the first one, two, and three layers. (b) LLM latency comparison between leveraging the first one, two, and three layers.

Figure 5: Adaptive selection strategy for frame comparison.

Figure 6: (a) The task process consists of four stages, categorized into coarse- and fine-grained actions based on velocity. (b) Typical failures in fine-grained stages.

Figure 7: (a) and (b) Ablation study on base K value. K values are chosen to maximize the Recall rate. (c) and (d) Ablation study on prune rate.

LIBERO benchmark	Success Rate (%)				Avg. Speedup	Avg. SR(%)	FLOPs
Method	Spatial	Object	Goal	Long	Avg. Speedup	Avg. SR(%)	FLOPs
OpenVLA-OFT	97.6%	96.5%	97.9%	94.5%	1.00 $\times$	96.6%	100%
+ FastV [ECCV24]	94.6%	95.8%	94.0%	88.8%	1.44 $\times$	93.3%	57%
+ DivPrune [CVPR25]	92.4%	91.2%	89.0%	84.8%	1.46 $\times$	89.4%	54%
+ SparseVLM [ICML25]	96.8%	94.2%	97.6%	93.6%	1.28 $\times$	95.6%	77%
+ VLA-Cache [NIPS25]	99.0%	97.7%	97.4%	93.6%	1.07 $\times$	96.9%	83%

Table 1: Performance Evaluation (Success Rate and Average Speedup)

Figure 8: Ablation study of three techniques in LIBERO-spatial

Real world task	Success Rate (%)				Avg. Latency (ms)	Avg. Speedup
Method	Pick&Place	TransferTube	PickUpCup	MultiCubeTask	Avg. Latency (ms)	Avg. Speedup
OpenVLA-OFT	96.7%	85.0%	91.7%	95.0%	187.7	$1.00\times$
Ours	96.7%	82.0%	90.0%	95.0%	110.2	$1.70\times$

Table 2: Performance comparison on real-world robot tasks.

Table 11

Figure 9: Extended evaluation on NVIDIA 3090 GPU

Figure 10: Left: Real world robot, a Flexiv Rizon4 arm equipped with a gripper and three Intel cameras. The cameras are mounted separately on the wrist, on its side, and above its head. Right: Four real world evaluation tasks.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

SimplerEnv	Success Rate (%)				Avg. Speedup	Avg. SR(%)
Method	Spatial	Goal	Object	Long	Avg. Speedup	Avg. SR(%)
OpenVLA-OFT	92.8%	97.2%	97.6%	95.2%	1.00 $\times$	95.7%
+ Ours ( $\alpha$ =0.8)	92.8%	96.6%	97.2%	94.4%	1.48 $\times$	95.3%

Appendix

Table 5: Performance Evaluation (Success Rate and Speedup) on RTX 4090 GPU

Figure 11: Performance evaluation on LIBERO-plus and VLA-Arena benchmark.

Figure 12: The task process is categorized into coarse- and fine-grained actions based on velocity.

SimplerEnv Visual Matching	Success Rate (%)				Avg. Speedup	Avg. SR(%)
Method	PickCokeCan	MoveNear	Open/CloseDrawer	AppleDrawer	Avg. Speedup	Avg. SR(%)
CogACT	91.3%	85.0%	71.8%	50.8%	1.00 $\times$	74.7%
+ FastV [ECCV24]	92.6%	81.4%	69.8%	52.4%	1.21 $\times$	74.1%
+ VLA-Cache [NIPS25]	92.0%	83.3%	70.5%	51.6%	1.20 $\times$	74.4%
+ Ours ( $\alpha$ =0.8)	91.3%	83.8%	71.6%	50.2%	1.28 $\times$	74.2%
+ Ours&SC ( $\alpha$ =0.8)	91.7%	83.8%	71.2%	50.0%	1.42 $\times$	74.1%

Appendix

Table 6: Performance Evaluation of CogACT (Success Rate and Average Speedup) in the SimplerEnv benchmark, visual matching task

Table 7: Ablation study of success rate under different threshold settings on LIBERO-spatial.

Figure 13: The prefilling latency of eager attention and flash attention under different input length on A800 GPU. When original input length is less than 2048 (4 images) and 30% sparsity is applied on input, eager attention achieves consistently lower latency.

Prune Ratio $\alpha$	0.6	0.8	1.0
Pruned Tokens	382	360	336
Visual Retention	25.4%	29.8%	34.4%

Appendix

Table 8: Number of pruned tokens and visual retention rate under different prune ratios.

Similarity calculation	First 2 layers	Top-k sorting	Attention Entropy	Our Method	Original Model
$1.8\pm 0.1\,\text{ms}$	$5.5\pm 0.2\,\text{ms}$	$2.5\pm 0.1\,\text{ms}$	$1.0\pm 0.2\,\text{ms}$	$72\text{--}78\,\text{ms}$	$109.0\,\text{ms}$

Appendix

Table 9: Latency breakdown per module and comparison with the original model.

Method	Latency (ms) / Speedup				Avg. Latency(ms)	Avg. Speedup
Method	Spatial	Object	Goal	Long	Avg. Latency(ms)	Avg. Speedup
OpenVLA-OFT	109.0 / 1.00 $\times$	109.0 / 1.00 $\times$	109.0 / 1.00 $\times$	109.0 / 1.00 $\times$	109.0	1.00 $\times$
SparseVLM	85.3 / 1.28 $\times$	85.3 / 1.28 $\times$	85.3 / 1.28 $\times$	85.3 / 1.28 $\times$	85.3	1.28 $\times$
VLA-Cache	101.8 / 1.07 $\times$	101.8 / 1.07 $\times$	101.8 / 1.07 $\times$	101.8 / 1.07 $\times$	101.8	1.07 $\times$
EfficientVLA	71.7 / 1.52 $\times$	71.7 / 1.52 $\times$	71.7 / 1.52 $\times$	71.7 / 1.52 $\times$	71.7	1.52 $\times$
Ours( $\alpha$ = 0.8)	74.8 / 1.46 $\times$	74.5 / 1.46 $\times$	74.8 / 1.46 $\times$	75.8 / 1.44 $\times$	75.1	1.46 $\times$

Appendix

Table 10: Performance Evaluation (Latency and Speedup)

Prune Ratio $\alpha$	Success Rate (%) / Speedup				Avg. SR (%)	Avg. Speedup
Prune Ratio $\alpha$	Spatial	Object	Goal	10	Avg. SR (%)	Avg. Speedup
None (OpenVLA-OFT)	97.6 / 1.00 $\times$	96.5 / 1.00 $\times$	97.9 / 1.00 $\times$	94.5 / 1.00 $\times$	96.6	1.00 $\times$
$\alpha$ =1.0	97.6 / 1.41 $\times$	96.3 / 1.43 $\times$	97.9 / 1.40 $\times$	94.0 / 1.38 $\times$	96.5	1.40 $\times$
$\alpha$ =0.8	97.6 / 1.46 $\times$	95.8 / 1.46 $\times$	97.7 / 1.46 $\times$	93.4 / 1.44 $\times$	96.1	1.46 $\times$
$\alpha$ =0.6	97.4 / 1.52 $\times$	93.2 / 1.50 $\times$	96.0 / 1.52 $\times$	92.2 / 1.51 $\times$	94.7	1.51 $\times$

Appendix

Table 11: Ablation study on prune rate

\alpha

. Success rates (%) and speedup factors are reported per task. Our method achieves optimal trade-off at

\alpha=0.8

, achieving

1.46\times

average speedup with minimal accuracy drop.

Hyperparameter	Value
# GPUs	8 × NVIDIA H200 (141GB VRAM)
Learning rate (LR)	5e-4 (decays to 5e-5 after 50K steps for all tasks)
Total batch size	64 (8 per GPU)
# Train steps	“Pick up the cube and place it in the pan.”: 80K
	“Remove the cube from the tube rack.”: 100K
	“Pick up the purple cube and put it in the pot.”: 80K

Appendix

Table 12: OpenVLA-OFT hyperparameters for real-world experiments. Includes parallel decoding, action chunking, continuous actions with L1 regression, and additional inputs (wrist/side cameras + robot state).

Figure 14: Retained tokens during tasks across four datasets and different views.

Figure 15: The real-world experiment visualization of OpenVLA-OFT and our method across various tasks.

Efficient VLA Model Inference

Efficient vision-language-action (VLA) model inference reduces robot-control latency, memory use, and energy while preserving task performance.

See all 199 papers

SAFE-Pruner: Semantic Attention-Guided Future-Aware Token Pruning for Efficient Vision-Language-Action Manipulation
May 28, 2026Shilin Ma, Chubin Zhang, Changyuan Wang +6Efficient VLM Inference Robotic Manipulation
Tsinghua Shenzhen International Graduate School, Tsinghua University · GigaAI
VLA-ACL: Action-Consistent Visual Token Pruning for Efficient Vision-Language-Action Models
Oct 6, 2026Owen Du, Yang Yue, Jie Zhang +3Efficient VLA Models VLMs for Robotics
ETH Zurich · Tsinghua University
When and What to Prune? Stage-Aware Visual Token Pruning for Efficient VLA
Oct 4, 2026Tianjun Shi, Haotian Xiong, Ziyu Gong +2Efficient VLA Model Inference Vision-Language-Action Models
University of Macau

May 28, 2026J/K move · Enter open · S save

SAFE-Pruner: Semantic Attention-Guided Future-Aware Token Pruning for Efficient Vision-Language-Action Manipulation

Shilin Ma, Chubin Zhang, Changyuan Wang, Yuji Wang+5

Tsinghua Shenzhen International Graduate School, Tsinghua University · GigaAI

Efficient VLM Inference Robotic Manipulation Vision-Language-Action Models VLMs for Robotics Visual Token Pruning

Open paper AskPDF arXiv

Real-time inference of vision-language-action (VLA) models is essential for robotic control. While visual token pruning has shown strong potential for accelerating inference, most existing methods mainly base pruning decisions on shallow-layer cues and risk discarding visual information required by deep layers. To address this issue, we propose SAFE-Pruner, a plug-and-play pruning framework that incorporates attention cues of future layers into pruning decisions. Specifically, we identify semantic attention consistency, the tendency that VLA models concentrate their attention probability mass on the same semantic entity across execution steps. Based on this observation, we design a forward-looking strategy to forecast the token saliency in deep layers, which prevents the premature removal of critical tokens and leads to more stable acceleration. We further introduce an adaptive subtask division strategy to detect abrupt attention shifts, thereby improving forecasting accuracy and pruning reliability. Extensive experiments in simulation and real-world settings demonstrate that our method achieves up to 1.89x speedup with a minimal degradation in success rate of less than 1.7%, while outperforming state-of-the-art methods by up to 1.9%.

Abstract

Efficient VLA Model Inference

Explore similar work

SAFE-Pruner: Semantic Attention-Guided Future-Aware Token Pruning for Efficient Vision-Language-Action Manipulation

VLA-ACL: Action-Consistent Visual Token Pruning for Efficient Vision-Language-Action Models

When and What to Prune? Stage-Aware Visual Token Pruning for Efficient VLA