HeiSD: Hybrid Speculative Decoding for Embodied Vision-Language-Action Models with Kinematic Awareness

Zihao Zheng; Zhihao Mao; Sicheng Tian; Maoliang Li; Jiayu Chen; Xinhao Sun; Zhaobo Zhang; Xuanzhe Liu

cs.ROMar 18, 2026

HeiSD: Hybrid Speculative Decoding for Embodied Vision-Language-Action Models with Kinematic Awareness

Authors: Zihao Zheng, Zhihao Mao, Sicheng Tian, Maoliang Li, Jiayu Chen, Xinhao Sun, Zhaobo Zhang, Xuanzhe Liu, +3 more

Organizations: School of Computer Science, Peking University · School of Computer Science, China University of Geosciences (Wuhan) · School of Artificial Intelligence, Beijing Normal University · School of EECS, Peking University

Abstract

Vision-Language-Action (VLA) Models have become the mainstream solution for robot control, but suffer from slow inference speeds. Speculative Decoding (SD) is a promising acceleration method which can be divided into two categories: drafter-based SD and retrieval-based SD. Each of the two methods demonstrates complementary advantages and limitations when applied to VLA models, leading to the hypothesis that a hybrid approach integrating these two methods will yield better performance. In this paper, we first conduct a series of detailed analyses to reveal the advantages and feasibility of hybrid utilization. However, even with the aforementioned key insights, implementing hybrid SD in VLA models presents several challenges: (1) draft rejection and persistent errors in retrieval-based SD; (2) difficulty in determining the hybrid boundary. To address these, we propose the HeiSD framework. We propose a retrieval-based SD optimization method in HeiSD, which contains a verify-skip mechanism and a sequence-wise relaxed acceptance strategy. Moreover, we proposed a kinematic-based fused metric in HeiSD to automatically determine the hybrid boundary. Experimental results demonstrate that HeiSD attains a speedup of up to 2.45x in simulation benchmarks and 2.06x~2.41x in real-world scenarios, while sustaining a high task success rate.

Figures & tables

Figure 1. Overview of the Proposed HeiSD Framework Fig.~(a) illustrates the implementation approaches of Drafter-based SD and Retrieval-based SD, while Fig.~(b) depicts the proposed \textit{HeiSD} framework. — Figure 1. Overview of the Proposed HeiSD Framework Fig.~~(a) illustrates the implementation approaches of Drafter-based SD and Retrieval-based SD, while Fig.~~(b) depicts the proposed *HeiSD* framework.

Figure 2. Database Building Process and Corresponding Tests This figure illustrates the establishment process of our database and the testing for its correctness.

Figure 3. Trajectory Analysis and Real-world/Simulation Validation for both VLA Inference and Database Retrieval xxx

Model	Environment	VLA Inference		Database Retrieval
Model	Environment	SR	Speed	SR	Speed
OpenVLA	LIBERO-Goal	77.0%	1.00 $\times$	62.0%	4.17 $\times$
	LIBERO-Object	71.2%	1.00 $\times$	68.0%	4.83 $\times$
	LIBERO-Spatial	82.8%	1.00 $\times$	53.0%	3.98 $\times$
	LIBERO-Long	54.4%	1.00 $\times$	18.0%	3.74 $\times$

Table 1. Database Performance on LIBERO Benchmark

Dataset	VLA+SD (SpecVLA)			Retrieval-Based SD
Dataset	SR	AL	Speed	SR	AL	Speed
LIBERO-Goal	71.0%	2.97	1.00 $\times$	77.0%	1.03	0.94 $\times$
LIBERO-Object	62.4%	3.25	1.00 $\times$	76.0%	0.93	0.92 $\times$
LIBERO-Spatial	80.4%	3.27	1.00 $\times$	78.0%	0.81	0.78 $\times$
LIBERO-Long	46.2%	2.82	1.00 $\times$	50.2%	0.82	0.97 $\times$

Table 2. Performance Comparison of Two Types of SD

Figure 4. Adaptive Verify-Skip Mechanism xxx

Figure 5. Sequence-Wise Relaxed Acceptance Strategy xxx

Figure 6. Kinematic-Based Metric for Achieving Hybrid SD xxx

Figure 7. System Implementation of HeiSD Framework xxx

Env.	Method	SR	Speed	AL	Steps	HW
LIBERO-Goal	AR w/o SD	77.0%	1.00 $\times$	–	157.6	GPU
	Pure R-SD	77.0%	0.96 $\times$	1.03	152.6	CPU+GPU
	Pure D-SD	76.2%	0.87 $\times$	1.68	159.2	GPU
	SpecVLA	71.0%	1.23 $\times$	3.63	166.8	GPU
	HeiSD	73.0%	2.38 $\times$	4.75	156.3	CPU+GPU
LIBERO-Object	AR w/o SD	71.2%	1.00 $\times$	–	191.7	GPU

Table 3. Simulation Results of HeiSD

Figure 8. Hybrid Raito and Verify-Skip Ratio in HeiSD xxx

Task Category	Fine-Tune	HeiSD SR	HeiSD Speedup	HeiSD AL
Atomic Grasping	87.2%	86.0%	2.33 $\times$	4.47
Spatial Displacement	77.3%	75.1%	2.41 $\times$	4.39
Composite Sequential	71.7%	67.8%	2.06 $\times$	4.15

Table 4. Real-World Results of HeiSD

Figure 9. A Case of HeiSD Framework Completing Real-World Tasks (Task Name: Pick up the Banana and Put It on the Plate) xxx

	SR	Speedup	AL
Only Hybrid SD	74.0%	1.05 $\times$	1.05
+ Adaptive Verify-Skip	73.0% $\downarrow$ 1.0%	2.08 $\times$ $\uparrow$ 1.03 $\times$	4.04 $\uparrow$ 2.99
+ Seq-Wise Relaxed Acceptance	73.0% $\uparrow$ 0.0%	2.38 $\times$ $\uparrow$ 0.30 $\times$	4.50 $\uparrow$ 0.46

Table 5. Ablation Studies of HeiSD on LIBERO-Goal

Figure 10. Discussion of Hyper-Parameter

w

in HeiSD xxx

Details	Goal	Object	Spatial	Long
HeiSD on GPU	7388.9 s	7731.7 s	6557.0 s	23318.5 s
HeiSD on CPU+GPU	7113.6 s	7328.4 s	6238.5 s	21352.7 s
Acceleration	1.04 $\times$	1.06 $\times$	1.05 $\times$	1.09 $\times$

Table 6. Hardware Implementation Analysis of HeiSD

Figure 11. Discussion of Hyper-Parameter

\alpha

in HeiSD xxx

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 12. Common Structure of Vision-Language-Action Models.

Figure 13. Representative initial scenes from the four LIBERO subsets. From left to right: LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-Long. These MuJoCo tabletop environments, manipulated by a 7-DoF Franka Panda arm, illustrate distribution shifts across spatial configurations, object instances,and goal specifications.

Dataset	Tasks	Episodes	Total Steps	Avg. Steps/Episode	Size	Files
LIBERO-Goal	10	428	52,042	121.59 $\pm$ 37.62	1.7 GB	18
LIBERO-Spatial	10	432	52,970	122.62 $\pm$ 20.75	1.8 GB	18
LIBERO-Object	10	454	66,984	147.54 $\pm$ 18.53	2.6 GB	34
LIBERO-Long	10	379	101,469	267.73 $\pm$ 56.74	3.4 GB	34

Appendix

Table 7. LIBERO Dataset Statistics

Figure 14. 3D visualization of database embedding vectors

Dataset	Ttl. Vectors	Size	Ttl. Size
LIBERO-Goal	52,042	1.38 GB	6.5 GB
LIBERO-Object	66,984	1.58 GB
LIBERO-Spatial	52,970	1.38 GB
LIBERO-Long	101,469	2.16 GB

Appendix

Table 8. Vector Database Storage Statistics

Figure 15. Distribution of retrieval confidence (cosine similarity scores) across the four LIBERO task suites. The consistently high scores ( $>$ 0.90 for most queries) demonstrate the reliability of our constructed database for action retrieval.

Figure 16. Empirical distributions of Cumulative Spatial Displacement ( $D^{[w]}$ ) and Radius of Curvature ( $R^{[w]}$ ) across the four LIBERO task suites. The dashed lines indicate the 95 ${}^{\textnormal{th}}$ percentile thresholds used for normalization.

Task Suite	$D_{\min}$	$D_{\max}$ (95%)	$R_{\min}$	$R_{\max}$ (95%)
LIBERO-Goal	0.000009	0.123381	0.000001	0.014989
LIBERO-Spatial	0.000027	0.128629	0.000019	0.015654
LIBERO-Object	0.000098	0.116458	0.000010	0.014151
LIBERO-Long	0.000008	0.102298	0.000001	0.012479

Appendix

Table 9. Normalization bounds for

D^{[w]}

and

R^{[w]}

across LIBERO task suites. All values are derived from the training demonstrations, with the upper bound set at the 95

{}^{\textnormal{th}}

percentile.

Figure 17. Our Tabletop Operation Environment.

Dataset	Domain	Hardware Setup	Task Complexity	Obj. & Env. Diversity
OXE ( O’Neill et al., 2024 )	Real	Heterogeneous	Atomic / Short-horizon	High
RoboCasa ( Nasiriany et al., 2024 )	Simulation	Standardized	Long-horizon	High
SimplerEnv ( Li et al., 2024a )	Sim / Real	Standardized	Atomic	Low
Ours	Real	Standardized	Hierarchical	High

Appendix

Table 10. Task Comparison with Existing Popular Datasets

Task Category	Description	Instruction Examples	Complexity
Atomic Grasping	Grasping a specific target object from the tabletop without interacting with other objects.	$\bullet$ “Pick up the apple.” $\bullet$ “Pick up the banana.”	Low
Spatial Displacement	Moving an object from a starting position to a target region or a specific container.	$\bullet$ “Move the apple from point A to point B.” $\bullet$ “Pick up the mango from the plate and put it into the bowl.”	Medium
Composite Sequential	Long-horizon tasks requiring multi-step planning and state memory to manipulate multiple objects in sequence.	$\bullet$ “Take the apple out of the plate and put it in the bowl, then place the banana onto the plate.”	High

Appendix

Table 11. Task Categories and Examples

Explore similar work

KERV: Kinematic-Rectified Speculative Decoding for Embodied VLA Models
Mar 2, 2026Zihao Zheng, Zhihao Mao, Maoliang Li +6Efficient VLA Model Inference Vision-Language-Action Models
School of Computer Science, Peking University, Beijing, China · School of Computer Science, China University of Geosciences (Wuhan), Wuhan, China
WA-SpecDec: World-Aware Speculative Decoding for Vision-Language-Action Models
Aug 9, 2026Zikang Wen, Yuning Zhang, Dong YuanEfficient VLA Model Inference Speculative Decoding
The University of Sydney
Reducing Temporal Redundancy for Efficient Vision-Language-Action Inference
Jul 14, 2026Yuzhou Wu, Yuxin Zheng, Muchun Niu +6Efficient VLA Model Inference Efficient VLA Models
1Tianji KernalMind co ltd · 2Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · 3Shanghai Jiao Tong University, Shanghai, China +2

Mar 2, 2026J/K move · Enter open · S save

KERV: Kinematic-Rectified Speculative Decoding for Embodied VLA Models

Zihao Zheng, Zhihao Mao, Maoliang Li, Jiayu Chen+5

School of Computer Science, Peking University, Beijing, China · School of Computer Science, China University of Geosciences (Wuhan), Wuhan, China

Efficient VLA Model Inference Vision-Language-Action Models Robotics

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models build a token-domain robot control paradigm, yet suffer from low speed. Speculative Decoding (SD) is an optimization strategy that can boost inference speed. Two key issues emerge when integrating VLA and SD: first, SD relies on re-inference to address token errors, which is computationally expensive; second, to mitigate token errors, the acceptance threshold in SD requires careful adjustment. Existing works fail to address the above two issues effectively. Meanwhile, as the bridge between AI and the physical world, existing embodied intelligence has overlooked the application of robotic kinematics. To address these issues, we innovatively combine token-domain VLA models with kinematic-domain prediction for SD, proposing a kinematic-rectified SD framework named KERV. We employ a kinematics-based Kalman Filter to predict actions and compensate for SD errors, avoiding costly re-inference. Moreover, we design a kinematics-based adjustment strategy to dynamically rectify the acceptance threshold, addressing the difficulty of threshold determination. Experimental results across diverse tasks and environments demonstrate that KERV achieves 27%~37% acceleration with nearly no Success Rate loss.

Abstract

Explore similar work

KERV: Kinematic-Rectified Speculative Decoding for Embodied VLA Models

WA-SpecDec: World-Aware Speculative Decoding for Vision-Language-Action Models

Reducing Temporal Redundancy for Efficient Vision-Language-Action Inference