Vision-language-action (VLA) models enable robots to understand and execute complex instructions, but their autoregressive (AR) action generation introduces significant latency on resource-constrained robotic platforms. Speculative decoding can accelerate AR-based inference by parallelizing action generation and verification; however, existing approaches rely on costly draft models, which are inefficient on edge platforms. In this paper, we propose Zora, a hardware-software co-design framework for accelerating AR-based VLA inference. Specifically, Zora introduces a lightweight retrieval-augmented generation (RAG)-based draft model that exploits robotic dynamics without additional neural draft models. We further reduce batch verification overhead through a kinematics-guided action-acceptance strategy and a KV-cache-sharing attention mechanism. To address the retrieval overhead introduced by RAG-based drafting, we co-design dedicated hardware support, Zora core, to address the memory irregularity in RAG-based drafting. We evaluate Zora on four VLA models with the LIBERO benchmark. Zora retains comparable accuracy while achieving up to 3.4× speedup with software-only optimization and up to 7.6× speedup with hardware support over GPUs.
Figures & tables
Fig. 1: Inference process of single-action autoregressive VLA models. With the visual observation ot and language instruction I , the model first performs a prefill step to encode the input context and generate the KV cache. During decoding, action tokens are generated autoregressively and appended to the KV cache. Once all action tokens are generated, including relative movement Δxt+1 , rotation change Δrt+1 , and gripper state gt+1 , the VLA model then uses a small action detokenizer to decode the action and control the robot. Lastly, the execution updates the next observation for subsequent action generation.
Fig. 2: The HNSW-based retrieval process. The input is first encoded into a query embedding, which is used to search the vector database before selecting the top- K retrieved results for downstream VLA models. The search database is implemented with a HNSW graph, where search proceeds in a coarse-to-fine manner, from the top layer to the bottom layer.
Figure 3Figure 4
Fig. 7: Zora overview. Given the current robot state st , visual observation ot , and language instruction I , Zora first encodes them into tokens in the latent space. The dynamics-guided retrieval module retrieves top- k candidate action sequences with a length of w from past execution contexts. An action checker then sequentially checks the retrieved action tokens and classifies them into high-confidence and low-confidence tokens. High-confidence tokens are directly committed for robotic execution, while low-confidence tokens are verified by the original VLA model in batches. Verified tokens are also committed for robot execution, whereas failed tokens trigger fallback inference to re-execute the original VLA model. After each action is executed, current actions are updated, and the KV cache is recomputed. The top- k candidate sequences are refreshed whenever fallback is triggered, or candidate sequences are all verified.
Fig. 8: The process of DG retrieval. DG retrieval first constructs a query embedding from the current visual observations and robot state. DG retrieval then performs HNSW-based search over the retrieval database to find the top- k keys that are most similar to the query. The associated values are returned as candidate action sequences and passed to the consensus checker.
Figure 7
Fig. 11: Overview of the Zora accelerator architecture. The accelerator contains a Zora core to accelerate DG retrieval, a checker unit, and a DNN accelerator. Yellow blocks highlight our key contributions.
Fig. 12: Overview of the Zora core architecture. The core accelerates HNSW-based DG retrieval through a four-stage pipeline: fetch, compute, accumulation, and update. The PE array exploits node-level parallelism and embedding-level parallelism to compute partial similarity scores between the query embedding and neighbor embeddings. Partial similarity scores are accumulated and then consumed by either the reduction unit for greedy search in upper HNSW layers or the comparison units for top- k selection in the bottom layer. Lastly, the neighbor index fetch unit updates the search frontier for unvisited nodes.
Fig. 13: Design of the checker unit. It computes kinematic or consistency discrepancy scores with FΔ and compares them with thresholds to decide whether to commit an action or send it for model verification.
Fig. 14: Accuracy comparison over different speculative methods. Red crosses show SpecVLA is unavailable for the corresponding models.
Figure 12
Fig. 17: Robustness evaluation on Zora over different dataset coverage. All results are measured on OpenVLA . We show (a) accuracy and GPU performance, (b) execution path breakdown, and (c) acceptance length (AL).
Fig. 18: GPU performance of Zora variants. Base+DG applies only DG retrieval; Base+DG+BV applies conventional batch verification; Base+DG+MABV further applies masked attention for batched verification with shared prefix KV cache; Base+DG+MABV+K includes Kinematic Check; the full-fledged design, Zora(+C) , includes Consistency Check.
Fig. 19: The performance and energy savings over different hardware variants. Bars show relative gains over the per-model GPU baseline, and markers show absolute measurements, i.e., FPS and energy per task.
Figure 16Figure 17
Fig. 24: Sensitivity of performance and task success rate to different algorithmic hyperparameters in the Zora framework.
Vision-Language-Action (VLA) models have recently demonstrated impressive capabilities across various embodied AI tasks. While deploying VLA models on real-world robots imposes strict real-time inference constraints, the inference performance landscape of VLA remains poorly understood due to the large combinatorial space of model architectures and inference systems. In this paper, we ask a fundamental research question: How should we design future VLA models and systems to support real-time inference? To address this question, we first introduce VLA-Perf, an analytical performance model that can analyze inference performance for arbitrary combinations of VLA models and inference systems. Using VLA-Perf, we conduct the first systematic study of the VLA inference performance landscape. From a model-design perspective, we examine how inference performance is affected by model scaling, model architectural choices, long-context video inputs, asynchronous inference, and dual-system model pipelines. From the deployment perspective, we analyze where VLA inference should be executed -- on-device, on edge servers, or in the cloud -- and how hardware capability and network performance jointly determine end-to-end latency. By distilling 15 key takeaways from our comprehensive evaluation, we hope this work can provide practical guidance for the design of future VLA models and inference systems.
Vision-Language-Action (VLA) models have demonstrated remarkable capabilities in the field of embodied AI, but their high computational cost and limited predicted action length hinder real-time deployment. Although Dadu-Corki, a dedicated accelerator for efficient embodied AI, has been introduced, it does not exploit the inherent interaction patterns between the robot and its environment, which results in a relatively short predicted action length. We observe that robotic environments naturally alternate between active states-where precise actions are crucial-and inactive states-where actions have limited impact on task success. This insight enables a new scheduling opportunity: long-action-length speculative prediction in inactive states, paired with selective verification in active states. We propose SpecVLA, an algorithm-system co-design framework that adaptively balances action length, inference latency, and task reliability. On the algorithm side, SpecVLA introduces a state-aware VLA inference execution paradigm and a hardware-friendly construction of a smaller verification model (sVLA) using differential residuals and block-wise mixed-precision quantization. On the system side, we develop a heterogeneous architecture consisting of a GPU and a robotic-specific hardware module, along with a speculative dataflow that decouples VLA and sVLA through parallel execution. Comprehensive evaluations on OpenVLA and RDT across LIBERO and ManiSkill benchmarks show that SpecVLA reduces end-to-end latency significantly while preserving task success rate. By enabling long-action-length speculative prediction with timely verification, SpecVLA achieves real-time robotic manipulation with both high efficiency and reliability.
Chunyu Qi, Zhuoran Song, Jian Weng +6
School of Computer Science, Shanghai Jiao Tong University Shanghai, China · Computer Science, King Abdullah University of Science and Technology Thuwal, Mecca, Saudi Arabia
Vision-Language-Action (VLA) models exhibit strong generalization for robotic manipulation, yet their high inference latency limits real time deployment. We identify two primary sources of temporal redundancy in existing VLA pipelines: repeated visual encoding of highly similar consecutive frames and multi step iterative sampling in diffusion based policies. To address this, we propose a system level acceleration strategy that reduces computation in both perception and action generation. On the perception side, we incrementally update only tokens corresponding to dynamic scene regions instead of re-encoding entire frames. On the policy side, we compress diffusion sampling into a compact 2-step schedule through efficiency oriented training while preserving action precision. Experiments on Libero, RobotWin, and Real Robot Platforms demonstrate over 2 times speedup while maintaining high performance, achieving up to 98% success rate on general manipulation benchmarks. Our codes will be released on Github.
Yuzhou Wu, Yuxin Zheng, Muchun Niu +6
1Tianji KernalMind co ltd · 2Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · 3Shanghai Jiao Tong University, Shanghai, China +2