OxyGen: Unified KV Cache Management for VLA Inference under Multi-Task Parallelism

Xiangyu Li; Huaizhi Tang; Xin Ding; Weijun Wang; Ting Cao; Yunxin Liu

cs.ROMar 15, 2026

OxyGen: Unified KV Cache Management for VLA Inference under Multi-Task Parallelism

Authors: Xiangyu Li, Huaizhi Tang, Xin Ding, Weijun Wang, Ting Cao, Yunxin Liu

Organizations: Institute for AI Industry Research (AIR), Tsinghua University · Department of Electronic Engineering, Tsinghua University · University of Science and Technology of China

Abstract

Embodied AI agents increasingly require parallel execution of multiple tasks, such as manipulation, conversation, and memory construction, from shared observations under distinct time constraints. Recent Mixture-of-Transformers (MoT) Vision-Language-Action Models (VLAs) support such heterogeneous outputs, yet existing inference systems incur redundant computation and resource contention when serving concurrent tasks on-device. We identify isolated KV cache management as the root cause. To address this, we propose unified KV cache management, an inference design that treats the KV cache as a first-class shared resource across tasks and control cycles. This abstraction enables two key optimizations: cross-task KV sharing eliminates redundant prefill of shared observations, while cross-frame continuous batching batches unfinished language requests from different frames and advances them within a per-frame decoding budget. We implement this design in OxyGen and evaluate $π_{0.5}$ on NVIDIA GeForce RTX 4090 and Jetson AGX Thor. OxyGen achieves up to 3.7 $\times$ speedup over isolated execution, delivering over 200 tokens/s language throughput and 70 Hz action frequency simultaneously on RTX 4090. We also validate the design on Xiaomi-Robotics-0 and StarVLA across different backbones and expert configurations. We further demonstrate the inference gains on a humanoid robot with on-board Jetson AGX Thor.

Figures & tables

Figure 1: Left: An example of deploying a Mixture-of-Transformers (MoT) Vision-Language-Action (VLA) model for parallel multi-task inference: from per-frame observations, the VLA generates robot actions within each frame while continuously generating language-based memories across frames [ 52 ] . Right: Comparison between paradigms of MoT VLA inference: existing systems manage the KV cache in isolation, slowing down inference through redundant computation and resource contention; our method adopts unified KV cache management , achieving up to 3.7 $\times$ speedup via cross-task KV sharing and cross-frame continuous batching.

Figure 2: KV-centric dataflow at frame $t$ with a unified KV cache manager. KV[t] tracks the cache originating from frame $t$ : its prefill portion is defined in Eq. 1 , and language decoding adds token KVs. $\Delta$ Language[t] represents incremental language tokens in $\mathbf{y}_{t}$ .

Figure 3: Timeline comparison of OxyGen and isolated execution with

N=12

tokens per request. After warmup, OxyGen advances

B=3

active requests by

k=4

tokens each per frame. Both systems produce 12 language tokens per frame, but OxyGen reduces frame latency from

T+\Delta T

T

, increasing both action frequency and aggregate language throughput by a factor of

1+\Delta T/T

. Each language request in OxyGen spans three frames, completing in

3T

Figure 4: Action-frequency vs. language-throughput tradeoff on the LIBERO configuration. We sweep decoding steps per frame

k\in\{1,5,10\}

and maximum total decoding steps

N\in\{5,10,15,20,30\}

; each OxyGen line fixes

k

and varies

N

(markers are individual settings). Each baseline curve must trade one axis for the other as

N

grows; OxyGen improves both metrics by up to

3.7\times

over sequential isolated execution at matched output lengths.

Figure 5: Speedup ratio for action frequency of OxyGen vs. baseline across LIBERO, ALOHA, and DROID configurations (subplots), action denoising steps $S$ (y-axis), and decoding steps per frame $k$ (x-axis), with total decoding steps fixed at $N=30$ . The speedup is primarily affected by $k$ , while the configuration choice and $S$ have smaller impact.

Figure 6: Ablation of action frequency on LIBERO across language decoding steps (5 steps per frame), measured on GeForce RTX 4090 (left) and Jetson AGX Thor (right). Cross-task KV sharing (“Ours w/o Batching”) provides the initial speedup for short decoding steps, while cross-frame continuous batching (“Ours”) maintains high action frequency as decoding steps increase, on both platforms. The dash-dotted line marks the single-frame decoding oracle upper bound.

Configuration	$E$	Baseline (ms)	$k=1$	$k=5$
Xiaomi-Robotics-0	2	562.8	5.83 $\boldsymbol{\times}$	3.23 $\times$
Qwen3VL-PI_v3 (StarVLA)	2	478.0	6.64 $\boldsymbol{\times}$	3.60 $\times$
+ GR00T	3	515.8	6.05 $\boldsymbol{\times}$	3.53 $\times$
+ GR00T + OFT	4	541.2	6.30 $\boldsymbol{\times}$	3.69 $\times$

Table 1: Transfer across backbones and experts on RTX 4090, with

N=30

E

counts all language and action experts. Speedup is relative to isolated execution within each configuration.

Figure 7: Action frequency and average language batch size under different request arrival patterns. Dashed lines show the isolated-execution baseline with one new request per frame and 30 language tokens per request.

Figure 8: Humanoid robot with on-board Jetson AGX Thor (left) and measured stage latencies (right). The action path includes one prefill and action denoising in both methods. The language path includes a second prefill and autoregressive decoding in Baseline; OxyGen reuses the action-path prefill for batched decoding. The language-path reduction therefore reflects both KV sharing and batching.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: Action-frequency vs. language-throughput tradeoff on the ALOHA and DROID configurations, sweeping total decoding steps $N$ and decoding steps per frame $k$ . Trends mirror the LIBERO results in Fig. 4 : OxyGen extends the Pareto frontier beyond what either baseline can reach.

Figure 10: Speedup ratio for action frequency of OxyGen vs. baseline across LIBERO, ALOHA, and DROID configurations (subplots), action denoising steps $S$ (y-axis), and total decoding steps $N$ (x-axis), with decoding steps per frame fixed at $k=5$ . The speedup increases with larger $N$ , while the configuration choice and $S$ have smaller impact.

Panel of Figure 11 — Figure 11: Action-frequency and language-throughput curves. Panel (a) varies total decoding steps $N$ at fixed $k=5$ , and panel (b) varies decoding steps per frame $k$ at fixed $N=30$ . OxyGen reaches over 200 tokens/s and 70 Hz simultaneously.

Suite	openpi	OxyGen mean	OxyGen 95% CI
Spatial	98.80	98.42	[98.12, 98.72]
Object	98.20	98.52	[97.95, 99.09]
Goal	98.00	97.24	[96.66, 97.82]
LIBERO-10	92.40	92.86	[92.40, 93.32]
Overall	96.85	96.76	[96.55, 96.97]

Appendix

Table 2: LIBERO success rates (%). OxyGen results are means and 95% confidence intervals across ten seeds; openpi values are published point estimates.

Setting	Peak Mem. (GiB)	Avg. Power (W)	Energy /Req. (J)	$\Delta$ Energy /Req. (%)
Baseline	6.43	293.5	117.4	0
Parallel	12.49	324.4	120.9	$+$ 3.0
Ours w/o Batching	6.43	287.9	97.7	$-$ 16.7
Ours (batch: 2)	7.35	173.3	39.1	$-$ 66.7
Ours (batch: 4)	7.41	154.4	25.8	$-$ 78.0

Appendix

Table 3: Comparison of memory and energy cost. OxyGen adds modest memory overhead while achieving substantial energy savings.

Model	$E$	$N$	Baseline (ms)	Ours w/o Batching	Ours ( $k=1$ )	Ours ( $k=5$ )	Ours ( $k=10$ )
Xiaomi-Robotics-0	2	10	249.9	1.16 $\times$	2.91 $\boldsymbol{\times}$	1.49 $\times$	0.96 $\times$
		20	406.3	1.09 $\times$	4.47 $\boldsymbol{\times}$	2.39 $\times$	1.47 $\times$
		30	562.8	1.07 $\times$	5.83 $\boldsymbol{\times}$	3.23 $\times$	2.03 $\times$
StarVLA Qwen3VL-PI_v3	2	10	211.8	1.13 $\times$	3.20 $\boldsymbol{\times}$	1.71 $\times$	1.13 $\times$
		20	344.9	1.08 $\times$	5.00 $\boldsymbol{\times}$	2.73 $\times$	1.73 $\times$
		30	478.0	1.05 $\times$	6.64 $\boldsymbol{\times}$	3.60 $\times$	2.35 $\times$

Appendix

Table 4: Cross-model and multi-expert results on RTX 4090. Baseline reports frame latency in ms; all other method columns report speedup over Baseline.

E

counts language and action experts. For StarVLA,

E=2

is Qwen3VL-PI_v3, with GR00T added at

E=3

and OFT at

E=4

Backbone	$N$	Baseline (ms)	Ours w/o Batching	Ours ( $k=1$ )	Ours ( $k=5$ )	Ours ( $k=10$ )
0.8B	10	196.4	1.15 $\times$	2.91 $\boldsymbol{\times}$	1.65 $\times$	1.12 $\times$
	20	313.6	1.10 $\times$	4.62 $\boldsymbol{\times}$	2.62 $\times$	1.70 $\times$
	30	429.0	1.06 $\times$	6.30 $\boldsymbol{\times}$	3.58 $\times$	2.32 $\times$
2B	10	211.3	1.16 $\times$	2.99 $\boldsymbol{\times}$	1.71 $\times$	1.14 $\times$
	20	331.2	1.09 $\times$	4.65 $\boldsymbol{\times}$	2.65 $\times$	1.73 $\times$
	30	453.3	1.07 $\times$	6.31 $\boldsymbol{\times}$	3.62 $\times$	2.34 $\times$

Appendix

Table 5: Qwen3.5 model-size comparison on RTX 4090. Baseline reports frame latency in ms; all other columns report speedup over Baseline.

$N$	Prefill (ms)	Denoising (ms)	Decoding (ms)	E2E reduction (%)
1	44.04	20.50	14.51	36.4
5	44.04	20.46	66.61	25.9
10	43.99	20.39	132.03	19.2
20	44.01	20.41	263.15	12.5
30	44.22	20.49	393.85	9.4

Appendix

Table 6: Inference latency breakdown and prefix-reuse benefits for

\pi_{0.5}

-LIBERO (

S=10

). The breakdown reports one prefill, action denoising, and

N

language decoding steps in Ours w/o Batching. The final column gives its end-to-end latency reduction relative to Baseline.

	JAX		PyTorch
Measurement	Batch 6 ( $k=5$ )	Batch 30 ( $k=1$ )	Batch 6 ( $k=5$ )	Batch 30 ( $k=1$ )
KV stacking (ms)	0.77	2.82	0.34	1.28
KV splitting (ms)	4.24	18.24	1.73	8.53
Manager total (ms)	5.01	21.08	2.08	9.82
Frame latency (ms)	147.59	120.62	168.32	148.07
Manager share (%)	3.40	17.48	1.24	6.63

Appendix

Table 7: KV-management overhead for

\pi_{0.5}

-LIBERO on RTX 4090 (

S=10

N=30

). Manager total includes stacking, splitting, and bookkeeping. Manager share is the percentage of frame latency spent on these operations.

Figure 12: Memory and latency under the fluctuating

\pi_{0.5}

-LIBERO workload on RTX 4090 (

k=4

). Solid curves show the mean frame latency for each complete 119-frame workload cycle. Dashed curves and shaded bands show mean and min–max active memory in 50-frame windows.

Figure 13: Textual-memory generation alongside LIBERO control. OxyGen (left) makes faster task progress than isolated execution (right) while generating textual memory. The replay combines measured inference time with real-time simulator execution.

Efficient VLA Model Inference

Efficient vision-language-action (VLA) model inference reduces robot-control latency, memory use, and energy while preserving task performance.

See all 190 papers

Jul 27, 2026

A Motion-Aware Vector Quantization Framework with Centroid Reuse for Efficient VLA Inference

Vision-Language-Action (VLA) models have demonstrated strong potential for embodied AI, yet their high inference latency on GPUs limits real-time deployment. Existing accelerators, such as Dadu-Corki, improve efficiency but treat VLA models as full-precision workloads, leaving substantial redundancy in both memory and computation underexploited. In this paper, we propose VQVLA, an algorithm-hardware co-design framework that accelerates VLA inference by exploiting weight similarity and execution dynamics. We first introduce MotionVQ, a motion-aware vector quantization scheme that dynamically adjusts quantization precision based on the robot's execution state, reducing memory access while preserving task success rate. We then propose a merged-centroid vectorized GEMM paradigm that operates on the codebook-index representation, eliminating redundant multiplications through spatial aggregation and temporal reuse of centroids. To realize these optimizations, we design an accelerator that efficiently supports dynamic precision selection and centroid-reuse computation. Experimental results show that VQVLA achieves 6.5x, 2.8x, 1.9x, 3.3x, and 4.3x speedup over the A100 GPU, Dadu-Corki, LUT-DLA, CodeGEMM, and ShiftAddLLM, respectively, with negligible accuracy degradation.

Zhuoran Song, Haozhe Jiang, Chunyu Qi +4

School of Computer Science, Shanghai Jiao Tong University, 2 Institute of Automation, Chinese Academy of Sciences China

AI Accelerator Inference Efficient VLA Model Inference Vector Quantization Vision-Language-Action Models

Ask PDF

Jul 7, 2026

Training-Free Acceleration for Vision-Language-Action Models with Action Caching and Refinement

Vision-Language-Action (VLA) models have emerged as a promising approach for generalizable robotic manipulations. In particular, flow matching-based VLA models have shown remarkable success due to their capability to generate precise and smooth action sequences and capture multimodal distributions. However, the iterative denoising process in the action head acts as a major computational bottleneck, posing a critical challenge for real-time deployment. To address this challenge, we propose ActionCache, a plug-and-play external cache that opportunistically reuses past intermediate actions to warm-start generations from the vicinity of target actions, thereby drastically reducing the inference latency. Specifically, ActionCache stores the intermediate actions with compact multimodal keys, which enables retrieval from similar past contexts across different episodes or even different tasks. Experimental results in simulation and real-world environments demonstrate that ActionCache maintains high task success rates in a low-latency regime, achieving inference acceleration of up to

11.75\times

and

34.43\times

for representative flow-based VLA models,

π_{0.5}

and GR00T-N1.6, respectively.

Ryuji Oi, Hikari Otsuka, Kosuke Matsushima +4

Institute of Science Tokyo

Efficient VLA Model Inference Vision-Language-Action Models VLMs for Robotics

Ask PDF

Jul 7, 2025

VOTE: Vision-Language-Action Optimization with Trajectory Ensemble Voting

Recent large-scale Vision Language Action (VLA) models have shown superior performance in robotic manipulation tasks guided by natural language. However, current VLA models suffer from two drawbacks: (i) generation of massive tokens leading to high inference latency and increased training cost, and (ii) insufficient utilization of generated actions resulting in potential performance loss. To address these issues, we develop a training framework to finetune VLA models for generating significantly fewer action tokens with high parallelism, effectively reducing inference latency and training cost. Furthermore, we introduce an inference optimization technique with a novel voting-based ensemble strategy to combine current and previous action predictions, improving the utilization of generated actions and overall performance. Our results demonstrate that we achieve superior performance compared with state-of-the-art VLA models, achieving significantly higher success rates and 39

\times

faster inference than OpenVLA with 46 Hz throughput on edge platforms, demonstrating practical deployability. The code is available at https://github.com/LukeLIN-web/VOTE.

Juyi Lin, Amir Taherin, Arash Akbari +11

Northeastern University, Boston, USA · EmbodyX,San Mateo,USA

Robotic Control Efficient VLA Model Inference Efficient VLA Models Vision-Language-Action Models VLMs for Robotics VLA Model Training

Ask PDF

Abstract

Efficient VLA Model Inference

Explore similar work

A Motion-Aware Vector Quantization Framework with Centroid Reuse for Efficient VLA Inference

Training-Free Acceleration for Vision-Language-Action Models with Action Caching and Refinement

VOTE: Vision-Language-Action Optimization with Trajectory Ensemble Voting