Fast-VLA: You Only Think Once for Robot Manipulation

Kehong Gong; Guanli Hou; Xiaoxu Zheng; Qi Wang; Mingxi Xu; Anbang Zhai; Xiaoyu He; Mingyuan Zhang

cs.ROOct 15, 2026

Fast-VLA: You Only Think Once for Robot Manipulation

Authors: Kehong Gong, Guanli Hou, Xiaoxu Zheng, Qi Wang, Mingxi Xu, Anbang Zhai, Xiaoyu He, Mingyuan Zhang,

Organizations: Huawei Technologies Co., Ltd · Huawei Central Research Institute

Abstract

Vision-language-action (VLA) policies commonly place a multi-billion-parameter model inside the feedback loop, repeatedly recomputing task context whenever they predict a new action chunk. Yet task understanding changes far more slowly than the robot state. We introduce Fast-VLA, which invokes a frozen 4B VLM once per episode to contextualize the instruction and initial views as fixed, multi-layer key-value task memory. During execution, a lightweight online actor fuses learned time- and arm-indexed queries with current visual and proprioceptive feedback to form action queries. These queries retrieve task context from the fixed memory and predict action chunks without re-running the foundation model. Fast-VLA achieves 92.46% and 92.40% success on RoboTwin 2.0 Clean and Randomized, and 95.95% and 79.11% on LIBERO and LIBERO-Plus evaluation. Against three continually conditioned baselines on the same hardware, it reduces RoboTwin amortized policy inference time by 10.9-17.1x and requires 1.22 GiB of post-prefill online resident memory. Task-memory swaps further show that the actor relies on episode-matched task memory. These results show that task context can be established once while closed-loop control remains responsive throughout execution. Project page: https://garfield-kh.github.io/FastVLA

Figures & tables

Figure 1: Fast-VLA thinks once per episode. (a) Success versus amortized inference latency on RoboTwin 2.0; bubble area denotes online resident memory and colour denotes method family. (b) A frozen 4B vision–language model (VLM) encodes the instruction and initial multi-view frames into fixed, layer-wise KV task memory once at episode start. At every action chunk, the online actor uses live observations and robot state to query this memory and predict 50 absolute joint commands without re-running the VLM.

Figure 2: Memory-conditioned online control in Fast-VLA. A frozen VLM constructs layer-wise KV task memory from the instruction and initial views once per episode. In the online actor, query fusion combines learned queries with arm state and wrist-view CLS tokens to form action queries. The action decoder reads visual tokens from current and historical views and retrieves task context from the fixed memory. The action head predicts a 50-step bimanual action chunk.

Setting	RoboTwin 2.0	LIBERO
Learning rate	$3\times 10^{-4}$	$3\times 10^{-4}$
Batch size	256	32
Training steps	200,000	30,000
Training compute (accelerator-hours)	$\approx 150$	$\approx 5$

Table 1: Training settings. Batch size is the effective global batch size; training compute is reported in accelerator-hours.

Type	Method	Action PT	Pred./ exec.	ms/chunk $\downarrow$	ms/step $\downarrow$	GiB $\downarrow$	Clean $\uparrow$	Rand. $\uparrow$	Mean $\uparrow$
VLA	$\pi_{0}$ ( Black et al., 2025 )	$\checkmark$	50/50	72.38	1.580	6.26	65.92 ♭	58.40 ♭	62.16
	$\pi_{0.5}$ ( Physical Intelligence et al., 2025 )	$\checkmark$	50/50	80.51	1.758	6.47	82.74 ♭	76.76 ♭	79.75
	X-VLA ( Zheng et al., 2025 )	–	30/30	211.03	7.452	3.29	72.90 ♭	72.80 ♭	72.85
	LingBot-VLA ( Wu et al., 2026a )	$\checkmark$	50/25	740.35	30.919	7.99	86.50	85.34	85.92
	Qwen-VLA ( Wang et al., 2026 )	$\checkmark$	–	–	–	–	86.10	87.20	86.65
	LingBot-VLA 2.0 ( Wu et al., 2026b )	$\checkmark$	50/25	931.29	38.892	11.98	93.52	92.80	93.16

Table 2: RoboTwin 2.0 success rates (%) and inference costs. Action PT denotes pretraining of action-generating weights on external large-scale action or latent-action data before benchmark adaptation; VLM or visual-backbone pretraining alone does not count.

Type	Method	Action PT	Pred./ exec.	ms/chunk $\downarrow$	Amort. ms/step $\downarrow$	GiB $\downarrow$	LIBERO Std. $\uparrow$	LIBERO Plus $\uparrow$
VLA	$\pi_{0}$ ( Black et al., 2025 )	$\checkmark$	50/5	81.96	16.577	6.26	94.10	53.60
	$\pi_{0}$ -FAST ( Pertsch et al., 2025 )	$\checkmark$	10/5	353.06	71.885	5.57	85.50 ♭	61.60
	OpenVLA-OFT ( Kim et al., 2025 )	$\checkmark$	8/8	107.00	13.685	14.41	97.10 ♭	69.60
	$\pi_{0.5}$ ( Physical Intelligence et al., 2025 )	$\checkmark$	10/5	86.35	17.466	6.47	96.90	85.70
	UniVLA ( Bu et al., 2025 )	$\checkmark$	12/1	241.96	241.960	14.15	95.50 ♭	43.90
	X-VLA ( Zheng et al., 2025 )	–	30/30	199.40	7.220	3.29	98.10	72.56 ∥

Table 3: Standard LIBERO and LIBERO-Plus success rates (%) with inference cost.

Panel of Figure 3 — Figure 3: Successful rollouts with fixed task memory. Each row shows five frames sampled at equal intervals from one episode; corner labels indicate frame indices. Wrist views are inset below the third-person view (two for RoboTwin and one for LIBERO). Task memory is constructed once at episode start and reused throughout each rollout. In the LIBERO-Plus row, sensor noise corrupts the third-person view while the wrist view remains unperturbed; memory is built from the perturbed initial observation.

Figure 4: Attention follows the active sub-goal. A successful 300-step stack_blocks_two rollout, shown in successive 50-step action chunks. As control passes from the left arm to the right, head-camera attention shifts from the red block to the green block. Each wrist focuses on the block handled by its own arm, with the right wrist attending to the green block before the second sub-goal begins. Crosshairs mark attention peaks, and enrichment above $1$ indicates preferential attention to the corresponding block; the final open-loop chunk has no new attention map.

Table 8

Table 9

Memory producer	Attention	Memory $\downarrow$	Std. $\uparrow$	Plus $\uparrow$
PaliGemma v1 ( Beyer et al., 2024 )	18L, MQA	9.8 MB	95.35	73.15
$\pi_{0.5}$	18L, MQA	9.8 MB	94.90	71.48
Qwen2.5-VL-7B ( Bai et al., 2025 )	28L, GQA-4	31.8 MB	95.35	75.55
DM0.5 (default)	34L, GQA-4	79.7 MB	95.95	79.11

Table 9: Task-memory encoders on LIBERO (%). The actor architecture is unchanged across rows; memory denotes the bf16 KV cache stored per episode in the LIBERO configuration.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	RoboTwin 2.0	LIBERO
Visual encoder	DINOv2 ViT-B/14	DINOv2 ViT-B/14
Actor image resolution	$168\times 224$	$224\times 224$
VLM prefill images	$448\times 448$ , 3 cameras	$448\times 448$ , 2 cameras
VLM image tokens	768	512
Decoder width / layers / heads	256 / 4 / 8	256 / 4 / 8
Cross-memory GQA KV heads	8	8

Appendix

Table 10: Architecture and training configuration. Each camera has a separate, trainable DINOv2 encoder. Batch sizes are effective global batch sizes.

Item	Protocol
Execution precision	BF16 for the VLM; BF16 autocast with gradient computation disabled for the actor.
Batch size
Warmup and timing	3 warmup calls followed by 10 timed calls; report mean latency.
Device synchronization	Synchronized immediately before and after each timed call.
Preprocessing and transfer	Timing starts from raw uint8 HWC NumPy images and includes resizing, normalization, and host-to-device transfer.
Inference backend	PyTorch 2.7.1+cu128; no torch.compile or TensorRT.

Appendix

Table 11: Inference latency and memory measurement protocol.

Panel of Figure 5 — Figure 5: Successful rollouts of Fast-VLA. Each row samples five frames at equal intervals from one episode; the corner label is the frame index. Each frame shows the third-person view at full size with the actor’s wrist views inset at the bottom, two for RoboTwin and one for LIBERO. All actions in a row are produced from a single task memory built before the first frame and never refreshed.

Panel of Figure 6 — Figure 6: Additional successful LIBERO-Plus rollouts for “put the black bowl in the bottom drawer of the cabinet and close it.” Rows show camera, background, lighting, object-layout, and initial-state perturbations, respectively. Each row contains five equally spaced frames, with frame indices in the corners and the wrist view inset. Task memory is built once from the perturbed initial observations and remains fixed for the episode. The sensor-noise variant appears in Figure 3 .

Vision-Language-Action Models

Vision-language-action models turn visual observations and natural-language instructions into actions for robots and other embodied agents.

See all 1,168 papers

IMLE-VLA: Fast Single-Step Action Generation for Vision-Language-Action Policies
Sep 10, 2026Kian Hosseinkhani, Qinhe Peng, George Shramko +6Efficient VLA Model Inference Vision-Language-Action Models
Simon Fraser University · University of Pennsylvania · Alberta Machine Intelligence Institute +1
ICI-VLA: In-Context Imitation with Spatiotemporally Aligned Demonstrations for Vision-Language-Action Models
Sep 7, 2026Songhua Yang, Ziyu Liu, Xuetao Li +3Imitation Learning Robotic Manipulation
Wuhan University · Institute of Technological Sciences, Wuhan University
TurboVLA: Real-Time Vision-Language-Action Model at 32 Hz on an RTX 4090 with <1 GB VRAM
Jul 29, 2026Hengyi Xie, Chenfei Yao, Xianjin Wu +4Efficient VLA Model Inference Efficient VLA Models
Huazhong University of Science and Technology

Sep 10, 2026J/K move · Enter open · S save

IMLE-VLA: Fast Single-Step Action Generation for Vision-Language-Action Policies

Kian Hosseinkhani, Qinhe Peng, George Shramko, Mehran Aghabozorgi+5

Simon Fraser University · University of Pennsylvania · Alberta Machine Intelligence Institute+1

Efficient VLA Model Inference Vision-Language-Action Models Policy Learning VLA Model Training

Open paper AskPDF arXiv

Vision-language-action (VLA) policies leverage pretrained vision-language backbones to achieve strong cross-task generalization. A leading design couples this backbone with a dedicated continuous action head trained via diffusion or flow matching. However, such heads rely on iterative multi-step sampling, for example 10 Euler steps in

π_{0.5}

. This creates an inference bottleneck that produces stop-and-go movement in the robot and slower task completion. We introduce IMLE-VLA, which replaces the iterative action head with a single-step conditional generator trained via conditional Implicit Maximum Likelihood Estimation (cIMLE). The cIMLE objective promotes multimodal action coverage, avoiding the mode collapse of naive regression heads while eliminating multi-step sampling entirely. When IMLE-VLA is applied to

π_{0.5}

, it increases inference frequency 3.67x (55 Hz vs. 15 Hz), enabling up to 11x higher action throughput. On the 40-task LIBERO benchmark, IMLE-VLA achieves the highest average success rate (98.0%) among all baselines while leading in inference frequency. Under the test-time perturbations of LIBERO-plus, IMLE-VLA retains

π_{0.5}

's robustness while other baselines degrade sharply, confirming that the cIMLE head preserves generalization. Real-world experiments on a Franka Emika Panda across four tasks demonstrate smoother motion (2.2x to 3.0x lower jerk) and faster task completion, with IMLE-VLA outperforming

π_{0.5}

on every task and reducing average VLA inference time per episode by 3.9x to 6.6x. Videos and code are available at https://kianhk6.github.io/IMLE-VLA/

Abstract

Vision-Language-Action Models

Explore similar work

IMLE-VLA: Fast Single-Step Action Generation for Vision-Language-Action Policies

ICI-VLA: In-Context Imitation with Spatiotemporally Aligned Demonstrations for Vision-Language-Action Models

TurboVLA: Real-Time Vision-Language-Action Model at 32 Hz on an RTX 4090 with <1 GB VRAM