REKI: Registering Knowledge in VLAs

Juliette Marrie; Paul Pacaud; Adrien Ramanana Rahary; Shizhe Chen; Moritz Böhle

cs.ROOct 15, 2026

REKI: Registering Knowledge in VLAs

Authors: Juliette Marrie, Paul Pacaud, Adrien Ramanana Rahary, Shizhe Chen, Moritz Böhle

Organizations: Kyutai · Inria · École nationale des ponts et chaussées

Abstract

We introduce REKI, a 2.4B vision-language-action model (VLA) pretrained exclusively on public data that sets a new Pareto frontier between performance and model size: on LIBERO-Plus, RoboTwin Clean2Rand and RoboCasa365, it is outperformed only by models about twice its size. REKI consists of a vision-language model (VLM) followed by a flow-matching action expert, and is pretrained on a curated mixture of public robot, egocentric human and vision-language data. Additionally, we endow it with a memory that gives the policy access to its recent past at negligible cost: a few register tokens in the action expert summarise the output of the VLM at each frame and persist in the expert's cache. This memory improves success in simulation and on real robots, and makes oscillations between poses about four times rarer. REKI uses a simple model design and training paradigm, and is optimised to be efficient at training and inference. We release the full code across data curation, training, inference and evaluation, deployment, and all pretrained and finetuned model weights: https://github.com/kyutai-labs/reki.

Figures & tables

Figure 1: Success rate against model size. REKI sets a new Pareto frontier between performance and model size: it scores highest on LIBERO-Plus, and is only outperformed by models about twice its size on RoboTwin Clean2Rand and RoboCasa365. We report full results in Fig. 4 .

Figure 2: How VLAs carry the past. Each panel shows one training window of three frames, with a loss under each frame that receives one. (a) Past frames enter the VLM with the current one, in full ( Yuan et al., 2026 ) , compressed ( Torne et al., 2026 ; Physical Intelligence et al., 2026 ) or from a cache ( Jang et al., 2025 ) . The VLM takes the whole window as input, and only the last frame is supervised. (b) A state is written at each step and read at the next, in the backbone ( Cherepanov et al., 2026 ; Xiao et al., 2026 ) or as fast weights in the head ( Jiang et al., 2026 ) . Each step needs the output of the previous one, so the window is unrolled step by step. (c) Each frame is encoded on its own, and an added module mixes the memory tokens of past frames into the prediction for the current one. HAMLET ( Koo et al., 2026 ) supervises the last frame only, and MemoryVLA ( Shi et al., 2026 ) every frame, with the bank entries stored without gradient. (d) In REKI, the memory tokens are registers in the action expert. Their keys and values are kept across frames and read by the self-attention already present. The window is one batched pass, and every frame receives a loss.

Panel of Figure 3 — Figure 3: REKI. ( 3(a) ) Camera views and instruction are encoded as context $\bm{E}_{t}$ , which every expert block reads by cross-attention. The state embedding $\bm{g}_{t}$ and the flow time modulate the actions through adaptive normalisation. The noisy chunk enters the expert’s stream, and the flow loss compares the predicted velocity with its target. The register keys and values of past frames are kept in a cache and read by self-attention, which is the memory of Section 4.1 . ( 3(b) ) One of the ten blocks, holding $n_{r}\mkern 1.25mu{=}\mkern 1.25mu32$ registers and the action tokens of the current frame. Arrows run from a query to what it reads. Action tokens read the current and past registers and themselves, and registers never read them.

Figure 4: Benchmark results. Out of distribution, on LIBERO-Plus (a) and RoboTwin Clean2Rand (c), REKI leads the compared VLAs pretrained on public data; in distribution, on RoboTwin (b), RoboCasa365 (d) and RoboCasa-GR1 (e), it is competitive with larger models. Hatched bars mark models pretrained on in-house data and outlined bars world-action models. Each bar gives the overall score and the ticks the subset scores. Full results for (a) to (d) are in Appendix Tables 4 , 5 and 6 .

Figure 5: Memory window, oscillation and inference cost. REKI trained with

K\mkern 1.25mu{=}\mkern 1.25mu3

and evaluated with

K\mkern 1.25mu{=}\mkern 1.25mu3,2,1

(

3\to 3

3\to 2

3\to 1

), and a memoryless model (

1\to 1

). (a) On RoboCasa365, the black line gives the average. (b) RoboTwin Clean episodes in which an arm reverses its motion at three or more consecutive chunks (50 tasks, 20 matched seeds each). (c) Two-arm tasks move both arms in at least 80% of the successful episodes with memory, one-arm tasks a single arm. The three other tasks are left out. (d) On one H100. The numbers give the difference to

K\mkern 1.25mu{=}\mkern 1.25mu1

in ms.

Figure 6: Inference latency (H100) , from the raw observation to the actions of one chunk, with LIBERO inputs, without compilation (left) and compiled (right). Dashed bars are compiled code that we built. ABot-M0.5 runs with its RoboCasa365 weights, the only ones released, and with its own LIBERO setting, which takes two

128{\times}128

images.

Figure 7: Oscillation without memory. Two RoboTwin rollouts of REKI trained without memory, stacking three blocks. Frames are one chunk apart (1 s), and A and B mark the two poses the arm alternates between. In the top row, the two arms swap at every chunk, one reaching while the other retracts. In the bottom row, a single arm moves back and forth. The blocks do not move in either row.

Figure 8: Real-robot experiments. Left: the UR5 arm with a WSG50 gripper, observed by a fixed Orbbec Femto Mega RGB camera, with the task objects and distractors. Right: task score of REKI with and without memory and of

\pi_{0.5}

( Physical Intelligence et al., 2025 ) on two shorter- and three longer-horizon tasks, over 10 episodes each. Solid bars give full successes, hatched bars partial credit.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: (a) The VLM encoder. It feeds the expert of Fig. 3 the encoded context. A vision–language model encodes the views and the prompt; its final-layer hidden states are the encoded context and its LM head predicts the subtask. (b) The registers and noisy action tokens of a frame cross-attend the context of that frame and self-attend the registers of the $K$ frames up to it, so the past reaches the current chunk only through the registers, and only these are kept for later frames.

Figure 10: Register cache at inference (

K\mkern 1.25mu{=}\mkern 1.25mu3

). (a) In the first Euler step of a chunk, the VLM encodes the current frame

t

. The registers of frame

t

read its context and the cached registers of frames

t{-}1

and

t{-}2

. The action tokens read the context and the registers of the three frames; past frames are not encoded again. (b) In the

N{-}1

other Euler steps, only the action tokens are updated, and they read the context and the register keys and values from the cache. The registers do not read the action tokens and skip adaptive normalisation, so they do not depend on the flow time and are computed once per frame. When the next frame enters, the registers of frame

t{-}2

are dropped.

Robot	Action type	Sources
Single-arm
WidowX	$\Delta$ EE + G	( Walke et al., 2023 )
Google Robot	$\Delta$ EE + G	( Brohan et al., 2023 ; Jang et al., 2022 )
Franka Panda	$\Delta$ EE + G; J + G	( Open X-Embodiment Collaboration, 2024 ; Bharadhwaj et al., 2024 ; Wu et al., 2025a ; Khazatsky et al., 2024 ; Tian et al., 2026 )
UR5e	J + G	( Wu et al., 2025a )
Dual-arm

Appendix

Table 1: Robot embodiments in the training mixture. “EE” = end-effector pose; “J” = joint angles; “

\Delta

” = delta (relative) commands, otherwise absolute; “G” = parallel gripper; “DH” = dexterous hand; “W” = waist/torso; “B” = mobile base. A robot may appear under several conventions across its source datasets.

Encoder
Setting	Value
Vision–language backbone	Qwen3.5-2B (18 gated-DeltaNet + 6 full-attention layers)
Image resolution	native, up to $448{\times}448$
Expert conditioning	cross-attention to the final-layer backbone states in every expert block
Trainable components	backbone at a reduced LR ( Table 3 ), last $4$ vision blocks, expert
Vision–language co-training	$10\%$ : FineVision (subset), LLaVA-OneVision-1.5 (subset), RoboPoint, RefSpatial, PixMo

Appendix

Table 2: Training configuration.

Method	#Params	Atomic	Composite		Avg
Method	#Params	Atomic	Seen	Unseen	Avg
Xiaomi-Robotics-1 ∗ ( Xiaomi Robotics Team, 2026 )	5.1B	80.2	57.1	32.1	57.4
ABot-M0.5 ( Chen et al., 2026 )	5B	75.9	38.3	2.7	40.4
RLDX-1 ∗ ( Kim et al., 2026 )	6.9B	67.6	27.9	8.5	36.0
Qwen-RobotManip ∗ ( Yuan et al., 2026 )	4.7B	68.6	20.1	14.9	35.9
GR00T-N1.5 ∗ ( Bjorck et al., 2025 )	2.4B	50.7	14.8	2.7	23.9

Appendix

Table 6: RoboCasa365 results behind Fig. 4 , on the 50 target tasks of the multi-task setting. Hatched in Fig. 4 , and marked

\ast

here: pretrained on in-house data. Italic : model not released at the time of writing.

Figure 11: Pretraining. RoboTwin Clean2Rand success rate of REKI against the number of pretraining steps. The bars give the mean over the six conditions, and the dots the individual conditions.

RoboCasa365	Atomic	Composite		Avg
RoboCasa365	Atomic	Seen	Unseen	Avg
Without mirroring	59.6	24.1	13.5	33.5
With mirroring	65.8	32.0	14.1	38.4

Appendix

Table 8: Finetuning ablations , memory window

K\mkern 1.25mu{=}\mkern 1.25mu3

. Each row changes one setting of the reported configuration (bold).

	REKI	InternVLA-A1.5
Phase	compiled	optimized	compiled by us
Vision encoder	9.8	11.5	2.1
VLM prefill	9.8	28.7	4.7
Euler steps (10)	8.2	46.0	34.9
Other	2.0	2.9	2.7
Model time	19.7	88.2	44.1

Appendix

Table 10: Phase times of one chunk with LIBERO inputs, in ms of wall time, from a separate run. REKI runs the vision encoder and the VLM prefill as one CUDA graph, timed together.

	REKI	$\pi_{0.5}$
Optimiser steps	3,000	3,000
Global batch size	360	360
Learning rate (action expert)	$5\times 10^{-5}$	$5\times 10^{-5}$
Learning rate (backbone)	$5\times 10^{-6}$	$5\times 10^{-6}$
LR schedule	warmup, then constant	warmup, then constant
Trainable vision blocks	last 4	last 4

Appendix

Table 11: Finetuning hyperparameters for the real-robot experiments.

Figure 12: Qualitative examples. Rollouts of REKI on our UR5 robotic arm across the five manipulation tasks.

Vision-Language-Action Models

Vision-language-action models turn visual observations and natural-language instructions into actions for robots and other embodied agents.

See all 1,168 papers

TurboVLA: Real-Time Vision-Language-Action Model at 32 Hz on an RTX 4090 with <1 GB VRAM
Jul 29, 2026Hengyi Xie, Chenfei Yao, Xianjin Wu +4Efficient VLA Model Inference Efficient VLA Models
Huazhong University of Science and Technology
VLA- $Δ$ : Learning Physical State Transition for Vision-Language-Action Models
Oct 15, 2026Qi Lu, Jiarun Li, Ce Zhang +5VLA Model Training Vision-Language-Action Models
Zhejiang University (zju.edu.cn) · Central South University (csu.edu.cn)
CoTinyVLA: Chain-of-Thought Distillation for a Sub-Billion-Parameter Vision-Language-Action Model
Jul 28, 2026Minhyeok Lee, Chiyoung Kim, Chanhoe Gu +5Reasoning for VLA Models Efficient VLA Model Inference
Chung-Ang University, Republic of Korea

Jul 29, 2026J/K move · Enter open · S save

TurboVLA: Real-Time Vision-Language-Action Model at 32 Hz on an RTX 4090 with <1 GB VRAM

Hengyi Xie, Chenfei Yao, Xianjin Wu, Yingying Zhu+3

Huazhong University of Science and Technology

Efficient VLA Model Inference Efficient VLA Models Robotic Manipulation Vision-Language-Action Models

Open paper AskPDF arXiv

Vision-language-action (VLA) models commonly adopt an LLM-centric

V \to L \to A

pathway, processing visual observations and language instructions through a large language model before predicting robot actions. Although effective, this design incurs substantial computation and memory overhead. In this work, we introduce TurboVLA, a compact VLA architecture built on a direct

V + L \to A

mapping. Instead of using a large language model as the central interface between perception and action, TurboVLA independently encodes visual observations and language instructions, directly exchanges information between them through lightweight bidirectional vision-language interaction, and predicts continuous action chunks with a compact decoder. This simple design directly constructs task-conditioned representations while avoiding the overhead of an LLM-centered execution pathway. On LIBERO, TurboVLA achieves 97.6% average success with only 0.2B parameters, 31.2 ms inference latency, and 0.9 GiB inference VRAM on a consumer-grade RTX 4090. Notably, a 0.4B TurboVLA achieves 88.06% success on RoboTwin 2.0, even matching or outperforming substantially larger VLA policies. These results demonstrate that the simple

V + L \to A

design of TurboVLA can achieve high performance without requiring an LLM-centric execution pathway, offering a new perspective on how vision, language, and action can be connected for efficient robotic manipulation. Code is available at https://github.com/H-EmbodVis/TurboVLA.

Abstract

Vision-Language-Action Models

Explore similar work

TurboVLA: Real-Time Vision-Language-Action Model at 32 Hz on an RTX 4090 with <1 GB VRAM

VLA-ΔΔΔ: Learning Physical State Transition for Vision-Language-Action Models

CoTinyVLA: Chain-of-Thought Distillation for a Sub-Billion-Parameter Vision-Language-Action Model

VLA- $Δ$ : Learning Physical State Transition for Vision-Language-Action Models