TempoFit: Plug-and-Play Layer-Wise Temporal KV Memory for Long-Horizon Vision-Language-Action Manipulation

Jun Sun; Boyu Yang; Jiahao Zhang; Ning Ma; Chencheng Wu; Siqing Zhang; Yiou Huang; Qiufeng Wang

cs.ROMar 8, 2026

TempoFit: Plug-and-Play Layer-Wise Temporal KV Memory for Long-Horizon Vision-Language-Action Manipulation

Authors: Jun Sun, Boyu Yang, Jiahao Zhang, Ning Ma, Chencheng Wu, Siqing Zhang, Yiou Huang, Qiufeng Wang, +2 more

Organizations: Xi’an Jiaotong-Liverpool University

Abstract

Pretrained Vision-Language-Action (VLA) policies have achieved strong single-step manipulation, but their inference remains largely memoryless, which is brittle in non-Markovian long-horizon settings with occlusion, state aliasing, and subtle post-action changes. Prior approaches inject history either by stacking frames, which scales visual tokens and latency while adding near-duplicate pixels, or by learning additional temporal interfaces that require (re-)training and may break the original single-frame inference graph. We present TempoFit, a training-free temporal retrofit that upgrades frozen VLAs through state-level memory. Our key insight is that prefix attention K/V already form a model-native, content-addressable runtime state; reusing them across timesteps introduces history without new tokens or trainable modules. TempoFit stores layer-wise FIFO prefix K/V at selected intermediate layers, performs parameter-free K-to-K retrieval with Frame-Gap Temporal Bias (FGTB), a fixed recency bias inspired by positional biases in NLP, to keep decisions present-dominant, and injects the retrieved context via pre-attention residual loading with norm-preserving rescaling to avoid distribution shift under frozen weights. On LIBERO-LONG, TempoFit improves strong pretrained backbones by up to +4.0% average success rate while maintaining near-real-time latency, and it transfers consistently to CALVIN and real-robot long-horizon tasks.

Figures & tables

Fig. 1: TempoFit overview. At each timestep, TempoFit caches prefix $K/V$ at selected intermediate layers, retrieves relevant history via K-to-K matching with FGTB , and injects the retrieved context through pre-attention residual loading (optionally with norm-preserving rescaling), enabling training-free temporal retrofitting without expanding input context length.

Fig. 2: TempoFit Pipeline. (a) In Layer-Wise FIFO KV Cache (see Sec. III-C ), TempoFit caches prefix $K/V$ states at selected intermediate layers, preserving historical context without expanding the input token sequence. (b) In K-to-K Retrieval with FGTB (see Sec. III-D & III-E ), the module utilizes current keys to retrieve relevant historical features via address-space matching, applying a fixed Frame-Gap Temporal Bias ( FGTB ) to down-weight stale history and minimize interference. (c) Finally, via Norm-Preserving Residual Loading (see Sec. III-F ), the retrieved history is injected into the current state through a rescaled residual update, enabling the frozen backbone to generate temporally consistent actions without parameter updates.

Method	Memory?	Training-free?	Avg. SR	Put soup and box in basket	Put box and butter in basket	Turn on stove and put pot	Put bowl in drawer and close	Put mugs on left and right plates	Pick book and place it in back	Put mug on plate, pudding right	Put soup and sauce in basket	Put both pots on stove	Put mug in microwave and close
Seer (scratch) [ 18 ]	$\times$	-	78.7	80.0	90.0	91.7	81.7	85.0	65.0	86.7	88.3	51.7	66.7
Seer [ 18 ]	$\times$	-	87.7	91.7	90.0	98.3	100	91.7	93.3	85.0	88.3	61.7	71.7
UniVLA [ 19 ]	$\times$	-	90.0	100	92.0	94.0	98.0	86.0	100	80.0	100	70.0	82.0
OpenVLA-OFT [ 15 ]	$\times$	-	94.0	90.0	98.0	98.0	98.0	96.0	100	92.0	100	72.0	96.0
MemoryVLA [ 13 ]	$\checkmark$	$\times$	93.4	92.0	96.0	96.0	100	100	100	96.0	96.0	62.0	96.0
HiF-VLA [ 10 ]	$\checkmark$	$\times$	96.4	88.0	98.0	100	100	100	100	96.0	100	82.0	100

TABLE I: Ablation study of our temporal memory module on the LIBERO-Long benchmark with two backbone VLAs (

\pi_{0.5}

and QwenGr00t). We report the per-task and average success rate (%) across 10 tasks (50 trials each). “Memory” indicates whether temporal memory is enabled. “Memory Type” indicates whether it is not used (-), training-based (

\times

), or training-free (

\checkmark

). Bold indicates the best performance within each backbone group.

Method	1	2	3	4	5	Avg. Len. $\uparrow$
CALVIN D-D
$\pi_{0}$ [ 3 ]	84.8	70.4	55.9	46.6	37.7	2.95
QwenPI [ 20 ]	90.9	79.5	69.6	62.2	55.4	3.58
QwenGR00T [ 20 ]	92.5	83.9	74.4	67.9	59.8	3.78
TempoFit ${}_{\text{QwenGr00t}}$ (Ours)	92.0	83.8	75.7	70.3	62.3	3.84
CALVIN ABC-D

TABLE II: Performance comparison on the CALVIN D-D and CALVIN ABC-D benchmarks. We report the average number of successfully completed tasks across five consecutive instructions. Bold indicates the best performance within each benchmark.

Method	History Length	Latency (ms, $\downarrow$ )	Peak memory (MB, $\downarrow$ )
$\pi_{0.5}$ [ 21 ]	1	71.2 ( $1.00\times$ )	6,396 ( $1.00\times$ )
+ Multi-frames	4	94.8 ( $1.33\times$ )	22,640 ( $3.54\times$ )
+ TempoFit (Ours)	4	73.4 ( $\mathbf{1.02\times}$ )	6,498 ( $\mathbf{1.02\times}$ )
+ Multi-frames	8	176.3 ( $2.48\times$ )	45,980 ( $7.19\times$ )
+ TempoFit (Ours)	8	74.4 ( $\mathbf{1.04\times}$ )	6,600 ( $\mathbf{1.03\times}$ )
+ TempoFit (Ours)	16	81.4 ( $\mathbf{1.13\times}$ )	6,761 ( $\mathbf{1.06\times}$ )

TABLE III: Efficiency analysis. Average latency and peak memory usage measured on LIBERO-long datasets. Both metrics are computed at each timestep within an episode and then averaged. All measurements were on an NVIDIA RTX5090 GPU.

\downarrow

indicates lower values are better.

Configuration	Avg. SR (%)
Baseline (no memory)	92.6
Component contribution
+ KV Memory only	93.8
+ KV Memory + FGTB	96.6
Retrieval strategy
Q-to-K retrieval	93.3

TABLE IV: Detailed ablation study of TempoFit on LIBERO-Long (

\pi_{0.5}

backbone). Each row modifies one design choice from the full method (last row). We report average success rate (%) over 500 trials.

Fig. 3: Real-world evaluation on Realman RM-65B. Left. Hardware and multi-view sensing setup. Right. Quantitative success rates and/or qualitative rollouts on three long-horizon manipulation tasks.

Explore similar work

Temporal Forcing: 4D Representation Alignment for Vision-Language-Action Models
Aug 31, 2026Xingyu Ding, Yuzhong Zhao, Chunhai Zhao +3Representation Learning Robotic Manipulation
Nanjing University, Nanjing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation
Jul 8, 2026Hongyu Qu, Jianzhe Gao, Xiaobin Hu +6Memory-Augmented VLMs Robotic Manipulation
Nanjing University of Science and Technology · Zhejiang University · National University of Singapore
NativeMEM: Native Memory Compression for Long-Horizon Robotic Manipulation
Jul 7, 2026Ziye Wang, Modi Shi, Chaojun Ni +5Memory-Augmented VLMs Long-Horizon Robotic Manipulation
The University of Hong Kong · Beihang University · Peking University +2

Aug 31, 2026J/K move · Enter open · S save

Temporal Forcing: 4D Representation Alignment for Vision-Language-Action Models

Xingyu Ding, Yuzhong Zhao, Chunhai Zhao, Yinghuan Shi+2

Nanjing University, Nanjing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

Representation Learning Robotic Manipulation Vision-Language-Action Models Latent Space Alignment

Open paper AskPDF arXiv

Long-horizon robotic manipulation requires vision-language-action (VLA) models to track scene states and their evolution beyond the current observation. However, simply conditioning policies on observation history does not guarantee that the history is effectively utilized: action supervision constrains what the policy should do, but only indirectly constrains what its history representations should retain. To resolve this, we present Temporal Forcing, a 4D representation alignment framework that explicitly supervises latent temporal states and their transitions. Specifically, we first introduce a history pathway that compresses past observations into compact latent tokens. We then align these tokens and current-frame features with geometric targets from a pretrained 4D foundation model, providing direct supervision at both the state and transition levels. The 4D foundation model and alignment heads are used only for training-time supervision. Temporal Forcing improves average success from 96.6% to 98.8% on LIBERO, with the largest gain on LIBERO-Long (93.8% to 97.2%), and from 53.5% to 62.8% across twelve RoboTwin 2.0 tasks. Furthermore, Temporal Forcing increases full-task success from 20.0% to 43.3% on a physical multi-stage hidden-placement task. Controlled experiments show that 4D representation alignment is crucial for making observation history beneficial to the model. Code will be publicly available.

Abstract

Explore similar work

Temporal Forcing: 4D Representation Alignment for Vision-Language-Action Models

Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation

NativeMEM: Native Memory Compression for Long-Horizon Robotic Manipulation