Pretrained Vision-Language-Action (VLA) policies have achieved strong single-step manipulation, but their inference remains largely memoryless, which is brittle in non-Markovian long-horizon settings with occlusion, state aliasing, and subtle post-action changes. Prior approaches inject history either by stacking frames, which scales visual tokens and latency while adding near-duplicate pixels, or by learning additional temporal interfaces that require (re-)training and may break the original single-frame inference graph. We present TempoFit, a training-free temporal retrofit that upgrades frozen VLAs through state-level memory. Our key insight is that prefix attention K/V already form a model-native, content-addressable runtime state; reusing them across timesteps introduces history without new tokens or trainable modules. TempoFit stores layer-wise FIFO prefix K/V at selected intermediate layers, performs parameter-free K-to-K retrieval with Frame-Gap Temporal Bias (FGTB), a fixed recency bias inspired by positional biases in NLP, to keep decisions present-dominant, and injects the retrieved context via pre-attention residual loading with norm-preserving rescaling to avoid distribution shift under frozen weights. On LIBERO-LONG, TempoFit improves strong pretrained backbones by up to +4.0% average success rate while maintaining near-real-time latency, and it transfers consistently to CALVIN and real-robot long-horizon tasks.
Figures & tables
Fig. 1: TempoFit overview. At each timestep, TempoFit caches prefix K/V at selected intermediate layers, retrieves relevant history via K-to-K matching with FGTB , and injects the retrieved context through pre-attention residual loading (optionally with norm-preserving rescaling), enabling training-free temporal retrofitting without expanding input context length.
Fig. 2: TempoFit Pipeline. (a) In Layer-Wise FIFO KV Cache (see Sec. III-C ), TempoFit caches prefix K/V states at selected intermediate layers, preserving historical context without expanding the input token sequence. (b) In K-to-K Retrieval with FGTB (see Sec. III-D & III-E ), the module utilizes current keys to retrieve relevant historical features via address-space matching, applying a fixed Frame-Gap Temporal Bias ( FGTB ) to down-weight stale history and minimize interference. (c) Finally, via Norm-Preserving Residual Loading (see Sec. III-F ), the retrieved history is injected into the current state through a rescaled residual update, enabling the frozen backbone to generate temporally consistent actions without parameter updates.
Method
Memory?
Training-free?
Avg. SR
Put soup and box in basket
Put box and butter in basket
Turn on stove and put pot
Put bowl in drawer and close
Put mugs on left and right plates
Pick book and place it in back
Put mug on plate, pudding right
Put soup and sauce in basket
Put both pots on stove
Put mug in microwave and close
Seer (scratch) [ 18 ]
×
-
78.7
80.0
90.0
91.7
81.7
85.0
65.0
86.7
88.3
51.7
66.7
Seer [ 18 ]
×
-
87.7
91.7
90.0
98.3
100
91.7
93.3
85.0
88.3
61.7
71.7
UniVLA [ 19 ]
×
-
90.0
100
92.0
94.0
98.0
86.0
100
80.0
100
70.0
82.0
OpenVLA-OFT [ 15 ]
×
-
94.0
90.0
98.0
98.0
98.0
96.0
100
92.0
100
72.0
96.0
MemoryVLA [ 13 ]
✓
×
93.4
92.0
96.0
96.0
100
100
100
96.0
96.0
62.0
96.0
HiF-VLA [ 10 ]
✓
×
96.4
88.0
98.0
100
100
100
100
96.0
100
82.0
100
TABLE I: Ablation study of our temporal memory module on the LIBERO-Long benchmark with two backbone VLAs ( π0.5 and QwenGr00t). We report the per-task and average success rate (%) across 10 tasks (50 trials each). “Memory” indicates whether temporal memory is enabled. “Memory Type” indicates whether it is not used (-), training-based ( × ), or training-free ( ✓ ). Bold indicates the best performance within each backbone group.
Method
1
2
3
4
5
Avg. Len. ↑
CALVIN D-D
π0 [ 3 ]
84.8
70.4
55.9
46.6
37.7
2.95
QwenPI [ 20 ]
90.9
79.5
69.6
62.2
55.4
3.58
QwenGR00T [ 20 ]
92.5
83.9
74.4
67.9
59.8
3.78
TempoFit QwenGr00t (Ours)
92.0
83.8
75.7
70.3
62.3
3.84
CALVIN ABC-D
TABLE II: Performance comparison on the CALVIN D-D and CALVIN ABC-D benchmarks. We report the average number of successfully completed tasks across five consecutive instructions. Bold indicates the best performance within each benchmark.
Method
History Length
Latency (ms, ↓ )
Peak memory (MB, ↓ )
π0.5 [ 21 ]
1
71.2 ( 1.00× )
6,396 ( 1.00× )
+ Multi-frames
4
94.8 ( 1.33× )
22,640 ( 3.54× )
+ TempoFit (Ours)
4
73.4 ( 1.02× )
6,498 ( 1.02× )
+ Multi-frames
8
176.3 ( 2.48× )
45,980 ( 7.19× )
+ TempoFit (Ours)
8
74.4 ( 1.04× )
6,600 ( 1.03× )
+ TempoFit (Ours)
16
81.4 ( 1.13× )
6,761 ( 1.06× )
TABLE III: Efficiency analysis. Average latency and peak memory usage measured on LIBERO-long datasets. Both metrics are computed at each timestep within an episode and then averaged. All measurements were on an NVIDIA RTX5090 GPU. ↓ indicates lower values are better.
Configuration
Avg. SR (%)
Baseline (no memory)
92.6
Component contribution
+ KV Memory only
93.8
+ KV Memory + FGTB
96.6
Retrieval strategy
Q-to-K retrieval
93.3
TABLE IV: Detailed ablation study of TempoFit on LIBERO-Long ( π0.5 backbone). Each row modifies one design choice from the full method (last row). We report average success rate (%) over 500 trials.
Fig. 3: Real-world evaluation on Realman RM-65B. Left. Hardware and multi-view sensing setup. Right. Quantitative success rates and/or qualitative rollouts on three long-horizon manipulation tasks.
Long-horizon robotic manipulation requires vision-language-action (VLA) models to track scene states and their evolution beyond the current observation. However, simply conditioning policies on observation history does not guarantee that the history is effectively utilized: action supervision constrains what the policy should do, but only indirectly constrains what its history representations should retain. To resolve this, we present Temporal Forcing, a 4D representation alignment framework that explicitly supervises latent temporal states and their transitions. Specifically, we first introduce a history pathway that compresses past observations into compact latent tokens. We then align these tokens and current-frame features with geometric targets from a pretrained 4D foundation model, providing direct supervision at both the state and transition levels. The 4D foundation model and alignment heads are used only for training-time supervision. Temporal Forcing improves average success from 96.6% to 98.8% on LIBERO, with the largest gain on LIBERO-Long (93.8% to 97.2%), and from 53.5% to 62.8% across twelve RoboTwin 2.0 tasks. Furthermore, Temporal Forcing increases full-task success from 20.0% to 43.3% on a physical multi-stage hidden-placement task. Controlled experiments show that 4D representation alignment is crucial for making observation history beneficial to the model. Code will be publicly available.
Xingyu Ding, Yuzhong Zhao, Chunhai Zhao +3
Nanjing University, Nanjing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Mainstream Vision-Language-Action (VLA) models predict actions primarily from the current observation under a Markovian assumption, thus struggling with long-horizon, temporally dependent tasks. Existing memory-augmented VLAs either expand the observation window or retrieve history from the memory bank as auxiliary policy-side context. However, they leave memory outside the native latent embedding space of VLA reasoning, preventing historical experience from being fluidly interleaved with multimodal reasoning and action formation. To this end, we introduce LaMem-VLA, a latent-memory-native framework that reconstructs historical experience into latent memory tokens and directly interweaves them with VLA reasoning. At its core, LaMem-VLA introduces four coordinated components: (i) a curator that organizes historical experience into two complementary short-term and long-term memory vaults; (ii) a seeker that queries both vaults using the multimodal cognition to retrieve context-relevant evidence; (iii) a condenser that reconstructs the retrieved evidence into compact short-term and long-term latent memory tokens; and (iv) a weaver that injects these memory tokens with the current observation and instruction into one continuous embedding sequence. By representing, retrieving, and consuming historical experience entirely in the same continuous latent space, LaMem-VLA enables memory to directly participate in VLA reasoning and guide action generation under a bounded context. Extensive experiments on SimplerEnv and LIBERO demonstrate the superiority of our LaMem-VLA.
Hongyu Qu, Jianzhe Gao, Xiaobin Hu +6
Nanjing University of Science and Technology · Zhejiang University · National University of Singapore
How can pretrained Vision-Language-Action (VLA) models retain long-horizon visual histories with high-frequency updates without sacrificing efficiency? Existing approaches rely on external memory management, which restrains either the memory horizon or the reactiveness of pretrained policies. To this end, we present NativeMEM, a VLA policy that features long-term and real-time updated memory. At its core is an efficient memory encoding scheme, Native Memory Compression, which repurposes the VLA's own vision encoder to compress each historical frame from each camera view into a single token. Appended to the input sequence, these memory tokens enable the pretrained VLA to attend over long-term history with negligible latency overhead, requiring neither an external planner nor a freshly initialized memory module. To align the memory tokens with the pretrained policy, we first develop a generic memory tokenizer under the supervision of a frozen VLA on memory-demanding data, and then unfreeze the VLA for task-specific fine-tuning. NativeMEM consistently outperforms prior methods, boosting success rates from 32.4% to 84.0% in simulation and up to 98.7% on real robots, while maintaining low inference latency and GPU memory usage. Notably, NativeMEM exhibits high data efficiency by achieving competitive results with prior arts using only 20% of the training data.
Ziye Wang, Modi Shi, Chaojun Ni +5
The University of Hong Kong · Beihang University · Peking University +2