Long-horizon, multi-stage Vision-Language-Action (VLA) policies often require visual information that is no longer present in the current observation. Retaining the complete observation history is computationally expensive and highly redundant, motivating selective visual memory. We introduce RePlanMem-VLA (Replanning-Guided Keyframe Memory for VLA), which turns a policy's own plan revisions into an online memory-write signal. Under receding-horizon execution, consecutive action chunks predict some of the same future actions. RePlanMem-VLA treats unusually large disagreement between these temporally aligned predictions as evidence of a task-relevant event. We first compare 35 disagreement measures across single-task autoregressive decoder-only and Flow-Matching Transformers trained from scratch, then validate the selected signals on pretrained OpenVLA-OFT and π0.5 policies over MIKASA-Robo-VLA tasks. The resulting detectors achieve ROC-AUC of 0.981-0.997. We then use the selected signals in RePlanMem-VLA to trigger updates to a bounded visual memory, with a robot-masked write filter suppressing redundant frames. In the evaluated memory-retention experiments, RePlanMem-VLA preserves task-relevant observations until the later decision point in 100% of episodes. Real-robot replay data further show that event selection and redundancy suppression extend beyond simulation. These results establish plan revision as a policy-internal signal for controlling what a VLA remembers. Website: https://vika787.github.io/RePlanMem/.
Figures & tables
Figure 2: Real-robot delayed-cue setup. The robot observes the target object’s color. The cue then leaves the observation, and the robot must later select the matching object among distractors using stored visual information.
Figure 3: Surprise-gated memory updates on the RememberColor task. The surprise score Sq and task-calibrated percentile pq identify candidate update times near transitions between the cue, empty, and action phases.
Figure 4: RePlanMem-VLA overview. At replanning step q , the VLA first predicts a chunk Cq− from the current observation, instruction, and existing memory. The temporally aligned overlap with the preceding plan is converted into a task-calibrated surprise percentile pq . If pq>τ , a selected observation is written to FIFO memory and the policy performs a second forward pass with the updated memory, yielding Cqfinal=Cq+ , otherwise Cqfinal=Cq− .
Figure 5: Examples of robot masking in simulation and real-robot observations. Each example shows the original observation, the estimated robot mask, and the masked image used for redundancy comparison. The policy receives the original RGB frame rather than the masked image.
Overlap L2 Mean L2 distance between aligned action predictions.
BiD lower bound Weighted sum of L2 distances.
Causal Z-score First-action revision relative to its causal baseline.
CUSUM Accumulated one-sided temporal change.
EMA deviation Deviation of the first-action change from its causal exponential moving average.
Translation cosine distance Mean cosine distance.
Translation disagreement Difference between translation predictions in controller-physical space.
Table 1: Detector features used in RQ2.
Policy
Regime
Classifier
PR-AUC
ROC-AUC
OpenVLA-OFT
RH1
XGBoost
0.932
0.982
OpenVLA-OFT
RH1
CatBoost
0.929
0.981
OpenVLA-OFT
RH1
LSTM
0.962
0.989
OpenVLA-OFT
EXEC4
XGBoost
0.996
0.997
OpenVLA-OFT
EXEC4
CatBoost
0.995
0.996
OpenVLA-OFT
EXEC4
LSTM
0.984
0.990
Table 2: Keyframe detection under RH1 and EXEC4.
Figure 6: Real-robot replay of the surprise-based event detector and redundancy-aware memory-selection pipeline. White highlights indicate memory updates, while red crosses mark detector activations for which the redundancy filter prevents overwriting existing memory frames.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Configuration
Shared settings
Visual encoder
NatureCNN, 512-D
Proprioception encoder
MLP 7→128→128
Context Transformer
4 layers, 8 attention heads
Hidden / FFN dimension
256 / 1024
Dropout
0.1
Appendix
Table 3: Architecture and training settings for the single-task policies in RQ1.
Autoregressive
Flow Matching
Task
Full-context
Current-only
Full-context
Current-only
RememberColor3
First NMSE 0.8480
Energy distance 0.8813
Energy distance 0.9427
Huber 0.9173
ShellGameColor LampTouch
Causal Z-score 0.7980
Gripper disagreement 0.8335
Gripper disagreement 0.9049
Gripper disagreement 0.8819
ShellGameShuffle ColorLampTouch
EMA deviation 0.5770
EMA deviation 0.6170
EMA deviation 0.6014
EMA deviation 0.6174
ShellGameShuffle Touch
Causal Z-score 0.7063
Relative TV gap 0.5678
Causal Z-score 0.8051
EMA deviation 0.7057
FindImposterShape3
EMA deviation 0.7332
Energy distance 0.8496
EMA deviation 0.7916
Wasserstein-1 0.8558
Appendix
Table 4: Best-performing disagreement metrics across the eight RQ1 tasks.
Task
Reference events
RememberColor3
Cue disappearance and onset of manipulation.
ShellGameColor LampTouch
Transition from cue presentation to manipulation.
ShellGameShuffle ColorLampTouch
Transitions from cue presentation to shuffling and from shuffling to manipulation.
ShellGameShuffle Touch
Transitions from cue presentation to shuffling and from shuffling to manipulation.
FindImposter Shape3
Transitions from cue presentation to the empty phase and from the empty phase to manipulation.
InterceptSlow
Transition from approach to deflection, onset of goal-directed deflection, and the success-achieving action.
Appendix
Table 5: Task-specific reference events used in RQ1.
Figure 7: Correlation structure of the disagreement measures. Pairwise within-episode Spearman correlations are aggregated across eight tasks with equal task weights. The top row shows autoregressive policies and the bottom row shows Flow-Matching policies. Columns correspond to full-context and current-only conditioning.
Figure 8: Within-episode keyframe detection on RememberColor5 . Each panel shows example observations, selected disagreement features standardized within the episode, and keyframe probabilities predicted by XGBoost, CatBoost, and LSTM. Horizontal dotted lines indicate detector decision thresholds.
#
Metric
Definition
1
overlap_l2_mean
L1i∑∥Ei∥2
2
bid_lb
i∑0.5i∥Ei∥2
3
causal_z_exec_l2
σq−1s+ϵtsq−μq−1s
4
cusum_exec_l2
cqCUSUM=max(0,cq−1CUSUM+sq−μ0s−κσ0s)
5
ema_dev_exec_l2
sq−sˉq−1,sˉq=αsq+(1−α)sˉq−1
6
a7_cos_trans
∣I∣1i∈I∑c(ui,vi)
Appendix
Table 6: Definitions of the 35 disagreement signals. Colored names mark the nine action-derived features used in RQ2 and RePlanMem-VLA.
Vision-language-action (VLA) models for closed-loop robot control are typically cast under the Markov assumption, making them prone to errors on tasks requiring historical context. To incorporate memory, existing VLAs either retrieve from a memory bank, which can be misled by distractors, or extend the frame window, whose fixed horizon still limits long-term retention. In this paper, we introduce ReMem-VLA, a Recurrent Memory VLA model equipped with two sets of learnable queries: frame-level recurrent memory queries for propagating information across consecutive frames to support short-term memory, and chunk-level recurrent memory queries for carrying context across temporal chunks for long-term memory. These queries are trained end-to-end to aggregate and maintain relevant context over time, implicitly guiding the model's decisions without additional training or inference cost. Furthermore, to enhance visual memory, we introduce Past Observation Prediction as an auxiliary training objective. Through extensive memory-centric simulation and real-world robot experiments, we demonstrate that ReMem-VLA exhibits strong memory capabilities across multiple dimensions, including spatial, sequential, episodic, temporal, and visual memory. ReMem-VLA significantly outperforms memory-free VLA baselines π0.5 and OpenVLA-OFT and surpasses MemoryVLA on memory-dependent tasks by a large margin.
Hang Li, Fengyi Shen, Dong Chen +6
Technical University of Munich, Munich, Germany · Huawei Heisenberg Research Center, Munich, Germany · University of Freiburg, Freiburg, Germany +2
Vision-language-action (VLA) models have driven rapid progress in robotic manipulation, demonstrating strong fine-grained control and promising performance on long-horizon tasks. However, many existing VLAs lack explicit access to interaction history, making them vulnerable to perceptual aliasing: similar current observations and robot states at different task stages may induce action ambiguity and lower success rate. Existing methods incorporate temporal or progress cues through feature conditioning, action-prior modification, or sampling guidance. However, methods that jointly fine-tune memory modules and the base VLA incur additional policy-training costs, motivating the separation of trainable history-conditioned steering from frozen base-policy refinement. We propose ActMem-VLA, a dual-expert handover architecture that augments a frozen, fine-tuned VLA with a memory plugin comprising a Mamba-based memory module and a lightweight PreAction Expert (PAE). Specifically, Mamba encodes executed-action history into memory that conditions PAE alongside current context. With these inputs, PAE steers task progression during early, high-noise denoising, then passes the partially denoised action to the frozen Action Expert (AE) to refine action details during the remaining low-noise steps. The fine-tuned base VLA remains frozen throughout training, while only the Mamba module and PAE are jointly optimized. On LIBERO-Mem, ActMem-VLA achieves 80.8% average success across all ten tasks, compared with 65.2% for π0.5 and 49.5% for MemoryVLA, while introducing only 3.45% additional parameters. Across four real-world tasks, it improves the average success rate over π0.5 by 28.8%.
Yaxin Zhao, Dianye Huang, Chenwei Wang +2
Harbin Institute of Technology, Harbin, China. · Medical Intelligence and Robotic Cognition (MIRoC) Lab, Department of Mechanical Engineering, The University of Hong Kong (HKU), Hong Kong SAR, China. · Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China
Vision-language-action (VLA) models provide a unified paradigm for connecting visual perception, language understanding, and robotic control. However, existing VLA models still face major challenges in long-horizon tasks: sparse expert demonstrations constrain cross-task compositional generalization; the non-Markovian nature of long-horizon tasks makes it difficult for policies conditioned only on current observations to maintain temporal consistency; limited closed-loop error correction allows execution errors to accumulate; and end-to-end action fine-tuning may weaken the high-level semantic representations of vision-language model (VLM) backbones. To address these issues, we propose a hierarchical long-horizon VLA architecture with an explicit language-memory module. The central idea is to convert discrete temporal observations into a coherent textual memory sequence with temporal logic. The system is decoupled into a high-level VLM and a low-level VLA: the high-level VLM performs semantic reasoning through a visual question answering training paradigm, while the low-level VLA executes precise continuous control conditioned on subtask instructions and visual observations. The high-level VLM recursively updates both language memory and subtask instructions using the previous memory as a contextual anchor, enabling persistent temporal tracking and dynamic correction during long-horizon execution. We evaluate the proposed method in multiple simulation environments and conduct sim-to-real experiments on a real robotic platform. The results demonstrate that explicit language memory improves the success rate and robustness of VLA models on complex long-horizon tasks while providing an interpretable semantic account of the decision process.