When Plans Change, Remember: Replanning-Guided Visual Memory for Vision-Language-Action Models

Victoria Smelova; Daniil Kazachkov; Georgiy Kazantsev; Iuliia Tsvetkova; Egor Cherepanov; Nikita Kachaev; Artem Latyshev; Aleksandr I. Panov

cs.ROOct 15, 2026

When Plans Change, Remember: Replanning-Guided Visual Memory for Vision-Language-Action Models

Authors: Victoria Smelova, Daniil Kazachkov, Georgiy Kazantsev, Iuliia Tsvetkova, Egor Cherepanov, Nikita Kachaev, Artem Latyshev, Aleksandr I. Panov,

Organizations: MIRAI · Independent Researcher · Central University · AXXX ·

Abstract

Long-horizon, multi-stage Vision-Language-Action (VLA) policies often require visual information that is no longer present in the current observation. Retaining the complete observation history is computationally expensive and highly redundant, motivating selective visual memory. We introduce RePlanMem-VLA (Replanning-Guided Keyframe Memory for VLA), which turns a policy's own plan revisions into an online memory-write signal. Under receding-horizon execution, consecutive action chunks predict some of the same future actions. RePlanMem-VLA treats unusually large disagreement between these temporally aligned predictions as evidence of a task-relevant event. We first compare 35 disagreement measures across single-task autoregressive decoder-only and Flow-Matching Transformers trained from scratch, then validate the selected signals on pretrained OpenVLA-OFT and $π_{0.5}$ policies over MIKASA-Robo-VLA tasks. The resulting detectors achieve ROC-AUC of 0.981-0.997. We then use the selected signals in RePlanMem-VLA to trigger updates to a bounded visual memory, with a robot-masked write filter suppressing redundant frames. In the evaluated memory-retention experiments, RePlanMem-VLA preserves task-relevant observations until the later decision point in 100% of episodes. Real-robot replay data further show that event selection and redundancy suppression extend beyond simulation. These results establish plan revision as a policy-internal signal for controlling what a VLA remembers. Website: https://vika787.github.io/RePlanMem/.

Figures & tables

Figure 2: Real-robot delayed-cue setup. The robot observes the target object’s color. The cue then leaves the observation, and the robot must later select the matching object among distractors using stored visual information.

Figure 3: Surprise-gated memory updates on the RememberColor task. The surprise score $S_{q}$ and task-calibrated percentile $p_{q}$ identify candidate update times near transitions between the cue, empty, and action phases.

Figure 4: RePlanMem-VLA overview. At replanning step $q$ , the VLA first predicts a chunk $C_{q}^{-}$ from the current observation, instruction, and existing memory. The temporally aligned overlap with the preceding plan is converted into a task-calibrated surprise percentile $p_{q}$ . If $p_{q}>\tau$ , a selected observation is written to FIFO memory and the policy performs a second forward pass with the updated memory, yielding $C_{q}^{\mathrm{final}}=C_{q}^{+}$ , otherwise $C_{q}^{\mathrm{final}}=C_{q}^{-}$ .

Figure 5: Examples of robot masking in simulation and real-robot observations. Each example shows the original observation, the estimated robot mask, and the masked image used for redundancy comparison. The policy receives the original RGB frame rather than the masked image.

Overlap L2 Mean L2 distance between aligned action predictions.

BiD lower bound Weighted sum of L2 distances.

Causal Z-score First-action revision relative to its causal baseline.

CUSUM Accumulated one-sided temporal change.

EMA deviation Deviation of the first-action change from its causal exponential moving average.

Translation cosine distance Mean cosine distance.

Translation disagreement Difference between translation predictions in controller-physical space.

Table 1: Detector features used in RQ2.

Policy	Regime	Classifier	PR-AUC	ROC-AUC
OpenVLA-OFT	RH1	XGBoost	0.932	0.982
OpenVLA-OFT	RH1	CatBoost	0.929	0.981
OpenVLA-OFT	RH1	LSTM	0.962	0.989
OpenVLA-OFT	EXEC4	XGBoost	0.996	0.997
OpenVLA-OFT	EXEC4	CatBoost	0.995	0.996
OpenVLA-OFT	EXEC4	LSTM	0.984	0.990

Table 2: Keyframe detection under RH1 and EXEC4.

Figure 6: Real-robot replay of the surprise-based event detector and redundancy-aware memory-selection pipeline. White highlights indicate memory updates, while red crosses mark detector activations for which the redundancy filter prevents overwriting existing memory frames.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Configuration
Shared settings
Visual encoder	NatureCNN, 512-D
Proprioception encoder	MLP $7\rightarrow 128\rightarrow 128$
Context Transformer	4 layers, 8 attention heads
Hidden / FFN dimension	256 / 1024
Dropout	0.1

Appendix

Table 3: Architecture and training settings for the single-task policies in RQ1.

	Autoregressive		Flow Matching
Task	Full-context	Current-only	Full-context	Current-only
RememberColor3	First NMSE 0.8480	Energy distance 0.8813	Energy distance 0.9427	Huber 0.9173
ShellGameColor LampTouch	Causal Z-score 0.7980	Gripper disagreement 0.8335	Gripper disagreement 0.9049	Gripper disagreement 0.8819
ShellGameShuffle ColorLampTouch	EMA deviation 0.5770	EMA deviation 0.6170	EMA deviation 0.6014	EMA deviation 0.6174
ShellGameShuffle Touch	Causal Z-score 0.7063	Relative TV gap 0.5678	Causal Z-score 0.8051	EMA deviation 0.7057
FindImposterShape3	EMA deviation 0.7332	Energy distance 0.8496	EMA deviation 0.7916	Wasserstein-1 0.8558

Appendix

Table 4: Best-performing disagreement metrics across the eight RQ1 tasks.

Task	Reference events
RememberColor3	Cue disappearance and onset of manipulation.
ShellGameColor LampTouch	Transition from cue presentation to manipulation.
ShellGameShuffle ColorLampTouch	Transitions from cue presentation to shuffling and from shuffling to manipulation.
ShellGameShuffle Touch	Transitions from cue presentation to shuffling and from shuffling to manipulation.
FindImposter Shape3	Transitions from cue presentation to the empty phase and from the empty phase to manipulation.
InterceptSlow	Transition from approach to deflection, onset of goal-directed deflection, and the success-achieving action.

Appendix

Table 5: Task-specific reference events used in RQ1.

Figure 7: Correlation structure of the disagreement measures. Pairwise within-episode Spearman correlations are aggregated across eight tasks with equal task weights. The top row shows autoregressive policies and the bottom row shows Flow-Matching policies. Columns correspond to full-context and current-only conditioning.

Panel of Figure 8 — Figure 8: Within-episode keyframe detection on RememberColor5 . Each panel shows example observations, selected disagreement features standardized within the episode, and keyframe probabilities predicted by XGBoost, CatBoost, and LSTM. Horizontal dotted lines indicate detector decision thresholds.

#	Metric	Definition
1	overlap_l2_mean	$\displaystyle\frac{1}{L}\sum_{i}\\|E_{i}\\|_{2}$
2	bid_lb	$\displaystyle\sum_{i}0.5^{i}\\|E_{i}\\|_{2}$
3	causal_z_exec_l2	$\displaystyle\frac{s_{q}-\mu_{q-1}^{s}}{\sigma_{q-1}^{s}+\epsilon_{t}}$
4	cusum_exec_l2	$\displaystyle\begin{aligned} c_{q}^{\mathrm{CUSUM}}=\max\bigl(&0,\,c_{q-1}^{\mathrm{CUSUM}}+s_{q}-\mu_{0}^{s}-\kappa\sigma_{0}^{s}\bigr)\end{aligned}$
5	ema_dev_exec_l2	$\displaystyle\begin{aligned} &s_{q}-\bar{s}_{q-1},\\ &\bar{s}_{q}=\alpha s_{q}+(1-\alpha)\bar{s}_{q-1}\end{aligned}$
6	a7_cos_trans	$\displaystyle\frac{1}{\|\mathcal{I}\|}\sum_{i\in\mathcal{I}}c(u_{i},v_{i})$

Appendix

Table 6: Definitions of the 35 disagreement signals. Colored names mark the nine action-derived features used in RQ2 and RePlanMem-VLA.

Memory-Augmented VLMs

Memory-augmented vision-language models use persistent or retrieved context to improve reasoning across long inputs and interactions.

See all 151 papers

ReMem-VLA: Empowering Vision-Language-Action Model with Memory via Dual-Level Recurrent Queries
Mar 13, 2026Hang Li, Fengyi Shen, Dong Chen +6Memory-Augmented VLMs Vision-Language-Action Models
Technical University of Munich, Munich, Germany · Huawei Heisenberg Research Center, Munich, Germany · University of Freiburg, Freiburg, Germany +2
ActMem-VLA: Action-History Memory with Dual-Expert Denoising for Long-Horizon Vision-Language-Action Policies
Sep 29, 2026Yaxin Zhao, Dianye Huang, Chenwei Wang +2Memory-Augmented VLMs Long-Horizon Robotic Manipulation
Harbin Institute of Technology, Harbin, China. · Medical Intelligence and Robotic Cognition (MIRoC) Lab, Department of Mechanical Engineering, The University of Hong Kong (HKU), Hong Kong SAR, China. · Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China
Explicit Language Memory for Long-Horizon Planning in Vision-Language-Action Models
Aug 5, 2026Houze Xu, Jizhong Li, Ziyi YeMemory-Augmented VLMs Vision-Language-Action Models
Fudan University

Mar 13, 2026J/K move · Enter open · S save

ReMem-VLA: Empowering Vision-Language-Action Model with Memory via Dual-Level Recurrent Queries

Hang Li, Fengyi Shen, Dong Chen, Liudi Yang+5

Technical University of Munich, Munich, Germany · Huawei Heisenberg Research Center, Munich, Germany · University of Freiburg, Freiburg, Germany+2

Memory-Augmented VLMs Vision-Language-Action Models VLMs for Robotics

Open paper AskPDF arXiv

Vision-language-action (VLA) models for closed-loop robot control are typically cast under the Markov assumption, making them prone to errors on tasks requiring historical context. To incorporate memory, existing VLAs either retrieve from a memory bank, which can be misled by distractors, or extend the frame window, whose fixed horizon still limits long-term retention. In this paper, we introduce ReMem-VLA, a Recurrent Memory VLA model equipped with two sets of learnable queries: frame-level recurrent memory queries for propagating information across consecutive frames to support short-term memory, and chunk-level recurrent memory queries for carrying context across temporal chunks for long-term memory. These queries are trained end-to-end to aggregate and maintain relevant context over time, implicitly guiding the model's decisions without additional training or inference cost. Furthermore, to enhance visual memory, we introduce Past Observation Prediction as an auxiliary training objective. Through extensive memory-centric simulation and real-world robot experiments, we demonstrate that ReMem-VLA exhibits strong memory capabilities across multiple dimensions, including spatial, sequential, episodic, temporal, and visual memory. ReMem-VLA significantly outperforms memory-free VLA baselines

π

0.5 and OpenVLA-OFT and surpasses MemoryVLA on memory-dependent tasks by a large margin.

Abstract

Memory-Augmented VLMs

Explore similar work

ReMem-VLA: Empowering Vision-Language-Action Model with Memory via Dual-Level Recurrent Queries

ActMem-VLA: Action-History Memory with Dual-Expert Denoising for Long-Horizon Vision-Language-Action Policies

Explicit Language Memory for Long-Horizon Planning in Vision-Language-Action Models