Memory Anchors for Continual Robot Learning

Maximilian Du; Zhanyi Sun; Chen Xu; Paarth Shah; Masha Itkina; Shuran Song

cs.ROAug 27, 2026

Memory Anchors for Continual Robot Learning

Authors: Maximilian Du, Zhanyi Sun, Chen Xu, Paarth Shah, Masha Itkina, Shuran Song

Organizations: Stanford University · Toyota Research Institute

Abstract

Robot policies deployed in the wild should have the capability to continually learn new tasks without forgetting existing behaviors. A common approach to combat such catastrophic forgetting is to train on new task data with a replay buffer of previously learned task data. Although this buffer is commonly sampled randomly from all prior experiences, we show that a small set of these experiences contributes greatly in anchoring past performance. We call these experiences Memory Anchors. We identify Memory Anchors in regions where representations of new-task observations collapse onto those of old-task observations even though the tasks require conflicting actions, like when a familiar object must be manipulated in a new way. Rehearsing old data in this region plays a key role in preventing destructive overwriting of past task knowledge, serving as this critical Memory Anchor role. Excluding only 10% Memory Anchors before sampling the buffer leads to more than a 4.5x increase in catastrophic forgetting on the LIBERO benchmark suites. Conversely, enriching the replay buffer with Memory Anchors can decrease high-conflict task forgetting by 63% and enables successful continual learning of two task sequences on a real robot. Videos and additional visualizations can be found at https://robot-adaptation.github.io/MemoryAnchors

Figures & tables

Figure 1: Memory Anchors . New tasks may be visually similar to old tasks while requiring different, sometimes even opposite actions, causing catastrophic forgetting (Right). We claim that a small proportion of old data plays a large role in anchoring past task performance. While these Memory Anchors may appear naturally in randomly selected memories, their concentration is especially important with conflicting tasks. We propose a method for finding Memory Anchors and supplementing the ER buffer with them to reduce catastrophic forgetting for these worst-case scenarios (Left).

Figure 2: Sensitivity of Experience Replay (ER) to Data Selection. The standard ER practice ( FixedER ) samples at the trajectory level and does not resample $\mathcal{D}^{ER}$ after a new task. If we sample uniformly at the transition level and resample $\mathcal{D}^{ER}$ for every new task ( RandER ), we get more consistent performance across tasks (A). However, remaining variance exists between samples of $\mathcal{D}^{ER}$ (B), indicating sensitivity to the exact data present in $\mathcal{D}^{ER}$ .

Figure 3: Task Conflicts Persist Across Permutations. With an ER regularization ( RandER ), task performance stays steady for some new tasks and drops significantly for others. Shown above, Cream Cheese to Bowl drops in performance when the policy learns an additional task involving picking the Bowl instead.

Figure 4: New Data Overlaps Past Tasks. When new and old tasks share visual or semantic similarities, a policy can collapse a new task ( red ) into an existing task’s representation space ( orange ), causing conflict.

Figure 5: Finding Memory Anchors . From the overlapping latent state spaces of old and new task policy (1), we isolate new task points yielding the highest action disagreement with the old data (2). To do this, we predict new task actions using the current policy (2 Upper) and compare to the action labels to measure disagreement (2 Lower). Using these points, we extract old task data that exhibits the most similarity (3). These Memory Anchor candidates have state representation similarities and action disagreement with the new task data (§ 4 ).

Figure 6: Impact of Memory Anchors on Forgetting . Reducing access to a small proportion of the best Memory Anchors (§ 5.1 ) during the ER sampling process leads to large increases in catastrophic forgetting (Left). Symmetrically, increasing the concentration of Memory Anchors in an ER buffer reduces forgetting on high-conflict task pairs (Right).

Table 1: Impact of Memory Anchors on VLA. The availability of Memory Anchors influences the continual learning performance of the

\pi_{0.5}

VLA.

Figure 7: OpenJar Real Robot Task Suite . We propose a set of three tasks with visually similar jars that require opposing strategies to open (Left), leading to many instances of catastrophic forgetting and plasticity loss when trained improperly (Middle). Using Anchor ER, the robot achieves less forgetting and 1.7x higher overall success rate compared to a randomly-sampled ER buffer, RandER (Right).

Figure 8: SweaterFold Real Robot Task Suite . We further test continual learning strategies on a realistic set of language-conditioned garment-folding tasks. Consistent with the results in OpenJar , we find that Anchor ER achieves a better balance of plasticity and task retention.

	Ours	MIR	GSS	Ours-ActionDis	RandER
Avg. NBT $\downarrow$	.11 $\pm$ .01	.14 $\pm$ .01	.17 $\pm$ .03	.13 $\pm$ .01	.18 $\pm$ .02
Anchor Conc. $\uparrow$	.27	.13	.08	.25	.10

Table 2: Baselines. Anchor ER selection outperforms relevant baselines and ablations. A correlation also exists between the number of Memory Anchors in a method’s buffer and its task retention ability.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Robot Setup. We use the UMI setup to collect data and then deploy a trained policy on an ARX robot arm for OpenJar tasks.

Figure 10: Examples of Action Disagreement and Selected Memory Anchors in Real-World Task . High action disagreement in the OpenJar task happens near the jar lid contact (Left column), and the corresponding Memory Anchors also focus on the jar lid contact of the old tasks (Right column). The images are sampled from the top $100$ action disagreement and memory anchors, respectively.

Figure 11: Real World Results by Correct Behavior. Some failures for both Anchor ER and RandER are due to lack of precision errors, not errors of behavior.

Figure 12: Forgetting with Smaller Buffer Sizes. As the available buffer size decreases, data retrieval through Anchor ER becomes more important.

Access to Memory Anchors	FWT( $\uparrow$ )	NBT( $\downarrow$ )	AUC( $\uparrow$ )	FWT( $\uparrow$ )	NBT( $\downarrow$ )	AUC( $\uparrow$ )
	LIBERO-Long			LIBERO-Spatial
100% ( RandER )	0.87 $\pm$ 0.02	0.20 $\pm$ 0.02	0.73 $\pm$ 0.03	0.90 $\pm$ 0.02	0.05 $\pm$ 0.01	0.87 $\pm$ 0.01
99% ( RandER -1%)	0.86 $\pm$ 0.02	0.17 $\pm$ 0.02	0.74 $\pm$ 0.02	0.90 $\pm$ 0.02	0.07 $\pm$ 0.02	0.85 $\pm$ 0.02
95% ( RandER -5%)	0.87 $\pm$ 0.02	0.28 $\pm$ 0.03	0.67 $\pm$ 0.03	0.88 $\pm$ 0.02	0.14 $\pm$ 0.03	0.78 $\pm$ 0.03
90% ( RandER -10%)	0.86 $\pm$ 0.02	0.34 $\pm$ 0.04	0.63 $\pm$ 0.04	0.90 $\pm$ 0.01	0.33 $\pm$ 0.04	0.66 $\pm$ 0.04
	LIBERO-Object			LIBERO-Goal

Appendix

Table 3: Additional Metrics for Reducing Memory Anchor Access (§ 5.1 ) . Reducing access to Memory Anchors reduces the ability to preserve past tasks but not the ability to learn new tasks. All numbers are from an ER buffer size of

1000

sampled without replacement from the reduced past task dataset.

Method	FWT( $\uparrow$ )	NBT( $\downarrow$ )	AUC( $\uparrow$ )	FWT( $\uparrow$ )	NBT( $\downarrow$ )	AUC( $\uparrow$ )
	LIBERO-Long			LIBERO-Spatial
RandER	0.85 $\pm$ 0.02	0.25 $\pm$ 0.02	0.67 $\pm$ 0.03	0.88 $\pm$ 0.01	0.18 $\pm$ 0.02	0.75 $\pm$ 0.01
Anchor ER (10%)	0.86 $\pm$ 0.02	0.26 $\pm$ 0.03	0.67 $\pm$ 0.03	0.90 $\pm$ 0.01	0.14 $\pm$ 0.01	0.79 $\pm$ 0.02
Anchor ER (20%)	0.87 $\pm$ 0.02	0.27 $\pm$ 0.03	0.68 $\pm$ 0.03	0.88 $\pm$ 0.02	0.16 $\pm$ 0.02	0.76 $\pm$ 0.02
	LIBERO-Object			LIBERO-Goal
RandER	0.96 $\pm$ 0.01	0.05 $\pm$ 0.01	0.92 $\pm$ 0.01	0.92 $\pm$ 0.01	0.18 $\pm$ 0.03	0.79 $\pm$ 0.03

Appendix

Table 4: Additional Metrics for Adding Memory Anchors (§ 5.2 ) . This table shows the average CL metrics with a 1% ER buffer (notice different buffer selection size than Table 3 ). Best results bolded , best results 1 SEM over baseline in green. The homogeneous (Goal) task suite is affected more by additional Memory Anchors.

Figure 13: Histogram of Task-Task Interactions ( $\Delta_{i,j}$ ) Most task-task interactions yield little forgetting, and a small proportion conflict heavily, leading to high forgetting. Many negative $\Delta_{i,j}$ instances exist even with an overall positive NBT because tasks will rebound after interacting with a conflicting task.

Task Interaction	Mean $\Delta$
put the cream cheese in the bowl $\rightarrow$ put the bowl on the plate	0.240
put the bowl on top of the cabinet $\rightarrow$ put the wine bottle on top of the cabinet	0.240
put the wine bottle on the rack $\rightarrow$ turn on the stove	0.180
open the top drawer and put the bowl inside $\rightarrow$ put the wine bottle on top of the cabinet	0.180
turn on the stove $\rightarrow$ open the middle drawer of the cabinet	0.120

Appendix

Table 5: Top five task-task conflicts in LIBERO-Goal. This table shows the task-task interactions that have the highest forgetting (§ B.4 ).

Figure 14: Other metrics for comparing FixedER and RandER . In addition to NBT shown in Fig. 2 , we show the AUC and FWT here.

Figure 15: $\pi_{0.5}$ on 1000 memories (20%) buffer. We verify that with a large ER buffer, $\pi_{0.5}$ achieves high task retention (NBT = 0.02 on LIBERO-Goal).

Continual Robot Learning

Continual robot learning develops robots that acquire new skills and adapt to changing conditions while retaining previously learned behavior.

See all 89 papers

Jun 7, 2026

Not Just After One: Sleep-Inspired Replay Prevents Catastrophic Forgetting After Sequential Tasks

One of the critical limitations of artificial neural networks is their lack of ability to continually learn: training on new tasks often leads to interference and forgetting of the previous ones. While several algorithms have been proposed to protect old memories from interference, they are typically applied during or immediately after each new episode of training. In contrast, humans and animals can learn continuously, acquiring multiple new memories during active learning before consolidating all of them into long-term storage. Here we show that multiple new tasks can be trained sequentially before an unsupervised sleep-like replay phase is applied to partially restore performance across all previously learned tasks. Our study further suggests that task-specific information remains resilient to new training but decays gradually as network is trained on new tasks. These findings point to novel principles for developing a broad range of continual learning AI solutions.

Anthony Bazhenov, Jean Erik Delanois, Giri P. Krishnan

Khoury College of Computer Sciences, Northeastern University, Boston, MA · Department of Medicine, University of California, San Diego, La Jolla, CA, USA · ARTISAN, Georgia Institute of Technology, Atlanta, GA USA

Replay-Based Continual Learning Memory Consolidation Catastrophic Forgetting

Ask PDF

Oct 2, 2026

RIFAR: Reliability-Informed and Forgetting-Aware Replay for Continual Robot Learning with World-Action Models

Genuine embodied agency requires robots to turn continuous real-world experience into lasting, transferable skills. This demands continual learning that integrates new capabilities while preserving prior knowledge in evolving environments. Experience replay mitigates catastrophic forgetting, but storing complete demonstrations becomes costly as tasks accumulate. World-action models provide an alternative through generative replay. However, visually coherent rollouts may contain actions that cannot realize the predicted transitions, while new-task adaptation can disrupt learned behavior. To address these problems, we propose RIFAR (Reliability-Informed and Forgetting-Aware Replay), a reliability-informed and drift-aware generative replay framework. It reconstructs trajectories from compact historical demonstration prefixes and uses a trainable inverse-dynamics model to assess action-visual consistency. It first adapts to new tasks using current-task demonstrations and high-reliability replay, then evaluates how this adaptation changes action predictions on historical inputs and prioritizes reliable trajectories with greater action drift for further training. Across three LIBERO suites and real-world experiments, RIFAR outperforms the previous state of the art in WAM-based generative replay. On LIBERO-Goal, it achieves an AUC of 90.97 while retaining only 320 historical time steps per task, about 4.9% of those retained by 50-demonstration replay.

Tu Lin, Zheng Lu, Zirui Song +7

Tsinghua University · MBZUAI · Zhejiang University

Continual Robot Learning Replay-Based Continual Learning Generative Replay World Action Models

Ask PDF

Mar 12, 2026

ARROW: Augmented Replay for RObust World models

Continual reinforcement learning challenges agents to acquire new skills while retaining previously learned ones with the goal of improving performance in both past and future tasks. Most existing approaches rely on model-free methods with replay buffers to mitigate catastrophic forgetting; however, these solutions often face significant scalability challenges due to large memory demands. Drawing inspiration from neuroscience, where the brain replays experiences to a predictive World Model rather than directly to the policy, we present ARROW (Augmented Replay for RObust World models), a model-based continual RL algorithm that extends DreamerV3 with a memory-efficient, distribution-matching replay buffer. Unlike standard fixed-size FIFO buffers, ARROW maintains two complementary buffers: a short-term buffer for recent experiences and a long-term buffer that preserves task diversity through intelligent sampling. We evaluate ARROW on two challenging continual RL settings: Tasks without shared structure (Atari), and tasks with shared structure, where knowledge transfer is possible (Procgen CoinRun variants). Compared to model-free and model-based baselines with replay buffers of the same-size, ARROW demonstrates substantially less forgetting on tasks without shared structure, while maintaining comparable forward transfer. Our findings highlight the potential of model-based RL and bio-inspired approaches for continual reinforcement learning, warranting further research.

Abdulaziz Alyahya, Abdallah Al Siyabi, Markus R. Ernst +3

Department of Information Systems, College of Computer and Information Sciences, Imam Mohammad Ibn Saud Islamic University (IMSIU) · Department of Data Science & AI, Faculty of Information Technology, Monash University · School of Computer Science and Engineering, Faculty of Engineering, University of New South Wales, Sydney

Continual RL Replay-Based Continual Learning Model-Based RL World Models

Ask PDF

Abstract

Continual Robot Learning

Explore similar work

Not Just After One: Sleep-Inspired Replay Prevents Catastrophic Forgetting After Sequential Tasks

RIFAR: Reliability-Informed and Forgetting-Aware Replay for Continual Robot Learning with World-Action Models

ARROW: Augmented Replay for RObust World models