Authors: Abdulaziz Alyahya, Abdallah Al Siyabi, Markus R. Ernst, Luke Yang, Levin Kuhlmann, Gideon Kowadlo
Organizations: Department of Information Systems, College of Computer and Information Sciences, Imam Mohammad Ibn Saud Islamic University (IMSIU) · Department of Data Science & AI, Faculty of Information Technology, Monash University · School of Computer Science and Engineering, Faculty of Engineering, University of New South Wales, Sydney
Continual reinforcement learning challenges agents to acquire new skills while retaining previously learned ones with the goal of improving performance in both past and future tasks. Most existing approaches rely on model-free methods with replay buffers to mitigate catastrophic forgetting; however, these solutions often face significant scalability challenges due to large memory demands. Drawing inspiration from neuroscience, where the brain replays experiences to a predictive World Model rather than directly to the policy, we present ARROW (Augmented Replay for RObust World models), a model-based continual RL algorithm that extends DreamerV3 with a memory-efficient, distribution-matching replay buffer. Unlike standard fixed-size FIFO buffers, ARROW maintains two complementary buffers: a short-term buffer for recent experiences and a long-term buffer that preserves task diversity through intelligent sampling. We evaluate ARROW on two challenging continual RL settings: Tasks without shared structure (Atari), and tasks with shared structure, where knowledge transfer is possible (Procgen CoinRun variants). Compared to model-free and model-based baselines with replay buffers of the same-size, ARROW demonstrates substantially less forgetting on tasks without shared structure, while maintaining comparable forward transfer. Our findings highlight the potential of model-based RL and bio-inspired approaches for continual reinforcement learning, warranting further research.
Figures & tables
Figure 1: World Model Learning. (A) Images drawn from the replay buffer are encoded to and reconstructed from a latent space using a recurrent state space model. (B) Learning the policy is achieved with Actor ( A ) and Critic ( C ) networks applied to latent states “dreamt-up” by the model.
Figure 2: Experiment setup. (A) Augmented buffer used in ARROW. (B) Continual learning tasks with and without shared structure. NB: no background, RT: restricted themes, GA: generated assets, MA: monochrome assets, CA: centered agent.
Figure 3: Atari median normalized performance (Eq. 1 ). Shaded area depicts 0.25 and 0.75 quartiles of 5 seeds. Bold line segments indicate training of task. (A) Default order of tasks (one-cycle). (B) Reversed order of tasks (one-cycle). (C) Default order of tasks (two-cycle). The dotted, vertical line marks the end of cycle 1 and the beginning of cycle 2.
Figure 4: Atari metrics shown as median with (0.25 - 0.75) quartile confidence intervals, across 5 seeds, and calculated using normalized scores (Eq. 1 ). (A) Default task order (one-cycle). (B) Reversed task order (one-cycle). (C) Default task order (two-cycle).
Figure 5: CoinRun median normalized performance (Eq. equation 1 ). Shaded area depicts 0.25 and 0.75 quartiles of 5 seeds. Bold line segments indicate training of task. (A) Default order of tasks (one-cycle). (B) Reversed order of tasks (one-cycle). (C) Default order of tasks (two-cycle). The dotted vertical line marks the end of cycle 1 and the beginning of cycle 2.
Figure 6: CoinRun metrics shown as median with (0.25 - 0.75) quartile confidence intervals, across 5 seeds, and calculated using normalized scores, Eq. 1 . (A) Default order of tasks (one-cycle). (B) Reversed order of tasks (one-cycle). (C) Default order of tasks (two-cycle).
Figure 7: Atari median normalized performance (Eq. 1 ) of the ARROW buffer-ratio variants AR25, AR50, and AR75. Shaded area depicts 0.25 and 0.75 quartiles of 5 seeds. Bold line segments indicate training of task. (A) Default order of tasks (one-cycle). (B) Reversed order of tasks (one-cycle). (C) Default order of tasks (two-cycle). The dotted, vertical line marks the end of cycle 1 and the beginning of cycle 2.
Figure 8: CoinRun median normalized performance (Eq. equation 1 ) of the ARROW buffer-ratio variants AR25, AR50, and AR75. Shaded area depicts 0.25 and 0.75 quartiles of 5 seeds. Bold line segments indicate training of task. (A) Default order of tasks (one-cycle). (B) Reversed order of tasks (one-cycle). (C) Default order of tasks (two-cycle). The dotted vertical line marks the end of cycle 1 and the beginning of cycle 2.
CoinRun: 1 env. step per frame; Atari: 4 env. steps per frame
Default / Reversed schedule
90 epochs per task, then shift to next task (single pass over 540 epochs)
Two-cycle schedule
Task shift every 45 epochs; cycle 1 ends at 270 epochs; cycle 2 uses remaining 270 epochs
Evaluation frequency
Every 10 epochs (55 checkpoints over 540 epochs, including epoch 0)
Evaluation scope
All tasks in the current sequence at each checkpoint
Appendix
Table A.2: Training schedule and evaluation protocol.
Table 11Table 12Table 13Table 14
85% Threshold
Frame at Middle
Method
Max Perf.
Env. Frames (median [q25–q75])
Runs ≥ 85%
Default Task Order
1.71
4,423,680
ARROW
0.80
Never reached threshold
0/5
DV3
2.01
5,652,480 [5,570,560 – 5,775,360]
4/5
TES-SAC
0.16
Never reached threshold
0/5
Reversed Task Order
0.69
4,423,680
ARROW
0.81
3,604,480 [3,522,560 – 5,079,040]
3/5
Appendix
Table A.7: Atari (Normalized) continual learning sample-efficiency. Best performance is written in bold.
85% Threshold
Frame at Middle
Method
Max Perf.
Env. Frames (median [q25–q75])
Runs ≥ 85%
Default Task Order
0.88
4,423,680
ARROW
1.03
1,638,400 [655,360 – 2,293,760]
5/5
DV3
1.04
491,520 [327,680 – 819,200]
5/5
TES-SAC
0.75
Never reached threshold
0/5
Reversed Task Order
0.99
4,423,680
ARROW
1.16
3,604,480 [3,112,960 – 3,768,320]
5/5
Appendix
Table A.8: CoinRun (Normalized) continual learning sample-efficiency. Best performance is written in bold.
85% Threshold
Frame at Middle
Method
Max Perf.
Env. Frames (median [q25–q75])
Runs ≥ 85%
Default Task Order
1.69
4,423,680
AR25
1.99
5,734,400 [5,488,640 – 5,898,240]
3/5
AR50
0.80
Never reached threshold
0/5
AR75
0.77
Never reached threshold
0/5
Reversed Task Order
0.69
4,423,680
AR25
0.64
3,686,400 [3,563,520 – 3,809,280]
2/5
Appendix
Table A.9: ARROW Buffer-ratio ablation on Atari - (Normalized) continual learning sample-efficiency. Best performance is written in bold.
85% Threshold
Frame at Middle
Method
Max Perf.
Env. Frames (median [q25–q75])
Runs ≥ 85%
Default Task Order
1.00
4,423,680
AR25
1.17
2,621,440 [2,457,600 – 4,751,360]
5/5
AR50
1.03
4,014,080 [3,072,000 – 4,751,360]
4/5
AR75
1.07
4,751,360 [4,259,840 – 5,242,880]
5/5
Reversed Task Order
0.99
4,423,680
AR25
1.16
4,587,520 [3,768,320 – 4,587,520]
5/5
Appendix
Table A.10: ARROW Buffer-ratio ablation on CoinRun - (Normalized) continual learning sample-efficiency. Best performance is written in bold.
Figure A.1: Atari metrics for the ARROW buffer-ratio variants AR25, AR50, and AR75, shown as median with (0.25 - 0.75) quartile confidence intervals across 5 seeds, and calculated using normalized scores (Eq. 1 ). (A) Default task order (one-cycle). (B) Reversed task order (one-cycle). (C) Default task order (two-cycle).
Figure A.2: CoinRun metrics for the ARROW buffer-ratio variants AR25, AR50, and AR75, shown as median with (0.25 - 0.75) quartile confidence intervals across 5 seeds, and calculated using normalized scores (Eq. 1 ). (A) Default order of tasks (one-cycle). (B) Reversed order of tasks (one-cycle). (C) Default order of tasks (two-cycle).
Task
Task-Specific Threshold (85%)
Method
Max Perf.
Env. Frames (median [q25–q75])
Runs ≥ 85%
Ms. Pac-Man
131.43
ARROW
96.26
Never reached 85% of peak
0/5
DV3
154.62
1,310,720 [1,310,720 – 1,310,720]
1/5
TES-SAC
46.00
Never reached 85% of peak
0/5
Boxing
84.05
ARROW
98.88
819,200 [655,360 – 983,040]
5/5
DV3
98.71
983,040 [819,200 – 983,040]
5/5
TES-SAC
3.12
Never reached 85% of peak
0/5
Appendix
Table A.11: Atari single-task sample-efficiency (raw rewards). Best performance is written in bold.
Task
Task-Specific Threshold (85%)
Method
Max Perf.
Env. Frames (median [q25–q75])
Runs ≥ 85%
CoinRun
6.08
ARROW
6.99
491,520 [491,520 – 819,200]
5/5
DV3
7.15
491,520 [491,520 – 819,200]
5/5
TES-SAC
6.56
655,360 [655,360 – 655,360]
2/5
+NB
6.84
ARROW
7.46
983,040 [901,120 – 983,040]
4/5
DV3
8.05
1,146,880 [983,040 – 1,310,720]
5/5
TES-SAC
6.45
Never reached threshold
0/5
Appendix
Table A.12: CoinRun single-task sample-efficiency (raw rewards). Best performance is written in bold.
Figure A.3: Performance of DreamerV3, with bold line segments denoting the periods in which certain tasks are being trained. Scores are normalized using min-max normalization. The line is the median and the shaded area is between the 0.25 and 0.75 quantiles, of 5 seeds.
Env. frames
Env. steps
Replay buffer capacity
ARROW
1.47M
1.47M
2×512 sequences ×T=512 ( 219 obs.)
DreamerV3
1.47M
1.47M
1024 sequences ×T=512 ( 219 obs.)
TES-SAC
1.47M
1.47M
1024 sequences ×T=512 ( 219 obs.)
Appendix
Table A.13: CoinRun single-task training parameters.
Env. frames
Env. steps
Replay buffer capacity
ARROW
1.47M
5.89M
2×512 sequences ×T=512 ( 219 obs.)
DreamerV3
1.47M
5.89M
1024 sequences ×T=512 ( 219 obs.)
TES-SAC
1.47M
5.89M
1024 sequences ×T=512 ( 219 obs.)
Appendix
Table A.14: Atari single-task training parameters.
Task
Reward scale
Random
ARROW
Ms. Pac-Man
0.05
12.40
1540.30
Boxing
1
0.51
90.27
Crazy Climber
0.001
7.49
109245.16
Frostbite
0.2
14.48
297.83
Seaquest
0.5
38.47
439.62
Enduro
0.5
0.01
707.47
Appendix
Table A.15: Atari single-task experimental results, median across 5 random seeds. Scores are unnormalized at the end of training.
Task
Random
ARROW
CoinRun
2.78
6.09
+NB
2.45
7.14
+NB+RT
2.70
6.89
+NB+RT+GA
2.62
6.85
+NB+RT+GA+MA
2.50
7.89
+NB+RT+GA+MA+CA
2.69
5.78
Appendix
Table A.16: Procgen CoinRun single-task experimental results, median across 5 random seeds. Scores are unnormalized at the end of training.
Figure B.1: Actor critic definition.
Name
ARROW
DreamerV3
TES-SAC
Replay capacity (FIFO)
0.26M
0.52M
0.52M
Replay capacity (long-term)
0.26M
0
0
Batch size
16
16
128
Learning rate
1×10−4
1×10−4
5×10−4
Activation (MLP)
LayerNorm+SiLU
LayerNorm+SiLU
ReLU
Activation (GRU)
tanh
tanh
–
Appendix
Table C.1: Hyperparameters.
Parameter
Atari
CoinRun
Data Collection
Parallel environments
4
4
Sequence length
4096
4096
Environment repeat
4
1
Data sequences per batch
32
32
Max sequences in buffer
512
512
Appendix
Table C.2: General training and experimental parameters.
Parameter
Value
Learning Rates
Policy learning rate
5×10−4
Q-network learning rate
5×10−4
Entropy coefficient (alpha) learning rate
5×10−4
Core SAC Hyperparameters
Batch size
128
Appendix
Table C.3: TES-SAC Hyperparameters
Benchmark
Task
Runs
Total
Atari
Single-task
6 tasks ×5 seeds
30
Continual learning
3 settings (A, B, C) ×5 seeds
15
CoinRun
Single-task
6 tasks ×5 seeds
30
Continual learning
3 settings (A, B, C) ×5 seeds
15
Per method, per benchmark
(30+15)
45
Per benchmark, all methods (ARROW, DV3, TES-SAC)
45 runs ×3 methods
135
Appendix
Table C.4: Experimental run breakdown across benchmarks and methods.
Model-based reinforcement-learning agents of the DreamerV3 family forget catastrophically when trained on task sequences, even when an unbounded replay buffer preserves every earlier experience. We ask a question the continual-RL literature has assumed an answer to but never measured: which component forgets? Under never-clear replay, pre-registered component-level probes (n=3 seeds throughout) show that the world model retains essentially everything measurable about old tasks -- reward discrimination (retention ratio ~1.0), value estimates, and termination structure -- while the actor's behavior collapses. Forgetting in this regime is a channel problem, not a memory problem. We demonstrate this by intervention: with the world model frozen and identical imagined rollouts, reinforcement learning in imagination fails to recover a lost skill (0/3 seeds), while supervised self-imitation on the world model's own graded dreams recovers it on 3/3 seeds with zero environment interaction. Interleaved during training, this graded dream rehearsal yields a task-label-free, parameter-constant continual learner: 3/3 four-task chains retained where plain replay passes 0/3, 3/3 eight-task chains, and consistent gains over matched real-episode cloning (paired difference +0.13, bootstrap 95% CI [0.07, 0.24], complete seed separation). The dream-grading step is load-bearing: we characterize two scoring failure modes, provide an offline selection gauge that caught both before they contaminated results, and give a realized-first grading rule that closes them. All experiments were pre-registered with committed protocols; every refuted hypothesis is reported.
Genuine embodied agency requires robots to turn continuous real-world experience into lasting, transferable skills. This demands continual learning that integrates new capabilities while preserving prior knowledge in evolving environments. Experience replay mitigates catastrophic forgetting, but storing complete demonstrations becomes costly as tasks accumulate. World-action models provide an alternative through generative replay. However, visually coherent rollouts may contain actions that cannot realize the predicted transitions, while new-task adaptation can disrupt learned behavior. To address these problems, we propose RIFAR (Reliability-Informed and Forgetting-Aware Replay), a reliability-informed and drift-aware generative replay framework. It reconstructs trajectories from compact historical demonstration prefixes and uses a trainable inverse-dynamics model to assess action-visual consistency. It first adapts to new tasks using current-task demonstrations and high-reliability replay, then evaluates how this adaptation changes action predictions on historical inputs and prioritizes reliable trajectories with greater action drift for further training. Across three LIBERO suites and real-world experiments, RIFAR outperforms the previous state of the art in WAM-based generative replay. On LIBERO-Goal, it achieves an AUC of 90.97 while retaining only 320 historical time steps per task, about 4.9% of those retained by 50-demonstration replay.
Tu Lin, Zheng Lu, Zirui Song +7
Tsinghua University · MBZUAI · Zhejiang University
Continual reinforcement learning must balance retention with adaptation, yet many methods still rely on single-model preservation, committing to one evolving policy as the main reusable solution across tasks. Even when a previously successful policy is retained, it may no longer provide a reliable starting point for rapid adaptation after interference, reflecting a form of loss of plasticity that single-policy preservation cannot address. Inspired by quality-diversity methods, we introduce TeLAPA (Transfer-Enabled Latent-Aligned Policy Archives), a continual RL framework that organizes behaviorally diverse policy neighborhoods into per-task archives and maintains a shared latent space so that archived policies remain comparable and reusable under non-stationary drift. This perspective shifts continual RL from retaining isolated solutions to maintaining skill-aligned neighborhoods with competent and behaviorally related policies that support future relearning. In our MiniGrid CL setting, TeLAPA learns more tasks successfully, recovers competence faster on revisited tasks after interference, and retains higher performance across a sequence of tasks. Our analyses show that source-optimal policies are often not transfer-optimal, even within a local competent neighborhood, and that effective reuse depends on retaining and selecting among multiple nearby alternatives rather than collapsing them to one representative. Together, these results reframe continual RL around reusable and competent policy neighborhoods, providing a route beyond single-model preservation toward more plastic lifelong agents.
Lute Lillo, Nick Cheney
Department of Computer Science, University of Vermont, USA