ARROW: Augmented Replay for RObust World models

Abdulaziz Alyahya; Abdallah Al Siyabi; Markus R. Ernst; Luke Yang; Levin Kuhlmann; Gideon Kowadlo

cs.LGMar 12, 2026

ARROW: Augmented Replay for RObust World models

Authors: Abdulaziz Alyahya, Abdallah Al Siyabi, Markus R. Ernst, Luke Yang, Levin Kuhlmann, Gideon Kowadlo

Organizations: Department of Information Systems, College of Computer and Information Sciences, Imam Mohammad Ibn Saud Islamic University (IMSIU) · Department of Data Science & AI, Faculty of Information Technology, Monash University · School of Computer Science and Engineering, Faculty of Engineering, University of New South Wales, Sydney

Abstract

Continual reinforcement learning challenges agents to acquire new skills while retaining previously learned ones with the goal of improving performance in both past and future tasks. Most existing approaches rely on model-free methods with replay buffers to mitigate catastrophic forgetting; however, these solutions often face significant scalability challenges due to large memory demands. Drawing inspiration from neuroscience, where the brain replays experiences to a predictive World Model rather than directly to the policy, we present ARROW (Augmented Replay for RObust World models), a model-based continual RL algorithm that extends DreamerV3 with a memory-efficient, distribution-matching replay buffer. Unlike standard fixed-size FIFO buffers, ARROW maintains two complementary buffers: a short-term buffer for recent experiences and a long-term buffer that preserves task diversity through intelligent sampling. We evaluate ARROW on two challenging continual RL settings: Tasks without shared structure (Atari), and tasks with shared structure, where knowledge transfer is possible (Procgen CoinRun variants). Compared to model-free and model-based baselines with replay buffers of the same-size, ARROW demonstrates substantially less forgetting on tasks without shared structure, while maintaining comparable forward transfer. Our findings highlight the potential of model-based RL and bio-inspired approaches for continual reinforcement learning, warranting further research.

Figures & tables

Figure 1: World Model Learning. (A) Images drawn from the replay buffer are encoded to and reconstructed from a latent space using a recurrent state space model. (B) Learning the policy is achieved with Actor ( $\mathcal{A}$ ) and Critic ( $\mathcal{C}$ ) networks applied to latent states “dreamt-up” by the model.

Figure 2: Experiment setup. (A) Augmented buffer used in ARROW. (B) Continual learning tasks with and without shared structure. NB: no background, RT: restricted themes, GA: generated assets, MA: monochrome assets, CA: centered agent.

Figure 3: Atari median normalized performance (Eq. 1 ). Shaded area depicts 0.25 and 0.75 quartiles of 5 seeds. Bold line segments indicate training of task. (A) Default order of tasks (one-cycle). (B) Reversed order of tasks (one-cycle). (C) Default order of tasks (two-cycle). The dotted, vertical line marks the end of cycle 1 and the beginning of cycle 2.

Figure 4: Atari metrics shown as median with (0.25 - 0.75) quartile confidence intervals, across 5 seeds, and calculated using normalized scores (Eq. 1 ). (A) Default task order (one-cycle). (B) Reversed task order (one-cycle). (C) Default task order (two-cycle).

Figure 5: CoinRun median normalized performance (Eq. equation 1 ). Shaded area depicts 0.25 and 0.75 quartiles of 5 seeds. Bold line segments indicate training of task. (A) Default order of tasks (one-cycle). (B) Reversed order of tasks (one-cycle). (C) Default order of tasks (two-cycle). The dotted vertical line marks the end of cycle 1 and the beginning of cycle 2.

Figure 6: CoinRun metrics shown as median with (0.25 - 0.75) quartile confidence intervals, across 5 seeds, and calculated using normalized scores, Eq. 1 . (A) Default order of tasks (one-cycle). (B) Reversed order of tasks (one-cycle). (C) Default order of tasks (two-cycle).

Figure 7: Atari median normalized performance (Eq. 1 ) of the ARROW buffer-ratio variants AR25, AR50, and AR75. Shaded area depicts 0.25 and 0.75 quartiles of 5 seeds. Bold line segments indicate training of task. (A) Default order of tasks (one-cycle). (B) Reversed order of tasks (one-cycle). (C) Default order of tasks (two-cycle). The dotted, vertical line marks the end of cycle 1 and the beginning of cycle 2.

Figure 8: CoinRun median normalized performance (Eq. equation 1 ) of the ARROW buffer-ratio variants AR25, AR50, and AR75. Shaded area depicts 0.25 and 0.75 quartiles of 5 seeds. Bold line segments indicate training of task. (A) Default order of tasks (one-cycle). (B) Reversed order of tasks (one-cycle). (C) Default order of tasks (two-cycle). The dotted vertical line marks the end of cycle 1 and the beginning of cycle 2.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Variant	Procgen flag	Description
Coinrun	—	regularly rendered game
+NB	use_backgrounds = False	removes decorative backgrounds
+RT	restrict_themes = True	restricts the set of level themes
+GA	use_generated_assets = True	enables procedurally generated assets
+MA	use_monochrome_assets = True	enables monochrome assets
+CA	center_agent = False	camera does not remain centered on the agent

Appendix

Table A.1: CoinRun task variations (Procgen configuration flags).

Item	Specification
Training budget	8.84 million environment frames over 540 epochs
Frame definition	CoinRun: 1 env. step per frame; Atari: 4 env. steps per frame
Default / Reversed schedule	90 epochs per task, then shift to next task (single pass over 540 epochs)
Two-cycle schedule	Task shift every 45 epochs; cycle 1 ends at 270 epochs; cycle 2 uses remaining 270 epochs
Evaluation frequency	Every 10 epochs (55 checkpoints over 540 epochs, including epoch 0)
Evaluation scope	All tasks in the current sequence at each checkpoint

Appendix

Table A.2: Training schedule and evaluation protocol.

Table 11

Table 12

Table 13

Table 14

85% Threshold	Frame at Middle	Method	Max Perf.	Env. Frames (median [q25–q75])	Runs $\geq$ 85%
Default Task Order
1.71	4,423,680	ARROW	0.80	Never reached threshold	0/5
		DV3	2.01	5,652,480 [5,570,560 – 5,775,360]	4/5
		TES-SAC	0.16	Never reached threshold	0/5
Reversed Task Order
0.69	4,423,680	ARROW	0.81	3,604,480 [3,522,560 – 5,079,040]	3/5

Appendix

Table A.7: Atari (Normalized) continual learning sample-efficiency. Best performance is written in bold.

85% Threshold	Frame at Middle	Method	Max Perf.	Env. Frames (median [q25–q75])	Runs $\geq$ 85%
Default Task Order
0.88	4,423,680	ARROW	1.03	1,638,400 [655,360 – 2,293,760]	5/5
		DV3	1.04	491,520 [327,680 – 819,200]	5/5
		TES-SAC	0.75	Never reached threshold	0/5
Reversed Task Order
0.99	4,423,680	ARROW	1.16	3,604,480 [3,112,960 – 3,768,320]	5/5

Appendix

Table A.8: CoinRun (Normalized) continual learning sample-efficiency. Best performance is written in bold.

85% Threshold	Frame at Middle	Method	Max Perf.	Env. Frames (median [q25–q75])	Runs $\geq$ 85%
Default Task Order
1.69	4,423,680	AR25	1.99	5,734,400 [5,488,640 – 5,898,240]	3/5
		AR50	0.80	Never reached threshold	0/5
		AR75	0.77	Never reached threshold	0/5
Reversed Task Order
0.69	4,423,680	AR25	0.64	3,686,400 [3,563,520 – 3,809,280]	2/5

Appendix

Table A.9: ARROW Buffer-ratio ablation on Atari - (Normalized) continual learning sample-efficiency. Best performance is written in bold.

85% Threshold	Frame at Middle	Method	Max Perf.	Env. Frames (median [q25–q75])	Runs $\geq$ 85%
Default Task Order
1.00	4,423,680	AR25	1.17	2,621,440 [2,457,600 – 4,751,360]	5/5
		AR50	1.03	4,014,080 [3,072,000 – 4,751,360]	4/5
		AR75	1.07	4,751,360 [4,259,840 – 5,242,880]	5/5
Reversed Task Order
0.99	4,423,680	AR25	1.16	4,587,520 [3,768,320 – 4,587,520]	5/5

Appendix

Table A.10: ARROW Buffer-ratio ablation on CoinRun - (Normalized) continual learning sample-efficiency. Best performance is written in bold.

Figure A.1: Atari metrics for the ARROW buffer-ratio variants AR25, AR50, and AR75, shown as median with (0.25 - 0.75) quartile confidence intervals across 5 seeds, and calculated using normalized scores (Eq. 1 ). (A) Default task order (one-cycle). (B) Reversed task order (one-cycle). (C) Default task order (two-cycle).

Figure A.2: CoinRun metrics for the ARROW buffer-ratio variants AR25, AR50, and AR75, shown as median with (0.25 - 0.75) quartile confidence intervals across 5 seeds, and calculated using normalized scores (Eq. 1 ). (A) Default order of tasks (one-cycle). (B) Reversed order of tasks (one-cycle). (C) Default order of tasks (two-cycle).

Task	Task-Specific Threshold (85%)	Method	Max Perf.	Env. Frames (median [q25–q75])	Runs $\geq$ 85%
Ms. Pac-Man	131.43	ARROW	96.26	Never reached 85% of peak	0/5
		DV3	154.62	1,310,720 [1,310,720 – 1,310,720]	1/5
		TES-SAC	46.00	Never reached 85% of peak	0/5
Boxing	84.05	ARROW	98.88	819,200 [655,360 – 983,040]	5/5
		DV3	98.71	983,040 [819,200 – 983,040]	5/5
		TES-SAC	3.12	Never reached 85% of peak	0/5

Appendix

Table A.11: Atari single-task sample-efficiency (raw rewards). Best performance is written in bold.

Task	Task-Specific Threshold (85%)	Method	Max Perf.	Env. Frames (median [q25–q75])	Runs $\geq$ 85%
CoinRun	6.08	ARROW	6.99	491,520 [491,520 – 819,200]	5/5
		DV3	7.15	491,520 [491,520 – 819,200]	5/5
		TES-SAC	6.56	655,360 [655,360 – 655,360]	2/5
+NB	6.84	ARROW	7.46	983,040 [901,120 – 983,040]	4/5
		DV3	8.05	1,146,880 [983,040 – 1,310,720]	5/5
		TES-SAC	6.45	Never reached threshold	0/5

Appendix

Table A.12: CoinRun single-task sample-efficiency (raw rewards). Best performance is written in bold.

Figure A.3: Performance of DreamerV3, with bold line segments denoting the periods in which certain tasks are being trained. Scores are normalized using min-max normalization. The line is the median and the shaded area is between the 0.25 and 0.75 quantiles, of 5 seeds.

	Env. frames	Env. steps	Replay buffer capacity
ARROW	1.47M	1.47M	$2\times 512$ sequences $\times\,T{=}512$ ( $2^{19}$ obs.)
DreamerV3	1.47M	1.47M	$1024$ sequences $\times\,T{=}512$ ( $2^{19}$ obs.)
TES-SAC	1.47M	1.47M	$1024$ sequences $\times\,T{=}512$ ( $2^{19}$ obs.)

Appendix

Table A.13: CoinRun single-task training parameters.

	Env. frames	Env. steps	Replay buffer capacity
ARROW	1.47M	5.89M	$2\times 512$ sequences $\times\,T{=}512$ ( $2^{19}$ obs.)
DreamerV3	1.47M	5.89M	$1024$ sequences $\times\,T{=}512$ ( $2^{19}$ obs.)
TES-SAC	1.47M	5.89M	$1024$ sequences $\times\,T{=}512$ ( $2^{19}$ obs.)

Appendix

Table A.14: Atari single-task training parameters.

Task	Reward scale	Random	ARROW
Ms. Pac-Man	0.05	12.40	1540.30
Boxing	1	0.51	90.27
Crazy Climber	0.001	7.49	109245.16
Frostbite	0.2	14.48	297.83
Seaquest	0.5	38.47	439.62
Enduro	0.5	0.01	707.47

Appendix

Table A.15: Atari single-task experimental results, median across 5 random seeds. Scores are unnormalized at the end of training.

Task	Random	ARROW
CoinRun	2.78	6.09
+NB	2.45	7.14
+NB+RT	2.70	6.89
+NB+RT+GA	2.62	6.85
+NB+RT+GA+MA	2.50	7.89
+NB+RT+GA+MA+CA	2.69	5.78

Appendix

Table A.16: Procgen CoinRun single-task experimental results, median across 5 random seeds. Scores are unnormalized at the end of training.

Figure B.1: Actor critic definition.

Name	ARROW	DreamerV3	TES-SAC
Replay capacity (FIFO)	0.26M	0.52M	0.52M
Replay capacity (long-term)	0.26M	0	0
Batch size	16	16	128
Learning rate	$1\times 10^{-4}$	$1\times 10^{-4}$	$5\times 10^{-4}$
Activation (MLP)	LayerNorm+SiLU	LayerNorm+SiLU	ReLU
Activation (GRU)	tanh	tanh	–

Appendix

Table C.1: Hyperparameters.

Parameter	Atari	CoinRun
Data Collection
Parallel environments	4	4
Sequence length	4096	4096
Environment repeat	4	1
Data sequences per batch	32	32
Max sequences in buffer	512	512

Appendix

Table C.2: General training and experimental parameters.

Parameter	Value
Learning Rates
Policy learning rate	$5\times 10^{-4}$
Q-network learning rate	$5\times 10^{-4}$
Entropy coefficient (alpha) learning rate	$5\times 10^{-4}$
Core SAC Hyperparameters
Batch size	128

Appendix

Table C.3: TES-SAC Hyperparameters

Benchmark	Task	Runs	Total
Atari	Single-task	$6$ tasks $\times$ $5$ seeds	$30$
Atari	Continual learning	$3$ settings (A, B, C) $\times$ $5$ seeds	$15$
CoinRun	Single-task	$6$ tasks $\times$ $5$ seeds	$30$
CoinRun	Continual learning	$3$ settings (A, B, C) $\times$ $5$ seeds	$15$
Per method, per benchmark		$(30+15)$	$45$
Per benchmark, all methods (ARROW, DV3, TES-SAC)		$45$ runs $\times$ $3$ methods	$135$

Appendix

Table C.4: Experimental run breakdown across benchmarks and methods.

Method	Single-task wall time	Continual learning wall time
ARROW / DV3	6 hours	1–2 days (CoinRun CL: $\sim$ 30 hours; Atari CL: $\sim$ 50 hours)
TES-SAC	3 hours	17 hours

Appendix

Table C.5: Wall-clock time per method and setting.

Continual RL

Continual reinforcement learning studies how agents adapt to changing tasks while retaining skills learned earlier.

See all 50 papers

The World Model Remembers, the Actor Forgets: Dream Rehearsal for Continual Model-Based RL
Jul 22, 2026Gurp NijjerGenerative Replay Reinforcement Learning
Quantegra Research, Canada
RIFAR: Reliability-Informed and Forgetting-Aware Replay for Continual Robot Learning with World-Action Models
Oct 2, 2026Tu Lin, Zheng Lu, Zirui Song +7Continual Robot Learning Replay-Based Continual Learning
Tsinghua University · MBZUAI · Zhejiang University
Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning
Apr 16, 2026Lute Lillo, Nick CheneyNon-Stationary RL Reinforcement Learning
Department of Computer Science, University of Vermont, USA

Jul 22, 2026J/K move · Enter open · S save

The World Model Remembers, the Actor Forgets: Dream Rehearsal for Continual Model-Based RL

Gurp Nijjer

Quantegra Research, Canada

Generative Replay Reinforcement Learning Replay-Based Continual Learning Continual RL Model-Based RL Catastrophic Forgetting

Open paper AskPDF arXiv

Model-based reinforcement-learning agents of the DreamerV3 family forget catastrophically when trained on task sequences, even when an unbounded replay buffer preserves every earlier experience. We ask a question the continual-RL literature has assumed an answer to but never measured: which component forgets? Under never-clear replay, pre-registered component-level probes (n=3 seeds throughout) show that the world model retains essentially everything measurable about old tasks -- reward discrimination (retention ratio ~1.0), value estimates, and termination structure -- while the actor's behavior collapses. Forgetting in this regime is a channel problem, not a memory problem. We demonstrate this by intervention: with the world model frozen and identical imagined rollouts, reinforcement learning in imagination fails to recover a lost skill (0/3 seeds), while supervised self-imitation on the world model's own graded dreams recovers it on 3/3 seeds with zero environment interaction. Interleaved during training, this graded dream rehearsal yields a task-label-free, parameter-constant continual learner: 3/3 four-task chains retained where plain replay passes 0/3, 3/3 eight-task chains, and consistent gains over matched real-episode cloning (paired difference +0.13, bootstrap 95% CI [0.07, 0.24], complete seed separation). The dream-grading step is load-bearing: we characterize two scoring failure modes, provide an offline selection gauge that caught both before they contaminated results, and give a realized-first grading rule that closes them. All experiments were pre-registered with committed protocols; every refuted hypothesis is reported.

Abstract

Continual RL

Explore similar work

The World Model Remembers, the Actor Forgets: Dream Rehearsal for Continual Model-Based RL

RIFAR: Reliability-Informed and Forgetting-Aware Replay for Continual Robot Learning with World-Action Models

Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning