Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learning

Ke Sun; Hongming Zhang; Jun Jin; Chao Gao; Xi Chen; Wulong Liu; Linglong Kong

cs.LGMar 1, 2026

Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learning

Authors: Ke Sun, Hongming Zhang, Jun Jin, Chao Gao, Xi Chen, Wulong Liu, Linglong Kong

Organizations: University of Alberta · National Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institution of Automation, Chinese Academy of Sciences · Edmonton Research Center, Huawei Canada · Huawei Noah’s Ark Lab ·

Abstract

Inspired by the human learning and memory system, particularly the interplay between the hippocampus and cerebral cortex, this study proposes a dual-learner framework comprising a fast learner and a meta learner to address continual Reinforcement Learning~(RL) problems. These two learners are coupled to perform distinct yet complementary roles: the fast learner focuses on knowledge transfer, while the meta learner ensures knowledge integration. In contrast to traditional multi-task RL approaches that share knowledge through average return maximization, our meta learner incrementally integrates new experiences by explicitly minimizing catastrophic forgetting, thereby supporting efficient cumulative knowledge transfer for the fast learner. To facilitate rapid adaptation in new environments, we introduce an adaptive meta warm-up mechanism that selectively harnesses past knowledge. We conduct experiments in various pixel-based and continuous control benchmarks, revealing the superior performance of continual learning for our proposed dual-learner approach relative to baseline methods. The code is released in https://github.com/datake/FAME.

Figures & tables

Figure 1: Illustration of FAME. In value-based continual RL, the fast learner can be denoted by $\{Q_{k}\}_{k=1}^{K}$ accordingly instead of $\{\pi_{k}\}_{k=1}^{K}$ .

Method	Ave. Perf $\uparrow$			FT $\uparrow$	Forgetting $\downarrow$
Method	Breakout	Spaceinvader	Freeway	FT $\uparrow$	Forgetting $\downarrow$
Reset	6.51 $\pm$ 1.67	3.29 $\pm$ 3.09	0.74 $\pm$ 0.38	0.00 $\pm$ 0.00	1.31 $\pm$ 0.23
Finetune	10.62 $\pm$ 2.75	4.95 $\pm$ 2.92	0.89 $\pm$ 0.49	0.13 $\pm$ 0.03	1.26 $\pm$ 0.32
MultiHead	6.85 $\pm$ 1.76	3.26 $\pm$ 2.99	0.94 $\pm$ 0.42	-0.01 $\pm$ 0.00	1.25 $\pm$ 0.22
LargeBuffer	10.71 $\pm$ 2.84	3.24 $\pm$ 2.91	1.16 $\pm$ 0.59	0.16 $\pm$ 0.02	1.65 $\pm$ 0.33
PT-DQN	0.39 $\pm$ 0.02	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	0.07 $\pm$ 0.02	1.64 $\pm$ 0.02

Table 1: Main results on MinAtar on Average Performance ( Avg. Perf ), Forward Transfer ( FT ), and Forgetting. Results (Mean

\pm

SE) are averaged over 10 sequences, each with 3 seeds.

\uparrow

denotes a positive metric (more is better), while

\downarrow

is a negative one (less is better). Reset is the baseline for evaluating FT. Forgetting is normalized by the standard deviation in each task.

Panel of Figure 2 — Figure 2: (Left) Average performance of the policy across each task across 10 sequences on MinAtar . Results are averaged over 3 seeds. The vertical lines at each point represent the standard errors. (Right) The selection ratio among three warm-up strategies when the arriving environment is previously encountered or novel.

Method	Freeway		SpaceInvader
Method	Avg. Perf $\uparrow$	FT $\uparrow$	Avg. Perf $\uparrow$	FT $\uparrow$
Reset	0.16 $\pm$ 0.18	0.00	0.10 $\pm$ 0.22	0.00
Finetune	0.21 $\pm$ 0.17	0.53	0.61 $\pm$ 0.41	0.65
ProgressiveNet	0.39 $\pm$ 0.25	0.21	0.61 $\pm$ 0.03	0.06
PackNet	0.41 $\pm$ 0.24	0.18	0.47 $\pm$ 0.06	0.17
FAME	0.90 $\pm$ 0.12	0.68	0.96 $\pm$ 0.02	0.63

Table 2: Main results on Atari games on Average Performance ( Avg. Perf ) and Forward Transfer ( FT ). Results (Mean

\pm

SE) are averaged over 3 seeds. The Forgetting metric is omitted as PackNet and ProgressiveNet store past model parameters and have zero forgetting. Reset is the baseline for FT.

Methods	Avg. Perf $\uparrow$	FT $\uparrow$	Forgetting $\downarrow$
Reset	0.093 $\pm$ 0.017	0.000 $\pm$ 0.000	0.710 $\pm$ 0.030
Finetune	0.037 $\pm$ 0.011	-0.265 $\pm$ 0.028	0.427 $\pm$ 0.033
Average	0.013 $\pm$ 0.007	-0.530 $\pm$ 0.024	0.070 $\pm$ 0.022
PackNet	0.491 $\pm$ 0.025	-0.194 $\pm$ 0.018	0.000 $\pm$ 0.000
FAME-KL	0.733 $\pm$ 0.026	0.022 $\pm$ 0.015	0.073 $\pm$ 0.019
FAME-WD	0.767 $\pm$ 0.024	-0.003 $\pm$ 0.014	0.023 $\pm$ 0.015

Table 3: Main results on Meta-World on Average Performance ( Ave. Perf ), Forward Transfer ( FT ), and Forgetting averaged over 3 sequences. Results are presented as averages and standard errors across 10 seeds.

Panel of Figure 3 — Figure 3: (Left) Performance profile of the fast learner across tasks, where the y-axis shows the proportion of tasks that achieve a success rate greater than or equal to the x-axis value. (Right) Average performance by evaluating the average success rates in past tasks across 10 seeds.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Ave. Perf $\uparrow$			FT $\uparrow$	Forgetting $\downarrow$
Method	Breakout	Spaceinvader	Freeway	FT $\uparrow$	Forgetting $\downarrow$
FAME ( $\lambda$ =0.1)	12.79 $\pm$ 0.42	19.52 $\pm$ 0.50	1.71 $\pm$ 0.17	0.13 $\pm$ 0.03	0.77 $\pm$ 0.08
FAME ( $\lambda$ =1.0)	14.54 $\pm$ 0.58	18.72 $\pm$ 0.52	1.69 $\pm$ 0.17	0.16 $\pm$ 0.03	0.72 $\pm$ 0.13
FAME ( $\lambda$ =5.0)	13.90 $\pm$ 0.55	19.38 $\pm$ 0.62	1.62 $\pm$ 0.16	0.14 $\pm$ 0.03	0.64 $\pm$ 0.07
FAME ( $\lambda$ =10.0)	13.88 $\pm$ 0.55	19.52 $\pm$ 0.57	1.63 $\pm$ 0.16	0.14 $\pm$ 0.03	0.67 $\pm$ 0.08

Appendix

Table 4: Ablation Study of Regularization Hyperparameter

\lambda

on MinAtar on Average Performance ( Avg. Perf ), Forward Transfer ( FT ), and Forgetting. Results are averaged over 10 sequences, each with 3 seeds.

Method	Ave. Perf $\uparrow$			FT $\uparrow$	Forgetting $\downarrow$
Method	Breakout	Spaceinvader	Freeway	FT $\uparrow$	Forgetting $\downarrow$
FAME ( $L=1\times 10^{4}$ )	13.34 $\pm$ 0.52	19.17 $\pm$ 0.68	1.64 $\pm$ 0.16	0.13 $\pm$ 0.03	0.74 $\pm$ 0.08
FAME ( $L=5\times 10^{4}$ )	14.54 $\pm$ 0.58	18.72 $\pm$ 0.52	1.69 $\pm$ 0.17	0.16 $\pm$ 0.03	0.72 $\pm$ 0.13
FAME ( $L=20\times 10^{4}$ )	13.28 $\pm$ 0.50	19.87 $\pm$ 0.66	1.65 $\pm$ 0.16	0.17 $\pm$ 0.03	0.72 $\pm$ 0.08
FAME ( $L=50\times 10^{4}$ )	11.83 $\pm$ 0.71	20.00 $\pm$ 0.96	1.87 $\pm$ 0.20	0.17 $\pm$ 0.03	0.78 $\pm$ 0.09

Appendix

Table 5: Ablation Study of Warm-Up Step

L

on MinAtar on Average Performance ( Avg. Perf ), Forward Transfer ( FT ), and Forgetting. Results are averaged over 10 sequences, each with 3 seeds.

Method	Ave. Perf $\uparrow$			FT $\uparrow$	Forgetting $\downarrow$
Method	Breakout	Spaceinvader	Freeway	FT $\uparrow$	Forgetting $\downarrow$
FAME ( $N$ =4000)	11.95 $\pm$ 0.47	14.69 $\pm$ 0.55	1.54 $\pm$ 0.17	0.14 $\pm$ 0.03	0.93 $\pm$ 0.08
FAME ( $N$ =8000)	12.49 $\pm$ 0.37	17.99 $\pm$ 0.58	1.67 $\pm$ 0.17	0.15 $\pm$ 0.03	0.86 $\pm$ 0.08
FAME ( $N$ =12000)	14.54 $\pm$ 0.58	18.72 $\pm$ 0.52	1.69 $\pm$ 0.17	0.16 $\pm$ 0.03	0.72 $\pm$ 0.13

Appendix

Table 6: Ablation Study of Weight Estimation Step

N

on MinAtar on Average Performance ( Avg. Perf ), Forward Transfer ( FT ), and Forgetting. Results are averaged over 10 sequences, each with 3 seeds.

Method	Ave. Perf $\uparrow$			FT $\uparrow$	Forgetting $\downarrow$
Method	Breakout	Spaceinvader	Freeway	FT $\uparrow$	Forgetting $\downarrow$
FAME ( $n$ =300)	13.02 $\pm$ 0.49	19.19 $\pm$ 0.65	1.42 $\pm$ 0.11	0.12 $\pm$ 0.03	0.69 $\pm$ 0.07
FAME ( $n$ =600)	14.54 $\pm$ 0.58	18.72 $\pm$ 0.52	1.69 $\pm$ 0.17	0.16 $\pm$ 0.03	0.72 $\pm$ 0.13
FAME ( $n$ =1200)	13.46 $\pm$ 0.63	19.39 $\pm$ 0.42	1.83 $\pm$ 0.20	0.16 $\pm$ 0.03	0.79 $\pm$ 0.08
FAME ( $n$ =5000)	13.09 $\pm$ 0.41	19.32 $\pm$ 0.52	1.84 $\pm$ 0.18	0.14 $\pm$ 0.03	0.76 $\pm$ 0.07

Appendix

Table 7: Ablation Study of Policy Evaluation Step

n

on MinAtar on Average Performance ( Avg. Perf ), Forward Transfer ( FT ), and Forgetting. Results are averaged over 10 sequences, each with 3 seeds.

Panel of Figure 4 — Figure 4: Learning curves of the fast learner in FAME on MinAtar Environments across 10 sequences of tasks.

Figure 5: Learning curves of the fast learner in FAME on the SpaceIvader environment averaged over 3 seeds.

Figure 6: Learning curves of the fast learner in FAME on the Freeway environment averaged over 3 seeds.

Methods	Avg. Perf $\uparrow$	FT $\uparrow$	Forgetting $\downarrow$
Reset	0.090 $\pm$ 0.029	0.000 $\pm$ 0.000	0.800 $\pm$ 0.040
Finetune	0.070 $\pm$ 0.026	-0.294 $\pm$ 0.039	0.480 $\pm$ 0.050
Average	0.020 $\pm$ 0.014	-0.584 $\pm$ 0.035	0.100 $\pm$ 0.030
PackNet	0.703 $\pm$ 0.041	-0.111 $\pm$ 0.028	0.000 $\pm$ 0.000
FAME-WD	0.87 $\pm$ 0.034	0.004 $\pm$ 0.022	0.010 $\pm$ 0.017
FAME-KL	0.86 $\pm$ 0.035	0.042 $\pm$ 0.019	0.050 $\pm$ 0.026

Appendix

Table 8: Results on Meta-World on Average Performance ( Ave. Perf ), Forward Transfer ( FT ), and Forgetting for each sequence of tasks . Results are presented as averages and standard errors across 10 seeds. Each table, from top to bottom, represents each sequence of tasks, respectively. The best results are highlighted in bold, and the second best are underlined.

\uparrow

denotes a positive metric (more is better), while

\downarrow

is a negative one (less is better). Reset is the baseline for evaluating FT.

Methods	Avg. Perf $\uparrow$	FT $\uparrow$	Forgetting $\downarrow$
Reset	0.093 $\pm$ 0.017	0.000 $\pm$ 0.000	0.710 $\pm$ 0.030
Finetune	0.037 $\pm$ 0.011	-0.265 $\pm$ 0.028	0.427 $\pm$ 0.033
Average	0.013 $\pm$ 0.007	-0.530 $\pm$ 0.024	0.070 $\pm$ 0.022
PackNet	0.491 $\pm$ 0.025	-0.194 $\pm$ 0.018	0.000 $\pm$ 0.000
FAME-WD	0.767 $\pm$ 0.024	-0.003 $\pm$ 0.014	0.023 $\pm$ 0.015
FAME-KL	0.733 $\pm$ 0.026	0.022 $\pm$ 0.015	0.073 $\pm$ 0.019

Appendix

Table 9: Results on Meta-World averaged over the three sequences on Average Performance ( Ave. Perf ), Forward Transfer ( FT ), and Forgetting. Results are presented as averages and standard errors across 10 seeds. The best results are highlighted in bold, and the second best are underlined.

\uparrow

denotes a positive metric (more is better), while

\downarrow

is a negative one (less is better). Reset is the baseline for evaluating FT.

Panel of Figure 7 — Figure 7: (Left) Performance profile of the fast learner across tasks, where the y-axis shows the proportion of tasks that achieve a success rate greater than or equal to the x-axis value. (Right) Average performance over time by evaluating the average success rates in the past tasks. Each Row represents the result for one sequence of tasks. The last row shows the average performance of the three sequences.

Panel of Figure 8 — Figure 8: Learning curves of the fast learner on the Meta-World benchmark. The x-axis shows the total number of environment interactions, and the y-axis indicates the success rate. Each Row represents the result for one sequence of tasks. The last row shows the average performance of the three sequences.

Methods	Avg. Perf $\uparrow$	FT $\uparrow$	Forgetting $\downarrow$
Reset	0.090 $\pm$ 0.029	0.000 $\pm$ 0.000	0.800 $\pm$ 0.040
Finetune	0.070 $\pm$ 0.026	-0.294 $\pm$ 0.039	0.480 $\pm$ 0.050
Average	0.020 $\pm$ 0.014	-0.584 $\pm$ 0.035	0.100 $\pm$ 0.030
PackNet	0.703 $\pm$ 0.041	-0.111 $\pm$ 0.028	0.000 $\pm$ 0.000
ProgressiveNet	0.760 $\pm$ 0.038	-0.063 $\pm$ 0.031	0.000 $\pm$ 0.000
CompoNet	0.803 $\pm$ 0.036	-0.007 $\pm$ 0.022	0.000 $\pm$ 0.000

Appendix

Table 10: Results on Meta-World on Average Performance ( Ave. Perf ), Forward Transfer ( FT ), and Forgetting for each sequence of tasks with more baselines . Results are presented as averages and standard errors across 10 seeds. Each table, from top to bottom, represents each sequence of tasks, respectively. The best results are highlighted in bold, and the second best are underlined.

\uparrow

denotes a positive metric (more is better), while

\downarrow

is a negative one (less is better). Reset is the baseline for evaluating FT.

Methods	Avg. Perf $\uparrow$	FT $\uparrow$	Forgetting $\downarrow$
Reset	0.093 $\pm$ 0.017	0.000 $\pm$ 0.000	0.710 $\pm$ 0.030
Finetune	0.037 $\pm$ 0.011	-0.265 $\pm$ 0.028	0.427 $\pm$ 0.033
Average	0.013 $\pm$ 0.007	-0.530 $\pm$ 0.024	0.070 $\pm$ 0.022
PackNet	0.491 $\pm$ 0.025	-0.194 $\pm$ 0.018	0.000 $\pm$ 0.000
ProgressiveNet	0.619 $\pm$ 0.025	-0.111 $\pm$ 0.016	0.000 $\pm$ 0.000
CompoNet	0.612 $\pm$ 0.025	-0.089 $\pm$ 0.016	0.000 $\pm$ 0.000

Appendix

Table 11: Results on Meta-World averaged over the three sequences with more baselines on Average Performance ( Ave. Perf ), Forward Transfer ( FT ), and Forgetting. Results are presented as averages and standard errors across 10 seeds. The best results are highlighted in bold, and the second best are underlined.

\uparrow

denotes a positive metric (more is better), while

\downarrow

is a negative one (less is better). Reset is the baseline for evaluating FT.

Panel of Figure 9 — Figure 9: (Left) Performance profile of the fast learner across tasks with more baselines , where the y-axis shows the proportion of tasks that achieve a success rate greater than or equal to the x-axis value. (Right) Average performance over time by evaluating the average success rates in the past tasks. Each Row represents the result for one sequence of tasks. The last row shows the average performance of the three sequences.

Panel of Figure 10 — Figure 10: Learning curves of the fast learner on the Meta-World benchmark with more baselines . The x-axis shows the total number of environment interactions, and the y-axis indicates the success rate. Each Row represents the result for one sequence of tasks. The last row shows the average performance of the three sequences.

Methods	Avg. Perf $\uparrow$	FT $\uparrow$	Forgetting $\downarrow$
Reset	0.020 $\pm$ 0.014	0.000 $\pm$ 0.000	0.370 $\pm$ 0.049
Finetune	0.020 $\pm$ 0.014	-0.123 $\pm$ 0.028	0.240 $\pm$ 0.047
Average	0.010 $\pm$ 0.01	-0.288 $\pm$ 0.039	0.010 $\pm$ 0.017
PackNet	0.292 $\pm$ 0.071	-0.086 $\pm$ 0.019	0.000 $\pm$ 0.000
ProgressiveNet	0.325 $\pm$ 0.074	-0.034 $\pm$ 0.016	0.000 $\pm$ 0.000
CompoNet	0.208 $\pm$ 0.063	-0.129 $\pm$ 0.029	0.000 $\pm$ 0.000

Appendix

Table 12: Results on the sequence of tasks in CW10 ( Wolczyk et al., 2021 ) on Average Performance ( Ave. Perf ), Forward Transfer ( FT ), and Forgetting. Results are presented as averages and standard errors across 10 seeds. The best results are highlighted in bold, and the second best are underlined.

\uparrow

denotes a positive metric (more is better), while

\downarrow

is a negative one (less is better). Reset is the baseline for evaluating FT.

Panel of Figure 11 — Figure 11: Results on the CW10 ( Wolczyk et al., 2021 ) sequence of tasks. (Left) Performance profile of the fast learner across tasks, where the y-axis shows the proportion of tasks that achieve a success rate greater than or equal to the x-axis value. (Right) Average performance over time by evaluating the average success rates in the past tasks.

Panel of Figure 12 — Figure 12: Learning curves of the fast learner on the CW10 ( Wolczyk et al., 2021 ) sequence of tasks. The x-axis shows the total number of environment interactions, and the y-axis indicates the success rate. Each Row represents the result for one sequence of tasks. The last row shows the average performance of the three sequences.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

When Does Continual Learning Require Learning
Jul 8, 2026Anne Harrington, Nayan Saxena, Michael Murphy +7Continual Learning for LLMs Concept Drift
1UC Berkeley · 2Independent · 3Capital Fund Management
Reliable Knowledge Acquisition for Continual Learning Agents
Oct 15, 2026Yixu Huang, Jiaqi W. MaContinual Learning for LLM Agents Continual Learning
University of California, Davis · University of Illinois Urbana-Champaign
ARROW: Augmented Replay for RObust World models
Mar 12, 2026Abdulaziz Alyahya, Abdallah Al Siyabi, Markus R. Ernst +3Continual RL Replay-Based Continual Learning
Department of Information Systems, College of Computer and Information Sciences, Imam Mohammad Ibn Saud Islamic University (IMSIU) · Department of Data Science & AI, Faculty of Information Technology, Monash University · School of Computer Science and Engineering, Faculty of Engineering, University of New South Wales, Sydney

Jul 8, 2026J/K move · Enter open · S save

When Does Continual Learning Require Learning

Anne Harrington, Nayan Saxena, Michael Murphy, Anastasia Borovykh+6

1UC Berkeley · 2Independent · 3Capital Fund Management

Continual Learning for LLMs Concept Drift Domain-Incremental Learning

Open paper AskPDF arXiv

As large language models (LLMs) become increasingly capable, the next question is how can we enable models to continually learn? Today, the field largely frames this as a problem of context management and mitigating forgetting. We argue this framing is incomplete: continual learning is fundamentally about increasing model competence as the world changes. We disentangle this change along two axes -- space, where the model encounters new domains, and time, where the underlying data drifts under a fixed task. This framing lets us study continual learning under realistic conditions: new domains arrive over time, facts drift past their training cutoff, and agentic interactions accumulate state across episodes. To evaluate methods under this setting, we recast widely used LLM benchmarks as sequential problems and introduce a single mechanism-agnostic protocol that compares prompt-based methods (GEPA, ACE), supervised learning (SFT, SDFT), reinforcement learning (GRPO, SDPO), and context compression (Cartridges, In-place TTT). Prompt-based methods fit each new stage quickly but degrade on future tasks. Distillation-based methods accumulate knowledge stably but struggle to update outdated facts. Context compression improves efficiency without substantially improving the ability to learn new tasks. Online reinforcement learning adapts most effectively to knowledge updates but remains sensitive to noisy reward signals. Overall, our results suggest that continual learning is not a single capability: different patterns of environmental change require fundamentally different update behaviors, determining when adaptation must be learned inside model weights and when it can be achieved through external scaffolding. We hope that understanding where each method succeeds and fails will guide the design of stronger continual learning systems.

Abstract

Reinforcement Learning

Explore similar work

When Does Continual Learning Require Learning

Reliable Knowledge Acquisition for Continual Learning Agents

ARROW: Augmented Replay for RObust World models