Continual reinforcement learning requires adaptation to changing conditions while preserving performance under previously encountered dynamics. It was investigated whether specialists generated by evolutionary search can provide a reusable memory when combined with environment detection and failure-driven search. The Evolutionary Neuroplastic Agent (ENA) evolves small neural networks without gradients, archives selected policies by detected dynamic regime, and retrieves them through Bayesian policy reuse. The architecture was evaluated in non-stationary CartPole and HalfCheetah environments. In the 30-replicate CartPole ablation, the complete architecture achieved a mean success rate of 83.2% across the five test environments. Among variants that learned the harder CartPole environments, the complete architecture showed the most consistent retention. A selection ablation also showed that the retrieval rule affected performance. In the evaluated sequences, ENA adapted more slowly than PPO on CartPole and achieved lower returns than SAC on HalfCheetah. The evidence concerns the complete architecture and does not isolate gradient-free optimization as the cause of retention or establish superiority over other continual-learning methods.
Figures & tables
Method
Memory
Online detection
Recurrence
Gradient-free
Probing
EWC [ 3 ]
regularized shared weights
–
–
–
–
CLEAR [ 4 ]
replay into shared weights
–
–
–
–
OWL [ 5 ]
shared extractor (EWC), one head per task
–
bandit at test
–
–
RL-CD [ 6 ]
model and policy per context
✓
✓
tabular
–
MBCD [ 7 ]
model and policy per context
✓
✓
–
–
SWOKS [ 8 ]
policy per context
✓
✓
–
–
TABLE I: How Related Methods Keep and Recall Earlier Behavior
Variant
Plasticity
HoF
Detection
ENA-NoSkill
–
–
–
ENA-Plasticity
✓
–
–
ENA-HoF
–
✓
–
ENA-HoF+EnvDetection
–
✓
✓
ENA-Plasticity+HoF
✓
✓
–
ENA-Combined
✓
✓
✓
TABLE IV: ENA Variants in the Skill Ablation
RQ
Evidence
Section
RQ1
skill ablation (reliability and retention); forgetting of single-policy learners
V-A , V-E , V-G
RQ1a
ablation: Hall of Fame with and without detection; selection ablation
V-A , V-D
RQ1b
ablation: candidate steps with and without the archive
V-A
RQ1c
recognition per phase (both domains); detection benchmark (HalfCheetah)
A-C , V-F
RQ2
ablation; compute study; controller study
V-A , V-B , V-C
TABLE V: Research Questions and the Evidence for Each
Fig. 2: Test reliability of each ENA variant; each dot is one of the 30 replicates.
Fig. 3: Test reliability on CartPole across 50 replicates, under equal budgets of environment steps; each dot is one replicate.
Fig. 4: Left-behind retention on HalfCheetah (20 replicates); each dot is one replicate. The dotted line marks 100%.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. 5: Frozen test on CartPole: mean return per episode across 50 replicates (line) and the middle 50% of replicates (band). Dashed lines mark environment switches; Neptune and Brown-dwarf were never seen in training.
Fig. 6: Frozen test on HalfCheetah: mean return per episode across 20 replicates (line) and the middle 50% of replicates (band). Dashed vertical lines mark environment switches; Ice-Floor and High-Wind were never seen in training.
Fig. 7: Training return on HalfCheetah against environment steps (20 replicates; line is the mean, band the middle 50% of replicates). Dashed vertical lines mark the switches between the three training environments (Earth-Standard → Heavy-Rough → Mars-Slippery).
Despite many studies about causes and remedies of plasticity loss in Reinforcement Learning (RL) under continual task changes, no RL method has yet consistently achieved a good balance between adaptation and forgetting. Here we turn to an alternative optimization paradigm, neuroevolution (NE): algorithms that search directly in weight space through mutation and selection over a population of neural networks. Across a wide array of environments and environmental changes, with policies ranging from a few hundred parameters to million-parameter networks, we compare evolution strategies (ES) and genetic algorithms (GAs) against state-of-the-art continual RL variants and population-based RL. ES most consistently achieves a good stability-plasticity trade-off, while the GA is the most plastic method but forgets more than ES. To explain this, we study the return landscape around each method's solutions. ES finds the widest neighborhoods, i.e.\ regions of weight space in which perturbed policies still solve the task, and the size of the overlap between the neighborhoods of consecutive tasks correlates with a method's stability-plasticity trade-off. Rewarding behavioral diversity in a GA through novelty search makes the population even more plastic, at the cost of forgetting. Finally, symptoms of plasticity loss commonly reported in RL do not transfer to NE. Overall, these results establish NE as a competitive alternative to RL under continual task changes, and suggest that training under perturbations in weight space may be a useful mechanism for continual learning more broadly.
Eleni Nisioti, Andrea Cossu, Kathrin Korte +1
IT University of Copenhagen · University of Pisa · Sakana AI
Continual reinforcement learning must balance retention with adaptation, yet many methods still rely on single-model preservation, committing to one evolving policy as the main reusable solution across tasks. Even when a previously successful policy is retained, it may no longer provide a reliable starting point for rapid adaptation after interference, reflecting a form of loss of plasticity that single-policy preservation cannot address. Inspired by quality-diversity methods, we introduce TeLAPA (Transfer-Enabled Latent-Aligned Policy Archives), a continual RL framework that organizes behaviorally diverse policy neighborhoods into per-task archives and maintains a shared latent space so that archived policies remain comparable and reusable under non-stationary drift. This perspective shifts continual RL from retaining isolated solutions to maintaining skill-aligned neighborhoods with competent and behaviorally related policies that support future relearning. In our MiniGrid CL setting, TeLAPA learns more tasks successfully, recovers competence faster on revisited tasks after interference, and retains higher performance across a sequence of tasks. Our analyses show that source-optimal policies are often not transfer-optimal, even within a local competent neighborhood, and that effective reuse depends on retaining and selecting among multiple nearby alternatives rather than collapsing them to one representative. Together, these results reframe continual RL around reusable and competent policy neighborhoods, providing a route beyond single-model preservation toward more plastic lifelong agents.
Lute Lillo, Nick Cheney
Department of Computer Science, University of Vermont, USA
Artificial Intelligence (AI) systems often perform well on isolated tasks but struggle under continual learning conditions, where training on new tasks can overwrite previously acquired knowledge, a failure mode known as catastrophic forgetting. Biological learning systems reduce this interference through complementary memory processes involving rapid hippocampal encoding and slower cortical consolidation. This study introduces NeuroSynth, a brain-inspired continual reinforcement learning architecture designed to mitigate catastrophic forgetting through a dual-pathway consolidation mechanism. NeuroSynth separates rapid task acquisition from long-term retention using distinct "plan" and "habit" pathways combined with replay and knowledge distillation. NeuroSynth was evaluated against Proximal Policy Optimization (PPO) and Elastic Weight Consolidation (EWC) across three sequential navigation tasks with changing goal locations in a non-revisitation continual learning setting. Across six independent seeds, NeuroSynth preserved substantially more early-task knowledge than PPO after sequential training, achieving 18.00% Task A success rate compared to 0.33% for PPO (p = 0.014929, Cohen's d = 1.49) and 35.33% Task B success rate compared to 0.00% for PPO (p = 0.002376, Cohen's d = 2.31). NeuroSynth also demonstrated higher final Task C performance than EWC, achieving 9.00% compared to 2.00% (p = 0.226643, Cohen's d = 0.56), indicating a moderate but not statistically significant advantage. These findings suggest that biologically inspired consolidation mechanisms may improve the stability-plasticity balance in continual reinforcement learning systems.