Engram Neuron Inspired Adaptive Memorization for Lifelong Learning

Suiyang Jiang; Bomin Wang; Hangqi Zhou; Jianfeng Feng; Xiahai Zhuang

cs.CVOct 10, 2026

Engram Neuron Inspired Adaptive Memorization for Lifelong Learning

Authors: Suiyang Jiang, Bomin Wang, Hangqi Zhou, Jianfeng Feng, Xiahai Zhuang

Organizations: School of Data Science, Fudan University, 220 Handan Road, Shanghai, 200433, China · Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, 220 Handan Road, Shanghai, 200433, China

Abstract

Lifelong learning enables artificial systems to adapt to changing environments. To achieve this, an effective artificial learner should dynamically reorganize past and new experiences, maintaining stability while enabling flexible knowledge integration. However, existing efforts primarily mitigate catastrophic forgetting, lacking the ability to adaptively update memory akin to biological systems. Here, by integrating the full memory life cycle of engram neurons into artificial learners, we propose a framework that constructs a structured memory from gradient representations, which regulates parameter updates and guides memory management. Through comprehensive evaluations on four benchmarks, our approach shows promising lifelong learning capability, maintaining stability over long task sequences and supporting flexible adaptation. Beyond the performance gains, our approach offers a computational perspective on robust and flexible lifelong learning, echoing mechanisms observed in biological memory systems.

Figures & tables

Panel of Figure 1 — Figure 1: Illustration of the problem in lifelong learning (LL) and how biological intelligence solves this problem: (a) Biological memory adaptively balances old and new contexts through the lifecycle of engrams. Encoding involves CREB-regulated excitability that biases the allocation of neurons into sparse memory engrams. Synaptic plasticity underlies both consolidation and forgetting by stabilizing relevant connections and weakening or reorganizing others. Retrieval follows the encoding specificity principle, whereby cues overlapping with the original memory encoding trace preferentially reactivate the corresponding synaptic connection. CREB: cAMP response element-binding protein. (b) Plasticity dominated lifelong learning methods face severe catastrophic forgetting of previous contexts, whereas stability dominated learning restricts updating of new knowledge for new contexts. By contrast, stability-plasticity balanced LL methods adapt well to new knowledge and consolidate important prior knowledge.

Panel of Figure 2 — Figure 2: Overview of the proposed AdaMLearn framework: (a) Overall workflow of the proposed AdaMLearn framework across the lifelong learning stream. (b) Diagram of how the model retrieves memory during training. The initial gradient obtained from backpropagation (treated as the cue) is orthogonally projected (matching) onto the old memory (treated as the trace). The residual gradient component lies in the new memory, enabling the model to learn new knowledge without compromising previously acquired knowledge. (c) Diagram of the encoding process. we design a core set optimization procedure. In this procedure, representative samples are chosen so that the loss landscape they produce closely matches that of the full dataset. This ensures that the most important memories of the context are preserved. (d) Illustration of the proposed process in gradient space. For input data from Context B, AdaMLearn projects features onto both the old (Context A) and new (Context B) gradient subspaces, applies SVD, then consolidates high-singular-value components and forgets low-singular-value ones.

Panel of Figure 3 — Figure 3: Overall performance in visual classification tasks: (a) Lifelong learning performance is evaluated from multiple perspectives, including sensitivity to task ordering, cross-domain transferability, and robustness to higher-resolution inputs. Colors represent the category of different LL methods. All results are averaged over five runs with different random seeds. (b) The learner’s performance on all previously learned tasks at each stage of the full task sequence is reported, averaged over five runs. (c) Plasticity is assessed by measuring the test accuracy on a new task immediately after its initial learning, with experiments conducted on CIFAR-100. (d) Comparison with other replay-based lifelong learning methods shows that our approach maintains strong performance across a wide range of buffer size settings. Notably, even when the number of available samples is small, it does not suffer from catastrophic forgetting.

Panel of Figure 4 — Figure 4: Achieving lifelong adaptive memorization within the framework of synaptic plasticity: (a) Integration of consolidation and forgetting within the neural synaptic plasticity. The color and intensity of nodes reflect the activation level of neurons. Learning context B introduces new potential synapses that partially weaken previously consolidated connections from context A. (b) Ablation experiments evaluate average accuracy and backward transfer. For AdaMLearn (M1), although synaptic remodeling increases the degree of forgetting, it leads to an overall improvement in average accuracy. All results are averaged over five runs. (c) Effects of consolidation and forgetting on memory efficiency. Without consolidation and forgetting (M4), memory grows rapidly and eventually saturates the budget. For AdaMLearn (M1), memory usage remains low and stable across tasks, adaptively converging to compact memory budgets that reflect efficient and context-aware resource allocation in CIFAR scenarios. The memory usage is computed at the level of the final linear hidden layer of the ANN. A value of 100% indicates that subsequent learning does not alter the parameters of this layer.

Panel of Figure 5 — Figure 5: Implementing the selective encoding of contextual memory: (a) Effects of the core set selection procedure. Varying the fraction of core set selection demonstrates that our encoding process yields substantial benefits across a wide range of selection ratios. The baseline (Random) means selecting samples randomly. (b) The t-SNE plot of the CIFAR-100 dataset illustrates the representative samples chosen for each class after core set optimization. Samples selected as core set are marked with distinctive shapes, whereas the unselected ones appear as small circular points.

Figure 6: Applying retrieval during LL: (a) Effects of the retrieval process on preserving prior knowledge. Without any constraint (SGD), the model rapidly forgets previously acquired knowledge when learning a new task. When perturbations are introduced during retrieval (Disturb), causing the parameter update direction to deviate from being strictly orthogonal to the old memory, the stability degrades.

Layer	Channels	Kernel	Stride	Padding	Dropout
Input	$3$	-	-	-	-
Conv1 + BN	$128$	$4\times 4$	$1$	-	0.2
MaxPool1	$128$	$2\times 2$	$2$	-	-
Conv2 + BN	$256$	$3\times 3$	$1$	-	0.2
MaxPool2	$256$	$2\times 2$	$2$	-	-
Conv3 + BN	$512$	$3\times 3$	$1$	-	0.5

Supplementary Table 1: Neural network architecture (AlexNet) for CIFAR-100.

Layer	Channels	Kernel	Stride	Padding	Dropout
Input	$3$	-	-	-	-
Conv1 + ReLU + LRN	$40$	$5\times 5$	$1$	$2$	-
MaxPool1	$40$	$3\times 3$	$2$	$1$	-
Conv2 + ReLU + LRN	$100$	$5\times 5$	$1$	$2$	-
MaxPool2	$100$	$3\times 3$	$2$	$1$	-
Flatten	$6400$	-	-	-	-

Supplementary Table 2: Neural network architecture (LeNet) for CIFAR-superclass.

Layer	Channels	Kernel	Stride	Padding	Dropout
Conv1 + BN + ReLU	$C_{in}\rightarrow C_{out}$	$3\times 3$	$s$	1	-
Conv2 + BN	$C_{out}\rightarrow C_{out}$	$3\times 3$	1	1	-
Shortcut	$C_{in}\rightarrow C_{out}$	$1\times 1$ (if needed)	$s$	0	-
ReLU (post-addition)	-	-	-	-	-

Supplementary Table 3: Structure of BasicBlock used in ResNet18.

Layer	Channels	Kernel	Stride	Padding	Dropout
Input	$C$	-	-	-	-
Conv1 + BN + ReLU	64	$3\times 3$	1	1	-
Stage 1: BasicBlock × 2
Block1	64	$3\times 3$	1	1	-
Block2	64	$3\times 3$	1	1	-
Stage 2: BasicBlock × 2

Supplementary Table 4: Neural network architecture (ResNet18(32×32)) for FIVE benchmark.

Layer	Channels	Kernel	Stride	Padding	Dropout
Input	$3$	-	-	-	-
Conv1 + BN + ReLU	64	$3\times 3$	1	1	-
Stage 1: BasicBlock × 2
Block1	64	$3\times 3$	1	1	-
Block2	64	$3\times 3$	1	1	-
Stage 2: BasicBlock × 2

Supplementary Table 5: Neural network architecture (ResNet18(84×84)) for Mini-imagenet.

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 304 papers

Tell Me What To Learn: Generalizing Neural Memory to be Controllable in Natural Language
Feb 26, 2026Max S. Bennett, Thomas P. Zollo, Richard ZemelMemory-Augmented Neural Networks Lifelong Learning Agents
Columbia University
LifeMem: Enabling Lifelong Experience Reuse for LLM Agents
Sep 11, 2026Yuli Qiu, Yutong Li, Wei Su +5Continual Learning for LLM Agents LLM Agent Skill Learning
School of Computer Science and Technology, Beijing Institute of Technology · School of Computer Science and Engineering, Beihang University · Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology +1
NeuroSynth: A Biologically Inspired Continual Reinforcement Learning Architecture for Mitigating Catastrophic Forgetting
Jul 24, 2026Yash KiniContinual Learning Continual RL
James Madison High School Vienna, Virginia, USA

Feb 26, 2026J/K move · Enter open · S save

Tell Me What To Learn: Generalizing Neural Memory to be Controllable in Natural Language

Max S. Bennett, Thomas P. Zollo, Richard Zemel

Columbia University

Memory-Augmented Neural Networks Lifelong Learning Agents Continual Learning

Open paper AskPDF arXiv

Modern machine learning models are deployed in diverse, non-stationary environments where they must continually adapt to new tasks and evolving knowledge. Continual fine-tuning and in-context learning are costly and brittle, whereas neural memory methods promise lightweight updates with minimal forgetting. However, existing neural memory models typically assume a single fixed objective and homogeneous information streams, leaving users with no control over what the model remembers or ignores over time. To address this challenge, we propose a generalized neural memory system that performs flexible updates based on learning instructions specified in natural language. Our approach enables adaptive agents to learn selectively from heterogeneous information sources, supporting settings, such as healthcare and customer service, where fixed-objective memory updates are insufficient.

Abstract

Continual Learning

Explore similar work

Tell Me What To Learn: Generalizing Neural Memory to be Controllable in Natural Language

LifeMem: Enabling Lifelong Experience Reuse for LLM Agents

NeuroSynth: A Biologically Inspired Continual Reinforcement Learning Architecture for Mitigating Catastrophic Forgetting