Stochastic Engrams for Efficient Continual Learning

Isabelle Aguilar; Luis Fernando Herbozo Contreras; Omid Kavehei

cs.LGMar 27, 2025

Stochastic Engrams for Efficient Continual Learning

Authors: Isabelle Aguilar, Luis Fernando Herbozo Contreras, Omid Kavehei

Organizations: School of Biomedical Engineering, The University of Sydney, NSW, 2006, Australia

Abstract

The ability to learn continuously in artificial neural networks (ANNs) is often limited by catastrophic forgetting, a phenomenon in which new knowledge becomes dominant. By taking mechanisms of memory encoding in neuroscience (i.e., engrams) as inspiration, we propose a novel approach that integrates stochastically-activated engrams as a gating mechanism for metaplastic binarized neural networks (mBNNs). This method leverages the computational efficiency of mBNNs combined with the robustness of probabilistic memory traces to mitigate forgetting and maintain the model's reliability. Previously validated metaplastic optimization techniques have been incorporated to further enhance synaptic stability. Compared to baseline binarized models and benchmark fully connected continual learning approaches, our method is the only strategy capable of achieving average accuracies over 70% in both class-incremental and domain-incremental MNIST benchmarks, matching full-precision state-of-the-art methods. Furthermore, we achieve a significant reduction in peak GPU and RAM usage, under 5% and 20%, respectively, as well as an ~8x reduction in memory footprint compared to full precision counterparts. Our findings demonstrate (A) an improved stability vs. plasticity trade-off, (B) reduced memory intensiveness, and (C) enhanced performance in binarized architectures. By uniting principles of neuroscience and efficient computing, we offer new insights into the design of scalable and robust deep learning systems.

Figures & tables

Figure 1: Continual learning mechanisms in biological and artificial networks. (a) In the brain, varying experiences and stimuli are streamed into the system over time, (b) forming different engrams (green and blue) that encode and consolidate unique information. (c) As incoming memories flood the network, metaplasticity and synaptic plasticity allow the brain to adapt to additional knowledge. This process repeats throughout life, and our brains continually learn. \label fig:fig1

Figure 2: Engram block architecture. The engram block processes the input data block through a two-layer encoder to compress features into a one-layer latent space. This compressed representation is then passed through a linear layer and subjected to stochastic activation, yielding a discrete gating vector. Finally, this gating vector modulates the mBNN via element-wise multiplication across its hidden layers, dynamically routing the forward pass. \label fig:fig2

Model	Experiment	ACC (%) $\uparrow$	w ACC ∗ (%) $\uparrow$	FWT (%) $\uparrow$	BWT (%) $\uparrow$	REM (%) $\uparrow$
Vanilla BNN	CORe50-NI	$\mathbf{20.94}\pm{\scriptstyle 0.23}$	$11.94\pm{\scriptstyle 0.42}$	$14.33\pm{\scriptstyle 0.37}$	$-82.43\pm{\scriptstyle 02.52}$	$17.57\pm{\scriptstyle 02.52}$
	Split MNIST	$61.41\pm{\scriptstyle 0.39}$	$58.71\pm{\scriptstyle 0.40}$	$53.77\pm{\scriptstyle 0.93}$	$-42.87\pm{\scriptstyle 28.67}$	$57.13\pm{\scriptstyle 28.67}$
engramBNN	CORe50-NI	$17.88\pm{\scriptstyle 0.13}$	$\mathbf{19.06}\pm{\scriptstyle 0.21}$	$\mathbf{16.49}\pm{\scriptstyle 0.37}$	$-17.40\pm{\scriptstyle 15.98}$	$82.60\pm{\scriptstyle 15.98}$
	Split MNIST	$\mathbf{75.34}\pm{\scriptstyle 1.83}$	$\mathbf{79.47}\pm{\scriptstyle 2.61}$	$52.20\pm{\scriptstyle 1.03}$	$-25.58\pm{\scriptstyle 18.06}$	$74.42\pm{\scriptstyle 18.06}$
EWC	CORe50-NI	$19.82\pm{\scriptstyle 0.46}$	$11.43\pm{\scriptstyle 0.59}$	$12.63\pm{\scriptstyle 0.59}$	$\mathbf{2.23}\pm{\scriptstyle 1.11}$	$\mathbf{100.00}\pm{\scriptstyle 1.11}$
	Split MNIST	$66.19\pm{\scriptstyle 0.98}$	$64.12\pm{\scriptstyle 1.29}$	$53.25\pm{\scriptstyle 2.40}$	$-41.11\pm{\scriptstyle 27.02}$	$58.89\pm{\scriptstyle 27.02}$

Table 1: Domain-incremental experiment metrics on the CORe50 and MNIST datasets. Average values are reported in percentages along with their respective standard deviations. \label tab:tab1

Figure 3: Performance and resource efficiency comparison of engramBNN against baseline CL models. On the Class-IL Split MNIST benchmark, we present a (a) radar chart illustrating the final test accuracy on each of the five individual tasks after sequentially learning the entire task sequence, highlighting retention across early tasks. (b) Trade-off between final overall average test accuracy and computational overhead, quantified by peak GPU memory utilization in percent Error bars denote standard deviation. \label fig:fig3

Model	ACC (%) $\uparrow$	w ACC ∗ (%) $\uparrow$	FWT (%) $\uparrow$	BWT (%) $\uparrow$	REM (%) $\uparrow$
Vanilla BNN	$15.83\pm{\scriptstyle 0.32}$	$8.18\pm{\scriptstyle 0.19}$	$5.23\pm{\scriptstyle 3.31}$	$-70.33\pm{\scriptstyle 3.58}$	$29.67\pm{\scriptstyle 3.58}$
engramBNN	$\mathbf{72.65\pm{\scriptstyle 1.42}}$	$\mathbf{76.31\pm{\scriptstyle 2.30}}$	$0.55\pm{\scriptstyle 0.13}$	$-33.10\pm{\scriptstyle 11.96}$	$66.90\pm{\scriptstyle 11.96}$
EWC	$20.05\pm{\scriptstyle 0.77}$	$10.99\pm{\scriptstyle 0.59}$	$0.04\pm{\scriptstyle 1.87}$	$-99.25\pm{\scriptstyle 1.04}$	$0.75\pm{\scriptstyle 1.04}$
SI	$9.76\pm{\scriptstyle 1.69}$	$9.42\pm{\scriptstyle 1.48}$	$\mathbf{9.29\pm{\scriptstyle 2.88}}$	$\mathbf{-0.22\pm{\scriptstyle 0.77}}$	$\mathbf{99.78\pm{\scriptstyle 0.77}}$
LwF	$20.12\pm{\scriptstyle 3.97}$	$12.03\pm{\scriptstyle 4.06}$	$4.08\pm{\scriptstyle 4.48}$	$-64.77\pm{\scriptstyle 3.96}$	$35.23\pm{\scriptstyle 3.96}$
XdG	$19.74\pm{\scriptstyle 2.45}$	$13.97\pm{\scriptstyle 2.57}$	$4.64\pm{\scriptstyle 3.97}$	$-52.38\pm{\scriptstyle 3.94}$	$47.62\pm{\scriptstyle 3.94}$

Table 2: Class-incremental Split MNIST continual learning metrics. Average results are reported along with their standard deviations. \label tab:tab2

Model	Task-Class Split	ACC (%) $\uparrow$	w ACC ∗ (%) $\uparrow$	FWT (%) $\uparrow$	BWT (%) $\uparrow$	REM (%) $\uparrow$
Vanilla BNN	$20-5$	$4.05\pm{\scriptstyle 0.25}$	$0.50\pm{\scriptstyle 0.39}$	$0.02\pm{\scriptstyle 0.47}$	$-50.13\pm{\scriptstyle 7.30}$	$49.87\pm{\scriptstyle 7.30}$
	$10-10$	$5.90\pm{\scriptstyle 0.17}$	$2.74\pm{\scriptstyle 0.43}$	$0.01\pm{\scriptstyle 0.55}$	$-47.90\pm{\scriptstyle 4.40}$	$52.10\pm{\scriptstyle 4.40}$
engramBNN	$20-5$	$\mathbf{8.12}\pm{\scriptstyle 0.25}$	$\mathbf{3.74}\pm{\scriptstyle 0.43}$	$\mathbf{0.61}\pm{\scriptstyle 0.40}$	$-16.89\pm{\scriptstyle 11.24}$	$83.11\pm{\scriptstyle 11.24}$
	$10-10$	$\mathbf{9.66}\pm{\scriptstyle 0.24}$	$\mathbf{4.58}\pm{\scriptstyle 0.55}$	$\mathbf{0.58}\pm{\scriptstyle 0.46}$	$-16.95\pm{\scriptstyle 7.05}$	$83.05\pm{\scriptstyle 7.05}$
EWC	$20-5$	$3.30\pm{\scriptstyle 0.10}$	$0.00\pm{\scriptstyle 0.01}$	$0.00\pm{\scriptstyle 0.38}$	$-61.60\pm{\scriptstyle 6.63}$	$38.40\pm{\scriptstyle 6.63}$
	$10-10$	$5.39\pm{\scriptstyle 0.11}$	$0.00\pm{\scriptstyle 0.00}$	$0.00\pm{\scriptstyle 0.50}$	$-48.72\pm{\scriptstyle 4.33}$	$51.28\pm{\scriptstyle 4.33}$

Table 3: Class-incremental Split CIFAR-100 continual learning metrics. Average results are reported along with their standard deviations. \label tab:tab3

Model	Accuracy ( $\%$ ) $\uparrow$	Sparsity $\uparrow$	Average Mask Stability $\uparrow$
engramBNN	$\mathbf{72.06\pm{\scriptstyle 2.09}}$	$\mathbf{0.80\pm{\scriptstyle 0.00}}$	$\mathbf{0.70\pm{\scriptstyle 0.00}}$
m BNN	$30.20\pm{\scriptstyle 2.33}$	$0.00\pm{\scriptstyle 0.00}$	N/A
BNN with Dropout	$20.94\pm{\scriptstyle 3.88}$	$0.00\pm{\scriptstyle 0.00}$	N/A

Table 4: Ablation Metrics . All metrics are averaged over 6 runs. Best values are reported in bold. \label tab:tab4

Figure 4: Task Jaccard Overlap of Neuron Masks . Jaccard similarity between the sets of active neurons for task i and task j for a BNN with (a) Dropout, (b) metaplasticity, and (c) an engram gating module. Image is best viewed in color. \label fig:fig4

Figure 5: Analysis of the engram block on the Split MNIST task . After training on all sequential class-incremental Split MNIST tasks, t-SNE feature visualization and corresponding silhouette scores, $S$ , were computed for extracted features from the last hidden layer of the (a) Vanilla BNN, (b) m BNN, and (c) engramBNN. Each plot illustrates how samples taken from each task are clustered and separated in a 2D space, and each color is a different class. On the same task, representational dissimilarity matrix and dissimilarity score, $\bar{d}$ , were visualized for the (d) Vanilla BNN and (e) engramBNN. Image is best viewed in color. \label fig:fig5

Model	Peak GPU Usage (%) $\downarrow$	Peak RAM Usage (%) $\downarrow$
Vanilla BNN	$7.95$	$21.18$
engramBNN	$\mathbf{4.51}$	$19.51$
EWC	$92.74$	$21.79$
SI	$57.53$	$6.66$
LwF	$12.39$	$25.71$
XdG	$8.47$	$22.30$

Table 5: Resource Utilization Metrics. Measurements were taken from the Weights and Biases (WandB [ 14 ] ) system metric documentation, which is logged every 10 seconds. Metrics were measured in PyTorch, and no specialized inference engine was used. The total GPU memory is approximately 17 GB. Lowest values are bolded. \label tab:tab5

Model	Parameter Count $\downarrow$	Memory Size (MiB) $\downarrow$
engramBNN	$6,429,877$	$3.02$
Full Precision Counterpart	$6,429,877$	$24.53$
m BNN	$5,820,419$	$\mathbf{0.69}$
BI-Replay	$6,892,882$	$26.30$
Stochastic Synapses	$17,461,248$	$66.61$
PNN ∗	$113,003,570$	$430.97$

Table 6: Memory overhead metrics. The number of parameters and the memory size are taken into account during inference. Memory size is recorded in MiB. Lowest values are reported in bold. \label tab:tab6

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Model	ACC (%) $\uparrow$	w ACC ∗ (%) $\uparrow$	FWT (%) $\uparrow$	BWT (%) $\uparrow$	REM (%) $\uparrow$
engramBNN	$52.94\pm{\scriptstyle 0.76}$	$36.19\pm{\scriptstyle 1.15}$	$\mathbf{11.27\pm{\scriptstyle 0.84}}$	$-45.55\pm{\scriptstyle 20.00}$	$54.45\pm{\scriptstyle 20.00}$
Vanilla BNN	$45.75\pm{\scriptstyle 0.99}$	$24.90\pm{\scriptstyle 1.51}$	$10.06\pm{\scriptstyle 1.01}$	$-53.81\pm{\scriptstyle 24.66}$	$46.19\pm{\scriptstyle 24.66}$
LwF	$46.82\pm{\scriptstyle 0.88}$	$25.13\pm{\scriptstyle 1.86}$	$10.49\pm{\scriptstyle 0.89}$	$-52.50\pm{\scriptstyle 25.62}$	$47.50\pm{\scriptstyle 25.62}$
XdG	$56.98\pm{\scriptstyle 1.08}$	$40.52\pm{\scriptstyle 2.31}$	$10.61\pm{\scriptstyle 1.02}$	$-33.63\pm{\scriptstyle 19.54}$	$66.37\pm{\scriptstyle 19.54}$
m BNN	$49.62\pm{\scriptstyle 1.04}$	$30.22\pm{\scriptstyle 1.52}$	$9.92\pm{\scriptstyle 0.99}$	$-48.85\pm{\scriptstyle 24.20}$	$51.15\pm{\scriptstyle 24.20}$
SDMLP	$\mathbf{63.27\pm{\scriptstyle 0.91}}$	$\mathbf{56.47\pm{\scriptstyle 1.58}}$	$9.58\pm{\scriptstyle 1.13}$	$\mathbf{-14.87\pm{\scriptstyle 8.90}}$	$\mathbf{85.13\pm{\scriptstyle 8.90}}$

Appendix

Table 7: Permuted MNIST continual learning metrics. All values are averages plus or minus their standard deviations. \label tab:tab7

Explore similar work

Mistake gating leads to energy and memory efficient continual learning
Apr 15, 2026Aaron Pache, Mark CW van RossumContinual Learning Plasticity
School of Mathematical Sciences University of Nottingham, Nottingham NG7 2RD, United Kingdom · School of Psychology University of Nottingham, Nottingham NG7 2RD, United Kingdom
Shapley Neuron Values for Continual Learning: Which Neurons Matter Most?
May 15, 2026Mohammad Ali Vahedifar, Abhisek Ray, Qi ZhangReplay-Based Continual Learning Continual Learning
DIGIT and Department of Electrical and Computer Engineering, Aarhus University, Denmark
NeuMoSync: End-to-End Neuromodulatory Control for Plasticity and Adaptability in Continual Learning
Aug 5, 2026Seyed Roozbeh Razavi Rohani, Khashayar Khajavi, Wesley Chung +2Replay-Based Continual Learning Plasticity
Department of Computer Science Simon Fraser University · Department of Computer Science McGill Univeristy, Mila - Quebec AI Institute

Apr 15, 2026, cs.AIJ/K move · Enter open · S save

Mistake gating leads to energy and memory efficient continual learning

Aaron Pache, Mark CW van Rossum

School of Mathematical Sciences University of Nottingham, Nottingham NG7 2RD, United Kingdom · School of Psychology University of Nottingham, Nottingham NG7 2RD, United Kingdom

Continual Learning Plasticity Synaptic Plasticity Singular Learning Theory Gating

Open paper AskPDF arXiv

Synaptic plasticity is metabolically expensive, yet animals continuously update their internal models without exhausting energy reserves. However, when artificial neural networks are trained, the network parameters are typically updated on every sample that is presented, even if the sample was classified correctly. Inspired by the human negativity bias and error-related negativity, we propose 'memorized mistake-gated learning' -- a biologically plausible plasticity rule where synaptic updates are strictly gated by current and past classification errors. This reduces the number of updates the network needs to make by

50\%\sim80\%

. Mistake gating is particularly well suited in two cases: 1) For incremental learning where new knowledge is acquired on a background of pre-existing knowledge, 2) For online learning scenarios when data needs to be stored for later replay, as mistake-gating reduces storage buffer requirements. The algorithm can be implemented in a few lines of code, adds no hyper-parameters, and comes at negligible computational overhead. Learning on mistakes is an energy efficient and biologically relevant modification to commonly used learning rules that is well suited for continual learning.

Abstract

Explore similar work

Mistake gating leads to energy and memory efficient continual learning

Shapley Neuron Values for Continual Learning: Which Neurons Matter Most?

NeuMoSync: End-to-End Neuromodulatory Control for Plasticity and Adaptability in Continual Learning