Do Artificial Neural Networks Need Dendritic Nanotubes? Two Reasons for Shrinking Importance Estimates in Continual Learning

Saurav K. Aryal

cs.LGOct 9, 2026

Do Artificial Neural Networks Need Dendritic Nanotubes? Two Reasons for Shrinking Importance Estimates in Continual Learning

Authors: Saurav K. Aryal

Organizations: Howard University, Institute for Human Centered Artificial Intelligence, Washington, DC, USA

Abstract

Regularization-based continual learning protects each parameter in proportion to an estimate of its importance, yet these estimates come from finite samples and heuristics and are used as if they were exact. We report four studies that start from a biologically inspired mechanism and end with a statistical explanation of it. Motivated by dendritic nanotubes, we let transient lateral links diffuse importance tags between hidden units. Links between co-active units gave no significant gain, but random links with lifetimes of 25 or 200 steps raised final accuracy on 20-task Permuted MNIST by 1.5 to 2.6 points. We show that random diffusion implements linear shrinkage of importance toward the layer mean in expectation. A simulation of the link process alone predicts shrinkage weights of 0.23 and 0.52 for these lifetimes, and graph-free shrinkage at 0.25 and 0.5 reproduces both the accuracy and the dispersion of the consolidated importance, while every activity-dependent rule did worse than random links. A five-seed screen across EWC, SI, and MAS found nominal gains for SI and MAS on Permuted MNIST and losses on task-incremental Split CIFAR-100, where forgetting was small and shrinkage cost plasticity. Varying the number of importance samples then separates two mechanisms on Permuted MNIST. For EWC, the gain fell by about ten points per tenfold increase in samples, from 19.6 points with 10 samples to -0.4 with 1000, a variance effect. For MAS, a gain of about one point persisted at 1000 samples, consistent with a bias in the importance measure rather than noise. Shrinkage had no detectable effect for MAS on a domain-incremental CIFAR-100 benchmark. Finally, we show that a standard heteroscedastic empirical Bayes rule shrinks the largest Fisher-type estimates most, which may explain its poor results, and point to a log-scale alternative.

Figures & tables

Panel of Figure 1 — Figure 1. The consolidation model of Studies 1 and 2. (a) Each hidden unit carries an importance tag, accumulated from squared gradients during a task, that sets how strongly its incoming weights are anchored afterward. (b) Transient links form between random pairs of units, and the importance of each step diffuses over the live links without changing its total. (c) In expectation, the diffusion pulls every tag toward the layer mean by a weight $\beta$ that depends on how many links are alive (Proposition 3.1 ). This is linear shrinkage, and it needs no graph. Outlines show the tags before shrinkage, drawn here for $\beta=0.5$ . Three panels. A row of eight hidden units with bars of different heights above them representing importance tags. In the second panel, curved links connect random pairs of units. In the third panel, the bars are pulled toward a dashed line at the mean height.

Arm	$\lambda$	ACC	BWT	LA
MLP (269,322 parameters)	–	32.58 $\pm$ 1.42	$-67.97$	97.15
Wider MLP (401,110 parameters)	–	35.06 $\pm$ 2.31	$-65.43$	97.21
Lateral links with pruning (400,394 parameters)	–	32.34 $\pm$ 1.90	$-68.20$	97.13
Tag only	0.003	79.48 $\pm$ 1.42	$-15.15$	93.87
+ co-activity links, $L=200$	0.003	79.94 $\pm$ 1.41	$-14.65$	93.86
+ random links, $L=200$	0.003	82.05 $\pm$ 1.35	$-12.20$	93.64

Table 1. Study 1 on 20-task P-MNIST (10 seeds). ACC is the mean

\pm

standard deviation over seeds, and BWT and LA are means, all in percent. The lower panel varies the link lifetime at

\lambda=0.01

, with

\beta_{\mathrm{eff}}

the shrinkage weight implied by random links (Section 3.4 ) and density the fraction of possible links that are alive.

Figure 2. Random links behave as graph-free shrinkage at the weight implied by their lifetime (P-MNIST, 20 tasks, 10 seeds). (a) Shrinkage weight implied by the link process, from a simulation of the process alone (Appendix B ). (b) to (d) Final accuracy against the shrinkage weight for graph-free shrinkage of hidden-layer importance (blue, where

\beta=0

is tag only) and for random links placed at their simulated weight (orange), at three penalty strengths

\lambda

. In (d) the labels give the link lifetime, and lifetimes 50, 500, 1000, and 2000 come from Study 1. Error bars show

\pm 1

standard error over seeds. Lines connect measured weights only, so the blue curve was not sampled between 0.5 and 1. Four panels. Panel a shows the implied shrinkage weight rising from 0.23 at lifetime 25 to 0.83 at lifetime 2000. Panels b to d plot final accuracy against shrinkage weight for three penalty strengths. The graph-free shrinkage curve peaks between beta 0.25 and 0.5 and falls at beta 1. Random-link points sit on the curve at weights 0.23 and 0.52. In panel d, longer lifetimes sit at larger weights and lower accuracy.

Arm	ACC	$\Delta$ tag only	$\Delta$ random	CV	Coverage	Touched	Gini
Tag only	79.48 $\pm$ 1.42	–	–	0.543	–	–	–
Shrinkage, $\beta=0.1$	81.68 $\pm$ 1.07	$+2.20$ (0.44) †	–	0.450	–	–	–
Shrinkage, $\beta=0.25$	82.22 $\pm$ 1.03	$+2.74$ (0.43) †	$+0.10$ (0.30)	0.407	–	–	–
Shrinkage, $\beta=0.5$	81.89 $\pm$ 1.40	$+2.41$ (0.50) †	$-0.17$ (0.21)	0.320	–	–	–
Shrinkage, $\beta=1$	77.85 $\pm$ 1.53	$-1.63$ (0.61)	–	0.000	–	–	–
Random, $L=25$	82.12 $\pm$ 1.63	$+2.64$ (0.41) †	–	0.413	0.792	1.000	0.013

Table 2. Study 2 at

\lambda=0.003

(P-MNIST, 10 seeds). ACC is the mean

\pm

standard deviation in percent. Differences are seed-paired, in points, with the standard error in parentheses. Graph-free arms are compared with random links of the matched lifetime (

\beta=0.25

with

L=25

\beta=0.5

with

L=200

), and link rules with random links of the same lifetime. CV is the coefficient of variation of the consolidated importance. † marks Holm-adjusted

p<0.05

within the family. Appendix Table 8 gives the other two values of

\lambda

Method	Opt.	None	$\beta=0.25$	EB	$\Delta$ , $\beta=0.25$	$\Delta$ LA	$\Delta$ , EB	$\bar{B}$
P-MNIST
EWC	Adam	82.1 $\pm$ 1.2	82.6 $\pm$ 0.5	81.5 $\pm$ 0.5	$+0.47$ (0.65)	$-1.11$	$-0.61$ (0.83)	0.25
EWC-unit	Adam	80.8 $\pm$ 1.0	80.6 $\pm$ 1.4	81.6 $\pm$ 0.8	$-0.20$ (0.43)	$-0.09$	$+0.86$ (0.28) *	0.37
SI	Adam	82.3 $\pm$ 1.2	83.5 $\pm$ 1.2	81.9 $\pm$ 0.8	$+1.24$ (0.30) *	$-0.23$	$-0.41$ (0.36)	0.13
MAS	Adam	81.7 $\pm$ 0.5	83.6 $\pm$ 0.7	82.0 $\pm$ 0.8	$+1.92$ (0.34) *	$-0.11$	$+0.29$ (0.53)	0.03
EWC	SGD	82.8 $\pm$ 0.9	82.5 $\pm$ 1.0	82.6 $\pm$ 0.7	$-0.29$ (0.55)	$+2.48$	$-0.21$ (0.37)	0.29

Table 3. Study 3 (5 seeds). Final accuracy in percent, mean

\pm

standard deviation, at each arm’s validation-selected

\lambda

, with seed-paired differences from no shrinkage in points (SE in parentheses).

\Delta

LA is the change in learning accuracy under

\beta=0.25

, and

\bar{B}

is the mean EB weight. * marks unadjusted

p<0.05

. No difference survives Holm correction over the 27 comparisons. Appendix Table 6 lists the selected

\lambda

values.

Figure 3. Varying the number of importance samples

n

separates variance from bias (Study 4, 10 seeds). (a) to (c) Final accuracy of each arm, with error bars of

\pm 1

standard error over seeds. (d) to (f) Seed-paired gain over no shrinkage, mean

\pm 1

standard error. For EWC on P-MNIST the gain vanishes as

n

grows, the signature of variance. For MAS on P-MNIST a fixed weight of 0.25 helps at every

n

, the signature of bias. Points are offset horizontally for legibility. The CIFAR-100 superclass EWC cells are pending a rerun. Six panels in two rows. Top row shows final accuracy against the number of importance samples for EWC on P-MNIST, MAS on P-MNIST, and MAS on CIFAR-100 superclass, for no shrinkage, fixed beta 0.25, fixed beta 0.5, and empirical Bayes. For EWC, the unshrunk arm rises from 62 to 83 percent while the shrunk arms stay between 77 and 83 percent. Bottom row shows the gain over no shrinkage. For EWC the gain falls from between 16 and 20 points at 10 samples to between minus 2.4 and minus 0.4 points at 1000 samples. For MAS on P-MNIST, beta 0.25 stays positive at every n. For CIFAR-100 all gains are near zero.

P-MNIST, EWC
	$n=10$	$n=50$	$n=200$	$n=1000$	Slope per decade
$\beta=0.25$	$+19.60$ (0.87)	$+6.16$ (0.87)	$+0.12$ (0.37)	$-0.37$ (0.28)	$-9.99$ (0.42) †
$\beta=0.5$	$+18.20$ (1.04)	$+4.52$ (0.90)	$-1.64$ (0.37)	$-2.41$ (0.30) †	$-10.30$ (0.46) †
EB	$+15.89$ (0.96)	$+4.22$ (0.80)	$-1.20$ (0.39)	$-1.29$ (0.40)	$-8.62$ (0.62)
P-MNIST, MAS
$\beta=0.25$	$+2.06$ (0.36)	$+0.89$ (0.39)	$+1.85$ (0.61)	$+1.06$ (0.18) †	$-0.33$ (0.17)

Table 4. Study 4 (10 seeds). The upper panel gives the seed-paired gain in final accuracy over no shrinkage, in points (SE in parentheses), and the seed-level slope of the gain against

\log_{10}n

, in points per decade. † marks Holm-adjusted

p<0.05

over the 16 primary tests, which are the slopes and the gains at

n=1000

for the two fixed weights; EB rows are descriptive. The lower panel gives the accuracy of the unshrunk arm with its slope against

\log_{10}n

, the coefficient of variation (CV) of its raw importance estimates averaged over tensors and tasks, and the mean weight

\bar{B}

assigned by the EB arm.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

	Study 1	Study 2	Study 3	Study 4
Benchmarks	P-MNIST, 20 tasks	P-MNIST, 20 tasks	P-MNIST, 20 tasks; Split CIFAR-100, 10 tasks	P-MNIST, 20 tasks; CIFAR-100 superclass, 5 tasks
Model	MLP $2\times 256$ , wider MLP $2\times 350$	MLP $2\times 256$	MLP $2\times 256$ ; CNN	MLP $2\times 256$ ; CNN
Epochs, batch	3, 128	3, 128	3, 128; 5, 64	3, 128; 5, 64
Optimizer	Adam $10^{-3}$	Adam $10^{-3}$	Adam $10^{-3}$ ; SGD 0.01 with momentum 0.9 and proximal penalty	Adam $10^{-3}$
Importance	unit-level squared gradients	unit-level squared gradients	EWC, EWC-unit, MAS ( $n=200$ ); SI ( $\xi=0.1$ )	EWC, MAS ( $n\in\{10,50,200,1000\}$ )
Shrinkage	none	$\beta\in\{0.1,0.25,0.5,1\}$ , per hidden layer	none, $\beta=0.25$ , EB; per tensor	none, $\beta\in\{0.25,0.5\}$ , EB; per tensor

Appendix

Table 5. Settings of the four studies.

Benchmark	Method	Opt.	Pilot $\lambda^{*}$	None	$\beta=0.25$	EB
P-MNIST	EWC	Adam	0.01	0.01	0.01	0.00333 *
P-MNIST	EWC-unit	Adam	0.01	0.00333 *	0.00333 *	0.01
P-MNIST	SI	Adam	0.001	0.003 *	0.003 *	0.003 *
P-MNIST	MAS	Adam	0.001	0.003 *	0.003 *	0.003 *
P-MNIST	EWC	SGD	0.1	0.3 *	0.1	0.1
Split CIFAR-100	EWC	Adam	10	30 *	3.33 *	30 *

Appendix

Table 6. Study 3, pilot

\lambda^{*}

and the value selected for each arm on validation accuracy from

\lambda^{*}\times\{1/3,1,3\}

. * marks a selected value at the edge of the bracket.

$L$	$\beta_{\mathrm{eff}}$	$\beta_{\mathrm{eff}}$ , first task	Realized weight	$R^{2}$	Coverage	Density	Logged density, coverage
25	0.227	0.226	0.224	0.997	0.793	0.0061	0.006, 0.792
50	0.315	0.307	0.312	0.995	0.957	0.0122	0.012
200	0.518	0.508	0.510	0.982	0.999	0.0488	0.049, 0.999
500	0.652	0.608	0.635	0.957	0.999	0.1215	0.121
1000	0.748	0.648	0.706	0.939	0.999	0.2407	0.241
2000	0.832	0.652	0.828	0.906	0.999	0.4728	0.473

Appendix

Table 7. The link process of the random rule, simulated without training.

\beta_{\mathrm{eff}}

is averaged over 20 tasks; the first-task value includes the period in which links accumulate. The realized weight and

R^{2}

come from a linear fit of the accumulated operator over one task.

	$\lambda=0.001$				$\lambda=0.01$
Arm	ACC	$\Delta$ tag only	$\Delta$ random	CV	ACC	$\Delta$ tag only	$\Delta$ random	CV
Tag only	75.73	–	–	0.460	78.48	–	–	0.733
$\beta=0.1$	75.45	$-0.28$ (0.71)	–	0.423	81.09	$+2.61$ (0.46) †	–	0.507
$\beta=0.25$	77.04	$+1.31$ (0.73)	$-0.17$ (0.47)	0.389	81.20	$+2.72$ (0.50) †	$+0.16$ (0.13)	0.444
$\beta=0.5$	78.24	$+2.51$ (0.52) †	$-0.05$ (0.31)	0.302	80.70	$+2.22$ (0.45) †	$+0.18$ (0.14)	0.342
$\beta=1$	74.87	$-0.86$ (0.44)	–	0.000	76.85	$-1.63$ (0.47)	–	0.000

Appendix

Table 8. Study 2 at

\lambda=0.001

and

\lambda=0.01

(P-MNIST, 10 seeds). ACC is the mean in percent, and differences are seed-paired, in points, with SE in parentheses. Graph-free arms are compared with random links of the matched lifetime, and link rules with random links of the same lifetime. † marks Holm-adjusted

p<0.05

within the family.

Benchmark	Method	Opt.	None	$\beta=0.25$	EB	None	$\beta=0.25$	EB
			BWT			LA
P-MNIST	EWC	Adam	$-10.4$	$-8.7$	$-12.9$	92.0	90.9	93.8
	EWC-unit	Adam	$-13.5$	$-13.6$	$-8.9$	93.6	93.5	90.1
	SI	Adam	$-11.8$	$-10.3$	$-12.2$	93.5	93.3	93.5
	MAS	Adam	$-12.6$	$-10.5$	$-12.3$	93.7	93.6	93.7
	EWC	SGD	$-7.9$	$-10.8$	$-10.6$	90.3	92.8	92.6

Appendix

Table 9. Study 3 retention and plasticity at each arm’s selected

\lambda

(5 seeds, means in percent).

Setting	Arm	ACC	BWT	LA	ACC	BWT	LA
		$n=10$			$n=1000$
P-MNIST, EWC	None	61.7	$-34.6$	94.5	83.2	$-8.5$	91.3
	$\beta=0.25$	81.3	$-11.4$	92.1	82.8	$-8.2$	90.6
	$\beta=0.5$	79.9	$-11.7$	91.0	80.8	$-9.9$	90.2
	EB	77.6	$-14.1$	91.0	81.9	$-9.4$	90.8
P-MNIST, MAS	None	80.2	$-10.9$	90.5	81.5	$-9.2$	90.3

Appendix

Table 10. Study 4 retention and plasticity at the smallest and largest sample sizes (10 seeds, means in percent).

Regularization-Based Continual Learning

See all 13 papers

Aug 16, 2026

Layers Matter: Why Continual Learning Regularization Should Be Layer-Adaptive

Continual learning regularizers like EWC fight forgetting by penalizing changes from previous-task parameters with per-parameter importance, typically diagonal Fisher values. Per-parameter looks more flexible than per-layer, but each layer's diagonal Fisher is a weak summary of its actual curvature, missing the top-eigenvalue information that controls forgetting. Adversarial bit-flip attacks and Hessian-spectrum studies show that this missing per-layer sensitivity spans orders of magnitude in neural networks. Under a block-diagonal Hessian assumption, the layer-level analogue of EWC's existing diagonal assumption, we prove three things. Forgetting decomposes as a sum of per-layer terms weighted by each layer's top Hessian eigenvalue. Diagonal-Fisher weights cannot recover this eigenvalue. For instance, two layers with identical Fisher averages can have top eigenvalues differing by a factor as large as the layer width. For the same level of forgetting, uniform regularization loses new-task performance by an amount scaling with the layer condition number. Our theoretical analysis leads to a simple recipe: protect early layers strongly, let deeper layers move. We apply this recipe to EWC and SLCA and show clear improvements in average performance and forgetting metrics.

Brian B. Moser, Ahmed Anwar, Tobias Christian Nauen +5

German Research Center for Artificial Intelligence (DFKI) · RPTU University Kaiserslautern-Landau

Regularization-Based Continual Learning Continual Learning Incremental Learning Catastrophic Forgetting Elastic Weight Consolidation

Ask PDF

May 15, 2026

Shapley Neuron Values for Continual Learning: Which Neurons Matter Most?

Continual learning enables neural networks to learn tasks sequentially without forgetting previously acquired knowledge. However, neural networks suffer from catastrophic forgetting, where learning new tasks degrades performance on earlier ones. We address this problem with Shapley Neuron Valuation (SNV), a principled framework that quantifies Neuron importance in continual learning, grounded in cooperative game theory. SNV selectively freezes important Neurons while keeping others plastic, enabling buffer-free continual learning without expanding architecture. Experiments on ImageNet-1k show that SNV consistently outperforms existing buffer-free methods. In particular, SNV improves accuracy by +2.88% in the class incremental learning and +6.46% in the task incremental learning scenarios compared to the second baseline.

Mohammad Ali Vahedifar, Abhisek Ray, Qi Zhang

DIGIT and Department of Electrical and Computer Engineering, Aarhus University, Denmark

Task-Incremental Learning Class-Incremental Learning Continual Learning Shapley Value Attribution Incremental Learning Catastrophic Forgetting

Ask PDF

Aug 2, 2026

Plasticity of Growing and Elastic Neural Networks in Online Continual Learning

Neural networks that can grow or both grow and shrink during learning, referred to as growing neural networks and elastic neural networks, respectively, have recently been explored in offline continual learning with a particular focus on catastrophic forgetting. Driven by the observations that 1) online continual learning closely resembles how animals learn; 2) loss of plasticity---the progressive decline in a learning network's ability to learn---is another crucial challenge facing continual learning; and 3) incremental introduction of randomly initialized hidden units was recently shown to help preserve plasticity, in this paper, we study the plasticity of several foundational growing and elastic networks in online continual learning. Our experiments in supervised learning settings show that adaptive growing networks, which incrementally incorporate new, randomly initialized units to the network while keeping all existing connections adaptive, can maintain high prediction accuracy without losing plasticity despite the continuous increase in the dead hidden unit proportion. Furthermore, we demonstrate that adaptive elastic networks, which in addition to progressively adding new hidden units also prune estimated dead hidden units at the beginning of each new task, can achieve excellent accuracy without loss of plasticity while simultaneously maintaining a near-constant, compact size. Our results suggest that growing and elastic networks, which exhibit the ability to adapt its structure to the relevant learning objectives, can be a promising class of algorithms also for preserving high plasticity in online continual learning.

Jeong Min Kong, Richard S. Sutton

Department of Electrical and Computer Engineering University of California, Los Angeles (UCLA) · Department of Computing Science University of Alberta Alberta Machine Intelligence Institute (Amii)

Loss of Plasticity Dynamic Neural Networks Elastic Neural Networks Online Continual Learning Incremental Learning Neural Network Growth

Ask PDF

Abstract

Regularization-Based Continual Learning

Explore similar work

Layers Matter: Why Continual Learning Regularization Should Be Layer-Adaptive

Shapley Neuron Values for Continual Learning: Which Neurons Matter Most?

Plasticity of Growing and Elastic Neural Networks in Online Continual Learning