Isotropic Activation Functions Enable Deindividuated Neurons and Adaptive Topologies

George Bird

cs.NEFeb 26, 2026

Isotropic Activation Functions Enable Deindividuated Neurons and Adaptive Topologies

Authors: George Bird

Organizations: Department of Computer Science, University of Manchester, UK

Abstract

Varying the number of neurons while preserving function is difficult because neural networks' dense connectivity creates strong computational interdependence between units. Elementwise activation functions, particularly their limited equivariance properties, are identified as a central hurdle to achieving this. We propose activation functions designed around chosen equivariances, enabling dense neural networks to adapt via singular value decomposition of their affine maps. This diagonalises layers into one-to-one, ordered connections, making it easier to assess the impact of individual connections on function. Low-impact neurons can be removed (neurodegeneration), and a thresholded buffer of largely inactive scaffold' neurons is maintained (neurogenesis). These symmetry-led diagonalisation and structural changes prioritise function-invariance: neurogenesis exactly preserves function, neurodegeneration is asymptotically function-preserving with the pruned singular-value threshold. Moreover, this construction enables asymptotic 50\% parameter sparsification of dense networks while preserving function analytically. Thus, the approach enables real-time restructuring of the architecture in response to task demands, task appending, removal or changes. Conceptually, this motivates broader applications of symmetry prescriptions at the small scale, even when no data-driven model-scale symmetry adherence is required. Following this, we build activations around symmetry, not from classical assumptions about individual neurons. Unlike typical elementwise forms, these alternatives feature explicit basis independence and do not individuate neurons. We leverage this to decompose and reinterpret neurons across sequential layers, simplifying structure and providing a natural route to adaptive topologies. Additional developments include a new tunable parameter, the intrinsic length'.

Figures & tables

Figure 1: Illustrates the qualitative effects on a network undergoing full diagonalisation. The connectivity is simplified to a one-to-one correspondence between neurons on the chosen layer.

Neuroadaptation	$o_{\mathrm{none}},\ \vec{\psi}_{\mathrm{decay}}$ $\Sigma_{0}=240$		$o_{\mathrm{constant}},\ \vec{\psi}_{\mathrm{decay}}$ $\Sigma_{0}=240$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{decay}}$ $\Sigma_{0}=241$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{none}}$ $\Sigma_{0}=240$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{constant}}$ $\Sigma_{0}=237$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{trainable}}$ $\Sigma_{0}=237$		$o_{\mathrm{trainable}},\ \vec{b}_{\mathrm{adjusted}}$ $\Sigma_{0}=240$
	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)
$+75$	$240$	$(8.4\!\pm\!0.2)\!\times\!10^{-4}$	$240$	$(8.5\!\pm\!0.2)\!\times\!10^{-4}$	$241$	$\mathbf{(8.2\!\pm\!0.2)\!\times\!10^{-4}}$	$240$	$(8.6\!\pm\!0.1)\!\times\!10^{-4}$	$237$	$(8.5\!\pm\!0.2)\!\times\!10^{-4}$	$237$	$(8.3\!\pm\!0.2)\!\times\!10^{-4}$	$240$	$(8.3\!\pm\!0.1)\!\times\!10^{-4}$
$+50$	$240$	$(8.4\!\pm\!0.2)\!\times\!10^{-4}$	$240$	$(8.5\!\pm\!0.2)\!\times\!10^{-4}$	$241$	$\mathbf{(8.2\!\pm\!0.2)\!\times\!10^{-4}}$	$240$	$(8.6\!\pm\!0.1)\!\times\!10^{-4}$	$237$	$(8.5\!\pm\!0.2)\!\times\!10^{-4}$	$237$	$(8.3\!\pm\!0.2)\!\times\!10^{-4}$	$240$	$(8.3\!\pm\!0.1)\!\times\!10^{-4}$
$+25$	$240$	$(8.4\!\pm\!0.2)\!\times\!10^{-4}$	$240$	$(8.5\!\pm\!0.2)\!\times\!10^{-4}$	$241$	$\mathbf{(8.2\!\pm\!0.2)\!\times\!10^{-4}}$	$240$	$(8.6\!\pm\!0.1)\!\times\!10^{-4}$	$237$	$(8.5\!\pm\!0.2)\!\times\!10^{-4}$	$237$	$(8.3\!\pm\!0.2)\!\times\!10^{-4}$	$240$	$(8.3\!\pm\!0.1)\!\times\!10^{-4}$
$+5$	$240$	$(8.4\!\pm\!0.2)\!\times\!10^{-4}$	$240$	$(8.5\!\pm\!0.2)\!\times\!10^{-4}$	$241$	$\mathbf{(8.2\!\pm\!0.2)\!\times\!10^{-4}}$	$240$	$(8.6\!\pm\!0.1)\!\times\!10^{-4}$	$237$	$(8.5\!\pm\!0.2)\!\times\!10^{-4}$	$237$	$(8.3\!\pm\!0.2)\!\times\!10^{-4}$	$240$	$(8.3\!\pm\!0.1)\!\times\!10^{-4}$
$+1$	$240$	$(8.4\!\pm\!0.2)\!\times\!10^{-4}$	$240$	$(8.5\!\pm\!0.2)\!\times\!10^{-4}$	$241$	$\mathbf{(8.2\!\pm\!0.2)\!\times\!10^{-4}}$	$240$	$(8.6\!\pm\!0.1)\!\times\!10^{-4}$	$237$	$(8.5\!\pm\!0.2)\!\times\!10^{-4}$	$237$	$(8.3\!\pm\!0.2)\!\times\!10^{-4}$	$240$	$(8.3\!\pm\!0.1)\!\times\!10^{-4}$

Table 1: Experiment One ( App. K.1 ): measures function invariance after neuroadaptation: appending

+k

, removing

-k

, or diagonalisation

0

the

\mathbb{R}^{100}\rightarrow\mathbb{R}^{100}

layer of a

[12288,100,100,256]

Caltech256 trained classifier. Mean

\pm

standard error (SE) are reported over

20

independent repeats. Columns specify intrinsic length

o

and the

{\psi}

-vector approach, yielding

7\times 20

total networks.

\Sigma_{0}

and

\Sigma_{1}

denote the before- and after-sums of singular values, respectively (with their SE omitted for legibility; all

\approx 0.4

\epsilon_{\mathrm{rel}}

is the test-set mean

L_{2}

logit change divided by the initial value, reported as a percentage.

Figure 2: Experiment Two ( App. K.2 ) displays the evolving layerwise singular value and bias spectra for a $\left[3072,100,100,10\right]$ fixed isotropic MLP trained on CIFAR-10 [ 47 ] classification. The spectra are obtained from $40$ independent repeats, showing strong consistency. Singular values begin at one, indicated by the dashed line, due to orthogonal initialisation, and biases at zero due to initialisation.

Figure 3: Experiment Three ( App. K.3 ) demonstrates thresholded neural adaptation when an architecture of

\left[2352,\text{H1},\text{H2},\text{H3},30\right]

adapts to a changing task schedule, across

25

independent repeats for random hidden widths

\text{H1},\text{H2},\text{H3}\!\sim\!\mathrm{U}\left(500,2500\right)

. From epoch

0

through

150

, only MNIST [ 48 ] cross-entropy terms contribute to the loss; at T1 , the total loss is equally averaged between both MNIST and FMNIST [ 49 ] . At T2 , the total loss is equally averaged between MNIST, FMNIST, and A-J letters from EMNIST [ 50 ] , and finally, at T3 , only FMNIST’s loss contributes.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Variant	Functional Invariance	Computational Cost
$Q$ absorbed with full covariance	Exact, under assumptions	Loses diagonal BatchNorm efficiency
$Q$ absorbed with diagonal covariance	Approximate	Retains BatchNorm’s efficiency

Appendix

Table 2: Two primary approaches for a network in the ‘ICA-representation’ with respective trade-offs.

Neuroadaptation	$o_{\mathrm{none}},\ \vec{\psi}_{\mathrm{decay}}$ $\Sigma_{0}=240$		$o_{\mathrm{constant}},\ \vec{\psi}_{\mathrm{decay}}$ $\Sigma_{0}=240$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{decay}}$ $\Sigma_{0}=241$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{none}}$ $\Sigma_{0}=240$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{constant}}$ $\Sigma_{0}=237$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{trainable}}$ $\Sigma_{0}=237$		$o_{\mathrm{trainable}},\ \vec{b}_{\mathrm{adjusted}}$ $\Sigma_{0}=240$
	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)
$+75$	$240$	$(8.4\!\pm\!0.2)\!\times\!10^{-4}$	$240$	$(8.5\!\pm\!0.2)\!\times\!10^{-4}$	$241$	$\mathbf{(8.2\!\pm\!0.2)\!\times\!10^{-4}}$	$240$	$(8.6\!\pm\!0.1)\!\times\!10^{-4}$	$237$	$(8.5\!\pm\!0.2)\!\times\!10^{-4}$	$237$	$(8.3\!\pm\!0.2)\!\times\!10^{-4}$	$240$	$(8.3\!\pm\!0.1)\!\times\!10^{-4}$
$+50$	$240$	$(8.4\!\pm\!0.2)\!\times\!10^{-4}$	$240$	$(8.5\!\pm\!0.2)\!\times\!10^{-4}$	$241$	$\mathbf{(8.2\!\pm\!0.2)\!\times\!10^{-4}}$	$240$	$(8.6\!\pm\!0.1)\!\times\!10^{-4}$	$237$	$(8.5\!\pm\!0.2)\!\times\!10^{-4}$	$237$	$(8.3\!\pm\!0.2)\!\times\!10^{-4}$	$240$	$(8.3\!\pm\!0.1)\!\times\!10^{-4}$
$+25$	$240$	$(8.4\!\pm\!0.2)\!\times\!10^{-4}$	$240$	$(8.5\!\pm\!0.2)\!\times\!10^{-4}$	$241$	$\mathbf{(8.2\!\pm\!0.2)\!\times\!10^{-4}}$	$240$	$(8.6\!\pm\!0.1)\!\times\!10^{-4}$	$237$	$(8.5\!\pm\!0.2)\!\times\!10^{-4}$	$237$	$(8.3\!\pm\!0.2)\!\times\!10^{-4}$	$240$	$(8.3\!\pm\!0.1)\!\times\!10^{-4}$
$+5$	$240$	$(8.4\!\pm\!0.2)\!\times\!10^{-4}$	$240$	$(8.5\!\pm\!0.2)\!\times\!10^{-4}$	$241$	$\mathbf{(8.2\!\pm\!0.2)\!\times\!10^{-4}}$	$240$	$(8.6\!\pm\!0.1)\!\times\!10^{-4}$	$237$	$(8.5\!\pm\!0.2)\!\times\!10^{-4}$	$237$	$(8.3\!\pm\!0.2)\!\times\!10^{-4}$	$240$	$(8.3\!\pm\!0.1)\!\times\!10^{-4}$
$+1$	$240$	$(8.4\!\pm\!0.2)\!\times\!10^{-4}$	$240$	$(8.5\!\pm\!0.2)\!\times\!10^{-4}$	$241$	$\mathbf{(8.2\!\pm\!0.2)\!\times\!10^{-4}}$	$240$	$(8.6\!\pm\!0.1)\!\times\!10^{-4}$	$237$	$(8.5\!\pm\!0.2)\!\times\!10^{-4}$	$237$	$(8.3\!\pm\!0.2)\!\times\!10^{-4}$	$240$	$(8.3\!\pm\!0.1)\!\times\!10^{-4}$

Appendix

Table 3: Experiment One (Caltech-256): singular-value structure and relative function change under neuroadaptation at layer

L=1

of the

[12288,100,100,256]

classifier. Results are mean

\pm

standard error over 20 independent repeats. Row

0

applies diagonalisation only;

+k

subsequently applies

k

neurogenesis operations in a single step and

-k

applies

k

neurodegeneration operations in a single step.

\Sigma_{0}

and

\Sigma_{1}

denote the sums of singular values before and after neuroadaptation, respectively. Singular-value standard errors are omitted for legibility; the largest singular-value standard error is 0.38.

\epsilon_{\mathrm{rel}}

is the Eqn. 58 ’s metric, bold indicates the smallest value within each neuroadaptation condition.

Neuroadaptation	$o_{\mathrm{none}},\ \vec{\psi}_{\mathrm{decay}}$		$o_{\mathrm{constant}},\ \vec{\psi}_{\mathrm{decay}}$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{decay}}$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{none}}$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{constant}}$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{trainable}}$		$o_{\mathrm{trainable}},\ \vec{b}_{\mathrm{adjusted}}$
	$D_{\mathrm{KL}}$	$\Delta a$ (pp)	$D_{\mathrm{KL}}$	$\Delta a$ (pp)	$D_{\mathrm{KL}}$	$\Delta a$ (pp)	$D_{\mathrm{KL}}$	$\Delta a$ (pp)	$D_{\mathrm{KL}}$	$\Delta a$ (pp)	$D_{\mathrm{KL}}$	$\Delta a$ (pp)	$D_{\mathrm{KL}}$	$\Delta a$ (pp)
$+75$	$\mathbf{(1.0\!\pm\!0.3)\!\times\!10^{-9}}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.4\!\pm\!0.2)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.4\!\pm\!0.3)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.2\!\pm\!0.3)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.2\!\pm\!0.3)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.4\!\pm\!0.3)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.5\!\pm\!0.3)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$
$+50$	$(1.3\!\pm\!0.3)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$\mathbf{(9.0\!\pm\!3.2)\!\times\!10^{-10}}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.3\!\pm\!0.3)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.3\!\pm\!0.3)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.5\!\pm\!0.3)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.2\!\pm\!0.3)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.5\!\pm\!0.2)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$
$+25$	$(1.3\!\pm\!0.3)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$\mathbf{(1.1\!\pm\!0.3)\!\times\!10^{-9}}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.3\!\pm\!0.3)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.3\!\pm\!0.3)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.5\!\pm\!0.3)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.2\!\pm\!0.3)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.5\!\pm\!0.2)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$
$+5$	$(1.3\!\pm\!0.3)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$\mathbf{(1.1\!\pm\!0.3)\!\times\!10^{-9}}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.3\!\pm\!0.3)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.3\!\pm\!0.3)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.5\!\pm\!0.3)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.2\!\pm\!0.3)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.5\!\pm\!0.2)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$
$+1$	$(1.3\!\pm\!0.3)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$\mathbf{(1.1\!\pm\!0.3)\!\times\!10^{-9}}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.2\!\pm\!0.3)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.3\!\pm\!0.3)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.7\!\pm\!0.3)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.2\!\pm\!0.3)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.5\!\pm\!0.2)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$

Appendix

Table 4: Experiment One (Caltech-256): additional function-change measures for the same neuroadaptation conditions. Results are mean

\pm

standard error over 20 independent repeats.

D_{\mathrm{KL}}

is provided by Eqn. 59 and

\Delta a=a_{\mathrm{post}}-a_{\mathrm{pre}}

is the change in test accuracy in percentage points; positive values indicate increased accuracy. The baseline row reports the unadapted test accuracy

a_{0}

as a percentage. Bold values indicate the smallest mean

D_{\mathrm{KL}}

and the accuracy change closest to zero within each neuroadaptation condition.

Neuroadaptation	$o_{\mathrm{none}},\ \vec{\psi}_{\mathrm{decay}}$ $\Sigma_{0}=174$		$o_{\mathrm{constant}},\ \vec{\psi}_{\mathrm{decay}}$ $\Sigma_{0}=174$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{decay}}$ $\Sigma_{0}=174$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{none}}$ $\Sigma_{0}=174$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{constant}}$ $\Sigma_{0}=164$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{trainable}}$ $\Sigma_{0}=164$		$o_{\mathrm{trainable}},\ \vec{b}_{\mathrm{adjusted}}$ $\Sigma_{0}=174$
	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)
$+75$	$174$	$(3.1\!\pm\!0.1)\!\times\!10^{-3}$	$174$	$(2.9\!\pm\!0.1)\!\times\!10^{-3}$	$174$	$(2.9\!\pm\!0.1)\!\times\!10^{-3}$	$174$	$(3.0\!\pm\!0.1)\!\times\!10^{-3}$	$164$	$\mathbf{(2.5\!\pm\!0.1)\!\times\!10^{-3}}$	$164$	$(2.7\!\pm\!0.1)\!\times\!10^{-3}$	$174$	$(3.1\!\pm\!0.2)\!\times\!10^{-3}$
$+50$	$174$	$(3.1\!\pm\!0.1)\!\times\!10^{-3}$	$174$	$(2.9\!\pm\!0.1)\!\times\!10^{-3}$	$174$	$(2.9\!\pm\!0.1)\!\times\!10^{-3}$	$174$	$(3.0\!\pm\!0.1)\!\times\!10^{-3}$	$164$	$\mathbf{(2.5\!\pm\!0.1)\!\times\!10^{-3}}$	$164$	$(2.7\!\pm\!0.1)\!\times\!10^{-3}$	$174$	$(3.1\!\pm\!0.2)\!\times\!10^{-3}$
$+25$	$174$	$(3.1\!\pm\!0.1)\!\times\!10^{-3}$	$174$	$(2.9\!\pm\!0.1)\!\times\!10^{-3}$	$174$	$(2.9\!\pm\!0.1)\!\times\!10^{-3}$	$174$	$(3.0\!\pm\!0.1)\!\times\!10^{-3}$	$164$	$\mathbf{(2.5\!\pm\!0.1)\!\times\!10^{-3}}$	$164$	$(2.7\!\pm\!0.1)\!\times\!10^{-3}$	$174$	$(3.1\!\pm\!0.2)\!\times\!10^{-3}$
$+5$	$174$	$(3.1\!\pm\!0.1)\!\times\!10^{-3}$	$174$	$(2.9\!\pm\!0.1)\!\times\!10^{-3}$	$174$	$(2.9\!\pm\!0.1)\!\times\!10^{-3}$	$174$	$(3.0\!\pm\!0.1)\!\times\!10^{-3}$	$164$	$\mathbf{(2.5\!\pm\!0.1)\!\times\!10^{-3}}$	$164$	$(2.7\!\pm\!0.1)\!\times\!10^{-3}$	$174$	$(3.1\!\pm\!0.2)\!\times\!10^{-3}$
$+1$	$174$	$(3.1\!\pm\!0.1)\!\times\!10^{-3}$	$174$	$(2.9\!\pm\!0.1)\!\times\!10^{-3}$	$174$	$(2.9\!\pm\!0.1)\!\times\!10^{-3}$	$174$	$(3.0\!\pm\!0.1)\!\times\!10^{-3}$	$164$	$\mathbf{(2.5\!\pm\!0.1)\!\times\!10^{-3}}$	$164$	$(2.7\!\pm\!0.1)\!\times\!10^{-3}$	$174$	$(3.1\!\pm\!0.2)\!\times\!10^{-3}$

Appendix

Table 5: Experiment One (CIFAR-10): singular-value structure and relative function change under neuroadaptation at layer

L=1

of the

[3072,100,100,10]

classifier. Results are mean

\pm

standard error over 20 independent repeats. Row

0

applies diagonalisation only;

+k

subsequently applies

k

neurogenesis operations in a single step and

-k

applies

k

neurodegeneration operations in a single step.

\Sigma_{0}

and

\Sigma_{1}

denote the sums of singular values before and after neuroadaptation, respectively. Singular-value standard errors are omitted for legibility; the largest singular-value standard error is 0.32.

\epsilon_{\mathrm{rel}}

is the Eqn. 58 ’s metric, bold indicates the smallest value within each neuroadaptation condition.

Neuroadaptation	$o_{\mathrm{none}},\ \vec{\psi}_{\mathrm{decay}}$		$o_{\mathrm{constant}},\ \vec{\psi}_{\mathrm{decay}}$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{decay}}$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{none}}$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{constant}}$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{trainable}}$		$o_{\mathrm{trainable}},\ \vec{b}_{\mathrm{adjusted}}$
	$D_{\mathrm{KL}}$	$\Delta a$ (pp)	$D_{\mathrm{KL}}$	$\Delta a$ (pp)	$D_{\mathrm{KL}}$	$\Delta a$ (pp)	$D_{\mathrm{KL}}$	$\Delta a$ (pp)	$D_{\mathrm{KL}}$	$\Delta a$ (pp)	$D_{\mathrm{KL}}$	$\Delta a$ (pp)	$D_{\mathrm{KL}}$	$\Delta a$ (pp)
$+75$	$(1.3\!\pm\!0.2)\!\times\!10^{-9}$	$-0.0\!\pm\!0.0$	$(1.5\!\pm\!0.2)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.2\!\pm\!0.1)\!\times\!10^{-9}$	$-0.0\!\pm\!0.0$	$(1.3\!\pm\!0.2)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$	$\mathbf{(1.1\!\pm\!0.2)\!\times\!10^{-9}}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.2\!\pm\!0.1)\!\times\!10^{-9}$	$-0.0\!\pm\!0.0$	$(1.6\!\pm\!0.2)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$
$+50$	$(1.4\!\pm\!0.1)\!\times\!10^{-9}$	$-0.0\!\pm\!0.0$	$(1.5\!\pm\!0.2)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.5\!\pm\!0.2)\!\times\!10^{-9}$	$-0.0\!\pm\!0.0$	$(1.4\!\pm\!0.2)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$	$\mathbf{(1.1\!\pm\!0.2)\!\times\!10^{-9}}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.2\!\pm\!0.1)\!\times\!10^{-9}$	$-0.0\!\pm\!0.0$	$(1.7\!\pm\!0.2)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$
$+25$	$(1.4\!\pm\!0.1)\!\times\!10^{-9}$	$-0.0\!\pm\!0.0$	$(1.4\!\pm\!0.2)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.3\!\pm\!0.2)\!\times\!10^{-9}$	$-0.0\!\pm\!0.0$	$(1.2\!\pm\!0.1)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$	$\mathbf{(1.1\!\pm\!0.2)\!\times\!10^{-9}}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.3\!\pm\!0.2)\!\times\!10^{-9}$	$-0.0\!\pm\!0.0$	$(1.6\!\pm\!0.2)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$
$+5$	$(1.4\!\pm\!0.2)\!\times\!10^{-9}$	$-0.0\!\pm\!0.0$	$(1.3\!\pm\!0.2)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.3\!\pm\!0.2)\!\times\!10^{-9}$	$-0.0\!\pm\!0.0$	$(1.3\!\pm\!0.1)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$	$(1.1\!\pm\!0.2)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$\mathbf{(1.0\!\pm\!0.2)\!\times\!10^{-9}}$	$-0.0\!\pm\!0.0$	$(1.6\!\pm\!0.2)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$
$+1$	$(1.3\!\pm\!0.2)\!\times\!10^{-9}$	$-0.0\!\pm\!0.0$	$(1.4\!\pm\!0.3)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.3\!\pm\!0.2)\!\times\!10^{-9}$	$-0.0\!\pm\!0.0$	$(1.1\!\pm\!0.2)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$	$\mathbf{(1.0\!\pm\!0.1)\!\times\!10^{-9}}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.1\!\pm\!0.1)\!\times\!10^{-9}$	$-0.0\!\pm\!0.0$	$(1.7\!\pm\!0.2)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$

Appendix

Table 6: Experiment One (CIFAR-10): additional function-change measures for the same neuroadaptation conditions. Results are mean

\pm

standard error over 20 independent repeats.

D_{\mathrm{KL}}

is provided by Eqn. 59 and

\Delta a=a_{\mathrm{post}}-a_{\mathrm{pre}}

is the change in test accuracy in percentage points; positive values indicate increased accuracy. The baseline row reports the unadapted test accuracy

a_{0}

as a percentage. Bold values indicate the smallest mean

D_{\mathrm{KL}}

and the accuracy change closest to zero within each neuroadaptation condition.

Neuroadaptation	$o_{\mathrm{none}},\ \vec{\psi}_{\mathrm{decay}}$ $\Sigma_{0}=260$		$o_{\mathrm{constant}},\ \vec{\psi}_{\mathrm{decay}}$ $\Sigma_{0}=260$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{decay}}$ $\Sigma_{0}=260$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{none}}$ $\Sigma_{0}=260$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{constant}}$ $\Sigma_{0}=252$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{trainable}}$ $\Sigma_{0}=252$		$o_{\mathrm{trainable}},\ \vec{b}_{\mathrm{adjusted}}$ $\Sigma_{0}=260$
	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)	$\Sigma_{1}$	$\epsilon_{\mathrm{rel}}$ (%)
$+75$	$260$	$(1.7\!\pm\!0.1)\!\times\!10^{-3}$	$260$	$(1.7\!\pm\!0.1)\!\times\!10^{-3}$	$260$	$(1.6\!\pm\!0.1)\!\times\!10^{-3}$	$260$	$(1.6\!\pm\!0.0)\!\times\!10^{-3}$	$252$	$\mathbf{(1.4\!\pm\!0.1)\!\times\!10^{-3}}$	$252$	$(1.5\!\pm\!0.1)\!\times\!10^{-3}$	$260$	$(1.7\!\pm\!0.1)\!\times\!10^{-3}$
$+50$	$260$	$(1.7\!\pm\!0.1)\!\times\!10^{-3}$	$260$	$(1.7\!\pm\!0.1)\!\times\!10^{-3}$	$260$	$(1.6\!\pm\!0.1)\!\times\!10^{-3}$	$260$	$(1.6\!\pm\!0.0)\!\times\!10^{-3}$	$252$	$\mathbf{(1.4\!\pm\!0.1)\!\times\!10^{-3}}$	$252$	$(1.5\!\pm\!0.1)\!\times\!10^{-3}$	$260$	$(1.7\!\pm\!0.1)\!\times\!10^{-3}$
$+25$	$260$	$(1.7\!\pm\!0.1)\!\times\!10^{-3}$	$260$	$(1.7\!\pm\!0.1)\!\times\!10^{-3}$	$260$	$(1.6\!\pm\!0.1)\!\times\!10^{-3}$	$260$	$(1.6\!\pm\!0.0)\!\times\!10^{-3}$	$252$	$\mathbf{(1.4\!\pm\!0.1)\!\times\!10^{-3}}$	$252$	$(1.5\!\pm\!0.1)\!\times\!10^{-3}$	$260$	$(1.7\!\pm\!0.1)\!\times\!10^{-3}$
$+5$	$260$	$(1.7\!\pm\!0.1)\!\times\!10^{-3}$	$260$	$(1.7\!\pm\!0.1)\!\times\!10^{-3}$	$260$	$(1.6\!\pm\!0.1)\!\times\!10^{-3}$	$260$	$(1.6\!\pm\!0.0)\!\times\!10^{-3}$	$252$	$\mathbf{(1.4\!\pm\!0.1)\!\times\!10^{-3}}$	$252$	$(1.5\!\pm\!0.1)\!\times\!10^{-3}$	$260$	$(1.7\!\pm\!0.1)\!\times\!10^{-3}$
$+1$	$260$	$(1.7\!\pm\!0.1)\!\times\!10^{-3}$	$260$	$(1.7\!\pm\!0.1)\!\times\!10^{-3}$	$260$	$(1.6\!\pm\!0.1)\!\times\!10^{-3}$	$260$	$(1.6\!\pm\!0.0)\!\times\!10^{-3}$	$252$	$\mathbf{(1.4\!\pm\!0.1)\!\times\!10^{-3}}$	$252$	$(1.5\!\pm\!0.1)\!\times\!10^{-3}$	$260$	$(1.7\!\pm\!0.1)\!\times\!10^{-3}$

Appendix

Table 7: Experiment One (CIFAR-100): singular-value structure and relative function change under neuroadaptation at layer

L=1

of the

[3072,100,100,100]

classifier. Results are mean

\pm

standard error over 20 independent repeats. Row

0

applies diagonalisation only;

+k

subsequently applies

k

neurogenesis operations in a single step and

-k

applies

k

neurodegeneration operations in a single step.

\Sigma_{0}

and

\Sigma_{1}

denote the sums of singular values before and after neuroadaptation, respectively. Singular-value standard errors are omitted for legibility; the largest singular-value standard error is 0.23.

\epsilon_{\mathrm{rel}}

is the Eqn. 58 ’s metric, bold indicates the smallest value within each neuroadaptation condition.

Neuroadaptation	$o_{\mathrm{none}},\ \vec{\psi}_{\mathrm{decay}}$		$o_{\mathrm{constant}},\ \vec{\psi}_{\mathrm{decay}}$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{decay}}$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{none}}$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{constant}}$		$o_{\mathrm{trainable}},\ \vec{\psi}_{\mathrm{trainable}}$		$o_{\mathrm{trainable}},\ \vec{b}_{\mathrm{adjusted}}$
	$D_{\mathrm{KL}}$	$\Delta a$ (pp)	$D_{\mathrm{KL}}$	$\Delta a$ (pp)	$D_{\mathrm{KL}}$	$\Delta a$ (pp)	$D_{\mathrm{KL}}$	$\Delta a$ (pp)	$D_{\mathrm{KL}}$	$\Delta a$ (pp)	$D_{\mathrm{KL}}$	$\Delta a$ (pp)	$D_{\mathrm{KL}}$	$\Delta a$ (pp)
$+75$	$(1.6\!\pm\!0.2)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$	$(1.5\!\pm\!0.2)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$	$\mathbf{(1.2\!\pm\!0.2)\!\times\!10^{-9}}$	$-0.0\!\pm\!0.0$	$(1.6\!\pm\!0.2)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.2\!\pm\!0.1)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$	$(1.6\!\pm\!0.2)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.4\!\pm\!0.2)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$
$+50$	$(1.6\!\pm\!0.2)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$	$(1.6\!\pm\!0.2)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$	$(1.2\!\pm\!0.2)\!\times\!10^{-9}$	$-0.0\!\pm\!0.0$	$(1.6\!\pm\!0.2)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$\mathbf{(1.1\!\pm\!0.2)\!\times\!10^{-9}}$	$+0.0\!\pm\!0.0$	$(1.6\!\pm\!0.2)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.4\!\pm\!0.2)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$
$+25$	$(1.6\!\pm\!0.2)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$	$(1.5\!\pm\!0.2)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$	$\mathbf{(1.2\!\pm\!0.2)\!\times\!10^{-9}}$	$-0.0\!\pm\!0.0$	$(1.6\!\pm\!0.2)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.2\!\pm\!0.1)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$	$(1.6\!\pm\!0.2)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.4\!\pm\!0.2)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$
$+5$	$(1.5\!\pm\!0.1)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$	$(1.4\!\pm\!0.2)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$	$(1.2\!\pm\!0.2)\!\times\!10^{-9}$	$-0.0\!\pm\!0.0$	$(1.5\!\pm\!0.2)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$\mathbf{(1.2\!\pm\!0.1)\!\times\!10^{-9}}$	$+0.0\!\pm\!0.0$	$(1.6\!\pm\!0.2)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.5\!\pm\!0.2)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$
$+1$	$(1.6\!\pm\!0.2)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$	$(1.5\!\pm\!0.2)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$	$\mathbf{(1.2\!\pm\!0.2)\!\times\!10^{-9}}$	$-0.0\!\pm\!0.0$	$(1.9\!\pm\!0.2)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.2\!\pm\!0.2)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$	$(1.4\!\pm\!0.2)\!\times\!10^{-9}$	$\mathbf{0.0\!\pm\!0.0}$	$(1.4\!\pm\!0.2)\!\times\!10^{-9}$	$+0.0\!\pm\!0.0$

Appendix

Table 8: Experiment One (CIFAR-100): additional function-change measures for the same neuroadaptation conditions. Results are mean

\pm

standard error over 20 independent repeats.

D_{\mathrm{KL}}

is provided by Eqn. 59 and

\Delta a=a_{\mathrm{post}}-a_{\mathrm{pre}}

is the change in test accuracy in percentage points; positive values indicate increased accuracy. The baseline row reports the unadapted test accuracy

a_{0}

as a percentage. Bold values indicate the smallest mean

D_{\mathrm{KL}}

and the accuracy change closest to zero within each neuroadaptation condition.

Figure 4: Experiment Two ( App. K.2 ) evolving layerwise singular value and bias spectra for a $\left[3072,100,100,10\right]$ isotropic-tanh CIFAR-10 classifier.

Figure 5: Experiment Two ( App. K.2 ) evolving layerwise singular value and bias spectra for a $\left[3072,100,100,10\right]$ standard-tanh CIFAR-10 classifier.

Figure 6: Experiment Two ( App. K.2 ) evolving layerwise bias-magnitude spectra for a

\left[3072,100,100,10\right]

isotropic-tanh CIFAR-10 classifier.

Figure 7: Experiment Two ( App. K.2 ) evolving layerwise bias-magnitude spectra for a

\left[3072,100,100,10\right]

standard-tanh CIFAR-10 classifier.

Figure 8: Experiment Two ( App. K.2 ) evolving layerwise singular value and bias spectra for a $\left[3072,500,500,10\right]$ isotropic-tanh CIFAR-10 classifier.

Figure 9: Experiment Two ( App. K.2 ) evolving layerwise singular value and bias spectra for a $\left[3072,1000,1000,10\right]$ isotropic-tanh CIFAR-10 classifier.

Figure 10: Experiment Two ( App. K.2 ) evolving layerwise bias-magnitude spectra for a

\left[3072,1000,1000,10\right]

isotropic-tanh CIFAR-10 classifier.

Figure 11: Experiment Two ( App. K.2 ) evolving layerwise singular value and bias spectra for a $\left[3072,500,500,10\right]$ standard-tanh CIFAR-10 classifier.

Figure 12: Experiment Two ( App. K.2 ) evolving layerwise singular value and bias spectra for a $\left[3072,1000,1000,10\right]$ standard-tanh CIFAR-10 classifier.

Figure 13: Experiment Two ( App. K.2 ) evolving layerwise singular value and bias spectra for a $\left[12288,100,100,256\right]$ isotropic-tanh Caltech256 classifier.

Figure 14: Experiment Two ( App. K.2 ) evolving layerwise singular value and bias spectra for a $\left[12288,500,500,256\right]$ isotropic-tanh Caltech256 classifier.

Figure 15: Experiment Two ( App. K.2 ) evolving layerwise singular value and bias spectra for a $\left[12288,1000,1000,256\right]$ isotropic-tanh Caltech256 classifier.

Figure 16: Experiment Two ( App. K.2 ) evolving layerwise bias-magnitude spectra for a

\left[12288,100,100,256\right]

isotropic-tanh Caltech256 classifier.

Figure 17: Experiment Two ( App. K.2 ) evolving layerwise bias-magnitude spectra for a

\left[12288,1000,1000,256\right]

isotropic-tanh Caltech256 classifier.

Epoch range	Active dataset(s)
$0$ – $149$	MNIST
$150$ – $299$	MNIST + Fashion-MNIST
$300$ – $449$	MNIST + Fashion-MNIST + EMNIST A–J
$450$ – $599$	Fashion-MNIST

Appendix

Table 9: Task schedule used in Experiment Three.

Figure 18: Experiment Three ( App. K.3 ) of a single idealistic example

Figure 19: Experiment Three ( App. K.3 ) of a single non-idealistic example

Figure 20: Experiment Three ( App. K.3 ) of a single non-idealistic example

Start width $\downarrow$ $\backslash$ End width $\rightarrow$	$7$	$10$	$25$	$50$	$75$	$100$
$7$	$\mathbf{39.88{\pm}0.36}$	$41.65{\pm}0.12$	$\mathbf{43.83{\pm}0.02}$	$\mathbf{44.37{\pm}0.15}$	$43.95{\pm}0.42$	$43.91{\pm}0.36$
$10$	$39.64{\pm}0.25$	$\mathbf{41.88{\pm}0.36}$	$43.59{\pm}0.44$	$43.96{\pm}0.19$	$\mathbf{44.18{\pm}0.26}$	$43.90{\pm}0.47$
$25$	$38.94{\pm}0.30$	$41.30{\pm}0.50$	$43.41{\pm}0.51$	$43.50{\pm}0.34$	$43.57{\pm}0.07$	$43.74{\pm}0.53$
$50$	$38.84{\pm}0.44$	$41.14{\pm}0.41$	$43.05{\pm}0.32$	$43.43{\pm}0.29$	$43.74{\pm}0.25$	$43.59{\pm}0.34$
$75$	$38.48{\pm}0.53$	$41.32{\pm}0.59$	$43.19{\pm}0.43$	$43.70{\pm}0.23$	$43.12{\pm}0.46$	$\mathbf{43.95{\pm}0.17}$
$100$	$39.27{\pm}0.09$	$41.19{\pm}0.62$	$43.33{\pm}0.13$	$43.75{\pm}0.37$	$43.68{\pm}0.25$	$43.24{\pm}0.54$

Appendix

Table 10: Experiment Four ( App. K.4 ) shows CIFAR-10 test accuracy for scheduled neuroadaptation between two set widths,

m\!\rightarrow\!n

Start width $\downarrow$ $\backslash$ End width $\rightarrow$	$32$	$128$	$256$	$384$	$512$	$640$
$32$	$\mathbf{12.42{\pm}0.15}$	$\mathbf{11.85{\pm}0.16}$	$11.44{\pm}0.11$	$11.53{\pm}0.20$	$11.37{\pm}0.28$	$11.26{\pm}0.10$
$128$	$10.26{\pm}0.13$	$11.25{\pm}0.28$	$11.09{\pm}0.12$	$11.35{\pm}0.14$	$11.31{\pm}0.13$	$11.29{\pm}0.17$
$256$	$10.17{\pm}0.18$	$11.18{\pm}0.03$	$10.68{\pm}0.05$	$11.45{\pm}0.05$	$\mathbf{11.47{\pm}0.31}$	$11.35{\pm}0.26$
$384$	$10.27{\pm}0.18$	$11.02{\pm}0.19$	$11.40{\pm}0.26$	$10.76{\pm}0.13$	$11.30{\pm}0.23$	$11.57{\pm}0.10$
$512$	$10.33{\pm}0.07$	$11.11{\pm}0.10$	$\mathbf{11.49{\pm}0.17}$	$11.33{\pm}0.05$	$10.81{\pm}0.18$	$\mathbf{11.87{\pm}0.09}$
$640$	$10.69{\pm}0.18$	$11.42{\pm}0.16$	$11.33{\pm}0.20$	$\mathbf{11.61{\pm}0.16}$	$11.45{\pm}0.09$	$11.06{\pm}0.20$

Appendix

Table 11: Experiment Four ( App. K.4 ) shows Caltech256 64

\times

64 test accuracy for scheduled neuroadaptation between two set widths,

m\!\rightarrow\!n

. This generally indicates an adaption benefit when

m\!\neq\!n

Architecture	Acc. before	$\Delta a$ ( $\!\times\!10^{-2}\,\mathrm{pp}$ )	Params before	Params after	% Reduced	$\epsilon_{\mathrm{rel}}$ ( $\!\times\!10^{-2}\%$ )	$D_{\mathrm{KL}}$ ( $\!\times\!10^{-8}$ )
$[12288,500\!\times\!2,256]$	$15.8\!\pm\!0.1$	$+0.0\!\pm\!0.0$	$6.52\mathrm{M}$	$6.26\mathrm{M}$	$4.1\%$	$0.6\!\pm\!0.0$	$0.4\!\pm\!0.1$
$[12288,500\!\times\!3,256]$	$15.2\!\pm\!0.1$	$+0.0\!\pm\!0.0$	$6.77\mathrm{M}$	$6.51\mathrm{M}$	$3.9\%$	$0.6\!\pm\!0.0$	$0.8\!\pm\!0.1$
$[12288,500\!\times\!4,256]$	$14.0\!\pm\!0.2$	$-0.3\!\pm\!0.3$	$7.02\mathrm{M}$	$6.51\mathrm{M}$	$7.3\%$	$0.9\!\pm\!0.1$	$1.5\!\pm\!0.5$
$[12288,500\!\times\!5,256]$	$12.7\!\pm\!0.1$	$+0.3\!\pm\!0.3$	$7.27\mathrm{M}$	$6.76\mathrm{M}$	$7.0\%$	$0.5\!\pm\!0.0$	$0.5\!\pm\!0.1$
$[12288,500\!\times\!6,256]$	$11.0\!\pm\!0.3$	$-0.3\!\pm\!0.3$	$7.52\mathrm{M}$	$6.76\mathrm{M}$	$10.1\%$	$0.9\!\pm\!0.1$	$1.7\!\pm\!0.4$
$[12288,1000\!\times\!2,256]$	$15.6\!\pm\!0.2$	$-0.3\!\pm\!0.3$	$13.54\mathrm{M}$	$12.50\mathrm{M}$	$7.7\%$	$1.2\!\pm\!0.1$	$1.9\!\pm\!0.2$

Appendix

Table 12: Post-training sparsification of Caltech-256 classifiers. Architectures are notated compactly as

w\!\times\!k

for

k

hidden layers of width

w

\Delta a

is post-sparsification minus pre-sparsification accuracy in percentage points. Parameters count non-zero parameters, using a numerical threshold of

|\theta|>0.0001

\epsilon_{\mathrm{rel}}

is Eqn. 58 , and

D_{\mathrm{KL}}

is Eqn. 59 . Values reported as mean

\pm

standard error over 5 repeats.

Architecture	Acc. before	$\Delta a$ ( $\!\times\!10^{-2}\,\mathrm{pp}$ )	Params before	Params after	% Reduced	$\epsilon_{\mathrm{rel}}$ ( $\!\times\!10^{-2}\%$ )	$D_{\mathrm{KL}}$ ( $\!\times\!10^{-8}$ )
$[3072,500\!\times\!2,100]$	$19.0\!\pm\!0.1$	$+0.2\!\pm\!0.2$	$1.84\mathrm{M}$	$1.58\mathrm{M}$	$13.7\%$	$0.7\!\pm\!0.0$	$0.6\!\pm\!0.0$
$[3072,500\!\times\!3,100]$	$18.9\!\pm\!0.1$	$-0.2\!\pm\!0.4$	$2.09\mathrm{M}$	$1.84\mathrm{M}$	$12.1\%$	$1.2\!\pm\!0.1$	$2.5\!\pm\!0.3$
$[3072,500\!\times\!4,100]$	$17.5\!\pm\!0.2$	$-0.8\!\pm\!0.6$	$2.34\mathrm{M}$	$1.84\mathrm{M}$	$21.4\%$	$1.5\!\pm\!0.1$	$3.5\!\pm\!0.4$
$[3072,500\!\times\!5,100]$	$15.2\!\pm\!0.3$	$+0.4\!\pm\!0.4$	$2.59\mathrm{M}$	$2.09\mathrm{M}$	$19.4\%$	$1.3\!\pm\!0.1$	$2.9\!\pm\!0.2$
$[3072,500\!\times\!6,100]$	$14.0\!\pm\!0.1$	$+0.2\!\pm\!0.5$	$2.84\mathrm{M}$	$2.09\mathrm{M}$	$26.4\%$	$1.4\!\pm\!0.1$	$3.2\!\pm\!0.5$
$[3072,1000\!\times\!2,100]$	$18.9\!\pm\!0.1$	$+0.4\!\pm\!0.2$	$4.17\mathrm{M}$	$3.17\mathrm{M}$	$24.1\%$	$1.2\!\pm\!0.0$	$1.7\!\pm\!0.1$

Appendix

Table 13: Post-training sparsification of CIFAR-100 classifiers. Architectures are notated compactly as

w\!\times\!k

for

k

hidden layers of width

w

\Delta a

is post-sparsification minus pre-sparsification accuracy in percentage points. Parameters count non-zero parameters, using a numerical threshold of

|\theta|>0.0001

\epsilon_{\mathrm{rel}}

is Eqn. 58 , and

D_{\mathrm{KL}}

is Eqn. 59 . Values reported as mean

\pm

standard error over 5 repeats.

Architecture	Acc. before	$\Delta a$ ( $\!\times\!10^{-2}\,\mathrm{pp}$ )	Params before	Params after	% Reduced	$\epsilon_{\mathrm{rel}}$ ( $\!\times\!10^{-2}\%$ )	$D_{\mathrm{KL}}$ ( $\!\times\!10^{-8}$ )
$[3072,500\!\times\!2,10]$	$43.1\!\pm\!0.2$	$-0.4\!\pm\!0.2$	$1.79\mathrm{M}$	$1.54\mathrm{M}$	$14.0\%$	$1.3\!\pm\!0.2$	$1.3\!\pm\!0.4$
$[3072,500\!\times\!3,10]$	$44.9\!\pm\!0.2$	$-0.2\!\pm\!0.4$	$2.04\mathrm{M}$	$1.79\mathrm{M}$	$12.3\%$	$2.8\!\pm\!0.2$	$6.0\!\pm\!0.7$
$[3072,500\!\times\!4,10]$	$45.4\!\pm\!0.1$	$+0.0\!\pm\!0.3$	$2.29\mathrm{M}$	$1.79\mathrm{M}$	$21.8\%$	$2.3\!\pm\!0.1$	$5.6\!\pm\!0.7$
$[3072,500\!\times\!5,10]$	$45.1\!\pm\!0.2$	$+0.4\!\pm\!0.5$	$2.54\mathrm{M}$	$2.04\mathrm{M}$	$19.7\%$	$1.8\!\pm\!0.2$	$3.3\!\pm\!0.6$
$[3072,500\!\times\!6,10]$	$42.8\!\pm\!0.3$	$+0.6\!\pm\!0.6$	$2.79\mathrm{M}$	$2.04\mathrm{M}$	$26.9\%$	$2.2\!\pm\!0.5$	$4.4\!\pm\!2.3$
$[3072,1000\!\times\!2,10]$	$43.2\!\pm\!0.2$	$-0.2\!\pm\!0.5$	$4.08\mathrm{M}$	$3.08\mathrm{M}$	$24.6\%$	$1.8\!\pm\!0.2$	$2.2\!\pm\!0.6$

Appendix

Table 14: Post-training sparsification of CIFAR-10 classifiers. Architectures are notated compactly as

w\!\times\!k

for

k

hidden layers of width

w

\Delta a

is post-sparsification minus pre-sparsification accuracy in percentage points. Parameters count non-zero parameters, using a numerical threshold of

|\theta|>0.0001

\epsilon_{\mathrm{rel}}

is Eqn. 58 , and

D_{\mathrm{KL}}

is Eqn. 59 . Values reported as mean

\pm

standard error over 5 repeats.

Architecture	MNIST		EMNIST_AJ		FMNIST		CIFAR-10		CIFAR-100		CALTECH-256
	Standard	Isotropic	Standard	Isotropic	Standard	Isotropic	Standard	Isotropic	Standard	Isotropic	Standard	Isotropic
$500{\times}2$	$\boxed{\mathbf{97.18{\pm}0.03}}$	$94.38{\pm}0.04$	$\mathbf{93.26{\pm}0.09}$	$86.69{\pm}0.20$	$\mathbf{87.58{\pm}0.23}$	$85.69{\pm}0.14$	$\mathbf{44.70{\pm}0.11}$	$44.16{\pm}0.09$	$\boxed{\mathbf{18.70{\pm}0.07}}$	$18.60{\pm}0.11$	$\boxed{\mathbf{14.22{\pm}0.19}}$	$12.88{\pm}0.16$
$500{\times}3$	$\mathbf{97.07{\pm}0.04}$	$95.28{\pm}0.11$	$\mathbf{93.24{\pm}0.02}$	$87.81{\pm}0.18$	$\boxed{\mathbf{87.77{\pm}0.09}}$	$\boxed{86.13{\pm}0.16}$	$\boxed{45.19{\pm}0.13}$	$\mathbf{45.36{\pm}0.21}$	$18.62{\pm}0.15$	$\boxed{\mathbf{19.62{\pm}0.10}}$	$13.45{\pm}0.16$	$\boxed{\mathbf{13.63{\pm}0.12}}$
$500{\times}4$	$\mathbf{96.83{\pm}0.08}$	$95.25{\pm}0.09$	$\mathbf{93.26{\pm}0.05}$	$88.32{\pm}0.06$	$\mathbf{86.96{\pm}0.13}$	$86.04{\pm}0.25$	$44.83{\pm}0.18$	$\mathbf{45.92{\pm}0.16}$	$17.23{\pm}0.15$	$\mathbf{18.91{\pm}0.10}$	$12.63{\pm}0.21$	$\mathbf{12.82{\pm}0.15}$
$500{\times}5$	$\mathbf{96.61{\pm}0.08}$	$\boxed{95.37{\pm}0.07}$	$\boxed{\mathbf{93.27{\pm}0.09}}$	$\boxed{89.80{\pm}0.20}$	$\mathbf{86.66{\pm}0.30}$	$85.77{\pm}0.32$	$43.49{\pm}0.28$	$\mathbf{45.95{\pm}0.34}$	$15.97{\pm}0.09$	$\mathbf{17.22{\pm}0.05}$	$11.66{\pm}0.38$	$\mathbf{12.41{\pm}0.09}$
$1000{\times}2$	$\mathbf{96.98{\pm}0.05}$	$94.33{\pm}0.06$	$\mathbf{93.15{\pm}0.02}$	$86.62{\pm}0.17$	$\mathbf{86.66{\pm}0.27}$	$85.62{\pm}0.15$	$42.18{\pm}0.34$	$\mathbf{44.19{\pm}0.08}$	$17.74{\pm}0.06$	$\mathbf{18.68{\pm}0.10}$	$12.71{\pm}0.12$	$\mathbf{12.93{\pm}0.05}$

Appendix

Table 15: Experiment Six: displays the final test-set accuracy for standard-tanh and isotropic-tanh for otherwise identical multilayer perceptrons across various datasets and architectures. Values are reported as mean

\pm

standard error across

5

repeats. Boxed indicates the column-wise highest accuracy (e.g., best isotropic-tanh performance across architectures, and likewise for standard-tanh). Bold indicates the pairwise highest accuracy (e.g., which was best between standard-tanh and isotropic-tanh).

Dynamic Neural Networks

Dynamic neural networks adapt their computation, capacity, structure, or internal state to changing inputs, tasks, and resource budgets.

See all 84 papers

Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability
Jun 3, 2026Vincent Bürgin, Daniel Herbst, Ya-Wei Eileen Lin +1Neural Representation Geometry
Technical University of Munich; School of CIT, MCML, MDSI · MIT; Dept. of EECS, CSAIL
DADP: Dynamic Activity-Dependent Pruning, A Reverse Hebbian-Inspired Structural Pruning Method
Oct 8, 2026Bhushan DeshpandeDynamic Sparse Training Neural Network Pruning
Independent Researcher, Pune, India
Spectral structural distortion reveals redundant neurons in neural networks
May 14, 2026Yongyu WangLLM Pruning Neural Network Compression

Jun 3, 2026J/K move · Enter open · S save

Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability

Vincent Bürgin, Daniel Herbst, Ya-Wei Eileen Lin, Stefanie Jegelka

Technical University of Munich; School of CIT, MCML, MDSI · MIT; Dept. of EECS, CSAIL

Neural Representation Geometry

Open paper AskPDF arXiv

Many striking phenomena in deep learning, such as linear mode connectivity and the structured behavior of training dynamics, are closely tied to parameter symmetries: transformations that leave the realized function unchanged. Despite growing attention to parameter symmetries, the exact interplay between parameters, data, and representations remains underexplored. To investigate this, we develop a theoretical framework of effective function classes, i.e., the set of functions a neuron can realize on its input support, and the norm cost of realizing them. We then formalize effective symmetry breaking via neuron identifiability across independent training runs. Our analysis shows that neural networks can admit large families of approximately equivalent solutions even in structurally asymmetric models. We further show that neuron identifiability enables representation merging without prior alignment, and characterize when such merging admits a linear low-loss path. These findings highlight the role of effective function classes in affecting the loss landscape.

Abstract

Dynamic Neural Networks

Explore similar work

Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability

DADP: Dynamic Activity-Dependent Pruning, A Reverse Hebbian-Inspired Structural Pruning Method

Spectral structural distortion reveals redundant neurons in neural networks