Varying the number of neurons while preserving function is difficult because neural networks' dense connectivity creates strong computational interdependence between units. Elementwise activation functions, particularly their limited equivariance properties, are identified as a central hurdle to achieving this. We propose activation functions designed around chosen equivariances, enabling dense neural networks to adapt via singular value decomposition of their affine maps. This diagonalises layers into one-to-one, ordered connections, making it easier to assess the impact of individual connections on function. Low-impact neurons can be removed (neurodegeneration), and a thresholded buffer of largely inactive scaffold' neurons is maintained (neurogenesis). These symmetry-led diagonalisation and structural changes prioritise function-invariance: neurogenesis exactly preserves function, neurodegeneration is asymptotically function-preserving with the pruned singular-value threshold. Moreover, this construction enables asymptotic 50\% parameter sparsification of dense networks while preserving function analytically. Thus, the approach enables real-time restructuring of the architecture in response to task demands, task appending, removal or changes. Conceptually, this motivates broader applications of symmetry prescriptions at the small scale, even when no data-driven model-scale symmetry adherence is required. Following this, we build activations around symmetry, not from classical assumptions about individual neurons. Unlike typical elementwise forms, these alternatives feature explicit basis independence and do not individuate neurons. We leverage this to decompose and reinterpret neurons across sequential layers, simplifying structure and providing a natural route to adaptive topologies. Additional developments include a new tunable parameter, the intrinsic length'.
Figures & tables
Figure 1: Illustrates the qualitative effects on a network undergoing full diagonalisation. The connectivity is simplified to a one-to-one correspondence between neurons on the chosen layer.
Neuroadaptation
onone,ψdecayΣ0=240
oconstant,ψdecayΣ0=240
otrainable,ψdecayΣ0=241
otrainable,ψnoneΣ0=240
otrainable,ψconstantΣ0=237
otrainable,ψtrainableΣ0=237
otrainable,badjustedΣ0=240
Σ1
ϵrel (%)
Σ1
ϵrel (%)
Σ1
ϵrel (%)
Σ1
ϵrel (%)
Σ1
ϵrel (%)
Σ1
ϵrel (%)
Σ1
ϵrel (%)
+75
240
(8.4±0.2)×10−4
240
(8.5±0.2)×10−4
241
(8.2±0.2)×10−4
240
(8.6±0.1)×10−4
237
(8.5±0.2)×10−4
237
(8.3±0.2)×10−4
240
(8.3±0.1)×10−4
+50
240
(8.4±0.2)×10−4
240
(8.5±0.2)×10−4
241
(8.2±0.2)×10−4
240
(8.6±0.1)×10−4
237
(8.5±0.2)×10−4
237
(8.3±0.2)×10−4
240
(8.3±0.1)×10−4
+25
240
(8.4±0.2)×10−4
240
(8.5±0.2)×10−4
241
(8.2±0.2)×10−4
240
(8.6±0.1)×10−4
237
(8.5±0.2)×10−4
237
(8.3±0.2)×10−4
240
(8.3±0.1)×10−4
+5
240
(8.4±0.2)×10−4
240
(8.5±0.2)×10−4
241
(8.2±0.2)×10−4
240
(8.6±0.1)×10−4
237
(8.5±0.2)×10−4
237
(8.3±0.2)×10−4
240
(8.3±0.1)×10−4
+1
240
(8.4±0.2)×10−4
240
(8.5±0.2)×10−4
241
(8.2±0.2)×10−4
240
(8.6±0.1)×10−4
237
(8.5±0.2)×10−4
237
(8.3±0.2)×10−4
240
(8.3±0.1)×10−4
Table 1: Experiment One ( App. K.1 ): measures function invariance after neuroadaptation: appending +k , removing −k , or diagonalisation 0 the R100→R100 layer of a [12288,100,100,256] Caltech256 trained classifier. Mean ± standard error (SE) are reported over 20 independent repeats. Columns specify intrinsic length o and the ψ -vector approach, yielding 7×20 total networks. Σ0 and Σ1 denote the before- and after-sums of singular values, respectively (with their SE omitted for legibility; all ≈0.4 ). ϵrel is the test-set mean L2 logit change divided by the initial value, reported as a percentage.
Figure 2: Experiment Two ( App. K.2 ) displays the evolving layerwise singular value and bias spectra for a [3072,100,100,10] fixed isotropic MLP trained on CIFAR-10 [ 47 ] classification. The spectra are obtained from 40 independent repeats, showing strong consistency. Singular values begin at one, indicated by the dashed line, due to orthogonal initialisation, and biases at zero due to initialisation.
Figure 3: Experiment Three ( App. K.3 ) demonstrates thresholded neural adaptation when an architecture of [2352,H1,H2,H3,30] adapts to a changing task schedule, across 25 independent repeats for random hidden widths H1,H2,H3∼U(500,2500) . From epoch 0 through 150 , only MNIST [ 48 ] cross-entropy terms contribute to the loss; at T1 , the total loss is equally averaged between both MNIST and FMNIST [ 49 ] . At T2 , the total loss is equally averaged between MNIST, FMNIST, and A-J letters from EMNIST [ 50 ] , and finally, at T3 , only FMNIST’s loss contributes.
Appendix figures & tables31 assets
Supplementary material from the paper’s appendix.
Appendix
Variant
Functional Invariance
Computational Cost
Q absorbed with full covariance
Exact, under assumptions
Loses diagonal BatchNorm efficiency
Q absorbed with diagonal covariance
Approximate
Retains BatchNorm’s efficiency
Appendix
Table 2: Two primary approaches for a network in the ‘ICA-representation’ with respective trade-offs.
Neuroadaptation
onone,ψdecayΣ0=240
oconstant,ψdecayΣ0=240
otrainable,ψdecayΣ0=241
otrainable,ψnoneΣ0=240
otrainable,ψconstantΣ0=237
otrainable,ψtrainableΣ0=237
otrainable,badjustedΣ0=240
Σ1
ϵrel (%)
Σ1
ϵrel (%)
Σ1
ϵrel (%)
Σ1
ϵrel (%)
Σ1
ϵrel (%)
Σ1
ϵrel (%)
Σ1
ϵrel (%)
+75
240
(8.4±0.2)×10−4
240
(8.5±0.2)×10−4
241
(8.2±0.2)×10−4
240
(8.6±0.1)×10−4
237
(8.5±0.2)×10−4
237
(8.3±0.2)×10−4
240
(8.3±0.1)×10−4
+50
240
(8.4±0.2)×10−4
240
(8.5±0.2)×10−4
241
(8.2±0.2)×10−4
240
(8.6±0.1)×10−4
237
(8.5±0.2)×10−4
237
(8.3±0.2)×10−4
240
(8.3±0.1)×10−4
+25
240
(8.4±0.2)×10−4
240
(8.5±0.2)×10−4
241
(8.2±0.2)×10−4
240
(8.6±0.1)×10−4
237
(8.5±0.2)×10−4
237
(8.3±0.2)×10−4
240
(8.3±0.1)×10−4
+5
240
(8.4±0.2)×10−4
240
(8.5±0.2)×10−4
241
(8.2±0.2)×10−4
240
(8.6±0.1)×10−4
237
(8.5±0.2)×10−4
237
(8.3±0.2)×10−4
240
(8.3±0.1)×10−4
+1
240
(8.4±0.2)×10−4
240
(8.5±0.2)×10−4
241
(8.2±0.2)×10−4
240
(8.6±0.1)×10−4
237
(8.5±0.2)×10−4
237
(8.3±0.2)×10−4
240
(8.3±0.1)×10−4
Appendix
Table 3: Experiment One (Caltech-256): singular-value structure and relative function change under neuroadaptation at layer L=1 of the [12288,100,100,256] classifier. Results are mean ± standard error over 20 independent repeats. Row 0 applies diagonalisation only; +k subsequently applies k neurogenesis operations in a single step and −k applies k neurodegeneration operations in a single step. Σ0 and Σ1 denote the sums of singular values before and after neuroadaptation, respectively. Singular-value standard errors are omitted for legibility; the largest singular-value standard error is 0.38. ϵrel is the Eqn. 58 ’s metric, bold indicates the smallest value within each neuroadaptation condition.
Neuroadaptation
onone,ψdecay
oconstant,ψdecay
otrainable,ψdecay
otrainable,ψnone
otrainable,ψconstant
otrainable,ψtrainable
otrainable,badjusted
DKL
Δa (pp)
DKL
Δa (pp)
DKL
Δa (pp)
DKL
Δa (pp)
DKL
Δa (pp)
DKL
Δa (pp)
DKL
Δa (pp)
+75
(1.0±0.3)×10−9
0.0±0.0
(1.4±0.2)×10−9
0.0±0.0
(1.4±0.3)×10−9
0.0±0.0
(1.2±0.3)×10−9
0.0±0.0
(1.2±0.3)×10−9
0.0±0.0
(1.4±0.3)×10−9
0.0±0.0
(1.5±0.3)×10−9
0.0±0.0
+50
(1.3±0.3)×10−9
0.0±0.0
(9.0±3.2)×10−10
0.0±0.0
(1.3±0.3)×10−9
0.0±0.0
(1.3±0.3)×10−9
0.0±0.0
(1.5±0.3)×10−9
0.0±0.0
(1.2±0.3)×10−9
0.0±0.0
(1.5±0.2)×10−9
0.0±0.0
+25
(1.3±0.3)×10−9
0.0±0.0
(1.1±0.3)×10−9
0.0±0.0
(1.3±0.3)×10−9
0.0±0.0
(1.3±0.3)×10−9
0.0±0.0
(1.5±0.3)×10−9
0.0±0.0
(1.2±0.3)×10−9
0.0±0.0
(1.5±0.2)×10−9
0.0±0.0
+5
(1.3±0.3)×10−9
0.0±0.0
(1.1±0.3)×10−9
0.0±0.0
(1.3±0.3)×10−9
0.0±0.0
(1.3±0.3)×10−9
0.0±0.0
(1.5±0.3)×10−9
0.0±0.0
(1.2±0.3)×10−9
0.0±0.0
(1.5±0.2)×10−9
0.0±0.0
+1
(1.3±0.3)×10−9
0.0±0.0
(1.1±0.3)×10−9
0.0±0.0
(1.2±0.3)×10−9
0.0±0.0
(1.3±0.3)×10−9
0.0±0.0
(1.7±0.3)×10−9
0.0±0.0
(1.2±0.3)×10−9
0.0±0.0
(1.5±0.2)×10−9
0.0±0.0
Appendix
Table 4: Experiment One (Caltech-256): additional function-change measures for the same neuroadaptation conditions. Results are mean ± standard error over 20 independent repeats. DKL is provided by Eqn. 59 and Δa=apost−apre is the change in test accuracy in percentage points; positive values indicate increased accuracy. The baseline row reports the unadapted test accuracy a0 as a percentage. Bold values indicate the smallest mean DKL and the accuracy change closest to zero within each neuroadaptation condition.
Neuroadaptation
onone,ψdecayΣ0=174
oconstant,ψdecayΣ0=174
otrainable,ψdecayΣ0=174
otrainable,ψnoneΣ0=174
otrainable,ψconstantΣ0=164
otrainable,ψtrainableΣ0=164
otrainable,badjustedΣ0=174
Σ1
ϵrel (%)
Σ1
ϵrel (%)
Σ1
ϵrel (%)
Σ1
ϵrel (%)
Σ1
ϵrel (%)
Σ1
ϵrel (%)
Σ1
ϵrel (%)
+75
174
(3.1±0.1)×10−3
174
(2.9±0.1)×10−3
174
(2.9±0.1)×10−3
174
(3.0±0.1)×10−3
164
(2.5±0.1)×10−3
164
(2.7±0.1)×10−3
174
(3.1±0.2)×10−3
+50
174
(3.1±0.1)×10−3
174
(2.9±0.1)×10−3
174
(2.9±0.1)×10−3
174
(3.0±0.1)×10−3
164
(2.5±0.1)×10−3
164
(2.7±0.1)×10−3
174
(3.1±0.2)×10−3
+25
174
(3.1±0.1)×10−3
174
(2.9±0.1)×10−3
174
(2.9±0.1)×10−3
174
(3.0±0.1)×10−3
164
(2.5±0.1)×10−3
164
(2.7±0.1)×10−3
174
(3.1±0.2)×10−3
+5
174
(3.1±0.1)×10−3
174
(2.9±0.1)×10−3
174
(2.9±0.1)×10−3
174
(3.0±0.1)×10−3
164
(2.5±0.1)×10−3
164
(2.7±0.1)×10−3
174
(3.1±0.2)×10−3
+1
174
(3.1±0.1)×10−3
174
(2.9±0.1)×10−3
174
(2.9±0.1)×10−3
174
(3.0±0.1)×10−3
164
(2.5±0.1)×10−3
164
(2.7±0.1)×10−3
174
(3.1±0.2)×10−3
Appendix
Table 5: Experiment One (CIFAR-10): singular-value structure and relative function change under neuroadaptation at layer L=1 of the [3072,100,100,10] classifier. Results are mean ± standard error over 20 independent repeats. Row 0 applies diagonalisation only; +k subsequently applies k neurogenesis operations in a single step and −k applies k neurodegeneration operations in a single step. Σ0 and Σ1 denote the sums of singular values before and after neuroadaptation, respectively. Singular-value standard errors are omitted for legibility; the largest singular-value standard error is 0.32. ϵrel is the Eqn. 58 ’s metric, bold indicates the smallest value within each neuroadaptation condition.
Neuroadaptation
onone,ψdecay
oconstant,ψdecay
otrainable,ψdecay
otrainable,ψnone
otrainable,ψconstant
otrainable,ψtrainable
otrainable,badjusted
DKL
Δa (pp)
DKL
Δa (pp)
DKL
Δa (pp)
DKL
Δa (pp)
DKL
Δa (pp)
DKL
Δa (pp)
DKL
Δa (pp)
+75
(1.3±0.2)×10−9
−0.0±0.0
(1.5±0.2)×10−9
0.0±0.0
(1.2±0.1)×10−9
−0.0±0.0
(1.3±0.2)×10−9
+0.0±0.0
(1.1±0.2)×10−9
0.0±0.0
(1.2±0.1)×10−9
−0.0±0.0
(1.6±0.2)×10−9
+0.0±0.0
+50
(1.4±0.1)×10−9
−0.0±0.0
(1.5±0.2)×10−9
0.0±0.0
(1.5±0.2)×10−9
−0.0±0.0
(1.4±0.2)×10−9
+0.0±0.0
(1.1±0.2)×10−9
0.0±0.0
(1.2±0.1)×10−9
−0.0±0.0
(1.7±0.2)×10−9
+0.0±0.0
+25
(1.4±0.1)×10−9
−0.0±0.0
(1.4±0.2)×10−9
0.0±0.0
(1.3±0.2)×10−9
−0.0±0.0
(1.2±0.1)×10−9
+0.0±0.0
(1.1±0.2)×10−9
0.0±0.0
(1.3±0.2)×10−9
−0.0±0.0
(1.6±0.2)×10−9
+0.0±0.0
+5
(1.4±0.2)×10−9
−0.0±0.0
(1.3±0.2)×10−9
0.0±0.0
(1.3±0.2)×10−9
−0.0±0.0
(1.3±0.1)×10−9
+0.0±0.0
(1.1±0.2)×10−9
0.0±0.0
(1.0±0.2)×10−9
−0.0±0.0
(1.6±0.2)×10−9
+0.0±0.0
+1
(1.3±0.2)×10−9
−0.0±0.0
(1.4±0.3)×10−9
0.0±0.0
(1.3±0.2)×10−9
−0.0±0.0
(1.1±0.2)×10−9
+0.0±0.0
(1.0±0.1)×10−9
0.0±0.0
(1.1±0.1)×10−9
−0.0±0.0
(1.7±0.2)×10−9
+0.0±0.0
Appendix
Table 6: Experiment One (CIFAR-10): additional function-change measures for the same neuroadaptation conditions. Results are mean ± standard error over 20 independent repeats. DKL is provided by Eqn. 59 and Δa=apost−apre is the change in test accuracy in percentage points; positive values indicate increased accuracy. The baseline row reports the unadapted test accuracy a0 as a percentage. Bold values indicate the smallest mean DKL and the accuracy change closest to zero within each neuroadaptation condition.
Neuroadaptation
onone,ψdecayΣ0=260
oconstant,ψdecayΣ0=260
otrainable,ψdecayΣ0=260
otrainable,ψnoneΣ0=260
otrainable,ψconstantΣ0=252
otrainable,ψtrainableΣ0=252
otrainable,badjustedΣ0=260
Σ1
ϵrel (%)
Σ1
ϵrel (%)
Σ1
ϵrel (%)
Σ1
ϵrel (%)
Σ1
ϵrel (%)
Σ1
ϵrel (%)
Σ1
ϵrel (%)
+75
260
(1.7±0.1)×10−3
260
(1.7±0.1)×10−3
260
(1.6±0.1)×10−3
260
(1.6±0.0)×10−3
252
(1.4±0.1)×10−3
252
(1.5±0.1)×10−3
260
(1.7±0.1)×10−3
+50
260
(1.7±0.1)×10−3
260
(1.7±0.1)×10−3
260
(1.6±0.1)×10−3
260
(1.6±0.0)×10−3
252
(1.4±0.1)×10−3
252
(1.5±0.1)×10−3
260
(1.7±0.1)×10−3
+25
260
(1.7±0.1)×10−3
260
(1.7±0.1)×10−3
260
(1.6±0.1)×10−3
260
(1.6±0.0)×10−3
252
(1.4±0.1)×10−3
252
(1.5±0.1)×10−3
260
(1.7±0.1)×10−3
+5
260
(1.7±0.1)×10−3
260
(1.7±0.1)×10−3
260
(1.6±0.1)×10−3
260
(1.6±0.0)×10−3
252
(1.4±0.1)×10−3
252
(1.5±0.1)×10−3
260
(1.7±0.1)×10−3
+1
260
(1.7±0.1)×10−3
260
(1.7±0.1)×10−3
260
(1.6±0.1)×10−3
260
(1.6±0.0)×10−3
252
(1.4±0.1)×10−3
252
(1.5±0.1)×10−3
260
(1.7±0.1)×10−3
Appendix
Table 7: Experiment One (CIFAR-100): singular-value structure and relative function change under neuroadaptation at layer L=1 of the [3072,100,100,100] classifier. Results are mean ± standard error over 20 independent repeats. Row 0 applies diagonalisation only; +k subsequently applies k neurogenesis operations in a single step and −k applies k neurodegeneration operations in a single step. Σ0 and Σ1 denote the sums of singular values before and after neuroadaptation, respectively. Singular-value standard errors are omitted for legibility; the largest singular-value standard error is 0.23. ϵrel is the Eqn. 58 ’s metric, bold indicates the smallest value within each neuroadaptation condition.
Neuroadaptation
onone,ψdecay
oconstant,ψdecay
otrainable,ψdecay
otrainable,ψnone
otrainable,ψconstant
otrainable,ψtrainable
otrainable,badjusted
DKL
Δa (pp)
DKL
Δa (pp)
DKL
Δa (pp)
DKL
Δa (pp)
DKL
Δa (pp)
DKL
Δa (pp)
DKL
Δa (pp)
+75
(1.6±0.2)×10−9
+0.0±0.0
(1.5±0.2)×10−9
+0.0±0.0
(1.2±0.2)×10−9
−0.0±0.0
(1.6±0.2)×10−9
0.0±0.0
(1.2±0.1)×10−9
+0.0±0.0
(1.6±0.2)×10−9
0.0±0.0
(1.4±0.2)×10−9
+0.0±0.0
+50
(1.6±0.2)×10−9
+0.0±0.0
(1.6±0.2)×10−9
+0.0±0.0
(1.2±0.2)×10−9
−0.0±0.0
(1.6±0.2)×10−9
0.0±0.0
(1.1±0.2)×10−9
+0.0±0.0
(1.6±0.2)×10−9
0.0±0.0
(1.4±0.2)×10−9
+0.0±0.0
+25
(1.6±0.2)×10−9
+0.0±0.0
(1.5±0.2)×10−9
+0.0±0.0
(1.2±0.2)×10−9
−0.0±0.0
(1.6±0.2)×10−9
0.0±0.0
(1.2±0.1)×10−9
+0.0±0.0
(1.6±0.2)×10−9
0.0±0.0
(1.4±0.2)×10−9
+0.0±0.0
+5
(1.5±0.1)×10−9
+0.0±0.0
(1.4±0.2)×10−9
+0.0±0.0
(1.2±0.2)×10−9
−0.0±0.0
(1.5±0.2)×10−9
0.0±0.0
(1.2±0.1)×10−9
+0.0±0.0
(1.6±0.2)×10−9
0.0±0.0
(1.5±0.2)×10−9
+0.0±0.0
+1
(1.6±0.2)×10−9
+0.0±0.0
(1.5±0.2)×10−9
+0.0±0.0
(1.2±0.2)×10−9
−0.0±0.0
(1.9±0.2)×10−9
0.0±0.0
(1.2±0.2)×10−9
+0.0±0.0
(1.4±0.2)×10−9
0.0±0.0
(1.4±0.2)×10−9
+0.0±0.0
Appendix
Table 8: Experiment One (CIFAR-100): additional function-change measures for the same neuroadaptation conditions. Results are mean ± standard error over 20 independent repeats. DKL is provided by Eqn. 59 and Δa=apost−apre is the change in test accuracy in percentage points; positive values indicate increased accuracy. The baseline row reports the unadapted test accuracy a0 as a percentage. Bold values indicate the smallest mean DKL and the accuracy change closest to zero within each neuroadaptation condition.
Figure 4: Experiment Two ( App. K.2 ) evolving layerwise singular value and bias spectra for a [3072,100,100,10] isotropic-tanh CIFAR-10 classifier.
Figure 5: Experiment Two ( App. K.2 ) evolving layerwise singular value and bias spectra for a [3072,100,100,10] standard-tanh CIFAR-10 classifier.
Figure 6: Experiment Two ( App. K.2 ) evolving layerwise bias-magnitude spectra for a [3072,100,100,10] isotropic-tanh CIFAR-10 classifier.
Figure 7: Experiment Two ( App. K.2 ) evolving layerwise bias-magnitude spectra for a [3072,100,100,10] standard-tanh CIFAR-10 classifier.
Figure 8: Experiment Two ( App. K.2 ) evolving layerwise singular value and bias spectra for a [3072,500,500,10] isotropic-tanh CIFAR-10 classifier.
Figure 9: Experiment Two ( App. K.2 ) evolving layerwise singular value and bias spectra for a [3072,1000,1000,10] isotropic-tanh CIFAR-10 classifier.
Figure 10: Experiment Two ( App. K.2 ) evolving layerwise bias-magnitude spectra for a [3072,1000,1000,10] isotropic-tanh CIFAR-10 classifier.
Figure 11: Experiment Two ( App. K.2 ) evolving layerwise singular value and bias spectra for a [3072,500,500,10] standard-tanh CIFAR-10 classifier.
Figure 12: Experiment Two ( App. K.2 ) evolving layerwise singular value and bias spectra for a [3072,1000,1000,10] standard-tanh CIFAR-10 classifier.
Figure 13: Experiment Two ( App. K.2 ) evolving layerwise singular value and bias spectra for a [12288,100,100,256] isotropic-tanh Caltech256 classifier.
Figure 14: Experiment Two ( App. K.2 ) evolving layerwise singular value and bias spectra for a [12288,500,500,256] isotropic-tanh Caltech256 classifier.
Figure 15: Experiment Two ( App. K.2 ) evolving layerwise singular value and bias spectra for a [12288,1000,1000,256] isotropic-tanh Caltech256 classifier.
Figure 16: Experiment Two ( App. K.2 ) evolving layerwise bias-magnitude spectra for a [12288,100,100,256] isotropic-tanh Caltech256 classifier.
Figure 17: Experiment Two ( App. K.2 ) evolving layerwise bias-magnitude spectra for a [12288,1000,1000,256] isotropic-tanh Caltech256 classifier.
Epoch range
Active dataset(s)
0 – 149
MNIST
150 – 299
MNIST + Fashion-MNIST
300 – 449
MNIST + Fashion-MNIST + EMNIST A–J
450 – 599
Fashion-MNIST
Appendix
Table 9: Task schedule used in Experiment Three.
Figure 18: Experiment Three ( App. K.3 ) of a single idealistic example
Figure 19: Experiment Three ( App. K.3 ) of a single non-idealistic example
Figure 20: Experiment Three ( App. K.3 ) of a single non-idealistic example
Start width ↓\ End width →
7
10
25
50
75
100
7
39.88±0.36
41.65±0.12
43.83±0.02
44.37±0.15
43.95±0.42
43.91±0.36
10
39.64±0.25
41.88±0.36
43.59±0.44
43.96±0.19
44.18±0.26
43.90±0.47
25
38.94±0.30
41.30±0.50
43.41±0.51
43.50±0.34
43.57±0.07
43.74±0.53
50
38.84±0.44
41.14±0.41
43.05±0.32
43.43±0.29
43.74±0.25
43.59±0.34
75
38.48±0.53
41.32±0.59
43.19±0.43
43.70±0.23
43.12±0.46
43.95±0.17
100
39.27±0.09
41.19±0.62
43.33±0.13
43.75±0.37
43.68±0.25
43.24±0.54
Appendix
Table 10: Experiment Four ( App. K.4 ) shows CIFAR-10 test accuracy for scheduled neuroadaptation between two set widths, m→n .
Start width ↓\ End width →
32
128
256
384
512
640
32
12.42±0.15
11.85±0.16
11.44±0.11
11.53±0.20
11.37±0.28
11.26±0.10
128
10.26±0.13
11.25±0.28
11.09±0.12
11.35±0.14
11.31±0.13
11.29±0.17
256
10.17±0.18
11.18±0.03
10.68±0.05
11.45±0.05
11.47±0.31
11.35±0.26
384
10.27±0.18
11.02±0.19
11.40±0.26
10.76±0.13
11.30±0.23
11.57±0.10
512
10.33±0.07
11.11±0.10
11.49±0.17
11.33±0.05
10.81±0.18
11.87±0.09
640
10.69±0.18
11.42±0.16
11.33±0.20
11.61±0.16
11.45±0.09
11.06±0.20
Appendix
Table 11: Experiment Four ( App. K.4 ) shows Caltech256 64 × 64 test accuracy for scheduled neuroadaptation between two set widths, m→n . This generally indicates an adaption benefit when m=n .
Architecture
Acc. before
Δa ( ×10−2pp )
Params before
Params after
% Reduced
ϵrel ( ×10−2% )
DKL ( ×10−8 )
[12288,500×2,256]
15.8±0.1
+0.0±0.0
6.52M
6.26M
4.1%
0.6±0.0
0.4±0.1
[12288,500×3,256]
15.2±0.1
+0.0±0.0
6.77M
6.51M
3.9%
0.6±0.0
0.8±0.1
[12288,500×4,256]
14.0±0.2
−0.3±0.3
7.02M
6.51M
7.3%
0.9±0.1
1.5±0.5
[12288,500×5,256]
12.7±0.1
+0.3±0.3
7.27M
6.76M
7.0%
0.5±0.0
0.5±0.1
[12288,500×6,256]
11.0±0.3
−0.3±0.3
7.52M
6.76M
10.1%
0.9±0.1
1.7±0.4
[12288,1000×2,256]
15.6±0.2
−0.3±0.3
13.54M
12.50M
7.7%
1.2±0.1
1.9±0.2
Appendix
Table 12: Post-training sparsification of Caltech-256 classifiers. Architectures are notated compactly as w×k for k hidden layers of width w . Δa is post-sparsification minus pre-sparsification accuracy in percentage points. Parameters count non-zero parameters, using a numerical threshold of ∣θ∣>0.0001 . ϵrel is Eqn. 58 , and DKL is Eqn. 59 . Values reported as mean ± standard error over 5 repeats.
Architecture
Acc. before
Δa ( ×10−2pp )
Params before
Params after
% Reduced
ϵrel ( ×10−2% )
DKL ( ×10−8 )
[3072,500×2,100]
19.0±0.1
+0.2±0.2
1.84M
1.58M
13.7%
0.7±0.0
0.6±0.0
[3072,500×3,100]
18.9±0.1
−0.2±0.4
2.09M
1.84M
12.1%
1.2±0.1
2.5±0.3
[3072,500×4,100]
17.5±0.2
−0.8±0.6
2.34M
1.84M
21.4%
1.5±0.1
3.5±0.4
[3072,500×5,100]
15.2±0.3
+0.4±0.4
2.59M
2.09M
19.4%
1.3±0.1
2.9±0.2
[3072,500×6,100]
14.0±0.1
+0.2±0.5
2.84M
2.09M
26.4%
1.4±0.1
3.2±0.5
[3072,1000×2,100]
18.9±0.1
+0.4±0.2
4.17M
3.17M
24.1%
1.2±0.0
1.7±0.1
Appendix
Table 13: Post-training sparsification of CIFAR-100 classifiers. Architectures are notated compactly as w×k for k hidden layers of width w . Δa is post-sparsification minus pre-sparsification accuracy in percentage points. Parameters count non-zero parameters, using a numerical threshold of ∣θ∣>0.0001 . ϵrel is Eqn. 58 , and DKL is Eqn. 59 . Values reported as mean ± standard error over 5 repeats.
Architecture
Acc. before
Δa ( ×10−2pp )
Params before
Params after
% Reduced
ϵrel ( ×10−2% )
DKL ( ×10−8 )
[3072,500×2,10]
43.1±0.2
−0.4±0.2
1.79M
1.54M
14.0%
1.3±0.2
1.3±0.4
[3072,500×3,10]
44.9±0.2
−0.2±0.4
2.04M
1.79M
12.3%
2.8±0.2
6.0±0.7
[3072,500×4,10]
45.4±0.1
+0.0±0.3
2.29M
1.79M
21.8%
2.3±0.1
5.6±0.7
[3072,500×5,10]
45.1±0.2
+0.4±0.5
2.54M
2.04M
19.7%
1.8±0.2
3.3±0.6
[3072,500×6,10]
42.8±0.3
+0.6±0.6
2.79M
2.04M
26.9%
2.2±0.5
4.4±2.3
[3072,1000×2,10]
43.2±0.2
−0.2±0.5
4.08M
3.08M
24.6%
1.8±0.2
2.2±0.6
Appendix
Table 14: Post-training sparsification of CIFAR-10 classifiers. Architectures are notated compactly as w×k for k hidden layers of width w . Δa is post-sparsification minus pre-sparsification accuracy in percentage points. Parameters count non-zero parameters, using a numerical threshold of ∣θ∣>0.0001 . ϵrel is Eqn. 58 , and DKL is Eqn. 59 . Values reported as mean ± standard error over 5 repeats.
Architecture
MNIST
EMNIST_AJ
FMNIST
CIFAR-10
CIFAR-100
CALTECH-256
Standard
Isotropic
Standard
Isotropic
Standard
Isotropic
Standard
Isotropic
Standard
Isotropic
Standard
Isotropic
500×2
97.18±0.03
94.38±0.04
93.26±0.09
86.69±0.20
87.58±0.23
85.69±0.14
44.70±0.11
44.16±0.09
18.70±0.07
18.60±0.11
14.22±0.19
12.88±0.16
500×3
97.07±0.04
95.28±0.11
93.24±0.02
87.81±0.18
87.77±0.09
86.13±0.16
45.19±0.13
45.36±0.21
18.62±0.15
19.62±0.10
13.45±0.16
13.63±0.12
500×4
96.83±0.08
95.25±0.09
93.26±0.05
88.32±0.06
86.96±0.13
86.04±0.25
44.83±0.18
45.92±0.16
17.23±0.15
18.91±0.10
12.63±0.21
12.82±0.15
500×5
96.61±0.08
95.37±0.07
93.27±0.09
89.80±0.20
86.66±0.30
85.77±0.32
43.49±0.28
45.95±0.34
15.97±0.09
17.22±0.05
11.66±0.38
12.41±0.09
1000×2
96.98±0.05
94.33±0.06
93.15±0.02
86.62±0.17
86.66±0.27
85.62±0.15
42.18±0.34
44.19±0.08
17.74±0.06
18.68±0.10
12.71±0.12
12.93±0.05
Appendix
Table 15: Experiment Six: displays the final test-set accuracy for standard-tanh and isotropic-tanh for otherwise identical multilayer perceptrons across various datasets and architectures. Values are reported as mean ± standard error across 5 repeats. Boxed indicates the column-wise highest accuracy (e.g., best isotropic-tanh performance across architectures, and likewise for standard-tanh). Bold indicates the pairwise highest accuracy (e.g., which was best between standard-tanh and isotropic-tanh).
Many striking phenomena in deep learning, such as linear mode connectivity and the structured behavior of training dynamics, are closely tied to parameter symmetries: transformations that leave the realized function unchanged. Despite growing attention to parameter symmetries, the exact interplay between parameters, data, and representations remains underexplored. To investigate this, we develop a theoretical framework of effective function classes, i.e., the set of functions a neuron can realize on its input support, and the norm cost of realizing them. We then formalize effective symmetry breaking via neuron identifiability across independent training runs. Our analysis shows that neural networks can admit large families of approximately equivalent solutions even in structurally asymmetric models. We further show that neuron identifiability enables representation merging without prior alignment, and characterize when such merging admits a linear low-loss path. These findings highlight the role of effective function classes in affecting the loss landscape.
Vincent Bürgin, Daniel Herbst, Ya-Wei Eileen Lin +1
Technical University of Munich; School of CIT, MCML, MDSI · MIT; Dept. of EECS, CSAIL
Modern neural networks are heavily over-parameterized. This redundancy incurs substantial compute and memory overhead during training and inference. Existing pruning methods rely on post-hoc magnitude thresholds or static initialization heuristics. Consequently, they often require manual per-layer sparsity targets or expensive retraining cycles. We propose Dynamic Activity-Dependent Pruning (DADP), a biologically inspired structural plasticity mechanism. During training, DADP measures connection importance via the accumulated product of pre-synaptic activations and post-synaptic error gradients. Using a single global threshold instead of fixed layer budgets, DADP dynamically allocates sparsity across network depth while naturally inducing neuron- and channel-level pruning. Across MLP, VGG-16, ResNet-18, BiLSTM-CRF, and MiniBERT architectures, DADP matches or outperforms Magnitude, SNIP and RigL, retaining 73.67% accuracy (dense baseline: 76.06%) at 99% sparsity on ResNet-18. Finally, matrix-based Shannon entropy and effective rank measurements confirm that DADP preserves latent feature diversity at extreme sparsities without representation collapse.
Overparameterized neural networks often contain many removable neurons, yet what makes a neuron redundant remains poorly understood. Existing pruning criteria commonly rely on local quantities such as weight magnitude, activation strength, or gradient sensitivity, but these measures provide limited insight into the structural role of a neuron in the transformation performed by a layer. Here we show that neuronal redundancy can be characterized by weak participation in the spectral structural distortion induced by layer-wise representation transformations. For each hidden layer of a trained network, we record pre-activation and post-activation hidden states, model neurons as graph nodes, and construct input-side and output-side graphs that describe neuron-level relational structure before and after the layer transformation. We then define a spectral structural importance score that measures the contribution of each neuron to the dominant graph-spectral distortion between these two relational structures. Low-participation neurons are treated as structurally redundant and removed through an iterative pruning process in which scores are recomputed after each structural change. No parameter updates are performed during intermediate pruning rounds; after the target parameter reduction is reached, a single recovery fine-tuning stage is applied to the compact model. Direct ablation analysis and experiments across conventional neural networks, encoder-only Transformers, and decoder-only language models show that this graph-spectral criterion identifies removable neurons and Transformer units while preserving task performance after compression. These results suggest that neural redundancy is not merely a consequence of small weights or weak activations, but can be understood through weak participation in the spectral distortion of layer-wise relational structure.