NGN: Learning Neural Network Size as a Differentiable Count

Lixing Li

cs.LGSep 23, 2026

NGN: Learning Neural Network Size as a Differentiable Count

Authors: Lixing Li

Organizations: Cornell University Ithaca, NY 14853

Abstract

Neural network size is usually chosen before training, separating architecture selection from weight optimization. We introduce the Neurogenesis Network (NGN), a differentiable parameterization for learning how many ordered structural components a model should use. For each ordered component group, one learnable boundary selects an active prefix while the model parameters are trained. The boundary can grow from a compact initialization and can be deployed by discarding components beyond the learned boundary. Controlled experiments examine convergence of the learned boundary, the performance of deployed prefixes, and comparisons with fixed-size models and alternative approaches to learning capacity. We then apply the same mechanism to MLPs, convolutional and graph networks, Transformers, state-space models, LoRA, and adapters. Across these settings, deploying only the learned prefix usually changes performance little, and the selected architectures perform similarly to fixed models trained at the same size. These results show that structural capacity can be optimized directly as a count.

Figures & tables

Figure 1: Baseline setting for the controlled 1D regression. The target is $y^{*}(x)=\sin(3x)+0.6\sin(7x)+0.3\sin(13x)$ on $[-\pi,\pi]$ , and each $f_{k}$ is a tanh MLP. Unless stated otherwise, the NGN baseline uses $w_{k}=0.5^{k}$ , $\lambda_{0}=\alpha_{\max}=10^{-4}$ , linearly decreasing $\lambda$ , increasing $\alpha$ , and $\beta:4\!\to\!15$ . Left: total loss, task MSE, and learned $t$ during training. Right: $\|f_{k}\|_{2}$ for each function. Final $t=7.00$ , soft MSE $=6.27e-06$ , and truncation MSE $=6.28e-06$ .

Figure 2: $\lambda$ sweep from $10^{-6}$ to $10^{-2}$ over 500 models, with the other NGN settings fixed to the baseline. Increasing $\lambda$ trades task fit for a smaller count. The learned boundary forms integer plateaus, showing that snapping remains effective across the sweep.

Schedule robustness
setting	$t$	integer	soft MSE	truncated MSE
baseline	6.00	100%	1.31e-05	1.31e-05
$\lambda$ const	5.45	0%	1.84e-05	1.84e-05
$\alpha=10^{-4}$ const	7.00	100%	1.41e-05	1.41e-05
Component ablations
$\alpha=0$	4.49	0%	5.43e-05	4.83e-04

Table 1: Schedule ablation on the 1D task (10 seeds per setting; medians). The upper block tests robustness to schedule shape; the lower block removes or freezes components involved in truncation or growth. “Integer” is the percentage of final boundaries within

0.01

of an integer. Full mean

\,\pm\,

standard deviation results are in Table 6 .

Figure 3: Grown NGN models and fixed-size prefixes trained from scratch at matched retained counts over 50 seeds. Lines show geometric mean MSE, and asymmetric error bars show one standard deviation in $\log_{10}(\mathrm{MSE})$ . The two frontiers are similar, with fixed models typically slightly better at the same count.

Figure 4: Capacity-learning comparison over 50 seeds and 25,000 epochs per seed. Each point is one seed’s deployed MSE against retained count.

Figure 5: Grow/prune stability over $t_{\text{init}}\in[0,16]$ , with 50 seeds per initialization. Lines show mean final $t$ , and error bars indicate one standard deviation. With geometric weights ( $r=0.5$ , left) and delayed $\alpha$ , grow and prune initializations converge to a similar range. Uniform weights ( $r=1$ , middle) and constant $\alpha$ from the start (right) fail to prune reliably.

true $D$	learned $D$	exact rate	soft NMSE	trunc. NMSE
1	1.00	100%	9.94e-07	9.94e-07
2	2.00	100%	2.39e-06	2.39e-06
4	4.00	100%	1.14e-05	1.14e-05
6	6.00	100%	5.55e-05	5.55e-05
8	8.00	90%	1.57e-04	1.57e-04

Table 2: Capacity tracking across nonlinear manifolds embedded in the 16-dimensional ambient space, over ten optimization seeds (medians; full mean and standard deviation results are in Table 8 ). “Learned

D

” is the selected number of rank-one bottleneck terms after truncation, and “exact rate” is the fraction of seeds for which learned

D

equals the true intrinsic dimension. Normalized mean squared error is

\mathrm{NMSE}=\mathbb{E}\lVert\hat{x}-x\rVert_{2}^{2}/\mathbb{E}\lVert x\rVert_{2}^{2}.

task	$f_{k}$	learned $t$	integer	soft	trunc	fixed	prunes back?
MNIST	tanh MLP features	[8.00]	100%	96.93	96.93	96.81	yes
MNIST	conv branches	[4.00]	100%	99.18	99.18	99.13	partial
CIFAR-10	conv branches	[7.02]	50%	71.77	71.77	72.03	partial
Cora	2-hop GCN branches	[1.00]	100%	79.05	79.05	79.45	partial
TinyStories	attention heads ( $\times 2$ layers)	[2.11, 3.03]	0%	1.3055	1.3234	1.3011	no
TinyStories	FFN blocks ( $\times 2$ layers)	[1.14, 8.00]	50%	1.3454	1.3775	1.3402	partial

Table 3: NGN applied across architectures and tasks. Scores are test accuracy (%, higher is better) or TinyStories validation bits/character (lower is better). The four non-language-model rows report medians over ten seeds; the TinyStories rows remain single-seed experiments. “Fixed” retrains the hard architecture at each seed’s learned size; the final column starts from high capacity and reports whether pruning returns to the grow-from-low allocation. Full mean

\,\pm\,

standard deviation results for the ten-seed rows and detailed prune-from-high outcomes are in Table 7 . For each seed, “yes” means that every boundary returns to within one block of its grow-from-low allocation, “partial” means that every boundary moves more than one block below its high-capacity initialization without satisfying that return criterion, and “no” covers the remaining cases. The aggregate row is “yes” or “no” only when every seed has that outcome and “partial” otherwise. “Integer” is the percentage of final learned boundaries within

0.01

of an integer.

construction / setting	learned $t$	depth	soft MSE	trunc. MSE	fixed MSE
per-layer: const- $\lambda$ , const- $r$	[3,4,4,3,3,1,3,2,2,6]	10	2.28e-02	2.28e-02	1.72e-02
per-layer: const- $\lambda$ , inc- $r$	[4,2,3,3,1,3,2,3,4,9]	10	7.37e-04	7.74e-04	6.98e-04
per-layer: dec- $\lambda$ , const- $r$	[0,0,0,4,5,5,5,8,9,9]	7	1.57e-02	1.58e-02	3.28e-02
per-layer: dec- $\lambda$ , inc- $r$	[0,0,0,0,2,3,7,9,9,9]	6	2.51e-04	3.30e-04	3.78e-03
recursive	7.02	8	2.49e-05	3.56e-05	1.54e-05

Table 4: Four-setting sweep and a recursive count construction for depth on the iterated 2D twist task. Per-layer rows learn one boundary

t_{\ell}

at each of ten residual layers; the recursive row learns one global boundary over a chain of at most sixteen functions. For the per-layer construction, const-

r

fixes

r_{\ell}=0.5

, whereas inc-

r

increases

r_{\ell}

linearly from

0.5

in the first layer to

0.75

in the last. Const-

\lambda

uses the same capacity price at every layer, while dec-

\lambda

decreases its initial value geometrically across layers from

10^{-1}

10^{-5}

; every layer then follows the temporal power-decay schedule in Appendix Table 10 . Learned depth counts non-identity layers in the per-layer construction and retained composed functions in the recursive construction.

LoRA rank
task	base	NGN rank (e/m/l)	NGN soft/trunc.	fixed	SoRA	AdaLoRA
WikiSQL-easy	3.5	19 (2/9/8)	78.2 / 78.6	77.7	75.2	45.1
WikiSQL-hard	2.0	21 (4/11/6)	70.8 / 70.4	68.7	67.4	45.4
Spider	1.1	17 (5/7/5)	12.6 / 12.4	12.3	13.9	10.3
GSM8K	45.2	18 (7/8/3)	53.6 / 55.2	50.8	35.6	56.8
SQuAD	65.2	13 (2/7/4)	67.2 / 67.3	67.1	67.1	66.9

Table 5: Learned PEFT capacity on Llama-3-8B-Instruct (single seed per configuration). NGN reports a soft/truncated task score and a fixed model trained with the same rounded per-layer allocation; parentheses split NGN size across early/middle/late layer thirds. SoRA and AdaLoRA target NGN’s learned LoRA rank, but SoRA reaches a lower rank on Spider and GSM8K. NGN scores higher than SoRA on the three exact matches and higher than AdaLoRA on four of five tasks. WikiSQL-easy has one where condition; WikiSQL-hard combines all multi-condition queries with an equally sized seeded sample of single-condition queries. All four LoRA methods use 10 epochs on WikiSQL; the WikiSQL adapter runs use 20 epochs. Integer and forgetting diagnostics are in Appendix E .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Schedule robustness
setting	$t$	soft MSE	truncated MSE
baseline	6.10 $\pm$ 1.45	9.32e-04 $\pm$ 2.3e-03	9.32e-04 $\pm$ 2.3e-03
$\lambda$ const	5.75 $\pm$ 1.83	3.12e-04 $\pm$ 4.9e-04	3.12e-04 $\pm$ 4.9e-04
$\alpha=10^{-4}$ const	7.30 $\pm$ 0.78	2.24e-04 $\pm$ 3.2e-04	2.24e-04 $\pm$ 3.2e-04
Component ablations
$\alpha=0$	4.70 $\pm$ 1.49	3.25e-04 $\pm$ 4.9e-04	1.45e-03 $\pm$ 2.6e-03

Appendix

Table 6: Full schedule-ablation results corresponding to Table 1 ; mean

\,\pm\,

population standard deviation over ten seeds. The baseline MSE statistics are dominated by one high-error seed (

7.82\times 10^{-3}

); the median baseline MSE is

1.31\times 10^{-5}

task	$f_{k}$	learned $t$	soft acc.	trunc. acc.	fixed acc.	prunes back?
MNIST	tanh MLP features	[8.00 $\pm$ 0.00]	96.94 $\pm$ 0.07	96.94 $\pm$ 0.07	96.81 $\pm$ 0.09	yes ( [8.20 $\pm$ 0.40] ; Y/P/N=10/0/0)
MNIST	conv branches	[4.10 $\pm$ 0.30]	99.17 $\pm$ 0.03	99.17 $\pm$ 0.03	99.14 $\pm$ 0.04	partial ( [5.60 $\pm$ 0.49] ; Y/P/N=5/4/1)
CIFAR-10	conv branches	[7.41 $\pm$ 0.49]	71.74 $\pm$ 0.26	71.74 $\pm$ 0.26	72.08 $\pm$ 0.29	partial ( [10.00 $\pm$ 0.00] ; Y/P/N=0/10/0)
Cora	2-hop GCN branches	[1.20 $\pm$ 0.40]	78.68 $\pm$ 1.66	78.68 $\pm$ 1.66	79.43 $\pm$ 1.18	partial ( [2.60 $\pm$ 0.49] ; Y/P/N=5/5/0)

Appendix

Table 7: Full results for the non-language-model rows of Table 3 ; mean

\,\pm\,

population standard deviation over ten seeds. Fixed models are retrained independently at each seed’s rounded learned boundary. The final column reports the aggregate label followed by the prune-from-high boundary and counts of yes/partial/no seed outcomes (Y/P/N), using the definitions in Table 3 . Values are displayed to two decimal places; the MNIST MLP entry

8.00\,\pm\,0.00

has unrounded mean

7.99955

and standard deviation

0.00049

, reflecting consistent convergence to the integer boundary rather than identical runs.

true $D$	learned $D$	exact rate	soft NMSE	trunc. NMSE
1	1.00 $\pm$ 0.00	100%	9.79e-07 $\pm$ 5.7e-08	9.79e-07 $\pm$ 5.7e-08
2	2.00 $\pm$ 0.00	100%	1.39e-05 $\pm$ 1.5e-05	1.36e-03 $\pm$ 2.5e-03
4	4.00 $\pm$ 0.00	100%	2.86e-05 $\pm$ 3.8e-05	1.18e-04 $\pm$ 3.0e-04
6	6.00 $\pm$ 0.00	100%	6.79e-05 $\pm$ 4.3e-05	6.79e-05 $\pm$ 4.3e-05
8	7.90 $\pm$ 0.30	90%	1.25e-03 $\pm$ 2.9e-03	1.25e-03 $\pm$ 2.9e-03

Appendix

Table 8: Full manifold-dimension results corresponding to Table 2 ; mean

\,\pm\,

population standard deviation over ten seeds. The NMSE distributions are right-skewed: three

D=2

seeds and one

D=4

seed have visible soft–truncated gaps, while one

D=8

seed selects seven rather than eight terms and reaches truncated NMSE

9.81\times 10^{-3}

study	$K$	$\lambda_{0}$	$\lambda$ sched.	$\alpha_{\max}$	$\alpha$ sched.	$\beta$	$w_{k}$
1D standard	32	$10^{-4}$	power-decay	$10^{-4}$	delayed linear	$4\!\to\!15$	$0.5^{k}$
1D $\lambda$ sweep	32	$10^{-6}\!:\!10^{-2}$	power-decay	$\lambda_{0}$	delayed linear	$4\!\to\!15$	$0.5^{k}$
initialization sweep	32	$10^{-4}$	power-decay	$10^{-4}$	delayed linear	$4\!\to\!15$	$r^{k}$
schedule ablations	32	$10^{-4}$	Table 1	$10^{-4}$	Table 1	Table 1	$0.5^{k}$
capacity baselines	32	selected	method-specific	method-specific	method-specific	method-specific	method-specific
manifold rank	10	$2\!\times\!10^{-3}$	linear decay	$10^{-3}$	delayed linear	$4\!\to\!12$	$0.5^{k}$

Appendix

Table 9: Architectures and structural hyperparameters for the controlled studies in Section 4 .

task	$K$	$\lambda_{0}$	$\lambda$ sched.	$\alpha_{\max}$	$\alpha$ sched.	$\beta$	$w_{k}$
MNIST MLP	32	$10^{-4}$	power-decay	$10^{-4}$	delayed linear	$4\!\to\!12$	$0.5^{k}$
MNIST / CIFAR-10 CNN	8 / 16	$10^{-4}$	power-decay	$10^{-3}$	delayed linear	$2\!\to\!8$	$0.5^{k}$
Cora	8	$10^{-4}$	power-decay	$10^{-1}$	delayed linear	$4\!\to\!12$	$0.5^{k}$
TinyStories attention	8/layer	$10^{-4}$	power-decay	$10^{-2}$	delayed linear	$2\!\to\!12$	$0.5^{k}$
TinyStories FFN	16/layer	$5\!\times\!10^{-4}$	power-decay	$5\!\times\!10^{-4}$	delayed linear	$4\!\to\!12$	$0.5^{k}$
TinyStories joint	8/16/layer	$5\!\times\!10^{-4}$	power-decay	$5\!\times\!10^{-2}$	delayed linear	$2\!\to\!12$	$0.5^{k}$

Appendix

Table 10: Structural hyperparameters for Section 5 . “Per layer” means that each listed layer has its own count boundary; delayed-linear

\alpha

schedules begin at 50% unless noted. A range

(a\text{--}b)

means that the ramp starts at training fraction

a

and reaches full strength at fraction

b

; a single value gives the start fraction and ramps through the remainder of training.

setting	$r$	$t$	soft MSE	hard MSE	fixed MSE
standard ( $r=0.5$ , learn $t$ )	0.50	7.00	6.27e-06	6.28e-06	6.28e-06
learn $r$ and $t$	0.81 $\pm$ 0.04	5.77 $\pm$ 1.42	4.84e-05 $\pm$ 5.7e-05	3.01e-02 $\pm$ 6.3e-02	8.59e-05 $\pm$ 2.1e-04
learn $t$ , norm., fixed $r=0.5$	0.50	3.42 $\pm$ 1.32	6.36e-02 $\pm$ 4.7e-02	9.03e-02 $\pm$ 1.1e-01	2.30e-05 $\pm$ 2.6e-05
learn $r$ only	0.81 $\pm$ 0.02	–	1.26e-06 $\pm$ 9.8e-07	8.23e+00 $\pm$ 2.3e+00	4.12e-05 $\pm$ 8.5e-05
learn $r$ , normalized	0.89 $\pm$ 0.04	–	2.75e-02 $\pm$ 1.4e-02	2.75e-01 $\pm$ 1.5e-01	4.05e-05 $\pm$ 7.8e-05

Appendix

Table 11: Ablations of the learned-decay and normalization components used by AWNN, evaluated on the common MLP function bank. Alternative rows use ten seeds; the standard row is a single historical run. Its fixed-MSE cell is the hard MSE, not an independent retraining result.

LoRA rank
task	base	size	e/m/l	integer	NGN soft/trunc.	fixed	NGN ppl/base	fixed ppl/base
WikiSQL-easy	3.5	19	2/9/8	20%	78.2 / 78.6	77.7	1.016	0.996
WikiSQL-hard	2.0	21	4/11/6	28%	70.8 / 70.4	68.7	1.022	0.999
Spider	1.1	17	5/7/5	58%	12.6 / 12.4	12.3	0.997	0.984
GSM8K	45.2	18	7/8/3	62%	53.6 / 55.2	50.8	0.968	0.973
SQuAD	65.2	13	2/7/4	34%	67.2 / 67.3	67.1	0.993	0.980

Appendix

Table 12: Full NGN and matched-fixed diagnostics for Table 5 . Size is split across early/middle/late layer thirds; “integer” is the percentage of NGN boundaries within

0.01

of an integer. Perplexity is measured on Pile relative to the base model.

task	method	learned count	e/m/l	score	ppl $/$ base
WikiSQL-easy	NGN	19	2/9/8	78.6	1.016
WikiSQL-easy	SoRA	19	7/12/0	75.2	1.009
WikiSQL-easy	AdaLoRA	19	3/13/3	45.1	1.087
WikiSQL-hard	NGN	21	4/11/6	70.4	1.022
WikiSQL-hard	SoRA	21	9/12/0	67.4	1.000
WikiSQL-hard	AdaLoRA	21	5/14/2	45.4	1.063

Appendix

Table 13: Detailed NGN, SoRA, and AdaLoRA results corresponding to the compact score columns in Table 5 . Size is total retained rank, e/m/l splits it across early/middle/late layer thirds, and forgetting is Pile perplexity relative to the base model. GSM8K scores use the same flexible numerical extraction exact match as Table 5 .

dataset	use	stated license / source
MNIST	image classification	not specified on the original distribution page
CIFAR-10	image classification	not specified on the original distribution page
Cora	graph node classification	not specified by the Planetoid distribution documentation
TinyStories	language modeling	CDLA-Sharing-1.0 on the dataset card
WikiSQL	text-to-SQL	BSD-3-Clause in the official repository
Spider	text-to-SQL	CC BY-SA 4.0 on the dataset card

Appendix

Table 14: Licenses and distribution sources for all datasets used in the reported experiments and forgetting evaluations.

method	kept	e/m/l (%)	score	ppl $\times$	kept	e/m/l (%)	score	ppl $\times$
	task-only				mixed
NGN prune	14.1%	19.9/10.8/11.2	62.5	639	34.2%	36.4/35.8/30.0	70.0	42
random drop	14.1%	19.9/10.8/11.2	17.0	7.0e4	34.2%	36.4/35.8/30.0	17.5	1.6e3
Wanda (per-layer)	14.1%	19.9/10.8/11.2	69.5	356	34.2%	36.4/35.8/30.0	73.0	45

Appendix

Table 15: Direct FFN pruning of Llama-3-8B-Instruct on WikiSQL-easy. NGN learns the kept dimension; random removal is a matched-budget control, and Wanda ranks weights using weight magnitude and activation statistics ( Sun et al., 2024 ) . “Mixed” adds general-domain replay data to the task data to reduce catastrophic forgetting. Task-only and mixed columns use matched budgets; random and Wanda reuse NGN’s per-layer allocation and differ only in which neurons they retain within each layer. “Kept” is the percentage of the original FFN intermediate dimensions retained. The e/m/l entry gives the percentage retained within each of the early, middle, and late layer thirds. Forgetting is Pile perplexity relative to the base model. Among NGN’s final per-layer boundaries, 34% (task-only) and 100% (mixed) land within

0.01

of an integer.

Figure 6: Structured-pruning hyperparameter ablation for Pythia-410M on WikiSQL-easy. Marker size increases as $\lambda$ decreases; filled markers freeze surviving weights and open markers update the full model.

target	backbone	$\lambda$	kept	e/m/l (%)	EM (trunc)	Pile $\times$ base
attention heads	frozen	$10^{-2}$	78.0%	76.3/79.7/77.9	41.5	9.61
attention heads	frozen	$10^{-3}$	91.2%	91.4/91.9/90.4	50.5	1.80
attention heads	frozen	$10^{-4}$	94.9%	94.0/94.3/96.4	57.0	1.50
attention heads	updated	$10^{-2}$	72.9%	69.8/76.0/72.9	37.0	561.01
attention heads	updated	$10^{-3}$	84.7%	85.2/86.7/82.3	54.5	219.20
attention heads	updated	$10^{-4}$	89.3%	89.1/92.2/86.7	63.0	220.37

Appendix

Table 16: Numerical results for all frozen- and updated-backbone runs in the pruning hyperparameter ablation of Figure 6 (10 epochs). “Kept” is the percentage of the full 302M attention-plus-FFN parameter budget retained, and e/m/l gives the retained percentage within each layer third. We also report truncated exact match and Pile perplexity relative to the base model.

Figure 7: Continue-learning. A model is trained to convergence (stage 1, shared), then training continues with only $\lambda$ changed. After the stage-2 boundary (dashed), larger $\lambda$ produces stronger pruning, while $0.1\times$ the standard price preserves the original plateau rather than recruiting new capacity.

Explore similar work

On the Stability of Growth in Structural Plasticity
May 14, 2026Lute Lillo, Nick CheneyPlasticity Growth
University of Vermont
Every Feedforward Neural Network Definable in an o-Minimal Structure Has Finite Sample Complexity
May 8, 2026Anastasis Kratsios, Gregory Cousins, Haitz Sáez de Ocáriz Borde +2Feed-Forward Learnability
Department of Mathematics & Statistics, McMaster University, Hamilton, ON, Canada · University of Cambridge, United Kingdom · Graduate School of Engineering, The University of Tokyo, Tokyo, Japan +1
Rethinking Neural Width for Alternating Current Optimal Power Flow Proxies
Jun 2, 2026Dhruvi Khandelwal, Anurag Basistha, Ayushi Jolotia +1Optimal Power Flow Deep Network
Department of Electrical Engineering, National Institute of Technology Kurukshetra, India. · Indraprastha Institute of Information Technology Delhi, India. · Department of Electrical Engineering, Indian Institute of Technology Roorkee, India.

May 14, 2026, cs.LGJ/K move · Enter open · S save

On the Stability of Growth in Structural Plasticity

Lute Lillo, Nick Cheney

University of Vermont

Plasticity Growth

Open paper AskPDF arXiv

Standard deep-learning pipelines usually choose the network architecture before training and keep it fixed throughout optimization. In contrast, a model can also be adapted by editing its structure during training, for example by pruning existing hidden-neuron units or growing new ones. Although growth is appealing for adaptive and continual systems, we show that it is not simply the inverse of pruning. Pruning selects among units that have participated in training from the start, whereas growth inserts new units into an already specialized optimization trajectory. We isolate this insertion problem and show that newborn units are often forward-active but backward-starved: they participate in the forward computation, yet receive much weaker gradient signal than incumbent units. This disadvantage is minor in small MLP benchmarks, but becomes clear in harder image-classification settings with a convolutional trunk. In these settings, \textsc{Grow} can achieve high final accuracy during the structural-editing procedure, while \textsc{Prune} is stronger when performance is averaged over the training trajectory or when the final sparse network is retrained from scratch. Interventions targeting optimizer state, insertion, selection, and trainability show that improving the integration of newborn units can improve adaptive performance, but does not automatically produce better final subnetworks. In continual-learning benchmarks stressing plasticity loss, \textsc{Grow} becomes competitive mainly when new units have enough time to integrate. Together, these results suggest that \textsc{Grow} should be evaluated not only as an architecture-search operator, but as a time-sensitive optimization process whose success depends on insertion stability.

Abstract

Explore similar work

On the Stability of Growth in Structural Plasticity

Every Feedforward Neural Network Definable in an o-Minimal Structure Has Finite Sample Complexity

Rethinking Neural Width for Alternating Current Optimal Power Flow Proxies