Biologically Inspired Mechanisms for Facilitating Grokking in Multilayer Perceptrons

Florin Leon

cs.LGAug 28, 2026

Biologically Inspired Mechanisms for Facilitating Grokking in Multilayer Perceptrons

Authors: Florin Leon

Organizations: Faculty of Automatic Control and Computer Engineering “Gheorghe Asachi” Technical University of Ias,i Bd. Mangeron 27, 700050 Ias,i, Romania

Abstract

Grokking is a delayed transition from memorization to generalization that is often accompanied by substantial reorganization of internal representations. This paper studies whether biologically inspired mechanisms, many of which are not commonly incorporated into artificial neural networks, can actively promote this transition by regulating hidden-layer computation at the levels of neuronal activity, response, and effective connectivity. We augment a multilayer perceptron with input gating, structural plasticity, gain modulation, threshold modulation, homeostasis, lateral inhibition, and activation decorrelation, and evaluate these mechanisms through systematic ablations on two established grokking benchmarks: sparse parity and noisy XOR classification. The results show that the mechanisms contribute unequally to generalization. Homeostasis provides the strongest and most consistent benefit, while structural sparsification emerges as the second major mechanism. The remaining biologically inspired mechanisms have smaller or less consistent effects in the present experiments. For both problems, the results support the common principle that explicit regulation of neuron utilization and effective connectivity can improve the emergence of generalizable internal computation. These findings motivate broader investigation of biologically inspired activity regulation and adaptive sparsification, including in large language models, where they may accelerate the development of generalizable representations and reduce the optimization time required for robust generalization.

Figures & tables

Concept	Computation or value	Notation	Description
Input dimension	problem-dependent	$d$	Number of input features.
Hidden dimensions	problem-dependent	$n$	Number of hidden neurons.
Output dimension	problem-dependent	$q$	Number of output neurons.
Layer input	data-dependent	$x\in\mathbb{R}^{d}$	Input to the hidden layer.
Weight	learned	$W\in\mathbb{R}^{n\times d}$	Main hidden-layer weight matrix.
Bias	learned	$b\in\mathbb{R}^{n}$	Hidden-layer bias vector.

Table 1: List of model parameters.

Panel of Figure 1 — Figure 1: Overview of a BioNN hidden layer. The input-derived context controls input gating, gain modulation, and fast threshold modulation. Structural plasticity determines the connectivity mask used in the affine transformation. Gain modulation and the fast and homeostatic thresholds regulate primary activity, which provides the input to lateral inhibition before the final hidden activity is computed. During training, the final activity is used to update the homeostatic threshold and to compute the activation-decorrelation loss. Solid arrows indicate forward or control dependencies, while dashed arrows indicate training-time adaptation or regularization effects.

Experiment	IG	GM	TM	LI	HO	SP	AD	Train 0.9	Test 0.9	Train 1	Test 1	Lifetime Sparsity	Sample Hoyer	Neuron Hoyer	Dead Neuron	Mean Abs Corr	Effective Rank	Specialization	Purity	Centroid Sep	Within Jaccard	Between Jaccard	Jaccard Gap	Activation Sparsity	Structural Sparsity	Active Subnetwork Size
two off inhibition decorrelation	1	1	1	0	1	1	0	12	14	16	17	0.863	0.999	0.855	0.833	0.129	0.007	0.024	0.096	0.671	0.943	0.932	0.011	0.863	0.65	7
two off gain threshold	1	0	0	1	1	1	1	13	15	16	17	0.862	0.991	0.821	0.559	0.066	0.012	0.062	0.268	0.472	0.495	0.488	0.007	0.862	0.65	9
one off threshold	1	1	0	1	1	1	1	12	14	16	18	0.869	0.995	0.829	0.585	0.068	0.010	0.044	0.243	0.518	0.542	0.536	0.006	0.869	0.65	8
two off threshold decorrelation	1	1	0	1	1	1	0	12	14	16	18	0.927	0.999	0.909	0.830	0.089	0.007	0.030	0.106	0.670	0.916	0.896	0.020	0.927	0.65	7
one off decorrelation	1	1	1	1	1	1	0	12	14	16	18	0.934	0.999	0.924	0.841	0.072	0.007	0.053	0.111	0.670	0.914	0.891	0.023	0.934	0.65	7
one off inhibition	1	1	1	0	1	1	1	12	14	16	18	0.899	0.998	0.882	0.811	0.082	0.009	0.023	0.109	0.666	0.708	0.701	0.007	0.899	0.65	7

Table 2: Experimental results for the parity task.

Experiment	IG	GM	TM	LI	HO	SP	AD	Train 0.9	Test 0.9	Train 1	Test 1	Lifetime Sparsity	Sample Hoyer	Neuron Hoyer	Mean Abs. Corr.	Effective Rank	Specialization	Purity	Centroid Sep.	Within Jaccard	Between Jaccard	Jaccard Gap	Activation Sparsity
all on	1	1	1	1	1	1	1	99	1691	219	2310	0.832	0.823	0.745	0.139	0.197	0.720	0.858	28.779	0.392	0.170	0.222	0.818
two off threshold inhibition	1	1	0	0	1	1	1	96	1692	213	2463	0.845	0.828	0.756	0.139	0.185	0.745	0.871	29.973	0.401	0.173	0.227	0.820
one off inhibition	1	1	1	0	1	1	1	96	1698	215	2470	0.845	0.827	0.756	0.139	0.185	0.745	0.871	29.900	0.398	0.173	0.226	0.820
two off inhibition structural	1	1	1	0	1	0	1	96	1698	215	2470	0.845	0.827	0.756	0.139	0.185	0.745	0.871	29.900	0.398	0.173	0.226	0.820
one off structural	1	1	1	1	1	0	1	97	1703	215	2475	0.836	0.828	0.747	0.138	0.191	0.725	0.858	29.967	0.385	0.166	0.219	0.821
two off inhibition decorrelation	1	1	1	0	1	1	0	97	1693	214	2481	0.844	0.827	0.756	0.141	0.185	0.754	0.877	29.934	0.404	0.175	0.229	0.819

Table 3: Experimental results for the XOR task.

Experiment	IG	GM	TM	LI	HO	SP	AD	Train 0.9	Test 0.9	Train 1	Test 1	Lifetime Sparsity	Sample Hoyer	Neuron Hoyer	Mean Abs. Corr.	Effective Rank	Specialization	Purity	Centroid Sep.	Within Jaccard	Between Jaccard	Jaccard Gap	Activation Sparsity	Structural Sparsity
two off gain decorrelation	1	0	1	1	1	1	0	160	1427	310	2353	0.840	0.708	0.744	0.104	0.723	0.435	0.632	0.517	0.170	0.068	0.103	0.864	0.65
two off input gate gain	0	0	1	1	1	1	1	158	1445	310	2610	0.840	0.709	0.744	0.102	0.724	0.418	0.623	0.510	0.167	0.069	0.098	0.864	0.65
two off gain inhibition	1	0	1	0	1	1	1	157	1418	314	2818	0.849	0.717	0.752	0.104	0.711	0.444	0.637	0.512	0.166	0.063	0.103	0.864	0.65
one off gain	1	0	1	1	1	1	1	158	1431	307	2966	0.840	0.709	0.744	0.102	0.724	0.429	0.629	0.515	0.169	0.068	0.101	0.864	0.65
one off threshold	1	1	0	1	1	1	1	159	1424	311	3012	0.829	0.847	0.735	0.106	0.466	0.429	0.628	14.762	0.202	0.070	0.132	0.856	0.65
two off gain threshold	1	0	0	1	1	1	1	158	1442	313	3029	0.841	0.709	0.745	0.102	0.724	0.430	0.629	0.514	0.169	0.068	0.101	0.864	0.65

Table 4: Experimental results for the XOR structural density ablation with structural plasticity enabled at

\rho_{m}=0.35

Figure 2: Sparse parity with all bio-inspired mechanisms disabled. Training accuracy improves well before test accuracy, which produces a pronounced delayed generalization phase.

Figure 3: Sparse parity with all bio-inspired mechanisms enabled. Training and test accuracy rise in close succession, which results in substantially faster grokking than in the all-off case.

Figure 4: Sparse parity with only homeostasis and structural plasticity enabled. This best two-mechanism configuration shows rapid generalization and closely reproduces the fast grokking behavior of the fully enabled model.

Figure 5: Sparse parity with only structural plasticity enabled. Generalization remains delayed relative to training, but the transition occurs much earlier than in the all-off case.

Figure 6: Noisy XOR with all bio-inspired mechanisms disabled. Training accuracy eventually reaches 1, but test accuracy remains near chance throughout the experimental horizon, which indicates memorization without generalization.

Figure 7: Noisy XOR with all bio-inspired mechanisms enabled. Training accuracy reaches its maximum early, whereas test accuracy improves progressively over a much longer interval before reaching near-perfect performance. The separation between these two timescales illustrates delayed generalization.

Figure 8: Noisy XOR with only gain modulation and homeostasis enabled. This configuration is the best two-mechanism case according to “Test 1”.

Figure 9: Noisy XOR with only input gating and homeostasis enabled. This configuration is the second-best two-mechanism case according to “Test 1”.

Explore similar work

Sep 9, 2026cs.AI

Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking

Neural networks trained past memorization frequently undergo a delayed transition to generalization, a phenomenon known as grokking. Despite theoretical progress on \emph{why} this transition occurs, the quantitative structure of \emph{when} it occurs in hyperparameter space remains uncharacterized. We map the memorization-to-generalization boundary across 384 configurations of two-hidden-layer MLPs on modular arithmetic, fitting a power-law scaling relation for generalization onset time:

T_{\mathrm{grok}} \propto H^{-0.27}\, D^{-2.04}\, η^{-0.50}\, λ^{-0.64}

(

R^2 = 0.732

;

0.821

with interactions). The exponent hierarchy reveals that data complexity (

D^{-2.04}

) is the dominant driver of regime transition, not model capacity (

H^{-0.27}

): doubling data accelerates generalization by

{\sim}4\times

, while doubling width yields only

{\sim}1.2\times

. A sharp phase boundary at weight decay

λ\gtrsim 1.0

separates grokking from non-grokking configurations, and weight norm trajectories show monotonic compression during the transition, consistent with implicit regularization selecting low-complexity solutions. These results provide a quantitative foundation for predicting and controlling regime transitions in overparameterized networks.

Anish Kataria

Princeton University, Princeton, NJ, USA.

Multilayer Perceptrons Neural Network Memorization Neural Network Training Dynamics Neural Scaling Laws Grokking

Ask PDF

Jul 13, 2026cs.LG

How to Tame Grokking: Representation Geometry as a Control Signal

Grokking is a phenomenon in which neural networks initially memorize training data and only later exhibit strong generalization after prolonged optimization. Despite extensive recent study, the factors influencing the emergence and timing of grokking remain incompletely understood. We investigate the relationship between representation geometry and delayed generalization. We find that dimensionality collapse consistently precedes the onset of grokking in all evaluated settings. Motivated by these observations, we introduce Geometric Dimensionality Regularization (GeomDR), a simple spectral regularizer that modifies the effective dimensionality of hidden representations during training. Across modular addition, modular division, and permutation composition tasks, GeomDR consistently alters grokking dynamics and can substantially accelerate the onset of generalization depending on the intervention schedule and target dimensionality. In several settings, grokking is accelerated by up to 52 times relative to standard AdamW training. Similar qualitative effects are observed in both multilayer perceptrons and transformers. Together, these results suggest that representation geometry can serve as an effective control signal for grokking and provide evidence that geometric interventions offer a practical approach for studying and influencing delayed generalization in neural networks.

Maksim A Kazanskii

Independent Researcher

Neural Network Generalization Spectral Regularization Representation Collapse Neural Representation Geometry Neural Network Training Dynamics Grokking

Ask PDF

Jun 29, 2026stat.ML

A Stochastic--Geometric Theory of Scaling Laws in Grokking

Delayed generalization (\ie~grokking) refers to the phenomenon in which a neural network fits its training data early in training but only begins to generalize after a prolonged delay, often through an abrupt transition. Despite extensive empirical study, its underlying mechanism remains poorly understood. In this work, we first theoretically characterize a shell--core topological configuration of the reachable solution space induced by Adam's optimization dynamics with weight-shrinkage regularization, supported by empirical evidence. This optimization-induced topological configuration gives rise to grokking. In model's parameter space, random initialization solutions concentrate on a thin outer spherical shell, enclosing another spherical shell of memorization solutions, which in turn contains a core corresponding to the generalization solutions. Leveraging stopping-time theory, we then analyze the geometry of this topological configuration and the solution transition time at which optimization trajectories escape the memorization manifold and first reach the boundary of the generalization manifold. Our theoretical analysis derives grokking scaling laws for the learning rate, batch size, and

\ell_2

regularization coefficient, which are further validated through experiments and shown to recover results from prior literature.

Róisín Luo, Christian Gagné, Jonas Ngnawé +2

Research Ireland – Centre for Research Training in AI · University of Galway · Universit´e Laval +2

Neural Network Generalization Neural Network Optimization Neural Network Memorization Neural Scaling Laws Grokking

Ask PDF