Biologically Inspired Mechanisms for Facilitating Grokking in Multilayer Perceptrons

Florin Leon

cs.LGAug 28, 2026

Biologically Inspired Mechanisms for Facilitating Grokking in Multilayer Perceptrons

Authors: Florin Leon

Organizations: Faculty of Automatic Control and Computer Engineering “Gheorghe Asachi” Technical University of Ias,i Bd. Mangeron 27, 700050 Ias,i, Romania

Abstract

Grokking is a delayed transition from memorization to generalization that is often accompanied by substantial reorganization of internal representations. This paper studies whether biologically inspired mechanisms, many of which are not commonly incorporated into artificial neural networks, can actively promote this transition by regulating hidden-layer computation at the levels of neuronal activity, response, and effective connectivity. We augment a multilayer perceptron with input gating, structural plasticity, gain modulation, threshold modulation, homeostasis, lateral inhibition, and activation decorrelation, and evaluate these mechanisms through systematic ablations on two established grokking benchmarks: sparse parity and noisy XOR classification. The results show that the mechanisms contribute unequally to generalization. Homeostasis provides the strongest and most consistent benefit, while structural sparsification emerges as the second major mechanism. The remaining biologically inspired mechanisms have smaller or less consistent effects in the present experiments. For both problems, the results support the common principle that explicit regulation of neuron utilization and effective connectivity can improve the emergence of generalizable internal computation. These findings motivate broader investigation of biologically inspired activity regulation and adaptive sparsification, including in large language models, where they may accelerate the development of generalizable representations and reduce the optimization time required for robust generalization.

Figures & tables

Concept	Computation or value	Notation	Description
Input dimension	problem-dependent	$d$	Number of input features.
Hidden dimensions	problem-dependent	$n$	Number of hidden neurons.
Output dimension	problem-dependent	$q$	Number of output neurons.
Layer input	data-dependent	$x\in\mathbb{R}^{d}$	Input to the hidden layer.
Weight	learned	$W\in\mathbb{R}^{n\times d}$	Main hidden-layer weight matrix.
Bias	learned	$b\in\mathbb{R}^{n}$	Hidden-layer bias vector.

Table 1: List of model parameters.

Panel of Figure 1 — Figure 1: Overview of a BioNN hidden layer. The input-derived context controls input gating, gain modulation, and fast threshold modulation. Structural plasticity determines the connectivity mask used in the affine transformation. Gain modulation and the fast and homeostatic thresholds regulate primary activity, which provides the input to lateral inhibition before the final hidden activity is computed. During training, the final activity is used to update the homeostatic threshold and to compute the activation-decorrelation loss. Solid arrows indicate forward or control dependencies, while dashed arrows indicate training-time adaptation or regularization effects.

Experiment	IG	GM	TM	LI	HO	SP	AD	Train 0.9	Test 0.9	Train 1	Test 1	Lifetime Sparsity	Sample Hoyer	Neuron Hoyer	Dead Neuron	Mean Abs Corr	Effective Rank	Specialization	Purity	Centroid Sep	Within Jaccard	Between Jaccard	Jaccard Gap	Activation Sparsity	Structural Sparsity	Active Subnetwork Size
two off inhibition decorrelation	1	1	1	0	1	1	0	12	14	16	17	0.863	0.999	0.855	0.833	0.129	0.007	0.024	0.096	0.671	0.943	0.932	0.011	0.863	0.65	7
two off gain threshold	1	0	0	1	1	1	1	13	15	16	17	0.862	0.991	0.821	0.559	0.066	0.012	0.062	0.268	0.472	0.495	0.488	0.007	0.862	0.65	9
one off threshold	1	1	0	1	1	1	1	12	14	16	18	0.869	0.995	0.829	0.585	0.068	0.010	0.044	0.243	0.518	0.542	0.536	0.006	0.869	0.65	8
two off threshold decorrelation	1	1	0	1	1	1	0	12	14	16	18	0.927	0.999	0.909	0.830	0.089	0.007	0.030	0.106	0.670	0.916	0.896	0.020	0.927	0.65	7
one off decorrelation	1	1	1	1	1	1	0	12	14	16	18	0.934	0.999	0.924	0.841	0.072	0.007	0.053	0.111	0.670	0.914	0.891	0.023	0.934	0.65	7
one off inhibition	1	1	1	0	1	1	1	12	14	16	18	0.899	0.998	0.882	0.811	0.082	0.009	0.023	0.109	0.666	0.708	0.701	0.007	0.899	0.65	7

Table 2: Experimental results for the parity task.

Experiment	IG	GM	TM	LI	HO	SP	AD	Train 0.9	Test 0.9	Train 1	Test 1	Lifetime Sparsity	Sample Hoyer	Neuron Hoyer	Mean Abs. Corr.	Effective Rank	Specialization	Purity	Centroid Sep.	Within Jaccard	Between Jaccard	Jaccard Gap	Activation Sparsity
all on	1	1	1	1	1	1	1	99	1691	219	2310	0.832	0.823	0.745	0.139	0.197	0.720	0.858	28.779	0.392	0.170	0.222	0.818
two off threshold inhibition	1	1	0	0	1	1	1	96	1692	213	2463	0.845	0.828	0.756	0.139	0.185	0.745	0.871	29.973	0.401	0.173	0.227	0.820
one off inhibition	1	1	1	0	1	1	1	96	1698	215	2470	0.845	0.827	0.756	0.139	0.185	0.745	0.871	29.900	0.398	0.173	0.226	0.820
two off inhibition structural	1	1	1	0	1	0	1	96	1698	215	2470	0.845	0.827	0.756	0.139	0.185	0.745	0.871	29.900	0.398	0.173	0.226	0.820
one off structural	1	1	1	1	1	0	1	97	1703	215	2475	0.836	0.828	0.747	0.138	0.191	0.725	0.858	29.967	0.385	0.166	0.219	0.821
two off inhibition decorrelation	1	1	1	0	1	1	0	97	1693	214	2481	0.844	0.827	0.756	0.141	0.185	0.754	0.877	29.934	0.404	0.175	0.229	0.819

Table 3: Experimental results for the XOR task.

Experiment	IG	GM	TM	LI	HO	SP	AD	Train 0.9	Test 0.9	Train 1	Test 1	Lifetime Sparsity	Sample Hoyer	Neuron Hoyer	Mean Abs. Corr.	Effective Rank	Specialization	Purity	Centroid Sep.	Within Jaccard	Between Jaccard	Jaccard Gap	Activation Sparsity	Structural Sparsity
two off gain decorrelation	1	0	1	1	1	1	0	160	1427	310	2353	0.840	0.708	0.744	0.104	0.723	0.435	0.632	0.517	0.170	0.068	0.103	0.864	0.65
two off input gate gain	0	0	1	1	1	1	1	158	1445	310	2610	0.840	0.709	0.744	0.102	0.724	0.418	0.623	0.510	0.167	0.069	0.098	0.864	0.65
two off gain inhibition	1	0	1	0	1	1	1	157	1418	314	2818	0.849	0.717	0.752	0.104	0.711	0.444	0.637	0.512	0.166	0.063	0.103	0.864	0.65
one off gain	1	0	1	1	1	1	1	158	1431	307	2966	0.840	0.709	0.744	0.102	0.724	0.429	0.629	0.515	0.169	0.068	0.101	0.864	0.65
one off threshold	1	1	0	1	1	1	1	159	1424	311	3012	0.829	0.847	0.735	0.106	0.466	0.429	0.628	14.762	0.202	0.070	0.132	0.856	0.65
two off gain threshold	1	0	0	1	1	1	1	158	1442	313	3029	0.841	0.709	0.745	0.102	0.724	0.430	0.629	0.514	0.169	0.068	0.101	0.864	0.65

Table 4: Experimental results for the XOR structural density ablation with structural plasticity enabled at

\rho_{m}=0.35

Figure 2: Sparse parity with all bio-inspired mechanisms disabled. Training accuracy improves well before test accuracy, which produces a pronounced delayed generalization phase.

Figure 3: Sparse parity with all bio-inspired mechanisms enabled. Training and test accuracy rise in close succession, which results in substantially faster grokking than in the all-off case.

Figure 4: Sparse parity with only homeostasis and structural plasticity enabled. This best two-mechanism configuration shows rapid generalization and closely reproduces the fast grokking behavior of the fully enabled model.

Figure 5: Sparse parity with only structural plasticity enabled. Generalization remains delayed relative to training, but the transition occurs much earlier than in the all-off case.

Figure 6: Noisy XOR with all bio-inspired mechanisms disabled. Training accuracy eventually reaches 1, but test accuracy remains near chance throughout the experimental horizon, which indicates memorization without generalization.

Figure 7: Noisy XOR with all bio-inspired mechanisms enabled. Training accuracy reaches its maximum early, whereas test accuracy improves progressively over a much longer interval before reaching near-perfect performance. The separation between these two timescales illustrates delayed generalization.

Figure 8: Noisy XOR with only gain modulation and homeostasis enabled. This configuration is the best two-mechanism case according to “Test 1”.

Figure 9: Noisy XOR with only input gating and homeostasis enabled. This configuration is the second-best two-mechanism case according to “Test 1”.

Explore similar work

Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking
Sep 9, 2026Anish KatariaMultilayer Perceptrons Neural Network Memorization
Princeton University, Princeton, NJ, USA.
How to Tame Grokking: Representation Geometry as a Control Signal
Jul 13, 2026Maksim A KazanskiiNeural Network Generalization Spectral Regularization
Independent Researcher
A Stochastic--Geometric Theory of Scaling Laws in Grokking
Jun 29, 2026Róisín Luo, Christian Gagné, Jonas Ngnawé +2Neural Network Generalization Neural Network Optimization
Research Ireland – Centre for Research Training in AI · University of Galway · Universit´e Laval +2

Sep 9, 2026, cs.AIJ/K move · Enter open · S save

Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking

Anish Kataria

Princeton University, Princeton, NJ, USA.

Multilayer Perceptrons Neural Network Memorization Neural Network Training Dynamics Neural Scaling Laws Grokking

Open paper AskPDF arXiv

Neural networks trained past memorization frequently undergo a delayed transition to generalization, a phenomenon known as grokking. Despite theoretical progress on \emph{why} this transition occurs, the quantitative structure of \emph{when} it occurs in hyperparameter space remains uncharacterized. We map the memorization-to-generalization boundary across 384 configurations of two-hidden-layer MLPs on modular arithmetic, fitting a power-law scaling relation for generalization onset time:

T_{\mathrm{grok}} \propto H^{-0.27}\, D^{-2.04}\, η^{-0.50}\, λ^{-0.64}

(

R^2 = 0.732

;

0.821

with interactions). The exponent hierarchy reveals that data complexity (

D^{-2.04}

) is the dominant driver of regime transition, not model capacity (

H^{-0.27}

): doubling data accelerates generalization by

{\sim}4\times

, while doubling width yields only

{\sim}1.2\times

. A sharp phase boundary at weight decay

λ\gtrsim 1.0

separates grokking from non-grokking configurations, and weight norm trajectories show monotonic compression during the transition, consistent with implicit regularization selecting low-complexity solutions. These results provide a quantitative foundation for predicting and controlling regime transitions in overparameterized networks.

Abstract

Explore similar work

Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking

How to Tame Grokking: Representation Geometry as a Control Signal

A Stochastic--Geometric Theory of Scaling Laws in Grokking