Divine Benevolence is an $x^2$: GLUs scale asymptotically faster than MLPs

Alejandro Francisco Queiruga

cs.LGFeb 16, 2026

Divine Benevolence is an $x^2$ : GLUs scale asymptotically faster than MLPs

Authors: Alejandro Francisco Queiruga

Abstract

Scaling laws can be understood from ground-up numerical analysis, where traditional function approximation theory can explain shifts in model architecture choices. GLU variants now dominate frontier LLMs and similar outer-product architectures are prevalent in ranking models. The success of these architectures has mostly been left as an empirical discovery. In this paper, we apply the tools of numerical analysis to expose a key factor: these models have an $x^2$ which enables asymptotically faster scaling than MLPs. GLUs have piecewise quadratic functional forms that are sufficient to exhibit quadratic order of approximation. Our key contribution is to demonstrate that the $L(P)$ scaling slope is $L(P)\propto P^{-3}$ for GLUs but only $L(P)=P^{-2}$ for MLPs on function reconstruction problems. We provide a parameter construction and empirical verification of these slopes for 1D function approximation. From the first principles we discover, we make one stride and propose the ``Gated Quadratic Unit'' which has an even steeper $L(P)$ slope than the GLU and MLP. This opens the possibility of architecture design from first principles numerical theory to unlock superior scaling in large models. Replication code is available at https://github.com/afqueiruga/divine_scaling.

Figures & tables

Panel of Figure 1 — Figure 1: Illustration of the function space of randomly initialized MLPs and GLUs with ReLU activations. Left and middle: randomly initialized MLP and GLU on $\mathbb{R}^{2}\rightarrow\mathbb{R}$ . Black boundaries are the activation boundaries that break the domain into piecewise linear (MLP) or piecewise quadratic partitions. Right: in 1D, each neuron of the GLU forms a single piecewise quadratic function.

Panel of Figure 2 — Figure 2: Comparison of MLP and GLU fitting behavior under spline-based initialization (a,d) and full training (b, e). Vertical lines denote cell boundaries formed by $Gx+g=0$ . Partial optimization in (a,d) already achieves the same asymptotic scaling rate as the analytical construction. The activations of individual neurons of the fully trained MLP and GLU are shown in (c,f).

Figure 3: Experimental results for

L(P)

scaling on 1d function approximation. The x-axis on the left uses neuron count which corresponds to the number of spline knots; changing the variable to parameter count on the right translates the log-log lines but does not change the log-log slope.

Figure 4: MLPs and GLUs have similar approximation errors and order of convergence as their spline counterparts. Neuron count is analogous to the number of knots (control nodes) in a spline.

Figure 5: Novel architecture design to scale faster than a GLU: the Gated Quadratic Unit (GQU) has faster scaling than a GLU at

L(P)\propto P^{-3.5}

Problem	$x$ dim	MLP	GLU	GQU
$1/(1+cos(\pi x)^{2})$	1	$-2.13$	$-3.08$	-3.55
$sin(4x)sin(4y)$	2	$-0.91$	$-1.55$	-
Friedman1	5	$-0.55$	$-1.00$	-
Friedman2	4	$-0.75$	$-1.12$	-
Friedman3	4	$-0.31$	$-0.56$	-
Airfoil Self-drag	5	$-0.25$	$-0.39$	-

Table 1: Empirically measured slopes.

Figure 6: Higher dimension synthetic problems.

Figure 7: Real world benchmark regression datasets.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: ReLU activations for the uniform cell spline initializations

G=\pm 1

and

g=\pm linspace(-1,1,n)

. On the left is an initialization where all face the same way, which simplified the analytical construction. On the right is an alternating construction with the same partitioning but prevents all dead neurons.

Figure 9: Surfaces of randomly initialized MLP and GLU with four neurons. The black lines demarcate the hinge of the ReLU activations, breaking the surface into piecewise linear and piecewise quadratic regions.

Figure 10: Neuron activations of a randomly initialized MLP with four neurons form a piecewise linear basis set.

Figure 11: Neuron activations of a randomly initialized GLU with four neurons form a piecewise quadratic basis set.

Neural Scaling Laws

Neural scaling laws describe how model performance changes with training data, compute, model capacity, and other resources.

See all 79 papers

The Devil is in the Condition Numbers: Why is GLU Better than non-GLU Structure?
May 20, 2026Xingyu Lyu, Qianqian Xu, Zhiyong Yang +2Neural Network Optimization Neural Network Training Dynamics
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 101408, China · Beijing Academy of Artificial Intelligence (BAAI), Beijing, China
Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation
May 8, 2026Joshua Shay Kricheli, Alexander Lawrence Reid, Soumajyoti Sarkar +2Parameter Estimation Language Model Scaling Laws
Syracuse University · Amazon AGI Foundations
PowLU: An Activation Function for Stable Pre-Training of LLMs
May 25, 2026Peijie Jiang, Yuqi Feng, Cunyin Peng +5Language Model Pretraining Neural Network Activation Functions
Ling Team, Ant Group

May 20, 2026J/K move · Enter open · S save

The Devil is in the Condition Numbers: Why is GLU Better than non-GLU Structure?

Xingyu Lyu, Qianqian Xu, Zhiyong Yang, Peisong Wen+1

State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 101408, China · Beijing Academy of Artificial Intelligence (BAAI), Beijing, China

Neural Network Optimization Neural Network Training Dynamics

Open paper AskPDF arXiv

Gated Linear Units (GLU) and their variants are widely adopted in modern open-source large language model architectures and consistently outperform their non-gated counterparts, yet the underlying reasons for this advantage remain unclear. In this work, we study GLU by analyzing two-layer networks in the neural tangent kernel (NTK) regime. Our analysis reveals that the GLU structure reshapes the NTK spectrum, leading to a smaller condition number and a more compact eigenvalue distribution. Building on this finding, we further analyze the resulting training dynamics and show how the reshaped spectrum leads to faster convergence of GLU models, including a characteristic loss-crossing phenomenon observed between GLU and non-GLU models. Finally, we empirically observe that GLU has limited impact in reducing the generalization gap on various models, including ViT and GPT-2, suggesting that its primary benefit lies in accelerating optimization rather than reducing the generalization gap. The code is available at: https://github.com/Zemdalk/GLU-NTK.

Abstract

Neural Scaling Laws

Explore similar work

The Devil is in the Condition Numbers: Why is GLU Better than non-GLU Structure?

Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation

PowLU: An Activation Function for Stable Pre-Training of LLMs