Parallel Layer Normalization for Universal Approximation

Yunhao Ni; Yuxin Guo; Yuhe Liu; Wenxin Sun; Jie Luo; Wenjun Wu; Lei Huang

cs.LGMay 19, 2025

Parallel Layer Normalization for Universal Approximation

Authors: Yunhao Ni, Yuxin Guo, Yuhe Liu, Wenxin Sun, Jie Luo, Wenjun Wu, Lei Huang

Organizations: State Key Laboratory of Complex and Critical Software Environment (SKLCCSE), Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China

Abstract

Layer Normalization (LN) is usually studied for its role in optimization, but it also acts as a multivariate nonlinearity between affine layers. We investigate how grouping LN units affects the approximation power of neural networks as width grows. In Parallel Layer Normalization networks (PLN-Nets), fixed-dimensional LN units are applied independently to groups between two affine layers. We prove that a shallow LN-Net with one LN unit is non-universal regardless of width, whereas PLN-Nets become universal as the number of parallel groups grows. The mechanism extends to deep networks, Root Mean Square Normalization (RMSNorm) variants, position-wise feed-forward networks, and convolutional networks under suitable grouping structures. Controlled MLP and physics-informed neural network (PINN) experiments support PLN's practical viability, while CNN and ViT experiments show that group count shapes the finite-sample fitting envelope but does not act alone: the grouped axes, replacement site, and label structure lead to distinct fitting and generalization behavior.

Figures & tables

Figure 5: CNN normalization domains (schematic

P=HW=6

C=8

); boxes mark independent statistics. From left to right: (a) LN uses one

P\times C

domain; (b) GN groups channels across all positions ( Wu & He, 2018 ) ; (c) CPLN groups channels at each position; and (d) mixed-axis grouping (

G_{P}=2,G_{C}=C

) is studied empirically.

Width	PLN[4]	SAT	Tanh	ReLU
128	0.000035	0.000104	0.000250	0.993593
256	0.000036	0.000052	0.000188	0.998485

Table 1: Best relative

\mathcal{L}_{2}

error for the Helmholtz PINN (lower is better). The lowest error at each width is shown in bold.

Operator	Normalization domain	Fitting accuracy (%)
LN	$128\times 32\times 32$	24.6
Best GN	$2\times 32\times 32$	95.8
PLN[8]	$8\times 1\times 1$	100.0
PLN[4]	$4\times 1\times 1$	100.0

Table 2: Representative CNN normalization domains on the

50{,}000

-example CIFAR-10 training set with fixed random labels (three-seed means).

Setting	Random 20k	Clean train	Clean test
Baseline	48.2 / 72.8	100.0 / 100.0	71.6 / 72.3
LN $\to$ PLN	75.3 / 92.1	100.0 / 100.0	72.3 / 72.9
LN $\to$ Mix-Norm	89.5 / 91.2	100.0 / 100.0	73.5 / 74.5
GELU $\to$ PLN	73.7 / 97.3	99.4 / 100.0	69.7 / 71.0
GELU $\to$ Mix-Norm	91.7 / 98.2	100.0 / 100.0	70.2 / 70.7

Table 3: ViT accuracy (%). Baseline is the unchanged LN–GELU model; arrows indicate replacement of all LN layers or all FFN GELUs. Each cell reports the highest three-seed mean / highest single-run accuracy across configurations in that row, selected independently.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

	$\displaystyle d=4$		$\displaystyle d=8$
Nonlinearity	Depth 1	Depth 4	Depth 1	Depth 4
Identity	$\displaystyle 0.9982\pm 0.0071$	$\displaystyle 0.9979\pm 0.0072$	$\displaystyle 0.9989\pm 0.0159$	$\displaystyle 0.9972\pm 0.0144$
LN	$\displaystyle 0.9979\pm 0.0071$	$\displaystyle 0.8793\pm 0.0885$	$\displaystyle 0.9990\pm 0.0158$	$\displaystyle 0.9808\pm 0.0099$
PLN[4]	$\displaystyle\mathbf{0.0048\pm 0.0013}$	$\displaystyle 0.0018\pm 0.0006$	$\displaystyle 0.1171\pm 0.0197$	$\displaystyle 0.1034\pm 0.0244$
PLN[8]	$\displaystyle 0.0079\pm 0.0030$	$\displaystyle 0.00075\pm 0.00027$	$\displaystyle\mathbf{0.0803\pm 0.0068}$	$\displaystyle 0.0660\pm 0.0215$
ReLU	$\displaystyle 0.0118\pm 0.0043$	$\displaystyle 0.00081\pm 0.00025$	$\displaystyle 0.1196\pm 0.0082$	$\displaystyle 0.0282\pm 0.0036$

Appendix

Table I: Dense Fourier test MSE (five-target mean

\displaystyle\pm

standard deviation after averaging three model seeds; lower is better).

Width	Loss weight	ReLU	PLN[4]	PLN[8]	PLN[16]	PLN[32]	Tanh	SAT
128	$\displaystyle(1,10)$	0.999601	0.000035	0.001120	0.001176	0.007878	0.000250	0.000104
	$\displaystyle(1,1)$	0.995612	0.000941	0.002284	0.000970	0.004231	0.003733	0.000979
	$\displaystyle(3,1)$	0.993593	0.009553	0.003912	0.004856	0.007465	0.000440	0.000525
256	$\displaystyle(1,10)$	0.998639	0.000036	0.004040	0.001631	0.000567	0.000188	0.000052
	$\displaystyle(1,1)$	0.998485	0.001416	0.005552	0.001839	0.002643	0.002653	0.001219
	$\displaystyle(3,1)$	0.999727	0.018451	0.002966	0.002875	0.005476	0.009267	0.000410

Appendix

Table II: Best observed relative

\displaystyle\mathcal{L}_{2}

error for the Helmholtz PINN across recorded seeds and optimizer settings. Lower is better; the best and second-best results at each width and loss weight are bold and underlined, respectively.

Method	Initial 90% rank	Adam error	Adam + L-BFGS error
ReLU	1	0.99999	1.00000
Tanh	1	0.0018235	$\displaystyle 2.2481\times 10^{-6}$
SAT	1	0.00067438	$\displaystyle 1.2747\times 10^{-6}$
PLN[4]	12	0.00067266	$\displaystyle 4.8682\times 10^{-6}$
LN	2	0.64416	0.62834

Appendix

Table III: Exploratory Helmholtz PINN diagnostics. Each entry is the median over 24 matched width–learning-rate–seed configurations. The effective rank is measured at initialization; lower relative

\displaystyle\mathcal{L}_{2}

error is better. This protocol differs from Table II .

Figure A1: CIFAR-10 accuracy across mixed-axis domains in the depth- $\displaystyle 4$ , width- $\displaystyle 128$ CNN (three-seed means): random-label fit (top), clean-label train (middle), and clean-label test (bottom). Cyan marks GN ( $\displaystyle K=1$ ), magenta marks LN, dashed $\displaystyle K=32$ marks position-wise PLN, and white boxes mark panel maxima. Each cell shows its three-seed mean accuracy, rounded to the nearest percentage point.

Figure A2: CNN accuracy versus total group count on CIFAR-10: (a) fitting

\displaystyle 50{,}000

fixed random labels, (b) clean-label training, and (c) clean-label testing.

\displaystyle K

is the number of spatial patches per axis, so the total number of groups is

\displaystyle G_{C}K^{2}

. The GN (

\displaystyle K=1

) and position-wise PLN (

\displaystyle K=32

) directions are marked explicitly. Curves show three-seed means with sample standard deviations.

Figure A3: Mean CIFAR-10 test accuracy of VGG-16 models over three random seeds for each learning-rate–initialization pair. Each panel corresponds to one nonlinear layer, and darker green denotes higher accuracy. The learning rate increases from $\displaystyle 0.001$ (left) to $\displaystyle 1$ (right). The middle row uses the unmodified He initialization ( He et al., 2015 ) ; the remaining rows scale its weights from $\displaystyle 1/4$ (bottom) to $\displaystyle 4$ (top).

Figure A4: Random-label fitting over token–channel grouping domains in the depth- $\displaystyle 4$ , width- $\displaystyle 192$ ViT. Each cell shows a three-seed mean rounded to the nearest percentage point. The dashed top row ( $\displaystyle G_{T}=197$ ) is the position-wise PLN direction. Left: every LN is replaced while GELU is retained. Right: LN is retained while all FFN activations are replaced. The magenta box marks standard LN in the all-normalization panel, and white boxes mark panel maxima.

Figure A5: Three-seed random- $\displaystyle 20{,}000$ ViT fitting results (cell labels rounded to the nearest percentage point). The magenta box marks standard LN in the all-normalization panel, the dashed row is the position-wise PLN direction, and white boxes mark maxima.

Figure A6: Three-seed clean-label ViT results (cell labels rounded to the nearest percentage point). Training accuracy varies strongly with domain geometry, while test accuracy occupies a much narrower range except at degenerate, near-scalar endpoints. The magenta box marks standard LN and white boxes mark panel maxima.

Layer Normalization

Layer normalization centers and scales neural activations to stabilize training, shape representations, and condition inputs across layers.

See all 29 papers

Activation-Adjacent Layer Normalization: Effects on Approximation Power
Oct 15, 2026Yunhao Ni, Yuxin Guo, Jie Luo +2Layer Normalization Neural Network Approximation Theory
State Key Laboratory of Complex and Critical Software Environment, Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China
Enjoy Your Layer Normalization with the Computational Efficiency of RMSNorm
May 14, 2026Yuxin Guo, Yihao Yue, Yunhao Ni +4Efficient Neural Network Inference Layer Normalization
State Key Laboratory of Complex and Critical Software Environment (SKLCCSE), Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China
Approximation Theory for Neural Networks: Old and New
May 20, 2026Soumendu Sundar Mukherjee, Himasish TalukdarUniversal Approximation Neural Network Approximation Theory
Statistics and Mathematics Unit, Indian Statistical Institute, Kolkata

Oct 15, 2026J/K move · Enter open · S save

Activation-Adjacent Layer Normalization: Effects on Approximation Power

Yunhao Ni, Yuxin Guo, Jie Luo, Wenjun Wu+1

State Key Laboratory of Complex and Critical Software Environment, Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China

Layer Normalization Neural Network Approximation Theory Shallow Neural Networks

Open paper AskPDF arXiv

Layer normalization (LN) and root mean square normalization (RMSNorm) depend on their inputs and remain nonlinear at inference. Inserting either next to an activation may therefore change the function a network represents. We study when shallow activation-normalized (AN) and preactivation-normalized (PN) networks can approximate a prescribed width-

N

one-hidden-layer MLP. Viewing RMSNorm as input-dependent scaling, we use width augmentation to locally linearize normalization on compact feature sets. For AN networks, an unbounded activation range allows width

N+1

, while every continuous nonconstant activation admits width

N+r

with error

O_f(r^{-1})

. For PN networks, affine parameters or, without them, degree-one positive homogeneity recover width

N+1

; general continuous activations admit width

N+r

with error tending to zero. The corresponding LN constructions use centered coordinate pairs, yielding widths

N+2

and

N+2r

, respectively. We prove that

N+1

for RMSNorm and

N+2

for LN are also the worst-case bound for approximations, and show that applicable universal-approximation theorems for ordinary MLPs transfer to these normalized architectures. Numerical experiments recover the predicted approximation rates for both normalizations across all five settings and illustrate the shifted-ReLU finite-width obstruction.

Abstract

Layer Normalization

Explore similar work

Activation-Adjacent Layer Normalization: Effects on Approximation Power

Enjoy Your Layer Normalization with the Computational Efficiency of RMSNorm

Approximation Theory for Neural Networks: Old and New