Activation-Adjacent Layer Normalization: Effects on Approximation Power

Yunhao Ni; Yuxin Guo; Jie Luo; Wenjun Wu; Lei Huang

cs.LGOct 15, 2026

Activation-Adjacent Layer Normalization: Effects on Approximation Power

Authors: Yunhao Ni, Yuxin Guo, Jie Luo, Wenjun Wu, Lei Huang

Organizations: State Key Laboratory of Complex and Critical Software Environment, Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China

Abstract

Layer normalization (LN) and root mean square normalization (RMSNorm) depend on their inputs and remain nonlinear at inference. Inserting either next to an activation may therefore change the function a network represents. We study when shallow activation-normalized (AN) and preactivation-normalized (PN) networks can approximate a prescribed width- $N$ one-hidden-layer MLP. Viewing RMSNorm as input-dependent scaling, we use width augmentation to locally linearize normalization on compact feature sets. For AN networks, an unbounded activation range allows width $N+1$ , while every continuous nonconstant activation admits width $N+r$ with error $O_f(r^{-1})$ . For PN networks, affine parameters or, without them, degree-one positive homogeneity recover width $N+1$ ; general continuous activations admit width $N+r$ with error tending to zero. The corresponding LN constructions use centered coordinate pairs, yielding widths $N+2$ and $N+2r$ , respectively. We prove that $N+1$ for RMSNorm and $N+2$ for LN are also the worst-case bound for approximations, and show that applicable universal-approximation theorems for ordinary MLPs transfer to these normalized architectures. Numerical experiments recover the predicted approximation rates for both normalizations across all five settings and illustrate the shifted-ReLU finite-width obstruction.

Figures & tables

Figure 1: Near- and far-field geometry of RMSNorm. In (a), radial normalization maps equally spaced points in

\mathcal{H}

to unevenly spaced points on the sphere. In (b), the equally sized set

\mathcal{H}^{\prime}

lies farther from the origin. It subtends a smaller angle, and its normalized image occupies a shorter, nearly flat arc.

Mode	Parameters	Additional condition on $\phi$	RMSNorm (UB/LB)	LN (UB/LB)	Error
AN	$\checkmark/\times$	unbounded range	$N+1\,/\,N+1$	$N+2\,/\,N+2$	$<\varepsilon$
AN	$\checkmark/\times$	nonconstant	$N+r\,/\,\text{--}$	$N+2r\,/\,\text{--}$	$\leq C_{f}/r$
PN	$\checkmark$	none	$N+1\,/\,N+1$	$N+2\,/\,N+2$	$<\varepsilon$
PN	$\times$	degree-one homogeneous	$N+1\,/\,N+1$	$N+2\,/\,N+2$	$<\varepsilon$
PN	$\times$	none	$N+r\,/\,\text{--}$	$N+2r\,/\,\text{--}$	$\to 0$

Table 1: Width and error guarantees for a width-

N

target. Entries are constructive upper bounds/worst-case lower bounds (UB/LB); matching entries are sharp, and – denotes that no matching lower bound is claimed. All rows assume continuous

\phi

Panel of Figure 3 — Figure 3: Representative verification of the three $M$ -controlled constructions at $N=16$ . Each curve reports the mean of grid-based maximum absolute errors over six target MLPs generated from independent random seeds. ReLU is used for unbounded AN and homogeneous PN, whereas $\tanh$ is used for affine PN. Both axes use logarithmic scales. Markers distinguish LayerNorm and RMSNorm, and gray lines illustrate reference slopes rather than error bounds. The complete four-width activation sweep and the width-growing experiments appear in Appendix F .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Construction	Predicted RMS/LN	RMS observed	LN observed
AN, unbounded (ReLU)	$-2/-1$	$-1.998$ ( $-1.999,-1.997$ )	$-0.953$ ( $-0.977,-0.886$ )
AN, general ( $\tanh$ )	$-1/-1$	$-0.956$ ( $-0.975,-0.931$ )	$-0.968$ ( $-0.982,-0.940$ )
PN, affine ( $\tanh$ )	$-2/-2$	$-1.999$ ( $-1.999,-1.997$ )	$-1.999$ ( $-1.999,-1.998$ )
PN, homogeneous (ReLU)	$-2/-2$	$-1.997$ ( $-1.999,-1.994$ )	$-1.998$ ( $-1.999,-1.995$ )
PN, general ( $\tanh$ )	$-1/-1$	$-0.956$ ( $-0.970,-0.913$ )	$-0.966$ ( $-0.979,-0.937$ )

Appendix

Table 2: Predicted and observed tail slopes for one activation in each regime. Observed values are medians (interquartile ranges) over

24

target–width pairs.

Figure 4: Extended activation-family tests: unbounded AN, general AN, and affine PN. Each blue or pink curve is the median grid maximum error over

24

target–width pairs for LN or RMSNorm, and the shaded region is its interquartile range. Gray dashed and dotted lines illustrate reference slopes, not error bounds. Both axes are logarithmic.

Figure 5: Extended activation-family tests: homogeneous and general PN. Curves and shaded regions have the same meaning as in Figure 4 ; the gray dashed lines illustrate the common reference slopes.

Figure 6: Shifted-ReLU PN constructions for

f(x)=\operatorname{ReLU}(2x)

. (a,b) Function-level recovery as width grows. (c) The coordinate-bound barrier

d=10^{2}

and subsequent grid-error decay consistent with

d^{-1}

without affine parameters. (d) Fixed-width affine constructions with grid-error decay consistent with

M^{-2}

Figure 7: Shifted-ReLU PN verification over

24

random one-hidden-layer MLP targets. (a,b) One representative target and selected approximants. (c,d) Median relative grid maximum errors and interquartile ranges for the width-growing no-affine and fixed-width affine constructions, respectively. Gray lines illustrate reference slopes, not error bounds.

Layer Normalization

Layer normalization centers and scales neural activations to stabilize training, shape representations, and condition inputs across layers.

See all 29 papers

Parallel Layer Normalization for Universal Approximation
May 19, 2025Yunhao Ni, Yuxin Guo, Yuhe Liu +4Layer Normalization Neural Network Approximation Theory
State Key Laboratory of Complex and Critical Software Environment (SKLCCSE), Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China
Enjoy Your Layer Normalization with the Computational Efficiency of RMSNorm
May 14, 2026Yuxin Guo, Yihao Yue, Yunhao Ni +4Efficient Neural Network Inference Layer Normalization
State Key Laboratory of Complex and Critical Software Environment (SKLCCSE), Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China
Deep Network Approximation: Beyond ReLU to Diverse Activation Functions
Jul 13, 2023Shijun Zhang, Jianfeng Lu, Hongkai ZhaoReLU Neural Networks Neural Network Activation Functions
Department of Mathematics Duke University

May 19, 2025J/K move · Enter open · S save

Parallel Layer Normalization for Universal Approximation

Yunhao Ni, Yuxin Guo, Yuhe Liu, Wenxin Sun+3

State Key Laboratory of Complex and Critical Software Environment (SKLCCSE), Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China

Layer Normalization Neural Network Approximation Theory

Open paper AskPDF arXiv

Layer Normalization (LN) is usually studied for its role in optimization, but it also acts as a multivariate nonlinearity between affine layers. We investigate how grouping LN units affects the approximation power of neural networks as width grows. In Parallel Layer Normalization networks (PLN-Nets), fixed-dimensional LN units are applied independently to groups between two affine layers. We prove that a shallow LN-Net with one LN unit is non-universal regardless of width, whereas PLN-Nets become universal as the number of parallel groups grows. The mechanism extends to deep networks, Root Mean Square Normalization (RMSNorm) variants, position-wise feed-forward networks, and convolutional networks under suitable grouping structures. Controlled MLP and physics-informed neural network (PINN) experiments support PLN's practical viability, while CNN and ViT experiments show that group count shapes the finite-sample fitting envelope but does not act alone: the grouped axes, replacement site, and label structure lead to distinct fitting and generalization behavior.

Abstract

Layer Normalization

Explore similar work

Parallel Layer Normalization for Universal Approximation

Enjoy Your Layer Normalization with the Computational Efficiency of RMSNorm

Deep Network Approximation: Beyond ReLU to Diverse Activation Functions