Activation-Adjacent Layer Normalization: Effects on Approximation Power

Yunhao Ni; Yuxin Guo; Jie Luo; Wenjun Wu; Lei Huang

cs.LGOct 15, 2026

Activation-Adjacent Layer Normalization: Effects on Approximation Power

Authors: Yunhao Ni, Yuxin Guo, Jie Luo, Wenjun Wu, Lei Huang

Organizations: State Key Laboratory of Complex and Critical Software Environment, Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China

Abstract

Layer normalization (LN) and root mean square normalization (RMSNorm) depend on their inputs and remain nonlinear at inference. Inserting either next to an activation may therefore change the function a network represents. We study when shallow activation-normalized (AN) and preactivation-normalized (PN) networks can approximate a prescribed width- $N$ one-hidden-layer MLP. Viewing RMSNorm as input-dependent scaling, we use width augmentation to locally linearize normalization on compact feature sets. For AN networks, an unbounded activation range allows width $N+1$ , while every continuous nonconstant activation admits width $N+r$ with error $O_f(r^{-1})$ . For PN networks, affine parameters or, without them, degree-one positive homogeneity recover width $N+1$ ; general continuous activations admit width $N+r$ with error tending to zero. The corresponding LN constructions use centered coordinate pairs, yielding widths $N+2$ and $N+2r$ , respectively. We prove that $N+1$ for RMSNorm and $N+2$ for LN are also the worst-case bound for approximations, and show that applicable universal-approximation theorems for ordinary MLPs transfer to these normalized architectures. Numerical experiments recover the predicted approximation rates for both normalizations across all five settings and illustrate the shifted-ReLU finite-width obstruction.

Figures & tables

Figure 1: Near- and far-field geometry of RMSNorm. In (a), radial normalization maps equally spaced points in

\mathcal{H}

to unevenly spaced points on the sphere. In (b), the equally sized set

\mathcal{H}^{\prime}

lies farther from the origin. It subtends a smaller angle, and its normalized image occupies a shorter, nearly flat arc.

Mode	Parameters	Additional condition on $\phi$	RMSNorm (UB/LB)	LN (UB/LB)	Error
AN	$\checkmark/\times$	unbounded range	$N+1\,/\,N+1$	$N+2\,/\,N+2$	$<\varepsilon$
AN	$\checkmark/\times$	nonconstant	$N+r\,/\,\text{--}$	$N+2r\,/\,\text{--}$	$\leq C_{f}/r$
PN	$\checkmark$	none	$N+1\,/\,N+1$	$N+2\,/\,N+2$	$<\varepsilon$
PN	$\times$	degree-one homogeneous	$N+1\,/\,N+1$	$N+2\,/\,N+2$	$<\varepsilon$
PN	$\times$	none	$N+r\,/\,\text{--}$	$N+2r\,/\,\text{--}$	$\to 0$

Table 1: Width and error guarantees for a width-

N

target. Entries are constructive upper bounds/worst-case lower bounds (UB/LB); matching entries are sharp, and – denotes that no matching lower bound is claimed. All rows assume continuous

\phi

Panel of Figure 3 — Figure 3: Representative verification of the three $M$ -controlled constructions at $N=16$ . Each curve reports the mean of grid-based maximum absolute errors over six target MLPs generated from independent random seeds. ReLU is used for unbounded AN and homogeneous PN, whereas $\tanh$ is used for affine PN. Both axes use logarithmic scales. Markers distinguish LayerNorm and RMSNorm, and gray lines illustrate reference slopes rather than error bounds. The complete four-width activation sweep and the width-growing experiments appear in Appendix F .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Construction	Predicted RMS/LN	RMS observed	LN observed
AN, unbounded (ReLU)	$-2/-1$	$-1.998$ ( $-1.999,-1.997$ )	$-0.953$ ( $-0.977,-0.886$ )
AN, general ( $\tanh$ )	$-1/-1$	$-0.956$ ( $-0.975,-0.931$ )	$-0.968$ ( $-0.982,-0.940$ )
PN, affine ( $\tanh$ )	$-2/-2$	$-1.999$ ( $-1.999,-1.997$ )	$-1.999$ ( $-1.999,-1.998$ )
PN, homogeneous (ReLU)	$-2/-2$	$-1.997$ ( $-1.999,-1.994$ )	$-1.998$ ( $-1.999,-1.995$ )
PN, general ( $\tanh$ )	$-1/-1$	$-0.956$ ( $-0.970,-0.913$ )	$-0.966$ ( $-0.979,-0.937$ )

Appendix

Table 2: Predicted and observed tail slopes for one activation in each regime. Observed values are medians (interquartile ranges) over

24

target–width pairs.

Figure 4: Extended activation-family tests: unbounded AN, general AN, and affine PN. Each blue or pink curve is the median grid maximum error over

24

target–width pairs for LN or RMSNorm, and the shaded region is its interquartile range. Gray dashed and dotted lines illustrate reference slopes, not error bounds. Both axes are logarithmic.

Figure 5: Extended activation-family tests: homogeneous and general PN. Curves and shaded regions have the same meaning as in Figure 4 ; the gray dashed lines illustrate the common reference slopes.

Figure 6: Shifted-ReLU PN constructions for

f(x)=\operatorname{ReLU}(2x)

. (a,b) Function-level recovery as width grows. (c) The coordinate-bound barrier

d=10^{2}

and subsequent grid-error decay consistent with

d^{-1}

without affine parameters. (d) Fixed-width affine constructions with grid-error decay consistent with

M^{-2}

Figure 7: Shifted-ReLU PN verification over

24

random one-hidden-layer MLP targets. (a,b) One representative target and selected approximants. (c,d) Median relative grid maximum errors and interquartile ranges for the width-growing no-affine and fixed-width affine constructions, respectively. Gray lines illustrate reference slopes, not error bounds.

Layer Normalization

Layer normalization centers and scales neural activations to stabilize training, shape representations, and condition inputs across layers.

See all 29 papers

May 19, 2025

Parallel Layer Normalization for Universal Approximation

Layer Normalization (LN) is usually studied for its role in optimization, but it also acts as a multivariate nonlinearity between affine layers. We investigate how grouping LN units affects the approximation power of neural networks as width grows. In Parallel Layer Normalization networks (PLN-Nets), fixed-dimensional LN units are applied independently to groups between two affine layers. We prove that a shallow LN-Net with one LN unit is non-universal regardless of width, whereas PLN-Nets become universal as the number of parallel groups grows. The mechanism extends to deep networks, Root Mean Square Normalization (RMSNorm) variants, position-wise feed-forward networks, and convolutional networks under suitable grouping structures. Controlled MLP and physics-informed neural network (PINN) experiments support PLN's practical viability, while CNN and ViT experiments show that group count shapes the finite-sample fitting envelope but does not act alone: the grouped axes, replacement site, and label structure lead to distinct fitting and generalization behavior.

Yunhao Ni, Yuxin Guo, Yuhe Liu +4

State Key Laboratory of Complex and Critical Software Environment (SKLCCSE), Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China

Layer Normalization Neural Network Approximation Theory

Ask PDF

May 14, 2026

Enjoy Your Layer Normalization with the Computational Efficiency of RMSNorm

Layer normalization (LN) is a fundamental component in modern deep learning, but its per-sample centering and scaling introduce non-negligible inference overhead. RMSNorm improves efficiency by removing the centering operation, yet this may discard benefits associated with centering. This paper propose a framework to determine whether an LN in an arbitrary DNN can be replaced by RMSNorm without changing the model function. The key idea is to fold LN's centering operation into upstream general linear layers by enforcing zero-mean outputs through the column-centered constraint (CCC) and column-based weight centering (CBWC). We extend the analysis to arbitrary DNNs, define such LNs as foldable LNs, and develop a graph-based detection algorithm. Our analysis shows that many LNs in widely used architectures are foldable, enabling exact inference-time conversion and end-to-end acceleration of 2% to 12% without changing model predictions. Experiments across multiple task families further show that, when exact equivalence is partially broken in practical training settings, our method remains competitive with vanilla LN while improving efficiency.

Yuxin Guo, Yihao Yue, Yunhao Ni +4

State Key Laboratory of Complex and Critical Software Environment (SKLCCSE), Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China

Efficient Neural Network Inference Layer Normalization

Ask PDF

Jul 13, 2023

Deep Network Approximation: Beyond ReLU to Diverse Activation Functions

This paper explores the expressive power of deep neural networks for a diverse range of activation functions. An activation function set

\mathscr{A}

is defined to encompass the majority of commonly used activation functions, such as

\mathtt{ReLU}

\mathtt{LeakyReLU}

\mathtt{ReLU}^2

\mathtt{ELU}

\mathtt{CELU}

\mathtt{SELU}

\mathtt{Softplus}

\mathtt{GELU}

\mathtt{SiLU}

\mathtt{Swish}

\mathtt{Mish}

\mathtt{Sigmoid}

\mathtt{Tanh}

\mathtt{Arctan}

\mathtt{Softsign}

\mathtt{dSiLU}

, and

\mathtt{SRS}

. We demonstrate that for any activation function

\varrho\in \mathscr{A}

, a

\mathtt{ReLU}

network of width

N

and depth

L

can be approximated to arbitrary precision by a

\varrho

-activated network of width

3N

and depth

2L

on any bounded set. This finding enables the extension of most approximation results achieved with

\mathtt{ReLU}

networks to a wide variety of other activation functions, albeit with slightly increased constants. Significantly, we establish that the (width,

\,

depth) scaling factors can be further reduced from

(3,2)

(1,1)

\varrho

falls within a specific subset of

\mathscr{A}

. This subset includes activation functions such as

\mathtt{ELU}

\mathtt{CELU}

\mathtt{SELU}

\mathtt{Softplus}

\mathtt{GELU}

\mathtt{SiLU}

\mathtt{Swish}

, and

\mathtt{Mish}

Shijun Zhang, Jianfeng Lu, Hongkai Zhao

Department of Mathematics Duke University

ReLU Neural Networks Neural Network Activation Functions Neural Network Approximation Theory Neural Network Expressivity

Ask PDF

Abstract

Layer Normalization

Explore similar work

Parallel Layer Normalization for Universal Approximation

Enjoy Your Layer Normalization with the Computational Efficiency of RMSNorm

Deep Network Approximation: Beyond ReLU to Diverse Activation Functions