Layer Normalization

Follow this topic Ask about this topic

Layer normalization (LayerNorm) centers and scales activations within a layer, while related methods such as RMSNorm scale without centering. Research compares where normalization is placed, studies conditional variants such as AdaLN in diffusion models, and develops theory linking normalization to approximation power, representation rank, and training dynamics. Experiments span language models, vision, diffusion, and graph networks, alongside methods that fold or approximate normalization to reduce inference and hardware costs. A central tension is that normalization can improve stability and expressivity in some settings but hurt learning in others, depending on its placement, optimizer, model capacity, and data regime.

Latest papers

29 papers3 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Activation-Adjacent Layer Normalization: Effects on Approximation Power
Oct 15, 2026Yunhao Ni, Yuxin Guo, Jie Luo +2Layer Normalization Neural Network Approximation Theory
State Key Laboratory of Complex and Critical Software Environment, Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China
Counting with Attention and LayerNorm: Norm Scaling and Representation Geometry
Oct 12, 2026Ayyub Shaffy, Manoj Dhakal, Safal Shrestha +1Self-Attention Layer Normalization
New York University Abu Dhabi
LayerRoPE: Dynamic Depth-wise Magnitude & Angular Superposition
Oct 6, 2026Shikhar Srivastava, Christopher KananTransformer Positional Encoding
University of Rochester
Correcting the Dropout-LayerNorm Expectation Gap Improves Protein Structure Models
Sep 25, 2026Isaac Ellmen, David Errington, Matthew I. J. Raybould +1Protein Structure Prediction Layer Normalization
Department of Statistics, University of Oxford · Recursion
Tracing distinguishability through transformer processing with stochastic LayerNorm
Aug 31, 2026Kieran MurphyTransformer Interpretability Layer Normalization
New Jersey Institute of Technology
Beyond receptive fields: sequence-pooled normalization can supply most of a sequence labeler's context
Aug 19, 2026Qing TianConvolutional Neural Networks Layer Normalization
Department of Computer Science, University of Alabama at Birmingham, Birmingham, AL 35294, USA
Rethinking Normalization Placement for LLMs: Post-Norm under Curriculum Depth Growing
Aug 13, 2026Sheng Ren, Yadong Wang, Naiqiang Tan +7Transformer Layer Normalization
Nanjing University of Aeronautics and Astronautics, Nanjing, Jiangsu, China · Didichuxing Co. Ltd, Beijing, China
Unveiling the Secret of AdaLN-Zero in Diffusion Transformer
Aug 10, 2026Jie Zhu, Mingyu Ding, Boqiang Duan +2Diffusion Transformer Neural Network Initialization
Key Lab of High Confidence Software Technologies (Peking University), Ministry of Education, China, and School of Computer Science, Peking University, China. · UC Berkeley. · Baidu, Beijing, China.
Why Post-Norm Transformers Collapse: Attention Amplification and Gradient Repair Failure
Aug 10, 2026Xingjian Wang, Qingyu Han, Xiaodong Luo +1Transformer Self-Attention
The Chinese University of Hong Kong, Shenzhen · Shenzhen Research Institute of Big Data
One Model to Magnify Them All: Efficient Scale-Invariant Histopathology via Conditional Normalization and Continuous Magnification Training
Aug 10, 2026Agnieszka Florkowska, Henning Müller, Marek WodzinskiComputational Pathology Histopathology Image Segmentation
Sano Centre for Computational Medicine, Krakow, Poland · AGH University of Krakow, al. Adama Mickiewicza 30, 30-059 Kraków, Poland · Institute of Informatics, HES-SO Valais-Wallis, Sierre, Switzerland +1
Phasor Attention: Mean Root Square Normalization for Phase Manifold Preservation
Jul 20, 2026Sungwoo Goo, Hwi-yeol Yun, Sangkeun JungNeural Network Stability Gradient Clipping
College of Pharmacy, Chungnam National University, Daejeon, Republic of Korea · Department of Computer Science & Engineering, Chungnam National University, Daejeon, Republic of Korea
Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth
Jul 15, 2026Katie EverettTransformer FFNs Transformer
MIT CSAIL
LayerNorm as Implicit Gain Control in Looped Transformers
Jul 12, 2026Matthias M. M. BuehlmaierRecurrent Transformers Layer Normalization
Faculty of Business and Economics, The University of Hong Kong · Pokfulam Road, Hong Kong
AutoNorm: Understanding Adaptive Normalization in Transformers through Differentiable Gating
Jul 12, 2026Piyush Kaushik Bhattacharyya, Divyanshu Rai, Swastik Singh +3Dynamic Neural Networks Transformer
Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale
Jun 17, 2026Tejas Pradeep Shirodkar, P. J. NarayananInformation Geometry Transformer
IIIT, Hyderabad
MIVE: A Minimalist Integer Vector Engine for Softmax LayerNorm and RMSNorm Acceleration
Jun 16, 2026Kosmas Alexandridis, Giorgos DimitrakopoulosAI Accelerator Inference Inference Acceleration
Integrated Circuits Lab, Electrical and Computer Engineering, Democritus University of Thrace (DUTH), Greece
PostDeg: Placement Beats Parameterization in LayerNorm GNNs
Jun 12, 2026Yash Tomar, Aryav DasGraph Neural Networks Layer Normalization
Purdue University · Park Tudor High School
Looped Transformers with Layer Normalization Provably Learn the Power Method
May 30, 2026Lyumin Wu, Chenyang Zhang, Yuan CaoRecurrent Transformers Layer Normalization
School of Computing & Data Science, The University of Hong Kong
Enjoy Your Layer Normalization with the Computational Efficiency of RMSNorm
May 14, 2026Yuxin Guo, Yihao Yue, Yunhao Ni +4Efficient Neural Network Inference Layer Normalization
State Key Laboratory of Complex and Critical Software Environment (SKLCCSE), Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China
Evolving Layer-Specific Scalar Functions for Hardware-Aware Transformer Adaptation
May 13, 2026Kieran Carrigg, Sigur de Vries, Amirhossein Sadough +1Efficient Neural Network Inference Efficient ViTs
Department of Machine Learning and Neural Computing Donders Institute for Brain, Cognition, and Behaviour Thomas van Aquinostraat 4, 6525 GD Nijmegen, The Netherlands
Attention Drift: What Autoregressive Speculative Decoding Models Learn
May 11, 2026Doğaç Eldenk, Payal Mohapatra, Yigitcan Comlek +3Long-Context Language Model Inference Speculative Decoding
Northwestern University · GE Aerospace · fal +1
The Transformer as a Polar State Estimator
May 10, 2026Peter RacioppoTransformer Layer Normalization
Independent Researcher Los Angeles, CA
Hardware-Efficient Softmax and Layer Normalization with Guaranteed Normalization for Edge Devices
Apr 26, 2026Dawon Choi, Hana Kim, Ji-Hoon KimEfficient Neural Network Inference Layer Normalization
Dept. of Artificial Intelligence Semiconductor Engineering, Hanyang University, Seoul, Republic of Korea · Dept. of Electronic Engineering, Hanyang University, Seoul, Republic of Korea
When Does Removing LayerNorm Help? Activation Bounding as a Regime-Dependent Implicit Regularizer
Apr 25, 2026Lucky VermaNeural Network Activation Functions Layer Normalization
Independent Researcher
Does Your Optimizer Care How You Normalize? Normalization-Optimizer Coupling in LLM Training
Apr 2, 2026Abdelrahman AbouzeidNeural Network Optimization Layer Normalization
Georgia Institute of Technology
The Geometric Cost of Normalization: Affine Bounds on the Bayesian Complexity of Neural Networks
Mar 28, 2026Sungbae ChunLayer Normalization Neural Representation Geometry
Department of Mathematics, Universität Bonn
MXNorm: Reusing MXFP block scales for efficient tensor normalisation
Mar 13, 2026Callum McLean, Luke Y. Prince, Alexandre Payot +2Layer Normalization
Graphcore
DP-aware AdaLN-Zero: Taming Conditioning-Induced Heavy-Tailed Gradients in Differentially Private Diffusion
Feb 26, 2026Tao Huang, Jiayang Meng, Xu Yang +2Conditional Diffusion Models Privacy-Preserving ML
School of Computer Science and Big Data, Minjiang University, Fuzhou, Fujian, China. · School of Information, Renmin University of China, Beijing, China.
Parallel Layer Normalization for Universal Approximation
May 19, 2025Yunhao Ni, Yuxin Guo, Yuhe Liu +4Layer Normalization Neural Network Approximation Theory
State Key Laboratory of Complex and Critical Software Environment (SKLCCSE), Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China

Oct 15, 2026J/K move · Enter open · S save

Activation-Adjacent Layer Normalization: Effects on Approximation Power

Yunhao Ni, Yuxin Guo, Jie Luo, Wenjun Wu+1

State Key Laboratory of Complex and Critical Software Environment, Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China

Layer Normalization Neural Network Approximation Theory Shallow Neural Networks

Open paper AskPDF arXiv

Layer normalization (LN) and root mean square normalization (RMSNorm) depend on their inputs and remain nonlinear at inference. Inserting either next to an activation may therefore change the function a network represents. We study when shallow activation-normalized (AN) and preactivation-normalized (PN) networks can approximate a prescribed width-

N

one-hidden-layer MLP. Viewing RMSNorm as input-dependent scaling, we use width augmentation to locally linearize normalization on compact feature sets. For AN networks, an unbounded activation range allows width

N+1

, while every continuous nonconstant activation admits width

N+r

with error

O_f(r^{-1})

. For PN networks, affine parameters or, without them, degree-one positive homogeneity recover width

N+1

; general continuous activations admit width

N+r

with error tending to zero. The corresponding LN constructions use centered coordinate pairs, yielding widths

N+2

and

N+2r

, respectively. We prove that

N+1

for RMSNorm and

N+2

for LN are also the worst-case bound for approximations, and show that applicable universal-approximation theorems for ordinary MLPs transfer to these normalized architectures. Numerical experiments recover the predicted approximation rates for both normalizations across all five settings and illustrate the shifted-ReLU finite-width obstruction.