Layer Normalization
Overview
Layer normalization (LayerNorm) centers and scales activations within a layer, while related methods such as RMSNorm scale without centering. Research compares where normalization is placed, studies conditional variants such as AdaLN in diffusion models, and develops theory linking normalization to approximation power, representation rank, and training dynamics. Experiments span language models, vision, diffusion, and graph networks, alongside methods that fold or approximate normalization to reduce inference and hardware costs. A central tension is that normalization can improve stability and expressivity in some settings but hurt learning in others, depending on its placement, optimizer, model capacity, and data regime.
Latest papers
32 papers2 papers in the last four weeks, down 50% on the four weeks before. 0.0% of all new papers.