cs.LGNov 9, 2026

KiteNorm: Variance Regularisation for Stable and Scalable Post-LN Transformers

Authors: Leon A. Trochelmann, Sajad Movahedi, Shiwei Liu, Antonio Orvieto

Organizations: ELLIS Institute Tübingen · MPI for Intelligent Systems · Tübingen AI Center

Abstract

Post-LayerNorm Transformers have seen limited practical adoption due to enduring difficulties in scaling them to large depths. While prior research has focused on stabilising Post-LayerNorm by improving conditioning at initialisation, stability often deteriorates when models are trained with large learning rates, forcing additional compromises. In the decoder-only settings we study, existing Post-LayerNorm methods fail to outperform strong Pre-LayerNorm baselines. We propose KiteNorm, a post-norm training method designed to break this trend by constraining hidden-state variance. This approach is motivated by a bound on the LayerNorm Jacobian suggesting that variance growth contributes to gradient vanishing, alongside empirical evidence linking it to rank collapse. KiteNorm combines residual scaling and variance regularisation to maintain stability throughout training, supported by scalarised double-normalisation that provides separately learnable scales for the skip and residual branches. Across decoder-only Transformers up to 1.3B parameters, KiteNorm improves training stability and outperforms leading baselines at their optimal learning rates, with scaling laws favouring KiteNorm across depth, width, batch size, and training budget. Ablations show that each component is necessary for the full stability and performance gains.

Figures & tables

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Why Post-Norm Transformers Collapse: Attention Amplification and Gradient Repair Failure

    Aug 10, 2026Xingjian Wang, Qingyu Han, Xiaodong Luo +1TransformerSelf-Attention

  2. Spectral Margins and Memory Routing in Looped Transformers

    Jul 12, 2026Matthias M. M. BuehlmaierRecurrent TransformersLayer Normalization

  3. LayerRoPE: Dynamic Depth-wise Magnitude & Angular Superposition

    Oct 6, 2026Shikhar Srivastava, Christopher KananTransformerPositional Encoding