KiteNorm: Variance Regularisation for Stable and Scalable Post-LN Transformers
Organizations: ELLIS Institute Tübingen · MPI for Intelligent Systems · Tübingen AI Center
Abstract
Post-LayerNorm Transformers have seen limited practical adoption due to enduring difficulties in scaling them to large depths. While prior research has focused on stabilising Post-LayerNorm by improving conditioning at initialisation, stability often deteriorates when models are trained with large learning rates, forcing additional compromises. In the decoder-only settings we study, existing Post-LayerNorm methods fail to outperform strong Pre-LayerNorm baselines. We propose KiteNorm, a post-norm training method designed to break this trend by constraining hidden-state variance. This approach is motivated by a bound on the LayerNorm Jacobian suggesting that variance growth contributes to gradient vanishing, alongside empirical evidence linking it to rank collapse. KiteNorm combines residual scaling and variance regularisation to maintain stability throughout training, supported by scalarised double-normalisation that provides separately learnable scales for the skip and residual branches. Across decoder-only Transformers up to 1.3B parameters, KiteNorm improves training stability and outperforms leading baselines at their optimal learning rates, with scaling laws favouring KiteNorm across depth, width, batch size, and training budget. Ablations show that each component is necessary for the full stability and performance gains.
Figures & tables
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| HYPERPARAMETER | SHARED VALUE |
|---|---|
| training corpus | SlimPajama-627B |
| training objective | next-token cross entropy |
| sequence length | 2048 |
| vocabulary size | 50280 |
| 512 | |
| 1536 |
| METHOD | SUBLAYER IMPLEMENTATION | SCALING DETAILS |
|---|---|---|
| GPT-2 pre-norm | , with a final output LayerNorm. Standard LayerNorm includes the shift parameter. | Runtime skip and residual scales are both . The Attention output projection and GLU output projection in every Transformer layer are multiplied at initialisation by . |
| LayerNorm-Scaling | Pre-norm with LN replaced in both sublayers of Transformer layer by . The final output normalisation is standard LayerNorm, not LayerNorm-Scaling. | Runtime skip and residual scales are both . No residual output projection scaling is applied at initialisation. Standard LayerNorm includes the shift parameter. |
| Post-norm | , without a final output LayerNorm. Standard LayerNorm includes the shift parameter. | Runtime skip and residual scales are both . No residual output projection scaling is applied at initialisation. |
| KEEL | Double-norm: . The outer LayerNorm after the first Attention sublayer is omitted. Standard LayerNorm is used without the shift parameter. | For the two sublayers belonging to Transformer layer , . For all sublayers belonging to Transformer layers , . The residual branch scale is always . No residual output projection scaling is applied at initialisation. The main comparison uses GLUs. The equal-parameter MLP variant is only used in Appendix C.2 . |
| Peri-norm | . There is no final output LayerNorm. Standard LayerNorm includes the shift parameter. | Runtime skip and residual scales are both . No residual output projection scaling is applied at initialisation. |
| Mix-norm | The first Transformer layers are post-norm layers and the remaining Transformer layers are pre-norm layers. A final output LayerNorm is used because the model ends with a pre-norm suffix. Standard LayerNorm includes the shift parameter. | Runtime skip and residual scales are both . Let . Only the Attention output projection and GLU output projection in the pre-norm suffix are multiplied at initialisation by , following the partial GPT-2 initialisation selected in Appendix C.3 . |
| Method | 2e-3 | 5e-3 | 1e-2 | 2e-2 | 5e-2 |
|---|---|---|---|---|---|
| KiteNorm | – | 25.181 | 24.831 | 25.180 | – |
| GPT-2 pre-norm | – | 26.909 | 26.660 | div. | – |
| LayerNorm-Scaling | – | 26.991 | 26.645 | 26.615 | div. |
| KEEL | 28.296 | 27.644 | 28.364 | – | – |
| Method | 1e-3 | 2e-3 | 5e-3 | 1e-2 |
|---|---|---|---|---|
| KiteNorm | – | 16.371 | 16.189 | 16.719 |
| GPT-2 pre-norm | 16.991 | 16.755 | 16.995 | 17.616 |
| KEEL | 18.101 | 17.126 | 16.889 | div. |
| LayerNorm-Scaling | – | 18.153 | 17.133 | 17.167 |
| ARC-Easy | PIQA | |||
|---|---|---|---|---|
| Method | acc | acc_norm | acc | acc_norm |
| KiteNorm | 0.455 | 0.404 | 0.628 | 0.625 |
| GPT-2 pre-norm | 0.426 | 0.383 | 0.609 | 0.621 |
| LayerNorm-Scaling | 0.436 | 0.381 | 0.633 | 0.624 |
| KEEL | 0.426 | 0.385 | 0.612 | 0.616 |
| Peri-norm | 0.414 | 0.381 | 0.625 | 0.613 |
| Configuration | 2e-3 | 5e-3 | 1e-2 | 2e-2 |
|---|---|---|---|---|
| KEEL | 20.766 | 20.420 | div. | – |
| No double-norm | 21.382 | div. | div. | – |
| No exceptions | 20.379 | 19.612 | 19.640 | div. |
| Neither | 21.093 | 19.936 | div. | – |
| Residual scale | 2e-3 | 5e-3 | 1e-2 |
|---|---|---|---|
| 0.05 | 25.953 | 24.380 | div. |
| 0.02 | 27.141 | 24.941 | 26.176 |
| 0.01 | 28.615 | 26.242 | 27.840 |
| Scaling rule | 2e-3 | 5e-3 | 1e-2 |
|---|---|---|---|
| 19.726 | 19.080 | 19.100 | |
| – | 19.125 | 19.199 |
| 2e-3 | 5e-3 | 1e-2 | |
|---|---|---|---|
| 0.1 | 19.755 | 19.190 | 19.277 |
| 1.0 | 19.726 | 19.080 | 19.100 |
| 10.0 | 20.036 | 19.431 | 19.337 |
| Configuration | 2e-3 | 5e-3 | 1e-2 | 2e-2 |
|---|---|---|---|---|
| KiteNorm, scalarLN | 19.726 | 19.080 | 19.100 | 19.868 |
| KiteNorm, scalarRMSNorm | 19.811 | 19.158 | 19.253 | 21.113 |
| KEEL, LN | 20.766 | 20.420 | div. | – |
| KEEL, RMSNorm | 20.816 | 20.321 | div. | – |
| Configuration | 2e-3 | 5e-3 | 1e-2 | 2e-2 |
|---|---|---|---|---|
| KiteNorm, 6% | 19.726 | 19.080 | 19.100 | 19.868 |
| KiteNorm, 12% | 19.920 | 19.240 | 19.123 | 19.565 |
| GPT-2 pre-norm, 6% | 20.376 | 20.240 | 20.951 | div. |
| GPT-2 pre-norm, 12% | 20.265 | 20.172 | 20.139 | 21.743 |
| KEEL, 6% | 20.766 | 20.420 | div. | – |
| KEEL, 12% | 20.758 | 19.829 | 20.115 | – |
| Configuration | 2e-3 | 5e-3 | 1e-2 |
|---|---|---|---|
| KiteNorm, 0.9 | 20.072 | 19.400 | 19.290 |
| KiteNorm, 0.95 | 19.726 | 19.080 | 19.100 |
| KiteNorm, 0.975 | 19.725 | 19.219 | 19.589 |
| GPT-2 pre-norm, 0.9 | 20.699 | 20.515 | div. |
| GPT-2 pre-norm, 0.95 | 20.376 | 20.240 | 20.951 |
| GPT-2 pre-norm, 0.975 | 20.761 | 20.562 | 20.707 |
Transformer
Transformers are neural architectures that use attention to model relationships among sequence elements in language, vision, and scientific data.
See all 579 papers