Post-LayerNorm Transformers have seen limited practical adoption due to enduring difficulties in scaling them to large depths. While prior research has focused on stabilising Post-LayerNorm by improving conditioning at initialisation, stability often deteriorates when models are trained with large learning rates, forcing additional compromises. In the decoder-only settings we study, existing Post-LayerNorm methods fail to outperform strong Pre-LayerNorm baselines. We propose KiteNorm, a post-norm training method designed to break this trend by constraining hidden-state variance. This approach is motivated by a bound on the LayerNorm Jacobian suggesting that variance growth contributes to gradient vanishing, alongside empirical evidence linking it to rank collapse. KiteNorm combines residual scaling and variance regularisation to maintain stability throughout training, supported by scalarised double-normalisation that provides separately learnable scales for the skip and residual branches. Across decoder-only Transformers up to 1.3B parameters, KiteNorm improves training stability and outperforms leading baselines at their optimal learning rates, with scaling laws favouring KiteNorm across depth, width, batch size, and training budget. Ablations show that each component is necessary for the full stability and performance gains.
Figures & tables
Figure 1: Effect of residual scaling on training failures observed when training a 12-layer post-norm Transformer according to the setup in Appendix B with learning rate 10−2 . We report statistics over the hidden state after the residual addition in the final Attention sublayer. Figures 1(a) & 1(b) show the behaviour of vanilla post-norm with residual scaling factor c=1 , whereas figures 1(c) and 1(d) show the effects of other choices of c .
Figure 2: Final validation perplexities of KiteNorm and competitive LayerNorm methods in a 48-layer Transformer.
Figure 3: Depth scaling law.
Figure 4: Scaling behaviour of KiteNorm and GPT-2 pre-norm across training budget, batch size, and width in 12-layer Transformers.
Figure 5: Performance of 1B-parameter models in Pythia configuration.
Figure 6: Training stability of Llama-style 1.3B-parameter models.
Figure 7: Ablation studies on KiteNorm in decoder-only Transformers.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
HYPERPARAMETER
SHARED VALUE
training corpus
SlimPajama-627B
training objective
next-token cross entropy
sequence length
2048
vocabulary size
50280
dmodel
512
dGLU
1536
Appendix
Table 1: List of shared hyperparameters.
METHOD
SUBLAYER IMPLEMENTATION
SCALING DETAILS
GPT-2 pre-norm
Xl+1=Xl+Fl(LN(Xl)) , with a final output LayerNorm. Standard LayerNorm includes the shift parameter.
Runtime skip and residual scales are both 1 . The Attention output projection and GLU output projection in every Transformer layer are multiplied at initialisation by 2L1 .
LayerNorm-Scaling
Pre-norm with LN replaced in both sublayers of Transformer layer k by LNSk(X)=k−1/2LN(X) . The final output normalisation is standard LayerNorm, not LayerNorm-Scaling.
Runtime skip and residual scales are both 1 . No residual output projection scaling is applied at initialisation. Standard LayerNorm includes the shift parameter.
Post-norm
Xl+1=LN(Xl+Fl(Xl)) , without a final output LayerNorm. Standard LayerNorm includes the shift parameter.
Runtime skip and residual scales are both 1 . No residual output projection scaling is applied at initialisation.
KEEL
Double-norm: Xl+1=LN(Mixal,1(Xl,Fl(LN(Xl)))) . The outer LayerNorm after the first Attention sublayer is omitted. Standard LayerNorm is used without the shift parameter.
For the two sublayers belonging to Transformer layer k=1 , al=1 . For all sublayers belonging to Transformer layers k≥2 , al=2L . The residual branch scale is always 1 . No residual output projection scaling is applied at initialisation. The main comparison uses GLUs. The equal-parameter MLP variant is only used in Appendix C.2 .
Peri-norm
Xl+1=Xl+LN(Fl(LN(Xl))) . There is no final output LayerNorm. Standard LayerNorm includes the shift parameter.
Runtime skip and residual scales are both 1 . No residual output projection scaling is applied at initialisation.
Mix-norm
The first ⌊0.25L⌋ Transformer layers are post-norm layers and the remaining L−⌊0.25L⌋ Transformer layers are pre-norm layers. A final output LayerNorm is used because the model ends with a pre-norm suffix. Standard LayerNorm includes the shift parameter.
Runtime skip and residual scales are both 1 . Let Lpre=L−⌊0.25L⌋ . Only the Attention output projection and GLU output projection in the pre-norm suffix are multiplied at initialisation by 2Lpre1 , following the partial GPT-2 initialisation selected in Appendix C.3 .
Appendix
Table 2: Exact implementation of the normalisation and scaling schemes in the comparison of competitive approaches.
Figure 8: Learning-rate explorations for pre-norm variants.
Figure 9: Comparison between KEEL models with GLU and models with MLP in a 12-layer Transformer.
Figure 10: Comparison between vanilla mix-norm and mix-norm with GPT-2 style residual scaling in a 12-layer Transformer.
Figure 11: Rank collapse in hidden states after the residual addition in Attention sublayers when training a 12-layer Transformer with peri-norm.
Method
2e-3
5e-3
1e-2
2e-2
5e-2
KiteNorm
–
25.181
24.831
25.180
–
GPT-2 pre-norm
–
26.909
26.660
div.
–
LayerNorm-Scaling
–
26.991
26.645
26.615
div.
KEEL
28.296
27.644
28.364
–
–
Appendix
Table 3: Final validation perplexities across learning rates in 12-layer Transformers.
Method
1e-3
2e-3
5e-3
1e-2
KiteNorm
–
16.371
16.189
16.719
GPT-2 pre-norm
16.991
16.755
16.995
17.616
KEEL
18.101
17.126
16.889
div.
LayerNorm-Scaling
–
18.153
17.133
17.167
Appendix
Table 4: Final validation perplexities across learning rates in 96-layer Transformers.
ARC-Easy
PIQA
Method
acc
acc_norm
acc
acc_norm
KiteNorm
0.455
0.404
0.628
0.625
GPT-2 pre-norm
0.426
0.383
0.609
0.621
LayerNorm-Scaling
0.436
0.381
0.633
0.624
KEEL
0.426
0.385
0.612
0.616
Peri-norm
0.414
0.381
0.625
0.613
Appendix
Table 5: Zero-shot downstream evaluation of the best final checkpoints from the 48-layer comparison.
Figure 12: Complete training loss curves of Llama-style 1.3B-parameter models across learning rates.
Figure 13: Comparison between training of GPT-2 pre-norm with and without the ReLU regulariser enabled in a 12-layer Transformer.
Configuration
2e-3
5e-3
1e-2
2e-2
KEEL
20.766
20.420
div.
–
No double-norm
21.382
div.
div.
–
No exceptions
20.379
19.612
19.640
div.
Neither
21.093
19.936
div.
–
Appendix
Table 6: Final validation perplexities of KEEL architectural variants in a 48-layer Transformer across learning rates.
Residual scale
2e-3
5e-3
1e-2
0.05
25.953
24.380
div.
0.02
27.141
24.941
26.176
0.01
28.615
26.242
27.840
Appendix
Table 7: Final validation perplexities in 24-layer post-norm models with variance regularisation and varying residual scaling.
Scaling rule
2e-3
5e-3
1e-2
1/(2L)
19.726
19.080
19.100
1/L
–
19.125
19.199
Appendix
Table 8: Final validation perplexities in 48-layer KiteNorm models with varying residual scaling.
λ
2e-3
5e-3
1e-2
0.1
19.755
19.190
19.277
1.0
19.726
19.080
19.100
10.0
20.036
19.431
19.337
Appendix
Table 9: Final validation perplexities in 48-layer KiteNorm models with different regularisation weights.
Configuration
2e-3
5e-3
1e-2
2e-2
KiteNorm, scalarLN
19.726
19.080
19.100
19.868
KiteNorm, scalarRMSNorm
19.811
19.158
19.253
21.113
KEEL, LN
20.766
20.420
div.
–
KEEL, RMSNorm
20.816
20.321
div.
–
Appendix
Table 10: Final validation perplexities in 48-layer Transformers with differently parameterised LayerNorm and RMSNorm variants.
Configuration
2e-3
5e-3
1e-2
2e-2
KiteNorm, 6%
19.726
19.080
19.100
19.868
KiteNorm, 12%
19.920
19.240
19.123
19.565
GPT-2 pre-norm, 6%
20.376
20.240
20.951
div.
GPT-2 pre-norm, 12%
20.265
20.172
20.139
21.743
KEEL, 6%
20.766
20.420
div.
–
KEEL, 12%
20.758
19.829
20.115
–
Appendix
Table 11: Final validation perplexities in 48-layer Transformers with differing warm-up durations.
Configuration
2e-3
5e-3
1e-2
KiteNorm, 0.9
20.072
19.400
19.290
KiteNorm, 0.95
19.726
19.080
19.100
KiteNorm, 0.975
19.725
19.219
19.589
GPT-2 pre-norm, 0.9
20.699
20.515
div.
GPT-2 pre-norm, 0.95
20.376
20.240
20.951
GPT-2 pre-norm, 0.975
20.761
20.562
20.707
Appendix
Table 12: Final validation perplexities in 48-layer Transformers when trained with different Adam beta parameters.