cs.LGNov 9, 2026

KiteNorm: Variance Regularisation for Stable and Scalable Post-LN Transformers

Authors: Leon A. Trochelmann, Sajad Movahedi, Shiwei Liu, Antonio Orvieto

Organizations: ELLIS Institute Tübingen · MPI for Intelligent Systems · Tübingen AI Center

Abstract

Post-LayerNorm Transformers have seen limited practical adoption due to enduring difficulties in scaling them to large depths. While prior research has focused on stabilising Post-LayerNorm by improving conditioning at initialisation, stability often deteriorates when models are trained with large learning rates, forcing additional compromises. In the decoder-only settings we study, existing Post-LayerNorm methods fail to outperform strong Pre-LayerNorm baselines. We propose KiteNorm, a post-norm training method designed to break this trend by constraining hidden-state variance. This approach is motivated by a bound on the LayerNorm Jacobian suggesting that variance growth contributes to gradient vanishing, alongside empirical evidence linking it to rank collapse. KiteNorm combines residual scaling and variance regularisation to maintain stability throughout training, supported by scalarised double-normalisation that provides separately learnable scales for the skip and residual branches. Across decoder-only Transformers up to 1.3B parameters, KiteNorm improves training stability and outperforms leading baselines at their optimal learning rates, with scaling laws favouring KiteNorm across depth, width, batch size, and training budget. Ablations show that each component is necessary for the full stability and performance gains.

Figures & tables

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Aug 10, 2026

Why Post-Norm Transformers Collapse: Attention Amplification and Gradient Repair Failure

Deep decoder-only Transformers often replace the original Post-Norm architecture with Pre-Norm variants because Post-Norm training is highly sensitive to warmup and learning rate under conventional initialization schemes. Although prior work has identified rank collapse and gradient vanishing as related symptoms, it remains poorly understood how causal attention creates high-similarity representations and why training dynamics fail to repair them. We give a two-stage analysis of Post-Norm rank collapse using token similarity as a scalar state variable. First, at initialization, causal attention acts approximately as a prefix-averaging operator that increases token similarity across depth, while the SwiGLU branch contributes only a smaller damping effect. Second, once training enters a high-similarity regime, growth of pre-normalization residual norms makes the RMSNorm backward factor contractive; under mild conditions, gradients to earlier layers decay geometrically. As a complementary result, we characterize the properties of a collapsed network: its best predictor is frequency distribution with relatively high loss floor, and gradients in collapsed layers vanish at frequency distribution. Experiments on 48-layer decoder-only Transformers trained on C4 dataset match the predicted initialization-time similarity growth and collapse-time gradient contraction, and show that collapsed runs stay near the predicted frequency loss. Together, these results distinguish the forward similarity amplification and backward repair incapacity in Post-Norm collapse, while also characterizing the behavior of collapsed networks.
Jul 12, 2026

Spectral Margins and Memory Routing in Looped Transformers

Looped transformers apply one block of weights repeatedly. Each iteration keeps a fraction of the previous hidden state, the carry, and adds the block's output, so information can pass between iterations through either path. We study a small, LayerNorm-based variant of the Parcae architecture and ask two questions: does the iterated state settle, and does learning to remember across iterations require the carry? On stability, pre-LayerNorm makes the block less sensitive at its fixed point as the carry grows, but not enough to guarantee contraction: with a scalar carry, a perturbation can grow in a single iteration even at a locally stable fixed point. These fixed points are stable because the spectral radius of the loop's Jacobian stays below 11; its distance from 11, the spectral margin, narrows as the carry approaches 11 and, at high carry, as the width grows, and some initializations do not settle within the iteration budget. On memory, a multi-hop copy task trained at a fixed loop depth is learned as well with the carry frozen at 0.50.5, 0.020.02, or 00 as with a learned carry, which changes little during training. When the depth is sampled during training, a carry of 0.50.5 still trains but 0.020.02 and 00 do not. All experiments are small (d≤32d \le 32 for training).
Oct 6, 2026

LayerRoPE: Dynamic Depth-wise Magnitude & Angular Superposition

As data propagates through a Transformer, the norm of its hidden states grows by orders of magnitude with depth, a phenomenon framed as 'curse of depth' and nearly universally treated as a pathology to be suppressed. We take the opposite view. Across 16 pre-trained LLMs from 9 families, spanning dense, mixture-of-experts and hybrid architectures and Pre-, Peri- and Post-Norm designs, we find that this growth reflects an emergent depth-positional encoding, carried by the only learned per-layer gain on the residual stream, the normalization weight γγ: with depth, γγ grows in magnitude and rotates in direction, jointly encoding the layer index. We make this depth-conditioned encoding explicit with LayerRoPE, an implicit analog of RoPE along the depth axis, which replaces all layerwise γγ vectors with a single shared vector and depth-conditioned scalars, at a net reduction in parameters and <0.02%<0.02\% change in FLOPs. Across a model ladder scaled up to 100100B+ tokens, LayerRoPE consistently outperforms Pre-, Post- and Peri-Norm and Layer-Norm Scaling, reaching Pre-Norm's 1.3B loss with 3.4×3.4\times less compute; LayerRoPE is the only approach that shows strong convergence and improves near monotonically as depth scales to 512 layers. It improves learning-rate sensitivity by 33-10×10\times, and transfers naively to and consistently improves looped latent models and Vision Transformers. Inspecting its learned schedule inverts the prevailing premise: LayerRoPE does not shrink the residual stream but widens it, damping what each block reads while amplifying what it writes. Depth stability, our results suggest, calls not for suppressing the residual stream, but for depth-conditioned regulation of the computational blocks it feeds.