LayerNorm and RMSNorm impose fundamentally different geometric constraints on their outputs - and this difference has a precise, quantifiable consequence for model complexity. We prove that LayerNorm's mean-centering step, by confining data to a linear hyperplane (through the origin), reduces the Local Learning Coefficient (LLC) of the subsequent weight matrix by exactly m/2 (where m is its output dimension); RMSNorm's projection onto a sphere preserves the LLC entirely. This reduction is structurally guaranteed before any training begins, determined by data manifold geometry alone. The underlying condition is a geometric threshold: for the codimension-one manifolds we study, the LLC drop is binary -- any non-zero curvature, regardless of sign or magnitude, is sufficient to preserve the LLC, while only affinely flat manifolds cause the drop. At finite sample sizes this threshold acquires a smooth crossover whose width depends on how much of the data distribution actually experiences the curvature, not merely on whether curvature exists somewhere. We verify both predictions experimentally with controlled single-layer scaling experiments using the wrLLC framework. We further show that Softmax simplex data introduces a "smuggled bias" that activates the same m/2 LLC drop when paired with an explicit downstream bias, proved via the affine symmetry extension of the main theorem and confirmed empirically.
Figure 1: Curvature geometry and effective LLC. Left: Median LLC per manifold class ( d=5 , m=4 , 5 seeds). All curved surfaces match the Gaussian baseline ( ≈12.0 ); only the flat hyperplane drops to ≈9.9≈12.0−m/2 . Curvature sign is irrelevant: the threshold is between flat and non-flat. Right: LLC vs. bump amplitude A for wide ( α=0.1 ) and narrow ( α=10.0 ) bumps. The wide bump recovers near A∗≈0.1 ; the narrow bump stays near the flat bound until A≥1.0 , showing that the effective RLCT depends on data–curvature overlap, not just the existence of curvature.
Figure 2: LLC reduction under LayerNorm and RMSNorm . Left: Varying output dimension m with fixed d=12 . Δλ is consistent with the m/2 prediction for LayerNorm (with increasing variance at large m ); RMSNorm shows Δλ≈0 throughout. Right: Varying input dimension d with fixed m=4 . Δλ≈2.0 regardless of d , confirming that the drop depends only on the number of dimensions lost to the constraint.
Configuration
Baseline ( λ )
Simplex ( λ )
Δλ
Expected
Strictly Linear ( b=0 )
10.38
10.62
−0.24
0.0
Explicit Bias ( +b )
12.40
10.14
2.26
m/2=2.0
Table 3: LLC under Softmax simplex geometry. d=5 , m=4 . Δλ measured relative to Gaussian-input baseline (5-seed median). The simplex alone causes no drop; degeneracy requires a downstream operation sensitive to the affine offset. The Post- LayerNorm case is omitted: SGLD estimation is unreliable for the non-affine LN loss landscape (see Conjecture 1 ).
Why does weight decay work? We prove that, in any fixed-precision regime, the smallest weight norm of a looped neural network outputting a binary string equals the Kolmogorov complexity of that string, up to a logarithmic factor. This implies that weight decay induces a prior matching Solomonoff's universal prior, the optimal prior over computable functions, up to a polynomial factor. The result is norm-agnostic: in fixed precision, every weight norm collapses to the non-zero parameter count up to constants, so the same sandwich bound holds for any norm used as a regulariser. The proof has two short reductions: any program for a universal Turing machine can be encoded into neural weights at unit cost per program bit, and any fixed-precision network can be described by enumerating its non-zero parameters with logarithmic addressing overhead. Both bounds are tight up to constants, with the logarithmic factor realised by permutation encodings: a network whose parameters encode a permutation produces a string whose Kolmogorov complexity is the non-zero parameter count times its logarithm. The fixed-precision assumption is essential: with infinite precision, neural networks can encode non-computable functions and the weight norm loses its relevance.
Layer normalization (LN) is a fundamental component in modern deep learning, but its per-sample centering and scaling introduce non-negligible inference overhead. RMSNorm improves efficiency by removing the centering operation, yet this may discard benefits associated with centering. This paper propose a framework to determine whether an LN in an arbitrary DNN can be replaced by RMSNorm without changing the model function. The key idea is to fold LN's centering operation into upstream general linear layers by enforcing zero-mean outputs through the column-centered constraint (CCC) and column-based weight centering (CBWC). We extend the analysis to arbitrary DNNs, define such LNs as foldable LNs, and develop a graph-based detection algorithm. Our analysis shows that many LNs in widely used architectures are foldable, enabling exact inference-time conversion and end-to-end acceleration of 2% to 12% without changing model predictions. Experiments across multiple task families further show that, when exact equivalence is partially broken in practical training settings, our method remains competitive with vanilla LN while improving efficiency.
Yuxin Guo, Yihao Yue, Yunhao Ni +4
State Key Laboratory of Complex and Critical Software Environment (SKLCCSE), Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China
Low-rank adaptation (LoRA) is now a standard approach to parameter-efficient fine-tuning of large neural networks. Its low-rank factorization is inherently non-identifiable, making LoRA a singular statistical model whose learning complexity cannot be described by parameter dimension alone. We study this complexity using the real log canonical threshold (RLCT) from singular learning theory and how it depends on both the low-rank parameterization and the pretrained network. Under a local linearization, LoRA induces a low-rank trace regression model whose Kullback--Leibler geometry is controlled by a forward-backward covariance, constructed from activations and backpropagated sensitivities at the adapted layer. This covariance determines which low-rank perturbations are statistically observable. When it is nondegenerate along the relevant low-rank directions, we recover the classical learning complexity of reduced-rank regression; when its kernel meets these directions, additional statistically indistinguishable parameters change the singular geometry. We characterize this geometry locally through observable tangent directions and a residual singularity, and globally through affine linear sections of determinantal varieties. The global learning complexity need not be determined at a ground-truth parameter, as singularities on another rank stratum can dominate. We also introduce a curvature-corrected multi-temperature estimator of the RLCT. Experiments on multilayer perceptron and transformer architectures reveal strongly layer-dependent forward-backward geometry and show that ambient covariance degeneracy does not imply degeneracy along the low-rank directions relevant to adaptation.
Kamillo Ferry, Anthea Monod, Keisuke Yano
Department of Mathematics, Imperial College London, UK · Institute of Statistical Mathematics, Japan