Signed-Permutation Coordinate Transport for RMSNorm Transformers

John Sweeney

cs.LGJun 30, 2026

Signed-Permutation Coordinate Transport for RMSNorm Transformers

Authors: John Sweeney

Organizations: Sideplane AI

Abstract

Modern LLM workflows move coordinate-indexed objects across checkpoints: steering vectors, sparse autoencoders, top- $k$ neuron sets, attribution lists, and merge alignments. This is only well posed after fixing the model's residual-stream gauge, which we show is architecture-dependent: LayerNorm residual charts have permutation gauge $S_d$ (up to a global sign flip), while RMSNorm charts with generic per-channel gain have signed-permutation gauge $B_d = S_d \ltimes \{\pm 1\}^d$ . Permutation-only alignment is therefore symmetry-incomplete for RMSNorm models. We introduce sign-marginalized Hungarian matching and prove a sharp failure mode: with decorrelated coordinates, raw signed-correlation matching has a structural permutation-accuracy ceiling at the positive-sign fraction of the true gauge, which sign-marginalization removes. We then make coordinate-preserving transport, not function-level merging, the primary object: composing saved-checkpoint local $B_d$ gauges along same-base fine-tuning trajectories recovers 91.1% of cross-run coordinates at 1500 steps versus 60.3% for endpoint matching, and the gain is not explained by merely routing through the base. The recovered gauge transfers tools that permutation-only alignment breaks: TinyLlama SAE reconstruction has NMSE 0.004 under $B_d$ versus 1.08 under $S_d$ ; Qwen sentiment steering preserves 95.8% of its effect versus 17.2%; refusal steering reverses sign under $S_d$ ; coordinate-preserving merges behave the same way. The same covariance governs stateful training: signed transport of AdamW state preserves the resumed trajectory, while permutation-only state follows a different one from a functionally identical checkpoint. Finally, gauge-sweep audits show index-level interpretability claims are reproducible only relative to an explicit gauge.

Explore similar work

May 18, 2026cs.LG

A Geometric Analysis of Sign-Magnitude Asymmetry in a ReLU + RMSNorm Block under Ternary Quantization

Pre-norm Transformers with RMSNorm tolerate ternary {-1,0,+1} weight quantization with surprisingly small loss (Ma et al., 2024). We give a geometric explanation via sign-magnitude decomposition of weight perturbations. In a two-layer ReLU + RMSNorm model with i.i.d. Gaussian weights, sign-flips produce

π/(π-2) \approx 2.75

times more transverse output energy than sign-preserving magnitude perturbations of equal Frobenius norm, as the flip rate

p \to 0

(Theorem 3). The mechanism: ReLU creates a hidden-space directional asymmetry between the two perturbation types, which RMSNorm's transverse-projection Fréchet derivative selectively exposes. Sign-quantization error is itself a sign-preserving perturbation with angular alignment

\cos^2 \to 2/π

(Theorem 4); its post-ReLU radial fraction (

0.365

) matches the pre-ReLU value

1-2/π

within

0.4\%

, so ReLU is approximately transparent to ternary error. Multi-layer compounding of the

2.75\times

factor is not experimentally supported; the gap to real-model sign sensitivity arises from outlier features violating delocalization. For an input dimension with amplitude

α

, a single sign-flip produces post-ReLU energy amplified by

R \approx nα^2

relative to a delocalized entry. On TinyLlama-1.1B, at linear response (

p \leq 0.5\%

), count-matched NLL leverage stabilizes at

\sim 10\times \approx n\mathbb{E}[α^2]

, matching the per-entry theory; the all-column NLL ratio of

5.0\times

falls within

R_{\mathrm{col}} \leq 19

(

67\times

PPL gap reflects metric nonlinearity). Measured outlier

α

at layer 12 (median

0.024

, max

0.26

) confirms heavy-tailed concentration. The Bussgang constant

2/π

, RMSNorm geometry, and ReLU half-space structure together explain sign-magnitude asymmetry in pre-norm models, with

R \propto nα^2

accounting for real-model deviations.

Lei Dong

Batch Normalization Rectified Linear Unit Rectified Linear Unit Networks Model Weights Asymmetry

Ask PDF

Jun 17, 2026cs.LG

Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale

Pretrained transformers sit near singular minima of the loss, where the Fisher information metric degenerates along dead directions: directions in parameter space along which the directional Fisher vanishes. Locating such a direction normally needs a forward pass and an eigendecomposition of activations, or a sampling-based complexity estimate; none returns a direction computable from the network's parameters alone. We give one, for LayerNorm transformers. The inverse-scale direction

γ^{-1}/\|γ^{-1}\|

of the LayerNorm affine is an exact algebraic kernel of the post-final-norm centred activation covariance, for any input distribution, and induces a corresponding dead direction in parameter space. It is read from the LN scale parameter alone, with no forward or backward pass and no eigensolve: the cheapest dead-direction read, specific to LayerNorm. We test it on

14

pretrained transformers (

9

LayerNorm,

5

RMSNorm;

160

M-

35

B; language and vision objectives). At random initialisation the predicted direction matches the measured bottom singular direction (one forward pass, direct SVD) to four decimal places on

9/9

LayerNorm models, and is correctly absent on

5/5

RMSNorm models, which lack the mean-subtraction projector that creates it. On the trained checkpoint the covariance eigenvalue along this direction deepens by

{\sim}10^3\times

and further dead directions open; the random-init-to-trained gap is a one-forward-pass, per-checkpoint readout of singular structure along the predicted coordinate. Two consequences follow in closed form: the residual stream's smallest singular value is preserved block-to-block on

13/14

transformers measured on their own input distribution, the one exception (Gemma

4

31

B) a genuine dead direction the same read pinpoints; and the kernel direction's presence classifies a transformer's normalisation from the parameters alone.

Tejas Pradeep Shirodkar, P. J. Narayanan

Batch Normalization Transformer Architectures Singular Value Decomposition Fisher Information Covariance Estimation

Ask PDF

Aug 13, 2026cs.AI

Rethinking Normalization Placement for LLMs: Post-Norm under Curriculum Depth Growing

Pre-norm is the standard normalization placement in modern Transformers because it facilitates joint optimization of full-depth models. We ask whether this preference persists when depth is introduced through a curriculum. In curriculum depth growth, each appended block receives the boundary representation produced by a trained prefix, making normalization placement relevant to forward conditioning. We therefore test whether placement and training curriculum interact. In a controlled distillation study with a Qwen3-8B teacher and a nine-layer student, pre-norm and post-norm are indistinguishable under joint training, differing by

0.0004

validation CE, while post-norm improves over pre-norm by

0.0328

under curriculum growth, an order of magnitude larger. A post-joint control matched by student active-layer tokens remains worse than post-grow, which rules out compute as the sole explanation. The ranking crosses over during the curriculum: post-norm takes the lead once blocks are appended. Single-block and freeze controls localize the ranking change to block appending rather than shallow-block quality or retraining. Boundary diagnostics associate post-norm with stable residual scales and pre-norm with structural-token scale drift; on a fixed batch, the final pre-grow block is also nearly identity-mapped. Together with the phase-wise crossover, these observations are consistent with boundary-scale conditioning after new blocks are appended. The results motivate treating normalization placement and training curriculum as coupled design choices in this distillation setting.

Sheng Ren, Yadong Wang, Naiqiang Tan +7

Batch Normalization Normalization Norm Curriculum

Ask PDF