A Mesoscopic View of Transformer Weights Through Row and Column Scale Fields

Tiexin Ding

cs.LGSep 25, 2026

A Mesoscopic View of Transformer Weights Through Row and Column Scale Fields

Authors: Tiexin Ding

Abstract

Pooled statistics of Transformer weights obscure how magnitude is distributed across functional channels, while individual weights are too numerous to compare directly. We study the mesoscopic level between them: row and column scale fields, the median-centred log-RMS profiles of a weight matrix over its channels, which together with a global scale and a full balanced core represent the matrix exactly. Across public Pythia checkpoints at four sizes and controlled runs from three initialization families, balancing reveals similar measured core magnitude profiles. A mixture bridge, with its form fixed before the analysis and its coefficients fitted, predicts the pooled-shape departure from field width on held-out runs and data arms of the controlled grid. The indexed fields retain further structure: they align across projections that share a functional channel, and query/key profiles follow reassigned RoPE frequencies rather than fixed matrix coordinates. Training trajectories show early field formation followed by component-dependent broadening or recession. Extending the channel-based analysis to AdamW's second moment reveals related functional organization in its log-space row and column factors. Finally, edits of a frozen checkpoint separate reciprocal scale balance, which preserves the forward computation, from relative channel gain: flattening the gain increases in-distribution loss while preserving matrix norms and the balanced core. Row and column scale fields thus connect pooled magnitude statistics to channel organization and provide coordinates for tracking and testing trained weight structure.

Figures & tables

Figure 1: A mesoscopic view of Transformer weights through row and column scale fields. The training loop produces a sequence of projection matrices. At selected checkpoints, each matrix is represented by its global RMS, balancing factors and full balanced core. Direct row and column log-RMS fields are linked to these factors by the conversions in Section 3.1 . Field widths, core magnitude profiles and fitted pooled parameters are statistical read-outs of this representation; the fitted scale $\lambda$ tracks the global RMS $s$ through the measured ratio $\lambda/s$ . Coloured strips mark the empirically dominant field sides; an unmarked side need not be homogeneous. The dashed analysis path is not part of the training update.

Figure 2: Two-sided balancing reveals a common core across components, model sizes and initialization families. (a) Final public Pythia checkpoints; grey connectors pair the raw and identity-axis-normalized fits of the same component kind (filled and open markers), stars mark two-sided balancing of $v$ , and coloured bars span the normalized block medians. (b) Each quantile of the unit-RMS core $|Z|$ divided by the corresponding quantile of the unit-RMS half-normal (the magnitude of a Gaussian weight), minus one, for one initialization family at one checkpoint. At 30,000 steps each family lies within 0.26% of it throughout the displayed range (median pooled over the three families: 0.19%). (c) Two-sided core shape in nine LLaMA-style 70M runs. The horizontal line in (a) and (c) is the reference shape of a Gaussian weight under the middle-80% protocol, $k\approx 1.20$ , derived for a half-normal magnitude in Ding [2026a] .

Figure 3: Scale-field mixing predicts the pooled-shape departure on the controlled grid. Panels (a), (b) and (d) use the 1,440 row-identity fits (4 data arms $\times$ 3 seeds $\times$ 4 checkpoints $\times$ 5 kinds $\times$ 6 layers) and panel (c) the two-axis fits of all seven kinds on the same runs; colours and markers identify component kinds throughout. Panel (c) uses one pooled two-axis fit over the seven kinds. Panel (d) scores the one-axis prediction of $k_{\mathrm{raw}}$ with per-kind coefficients: bars give the in-sample error and the medians over held-out runs and held-out arms of the fold-mean relative error, and marks show the worst fold; the field width and normalized-core shape of each held-out matrix are measured inputs. Absolute leave-one-run-out errors of the one- and two-axis models are in Figure 9 (c).

projection	functional identity	matrix side	normalization removing most of the departure
$q$ , $k$	logit coordinate, RoPE frequency	rows	row
$v$	head/value output; input-side channel profile	rows and columns	two-sided
$o$	head/value input	columns	column
gate, up	FFN hidden output	rows	row
down	FFN hidden input	columns	column
$q$ , $k$ , $v$	residual-stream input channel	columns	none (column fields widest in $v$ ; column normalization leaves $k_{\mathrm{raw}}$ nearly unchanged)

Table 1: Where the computation graph places each functional identity (matrix side, a structural fact), and which one-sided normalization removes most of the pooled-shape departure on the controlled grid (an empirical result; values in Table 4 ).

Figure 4: Shared functional channels carry matched scale-field profiles across projections. Correlations are computed at step 5,000 over the 12 controlled LLaMA-style 70M runs, per layer; pale markers are single run–layer observations, lines connect per-layer medians over runs, and the overall median over the 72 layer

\times

run cells is given in each legend. The next-layer mismatch pairs the same projections across adjacent layers. Up against down matches as closely as gate against down (median

r

0.93). In (c), direct

q/k

row matching is weaker and layer dependent, whereas aggregating rows by RoPE frequency pair (32 pairs per head, median over heads) gives a consistently matched profile. Long-run and data-arm contrasts are reported in Appendix B.3 .

Figure 5: RoPE frequency indexes the

q/k

row-scale profiles. Profiles are cumulative median-centred changes in log row RMS. (a) Base and frequency-permuted LLaMA-style 70M runs of the representative seed pair, aggregated over heads and layers. (b) Frequency-indexing advantage

\Delta r_{\mathrm{id}}=r_{\mathrm{freq}}-r_{\mathrm{coord}}

, where

r_{\mathrm{coord}}

compares the paired profiles at the same matrix coordinates and

r_{\mathrm{freq}}

at the same assigned frequencies; one point per layer and seed pair, with the

v

control and the per-layer one-sided 95th-percentile permutation null. (c) A separately trained Pythia-protocol size series (70M, 160M, 410M; one data arm, 8,000 steps; rotary on 16 of 64 head dimensions, 8 pairs per head); each profile is divided by its maximum absolute value, comparing shape and peak location rather than amplitude.

Figure 6: Scale fields form early in every projection kind, but their later retention differs. Each kind is read on its identity side, and $k_{\mathrm{ident}}$ is the fit after normalizing that side only; the $v$ curve therefore reports its row-side contribution, its two-sided restoration being given in Section 4.1 . Each point is the median over 18 matrices (six layers $\times$ three seeds of the Gaussian family) at seven sampled steps; a matrix enters only if both fits pass $R^{2}\geq 0.99$ . (a) Identity-axis field width; the common initial value is the finite-sample floor when each marginal RMS is estimated from 512 entries, and triangles mark each kind’s largest sampled width. (b) Shape difference $k_{\mathrm{raw}}-k_{\mathrm{ident}}$ . (c) Mixture term $k_{\mathrm{raw}}^{-2}-k_{\mathrm{ident}}^{-2}$ against $(H^{W})^{2}$ ; in these median trajectories the receding kinds return along nearly the same path.

Figure 7: The row and column factors of AdamW’s second moment align with the decoder’s functional channel spaces. Spearman correlation $\rho$ among the 14 row and column factors of $\log\hat{V}$ for the seven projection kinds at steps 200, 800, 3,200 and 10,000 of one LLaMA-style 70M run (single seed; the step-400 dump enters the 210 fits but is not drawn); each factor is the centred mean of $\log\hat{V}$ along its axis, and each cell the median over six layers. Grey cells pair factors of different length and are undefined; negative values, none below $-0.10$ , are drawn as 0. At step 10,000 the median correlation is $+0.81$ over the 10 within-space pairs and $+0.01$ over the 48 comparable across-space pairs; the permutation null (0.07) is the median over cells of the 95th percentile of $|\rho|$ under 200 within-layer permutations.

Figure 8: Frozen-checkpoint edits separate an invariant balance mode from a loss-sensitive gain mode. Edits act on one of the $q/k$ , $v/o$ and up/down paths across all six layers of the 30,000-step base checkpoint. Filled and open markers show two disjoint evaluation slices from the training token stream (slice 1 and slice 2; values in Table 5 ). (a) Balance removal and gain flattening. (b) Loss changes relative to the flatten, divided by the corresponding ratios of squared relative Frobenius displacement $\Delta_{F}$ . (c) Flattening one gain decile alone, relative to the full flatten; responses below $10^{-3}$ of it are drawn at the floor, and the seven non-positive ones as open downward triangles. Edit formulas and displacement ratios are in Appendix B.6 .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

symbol	meaning	defined in
Matrix and decomposition
$W\in\mathbb{R}^{d_{\mathrm{out}}\times d_{\mathrm{in}}}$	one projection matrix; rows index output channels, columns input channels	Section 3.1
$W=s\,D_{r}\,Z\,D_{c}$	exact re-parameterization: global scale, row and column balancing factors, core	eq. ( 2 )
$s=\mathrm{RMS}(W)$	global scale, fitting-free	eq. ( 2 )
$D_{r},\,D_{c}$ ; $(D_{r})_{ii},\,(D_{c})_{jj}$	positive diagonal balancing factors; convertible to and from the scale fields given $s$ and $Z$	eqs. ( 2 ), ( 13 )
$Z$	two-sided RMS-balanced core, unit RMS in every row and column	eq. ( 3 )

Appendix

Table 2: Notation. Component kinds are written

q

k

v

o

, gate, up and down; the key projection

k

always appears alongside

q

, and the Weibull shape is always qualified as “shape

k

”.

source	role	model	runs	steps / ckpts	protocol-specific details
Controlled data grid	one-axis bridge; short trajectories	LLaMA 70M	4 arms $\times$ 3 seeds	5,000 / 4	shuffling $f\in\{0,1\}$ $\times$ repetition $n_{\mathrm{rep}}\in\{1,64\}$ : D1 $(f{=}0,n_{\mathrm{rep}}{=}1)$ , D2 $(1,1)$ , D3 $(1,64)$ , D4 $(0,64)$ ; sequential single pass; stopped at 5,000 steps of the 30,000-step schedule.
Paired RoPE reassignment	frequency-to-row identity (paired intervention)	LLaMA 70M	3 seed pairs	30,000 / 9; 3 shared (pairs 2–3)	base run against a run whose RoPE inverse-frequency table is permuted by a fixed $\pi$ at initialization; frequency set unchanged; base arms in the text.
Initialization families	core convergence; long-term field evolution	LLaMA 70M	3 families $\times$ 3 seeds	30,000 / 7 (0, 800, 3,200, 10k, 20k, 25k, 30k)	Gaussian, Laplace, uniform at equal initial RMS; windows sampled with replacement from the untransformed $10^{8}$ -token stream.
Optimizer-state records	row–column structure of $\hat{V}$ (descriptive)	LLaMA 70M	2	10,000 / 5 dumps; 30,000 / 150 logged windows	a dump run (batch 24) and a separate logger run (batch 16); see text.
No-position control	RoPE-versus-head organization (boundary)	LLaMA 70M	1	30,000 / 9 (6 shared with the base run)	base run with the rotary rotation removed ( $\cos=1$ , $\sin=0$ ); single-seed boundary.
Checkpoint field edits	function carried by the paired gain (screen)	LLaMA 70M	base run at 30k	no training	edits of the stored weights scored by mean loss on two disjoint slices of 48 sequences $\times$ 512 tokens from the training stream; 63 edits and 54 displacement-controlled edits (Appendix B.6 ).

Appendix

Table 3: Evidence sources, their roles and protocol-specific details; all LLaMA-style runs share the architecture, optimizer and tokenizer stated above, while sampling, schedule length and batch size differ by row. Rows denote evidence streams rather than independent sample counts: the Gaussian initialization runs are the base arms for RoPE seed pairs 2 and 3. Fit counts and the mapping to figures are in the data documentation.

kind	$k_{\mathrm{raw}}$	$k_{\mathrm{row}}$	$k_{\mathrm{col}}$	$k_{\mathrm{bi}}$	$H^{W}_{\mathrm{row}}$ / $H^{W}_{\mathrm{col}}$	$R_{0}^{2}$ row/col/both	identity side (Table 1 )
$q$	1.192	1.208	1.195	1.211	0.16 / 0.06	0.978 / 0.623 / 0.978	rows
$k$	1.188	1.207	1.192	1.210	0.17 / 0.08	0.977 / 0.657 / 0.977	rows
$v$	1.194	1.200	1.197	1.205	0.13 / 0.11	0.932 / 0.942 / 0.964	both
$o$	1.191	1.195	1.199	1.206	0.12 / 0.16	0.799 / 0.925 / 0.927	columns
gate	1.187	1.203	1.188	1.205	0.19 / 0.07	0.906 / 0.627 / 0.915	rows
up	1.191	1.203	1.192	1.206	0.18 / 0.08	0.890 / 0.726 / 0.893	rows

Appendix

Table 4: Medians over the 288 fits per kind on the controlled grid (12 runs

\times

4 checkpoints

\times

6 layers): the raw fit, the fit after row, column and two-sided normalization, the two heterogeneities, and

R_{0}^{2}

of the through-origin bridge

k_{\mathrm{raw}}^{-2}-k_{\mathrm{bi}}^{-2}

against

(H^{W}_{\mathrm{row}})^{2}

(H^{W}_{\mathrm{col}})^{2}

, and both.

Figure 9: Bridge coefficients per kind, on one axis and on two. (a) The one-axis protocol coefficient $\alpha_{P}$ on the 1,440 grid fits; all 12 runs share the sign in every kind. (b) Two-axis coefficients fitted per kind through the origin on the 2,016 two-axis fits. On the axis that carries each kind’s identity the coefficient is 0.75–0.87 ( $q$ , $k$ , $o$ , gate, up, down); $v$ , whose two fields are of similar size, has $\alpha_{r}=0.42$ and $\alpha_{c}=0.88$ . The minor-axis coefficient is poorly identified by these data: the minor-axis width has per-kind medians of 0.06–0.12, its square is 2–7 times below the major term, and the fitted minor coefficients include negative values (gate $\alpha_{c}=-1.14$ , down $\alpha_{r}=-1.13$ ); ranges beyond $\pm 2.5$ are truncated. (c) Each model is fitted per kind. The column term is what brings $v$ , $o$ and down into the same relation; it changes $q/k$ little and slightly worsens gate/up, and the cross term changes the error by at most 0.001 in every kind. The worst fold is a high-repetition arm in every kind and model, the shuffled one (D3) in all but two cases.

Figure 10: Scale-field distributions retain information beyond field width. Controlled data grid (Table 3 ), 504 matrices at step 5,000, identity-axis field of each matrix. The plotted quantity is $\tilde{h}$ , the field $h$ of ( 4 ) re-centred by its mean rather than by its median; the shift changes neither width, skewness nor kurtosis, and $\tilde{h}=0$ is the geometric-mean channel RMS. (a) Positive skewness is a tail of amplified channels, negative a tail of suppressed channels. (b) Channel histograms of five example matrices, chosen to show the range of shapes and not matched in width.

Figure 11: The RoPE permutation changes field arrangement while the pooled shape remains stable. Paired runs with base and reassigned RoPE frequencies, three seeds, same data and initialization. (a) Similarity between the permuted and the base row profile at each paired seed, pooled over heads and layers on the checkpoints shared by each pair (steps 3,200, 10,000 and 30,000); the pooled

\Delta r_{\mathrm{id}}

is the difference of the two bars. (b) Change under the permutation at seed 1 (the width change differs across seeds, see text), each bar a pooled median over 18 matrices per kind (6 layers

\times

steps 20,000, 25,000 and 30,000);

o

is not part of the row-side width comparison because its identity axis is the column axis.

Figure 12: A $q/k$ row field forms without RoPE, but its rotary-pair organization and the $q$ – $k$ pair coupling weaken. One run trained without positional rotation (rotary $\cos=1$ , $\sin=0$ ) against the base run: single seed, same data and initialization, terminal loss 3.174 against 2.977. In (d), $g=h_{1}+h_{2}$ and $b=h_{1}-h_{2}$ are the sum and difference of the two paired profiles on each shared path (Section 4.6 ), used here only as a pairing read-out; its grey marks are the upper 97.5% quantile of 100 within-layer re-pairings.

Figure 13: Core convergence and scale-field formation unfold together across initializations. Nine LLaMA-style 70M runs on one data setting (Gaussian, Laplace and uniform initialization $\times$ three seeds), in three columns of kinds ( $q/k$ , $v/o$ , gate/up/down). Top row: pooled shape $k_{\mathrm{raw}}$ ; middle row: identity-axis field width ( $H^{W}_{\mathrm{row}}$ for $q$ , $k$ , gate, up; $H^{W}_{\mathrm{col}}$ for $o$ , down; both for $v$ ); bottom row: two-sided core shape $k_{\mathrm{bi}}$ . Lines are family medians over 18 matrices (three seeds $\times$ six layers).

Figure 14: Matched scale fields accumulate predominantly in a shared gain mode. Gain $g$ and balance $b$ of a paired channel are defined in Section 4.6 ; their variance ratio exceeds 1 when the two profiles move together. Re-paired profiles give about 1 (0.5–2.0 on $q/k$ ); the adjacent-layer $q/k$ pairing stays near 4.6. Both run sources are defined in Table 3 . (a) The re-pairing null (shaded bars) and the next-layer pairings (crosses) are computed at step 5,000 of the data grid only and are drawn there, one per path. (b) Head order is a permutation-symmetric label, so $v/o$ has no reproducible index; up/down is shown as a sorted comparison only.

	$q/k$			$v/o$			up/down
edit	size	slice 1	slice 2	size	slice 1	slice 2	size	slice 1	slice 2
flatten (reference)	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
log-gain doubled ( $\tau=-1$ )	1.23	3.64	3.48	1.20	1.48	1.16	0.99	1.59	0.64
log-gain $\times 1.5$ ( $\tau=-0.5$ )	0.29	1.30	1.31	0.30	0.38	0.28	0.25	0.45	0.10
shuffle, size-matched (5 draws)	0.98	0.64	0.66	1.00	0.87	0.85	1.00	0.65	0.69
Gaussian direction (5 draws)	0.94	0.29	0.29	0.87	0.62	0.54	1.01	0.50	0.33

Appendix

Table 5: Control-edit displacements and loss responses. Unless marked as an absolute quantity, the slice columns report the change in loss of the edit relative to flattening the gain (

\tau=1

) on the same path and slice, averaged over the five random draws where marked; size is the squared displacement ratio

\Delta_{F}/\Delta_{F}^{\mathrm{flat}}

of Appendix B.6 . The shuffle at the histogram’s width is the unscaled permutation of the edit table in Appendix B.6 . The lower block reads the 63-condition set of the same two slices: the flatten cost in nats (base loss 3.347 on slice 1, 3.093 on slice 2), and, relative to it, the reversal

\tau=2

, the two extreme deciles of

g

edited alone, the sum of all ten decile edits, and the flatten cost divided by the squared gain width (0.275, 0.209, 0.068 on the three paths).

Transformer

Transformers are neural architectures that use attention to model relationships among sequence elements in language, vision, and scientific data.

See all 540 papers

Sep 30, 2026

The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidation

Massive activations, residual-stream coordinates with magnitudes far larger than typical activations, are associated with attention sinks in transformers, but how their scale is regulated during training remains incompletely understood. Combining training-trajectory analyses and controlled interventions, we trace their emergence, growth, and consolidation. Sink-carrying channels vary across random seeds but stabilize early within each run. Over longer training, surrounding channels erode and the sink concentrates onto a few redundant carriers. Across ablations, gradient attenuation follows the sink token's collective root-mean-square magnitude rather than any single channel, making collective scale central to understanding their effects. Our central result is that weight decay causally controls the turnover of global activation scale. In controlled continuations, removing decay near the peak allows this scale to keep rising, whereas retaining it produces decline even at constant learning rate. We develop a balance model for the rise and peak of massive-activation magnitude, in which AdamW-preconditioned growth opposes weight decay. Sweeping the decay coefficient

λ

shifts peak timing approximately log-linearly and yields peak magnitudes scaling approximately as

λ^{-1/2}

, consistent with this balance. Optimizer measurements further show that preconditioning sustains the large-channel cohort against decay even when raw maintaining forces are too small to do so. Together, these findings connect the observed life cycle to scale-regulating training dynamics and establish weight decay as a training-time lever on activation magnitude.

S. Aaron McClendon, Jorge Gallego-Feliciano, Antonios Saravanos

Aimpoint Digital Labs, Atlanta, GA, USA · New York University, New York, NY, USA

Weight Decay Attention Sinks Massive Activations Neural Network Training Dynamics

Ask PDF

Jun 11, 2026

Weibull Weight-Scale Parameter Evolution under AdamW Training Dynamics

Building on a two-parameter Weibull framework for diagnosing transformer weight distributions, we study why the Weibull weight-scale parameter

λ

grows, overshoots, and then relaxes during AdamW training. We derive a leading-order three-force decomposition of the squared weight norm from the AdamW update: an alignment force measuring the correlation between weights and the adaptive update direction, an injection force from adaptive step magnitude, and a decay force from decoupled weight decay. On self-trained Pythia-70M models with ground-truth optimizer moments, alignment dominates the rise phase, contributing 88-94% of the absolute force budget across four random seeds and remaining robust to super-weight removal. Near saturation, alignment and decay approach balance, explaining the transition from weight-scale growth to relaxation. These force dynamics directly govern the squared-norm component underlying

λ(t)

; the remaining RMS-to-Weibull reconstruction offset is measurable and decomposes into bridge and integration components, totaling approximately 5-6% in densely sampled regions. To extend the analysis to real models where optimizer moments are unavailable, we introduce a spline displacement method that recovers the alignment force from sparse checkpoints with approximately 92-94% accuracy, about twice the naive two-point baseline. We further observe that the peak value of

λ(t)

varies with training-data coherence in our experiments, suggesting a data-dependent component of weight-scale growth that we leave to a controlled follow-up study. Code and data are available at https://github.com/tiexinding/NPM-Weibull-public.

Tiexin Ding

Independent Researcher.

Weight-Space Analysis Adaptive Gradient Methods Neural Network Training Dynamics

Ask PDF

Oct 13, 2026

Same Function, Different Trajectory: No Standard Optimizer Is Equivariant to Transformer Reparameterization

We show that two Transformers representing the same function can follow different training trajectories. For example, doubling a query matrix and halving its key matrix preserves the model's function but changes the gradient scales seen by the optimizer. Our theoretical analysis establishes that standard block-local first-order optimizers fail to scale updates with parameters under a shared learning rate. For AdamW, transforming the first- and second-moment estimates alone does not correct the mismatch between parameter and update scales. To make runs restarted from equivalent checkpoints follow the same trajectory, we balance matrix pairs and transform their moment estimates together, bringing distinct but equivalent training states to the same starting point. To correct scale imbalance between paired Transformer matrices without changing predictions, we propose Symmetry-Aware Gradient Geometry Stabilization (SAGGS), which periodically equalizes the Frobenius norms of exact symmetry pairs, and its damped variant, Soft SAGGS. SAGGS reduces perplexity (PPL) by

0.374\pm0.026

at 101M parameters on WikiText-103, and Soft SAGGS reduces PPL by

0.780\pm0.040

in a 31M modern-architecture replication, both under scale imbalance. In a 123.8M checkpoint-resume experiment, reciprocally rescaling matrix pairs barely changes validation loss at restart but increases final PPL by

3.78\pm0.42

; hard and soft rebalancing recover to within

0.02

PPL of the control.

Bum Jun Kim

The University of Tokyo

Deep Learning Optimization Transformer Transformer Training Dynamics Transformer Attention

Ask PDF

Abstract

Transformer

Explore similar work

The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidation

Weibull Weight-Scale Parameter Evolution under AdamW Training Dynamics

Same Function, Different Trajectory: No Standard Optimizer Is Equivariant to Transformer Reparameterization