The Spectral Lifecycle of Transformer Training: Transient Compression Waves, Persistent Spectral Gradients, and the Q/K--V Asymmetry

Yi Liu

cs.LGApr 3, 2026

The Spectral Lifecycle of Transformer Training: Transient Compression Waves, Persistent Spectral Gradients, and the Q/K--V Asymmetry

Authors: Yi Liu

Abstract

We present the first systematic study of weight matrix singular value spectra during transformer pretraining, tracking full SVD decompositions of every weight matrix at 25-step intervals across three model scales (30M--285M parameters). We discover three phenomena: (1)~Transient Compression Waves: stable rank compression propagates as a traveling wave from early to late layers, creating a dramatic gradient that peaks early then reverses -- late layers eventually over-compress past early layers. (2)~Persistent Spectral Gradients: the power-law exponent~ $α$ develops a permanent depth gradient forming a non-monotonic inverted-U in deeper models, with peaks shifting toward earlier layers as depth increases. (3)~Q/K--V Functional Asymmetry: value/output projections compress uniformly while query/key projections carry the full depth-dependent dynamics. The dissociation between transient compression and persistent spectral shape reveals that rank and spectral shape encode fundamentally different information about training. We formalize this as a two-timescale dynamical model and derive scaling laws ( $Δα\propto L^{0.26}$ , $R^2{=}0.99$ ). We validate on nine models across three families (custom, GPT-2, Pythia; 30M--1B parameters; 8--36 layers), demonstrate that $α$ predicts layer importance ( $ρ{=}0.69$ -- $0.84$ , $p{<}0.02$ ), and show that spectral-guided pruning outperforms Last-N heuristics by $1.1{\times}$ -- $3.6{\times}$ across seven models in two families (GPT-2 124M--774M, Pythia 160M--1B), with worst-vs-best gaps up to $23.7{\times}$ confirming the causal role of spectral structure.

Figures & tables

Model	$d_{\text{model}}$	Layers	Heads	Params	SVD Snapshots
D8	512	8	8	30.4M	20,050
D12	768	12	12	92.8M	29,674
D16	1024	16	16	285.2M	35,142

Table 1: Model configurations.

Panel of Figure 1 — Figure 1: Transient compression waves. (a) Spatiotemporal heatmaps show the diagonal wave front. (b) The compression gradient transitions from negative to positive in all models, with D16 showing the largest magnitude swings.

	Step 250	Step 500	Step 1K	Step 2K	Step 5K	Final
D8	$-$ 23.1	$-$ 15.0	+2.9	+14.4	+19.7	+22.1
D12	$-$ 42.2	$-$ 25.9	$-$ 2.6	+8.1	+17.0	+17.2
D16	$-$ 41.7	$-$ 59.6	$-$ 19.5	$-$ 2.6	+17.8	+18.8

Table 2: Compression gradient evolution (L

{}_{\text{first}}-

{}_{\text{last}}

average SR). All models show a negative

\to

positive transition.

Panel of Figure 2 — Figure 2: Persistent spectral gradients. (a) D16 Q-projection $\alpha$ across layers showing the inverted-U. (b) Cross-model comparison on normalized depth: the peak shifts earlier in deeper models.

Panel of Figure 3 — Figure 3: The two-timescale dissociation. (a) Layers reach similar compression but divergent $\alpha$ . (b) SR gradient reverses while $\alpha$ gradient monotonically strengthens—the hallmark of two-timescale dynamics.

	D8	D12	D16
$\alpha_{\max}$	0.461	0.516	0.567
$\Delta\alpha$	0.259	0.284	0.310
Peak $l^{*}/L$	0.43	0.36	0.13
$v_{\text{wave}}$	102	131	142

Table 3: Scaling laws.

Figure 4: Cross-family Q-

\alpha

profiles (normalized depth). (a) Custom models: inverted-U with peak shifting left. (b) GPT-2: early-layer concentration after extended training. (c) Pythia: late-layer peaks at intermediate training. All show non-zero

\alpha

gradients.

Model	$k$	Spectral	Last-N	Magnitude	Random	Worst
D12	1	+0.010	+0.010	+0.010	+0.049	+0.052
	2	+0.036	+0.046	+0.046	+0.128	+0.166
	3	+0.102	+0.102	+0.102	+0.221	+0.330
	4	+0.220	+0.220	+0.220	+0.356	+0.460
D16	1	+0.009	+0.009	+0.009	+0.023	+0.082
	2	+0.038	+0.038	+0.038	+0.060	+0.088

Table 4: Custom model pruning:

\Delta

Loss by strategy. Spectral

\,{\approx}\,

Last-N on D16 (coincident ordering).

Model	$k$	Spectral	Last-N	Random	Worst	L-N/S
Medium (24L)	1	+0.84	+1.98	+1.06	+0.73	2.4 $\times$
	2	+2.67	+6.04	+2.36	+2.66	2.3 $\times$
	3	+4.89	+13.49	+3.79	+19.49	2.8 $\times$
	4	+9.65	+25.06	+5.62	+229.15	2.6 $\times$
Large (36L)	1	+0.50	+0.75	+0.44	+0.62	1.5 $\times$
	2	+1.04	+1.92	+1.06	+21.20	1.9 $\times$

Table 5: GPT-2 pruning (

\Delta

PPL): spectral vs. Last-N orderings differ . Spectral-guided achieves 1.5–2.8

\times

less degradation than Last-N across both model scales.

Figure 5: Cross-family pruning synthesis (7 models, 2 families). (a) Last-N/Spectral ratio at maximum

k

: spectral wins on 5/7 models. (b)

\Delta

PPL at

k{=}2

across all models. (c) Topology dependence: the

\alpha

-peak position modulates pruning advantage. Models with mid-network peaks (GPT-2 Medium, Pythia-1B) show the largest spectral advantage.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Optimizer	AdamW
$\beta_{1},\beta_{2}$	0.9, 0.95
Weight decay	0.1
Gradient clipping	1.0
Learning rate	$6\times 10^{-4}$
Schedule	Cosine with 200-step warmup

Appendix

Table 6: Complete training hyperparameters.

	Mean Compression		$\alpha$ Range		$\alpha$ Std
Model	Q	V	Q	V	Q	V
D8	91.2%	95.1%	0.26	0.08	0.095	0.025
D12	89.7%	94.8%	0.28	0.10	0.098	0.031
D16	85.4%	93.2%	0.31	0.09	0.102	0.028

Appendix

Table 7: Q/K–V asymmetry. V/O projections compress uniformly (

>

90%), while Q/K exhibit depth-dependent dynamics.

Figure 6: Q/K vs. V

\alpha

profiles. Q/K projections carry depth-dependent dynamics; V projections are nearly flat.

Panel of Figure 7 — Figure 7: Multi-seed reproducibility. All qualitative phenomena (compression waves, $\alpha$ gradients, inverted-U) are consistent across seeds.

Figure 8: Spectral entropy gradient mirrors

\alpha

: lower entropy (more concentrated spectra) at early-middle layers. This independently confirms the persistent gradient.

Method	Val Loss @5K	Initial SR	Initial $\alpha$	Final SR
Standard	3.720	${\sim}$ 130	${\sim}$ 0.10	${\sim}$ 15
Warmup	5.307	${\sim}$ 15	${\sim}$ 0.45	${\sim}$ 22

Appendix

Table 8: Spectral Warmup vs. Standard Init (D8, 5K steps).

Figure 9: Layer importance via ablation. Color = $\alpha$ . Among core layers, higher $\alpha$ $\to$ higher importance.

Figure 10: Layer importance–

\alpha

scatter plots. Boundary layers are structural outliers; core layers show positive

\alpha

–importance correlation.

	D8		D12		D16
Layer	$\alpha$	$\Delta$ Loss	$\alpha$	$\Delta$ Loss	$\alpha$	$\Delta$ Loss
L0	0.453	+3.42	0.418	+2.82	0.417	+3.56
L1	0.374	+1.72	0.361	+3.08	0.401	+6.91
L2	0.415	+5.10	0.506	+0.163	0.567	+0.083
L3	0.461	+0.096	0.514	+0.058	0.506	+0.024
L4	0.450	+0.091	0.516	+0.054	0.502	+0.022

Appendix

Table 9: Full layer ablation results.

Family	Model	$L$	Params	$\Delta\alpha$	Peak $l^{*}$	$l^{*}/L$	Training
Custom	D8	8	30M	0.259	L3	0.38	10K steps
	D12	12	93M	0.284	L4	0.33	10K steps
	D16	16	285M	0.310	L2	0.13	10K steps
GPT-2	Small	12	124M	0.092	L11	0.92	${\sim}$ 40B tok
	Medium	24	355M	0.285	L0	0.00	${\sim}$ 40B tok
	Large	36	774M	0.107	L1	0.03	${\sim}$ 40B tok

Appendix

Table 10: Complete cross-family comparison: nine models, three families, 30M–1B parameters.

Model	$k$	Spectral	Last-N	Random	Worst	W/S Ratio
Small (12L)	1	+4.48	+9.33	+4.62	+1.37	0.3 $\times$
	2	+38.72	+38.72	+14.43	+10.18	0.3 $\times$
	3	+93.72	+93.72	+34.69	+106.42	1.1 $\times$
	4	+259.53	+259.53	+97.34	+3318.96	12.8 $\times$
Medium (24L)	1	+0.84	+1.98	+1.06	+0.73	0.9 $\times$
	2	+2.67	+6.04	+2.36	+2.66	1.0 $\times$

Appendix

Table 11: Complete GPT-2 pruning results (

\Delta

PPL). Baseline PPL: Small = 24.36, Medium = 18.00, Large = 15.55.

Figure 11: GPT-2 family pruning comparison. Left: Small (12L)—spectral and Last-N overlap at

k{\geq}2

. Center: Medium (24L)—spectral outperforms Last-N by

2.3

–

2.8\times

. Right: Large (36L)—spectral outperforms Last-N by

1.5

–

2.0\times

with worst-vs-best gap up to

20.4\times

Figure 12: GPT-2 Large (36L, 774M) spectral pruning analysis. Left:

\alpha

profile showing low-

\alpha

layers (red, pruned first) concentrated in L25–L30, distinct from the Last-N targets. Center: pruning impact comparison on log scale. Right: discrimination ratios—Last-N/Spectral ratio stays

1.2

–

2.0\times

; Worst/Spectral reaches

20.4\times

k{=}2

Figure 13: Pythia-160M (12L) zone-aware spectral pruning. (a)

\alpha

profile showing rising trend with pruning targets (red, k=4). (b)

\Delta

PPL comparison on log scale. (c) Zone-aware advantage grows with

k

$k$	Zone-Aware	Last-N	Random	Worst	L-N/ZA
1	+35.21	+43.14	+14.50	+14.47	1.2 $\times$
2	+101.03	+111.94	+123.59	+34,875	1.1 $\times$
3	+195.23	+389.77	+591.06	+156,415	2.0 $\times$
4	+501.59	+633.19	+1,224	+5,959,385	1.3 $\times$

Appendix

Table 12: Pythia-160M pruning (

\Delta

PPL). Baseline PPL = 25.49.

Model	$k$	Zone-Aware	Last-N	Random	Worst	L-N/ZA
1B (16L)	1	+5.60	+6.10	+4.87	+40.40	1.1 $\times$
	2	+11.83	+15.23	+11.61	+86.58	1.3 $\times$
	3	+20.90	+53.36	+27.95	+11,703	2.6 $\times$
	4	+55.32	+199.12	+117.92	+15,146	3.6 $\times$
410M (24L)	1	+7.34	+4.40	+4.43	+4.40	0.6 $\times$
	2	+23.77	+10.12	+11.11	+7.00	0.4 $\times$

Appendix

Table 13: Pythia pruning (

\Delta

PPL). Zone-aware spectral outperforms Last-N on Pythia-1B (16L) and Pythia-160M (12L), and at aggressive pruning on Pythia-410M (24L). Baseline PPL: 1B = 12.41, 410M = 14.97.

Figure 14: Pythia-1B (16L, 1B) zone-aware spectral pruning. (a)

\alpha

profile with boundary (orange) and interior layers; red bars indicate pruning targets (k=4). (b)

\Delta

PPL comparison on log scale. (c) Last-N/Zone-Aware ratio: advantage grows from

1.1\times

3.6\times

with increasing

k

Figure 15: Pythia-410M (24L) pruning with crossover. (a) Monotonically rising

\alpha

profile. (b) Zone-aware vs. Last-N on log scale. (c) Ratio showing crossover at

k{=}6

: zone-aware dominates at aggressive pruning.

Figure 16: D8 per-layer Q-projection SR trajectories. Each color is one layer.

Figure 17: D8 per-layer Q-projection

\alpha

trajectories. Layers diverge early and never reconverge.

Figure 18:

\alpha

–loss correlation across models.

Figure 19: D16 three-panel dynamics.

Figure 21: D12

\alpha

profile (inverted-U).

Figure 22: Scaling laws: (a)

\Delta\alpha

vs. depth, (b)

\alpha

extremes, (c) peak position, (d) wave velocity.

Figure 23: Pruning strategy comparison (line plots).

Figure 24: Pruning strategy comparison (bar charts).

Transformer Training Dynamics

See all 24 papers

Spectral Edge Dynamics of Training Trajectories: Signal--Noise Geometry Across Scales
Mar 14, 2026Yongzhong XuTransformer Training Dynamics Neural Network Training Dynamics
Structural Sensitivity in Compressed Transformers: Relative Error Propagation and Layer Removal
Mar 22, 2026Abhinaba Basu, Kumkum Basu, Koushik DebTransformer Pruning LLM Compression
Indian Institute of Information Technology Allahabad, India · National Institute of Electronics and Information Technology (NIELIT), India · Indian Institute of Technology Patna, India +2
Spectral Outliers Reveal Dominant Learned Structure in Transformer Attention
Aug 8, 2026Kasun Dewage, Marianna Pensky, Suranadi De Silva +1Transformer Interpretability Transformer Attention
University of Central Florida Orlando, Florida, USA · Virtusa

Mar 14, 2026J/K move · Enter open · S save

Spectral Edge Dynamics of Training Trajectories: Signal--Noise Geometry Across Scales

Yongzhong Xu

Transformer Training Dynamics Neural Network Training Dynamics Gradient Descent Dynamics Spectral Methods

Open paper AskPDF arXiv

Despite hundreds of millions of parameters, transformer training trajectories evolve within only a few coherent directions. We introduce Spectral Edge Dynamics (SED) to quantify this structure: a rolling-window SVD of parameter updates reveals a sharp boundary -- the spectral edge -- between coherent optimization directions and stochastic noise, identified via the maximum consecutive singular value ratio

σ_k / σ_{k+1}

. Across a 51M-parameter TinyStories model (4 seeds) and GPT-2 124M under distribution shift, the spectral edge exhibits a universal three-phase pattern (rise, plateau, collapse). The effective signal rank adapts to task complexity (

k^* = 2

at 51M,

k^* = 3

at 124M), and the directional coupling between spectral geometry and validation loss reverses with window size -- a lag flip reflecting the timescale of trajectory integration. Johnson--Lindenstrauss projection to

d = 10W

dimensions (e.g.,

d = 100

for

W = 10

) preserves the spectral gap within

5.7\%

, making the framework applicable to models of arbitrary scale. In companion work, the same spectral geometry provides early-warning signals of grokking -- predicting generalization

600

1{,}700

steps before it occurs across modular arithmetic, Dyck languages, and the SCAN benchmark.

Abstract

Transformer Training Dynamics

Explore similar work

Spectral Edge Dynamics of Training Trajectories: Signal--Noise Geometry Across Scales

Structural Sensitivity in Compressed Transformers: Relative Error Propagation and Layer Removal

Spectral Outliers Reveal Dominant Learned Structure in Transformer Attention