NerVE: Nonlinear Eigenspectrum Dynamics in LLM Feed-Forward Networks

Nandan Kumar Jha; Brandon Reagen

cs.LGMar 6, 2026

NerVE: Nonlinear Eigenspectrum Dynamics in LLM Feed-Forward Networks

Authors: Nandan Kumar Jha, Brandon Reagen

Organizations: New York University

Abstract

We introduce NerVE, a unified eigenspectral framework for understanding how feed-forward networks (FFNs) in large language models (LLMs) organize and regulate information flow in high-dimensional latent space. Despite FFNs dominating the parameter budget, their high-dimensional dynamics remain poorly understood. NerVE addresses this gap through lightweight, memory-efficient tracking of eigenspectrum dynamics via four complementary metrics: Spectral Entropy (dispersion), Participation Ratio (effective dimensionality), Eigenvalue Early Enrichment (top-heaviness), and Jensen-Shannon divergence (distributional shifts). Our key insight is that FFN nonlinearities reinject variance across eigenmodes, fundamentally governing latent dimension utilization, and that optimizer geometry strongly modulates the extent of this variance reinjection. We validate NerVE across model scales, and diverse architectural and optimizer configurations, each uniquely shaping FFN dynamics: normalization schemes controlling variance flow; FFN weight geometries constraining latent space; positional encoding and activation functions regulating information flow; and optimizer choices redistributing effective capacity across depth. Across these settings, NerVE consistently recovers stable spectral signatures that correlate with model's generalization ability and respond predictably to design choices, generalizing beyond transformer to MLP-Mixer architectures, providing actionable insights for architectural and optimizer choices beyond trial-and-error.

Figures & tables

Figure 1: NerVE quantifies nonlinear eigenspectrum dynamics in FFNs of GPT-2. FFN nonlinearity (GELU) regulates information flow by reinjecting variance, reactivating under-utilized directions (post-activation SE

\uparrow

and PR

\uparrow

), and flattening the eigenspectrum, less top-heavy (post-activation EEE

\downarrow

). The JS heatmap shows a depth-localized transition band where redistribution is strongest.

Experimental Axis	Key Findings
Activation (GELU vs ReLU)	Similar trend, distinct dynamics; GELU explores broader subspace (§ 3.1 )
Norm-free models	GELU exhibits spectral inertia; ReLU compensates LayerNorms (§ 3.2 )
FFN weight geometry	Spectral dynamics matter; performance tracks sustained flattening (§ 3.3 )
Norm placement (Pre/Post/Mix)	PreLN: best return-on-width; PostLN: diminishing spectral returns (§ 3.4 )
Positional encoding	RoPE prevents mid-to-deep spectral collapse (depth utilization $\uparrow$ ) (§ 3.6 )
Optimizer (AdamW vs Muon)	Repair or refinement; performance follows mid-layer capacity trends (§ 3.7 )

Table 1: Summary of key NerVE findings per experimental axis.

Figure 2: Cumulative variance distribution across a 768-dimensional latent space. Higher values (shown on curves) indicate top-heavy concentration in a few dominant directions, while lower values reflect a more uniform distribution.

Figure 3: Eigenspectrum dynamics illustrate how FFN nonlinearities regulate information flow and reshape the eigenspectrum during training for GPT-2 (ReLU) on CodeParrot. Pre- and post-activation dynamics are shown for SE, PR, and EEE, highlighting how nonlinearities reinject variance and alter spectral structure. JS heatmaps (rightmost) capture the layer-wise distributional shift induced by nonlinearity. In-panel titles report Pearson correlations (

r

) between each metric and evaluation loss.

Figure 4: Eigenspectrum dynamics for norm-free GPT-2 (125M) models with GELU (top), ReLU (middle), and learnable-slope Leaky ReLU (bottom). Columns show layer-averaged SE (pre vs. post), PR gain (post to pre), post-activation EEE (yellow regions indicate top-heavy distribution), and JS (yellow regions highlight strong redistribution) across layers and training steps. Norm-free GELU exhibits spectral inertia in layers 0 to 5 (EEE $\rightarrow$ 1, JS $\rightarrow$ 0); whereas, ReLU and Leaky ReLU aggressively reinject variance (PR gain $>$ 200 $\times$ ) and flattening the spectrum (EEE $<$ 0.3).

	Baseline Models		Norm-free Models			Norm-free w/ FFN-Norm
	GELU	ReLU	GELU	ReLU	Leaky ReLU	WNorm	SNorm	HNorm
PPL	2.714	2.774	3.223	2.988	3.081	3.041	3.000	3.122

Table 2: Evaluation perplexity (PPL

\downarrow

) comparison across GPT-2 baseline models (GELU and ReLU), norm-free models (GELU, ReLU, learnable-slope Leaky ReLU). Parametric normalization (Weight, Spectral, Hyperspherical) are applied to FFNs of norm-free learnable-slope Leaky ReLU models. All models trained on 2.1B tokens from CodeParrot dataset.

Figure 5: Impact of FFN (parametric) normalization in norm-free GPT-2 with learnable-slope leaky ReLU. Eigenspectrum dynamics are quantified by latent capacity (PR_post), spectral regularization and flattening (

\Delta

EEE and EEE_post), and distributional shift (JS). Top to bottom: Weight, Spectral, and Hyperspherical Normalization. Each method exhibits distinct JS localization and spectral patterns, showing different influences on FFN internal dynamics.

Figure 6: LayerNorm positioning and FFN width sweep: Post-activation participation ratio is normalized by

D

for PreLN, MixLN, and PostLN configurations. PreLN sustains the highest and most stable utilization of FFN width across the sweep, PostLN incurs diminishing return at higher FFN width, MixLN lies in between but with greater layer-to-layer variability.

Metric	FFN Width Configuration (GPT-2 GELU)								GPT-2				NormFree GPT-2
Metric	D=1d	D=2d	D=3d	D=4d	D=5d	D=6d	D=7d	D=8d	GELU	ReLU	GeGLU	SwiGLU	GELU	ReLU	LReLU
SE_pre	-0.98	-0.98	-0.99	-0.99	-0.99	-0.99	-0.99	-0.99	-0.99	-0.98	-0.95	-0.97	-0.82	0.03	0.03
SE_post	-0.84	-0.84	-0.86	-0.87	-0.87	-0.87	-0.87	-0.87	-1.00	-1.00	-0.57	-0.85	-0.92	-0.99	-1.00
PR_pre	-0.97	-0.98	-0.98	-0.99	-0.98	-0.97	-0.98	-0.97	-0.99	-0.98	-0.97	-0.97	-0.93	-0.55	-0.60
PR_post	-0.85	-0.93	-0.94	-0.94	-0.95	-0.95	-0.93	-0.93	-1.00	-0.97	-0.94	-0.89	-0.99	-0.94	-0.99

Table 3: Correlation between eigenspectrum metrics (SE, PR) and generalization. Left (within-run): Pearson

r

between each metric and validation loss over checkpoints at each FFN width (

D

d

–8

d

). Right (cross-config): Pearson

r

between final metric values and perplexity across the eight width configurations for each architecture and activation. Higher SE and PR consistently implies lower loss.

Figure 7: Layerwise participation ratio (PR) comparison (RoPE vs NoPE) in GPT-2 models trained from scratch on 26B token form openwebtext dataset with 512 context length for 100K steps. RoPE sustains higher PR in the middle and deeper layers, indicating better utilization of latent space and network’s depth.

Figure 8: Optimizer-dependent FFN eigenspectrum dynamics in GPT2-350M. Rows show AdamW (top), Muon (middle), and Dion (bottom). AdamW has large early PR gains and high JS with relatively high EEE_post, indicating optimizer-induced pre-activation collapse followed by aggressive but incomplete nonlinear repair. Muon shows the smallest PR gains, lowest JS, and lowest EEE_post, and flatter post-spectra. Dion is intermediate and falls between these two regimes, improving over AdamW but not matching Muon’s pre-/post-spectral behavior. The perplexity ordering (Muon $>$ Dion $>$ AdamW) aligns with post-activation spectral flatness.

Panel of Figure 9 — Figure 9: Layerwise PR_pre (top) over training, and final PR_post per-layer (bottom) for AdamW, Muon, and Dion optimizer. Muon maintains the highest PR_pre across almost all layers, Dion is intermediate, and AdamW shows early-layer collapse. Moreover, Muon concentrates the largest effective dimensionality in middle FFNs.

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: For each FFN layer

\ell

, we first collect pre-activation (after

W_{\text{up}}

, before

\sigma

) and post-activation (after

\sigma

, before

W_{\text{down}}

). Then, tokens are flattened into

X\in\mathbb{R}^{(B\times S)\times D}

sample matrix, where

B

is the global batch size,

S

the sequence (context) length, and

D

the FFN hidden dimension. Finally, we compute the unbiased sample co-variance of mean-centered activations to get the eigenvalues

\{\lambda_{i}\}_{i=1}^{D}

, and sorted them into descending order. Three eigen-metrics are computed on each (pre-/post-) eigenspectrum: spectral entropy (SE) for dispersion, participation ratio (PR) for effective dimensionality, and eigenvalue early enrichment (EEE) for top-heaviness; while Jensen-Shannon Divergence (JS) quantifies the distributional shift between the pre- and post-activation spectra, capturing the geometric restructuring performed by the nonlinearity.

Metric	Input	Range	Sensitive to	Captures
SE	$\hat{\lambda}$ (normalized)	$[0,\ln D]$	Full spectrum (especially mid-to-tail)	Variance uniformity (dispersion)
PR	$\lambda$ (raw)	$[1,D]$	Large eigenvalues (suppresses small eigenvalues)	Effective dimensionality
EEE	$\lambda$ (raw)	$[0,1)$	Top eigenvalues (front-loaded concentration)	Top-heaviness (anisotropy)
JS	$\hat{\lambda}_{\text{pre}},\;\hat{\lambda}_{\text{post}}$	$[0,\ln 2]$	Shape difference between (pre-/post-)spectra	Geometric restructuring

Appendix

Table 4: Summary of the four NerVE eigen-metrics. Input indicates whether the metric operates on raw (

\lambda

) or normalized (

\hat{\lambda}

) eigenvalues, and range shows their (bounded) output values. SE, PR, and EEE characterize a single spectrum; while JS quantifies the divergence between the pre- and post-activation spectral shape. All four metrics are invariant to uniform scaling of eigenvalues.

Joint Signatures	Interpretation	References
SE $\uparrow$ PR $\uparrow$ EEE $\downarrow$	Healthy spectral flattening (Rank inflation)	Standard training (§ 3.1 )
SE $\downarrow$ PR $\downarrow$ EEE $\uparrow$	Spectral collapse	AdamW pre-activation (§ 3.7 )
JS $\downarrow$ across depth	Decreasing nonlinear restructuring	Healthy deep layers (§ 3.1 )
SE $\uparrow$ PR $\uparrow\uparrow$ JS $\downarrow$ across depth	Nonlinearity compensating for pathology	Norm-free ReLU models (§ 3.2 )
EEE $\downarrow\downarrow$ JS $\uparrow$ across depth	Sustained spectral flattening	Spectral norm. in FFNs (§ 3.3 )
Pre-act stable, post-act shifts	Repair regime	AdamW (§ 3.7 & Appendix L.1 )

Appendix

Table 5: Diagnostic reference for joint eigen-metric signatures. Each row describes a commonly observed pattern, its interpretation, and where in the paper it is empirically demonstrated.

JS	PR gain	$\Delta$ EEE	Interpretation
High	High	Strongly negative	Beneficial Restructuring
High	High	Strongly negative	Active expansion and flattening (norm-free ReLU, § 3.2 )
High	High	Weakly negative	Compensatory Repair
High	High	Weakly negative	Effort wasted undoing PreAct spectral collapse (AdamW, § 3.7 )
Moderate	High	Near zero or positive	Expansion without equalization
Moderate	High	Near zero or positive	New directions but top-heaviness persists (HNorm, § 3.3 )

Appendix

Table 6: Four regimes of nonlinear geometric work, distinguished by the joint signature of JS, PR(Post/Pre), and

\Delta

EEE. JS alone cannot distinguish beneficial restructuring from compensatory repair; PR gain alone cannot distinguish beneficial restructuring from expansion without equalization.

Panel of Figure 11 — Figure 11: Eigenspectral impact of LayerNorm placement (PreLN, PostLN, MixLN) in GPT-2 and LLaMA variants (70M, 130M). The spectral signatures are shown through post-activation spectral entropy ( $\uparrow$ ) and participation ratio ( $\uparrow$ ), and model’s perplexity is shown on top of each plots. GPT-2 models trained on CodeParrot and LLaMA variants on C4.

Panel of Figure 12 — Figure 12: Eigenspectral impact of LayerNorm placement (PreLN vs MixLN) in LLaMA-250 and LLaMA-1.3B models, trained from scratch on C4 dataset for 20K iterations with 256 context length. The spectral signatures are shown through SE_post ( $\uparrow$ ) and PR_post ( $\uparrow$ ), and model’s perplexity is shown on top of each plots. MixLN variant of LLaMA-1.3B destabilized after 7K iterations.

Panel of Figure 13 — Figure 13: Spectral Signature of Positional Encoding (RoPE vs NoPE) in GPT-2 models trained with 512 context size on 26B token form openwebtext dataset

Method	D=768	D=1536	D=2048	D=3072	D=3840	D=4608	D=5376	D=6144
PreLN	292 $\pm$ 62	573 $\pm$ 149	831 $\pm$ 238	1002 $\pm$ 354	1275 $\pm$ 333	1562 $\pm$ 429	1721 $\pm$ 244	1822 $\pm$ 370
MixLN	137 $\pm$ 111	184 $\pm$ 126	247 $\pm$ 233	274 $\pm$ 224	278 $\pm$ 234	159 $\pm$ 142	251 $\pm$ 201	233 $\pm$ 187
PostLN	95 $\pm$ 65	142 $\pm$ 104	151 $\pm$ 147	240 $\pm$ 231	117 $\pm$ 110	91 $\pm$ 83	94 $\pm$ 84	71 $\pm$ 62

Appendix

Table 7: PR_post (median

\pm

MAD across 12 layers) at final training checkpoint for different LayerNorm placements and FFN widths in GPT-2 (125M).

Figure 14: Eigen metrics in GPT-2 ( GELU , D=

1d

8d

) with Pearson

r

to eval loss (

r_{\mathrm{pre}},r_{\mathrm{post}}

)

Figure 15: Eigen metrics in GPT-2 ( ReLU , D=

1d

8d

) with Pearson

r

to eval loss (

r_{\mathrm{pre}},r_{\mathrm{post}}

)

Figure 16: SE and PR in Normalization-free GPT-2 ( GELU , D=

1d

8d

) with Pearson correlation to eval loss (

r_{\mathrm{pre}},r_{\mathrm{post}}

)

Figure 17: SE and PR in Normalization-free GPT-2 ( ReLU , D=

1d

8d

) with Pearson correlation to eval loss (

r_{\mathrm{pre}},r_{\mathrm{post}}

)

Metric	Sampling				RandSVD		Lanczos
Metric	5%	10%	25%	50%	256	512	256	512
SE_pre (%)	1.0	0.5	0.5	0.4	50.7	50.1	7.9	1.7
SE_post (%)	4.8	1.9	1.3	1.3	63.7	61.0	28.8	19.3
PR_pre (%)	4.9	4.1	5.1	4.3	90.4	90.2	18.0	3.6
PR_post (%)	16.7	10.2	33.5	30.2	94.9	94.8	68.4	53.9
EEE_pre (%)	0.4	0.1	0.1	0.1	4.2	1.6	38.8	26.3

Appendix

Table 8: Percentage of error when sampling and low-rank approximation methods are used in GPT-2.

Metric	Sampling				RandSVD		Lanczos
Metric	5%	10%	25%	50%	256	512	256	512
SE_pre	-0.97	-0.972	-0.971	-0.972	-0.106	-0.174	-0.822	-0.959
SE_post	-0.34	-0.376	-0.332	-0.363	0.612	0.568	0.335	0.08
PR_pre	-0.914	-0.915	-0.918	-0.912	0.014	0.047	-0.788	-0.901
PR_post	-0.122	-0.138	0.049	-0.235	0.594	0.575	0.314	0.136

Appendix

Table 9: Effect of token-sampling and low-rank approximation on metric–(eval)loss correlation, evaluated on GPT-2 (125M). Token-sampling preserves pre-metric trends but degrades post-metric correlations since tail eigenvalues are under-sampled; low-rank truncation distorts both.

Figure 18: SE and PR in GPT-2 with Pearson

r

to eval loss under sub-sampling (5, 10, 25, 50%)

Figure 19: SE and PR in GPT-2 with Pearson

r

to eval loss under low-rank approximation .

Figure 20: Impact of sampling (10%, 25%, 50%), and low-rank approximation on EEE (GPT-2)

Figure 21: Impact of sampling (10%, 25%, 50%), and low-rank approximation on JS metric (GPT-2)

Metric	Sampling				RandSVD		Lanczos		Full
Metric	5%	10%	25%	50%	256	512	256	512	batch
Wall-clock time	9.89s	9.92s	11.48s	12.62s	10.28s	11.19s	12.07s	12.20s	14.41s
Overhead-200 (%)	4.37	4.39	5.08	5.58	4.55	4.95	5.34	5.40	6.38
Overhead-1K (%)	0.87	0.88	1.02	1.12	0.91	0.99	1.07	1.08	1.28

Appendix

Table 10: Wall-clock time and relative overhead for computing eigenspectrum metrics at various logging frequencies. Overhead is reported as percentage of total training time when eigendecomposition is performed every 200 and 1000 steps on GPT-2 with 3072

\times

3072 FFN covariance matrix size running on AMD EPYC 7502 server with NVIDIA RTX 3090 GPU

Metric	D=1 $d$	D=2 $d$	D=3 $d$	D=4 $d$	D=5 $d$	D=6 $d$	D=7 $d$	D=8 $d$
Matrix dim.	[768,768]	[1536, 1536]	[2304, 2304]	[3072, 3072]	[3840, 3840]	[4608, 4608]	[5376, 5376]	[6144, 6144]
Matrix size	2.25MB	9MB	20.25MB	36MB	56.25MB	81MB	110.25MB	144MB
GPU Mem.	4.5MB	18MB	40.5MB	72MB	112.5MB	162MB	220.5MB	288MB

Appendix

Table 11: GPU memory overhead for full-batch eigen-computation across various FFN widths.

Figure 22: Eigenspectrum dynamics in MLP-Mixer under activation ablations. Rows correspond to the four activation configurations for token-mixing (FFN1) and channel-mixing (FFN2) layers, and columns (from left to right) show spectral entropy (SE), participation ratio (PR), eigenvalue early enrichment (EEE), and Jensen-Shannon divergence (JS) for the channel-mixing FFNs (FFN2). Each panel traces pre- and post-activation metrics over training, showing that ReLU in the channel-mixing MLP (3rd and 4th rows) most strongly increases SE/PR and reduces EEE, i.e., reinjects variance into low-energy directions and flattens the spectrum.

FFN1	FFN2	Acc. (%)
GELU	GELU	66.96
GELU	ReLU	66.99
ReLU	GELU	67.86
ReLU	ReLU	67.20

Appendix

Table 12: Validation accuracy on CIFAR-100 for different activation function configurations in MLP-Mixer. FFN1 refers to token-mixing MLP, FFN2 refers to channel-mixing MLP.

Panel of Figure 23 — Figure 23: Sequence-aware FFN eigenspectrum across token positions. For four GPT-2 (125M) variants (rows 1 to 4) and a non-transformer MLP-Mixer (row 5), tokens are grouped by position. Left : pre-activation $\mathrm{PR}_{\mathrm{pre}}$ ; Middle : post-activation $\mathrm{PR}_{\mathrm{post}}$ ; and Right : layer-wise $\mathrm{PR}_{\mathrm{post}}$ . In GPT-2 with baseline, both GELU and ReLU variants, early/middle/late tokens have similar $\mathrm{PR}_{\mathrm{pre}}$ but diverge strongly in $\mathrm{PR}_{\mathrm{post}}$ , with late tokens using substantially higher effective dimensionality, mainly visible in deeper layers. However, in the normalization-free GPT-2 variants, these position-conditioned differences are largely suppressed in both pre- and post-activations. On the other hand, in a non-transformer family model, MLP-Mixer trained with SGD optimizer, where we split patches into top vs bottom rows, position effects are weak throughout. This show that strong position-dependent FFN geometry is a characteristic of LayerNorm-based GPT-2

Figure 24: Effect of LayerNorm vs RMSNorm on FFN eigenspectrum . Top row: Post-activation participation ratio (

\mathrm{PR}_{\mathrm{post}}

) in GPT-2 (125M) for three FFN activations (GELU, ReLU, learnable leaky ReLU), three normalization placements (PreLN, PostLN, MixLN), and two normalization variants. Within each activation-placement pair, the LayerNorm and RMSNorm boxplots substantially overlap, while much larger shifts arise from changing the activation type or the norm placement. Bottom row: Layerwise comparison of

\text{PR}_{\text{post}}

under LayerNorm (

x

-axis) vs RMSNorm (

y

-axis). Each point represents one FFN layer. GELU and ReLU show similar PR ranges but weak layerwise correspondence (

R^{2}<0.2

), while learnable leaky ReLU exhibits strong alignment (

R^{2}\approx 0.77

). This indicates that FFN eigenspectrum patterns are primarily driven by activation type and are largely robust to the choice of normalization scheme.

Figure 25: Effect of LayerNorm vs RMSNorm in the MLP-Mixer models . Columns (from left to right) shows post-activation SE, PR, EEE, and finally JS metric trend for LayerNorm and RMSNorm configurations throughout the training when MLP-Mixer models (GELU in both MLPs) are trained from scratch on CIFAR-100 dataset using Adam optimizer. FFN eigenspectrum dynamics remain (qualitatively) stable when LayerNorm is replaced with RMSNorm. However, in the later phases of training, the LayerNorm configuration attains higher effective dimensionality (PR) than RMSNorm, and its spectrum becomes noticeably flatter.

Figure 26: Optimizer-dependent FFN eigenspectrum dynamics in GPT2-160M (context length 1024). Rows show AdamW (top), Muon (middle), and Dion (bottom).

Figure 27: Optimizer-dependent FFN eigenspectrum dynamics in GPT2-160M (context length 512). Rows show AdamW (top), Muon (middle), and Dion (bottom).

Panel of Figure 28 — Figure 28: Activation-compatibility of pre-activation eigenspectrum . Columns show the effective dimensionality of pre-activation spectrum (PR_pre) when GPT2-160M models are trained from scratch with AdamW (left), Muon (middle), and Dion (right) optimizers, with 512 (top) and 1024 (bottom) context length. Muon exhibits the highest PR_pre across layers, demonstrating well-conditioned pre-activation spectra that place less burden on the FFN nonlinearity to restore representational rank.

Optimizer	GPT2-350M	GPT2-160M	GPT2-160M
	(Context 512)	(Context 512)	(Context 1024)
AdamW ( Loshchilov and Hutter, 2019 )	33.24	39.26	34.33
Muon ( Jordan et al., 2024 )	25.68	30.95	27.09
Dion ( Ahn et al., 2025 )	27.68	33.60	29.34

Appendix

Table 13: Evaluation perplexity (PPL) for GPT-2 models trained from scratch on the FineWeb dataset ( Penedo et al., 2024 ) using AdamW, Muon, and Dion optimizer. Muon consistently achieves the lowest perplexity across all model sizes and context lengths, while AdamW results in worse perplexity.

Figure 29: Eigen-metrics (SE, PR, EEE, and JS) illustrate how FFN nonlinearities regulate information flow and reshape the eigenspectrum when GPT-2 (125M) models with GELU (top) and ReLU (bottom) are trained from scratch on CodeParrot (2.1B tokens) datasets using Adafactor optimizer ( Shazeer and Stern, 2018 ) . Pre- and post-activation dynamics are shown for SE, PR, and EEE, highlighting how nonlinearities reinject variance and alter spectral structure. JS heatmaps (rightmost) capture the layerwise distributional shift induced by nonlinearity.

Figure 30: Layerwise post-activation participation ratio (PR_post) comparison for AdamW and Adafactor across four GPT-2 (125M) configurations: baseline (PreLN) with GELU and ReLU (top), and normalization-free GELU and ReLU (bottom). Across settings, Adafactor systematically produces higher PR_post than AdamW, indicating higher FFN latent-space utilization, with the largest gains in the normalization-free ReLU model.

Figure 31: Training dynamics of post-activation participation ratio (PR) for all layers in GPT-2 (125M) across four configurations (columns) and two optimizers (rows). Deeper layers achieve and maintain higher PR_post throughout training, with Adafactor consistently producing larger PR_post than AdamW, especially in the deeper FFNs and in the normalization-free ReLU models.

Figure 32: Training dynamics of FFN-induced spectral flattening in GPT-2 (125M), quantified as $\Delta$ EEE = EEE ${}_{\text{post}}$ - EEE ${}_{\text{pre}}$ , under AdamW (top row) and Adafactor (bottom row) optimizer. More negative $\Delta$ EEE indicates stronger suppression of top eigenvalues and a flatter spectrum. Across settings, deeper layers under Adafactor tend to reach more negative $\Delta$ EEE than under AdamW, indicating more aggressive spectral flattening.

Figure 33: Optimizer effects in MLP-Mixer (Adam vs. SGD) . Columns (from left to right) shows post-activation SE, PR, EEE, and finally JS divergence metric trend when MLP-Mixer baseline model (GELU in both MLPs) is trained from scratch on CIFAR-100 dataset using Adam and SGD optimizer. Right from the epoch 10, the MLP-Mixer with the SGD optimizer starts exhibiting superior SE_post and PR_post showing significantly better utilization of FFN2 latent space.

Transformer FFNs

Transformer feed-forward networks (FFNs) are the within-token MLP blocks whose activations, structure, efficiency, and role in model computation are studied.

See all 42 papers

May 14, 2026

Dynamics of the Transformer Residual Stream: Coupling Spectral Geometry to Network Topology

Large language models are remarkably capable, yet how computation propagates through their layers remains poorly understood. A growing line of work treats depth as discrete time and the residual stream as a dynamical system, where each layer's nonlinear update has a local linear description. However, previous analyses have relied on scalar summaries or approximate linearizations, leaving the full spectral geometry of trained LLMs unknown. We perform full Jacobian eigendecomposition across three production--scale LLMs and show that training installs a monotonic spectral gradient through depth -- from non-normal, rotation-dominated early layers to near--symmetric late layers -- together with a cumulative low-rank bottleneck that funnels perturbations into a small fraction of the residual stream's effective dimensions. Our experiments reveal that this gradient and the dimensional collapse are learned rather than architectural, and is largely dissolved when structured non-normality is removed. We further show that the topological positioning of graph communities predicts whether the Jacobian amplifies or suppresses them, with the sign of the coupling determined by the local operator type, a relationship absent at initialization. These results map a learned spectral geometry in LLMs that links perturbation propagation and compression to the network's functional topology.

Jesseba Fernando, Grigori Guitchounts

Network Science Institute, Northeastern University · Flagship Pioneering

Dynamical Systems Transformer LLM Interpretability Spectral Methods Neural Network Training Dynamics

Ask PDF

May 7, 2026

Spectral Lens: Activation and Gradient Spectra as Diagnostics of LLM Optimization

Training loss and throughput can hide distinct internal representation in language-model training. To examine these hidden mechanics, we use spectral measurements as practical and operational diagnostics. Using a controlled family of decoder-only models adapted from the modded NanoGPT codebase, we introduce an empirical protocol based on activation covariance and per-sample gradient SVD spectra. This dual-view reveals three empirical findings and one mechanistic explanation. First, batch size acts as a latent determinant of representation geometry: runs that reach equal loss settle into systematically distinct activation spectra. Second, the activation covariance tail measured early in training reliably forecasts downstream token efficiency. Third, movement of the activation spectrum head (leading modes), together with gradient spectra, characterizes underlying learning-dynamics changes, separating learning-side architectural improvements from primarily execution-side gains. These predictive and diagnostic signals persist across the 12-, 36-, and 48-layer model tiers. Finally, a mechanistic model proves the main observations and explains how activation covariance spectra correlate with task-aligned feature learning.

Andy Zeyi Liu, Elliot Paquette, John Sous

Department of Applied Physics, Yale University, New Haven, CT 06511, United States · Department of Mathematics and Statistics, McGill University · Department of Applied Physics, Yale University, New Haven, Connecticut 06511, USA +1

Representation Learning Decoder-Only Language Models Neural Network Interpretability Neural Network Training Dynamics LLM Training

Ask PDF

May 29, 2026

Spectral Reach: Understanding Neural Scaling as Progress into the Spectral Tail

Neural scaling laws describe predictable power-law relationships between model size, dataset size, compute, and performance. While these laws guide the development of modern foundation models, the mechanisms underpinning them remain poorly understood, in part due to the absence of scalable analysis tools. To close this gap, we introduce "spectral position": a scalable measure of which eigenvalues of the empirical neural tangent kernel (eNTK) currently drive loss reduction. Applying this measure to scaling experiments, we find that spectral position decreases throughout training: learning shifts from dominant eigenmodes into the spectral tail. Larger models reach further into the tail than smaller models, revealing a size-dependent capacity we call "spectral reach". This suggests why larger models achieve lower losses: they sustain learning on weak spectral signals inaccessible to smaller models. We further identify feature learning as a key enabler of spectral reach. It adaptively amplifies gradient magnitudes as learning advances, sustaining progress where frozen representations stall. This points to concrete interventions through architecture and optimizer design.

Konstantin Nikolaou, Jonas Scheunemann, Sven Krippendorf +2

Institute for Computational Physics, University of Stuttgart, Stuttgart, Germany · Department of Applied Mathematics and Theoretical Physics and Department of Physics, University of Cambridge, Cambridge, UK

Representation Learning Neural Network Training Dynamics Neural Tangent Kernel Neural Scaling Laws

Ask PDF

Abstract

Transformer FFNs

Explore similar work

Dynamics of the Transformer Residual Stream: Coupling Spectral Geometry to Network Topology

Spectral Lens: Activation and Gradient Spectra as Diagnostics of LLM Optimization

Spectral Reach: Understanding Neural Scaling as Progress into the Spectral Tail