Transformer Training Dynamics

Follow this topic Ask about this topic

Latest papers

26 papers5 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Oct 18, 2026

T-MUON: Tensorizing Muon via t-SVD for Large Language Model Pre-training

Muon has recently emerged as a strong optimizer for large language model pretraining by exploiting matrix structure through polar orthogonalization. However, it processes each parameter matrix independently, whereas recent tensor-aware extensions of Muon jointly update groups of eligible matrix parameters through matrix unfoldings. In contrast, we propose T-MUON, a t-SVD-based generalization that groups structurally related Transformer gradients into third-order tensors and defines the corresponding Muon-style update directly through the

M

-transform. We show that the exact T-MUON update is a steepest-descent direction under the

\star_M

-spectral norm and establish a convergence guarantee. Across GPT-2-style and LLaMA-style models from 51M to 1.3B parameters trained on FineWeb and FineWeb-Edu, head-wise T-MUON with the identity transform consistently achieves lower validation perplexity than AdamW, Muon, and variants using DFT, DCT, or DST transforms. These gains persist in additional experiments with longer context lengths and larger token budgets, although the preferred tensorization mode varies across settings.

Dongyang Li, Mingsong Yan, Zheng Zhang +1

University of California, Santa Barbara Santa Barbara, CA 93106, USA

Muon Optimizer Language Model Pretraining Transformer Training Dynamics

Oct 16, 2026

A Conditional No-Hair Theorem for Transformers: How Language Models Forget Their Prompt

We ask how much an autoregressive transformer retains of its prompt after generating

t

tokens. Consider two different prompts followed by the same continuation text, so that any difference in their hidden states comes from the prompt alone, and we find that the distance between their hidden states decays approximately as a power law

t^{-α}

, where

α=1

is the rate at which a fixed prompt's share of uniform attention shrinks and smaller values mean longer memory. Trained models forget fast (

α\approx0.55

--

0.7

), untrained ones slowly (

α\approx0.2

--

0.5

). We prove that this offset is

O\big(P(\log t)^{L-1}/t\big)

for a prompt of length

P

and depth

L

. The bound holds for finite-depth pre-LN softmax transformers with bounded attention logits, under one additional condition: the pre-norm activations stay away from zero. The mechanism is geometric: the prompt's direct influence is diluted as the context grows, and relay --- continuation tokens passing on what earlier tokens read from the prompt --- can only add logarithmic factors. This is what we call the conditional no-hair theorem: given the text generated so far, the prompt's influence on the next token vanishes as

t

grows, so any property of the prompt that survives, such as its language, must be carried by the text itself. An exactly solvable linear relay model shows that the bound cannot be improved and reproduces the rise of the local exponent towards

1

seen in untrained networks. Cancelling the dilution with a logit bias stops the decay early in training and reverses it after, so dilution is necessary for the forgetting we measure; untrained networks and the solvable model show it is also sufficient.

Shuyang Xiang

Independent researcher

Transformer Attention Transformer Training Dynamics

Oct 14, 2026

Training Dynamics of Transformers on Unbounded Inputs: Attention Learning Can Lag Output Layer Learning

The success of transformers is attributed to several factors, including their capacity to express formal languages over variable-length inputs, and ability of the attention mechanism to focus on relevant input features. However, there is limited understanding of the training dynamics of shallow transformers, specifically how to decouple the learning of the attention and the output (feedforward) layer. A natural extension of the theory of one-step gradient feature learning in feedforward networks suggests that one could first learn the attention weights with a single gradient step, and then train the output layer. We show that this fails for a shallow transformer trained to recognize a star-free language from unbounded-length input. We prove that the transformer can be trained to recognize the language with a three-phase gradient descent: the output layer is updated first, followed by an attention update, and the output layer is fine-tuned in the third phase. In contrast, the language cannot be learned by training only the attention or output layer, or by a single attention-layer update followed by finetuning the output layer. We empirically show that ``attention learning lagging output layer learning" is beneficial in other language recognition, regression, and in-context learning tasks as well.

Yaomengxi Han, Debarghya Ghoshdastidar

Technical University of Munich

Transformer Training Dynamics Neural Network Training Dynamics Transformer Attention Transformer FFNs

Oct 14, 2026

Asymptotic Behavior of Transformers

In this paper we study the behavior of transformer networks generating text over long periods of time. We show that even a three-token, single-layer attention-only transformer can fail to converge. We then study a rich class of transformers with arbitrary finite depth, layer normalization, MLPs, RoPE encodings, and an arbitrary mix of local and global layers. Every transformer in this class has a softmax temperature threshold above which its empirical cache distribution converges to a unique limit, independent of initial conditions. We show further that, while each transformer has a temperature threshold, the supremum of this set is infinite, so there is no global minimal temperature that forces all transformers to converge.

Katy Blumer, Da-Cheng Juan, Jon Kleinberg +3

Google Research

Transformer Training Dynamics Transformer Text Generation Softmax Attention

Oct 13, 2026

Same Function, Different Trajectory: No Standard Optimizer Is Equivariant to Transformer Reparameterization

We show that two Transformers representing the same function can follow different training trajectories. For example, doubling a query matrix and halving its key matrix preserves the model's function but changes the gradient scales seen by the optimizer. Our theoretical analysis establishes that standard block-local first-order optimizers fail to scale updates with parameters under a shared learning rate. For AdamW, transforming the first- and second-moment estimates alone does not correct the mismatch between parameter and update scales. To make runs restarted from equivalent checkpoints follow the same trajectory, we balance matrix pairs and transform their moment estimates together, bringing distinct but equivalent training states to the same starting point. To correct scale imbalance between paired Transformer matrices without changing predictions, we propose Symmetry-Aware Gradient Geometry Stabilization (SAGGS), which periodically equalizes the Frobenius norms of exact symmetry pairs, and its damped variant, Soft SAGGS. SAGGS reduces perplexity (PPL) by

0.374\pm0.026

at 101M parameters on WikiText-103, and Soft SAGGS reduces PPL by

0.780\pm0.040

in a 31M modern-architecture replication, both under scale imbalance. In a 123.8M checkpoint-resume experiment, reciprocally rescaling matrix pairs barely changes validation loss at restart but increases final PPL by

3.78\pm0.42

; hard and soft rebalancing recover to within

0.02

PPL of the control.

Bum Jun Kim

The University of Tokyo

Deep Learning Optimization Transformer Transformer Training Dynamics Transformer Attention

Jul 24, 2026

Hidden Boundary Motion in Transformer Optimization: Function-Space Orthogonalization of Affine Weight and Bias Updates

Weights and biases are normally optimized as separate parameter tensors, yet they do not represent separate functions when the input to an affine layer has nonzero mean. For an affine map

z=Wx+b

with input mean

μ

, a weight update contains a sample-independent displacement

ΔWμ

that is functionally indistinguishable from a bias update. We call this hidden contribution boundary motion and decompose each update into a centered, sample-varying shape component and a shared boundary component. On a four-layer Transformer trained from scratch on IMDb, the bias-like term

g_bμ^\top

has a median norm equal to 0.664 of the raw weight-gradient norm across affine layers and training checkpoints. More strikingly, the median ratio

\norm{ΔWμ}/\norm{Δb}

is 134.7, while

\norm{ΔWμ}/\norm{Δb+ΔWμ}

is 0.994. Thus, under AdamW, the observed boundary motion is almost entirely realized through the weight matrix rather than the explicit bias. We implement a diagnostic optimizer, Shape--Boundary Orthogonal AdamW (SBO-AdamW), that optimizes

g_W-g_bμ^\top

and

g_b

with independent Adam states and compensates the weight-induced boundary displacement. In a single-seed experiment, SBO-AdamW raises validation accuracy from 81.68% to 85.81% and validation-selected test accuracy from 78.73% to 82.73%, with the best validation checkpoint occurring at step 800 instead of step 3000. However, the moving-batch-center compensation produces severe bias-coordinate drift and strongly reduces boundary energy. The present evidence therefore supports hidden boundary motion as an important optimization mechanism, but it does not yet establish a final general-purpose optimizer. A stable centered-affine parameterization is identified as the required next step.

Zhang Gongyue, Sheng Yixuan, Liu donghan +3

arXiv:2607.22927v1

Transformer Neural Network Optimization Adaptive Gradient Methods Transformer Training Dynamics

Jul 13, 2026

Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks

We present a theoretical framework to explain the emergence of inductive reasoning abilities in Transformer language models. While previous works on Transformer learning dynamics have so far been mostly tied to specific tasks, we study a generalized class of inductive tasks that unifies several synthetic tasks known in the literature, including in-context n-grams and multi-hop reasoning. In this class, we theoretically prove that the training dynamics of attention models can be confined to a highly interpretable, low-dimensional invariant manifold. On this manifold, the learning dynamics are captured by a handful of interpretable coordinates rather than millions of parameters, making both theoretical and empirical analysis more tractable. Using this framework, we characterize how data statistics govern the competition between in-context and in-weights learning, we study how random initializations determine the `winning' circuit when multiple solutions are possible, and we demonstrate that the coordinate frame associated with the manifold can be used to automatically detect which circuits have been learned in trained models. By casting circuit formation as a low-dimensional dynamical phenomenon, we take a step toward a predictive theory of how Transformers learn.

Tiberiu Musat, Tiago Pimentel, Nicolas Zucchet +1

ETH Zurich · Stanford University

Transformer Training Dynamics In-Context Learning Circuits in Language Models Neural Network Training Dynamics Inductive Reasoning in Language Models

Jun 16, 2026

Loop- $μ$ P: Residual Scaling and Hyperparameter Transfer in Looped Transformers

Looped Transformers have drawn growing attention because looping improves multi-step reasoning and increases the executed depth of a model without adding parameters. They achieve this by reusing a stack of

L

layers for

N

loops. However, increasing the loop count

N

not only destabilizes training but also shifts the best hyperparameters, so they must be searched again for every loop count. Inspired by Depth-

μ

P (Yang et al., 2024), which multiplies each residual branch by

L^{-1/2}

for depth

L

so that a learning rate tuned on a shallow untied network transfers to a deep one, we study how the residual branches and the learning rate should scale with the loop count

N

. We find that loops cannot simply be treated as extra depth, because the loops share weights. At initialization, distinct layers add nearly uncorrelated increments to the residual stream, so their sum grows as

\sqrt{L}

, whereas a shared layer adds positively correlated increments at every loop, so their sum grows linearly in

N

. For a single layer looped

N

times, we show in the infinite-width limit that multiplying the residual branch by

1/N

and keeping the learning rate independent of

N

is the only parameterization with both a stable forward pass and maximal residual-stream updates. We then generalize the analysis to a stack of

L

distinct layers looped

N

times to obtain a parameterization we call Loop-

μ

P, in which the loop factor

1/N

combines with the depth factor

L^{-1/2}

. Empirically, we find in language modeling that the base learning rate tuned on the smallest model stays best under Loop-

μ

P for loop counts up to

8

and executed depths up to

384

layers. These results make Loop-

μ

P a practical recipe for scaling looped Transformers.

Shaowen Wang, Bingrui Li, Tansheng Zhu +4

Tsinghua University · ByteDance Seed · M-A-P

Residual Learning Hyperparameter Transfer Recurrent Transformers Neural Network Training Dynamics Transformer Training Dynamics

Jun 15, 2026

Taming Curvature: Architecture Warm-Up for Stable Transformer Training

Training billion-parameter Transformers is often brittle, with transient loss spikes and divergence that waste compute. Even though the recently developed Edge of Stability (EoS) theory provides a powerful tool to understand and control the stability of optimization methods via the (preconditioned) curvature, these curvature-controlling methods are not popular in large-scale Transformer training due to the complexity of curvature estimation. To this end, we first introduce a fast online estimator of the largest (preconditioned) Hessian eigenvalue (i.e., curvature) based on a warm-started variant for power iteration with Hessian-vector products. We show theoretically, and verify empirically, that the proposed method makes per-iteration curvature tracking feasible at billion parameter scale while being more accurate. Using this tool, we find that training instabilities coincide with surges in preconditioned curvature and that curvature grows with depth. Motivated by these observations, we propose architecture warm-up: progressively growing network depth to carefully control the preconditioned Hessian and stabilize training. Experiments on large Transformers validate that our approach enables efficient curvature tracking and reduces instabilities compared to existing state-of-the-art stabilization techniques without slowing down convergence.

Sameera Ramasinghe, Ajanthan Thalaiyasingam, Hadi Mohaghegh Dolatabadi +6

Pluralis Research

Edge of Stability Transformer Second-Order Optimization Neural Network Training Dynamics Transformer Training Dynamics

May 23, 2026

Momentum Streams for Optimizer-Inspired Transformers

The residual update of a pre-norm Transformer layer admits an interpretation as one step of a first-order optimizer acting on a surrogate token energy, wherein the attention and MLP sublayers function as gradient oracles. Based on this observation, we build a family of optimizer-inspired Transformers (triple-momentum, Adam/AdamW, Muon, SOAP) and compare them under matched compute. In our main pretraining experiment, the triple-momentum TMMFormer achieves the lowest validation loss, outperforming the vanilla Transformer and prior architectural variants. A controlled ablation and supporting theory show that momentum, not preconditioning, is the main source of the gain. We further show that TMMFormer and other momentum-based designs reach flatter minima than the vanilla Transformer, which leads to less forgetting and better generalization.

Jingchu Gai, Nai-Chieh Huang, Jiayun Wu

Carnegie Mellon University

Deep Learning Optimization Transformer Momentum Methods Neural Network Optimization Flat Minima Transformer Training Dynamics

May 20, 2026

Large-Step Training Dynamics of a Two-Factor Linear Transformer Model

Gradient-flow analyses show that simplified linear transformers can learn the in-context linear-regression algorithm, but they do not explain the finite-step behavior of gradient descent at large learning rates. Motivated by empirical work on high-learning-rate transformer instabilities and by the cubic-map phase diagram for quadratic regression, we study an exactly reducible one-prompt linear-transformer training problem. After normalization, the dynamics reduce to a two-factor product map with an effective step-size parameter

μ

. On the balanced slice, this map recovers the known scalar cubic transition from monotone convergence to catapult convergence, periodic and chaotic bounded nonconvergence, and divergence. We then analyze the full two-dimensional system and show that, for

0<μ<2

, it has an explicit invariant Chebyshev ellipse separating forward-invariant regions; this ellipse carries off-balanced chaotic dynamics but is transversely repelling, while balanced scalar attractors can be transversely attracting. These results show that large constant learning rates can change the training attractor of the learned transformer rather than merely accelerating convergence: beyond sharp stability thresholds, finite-step training may settle into cycles, bounded chaos, or divergence instead of a single in-context linear-regression solution. We also discuss the consequences for mini-batch gradient descent based training methods.

Krishnakumar Balasubramanian

Department of Statistics, University of California, Davis

Transformer Phase Transitions Neural Network Training Dynamics Gradient Descent Dynamics Transformer Training Dynamics

May 19, 2026

Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics

Transformers trained on modular arithmetic exhibit sharp transitions between memorization, generalization, and collapse. We show that weight decay acts as a scalar empirical control parameter for these regimes, and introduce two cheap online diagnostics, mean pairwise attention-head cosine similarity and entropy standard deviation, that track training dynamics from attention activations alone and complement loss-landscape diagnostics at lower compute cost. Across eleven experimental conditions and three model scales (0.82M to 85M parameters), the weight-decay axis separates memorization, developmental grokking, and collapse. A near-transition logistic fit localizes the memorization-to-developmental boundary at

λ_c=0.0158

(95% CI [0.0109, 0.0200], N=210); a power-law fit gives an empirical exponent

ν=0.757

(CI [0.725, 0.799]). Reference exponents

ν=1/2

and 3D Ising

ν\approx 0.63

lie outside this empirical CI under our four-bin grid, so we report

ν

as empirical and defer universality-class identification to denser finite-size-scaling work. A horizon-matched multi-task replication (n=280, four modular operations) preserves the weight-decay control pattern; a paired attention-head re-initialization experiment at

λ=0.05

changes Phase-2 amplitude (Cohen's

d=-1.190

, n=10,

p_t=4.5 \times 10^{-3}

), while matched weight-norm clipping does not. Three cross-architecture probes (4L MLP, 4L LSTM, and 4L Mamba; each n=70) replicate the weight-decay-controlled transition with architecture-specific

λ_c

values. Main diagnostic claims are scoped to modular arithmetic in small transformer attention models; the non-attention experiments are scope probes, and architecture-wide, language-model, and universality-class claims are out of scope.

Lucky Verma

Independent Researcher

Weight Decay Neural Network Memorization Phase Transitions Neural Network Training Dynamics Grokking Transformer Training Dynamics

May 15, 2026

Multi-Headed Transformer Architectures as Time-dependent Wasserstein Gradient Flows

In recent years, transformer architectures have revolutionized the field of language processing, opening the door to previously unforeseen possibilities. However, from a theoretical point of view, the mathematical models proposed in the literature often lack direct contact with the actual architectures and depend on strong simplifying assumptions. In this paper, we reduce this gap by modelling the data flow in multi-headed transformer architectures as time-dependent gradient flows for a suitable interaction energy capturing the design of the attention mechanism. The explicit dependence on time allows us to consider different weights for each head and for each layer, without imposing constraints on the initialization method. Moreover, we prove that, under a suitable integrability assumption on the evolution of the weights, each element of the

ω

-limit set of the gradient flows is a stationary point of the interaction energy at a limiting weight distribution. Finally, we analyse the stability of the gradient flows considering perturbations of both the initial data and the weights. Specifically, on the one hand, we study the robustness of the proposed models with respect to noisy inputs, establishing a continuous dependence of the gradient flows on the initial data and uniqueness of the flows. On the other hand, we prove the

Γ

-convergence of the perturbed interaction energy to the unperturbed one, leading to the convergence of the corresponding gradient flows. We complement these theoretical results with numerical experiments that confirm the predicted energy-dissipation identity and clarify the asymptotic behavior of the dynamics in both the autonomous-like (Ornstein--Uhlenbeck) and the genuinely non-autonomous (oscillating-weights) regimes.

Alex Massucco, Leonardo Del Grande, Marcello Carioni +2

Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, UK · Department of Mathematics, University of Twente, Enschede, Netherlands

Transformer Transformer Training Dynamics Wasserstein Gradient Flows Transformer Attention Multi-Head Attention

May 8, 2026

Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers

Transformers perform inference by iteratively transforming token representations across layers. This layerwise computation has been studied empirically, and recent mean-field theories of Transformer dynamics explain how attention can drive token distributions toward clustering. However, existing analyses treat model parameters as prescribed, leaving open how training reshapes this clustering picture. We study this question in a noisy mean-field Transformer in which only a parameter-linear FFN is trained under

L^2

regularization. We identify a training-induced phase of the inference dynamics across layers: after clustering in the early layers, the token distribution can leave the clustered regime near the final layers. A single gradient-descent update already produces this escape, and minimizers of the training objective satisfy an exponential turnpike bound under which clustering persists through the interior layers. Our analysis is based on an entropy-regularized interaction energy that captures the clustering bias of attention, for which we establish a double-well landscape and a local Polyak-Łojasiewicz inequality. More broadly, our results point toward a training-aware mean-field theory in which training and inference dynamics of Transformers are treated together.

Noboru Isobe, Daisuke Inoue, Masaaki Imaizumi

RIKEN Center for Advanced Intelligence Project · Imperial College London · The University of Tokyo +1

Transformer Transformer Training Dynamics Clustering Transformer Attention Mean-Field Theory Neural Network Training Dynamics

May 6, 2026

Critical Windows of Complexity Control: When Transformers Decide to Reason or Memorize

Recent work has shown that Transformers' compositional generalization is governed by complexity control, initialization scale and weight decay, which steers training toward low-complexity reasoning solutions rather than high-complexity memorization. Existing analyses, however, treat complexity control as a single static hyperparameter choice, leaving open when during training this control is actually decisive. We show that the memorization-versus-reasoning fate of a Transformer is determined within a sharp, identifiable window of training. On a controlled compositional task we find that (i)~weight decay applied for a single 25%-of-training window matches full-training weight decay in out-of-distribution (OOD) accuracy (

0.93

vs

0.91

); (ii)~holding total regularization budget constant, placing it in the middle of training yields

5{-}9\times

higher OOD accuracy than placing it early; (iii)~the boundary of the critical window is remarkably sharp, window onset shifted by as little as

100

optimization steps causes mean OOD to jump from chance (

0.15

) to reasoning-regime (

0.61

); (iv)~the window's position depends systematically on initialization scale, but the basin of attraction for reasoning solutions shrinks at small initialization, contradicting the prevailing recommendation that smaller initialization is uniformly better. We further show that the critical-window phenomenon is task-specific: it does not appear on grokking with modular arithmetic, where properly tuned constant weight decay matches scheduled weight decay.

Sarwan Ali

Columbia University, Irving Medical Center, NY, USA

Deep Learning Optimization Weight Decay Transformer Training Dynamics OOD Generalization Neural Network Training Dynamics Compositional Generalization

May 4, 2026

Trust, but Verify: Peeling Low-Bit Transformer Networks for Training Monitoring

Understanding whether deep neural networks are effectively optimized remains challenging, as training occurs in highly nonconvex landscapes and standard metrics provide limited visibility into layer-wise learning quality. This challenge is particularly acute for transformer-based language models, where training is expensive, models are often reused in frozen form, and poorly optimized layers can silently degrade performance. We propose a layer-wise peeling framework for monitoring training dynamics, in which each transformer layer is locally optimized against intermediate representations of the trained model. By constructing lightweight, layer-specific reference solutions and projecting layers onto multiple intermediate outputs via different permutations, we obtain achievable baselines that enable fine-grained diagnosis of under-optimized layers. Experiments on decoder-only transformer models show that these layer-wise reference bounds can match or even surpass the trained model at various stages of training, exposing inefficiencies that remain hidden in aggregate loss curves. We further demonstrate that this analysis remains effective under binarization and quantized settings, where training dynamics are particularly fragile. Across all numerical results, the proposed bounds consistently separate apparent convergence from effective optimality, highlighting optimization opportunities that are invisible when relying on training loss alone.

Arian Eamaz, Farhang Yeganegi, Mojtaba Soltanalian

Department of Electrical and Computer Engineering University of Illinois Chicago Chicago, IL, USA

Deep Learning Optimization Neural Network Training Dynamics Transformer Training Dynamics

May 2, 2026

Focus and Dilution: The Multi-stage Learning Process of Attention

Transformer-based models have achieved remarkable success across a wide range of domains, yet our understanding of their training dynamics remains limited. In this work, we identify a recurrent focus-dilution cycle in attention learning and provide a rigorous explanation in a one-layer Transformer setting for Markovian data via gradient-flow analysis. Using stage-wise linearization around critical points, we show that a single focus-dilution cycle can be decomposed into a sequence of distinct stages. First, embedding and projection rapidly condense to a rank-one structure, while attention parameters remain effectively frozen. Then, the attention parameters begin to increase, inducing a frequency-driven focus toward high-frequency tokens. As attention continues to evolve, it generates next-order perturbations in embeddings, leading to a mass-redistribution mechanism that progressively dilutes this focus. Finally, small asymmetries among low-frequency tokens lift a degenerate critical point, opening new embedding directions and initiating the next cycle. Experiments on synthetic Markovian data as well as WikiText and TinyStories corroborate the predicted stages and cyclical dynamics.

Zheng-An Chen, Pengxiao Lin, Zhi-Qin John Xu +1

School of Mathematical Sciences, Shanghai Jiao Tong University. · Institute of Natural Sciences, Shanghai Jiao Tong University. · MOE-LSC, Shanghai Jiao Tong University. +2

Self-Attention Transformer Attention Neural Network Training Dynamics Transformer Training Dynamics

Apr 14, 2026

Distinct mechanisms underlying in-context learning in transformers

Modern distributed networks, notably transformers, acquire a remarkable ability (termed `in-context learning') to adapt their computation to input statistics, such that a fixed network can be applied to data from a broad range of systems. Here, we provide a complete mechanistic characterization of this behavior in transformers trained on a finite set

S

of discrete Markov chains. The transformer displays four algorithmic phases, characterized by whether the network memorizes and generalizes, and whether it uses 1-point or 2-point statistics. We show that the four phases are implemented by multi-layer subcircuits that exemplify two qualitatively distinct mechanisms for implementing context-adaptive computations. Minimal models isolate the key features of both motifs. Memorization and generalization phases are delineated by two boundaries that depend on data diversity,

K = |S|

. The first (

K_1^\ast

) is set by a kinetic competition between subcircuits and the second (

K_2^\ast

) is set by a representational bottleneck. A symmetry-constrained theory of a transformer's training dynamics explains the sharp transition from 1-point to 2-point generalization and identifies key features of the loss landscape that allow the network to generalize. Put together, we show that transformers develop distinct subcircuits to implement in-context learning and identify conditions that favor certain mechanisms over others.

Cole Gibson, Wenping Cui, Gautam Reddy

Lewis-Sigler Institute for Integrative Genomics, Princeton University, Princeton, NJ 08544, USA · Department of Physics, Princeton University, Princeton, NJ 08544, USA

In-Context Learning Transformer Training Dynamics Transformer Interpretability Neural Network Training Dynamics Mechanistic Interpretability Statistical Physics of Learning

Apr 3, 2026

The Spectral Lifecycle of Transformer Training: Transient Compression Waves, Persistent Spectral Gradients, and the Q/K--V Asymmetry

We present the first systematic study of weight matrix singular value spectra during transformer pretraining, tracking full SVD decompositions of every weight matrix at 25-step intervals across three model scales (30M--285M parameters). We discover three phenomena: (1)~Transient Compression Waves: stable rank compression propagates as a traveling wave from early to late layers, creating a dramatic gradient that peaks early then reverses -- late layers eventually over-compress past early layers. (2)~Persistent Spectral Gradients: the power-law exponent~

α

develops a permanent depth gradient forming a non-monotonic inverted-U in deeper models, with peaks shifting toward earlier layers as depth increases. (3)~Q/K--V Functional Asymmetry: value/output projections compress uniformly while query/key projections carry the full depth-dependent dynamics. The dissociation between transient compression and persistent spectral shape reveals that rank and spectral shape encode fundamentally different information about training. We formalize this as a two-timescale dynamical model and derive scaling laws (

Δα\propto L^{0.26}

,

R^2{=}0.99

). We validate on nine models across three families (custom, GPT-2, Pythia; 30M--1B parameters; 8--36 layers), demonstrate that

α

predicts layer importance (

ρ{=}0.69

--

0.84

,

p{<}0.02

), and show that spectral-guided pruning outperforms Last-N heuristics by

1.1{\times}

--

3.6{\times}

across seven models in two families (GPT-2 124M--774M, Pythia 160M--1B), with worst-vs-best gaps up to

23.7{\times}

confirming the causal role of spectral structure.

Yi Liu

Transformer Training Dynamics Transformer Neural Network Training Dynamics Spectral Methods

Mar 14, 2026

Spectral Edge Dynamics of Training Trajectories: Signal--Noise Geometry Across Scales

Despite hundreds of millions of parameters, transformer training trajectories evolve within only a few coherent directions. We introduce Spectral Edge Dynamics (SED) to quantify this structure: a rolling-window SVD of parameter updates reveals a sharp boundary -- the spectral edge -- between coherent optimization directions and stochastic noise, identified via the maximum consecutive singular value ratio

σ_k / σ_{k+1}

. Across a 51M-parameter TinyStories model (4 seeds) and GPT-2 124M under distribution shift, the spectral edge exhibits a universal three-phase pattern (rise, plateau, collapse). The effective signal rank adapts to task complexity (

k^* = 2

at 51M,

k^* = 3

at 124M), and the directional coupling between spectral geometry and validation loss reverses with window size -- a lag flip reflecting the timescale of trajectory integration. Johnson--Lindenstrauss projection to

d = 10W

dimensions (e.g.,

d = 100

for

W = 10

) preserves the spectral gap within

5.7\%

, making the framework applicable to models of arbitrary scale. In companion work, the same spectral geometry provides early-warning signals of grokking -- predicting generalization

600

--

1{,}700

steps before it occurs across modular arithmetic, Dyck languages, and the SCAN benchmark.

Yongzhong Xu

Transformer Training Dynamics Neural Network Training Dynamics Gradient Descent Dynamics Spectral Methods

Mar 10, 2026

An Optimal Control Approach To Transformer Training

In this paper, we develop a rigorous optimal control-theoretic approach to Transformer training that respects key structural constraints such as (i) realized-input-independence during execution, (ii) the ensemble control nature of the problem, and (iii) positional dependence. We model the Transformer architecture as a discrete-time controlled particle system with shared actions, exhibiting noise-free McKean-Vlasov dynamics. While the resulting dynamics is not Markovian, we show that lifting it to probability measures produces a fully-observed Markov decision process (MDP). Positional encodings are incorporated into the state space to preserve the sequence order under lifting. Using the dynamic programming principle, we establish the existence of globally optimal policies under mild assumptions of compactness. We further prove that closed-loop policies in the lifted is equivalent to an initial-distribution dependent open-loop policy, which are realized-input-independent and compatible with standard Transformer training. To train a Transformer, we propose a triply quantized training procedure for the lifted MDP by quantizing the state space, the space of probability measures, and the action space, and show that any optimal policy for the triply quantized model is near-optimal for the original training problem. Finally, we establish stability and empirical consistency properties of the lifted model by showing that the value function is continuous with respect to the perturbations of the initial empirical measures and convergence of policies as the data size increases. This approach provides a globally optimal and robust alternative to gradient-based training without requiring smoothness or convexity.

Kağan Akman, Naci Saldı, Serdar Yüksel

Department of Mathematics, Bilkent University, Ankara, 06800, Turkey · Department of Mathematics and Statistics, Queen’s University, Kingston K7L 3N6, ON, Canada

Transformer Transformer Training Dynamics Optimal Control

Feb 27, 2026

Optimizer-Induced Low-Dimensional Drift and Transverse Dynamics in Transformer Training

We analyze cumulative parameter trajectories of transformer training under AdamW and identify a dominant low-dimensional drift direction ("backbone") that captures 60--80% of long-horizon displacement from initialization. This direction is highly stable over rolling training windows yet reorients gradually across phases, particularly following objective reweighting. Per-batch gradients exhibit near-noise-floor alignment with the backbone, whereas optimizer-integrated updates align strongly with it, indicating that the structure emerges from accumulated optimizer dynamics rather than instantaneous gradient geometry. Replacing AdamW with SGD-family optimizers eliminates this structure, and reducing

β_2

smoothly degrades backbone dominance and reheating recoverability. Reheating experiments show that transverse probe modes can be transiently re-excited without substantially altering accumulated backbone drift. These results provide a trajectory-level characterization of optimizer-induced geometric structure in transformer training and shift attention from instantaneous gradient properties to cumulative update dynamics.

Yongzhong Xu

Transformer Training Dynamics Neural Network Training Dynamics Transformer Adaptive Gradient Methods Intrinsic Dimensionality

Feb 22, 2026

Incremental Learning of Sparse Attention Patterns in Transformers

This paper studies simple transformers trained on a high-order Markov chain, where the model must incorporate information from multiple past positions, each with different statistical importance. We show that transformers learn the task incrementally, with each stage corresponding to learning how to copy information from a subset of positions via a sparse attention pattern. Notably, the learning dynamics transition from a competitive phase, where all heads focus on the statistically most important positions, to a cooperative phase, where different heads specialize in different patterns. We model these dynamics with simplified differential equations and prove stage-wise convergence of the resulting system. Functionally, these stages correspond to a sequence of increasingly expressive misspecified models, with the full model class reached only at the end. Overall, we give a theoretical account of how structured attention patterns and head specialization emerge in stages without an explicit curriculum, with implications for generalization in sequential tasks.

Oğuz Kaan Yüksel, Rodrigo Alvarez Lucendo, Nicolas Flammarion

TML Lab, EPFL

Self-Attention Transformer Training Dynamics Transformer Attention Sparse Attention Incremental Learning Attention Mechanisms+3

Jan 10, 2026

Architecture Before the Formula: Training-Dependent Computational Semantics in Transformers

An operation's input-output function does not, by itself, determine its computational role within a represented process. That role also depends on the processes producing its inputs and receiving its outputs, a philosophical stance termed deep learning externalism (DLE). We develop a mathematical account of represented computation that distinguishes functional equivalence, retained predecessor distinctions, restricted receiver capability, and context-dependent effective architectures. The resulting equivalences depend on which computational relations a scientific description must preserve. We prove that two injective producer contexts can preserve every predecessor distinction while changing whether fixed local and attention receiver families are equivalent. We investigate these relations in a dimension-labelled PreNorm Transformer and their evolution under training. Residual bypasses restrict complete-state information identifications to two merges per block, while finer branches can change receiver accessibility. Exact gradient-descent models yield covariance-dependent geometric attenuation under learned producer routing and algebraic

(t^{-1/2})

attenuation under Q/K-softmax training, with optimal bounded-receiver recovery errors derived in both cases. Coupled producer-attention dynamics establish how learning jointly reorganizes the presented features and receiver allocation. Finally, present functional or receiver-profile equivalence need not determine future behavior under an optimizer. In an input-conditioned attention model, two accessibility measurements form an exact reduced predictive state, whereas one alone does not.

Luis F. Rosario Freytes

Transformer Training Dynamics Transformer Attention Neural Network Interpretability Neural Network Training Dynamics

Nov 24, 2025

Understanding the Staged Dynamics of Transformers in Learning Latent Structure

Language modeling has shown us that transformers can discover latent structure from context, but the dynamics of how they acquire different components of that structure remain poorly understood, leading to assertions that models just remix training data. In this work, we use the Alchemy benchmark in a controlled setting (Wang et al.,2021) to investigate latent structure learning. We train a small decoder-only transformer on three task variants: 1) inferring missing transitions from partial contextual information, 2) composing simple rules to solve multi-transition sequences, and 3) decomposing complex multi-step examples to infer intermediate transitions. By factorizing each task into interpretable components, we show that the model learns the different latent structure components in discrete stages. We also observe an asymmetry: the model composes fundamental transitions robustly, but struggles to decompose complex examples to discover the atomic transitions. Finally, using causal interventions, we identify layer-specific plasticity windows during which freezing substantially delays or prevents stage completion. These findings provide insight into how a transformer model acquires latent structure, offering a detailed view of how capabilities evolve during training.

Rohan Saha, Farzane Aminmansour, Alona Fyshe

Department of Computing Science University of Alberta Edmonton, Canada · Department of Computing Science and Psychology University of Alberta Edmonton, Canada

Transformer Training Dynamics Decoder-Only Language Models Neural Network Training Dynamics Compositional Generalization

Oct 31, 2024

A Mechanistic Study of Transformers Training Dynamics

Large-scale pretraining of transformers has been central to the success of foundation models. However, the scale of those models limits our understanding of the mechanisms at play during optimization. In this work, we study the training dynamics of transformers in a controlled and interpretable setting. On the sparse modular addition task, we demonstrate that specialized attention circuits, called clustering heads, can be implemented during gradient descent to solve the problem. Our experiments show that such pathways naturally emerge during training. By monitoring the evolution of tokens via a visual sandbox, we uncover a two-stage learning and the occurrences of loss spikes due to the high curvature of normalization layers. Our findings provide several insights into patterns observed in more practical settings, such as the pretraining of large language models.

Ambroise Odonnat, Wassim Bouaziz, Vivien Cabannes

Inria, Univ. Rennes 2 · Mistral AI; work done while at Meta. · FAIR at Meta.

Transformer Interpretability Attention Mechanisms Modular Addition Neural Network Training Dynamics Attention Head Specialization Transformer Training Dynamics