Transformer Training Dynamics

Follow this topic Ask about this topic

Latest papers

24 papers0 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Training Dynamics of Transformers on Unbounded Inputs: Attention Learning Can Lag Output Layer Learning
Oct 14, 2026Yaomengxi Han, Debarghya GhoshdastidarTransformer Training Dynamics Neural Network Training Dynamics
Technical University of Munich
Asymptotic Behavior of Transformers
Oct 14, 2026Katy Blumer, Da-Cheng Juan, Jon Kleinberg +3Transformer Training Dynamics Transformer
Google Research
Same Function, Different Trajectory: No Standard Optimizer Is Equivariant to Transformer Reparameterization
Oct 13, 2026Bum Jun KimDeep Learning Optimization Transformer
The University of Tokyo
Hidden Boundary Motion in Transformer Optimization: Function-Space Orthogonalization of Affine Weight and Bias Updates
Jul 24, 2026Zhang Gongyue, Sheng Yixuan, Liu donghan +3Transformer Neural Network Optimization
arXiv:2607.22927v1
Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks
Jul 13, 2026Tiberiu Musat, Tiago Pimentel, Nicolas Zucchet +1Transformer Training Dynamics In-Context Learning
ETH Zurich · Stanford University
On the Residual Scaling of Looped Transformers: Stability and Transferability
Jun 16, 2026Shaowen Wang, Bingrui Li, Ge Zhang +3Residual Learning Hyperparameter Transfer
Tsinghua University · ByteDance Seed · M-A-P
Taming Curvature: Architecture Warm-Up for Stable Transformer Training
Jun 15, 2026Sameera Ramasinghe, Ajanthan Thalaiyasingam, Hadi Mohaghegh Dolatabadi +6Edge of Stability Transformer
Pluralis Research
Momentum Streams for Optimizer-Inspired Transformers
May 23, 2026Jingchu Gai, Nai-Chieh Huang, Jiayun WuDeep Learning Optimization Transformer
Carnegie Mellon University
Large-Step Training Dynamics of a Two-Factor Linear Transformer Model
May 20, 2026Krishnakumar BalasubramanianTransformer Phase Transitions
Department of Statistics, University of California, Davis
Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics
May 19, 2026Lucky VermaWeight Decay Neural Network Memorization
Independent Researcher
Multi-Headed Transformer Architectures as Time-dependent Wasserstein Gradient Flows
May 15, 2026Alex Massucco, Leonardo Del Grande, Marcello Carioni +2Transformer Transformer Training Dynamics
Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, UK · Department of Mathematics, University of Twente, Enschede, Netherlands
Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers
May 8, 2026Noboru Isobe, Daisuke Inoue, Masaaki ImaizumiTransformer Transformer Training Dynamics
RIKEN Center for Advanced Intelligence Project · Imperial College London · The University of Tokyo +1
Critical Windows of Complexity Control: When Transformers Decide to Reason or Memorize
May 6, 2026Sarwan AliDeep Learning Optimization Weight Decay
Columbia University, Irving Medical Center, NY, USA
Trust, but Verify: Peeling Low-Bit Transformer Networks for Training Monitoring
May 4, 2026Arian Eamaz, Farhang Yeganegi, Mojtaba SoltanalianDeep Learning Optimization Neural Network Training Dynamics
Department of Electrical and Computer Engineering University of Illinois Chicago Chicago, IL, USA
Focus and Dilution: The Multi-stage Learning Process of Attention
May 2, 2026Zheng-An Chen, Pengxiao Lin, Zhi-Qin John Xu +1Self-Attention Transformer Attention
School of Mathematical Sciences, Shanghai Jiao Tong University. · Institute of Natural Sciences, Shanghai Jiao Tong University. · MOE-LSC, Shanghai Jiao Tong University. +2
Distinct mechanisms underlying in-context learning in transformers
Apr 14, 2026Cole Gibson, Wenping Cui, Gautam ReddyIn-Context Learning Transformer Training Dynamics
Lewis-Sigler Institute for Integrative Genomics, Princeton University, Princeton, NJ 08544, USA · Department of Physics, Princeton University, Princeton, NJ 08544, USA
The Spectral Lifecycle of Transformer Training: Transient Compression Waves, Persistent Spectral Gradients, and the Q/K--V Asymmetry
Apr 3, 2026Yi LiuTransformer Training Dynamics Transformer
Spectral Edge Dynamics of Training Trajectories: Signal--Noise Geometry Across Scales
Mar 14, 2026Yongzhong XuTransformer Training Dynamics Neural Network Training Dynamics
An Optimal Control Approach To Transformer Training
Mar 10, 2026Kağan Akman, Naci Saldı, Serdar YükselTransformer Transformer Training Dynamics
Department of Mathematics, Bilkent University, Ankara, 06800, Turkey · Department of Mathematics and Statistics, Queen’s University, Kingston K7L 3N6, ON, Canada
Optimizer-Induced Low-Dimensional Drift and Transverse Dynamics in Transformer Training
Feb 27, 2026Yongzhong XuTransformer Training Dynamics Neural Network Training Dynamics
Incremental Learning of Sparse Attention Patterns in Transformers
Feb 22, 2026Oğuz Kaan Yüksel, Rodrigo Alvarez Lucendo, Nicolas FlammarionSelf-Attention Transformer Training Dynamics
TML Lab, EPFL
Architecture Before the Formula: Training-Dependent Computational Semantics in Transformers
Jan 10, 2026Luis F. Rosario FreytesTransformer Training Dynamics Transformer Attention
Understanding the Staged Dynamics of Transformers in Learning Latent Structure
Nov 24, 2025Rohan Saha, Farzane Aminmansour, Alona FysheTransformer Training Dynamics Decoder-Only Language Models
Department of Computing Science University of Alberta Edmonton, Canada · Department of Computing Science and Psychology University of Alberta Edmonton, Canada
A Mechanistic Study of Transformers Training Dynamics
Oct 31, 2024Ambroise Odonnat, Wassim Bouaziz, Vivien CabannesTransformer Interpretability Attention Mechanisms
Inria, Univ. Rennes 2 · Mistral AI; work done while at Meta. · FAIR at Meta.

Oct 14, 2026J/K move · Enter open · S save

Training Dynamics of Transformers on Unbounded Inputs: Attention Learning Can Lag Output Layer Learning

Yaomengxi Han, Debarghya Ghoshdastidar

Technical University of Munich

Transformer Training Dynamics Neural Network Training Dynamics Transformer Attention Transformer FFNs

Open paper AskPDF arXiv

The success of transformers is attributed to several factors, including their capacity to express formal languages over variable-length inputs, and ability of the attention mechanism to focus on relevant input features. However, there is limited understanding of the training dynamics of shallow transformers, specifically how to decouple the learning of the attention and the output (feedforward) layer. A natural extension of the theory of one-step gradient feature learning in feedforward networks suggests that one could first learn the attention weights with a single gradient step, and then train the output layer. We show that this fails for a shallow transformer trained to recognize a star-free language from unbounded-length input. We prove that the transformer can be trained to recognize the language with a three-phase gradient descent: the output layer is updated first, followed by an attention update, and the output layer is fine-tuned in the third phase. In contrast, the language cannot be learned by training only the attention or output layer, or by a single attention-layer update followed by finetuning the output layer. We empirically show that ``attention learning lagging output layer learning" is beneficial in other language recognition, regression, and in-context learning tasks as well.