Transformer Training Dynamics

Latest papers

41 papers18 papers in the last four weeks, with none the four weeks before. 0.1% of all new papers.

Focus and Dilution: The Multi-stage Learning Process of Attention
May 2, 2026Zheng-An Chen, Pengxiao Lin, Zhi-Qin John Xu +1Self-Attention Transformer Attention
School of Mathematical Sciences, Shanghai Jiao Tong University. · Institute of Natural Sciences, Shanghai Jiao Tong University. · MOE-LSC, Shanghai Jiao Tong University. +2
Distinct mechanisms underlying in-context learning in transformers
Apr 14, 2026Cole Gibson, Wenping Cui, Gautam ReddyIn-Context Learning Transformer Training Dynamics
Lewis-Sigler Institute for Integrative Genomics, Princeton University, Princeton, NJ 08544, USA · Department of Physics, Princeton University, Princeton, NJ 08544, USA
The Spectral Lifecycle of Transformer Training: Transient Compression Waves, Persistent Spectral Gradients, and the Q/K--V Asymmetry
Apr 3, 2026Yi LiuTransformer Training Dynamics Transformer
Spectral Edge Dynamics of Training Trajectories: Signal--Noise Geometry Across Scales
Mar 14, 2026Yongzhong XuTransformer Training Dynamics Neural Network Training Dynamics
An Optimal Control Approach To Transformer Training
Mar 10, 2026Kağan Akman, Naci Saldi, Serdar YükselTransformer Transformer Training Dynamics
Department of Mathematics, Bilkent University, Ankara, 06800, Turkey · Department of Mathematics and Statistics, Queen’s University, Kingston K7L 3N6, ON, Canada
Optimizer-Induced Low-Dimensional Drift and Transverse Dynamics in Transformer Training
Feb 27, 2026Yongzhong XuTransformer Training Dynamics Neural Network Training Dynamics
Incremental Learning of Sparse Attention Patterns in Transformers
Feb 22, 2026Oğuz Kaan Yüksel, Rodrigo Alvarez Lucendo, Nicolas FlammarionSelf-Attention Transformer Training Dynamics
TML Lab, EPFL
Outcome-Based RL Provably Leads Transformers to Reason, but Only With the Right Data
Jan 21, 2026Yuval Ran-Milo, Yotam Alexander, Shahar Mendel +1RL for Language Model Reasoning Transformer Training Dynamics
Tel Aviv University
Architecture Before the Formula: Training-Dependent Computational Semantics in Transformers
Jan 10, 2026Luis F. Rosario FreytesTransformer Training Dynamics Transformer Attention
Understanding the Staged Dynamics of Transformers in Learning Latent Structure
Nov 24, 2025Rohan Saha, Farzane Aminmansour, Alona FysheTransformer Training Dynamics Decoder-Only Language Models
Department of Computing Science, University of Alberta, Edmonton, Canada · Department of Computing Science and Psychology, University of Alberta, Edmonton, Canada
A Mechanistic Study of Transformers Training Dynamics
Oct 31, 2024Ambroise Odonnat, Wassim Bouaziz, Vivien CabannesTransformer Interpretability Attention Mechanisms
Inria, Univ. Rennes 2 · Mistral AI; work done while at Meta. · FAIR at Meta.

May 2, 2026J/K move · Enter open · S save

Focus and Dilution: The Multi-stage Learning Process of Attention

Zheng-An Chen, Pengxiao Lin, Zhi-Qin John Xu, Tao Luo

School of Mathematical Sciences, Shanghai Jiao Tong University. · Institute of Natural Sciences, Shanghai Jiao Tong University. · MOE-LSC, Shanghai Jiao Tong University.+2

Self-Attention Transformer Attention Neural Network Training Dynamics Transformer Training Dynamics

Open paper AskPDF arXiv

Transformer-based models have achieved remarkable success across a wide range of domains, yet our understanding of their training dynamics remains limited. In this work, we identify a recurrent focus-dilution cycle in attention learning and provide a rigorous explanation in a one-layer Transformer setting for Markovian data via gradient-flow analysis. Using stage-wise linearization around critical points, we show that a single focus-dilution cycle can be decomposed into a sequence of distinct stages. First, embedding and projection rapidly condense to a rank-one structure, while attention parameters remain effectively frozen. Then, the attention parameters begin to increase, inducing a frequency-driven focus toward high-frequency tokens. As attention continues to evolve, it generates next-order perturbations in embeddings, leading to a mass-redistribution mechanism that progressively dilutes this focus. Finally, small asymmetries among low-frequency tokens lift a degenerate critical point, opening new embedding directions and initiating the next cycle. Experiments on synthetic Markovian data as well as WikiText and TinyStories corroborate the predicted stages and cyclical dynamics.