Transformer Training Dynamics

Latest papers

41 papers18 papers in the last four weeks, with none the four weeks before. 0.1% of all new papers.

Theoretical Analysis of Muon on Transformers for Next-Token Prediction
Nov 3, 2026Shengwei Wu, Yixuan Yang, Gao Huang +1Transformer Training Dynamics Muon Optimizer
School of Mathematical Sciences, Zhejiang University, Hangzhou, Zhejiang, China · Department of Mathematics, National University of Singapore, Singapore
Meta-memorization and memorization scaling laws in transformers
Nov 1, 2026Alex Nguyen, Kenneth A. Norman, Gautam ReddyNeural Network Memorization Transformer Training Dynamics
Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA · Princeton Neuroscience Institute, Department of Psychology, Princeton University, Princeton, NJ, USA · Department of Physics, Princeton University, Princeton, NJ, USA
Solvable dynamics model of in-context learning
Oct 31, 2026Ruoran Bai, Haiping HuangIn-Context Learning Transformer Training Dynamics
PMI Lab, School of Physics, Sun Yat-sen University, Guangzhou 510275, People’s Republic of China · Guangdong Provincial Key Laboratory of Magnetoelectric Physics and Devices, Sun Yat-sen University, Guangzhou 510275, People’s Republic of China
Closed Gates Leak: Decomposing Adjacent-Layer Cancellation in Gated Language Models
Oct 28, 2026Harvey Dam, Ganesh GopalakrishnanTransformer FFNs Multilayer Perceptrons
Kahlert School of Computing, University of Utah, Salt Lake City, USA.
Deeper Backpropagation or More Updates? Training Looped Language Models Across Budgets
Oct 28, 2026Jiarun Shen, Xiao LiLLM Training Efficient Language Model Training
The Chinese University of Hong Kong, Shenzhen
Replaying the Same Gradients at a Different $\varepsilon$ : A Registered Test of the Fixed-Trajectory Account of Adam's $\varepsilon$ for One Rare Row
Oct 28, 2026Lngel KyoAdaptive Gradient Methods Gradient Descent Dynamics
A2O Labs
Sometimes less is more: Investigating inductive biases of MLPs and Transformers on noisy k-parity features
Oct 27, 2026Alasdair ParenTransformer Multilayer Perceptrons
Chalmers University of Technology
Does LLM Pre-Training Typically Occur at the Edge of Stability?
Oct 27, 2026Yuhang Cai, Haodong Wen, Haofeng Huang +3Edge of Stability Neural Network Training Dynamics
UC Berkeley · Tsinghua University · ByteDance Seed
What a Weight Update Buys: Leverage Across Training Stages
Oct 27, 2026Shikhar ShiromaniTransformer Training Dynamics Language Model Post-Training
Subspace Crystallization in Grokking: Rank Compression Tracks the Phase Transition
Oct 26, 2026Shlok Mehendale, Paras ChopraGrokking Neural Network Training Dynamics
Lossfunk
Transformers with CoT Discover Heavy-Ball Acceleration Through Training
Oct 22, 2026Yuxuan Cai, Yuan CaoCoT Reasoning Neural Algorithmic Reasoning
School of Computing & Data Science, The University of Hong Kong
Reader--Writer Coupling in Transformer MLPs
Oct 21, 2026Giyeong Oh, Sunghee Ahn, Youngjae YuTransformer FFNs Transformer Interpretability
Seoul National University
How Much Gradient Does Truncated Backprop Lose in Looped Transformers?
Oct 20, 2026Aur Shalev MerinTruncated Backpropagation Through Time Backpropagation
Independent researcher
Transformers Learn Two Competing Sorting Algorithms, and Only One Length-Generalizes
Oct 18, 2026Khashayar Gatmiry, Nathan W. Henry, Chloe H. Su +5Length Generalization Transformer Training Dynamics
UC Berkeley & Apple · UC Berkeley · Kempner Institute, Harvard University +5
T-MUON: Tensorizing Muon via t-SVD for Large Language Model Pre-training
Oct 18, 2026Dongyang Li, Mingsong Yan, Zheng Zhang +1Muon Optimizer Language Model Pretraining
University of California, Santa Barbara Santa Barbara, CA 93106, USA
A Conditional No-Hair Theorem for Transformers: How Language Models Forget Their Prompt
Oct 16, 2026Shuyang XiangTransformer Attention Transformer Training Dynamics
Independent researcher
Training Dynamics of Transformers on Unbounded Inputs: Attention Learning Can Lag Output Layer Learning
Oct 14, 2026Yaomengxi Han, Debarghya GhoshdastidarTransformer Training Dynamics Neural Network Training Dynamics
Technical University of Munich
Asymptotic Behavior of Transformers
Oct 14, 2026Katy Blumer, Da-Cheng Juan, Jon Kleinberg +3Transformer Training Dynamics Transformer
Google Research
Same Function, Different Trajectory: No Standard Optimizer Is Equivariant to Transformer Reparameterization
Oct 13, 2026Bum Jun KimDeep Learning Optimization Transformer
The University of Tokyo
Hidden Boundary Motion in Transformer Optimization: Function-Space Orthogonalization of Affine Weight and Bias Updates
Jul 24, 2026Zhang Gongyue, Sheng Yixuan, Liu donghan +3Transformer Neural Network Optimization
arXiv:2607.22927v1
Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks
Jul 13, 2026Tiberiu Musat, Tiago Pimentel, Nicolas Zucchet +1Transformer Training Dynamics In-Context Learning
ETH Zurich · Stanford University
Loop- $μ$ P: Residual Scaling and Hyperparameter Transfer in Looped Transformers
Jun 16, 2026Shaowen Wang, Bingrui Li, Tansheng Zhu +4Residual Learning Hyperparameter Transfer
Tsinghua University · ByteDance Seed · M-A-P
Taming Curvature: Architecture Warm-Up for Stable Transformer Training
Jun 15, 2026Sameera Ramasinghe, Ajanthan Thalaiyasingam, Hadi Mohaghegh Dolatabadi +6Edge of Stability Transformer
Pluralis Research
Momentum Streams for Optimizer-Inspired Transformers
May 23, 2026Jingchu Gai, Nai-Chieh Huang, Jiayun WuDeep Learning Optimization Transformer
Carnegie Mellon University
Large-Step Training Dynamics of a Two-Factor Linear Transformer Model
May 20, 2026Krishnakumar BalasubramanianTransformer Phase Transitions
Department of Statistics, University of California, Davis
Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics
May 19, 2026Lucky VermaWeight Decay Neural Network Memorization
Independent Researcher
Multi-Headed Transformer Architectures as Time-dependent Wasserstein Gradient Flows
May 15, 2026Alex Massucco, Leonardo Del Grande, Marcello Carioni +2Transformer Transformer Training Dynamics
Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, UK · Department of Mathematics, University of Twente, Enschede, Netherlands
Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers
May 8, 2026Noboru Isobe, Daisuke Inoue, Masaaki ImaizumiTransformer Transformer Training Dynamics
RIKEN Center for Advanced Intelligence Project · Imperial College London · The University of Tokyo +1
Critical Windows of Complexity Control: When Transformers Decide to Reason or Memorize
May 6, 2026Sarwan AliDeep Learning Optimization Weight Decay
Columbia University, Irving Medical Center, NY, USA
Trust, but Verify: Peeling Low-Bit Transformer Networks for Training Monitoring
May 4, 2026Arian Eamaz, Farhang Yeganegi, Mojtaba SoltanalianDeep Learning Optimization Neural Network Training Dynamics
Department of Electrical and Computer Engineering University of Illinois Chicago Chicago, IL, USA

Nov 3, 2026J/K move · Enter open · S save

Theoretical Analysis of Muon on Transformers for Next-Token Prediction

Shengwei Wu, Yixuan Yang, Gao Huang, Song Li

School of Mathematical Sciences, Zhejiang University, Hangzhou, Zhejiang, China · Department of Mathematics, National University of Singapore, Singapore

Transformer Training Dynamics Muon Optimizer Neural Network Optimization Next-Token Prediction Transformer Attention

Open paper AskPDF arXiv

Muon has recently shown strong empirical performance in training Transformer-based large language models (LLMs), yet its optimization dynamics remain poorly understood from a theoretical perspective. In this paper, we analyze the training dynamics of Muon for next-token prediction (NTP) in Transformers. The training procedure consists of two phases: the preprocessing phase learns the output-value component, while the main phase learns the key-query component. For the preprocessing phase, we characterize the optimization trajectory exactly and show that the normalized spectral margin is optimal after each update. For the main phase, we establish nonasymptotic convergence rates for the training loss and the normalized margin gap under softmax attention. These results suggest that Muon's update rule can help models efficiently learn which tokens matter for prediction, offering a possible explanation for its strong performance in large language model training. The learned weight structure also yields a generalization guarantee for unseen contexts. Furthermore, we validate our theoretical insights with experiments on next-token prediction tasks.