Transformer Training Dynamics

Follow this topic Ask about this topic

Latest papers

24 papers5 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

T-MUON: Tensorizing Muon via t-SVD for Large Language Model Pre-training
Oct 18, 2026Dongyang Li, Mingsong Yan, Zheng Zhang +1Muon Optimizer Language Model Pretraining
A Conditional No-Hair Theorem for Transformers: How Language Models Forget Their Prompt
Oct 16, 2026Shuyang XiangTransformer Attention Transformer Training Dynamics
Training Dynamics of Transformers on Unbounded Inputs: Attention Learning Can Lag Output Layer Learning
Oct 14, 2026Yaomengxi Han, Debarghya GhoshdastidarTransformer Training Dynamics Neural Network Training Dynamics
Technical University of Munich
Asymptotic Behavior of Transformers
Oct 14, 2026Katy Blumer, Da-Cheng Juan, Jon Kleinberg +3Transformer Training Dynamics Transformer
Google Research
Same Function, Different Trajectory: No Standard Optimizer Is Equivariant to Transformer Reparameterization
Oct 13, 2026Bum Jun KimDeep Learning Optimization Transformer
The University of Tokyo
Hidden Boundary Motion in Transformer Optimization: Function-Space Orthogonalization of Affine Weight and Bias Updates
Jul 24, 2026Zhang Gongyue, Sheng Yixuan, Liu donghan +3Transformer Neural Network Optimization
arXiv:2607.22927v1
Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks
Jul 13, 2026Tiberiu Musat, Tiago Pimentel, Nicolas Zucchet +1Transformer Training Dynamics In-Context Learning
ETH Zurich · Stanford University
Loop- $μ$ P: Residual Scaling and Hyperparameter Transfer in Looped Transformers
Jun 16, 2026Shaowen Wang, Bingrui Li, Tansheng Zhu +4Residual Learning Hyperparameter Transfer
Tsinghua University · ByteDance Seed · M-A-P
Taming Curvature: Architecture Warm-Up for Stable Transformer Training
Jun 15, 2026Sameera Ramasinghe, Ajanthan Thalaiyasingam, Hadi Mohaghegh Dolatabadi +6Edge of Stability Transformer
Pluralis Research
Momentum Streams for Optimizer-Inspired Transformers
May 23, 2026Jingchu Gai, Nai-Chieh Huang, Jiayun WuDeep Learning Optimization Transformer
Carnegie Mellon University
Large-Step Training Dynamics of a Two-Factor Linear Transformer Model
May 20, 2026Krishnakumar BalasubramanianTransformer Phase Transitions
Department of Statistics, University of California, Davis
Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics
May 19, 2026Lucky VermaWeight Decay Neural Network Memorization
Independent Researcher
Multi-Headed Transformer Architectures as Time-dependent Wasserstein Gradient Flows
May 15, 2026Alex Massucco, Leonardo Del Grande, Marcello Carioni +2Transformer Transformer Training Dynamics
Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, UK · Department of Mathematics, University of Twente, Enschede, Netherlands
Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers
May 8, 2026Noboru Isobe, Daisuke Inoue, Masaaki ImaizumiTransformer Transformer Training Dynamics
RIKEN Center for Advanced Intelligence Project · Imperial College London · The University of Tokyo +1
Critical Windows of Complexity Control: When Transformers Decide to Reason or Memorize
May 6, 2026Sarwan AliDeep Learning Optimization Weight Decay
Columbia University, Irving Medical Center, NY, USA
Trust, but Verify: Peeling Low-Bit Transformer Networks for Training Monitoring
May 4, 2026Arian Eamaz, Farhang Yeganegi, Mojtaba SoltanalianDeep Learning Optimization Neural Network Training Dynamics
Department of Electrical and Computer Engineering University of Illinois Chicago Chicago, IL, USA
Focus and Dilution: The Multi-stage Learning Process of Attention
May 2, 2026Zheng-An Chen, Pengxiao Lin, Zhi-Qin John Xu +1Self-Attention Transformer Attention
School of Mathematical Sciences, Shanghai Jiao Tong University. · Institute of Natural Sciences, Shanghai Jiao Tong University. · MOE-LSC, Shanghai Jiao Tong University. +2
Distinct mechanisms underlying in-context learning in transformers
Apr 14, 2026Cole Gibson, Wenping Cui, Gautam ReddyIn-Context Learning Transformer Training Dynamics
Lewis-Sigler Institute for Integrative Genomics, Princeton University, Princeton, NJ 08544, USA · Department of Physics, Princeton University, Princeton, NJ 08544, USA
The Spectral Lifecycle of Transformer Training: Transient Compression Waves, Persistent Spectral Gradients, and the Q/K--V Asymmetry
Apr 3, 2026Yi LiuTransformer Training Dynamics Transformer
Spectral Edge Dynamics of Training Trajectories: Signal--Noise Geometry Across Scales
Mar 14, 2026Yongzhong XuTransformer Training Dynamics Neural Network Training Dynamics
An Optimal Control Approach To Transformer Training
Mar 10, 2026Kağan Akman, Naci Saldı, Serdar YükselTransformer Transformer Training Dynamics
Department of Mathematics, Bilkent University, Ankara, 06800, Turkey · Department of Mathematics and Statistics, Queen’s University, Kingston K7L 3N6, ON, Canada
Optimizer-Induced Low-Dimensional Drift and Transverse Dynamics in Transformer Training
Feb 27, 2026Yongzhong XuTransformer Training Dynamics Neural Network Training Dynamics
Incremental Learning of Sparse Attention Patterns in Transformers
Feb 22, 2026Oğuz Kaan Yüksel, Rodrigo Alvarez Lucendo, Nicolas FlammarionSelf-Attention Transformer Training Dynamics
TML Lab, EPFL
Architecture Before the Formula: Training-Dependent Computational Semantics in Transformers
Jan 10, 2026Luis F. Rosario FreytesTransformer Training Dynamics Transformer Attention
Understanding the Staged Dynamics of Transformers in Learning Latent Structure
Nov 24, 2025Rohan Saha, Farzane Aminmansour, Alona FysheTransformer Training Dynamics Decoder-Only Language Models
Department of Computing Science University of Alberta Edmonton, Canada · Department of Computing Science and Psychology University of Alberta Edmonton, Canada
A Mechanistic Study of Transformers Training Dynamics
Oct 31, 2024Ambroise Odonnat, Wassim Bouaziz, Vivien CabannesTransformer Interpretability Attention Mechanisms
Inria, Univ. Rennes 2 · Mistral AI; work done while at Meta. · FAIR at Meta.

Oct 18, 2026J/K move · Enter open · S save

T-MUON: Tensorizing Muon via t-SVD for Large Language Model Pre-training

Dongyang Li, Mingsong Yan, Zheng Zhang, Sui Tang

Muon Optimizer Language Model Pretraining Transformer Training Dynamics

Open paper AskPDF arXiv

Muon has recently emerged as a strong optimizer for large language model pretraining by exploiting matrix structure through polar orthogonalization. However, it processes each parameter matrix independently, whereas recent tensor-aware extensions of Muon jointly update groups of eligible matrix parameters through matrix unfoldings. In contrast, we propose T-MUON, a t-SVD-based generalization that groups structurally related Transformer gradients into third-order tensors and defines the corresponding Muon-style update directly through the

M

-transform. We show that the exact T-MUON update is a steepest-descent direction under the

\star_M

-spectral norm and establish a convergence guarantee. Across GPT-2-style and LLaMA-style models from 51M to 1.3B parameters trained on FineWeb and FineWeb-Edu, head-wise T-MUON with the identity transform consistently achieves lower validation perplexity than AdamW, Muon, and variants using DFT, DCT, or DST transforms. These gains persist in additional experiments with longer context lengths and larger token budgets, although the preferred tensorization mode varies across settings.