Transformer

Follow this topic Ask about this topic

Momentum

26 papers in the last four weeks, up 44% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 351

All topics

Reachability and asymptotics of Gaussian Transformer dynamics
May 29, 2026Albert Alcalde, Zhengping Ji, Enrique ZuazuaTransformer
†Chair for Dynamics, Control, Machine Learning & Numerics (Alexander von Humboldt Profes- sorship), Department of Mathematics, Friedrich–Alexander-Universität Erlangen–Nürnberg, 91058 Erlangen, Germany. · ‡Departamento de Matem´aticas, Universidad Aut´onoma de Madrid, 28049 Madrid, Spain. · Chair of Computational Mathematics, Fundaci´on Deusto. Av. de las Universidades, 24, 48007 Bilbao, Basque Country, Spain.
Revisiting Padded Transformer Expressivity: Which Architectural Choices Matter and Which Don't
May 28, 2026Anej Svete, William Merrill, Ryan Cotterell +1Transformer Expressivity Transformer
1. ETH Zürich · 2. Allen Institute for AI
Anti Mode-Collapse in Mean-Field Transformer via Auxiliary Variables
May 28, 2026Masaaki Imaizumi, Masanori Koyama, Noboru Isobe +1Transformer Self-Attention
The University of Tokyo, Tokyo, Japan · RIKEN Advanced Intelligene Project, Tokyo, Japan · Kyoto University, Kyoto, Japan
Transformers Provably Learn to Internalize Chain-of-Thought
May 27, 2026Yixiao Huang, Hanlin Zhu, Zixuan Wang +4Transformer CoT Reasoning
UC Berkeley · Princeton University
Measure-to-measure Regression with Transformers
May 27, 2026Matthew Vandergrift, Martha White, Yury Polyanskiy +2Transformer Operator Learning
University of Alberta · Canada CIFAR AI Chair · MIT +1
ClothTransformer: Unified Latent-Space Transformers for Scalable Cloth Simulation
May 27, 2026Yu Zhang, Yidi Shao, Wenqi Ouyang +5Transformer Physics-Based Simulation
1S-Lab, Nanyang Technological University, Singapore · 2Feeling AI · University of Oxford +2
Customer Churn Prediction on Structured Data Using FT-Transformer and Stacking Ensembles
May 26, 2026Joyjit Roy, Samaresh Kumar Singh, Laxmi ShawTransformer Ensemble Learning
Independent Researcher, Austin, TX, USA · Independent Researcher, Leander, TX · Texas A & M University-Victoria, Victoria, TX
Kan Extension Transformers: A Categorical Unification of Attention, Diffusion, and Predict-Detach Self-Conditioning
May 26, 2026Sridhar MahadevanTransformer Autoregressive Language Modeling
Adobe Research San Jose, CA · Academic affiliation: Research Professor, University of Massachusetts, Amherst
More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations
May 26, 2026Mingze Wang, Jinbo Wang, Yikuan Xia +2Transformer FFNs Transformer Expressivity
ByteDance Seed · Peking University
PRISM: Position-encoded Regressive Inverse Spectral Model for Multilayer Thin-Film Design
May 26, 2026Runtian Wang, Renhao Xue, Baige Chen +1Transformer Materials Science
Independent Researcher · Amazon
The Quantization Benefits of Residual-Free Transformers
May 25, 2026Yiping Ji, Mahalakshmi Sabanayagam, Peyman Moghadam +2Transformer Activation Outliers
Australian Institute for Machine Learning, Adelaide University · DATA61, CSIRO
Analogies between Transformer Layers and Power Method
May 25, 2026Chenglong Li, Claudio AltafiniTransformer Interpretability Transformer
Polymorphism Is Rotation: Operational Mechanistic Interpretability from a Two-Layer Transformer to Pythia-70m
May 23, 2026Jordan F. McCannTransformer Interpretability Transformer
Independent Researcher
Momentum Streams for Optimizer-Inspired Transformers
May 23, 2026Jingchu Gai, Nai-Chieh Huang, Jiayun WuDeep Learning Optimization Transformer
Carnegie Mellon University
Complete-muE: Optimal Hyperparameter Transfer and Scaling for MoE Models
May 22, 2026Hongwu Peng, Ohiremen Dibua, Yuanjun Xiong +3Transformer Neural Network Optimization
Adobe Research
Preisach Attention: A Hysteretic Model of Sequential Memory
May 22, 2026Piotr FrydrychTransformer Efficient Attention
The Metrology and Biomedical Engineering Institute · Faculty of Mechatronics, Warsaw University of Technology
How Many Different Outputs Can a Transformer Generate?
May 21, 2026Maxime Meyer, Mario Michelessa, Caroline Chaux +1Transformer Expressivity Transformer
Department of Mathematics, National University of Singapore, Singapore, 117543 · IPAL, IRL2955, Singapore · School of Computing, National University of Singapore, Singapore, 117543 +2
Towards Verifiable Transformers: Solver-Checkable Circuit Explanations
May 21, 2026Neel SomaniNeural Network Verification Transformer
Independent Researcher
Large-Step Training Dynamics of a Two-Factor Linear Transformer Model
May 20, 2026Krishnakumar BalasubramanianTransformer Phase Transitions
Department of Statistics, University of California, Davis
UniT: Unified Geometry Learning with Group Autoregressive Transformer
May 20, 2026Haotian Wang, Yusong Huang, Zhaonian Kuang +4Transformer Depth Estimation
Intelligent Transportation Thrust of the Systems Hub, The Hong Kong University of Science and Technology (GZ), Guangzhou, P.R.China. · The National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University, Xi’an, P.R.China. · Applied Science, Amazon.com, Inc., USA.
HORST: Composing Optimizer Geometries for Sparse Transformer Training
May 20, 2026Tom Jacobs, Rohan Jain, Rebekka BurkholzDeep Learning Optimization Transformer
CISPA Helmholtz Center for Information Security
Markovian Circuit Tracing for Transformer State Dynamic
May 20, 2026Abdullah XTransformer Interpretability Transformer
Project AWARE and Zephara AI
Most Transformer Modifications Still Do Not Transfer at 1-3B: A 2020-2026 Update to Narang et al. (2021) with Downstream Evaluation and a Noise Floor
May 20, 2026Yang Zhao, Jiahao Lu, Bin Huang +2LLM Evaluation Transformer
Tencent
Position: The Turing-Completeness of Autoregressive Transformers Relies Heavily on Context Management
May 19, 2026Guanyu Cui, Zhewei Wei, Kun HeTransformer Expressivity Transformer
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · DEKE Lab, Renmin University of China, Beijing, China
Chessformer: A Unified Architecture for Chess Modeling
May 18, 2026Daniel Monroe, George Eilender, Philip Chalmers +2Transformer Interpretability Game-Playing Agents
University of Toronto · Williams College
The Expressive Power of Low Precision Softmax Transformers with (Summarized) Chain-of-Thought
May 18, 2026Moritz Brösamle, Stephan EcksteinTransformer Expressivity Transformer
Department of Mathematics, University of Tübingen, Germany.
InfoFlow: A Framework for Multi-Layer Transformer Analysis
May 18, 2026Penghao Yu, Haotian Jiang, Zeyu Bao +1Transformer Expressivity Transformer
Department of Mathematics National University of Singapore · Institute for Functional Intelligent Materials National University of Singapore · National University of Singapore +1
Training Infinitely Deep and Wide Transformers
May 17, 2026Raphaël Barboni, Maarten V. de Hoop, Takashi Furuya +1Transformer Wasserstein Gradient Flows
Bocconi University · Rice University · Doshisha University, RIKEN AIP +1
A Two-Parameter Weibull Framework for Diagnosing Transformer Weight Distributions
May 17, 2026Tiexin DingTransformer Weight-Space Analysis
Independent Researcher
OPTNet: Ordering Point Transformer Network for Post-disaster 3D Semantic Segmentation
May 16, 2026Nhut Le, Ehsan Karimi, Maryam RahnemoonfarDisaster Damage Assessment Transformer
Computer Science and Engineering, Lehigh University, Bethlehem PA 18015, US · Civil and Environmental Engineering, Lehigh University, Bethlehem PA 18015, US

May 29, 2026, cs.LGJ/K move · Enter open · S save

Reachability and asymptotics of Gaussian Transformer dynamics

Albert Alcalde, Zhengping Ji, Enrique Zuazua

†Chair for Dynamics, Control, Machine Learning & Numerics (Alexander von Humboldt Profes- sorship), Department of Mathematics, Friedrich–Alexander-Universität Erlangen–Nürnberg, 91058 Erlangen, Germany. · ‡Departamento de Matem´aticas, Universidad Aut´onoma de Madrid, 28049 Madrid, Spain. · Chair of Computational Mathematics, Fundaci´on Deusto. Av. de las Universidades, 24, 48007 Bilbao, Basque Country, Spain.

Transformer

Open paper AskPDF arXiv

We formulate data propagation through the Transformer, the machine learning architecture powering large language models, as a nonlinear control system on the space of probability measures. For the mean-field Transformer model with self-attention and affine feed-forward layers, we prove that Gaussian distributions remain exactly Gaussian along the induced flow. This invariance reduces the infinite-dimensional measure dynamics to a finite-dimensional bilinear control system governing the evolution of the mean and covariance, reformulates the expressive capacity of Transformers as a reachability problem for prescribed Gaussian moments, and reveals a novel connection with Riccati-type equations from classical filtering and control. For time-varying controls, we prove exact finite-time reachability of any target Gaussian distribution whose covariance matrix has the same rank as the initial one, this rank constraint being an intrinsic invariant of the dynamics. For time-invariant parameters, we derive explicit spectral conditions leading either to asymptotic stability toward positive-definite equilibria or to finite-time blow-up of the covariance. Numerical experiments complement the theory by showing that practical Transformers with Gaussian inputs remain close to moment-matched Gaussian distributions through early and intermediate layers, while Transformers with prescribed attention matrices reproduce the predicted covariance regimes: bounded evolution in stabilizing configurations and blow-up in destabilizing ones.

Momentum

Related topics

Latest papers 351