Transformer

Follow this topic Ask about this topic

Momentum

26 papers in the last four weeks, up 44% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 351

All topics

Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers
May 8, 2026Noboru Isobe, Daisuke Inoue, Masaaki ImaizumiTransformer Clustering
1RIKEN Center for Advanced Intelligence Project · 2Imperial College London · 3The University of Tokyo +1
Encoder-Decoder Transformers: Logical Characterizations and Periodicity
May 8, 2026Veeti Ahvonen, Damian Heiman, Antti Kuusisto +2Transformer Expressivity Transformer
Mathematics Research Centre, Tampere University, Tampere, Finland
Revisiting Transformer Layer Parameterization Through Causal Energy Minimization
May 8, 2026Jin Xu, Camille Couturier, Victor Rühle +2Transformer Energy-Based Models
Microsoft · Microsoft Research Cambridge
Dynamic Mode Decomposition along Depth in Vision Transformers
May 8, 2026Nishant Suresh Aswani, Saif Eddin JabariTransformer Vision Transformer
NYU Tandon & NYU Abu Dhabi
Approximation Error Upper and Lower Bounds for Hölder Class with Transformers
May 8, 2026Xin He, Yuling Jiao, Xiliang Lu +1Transformer Neural Network Approximation Theory
School of Mathematics and Statistics, Wuhan University, Wuhan, China · School of Artificial Intelligence, Wuhan University, Wuhan, China · Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, China
Spectrum-Adaptive Generalization Bounds for Trained Deep Transformers
May 8, 2026Mana Sakai, Masaaki ImaizumiTransformer Generalization Bounds
The University of Tokyo · RIKEN Center for Advanced Intelligence Project · Kyoto University
Convergence and Emergence of In-Context Reinforcement Learning with Chain of Thought
May 8, 2026Zixuan Xie, Xinyu Liu, Rohan Chandra +1Reinforcement Learning Transformer
University of Virginia
When Symbol Names Should Not Matter: A Logistic Theory of Fresh-Symbol Classification
May 8, 2026Wenjie Guan, Jelena BradicTransformer Classification
Department of Statistics and Data Science, Cornell University
A Linear-Transformer Hybrid for SNP-Based Genotype-to-Phenotype Prediction in Grapevine
May 7, 2026Yibin Wang, Murukarthick Jayakodi, Silvas Kirubakaran +2Transformer
Department of Biological and Agricultural Engineering, Texas A&M AgriLife Research, Texas A&M University System, Dallas, TX, 75252, USA · Department of Soil and Crop Sciences, Texas A&M AgriLife Research, Texas A&M University System, Dallas, TX, 75252, USA · USDA-ARS, Grape Genetics Research Unit, 630 West North Street, Geneva, New York 14456, USA
Beyond Autoregressive RTG: Conditioning via Injection Outside Sequential Modeling in Decision Transformer
May 7, 2026Yongyi Wang, Hanyu Liu, Lingfeng Li +6Efficient Transformer Inference Reinforcement Learning
School of Computer Science Peking University Beijing, China 100871
The E $Δ$ -MHC-Geo Transformer: Adaptive Geodesic Operations with Guaranteed Orthogonality
May 7, 2026Arash ShahmansooriTransformer Residual Learning
Independent Researcher
Large Vision-Language Models Get Lost in Attention
May 7, 2026Gongli Xi, Ye Tian, Mengyu Yang +5Transformer FFNs Vision-Language Models
School of Cyberspace Security, Beijing University of Posts and Telecommunications, Beijing, China · State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China · School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications, Beijing, China +3
Average Attention Transformers and Arithmetic Circuits
May 6, 2026Lena Ehrmuth, Laura StriekerTransformer Expressivity Transformer
Institute for Theoretical Computer Science, Leibniz University Hanover, Hanover, Germany
FLUID: Continuous-Time Hyperconnected Sparse Transformer for Sink-Free Learning
May 6, 2026Waleed Razzaq, Yun-Bo ZhaoTransformer Transformer Attention
School of Automation University of Science and Technology China Hefei, Anhui
Domain-Adaptive Arrhythmia Classification Using a Hybrid Transformer on Wearable Heart Signals
May 5, 2026Maedeh H. Toosi, Siamak MohammadiTransformer Domain Adaptation
School of Electrical and Computer Engineering, University of Tehran, Tehran, Iran
Feature-Augmented Transformers for Robust AI-Text Detection Across Domains and Generators
May 5, 2026Mohamed Mady, Johannes Reschke, Björn SchullerAI-Generated Text Detection Transformer
Chair of Health Informatics, Technical University of Munich (TUM), Germany · OTH Regensburg, Germany · GLAM – Group on Language, Audio & Music, Imperial College London, United Kingdom
Transformers with Selective Access to Early Representations
May 5, 2026Skye Gunasekaran, Téa Wright, Rui-Jie Zhu +1Transformer Residual Learning
UC Santa Cruz · UC Berkeley
A Systematic Benchmark of Machine Transliteration Models for the Tajik-Farsi Language Pair: A Comparative Study from Rule-Based to Transformer Architectures
May 4, 2026Mullosharaf K. ArabovMultilingual Language Models Transformer
Institute of Computational Mathematics and Information Technologies Kazan Federal University Kazan, Russia
Projection-Free Transformers via Gaussian Kernel Attention
May 4, 2026Debarshi Kundu, Archisman Ghosh, Swaroop Ghosh +1Transformer Self-Attention
The Pennsylvania State University, University Park, USA
Linear-Time Global Visual Modeling without Explicit Attention
May 3, 2026Ruize He, Dongchen Han, Gao HuangTransformer Transformer Attention
Tsinghua University
Characterizing the Expressivity of Local Attention in Transformers
May 1, 2026Jiaoda Li, Ryan CotterellTransformer Expressivity Transformer
E $^2$ DT: Efficient and Effective Decision Transformer with Experience-Aware Sampling for Robotic Manipulation
Apr 30, 2026Kaiyan Zhao, Borong Zhang, Yiming Wang +4Long-Horizon Robotic Manipulation Transformer
School of Computer Science, Wuhan University, Wuhan 430072, China · State Key Lab of Internet of Things for Smart City and Department of Computer Information Science, University of Macau, Macao 999078, China
Hierarchical Fault Detection and Diagnosis for Transformer Architectures
Apr 30, 2026Sigma Jahan, Saurabh Singh Rajput, Tushar Sharma +1Software Engineering Automated Software Testing
Dalhousie University, Halifax, Canada
Sampling two-dimensional spin systems with transformers
Apr 30, 2026Piotr Białas, Piotr Korcyl, Tomasz Stebel +2Transformer Ising Model
Institute of Applied Computer Science, Jagiellonian University, ul. Łojasiewicza 11, 30-348 Kraków, Poland · Institute of Theoretical Physics, Jagiellonian University, ul. Łojasiewicza 11, 30-348 Kraków, Poland · Doctoral School of Exact and Natural Sciences, Jagiellonian University, ul. Łojasiewicza 11, 30-348 Kraków, Poland
PINN-Cast: Exploring the Role of Continuous-Depth NODE in Transformers and Physics Informed Loss as Soft Physical Constraints in Short-term Weather Forecasting
Apr 30, 2026Hira Saleem, Flora Salim, Cormac PurcellTransformer Physics-Informed ML
University of New South Wales, Sydney, NSW, Australia
Learning Rate Transfer in Normalized Transformers
Apr 29, 2026Boris Shigida, Boris Hanin, Andrey GromovTransformer Hyperparameter Transfer
Meta Superintelligence Labs · Princeton University
Stochastic Scaling Limits and Synchronization by Noise in Deep Transformer Models
Apr 29, 2026Andrea Agazzi, Giuseppe Bruno, Eloy Mosig García +2Dynamical Systems Transformer
Department of Mathematics and Statistics, University of Bern (CH) · Department of Mathematics, University of Pisa (IT)
Exploring the Potential of Probabilistic Transformer for Time Series Modeling: A Report on the ST-PT Framework
Apr 29, 2026Zhangzhi Xiong, Haoyi Wu, You Wu +3Transformer Conditional Random Fields
School of Information Science and Technology, ShanghaiTech University
Folding Tensor and Sequence Parallelism for Memory-Efficient Transformer Training & Inference
Apr 29, 2026Vasu Shyam, Anna Golubeva, Quentin AnthonyTransformer Inference Transformer
Zyphra
Efficient and Interpretable Transformer for Counterfactual Fairness
Apr 29, 2026Panyi Dong, Zhiyu QuanTransformer Interpretability Transformer
Acturial and Risk Mangament Sciences, University of Illinois Urbana-Champaign, Champaign, Illinois, USA.

May 8, 2026, cs.LGJ/K move · Enter open · S save

Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers

Noboru Isobe, Daisuke Inoue, Masaaki Imaizumi

1RIKEN Center for Advanced Intelligence Project · 2Imperial College London · 3The University of Tokyo+1

Transformer Clustering Transformer Attention Mean-Field Theory Neural Network Training Dynamics

Open paper AskPDF arXiv

Transformers perform inference by iteratively transforming token representations across layers. This layerwise computation has been studied empirically, and recent mean-field theories of Transformer dynamics explain how attention can drive token distributions toward clustering. However, existing mean-field analyses largely treat model parameters as prescribed, leaving open how training reshapes this clustering picture. We study this question in a noisy mean-field Transformer in which only a parameter-linear FFN is trained under

L^2

regularization. We find and analyze a training-induced phase in the dynamics: after initially following attention-driven clustering, the token distribution can leave the clustered regime near the final layers. Our mathematical analysis is based on an entropy-regularized interaction energy that captures the clustering bias of attention. More broadly, our results point toward a training-aware mean-field theory of Transformer dynamics, in which training and inference dynamics are treated together.

Momentum

Related topics

Latest papers 351