Asymptotic Behavior of Transformers

Katy Blumer; Da-Cheng Juan; Jon Kleinberg; Ravi Kumar; Cyrus Rashtchian; Andrew Tomkins

cs.LGOct 14, 2026

Asymptotic Behavior of Transformers

Authors: Katy Blumer, Da-Cheng Juan, Jon Kleinberg, Ravi Kumar, Cyrus Rashtchian, Andrew Tomkins

Organizations: Google Research

Abstract

In this paper we study the behavior of transformer networks generating text over long periods of time. We show that even a three-token, single-layer attention-only transformer can fail to converge. We then study a rich class of transformers with arbitrary finite depth, layer normalization, MLPs, RoPE encodings, and an arbitrary mix of local and global layers. Every transformer in this class has a softmax temperature threshold above which its empirical cache distribution converges to a unique limit, independent of initial conditions. We show further that, while each transformer has a temperature threshold, the supremum of this set is infinite, so there is no global minimal temperature that forces all transformers to converge.

Figures & tables

Panel of Figure 1 — Figure 1: A stripped-down model of a transformer with NoPE, no MLPs, and an unbounded context window parameterized by historical distribution $P_{t}$ .

Panel of Figure 2 — Figure 2: A two-token example illustrating the temperature transition: $x=2p-1$ and $dx/dt=\tanh((a/\tau)x)-x$ . For $\tau>a$ , the uniform state is the unique globally attracting equilibrium. For $\tau<a$ , it becomes unstable and two symmetric stable equilibria appear.

Panel of Figure 3 — Figure 3: Phase portrait of the three-token example in Section 4, with $c=7$ . The unique equilibrium $\bar{P}=(1/3,1/3,1/3)$ is a repelling focus. The teal curve is a numerically computed attracting periodic orbit of the limiting ODE. Trajectories starting inside (orange) and outside (gray) the orbit spiral toward it in a counterclockwise direction. The shaded sectors indicate the most probable next token: red for token 1, green for token 2, and blue for token 3. The trajectories represent the limiting deterministic dynamics, not individual stochastic sample paths.

Transformer Training Dynamics

See all 24 papers

Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime
May 11, 2026Albert Alcalde, Leon Bungert, Konstantin Riedl +1Transformer Inference Transformer
Department of Mathematics Friedrich-Alexander-University Erlangen-Nürnberg · Institute of Mathematics CAIDAS University of Würzburg · Mathematical Institute University of Oxford Reuben College +1
Stochastic Scaling Limits and Synchronization by Noise in Deep Transformer Models
Apr 29, 2026Andrea Agazzi, Giuseppe Bruno, Eloy Mosig García +2Dynamical Systems Transformer
Department of Mathematics and Statistics, University of Bern (CH) · Department of Mathematics, University of Pisa (IT)
Uniform Scaling Limits in AdamW-Trained Transformers
May 11, 2026William Gibson, Christoph ReisingerTransformer Self-Attention
Mathematical Institute · University of Oxford · Oxford, OX2 6GG, UK

May 11, 2026J/K move · Enter open · S save

Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime

Albert Alcalde, Leon Bungert, Konstantin Riedl, Tim Roith

Department of Mathematics Friedrich-Alexander-University Erlangen-Nürnberg · Institute of Mathematics CAIDAS University of Würzburg · Mathematical Institute University of Oxford Reuben College+1

Transformer Inference Transformer Transformer Attention Mean-Field Theory

Open paper AskPDF arXiv

Transformers with self-attention modules as their core components have become an integral architecture in modern large language and foundation models. In this paper, we study the evolution of tokens in deep encoder-only transformers at inference time which is described in the large-token limit by a mean-field continuity equation. Leveraging ideas from the convergence analysis of interacting multi-particle systems, with particles corresponding to tokens, we prove that the token distribution rapidly concentrates onto the push-forward of the initial distribution under a projection map induced by the key, query, and value matrices, and remains metastable for moderate times. Specifically, we show that the Wasserstein distance of the two distributions scales like

\sqrt{{\log(β+1)}/β}\exp(Ct)+\exp(-ct)

in terms of the temperature parameter

β^{-1}\to 0

and inference time

t\geq 0

. For the proof, we establish Lyapunov-type estimates for the zero-temperature equation, identify its limit as

t\to\infty

, and employ a stability estimate in Wasserstein space together with a quantitative Laplace principle to couple the two equations. Our result implies that for time scales of order

\logβ

the token distribution concentrates at the identified limiting distribution. Numerical experiments confirm this and, beyond that, complement our theory by showing that for finite

β

and large

t

the dynamics enter a different terminal phase, dominated by the spectrum of the value matrix.

Abstract

Transformer Training Dynamics

Explore similar work

Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime

Stochastic Scaling Limits and Synchronization by Noise in Deep Transformer Models

Uniform Scaling Limits in AdamW-Trained Transformers