Stochastic Scaling Limits and Synchronization by Noise in Deep Transformer Models

Andrea Agazzi; Giuseppe Bruno; Eloy Mosig García; Samuele Saviozzi; Marco Romito

math.PRApr 29, 2026

Stochastic Scaling Limits and Synchronization by Noise in Deep Transformer Models

Authors: Andrea Agazzi, Giuseppe Bruno, Eloy Mosig García, Samuele Saviozzi, Marco Romito

Organizations: Department of Mathematics and Statistics, University of Bern (CH) · Department of Mathematics, University of Pisa (IT)

Abstract

We prove pathwise convergence of the layerwise evolution of tokens in a finite-depth, finite-width transformer model with MultiLayer Perceptron (MLP) blocks to a continuous-time stochastic interacting particle system. We also identify the stochastic partial differential equation describing the evolution of the tokens' distribution in this limit and prove propagation of chaos when the number of such tokens is large. The bounds we establish are quantitative and the limits we consider commute. We further prove that the limiting stochastic model displays synchronization by noise and establish exponential dissipation of the interaction energy on average, provided that the common noise is sufficiently coercive relative to the deterministic self-attention drift. We finally characterize the activation functions satisfying the former condition.

Explore similar work

Uniform Scaling Limits in AdamW-Trained Transformers
May 11, 2026William Gibson, Christoph ReisingerTransformer Architectures Hidden States
Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime
May 11, 2026Albert Alcalde, Leon Bungert, Konstantin Riedl +1Transformer Encoder Transformer Architectures
On the Diverse Dynamical Behaviors Arising in Deep Linear Transformers
Jul 20, 2026Sixu Li, Thomas Jacob Maranzatto, Jan Peszek +5Transformer Architectures Softmax Attention

May 11, 2026 · stat.MLJ/K move · Enter open · S save

Uniform Scaling Limits in AdamW-Trained Transformers

William Gibson, Christoph Reisinger

Mathematical Institute · University of Oxford · Oxford, OX2 6GG, UK

We study the large-depth limit of transformers trained with AdamW, by modelling the hidden-state dynamics as an interacting particle system (IPS) coupled through the attention mechanism. Under appropriate scaling of the attention heads, we prove that the joint dynamics of the hidden states and backpropagated variables converge in

L^2

, uniformly over the initial condition, to the solution of a forward--backward system of ODEs at rate

\mathcal O(L^{-1}+L^{-1/3}H^{-1/2})

. Here,

L

and

H

denote the depth and number of heads of the transformer, respectively. The limiting system of ODEs can be identified with a McKean--Vlasov ODE (MVODE) when the attention heads do not incorporate causal masking. By using the flow maps associated with this MVODE and applying concentration of measure techniques, we obtain bounds on the difference between the discrete and continuous models that are uniform over compact sets of initial conditions. As this is achieved without resorting to a covering argument, the constants in our bounds are independent of the number of tokens. Furthermore, under a suitable adaptation to AdamW, the bounds become independent of the token embedding dimension.

Transformer Architectures Hidden States Neural Scaling Laws

Open paper Ask about it PDF

Abstract

Explore similar work

Uniform Scaling Limits in AdamW-Trained Transformers

Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime

On the Diverse Dynamical Behaviors Arising in Deep Linear Transformers