math.PROct 23, 2025

On the Structure of Stationary Solutions to McKean-Vlasov Equations with Applications to Noisy Transformers

Authors: Krishnakumar Balasubramanian, Sayan Banerjee, Philippe Rigollet

Organizations: Department of Statistics, University of California, Davis · Department of Statistics and Operations Research, University of North Carolina, Chapel Hill · Department of Mathematics, Massachusetts Institute of Technology

Abstract

We study stationary solutions of McKean-Vlasov equations on the circle. We establish an exact equivalence between solutions of the stationary McKean-Vlasov equation and an infinite-dimensional quadratic system of equations over Fourier coefficients, allowing explicit characterization of stationary states in a sequence space rather than a function space. This framework provides a transparent description of local bifurcations, their periodicity and resonance structures, while accommodating singular potentials. We derive analytic expressions for the emergence, form and shape (supercritical, critical, subcritical or transcritical) of bifurcations involving possibly multiple Fourier modes and connect them with discontinuous phase transitions. We also characterize, under suitable assumptions, the detailed structure of the stationary bifurcating solutions through leading small-amplitude coefficients for arbitrarily many fixed Fourier modes. At the global level, we establish regularity and concavity properties of the free energy landscape, proving existence, compactness, and coexistence of globally minimizing stationary measures, and identifying discontinuous phase transitions with points of non-differentiability of the minimum free energy map. We apply the theory to the Noisy Mean-Field Transformer model, where we show how changing the inverse temperature ββ affects the geometry of the infinitely many bifurcations from the uniform measure. We also discuss how increasing ββ motivates the study of approximate multi-mode stationary profiles as candidates for a separate metastability analysis; persistence times require additional dynamical estimates. Moreover, we determine the exact threshold β0≃2.447β_0\simeq2.447 separating continuous and discontinuous phase transitions in this model. The same threshold marks the change of the first bifurcation from supercritical to subcritical.

Figures & tables

Explore similar work

May 11, 2026

Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime

Transformers with self-attention modules as their core components have become an integral architecture in modern large language and foundation models. In this paper, we study the evolution of tokens in deep encoder-only transformers at inference time which is described in the large-token limit by a mean-field continuity equation. Leveraging ideas from the convergence analysis of interacting multi-particle systems, with particles corresponding to tokens, we prove that the token distribution rapidly concentrates onto the push-forward of the initial distribution under a projection map induced by the key, query, and value matrices, and remains metastable for moderate times. Specifically, we show that the Wasserstein distance of the two distributions scales like log⁡(β+1)/βexp⁡(Ct)+exp⁡(−ct)\sqrt{{\log(β+1)}/β}\exp(Ct)+\exp(-ct) in terms of the temperature parameter β−1→0β^{-1}\to 0 and inference time t≥0t\geq 0. For the proof, we establish Lyapunov-type estimates for the zero-temperature equation, identify its limit as t→∞t\to\infty, and employ a stability estimate in Wasserstein space together with a quantitative Laplace principle to couple the two equations. Our result implies that for time scales of order log⁡β\logβ, which correspond to the early-phase behavior most relevant for practical finite-depth models, the token distribution concentrates at the identified limiting distribution. Numerical experiments confirm this and, beyond that, complement our theory by showing that for finite ββ and large tt the dynamics enter a different terminal phase, dominated by the spectrum of the value matrix.
Nov 14, 2026

Equilibrium measures for anisotropic exponential interaction energies with applications to transformers

We study the minimization of anisotropic exponential interaction energies over probability measures on the sphere, motivated by mean-field models of self-attention in transformer architectures. For every symmetric positive definite interaction matrix, we relate the minimizer to a corresponding anisotropic Brownian motion and use the Feynman--Kac formula and elliptic regularity theory to show that it admits a smooth, strictly positive density on the sphere. We further show that it is the only stationary measure with full support for the dynamics considered. We use the stochastic viewpoint to derive an algorithm for sampling from the minimizer. In dimension two, we also propose a numerical scheme for approximating its density based on a Mathieu series expansion.
Apr 29, 2026

Stochastic Scaling Limits and Synchronization by Noise in Deep Transformer Models

We prove pathwise convergence of the layerwise evolution of tokens in a finite-depth, finite-width transformer model with MultiLayer Perceptron (MLP) blocks to a continuous-time stochastic interacting particle system. We also identify the stochastic partial differential equation describing the evolution of the tokens' distribution in this limit and prove propagation of chaos when the number of such tokens is large. The bounds we establish are quantitative and the limits we consider commute. We further prove that the limiting stochastic model displays synchronization by noise and establish exponential dissipation of the interaction energy on average, provided that the common noise is sufficiently coercive relative to the deterministic self-attention drift. We finally characterize the activation functions satisfying the former condition.