math.APNov 14, 2026

Equilibrium measures for anisotropic exponential interaction energies with applications to transformers

Authors: Oleksii Kachaiev, Martin Burger, Samira Kabri, Yury Korolev, Tim Roith

Organizations: MaLGa, Dipartimento di Matematica, Università degli Studi di Genova, Genoa, Italy. · Helmholtz Imaging, Deutsches Elektronen-Synchrotron DESY, Notkestr. 85, 22607 Hamburg, Germany. · Department of Mathematics, University of Hamburg, Bundesstr. 55, 20146 Hamburg, Germany. · Department of Mathematical Sciences, University of Bath, Bath, UK. · 

Abstract

We study the minimization of anisotropic exponential interaction energies over probability measures on the sphere, motivated by mean-field models of self-attention in transformer architectures. For every symmetric positive definite interaction matrix, we relate the minimizer to a corresponding anisotropic Brownian motion and use the Feynman--Kac formula and elliptic regularity theory to show that it admits a smooth, strictly positive density on the sphere. We further show that it is the only stationary measure with full support for the dynamics considered. We use the stochastic viewpoint to derive an algorithm for sampling from the minimizer. In dimension two, we also propose a numerical scheme for approximating its density based on a Mathieu series expansion.

Explore similar work

CardsList
  1. Sampling the XY model with an attention-based neural network

    Oct 16, 2026Piotr Białas, Christian D. Morales, Tomasz Stebel +1Boltzmann SamplingAutoregressive Transformer

  2. Measure Neural Mappings

    Oct 12, 2026Ricardo Baptista, Edoardo Calvello, Bohan Chen +2Operator LearningNeural Operators

  3. A Geometric Theory for Nonequilibrium Transformer Dynamics

    Nov 7, 2026Aythami Bethencourt de León, Darryl D. HolmTransformer AttentionSoftmax Attention