cs.LGNov 7, 2026

A Geometric Theory for Nonequilibrium Transformer Dynamics

Authors: Aythami Bethencourt de León, Darryl D. Holm

Organizations: Departamento de Análisis Matemático, Universidad de La Laguna · Department of Mathematics, Imperial College London

Abstract

We develop a variational and geometric theory based on statistical physics for the joint evolution of Transformer attention and token representations at fixed model parameters. To this end, we introduce a free energy for the attention and model the evolution of the attention weights by its Fisher--Rao gradient flow. At fixed token representations, we prove that the softmax matrix is the unique free-energy minimiser and that every positive attention trajectory converges to it. For prescribed time-dependent representations, we show that the attention dynamics becomes linear after a change to logit variables. This result yields tracking bounds relative to the instantaneous softmax and sufficient conditions for attention to converge to a fixed matrix. Hamiltonian dynamics for the representations and their momenta are then coupled to the attention relaxation driven by the same energy. This coupling results in an energy-dissipation law. We also develop two alternative Hamiltonians incorporating the attention value map. We prove global well-posedness of the attention flow and all three coupled systems.

Figures & tables

Explore similar work

CardsList
  1. A Mixture-Gated Theory of Anisotropic Representation Dynamics in Transformers

    Oct 16, 2026Nguyen Do, My T. ThaiTransformerSelf-Attention

  2. On the Statistical Mechanics of Attention

    Oct 12, 2026Jerry Yao-Chieh Hu, Yu-An Chen, Maojiang Su +1Transformer AttentionAttention Mechanisms

  3. Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime

    May 11, 2026Albert Alcalde, Leon Bungert, Konstantin Riedl +1Transformer InferenceTransformer