cs.CLJun 26, 2024

Clustering in pure-attention hardmax transformers and its role in sentiment analysis

Authors: Albert Alcalde, Giovanni Fantuzzi, Enrique Zuazua

Organizations: Chair for Dynamics, Control, Machine Learning, and Numerics (Alexander von Humboldt Professorship), Department of Mathematics, Friedrich–Alexander-Universität Erlangen–Nürnberg, 91058 Erlangen, Germany. · Departamento de Matemáticas, Universidad Autónoma de Madrid, 28049 Madrid, Spain. · Chair of Computational Mathematics, Fundación Deusto. Av. de las Universidades, 24, 48007 Bilbao, Basque Country, Spain.

Abstract

Transformers are extremely successful machine learning models whose mathematical properties remain poorly understood. Here, we rigorously characterize the behavior of transformers with hardmax self-attention and normalization sublayers as the number of layers tends to infinity. By viewing such transformers as discrete-time dynamical systems describing the evolution of points in a Euclidean space, and thanks to a geometric interpretation of the self-attention mechanism based on hyperplane separation, we show that the transformer inputs asymptotically converge to a clustered equilibrium determined by special points called leaders. We then leverage this theoretical understanding to solve sentiment analysis problems from language processing using a fully interpretable transformer model, which effectively captures `context' by clustering meaningless words around leader words carrying the most meaning. Finally, we outline remaining challenges to bridge the gap between the mathematical analysis of transformers and their real-life implementation.

Figures & tables

Explore similar work

CardsList
  1. Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers

    May 8, 2026Noboru Isobe, Daisuke Inoue, Masaaki ImaizumiTransformerTransformer Training Dynamics

  2. Leaner Transformers Can Easily Learn to Cluster

    Oct 7, 2026Charlotte Park, Kenneth L. Clarkson, Lior Horesh +2TransformerClustering

  3. Expressivity of Transformers: A Tropical Geometry Perspective

    Apr 16, 2026Ye Su, Yong LiuTransformer ExpressivityTransformer