Clustering in pure-attention hardmax transformers and its role in sentiment analysis
Authors: Albert Alcalde, Giovanni Fantuzzi, Enrique Zuazua
Organizations: Chair for Dynamics, Control, Machine Learning, and Numerics (Alexander von Humboldt Professorship), Department of Mathematics, Friedrich–Alexander-Universität Erlangen–Nürnberg, 91058 Erlangen, Germany. · Departamento de Matemáticas, Universidad Autónoma de Madrid, 28049 Madrid, Spain. · Chair of Computational Mathematics, Fundación Deusto. Av. de las Universidades, 24, 48007 Bilbao, Basque Country, Spain.
Transformers are extremely successful machine learning models whose mathematical properties remain poorly understood. Here, we rigorously characterize the behavior of transformers with hardmax self-attention and normalization sublayers as the number of layers tends to infinity. By viewing such transformers as discrete-time dynamical systems describing the evolution of points in a Euclidean space, and thanks to a geometric interpretation of the self-attention mechanism based on hyperplane separation, we show that the transformer inputs asymptotically converge to a clustered equilibrium determined by special points called leaders. We then leverage this theoretical understanding to solve sentiment analysis problems from language processing using a fully interpretable transformer model, which effectively captures `context' by clustering meaningless words around leader words carrying the most meaning. Finally, we outline remaining challenges to bridge the gap between the mathematical analysis of transformers and their real-life implementation.
Figures & tables
Figure 1. Geometric interpretation of eq. 1.1b for i=1 with (a) A=I and (b) A=(2111) . In (a), tokens z2 and z3 have the largest projection on the direction of Az1=z1 , so C1(Z)={2,3} . In (b), token z4 has the largest projection on the direction of Az1 , so C1(Z)={4} . In both cases, tokens attracting z1 lie in the half-space H1={z:⟨Az1,z−z1⟩≥0} (blue shading).
Figure 2. Simulations of eq. 1.1 with α=0.5 , A=I , and four different initial token values. In each panel, stars denote tokens zi satisfying Ci(Zt)={i} at layer t∈N , while circles denote all other tokens. Colors indicate which tokens are being followed. Tokens painted in two halves follow two tokens. Tokens whose interior and edge colors are different, instead, follow tokens of their interior color and are followed by tokens of their edge color. The shaded region in the last column is the closed convex hull of leaders.
Figure 3. Schematic illustrations of a deep neural network with normalization and feed-forward sublayers (top), a full transformer with self-attention, normalization, and feed-forward sublayers (middle), and a pure-attention transformer with only self-attention and normalization sublayers (bottom). Each model takes a matrix Z0∈Rn×d as its input and outputs a matrix ZK∈Rn×d after being processed by K transformer layers. Residual connections are incorporated within each feed-forward and self-attention layer.
Figure 4. Sketch of the δ -neighborhood of the attracting set S .
Figure 5. Geometric intuition behind Lemma 4.5 for S′={s1,s2} . For δ>0 small enough and all x∈Bδ(s0) , the hyperplane with normal direction x passing through x (green) separates s0 from the balls Bδ(s1) and Bδ(s2) .
Figure 6. Sketch of the construction used to prove Proposition 4.6 . Token zi remains in a neighborhood of s1∈S1 . Token za falls in case (a) of the analysis, so it remains in a neighborhood of s2∈S2 , while token zb falls in case (b), jumping from a neighborhood of s2′∈S2 to the neighborhood of s1 , where it remains for all future times.
Figure 7. Token values in example 5.2 for (a) initial time t=0 and (b) time t=1 . After one iteration, the token z2 has moved enough to be below the hyperplane separating the tokens influencing token z1 (in green), which causes token z1 to satisfy the leader condition C1(Z1)={1} at time t=1 .
Figure 8. Loss on the training set for encoder dimensions d∈{2,4} , calculated at every epoch with the regularized softmax model used for training, and every 10 epochs with our hardmax model.
Figure 9. Evolution of the words of a positive review (top) and a negative review (bottom), as they are processed by the transformer layers. Color coded as Figure 2 . Leaders are represented with stars and tagged with the word they encode, circles are the remaining tokens, and the black triangle is the average token. The dashed line represents the hyperplane H(z)=z⊤w+v=0 , which separates the half-plane identified with the positive class (shaded in green) from the one identified with the negative class (shaded in red).
Figure 10. Histogram of the 15 most frequent leaders zi of correctly classified test reviews and satisfying ∣H(zi)∣≥2 , meaning that they are situated far from the separating hyperplane H .
d
m
σm
mmin
mmax
m0/m
2
5.960
1.247
2
11
0.995
4
19.321
4.095
4
34
0.999
8
49.866
13.131
6
84
0.999
16
75.423
23.646
7
113
1.000
Table 1. Statistical information on leaders in the test set. For each encoder dimension d∈{2,4,8,16} , average number of leaders ( m ), standard deviation of the number of leaders ( σm ), minimum number of leaders ( mmin ), maximum number of leaders ( mmax ) and average proportion of leaders determined initially ( m0/m ).
Figure 11. Simulations of eq. 7.1 with n=50 tokens in R2 , A=I , and two choices for the vectors u , v . In panel (a), where u=v=(1,1)⊤ , tokens cluster along hyperplanes perpendicular to v . In panel (b), where u=(1/2,−1/2)⊤ and v=(1,1)⊤ , tokens appear to reach an equilibrium without clear clusters. Leaders are indicated by stars in all plots.
Transformers perform inference by iteratively transforming token representations across layers. This layerwise computation has been studied empirically, and recent mean-field theories of Transformer dynamics explain how attention can drive token distributions toward clustering. However, existing analyses treat model parameters as prescribed, leaving open how training reshapes this clustering picture. We study this question in a noisy mean-field Transformer in which only a parameter-linear FFN is trained under L2 regularization. We identify a training-induced phase of the inference dynamics across layers: after clustering in the early layers, the token distribution can leave the clustered regime near the final layers. A single gradient-descent update already produces this escape, and minimizers of the training objective satisfy an exponential turnpike bound under which clustering persists through the interior layers. Our analysis is based on an entropy-regularized interaction energy that captures the clustering bias of attention, for which we establish a double-well landscape and a local Polyak-Łojasiewicz inequality. More broadly, our results point toward a training-aware mean-field theory in which training and inference dynamics of Transformers are treated together.
Noboru Isobe, Daisuke Inoue, Masaaki Imaizumi
RIKEN Center for Advanced Intelligence Project · Imperial College London · The University of Tokyo +1
Transformers have in-context learning capabilities, where some known learning algorithms can be executed in the forward pass through the model. Recent work shows that transformers can exactly perform Lloyd's algorithm for k-means clustering with n points in d dimensions with an embedding size demb=d+k (thus, requiring attention projection matrices of size (d+k)2). In this work, we build upon this result in the following ways: First, we present an equally expressive but smaller transformer that executes Lloyd's algorithm with embedding size demb=(d+⌈log2k⌉). Next, we train these transformers to learn the clustering algorithms given a distribution of clustering tasks, and theoretically characterize and empirically validate the factors affecting the convergence and in-distribution generalization of learning algorithms based on stochastic gradients. Finally, we probe the general clustering abilities of these learned algorithms (in the form of transformers), and try to understand situations where they succeed and fail.
Charlotte Park, Kenneth L. Clarkson, Lior Horesh +2
To quantify the geometric expressivity of transformers, we introduce a tropical geometry framework to characterize their exact spatial partitioning capabilities. By modeling self-attention as a vector-valued tropical rational map, we prove it evaluates exactly to a Power Voronoi Diagram in the zero-temperature limit. Building on this equivalence, we establish a combinatorial rationale for Multi-Head Self-Attention (MHSA): via the Minkowski sum of Newton polytopes, multi-head aggregation expands the polyhedral complexity to O(NH), overcoming the O(N) bottleneck of single heads. Extending this to deep architectures, we derive the first tight asymptotic bounds on the number of linear regions in transformers (Θ(NdmodelL)), demonstrating a combinatorial explosion driven intrinsically by sequence length N, ambient embedding dimension dmodel, and network depth L. Importantly, we guarantee that this idealized polyhedral skeleton is geometrically stable: finite-temperature soft attention preserves these topological partitions via exponentially tight differential approximation bounds.
Ye Su, Yong Liu
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China · University of Chinese Academy of Sciences, Beijing 101407, China · Gaoling School of Artificial Intelligence, Renmin University of China, Beijing 100872, China