Two-Clustering Regime of Token Dynamics in Causal Attention
Authors: Trinh T. Nguyen, Duy-Tung Pham, Hoang-Son Do, Tan Minh Nguyen, Thieu N. Vo
Organizations: School of Mathematical Sciences Shanghai Jiao Tong University, China · Khoury College of Computer Sciences Northeastern University, USA · Institute of Mathematics Vietnam Academy of Science and Technology · University of Science and Technology of Hanoi Vietnam Academy of Science and Technology · Department of Mathematics National University of Singapore · Department of Computer Science University of Bath, United Kingdom
We partially resolve Conjecture 1 of [Karagodin et al., Clustering in Causal Attention Masking (NeurIPS 2024)] and extend it to time-dependent parameter regimes. The conjecture predicts that causal attention dynamics converge to two clusters aligned with the eigenvectors associated with the largest eigenvalue λmax of the value transformation. We prove that this behavior holds when λmax>0 and is generally false when λmax≤0. The resulting two-cluster regime is common in practice and substantially more challenging than the classical single-cluster setting studied previously. A central contribution of our work is the identification of a rigorous dimension reduction phenomenon: despite evolving in a high-dimensional embedding space, token trajectories collapse onto a two-dimensional structure governed by a small number of dominant directions. Building on this insight, we introduce a new analytical framework--instability within the flow--inspired by instability phenomena in physical systems. This framework enables the analysis of steady-regime behavior in non-monotone, non-autonomous attention dynamics. Our theoretical findings are illustrated through simulations and experimental studies using Group Query Attention (GQA) layers.
Figures & tables
Work
Parameters
Initial Data
Key Techniques
Theoretical Results
Karagodin et al. (2024)
V constant
Arbitrary
Explicit x1(t) formula;
Convergence of x1(t) to ±ξmax
Q,K general
on Sd−1
Schub genericity theorem
Full convergence if V=Id
Abella et al. (2025)
V(t) symmetric
Single
Lyapunov function;
Global convergence to
Q(t),K(t) general
hemisphere
Input-to-state stability
single cluster ξmax
Ours
V(t) symmetric
Arbitrary
Instability analysis;
Global convergence to
Q(t),K(t) general
on Sd−1
Hierarchy dim. deduction
two clusters ±ξmax ;
Table 1: Comparison of theoretical analyses for causal-attention dynamics. Our work resolves the two-cluster regime for diagonal V(t) without hemisphere constraints and provides the first theoretical justification for the dimensional deduction phenomenon conjectured from experiments.
Figure 1: Initial (circles) and final (squares) token positions after processing through weight-sharing causal GQA with RMSNorm blocks. Tokens organize into two opposing clusters on the circle.
Figure 2: Tokens initial (circle) and final (square) positions and trajectories across various choices of hidden dimension.
Fine-tuning weight
Q
K
V
Q,K
Q,V
Fine-tuning rank
8
8
8
4
4
WikiSQL ( ±0.5% )
70.4
70.0
73.0
71.4
73.7
MultiNLI ( ±0.1% )
91.0
90.8
91.0
91.3
91.3
Table 2: Validation accuracy ( %,↑ ) on WikiSQL and MultiNLI when fine-tuning GPT-3 with LoRA applied to different subsets of self-attention weight matrices. All configurations use 18M trainable parameters. The standard deviation across random seeds is consistent for each dataset and reported in the first column. For the same number of fine-tuned weight types, combinations that include V yield higher accuracy. Results reproduced in ( Hu et al., 2022 ) .
Transformer layers generate state-dependent interaction networks: token representations determine the attention matrix, which in turn updates the representations. We study this feedback in a minimal normalized self-attention dynamics and identify the overlap gap as the central quantity governing its attractor structure in the thermodynamic limit. When tokens form internally aligned clusters and their similarity to members of the same cluster exceeds that to every other cluster by a nonvanishing amount, inter-cluster attention is exponentially suppressed as the dimension increases. This mechanism produces a high-dimensional manifold of clustered fixed points, ranging from a few macroscopic clusters to extensive microscopic fragmentation, and also controls their stability against perturbations. Starting from an unstructured Gaussian state, we find that clustered states nucleate from the diffuse background only above a finite threshold in attention sharpness, giving rise to a dynamical attention-condensation transition.
Qucheng Gao, Zuyi Yang, Xiao Chen
Department of Physics, Boston College, Chestnut Hill, Massachusetts 02467, USA · Department of Physics, Tsinghua University, Beijing 100084, China
Transformer self-attention can be interpreted as a gradient flow on the unit sphere, in which tokens evolve under softmax interaction potentials and tend to form clusters. While prior work has established clustering behavior for single-head attention, the multi-head setting remains less understood due to geometric interference between heads, which invalidates standard monotonicity arguments. In this work, we develop a theoretical framework for multi-head self-attention dynamics and resolve several open questions. We show that, under suitable conditions on the score matrices, a natural multi-head energy functional is non-decreasing along both flat and spherical dynamics. We identify the key obstruction to per-head monotonicity as radial shadow terms, which are projections of each head's output onto token directions, persisting even under orthogonality assumptions. We introduce a sufficient condition ensuring monotonicity and establish robustness to approximate orthogonality. In a simplified scalar-head regime with equiangular token configurations, we derive a closed-form expression for the critical inverse temperature governing clustering behavior, and show that heterogeneous heads exhibit super-additive clustering rates. In this regime, we also prove a separation in clustering time between ReLU and softmax attention in the linearized dynamics. Finally, we establish an entropy production identity and show that attention entropy increases monotonically toward equilibrium as clustering progresses. Our results provide a unified perspective on the dynamics of multi-head attention and clarify the mechanisms underlying clustering and stability in transformer models.
Sparse attention reduces the quadratic cost of global self-attention while retaining strong empirical performance, but how its restricted interactions shape the evolution of token representations remains theoretically underexplored. Modeling tokens as particles on the unit sphere, we introduce opinion leader dynamics, a framework that identifies two mechanisms through which token groups converge internally while maintaining distinct limiting directions. In the explicit model, fixed representatives induce a potential that attracts tokens toward distinct local maxima. In the implicit model, disconnected interaction groups evolve toward separate consensus directions. We formulate both models as reverse Wasserstein gradient flows and establish exponential convergence under suitable conditions. We further connect these theoretical predictions to token evolution in frontier sparse-attention LLMs that motivate our framework. Across four benchmarks, Kimi-K3, MiniMax-M3, and DeepSeek-V4-Flash consistently exhibit clearer cluster separation and higher clustering scores than the dense-attention model GLM-4.7-Flash in projected token representations. These observations support the relevance of the predicted multiple-group structure to trained frontier LLMs, while finite-particle simulations illustrate the theoretical convergence behavior. Together, our results connect restricted token interactions to distinct group-level attractors, providing a dynamical account of how sparse attention can support alignment within groups while preserving separation between them.
Jingkun Liu, Yue Song
Tsinghua University · Shanghai Jiao Tong University