cs.AINov 13, 2026

Two-Clustering Regime of Token Dynamics in Causal Attention

Authors: Trinh T. Nguyen, Duy-Tung Pham, Hoang-Son Do, Tan Minh Nguyen, Thieu N. Vo

Organizations: School of Mathematical Sciences Shanghai Jiao Tong University, China · Khoury College of Computer Sciences Northeastern University, USA · Institute of Mathematics Vietnam Academy of Science and Technology · University of Science and Technology of Hanoi Vietnam Academy of Science and Technology · 

Abstract

We partially resolve Conjecture 1 of [Karagodin et al., Clustering in Causal Attention Masking (NeurIPS 2024)] and extend it to time-dependent parameter regimes. The conjecture predicts that causal attention dynamics converge to two clusters aligned with the eigenvectors associated with the largest eigenvalue λmax⁡λ_{\max} of the value transformation. We prove that this behavior holds when λmax⁡>0λ_{\max}>0 and is generally false when λmax⁡≤0λ_{\max}\le0. The resulting two-cluster regime is common in practice and substantially more challenging than the classical single-cluster setting studied previously. A central contribution of our work is the identification of a rigorous dimension reduction phenomenon: despite evolving in a high-dimensional embedding space, token trajectories collapse onto a two-dimensional structure governed by a small number of dominant directions. Building on this insight, we introduce a new analytical framework--instability within the flow--inspired by instability phenomena in physical systems. This framework enables the analysis of steady-regime behavior in non-monotone, non-autonomous attention dynamics. Our theoretical findings are illustrated through simulations and experimental studies using Group Query Attention (GQA) layers.

Figures & tables

Explore similar work

CardsList
  1. Clustered Attractor Manifolds and Dynamical Condensation in Self-Attention

    Aug 9, 2026Qucheng Gao, Zuyi Yang, Xiao ChenDynamical SystemsSelf-Attention

  2. Opinion Leader Dynamics: How Sparse Attention Shapes Token Clustering

    Sep 21, 2026Jingkun Liu, Yue SongOpinion DynamicsClustering