Standard self-attention compares tokens by similarity but does not account for how much of a past state survives to the present. We introduce Dynamical Kernel Attention (DKA), which derives attention from robust estimation of a latent state evolving under stochastic dynamics. As a past state decays toward equilibrium under the dynamics, the information it carries about the present vanishes, bounding the total evidence a head can draw. Unlike recency biases, DKA can abstain when relevant evidence is weak and guarantees a bounded error when old tokens are discarded. DKA preserves the parallel structure and computational cost of standard attention. We present the model and its theory; empirical evaluation is deferred to a subsequent version.
Figures & tables
Figure 1: A distant token restates the prior. A latent state (black) evolves under stable linear dynamics around an equilibrium at zero; the star marks the current state xi . Each past token predicts the present by carrying its value forward under the dynamics (blue curves), and the error bars on the right show the uncertainty of each prediction at the present. A nearby token (dark) predicts the current state closely. A distant token (light) has decayed to the equilibrium, and its uncertainty has grown to the stationary spread v∞ .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: A single Isotropic DKA head. The dynamical model (blue) depends only on the lag and a few per-head scalars, and supplies the decay kernel E , the observation coefficient h=E2 , and the conditioned covariance Σn . The query is compared in the key’s frame against the backward conditional mean EQ~i . Weights are not softmax-normalized: m is the row maximum of L , and the proximal prior on the current estimate contributes a null slot e−mγ−2 to the denominator, so a head abstains when the accumulated information S~ is small. The robust weight w may be the Student- t or exponential kernel, and the mask is causal (including the diagonal).
Figure 3: Counting evidence from distant tokens. One mode of the linear Gaussian model, with data drawn from its stationary distribution and nominal attention weights wij=1 ( μ=0.05 , v∞=2 , η2=0.5 ). (a) Information about the current state assigned to a single token. DKA’s falls exponentially with lag; RFA’s levels off at 1/v∞ , since each distant token restates the prior. The exact curve is the information a token adds given all newer tokens. (b) Total information in the N most recent tokens. The exact and DKA totals both saturate, DKA’s above the exact by a bounded factor; RFA’s grows linearly.
Figure 4: Accuracy and memory (setting as in Figure 3 ). We compare the estimates xˉi before the residual or proximal step. (a) MSE of each estimate of the current state, E[(xˉi−xi)2]/v∞ ; a value of one is what always guessing the equilibrium, xˉi=0 , achieves. With a few tokens, all three agree (RFA dips slightly below exact, trading a little bias for lower variance). Each further token adds a vote for the equilibrium in RFA’s average, so its error rises toward one. DKA’s levels off above the exact error, reflecting its bounded overcount. (b) Change in the estimate when only the T most recent tokens of a 2048 -token context are kept (RMS over the stationary data, relative to v∞ ). DKA’s change falls exponentially at a rate set by μ , so tokens beyond a fixed window can be evicted for every query. RFA’s falls slowly, since a distant token’s weight in the normalizer is not decayed. The exact change falls fastest: once recent states are pinned down, older tokens add almost nothing.
Attention mechanisms let neural networks weight information from other tokens or regions according to relevance, enabling context-dependent processing.
The scalability of Large Language Models (LLMs) to long contexts is fundamentally constrained by the quadratic complexity of standard attention, motivating the adoption of linear attention mechanisms with sub-quadratic cost. To improve representation capacity under long contexts, recent approaches organize memory in a multi-state manner. However, existing multi-state linear attention methods rely on fixed state merging policies that cannot adapt to dynamically varying token importance, irreversibly obscuring critical tokens and causing severe error accumulation over long sequences. To address this limitation, we propose DLA, a dynamic memory modeling framework for multi-state linear attention. DLA introduces (i) Information-Aware Dynamic State Merging, which adaptively determines state boundaries based on token-level information variation, preserving high-resolution representations around semantic transitions while aggressively summarizing stable regions, and (ii) Capacity-Bounded Memory Modeling, which maintains a fixed-size, chronologically ordered state cache by selectively merging adjacent low-information states to control memory growth with minimal information loss. We pre-train DLA on two different linear attention models and evaluate on 16 datasets across three categories. Experimental results demonstrate the superiority of DLA over state-of-the-art.
Xin Wang, Hui Shen, Boyuan Zheng +7
The Ohio State University · University of Michigan · ByteDance Seed
We introduce Robust Filter Attention (RFA), a formulation of self-attention as a robust state estimator. Each token is treated as a noisy observation of a latent trajectory governed by a linear stochastic differential equation (SDE), and attention weights are determined by consistency under this model rather than static feature similarity. Under isotropic noise and decay assumptions, RFA matches the computational complexity of standard attention. On language modeling benchmarks, RFA achieves lower perplexity than RoPE within the training window while remaining stable under zero-shot extrapolation to longer contexts. The framework also provides a dynamical interpretation of standard positional mechanisms, connecting rotational embeddings and recency biases to transport and uncertainty propagation induced by stochastic dynamics.
The theoretical understanding of self-attention (SA) has been steadily progressing. A prominent line of work studies a class of SA layers that admit an energy function decreased by state updates. While it provides valuable insights into inherent biases in signal propagation, it often relies on idealized assumptions or additional constraints not necessarily present in standard SA. Thus, to broaden our understanding, this work aims to relax these energy constraints and provide an energy-agnostic characterization of inference dynamics by dynamical systems analysis. In more detail, we first consider relaxing the symmetry and single-head constraints traditionally required in energy-based formulations. Next, we show that analyzing the Jacobian matrix of the state is highly valuable when investigating more general SA architectures without necessarily admitting an energy function. It reveals that the normalization layer plays an essential role in suppressing the Lipschitzness of SA and the Jacobian's complex eigenvalues, which correspond to the oscillatory components of the dynamics. In addition, the Lyapunov exponents computed from the Jacobians demonstrate that the normalized dynamics lie close to a critical state, and this criticality serves as a strong indicator of high inference performance. Furthermore, the Jacobian perspective also enables us to develop regularization methods for training and a pseudo-energy for monitoring inference dynamics.
Akiyoshi Tomihari, Ryo Karakida
Artificial Intelligence Research Center, AIST, Japan · Department of Computer Science, The University of Tokyo, Japan · RIKEN Center for Advanced Intelligence Project