Dynamical Kernel Attention: Estimating the Present from a Record of the Past

Peter Racioppo

cs.LGOct 16, 2026

Dynamical Kernel Attention: Estimating the Present from a Record of the Past

Authors: Peter Racioppo

Organizations: Independent Researcher Los Angeles, CA

Abstract

Standard self-attention compares tokens by similarity but does not account for how much of a past state survives to the present. We introduce Dynamical Kernel Attention (DKA), which derives attention from robust estimation of a latent state evolving under stochastic dynamics. As a past state decays toward equilibrium under the dynamics, the information it carries about the present vanishes, bounding the total evidence a head can draw. Unlike recency biases, DKA can abstain when relevant evidence is weak and guarantees a bounded error when old tokens are discarded. DKA preserves the parallel structure and computational cost of standard attention. We present the model and its theory; empirical evaluation is deferred to a subsequent version.

Figures & tables

Figure 1: A distant token restates the prior. A latent state (black) evolves under stable linear dynamics around an equilibrium at zero; the star marks the current state

x_{i}

. Each past token predicts the present by carrying its value forward under the dynamics (blue curves), and the error bars on the right show the uncertainty of each prediction at the present. A nearby token (dark) predicts the current state closely. A distant token (light) has decayed to the equilibrium, and its uncertainty has grown to the stationary spread

\sqrt{v_{\infty}}

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: A single Isotropic DKA head. The dynamical model (blue) depends only on the lag and a few per-head scalars, and supplies the decay kernel $\boldsymbol{E}$ , the observation coefficient $h=\boldsymbol{E}^{2}$ , and the conditioned covariance $\boldsymbol{\Sigma}_{n}$ . The query is compared in the key’s frame against the backward conditional mean $\boldsymbol{E}\tilde{\boldsymbol{Q}}_{i}$ . Weights are not softmax-normalized: $\boldsymbol{m}$ is the row maximum of $\boldsymbol{L}$ , and the proximal prior on the current estimate contributes a null slot $e^{-\boldsymbol{m}}\gamma^{-2}$ to the denominator, so a head abstains when the accumulated information $\tilde{\boldsymbol{S}}$ is small. The robust weight $w$ may be the Student- $t$ or exponential kernel, and the mask is causal (including the diagonal).

Panel of Figure 3 — Figure 3: Counting evidence from distant tokens. One mode of the linear Gaussian model, with data drawn from its stationary distribution and nominal attention weights $w_{ij}=1$ ( $\mu=0.05$ , $v_{\infty}=2$ , $\eta^{2}=0.5$ ). (a) Information about the current state assigned to a single token. DKA’s falls exponentially with lag; RFA’s levels off at $1/v_{\infty}$ , since each distant token restates the prior. The exact curve is the information a token adds given all newer tokens. (b) Total information in the $N$ most recent tokens. The exact and DKA totals both saturate, DKA’s above the exact by a bounded factor; RFA’s grows linearly.

Panel of Figure 4 — Figure 4: Accuracy and memory (setting as in Figure 3 ). We compare the estimates $\bar{x}_{i}$ before the residual or proximal step. (a) MSE of each estimate of the current state, $\mathbb{E}\big[(\bar{x}_{i}-x_{i})^{2}\big]/v_{\infty}$ ; a value of one is what always guessing the equilibrium, $\bar{x}_{i}=0$ , achieves. With a few tokens, all three agree (RFA dips slightly below exact, trading a little bias for lower variance). Each further token adds a vote for the equilibrium in RFA’s average, so its error rises toward one. DKA’s levels off above the exact error, reflecting its bounded overcount. (b) Change in the estimate when only the $T$ most recent tokens of a $2048$ -token context are kept (RMS over the stationary data, relative to $\sqrt{v_{\infty}}$ ). DKA’s change falls exponentially at a rate set by $\mu$ , so tokens beyond a fixed window can be evicted for every query. RFA’s falls slowly, since a distant token’s weight in the normalizer is not decayed. The exact change falls fastest: once recent states are pinned down, older tokens add almost nothing.

Attention Mechanisms

Attention mechanisms let neural networks weight information from other tokens or regions according to relevance, enabling context-dependent processing.

See all 512 papers

Dynamic Linear Attention
Jun 9, 2026Xin Wang, Hui Shen, Boyuan Zheng +7Long-Context Language Modeling Linear Attention
The Ohio State University · University of Michigan · ByteDance Seed
Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation
Sep 4, 2025Peter RacioppoSelf-Attention Long-Context Language Modeling
Independent Researcher, Los Angeles, CA, USA.
Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobians
May 26, 2025Akiyoshi Tomihari, Ryo KarakidaSelf-Attention Attention Mechanisms
Artificial Intelligence Research Center, AIST, Japan · Department of Computer Science, The University of Tokyo, Japan · RIKEN Center for Advanced Intelligence Project

Jun 9, 2026J/K move · Enter open · S save

Dynamic Linear Attention

Xin Wang, Hui Shen, Boyuan Zheng, Xueshen Liu+6

The Ohio State University · University of Michigan · ByteDance Seed

Long-Context Language Modeling Linear Attention

Open paper AskPDF arXiv

The scalability of Large Language Models (LLMs) to long contexts is fundamentally constrained by the quadratic complexity of standard attention, motivating the adoption of linear attention mechanisms with sub-quadratic cost. To improve representation capacity under long contexts, recent approaches organize memory in a multi-state manner. However, existing multi-state linear attention methods rely on fixed state merging policies that cannot adapt to dynamically varying token importance, irreversibly obscuring critical tokens and causing severe error accumulation over long sequences. To address this limitation, we propose DLA, a dynamic memory modeling framework for multi-state linear attention. DLA introduces (i) Information-Aware Dynamic State Merging, which adaptively determines state boundaries based on token-level information variation, preserving high-resolution representations around semantic transitions while aggressively summarizing stable regions, and (ii) Capacity-Bounded Memory Modeling, which maintains a fixed-size, chronologically ordered state cache by selectively merging adjacent low-information states to control memory growth with minimal information loss. We pre-train DLA on two different linear attention models and evaluate on 16 datasets across three categories. Experimental results demonstrate the superiority of DLA over state-of-the-art.

Abstract

Attention Mechanisms

Explore similar work

Dynamic Linear Attention

Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation

Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobians