SympFormer: Accelerated attention blocks via Inertial Dynamics on Density Manifolds

Viktor Stein; Wuchen Li; Gabriele Steidl

cs.LGMar 17, 2026

SympFormer: Accelerated attention blocks via Inertial Dynamics on Density Manifolds

Authors: Viktor Stein, Wuchen Li, Gabriele Steidl

Organizations: Institute of Mathematics, Technische Universität Berlin, Straße des 17. Juni 136, 10623 Berlin, Germany · Department of Mathematics, University of South Carolina, Columbia. 1523 Greene St, Columbia, SC 29208, USA

Abstract

Transformers owe much of their empirical success in natural language processing to the self-attention blocks. Recent perspectives interpret attention blocks as interacting particle systems, whose mean-field limits correspond to gradient flows of interaction energy functionals on probability density spaces equipped with Wasserstein- $2$ -type metrics. We extend this viewpoint by introducing accelerated attention blocks derived from inertial Nesterov-type dynamics on density spaces. In our proposed architecture, tokens carry both spatial (feature) and velocity variables. The time discretization and the approximation of accelerated density dynamics yield Hamiltonian momentum attention blocks, which constitute the proposed accelerated attention architectures. In particular, for linear self-attention, we show that the attention blocks approximate a Stein variational gradient flow, using a bilinear kernel, of a potential energy. In this setting, we prove that elliptically contoured probability distributions are preserved by the accelerated attention blocks. We present implementable particle-based algorithms and demonstrate that the proposed accelerated attention blocks converge faster than the classical attention blocks while preserving the number of oracle calls.

Figures & tables

Panel of Figure 1 — Figure 1: The accelerated attention block followed by an accelerated MLP block as described in Algorithm 1 .

model	last ( $\downarrow$ )	best ( $\downarrow$ )	wall clock time (seconds) ( $\downarrow$ )
Baseline	2.4687	2.4473	2409.4
YuriiFormer-Lie-Trotter	2.4041	2.3872	2835.2
Plain Euler	2.3247	2.3234	5036.1
Presymp Euler	2.4728	2.4592	4979.0
Presymp ExpEuler	2.3579	2.2523	5623.4
Presymp AB2	2.6653	2.6546	5249.4

Table 1: Validation loss on the tinystories data set after 10000 optimization steps.

Figure 2: Validation loss (circles) and training loss (lines) on the tinystories data set after 10000 optimization steps, illustrating the results from Table 1 .

model	last ( $\downarrow$ )	best ( $\downarrow$ )	wall clock time (seconds) ( $\downarrow$ )
Baseline	3.0830	2.961427	486.7
YuriiFormer	2.8783	2.765975	749.6
plain Euler	3.0399	2.920602	809.3
Presymp Euler	2.8467	2.731363	1172.8

Table 2: Validation loss of the SympFormer with linear attention (smaller configuration) on the TinyStories data set after 10000 optimization steps.

model	last ( $\downarrow$ )	best ( $\downarrow$ )	wall clock time (seconds) ( $\downarrow$ )
Baseline	3.0118	2.917168	8956.8
YuriiFormer	2.8486	2.765023	9601.3
plain Euler	2.7483	2.656692	10806.5
Presymp Euler	2.7862	2.690885	12620.0

Table 3: Validation loss of the SympFormer with linear attention (larger configuration) on the TinyStories data set after 10000 optimization steps.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

architecture	$\zeta_{1}$	$\zeta_{2}$
plain Euler	$\alpha$	$1$
Presymp Euler	$1-\alpha(t_{k})h_{Y}$	$1$
Presymp ExpEuler	$e^{-\Delta\eta_{k}}$	$\frac{1-e^{-\Delta\eta_{k}}}{\alpha_{\text{eff}}}$

Appendix

Table 4: Scalar prefactors

\zeta_{i}

for the different first-order time discretizations. Here,

\alpha\in(0,1)

is a freely learned scalar.

Explore similar work

Exact Flow Linear Attention: Exact Solution from Continuous-Time Dynamics
Dec 14, 2025Jingdi Lei, Di Zhang, Soujanya PoriaDynamical Systems Linear Attention
Nanyang Technological University · Fudan University
Selective Synchronization Attention
Feb 16, 2026Hasi HaysSelf-Attention Transformer Attention
Department of Chemical Engineering, University of Arkansas, Fayetteville, AR 72701, USA
Exact Linear Attention
May 13, 2026Weinuo OuEfficient Transformer Inference Self-Attention
Wuyi University

Dec 14, 2025J/K move · Enter open · S save

Exact Flow Linear Attention: Exact Solution from Continuous-Time Dynamics

Jingdi Lei, Di Zhang, Soujanya Poria

Nanyang Technological University · Fudan University

Dynamical Systems Linear Attention Delta-Rule Linear Attention Attention Mechanisms

Open paper AskPDF arXiv

In this paper, we introduce Exact Flow Linear Attention~(EFLA), an exact-flow formulation of delta-rule linear attention. We show that the delta-rule update can be interpreted as an explicit Euler discretization of an underlying continuous-time system. EFLA replaces this first-order update with the exact closed-form flow. By exploiting the rank-1 structure of the dynamics matrix, both the matrix exponential and the input integral collapse to a simple update that preserves delta-rule linear attention's algebraic structure, parameter count, linear-time complexity, and chunkwise parallelism. This attention mechanism removes the Euler discretization error of the delta-rule dynamics without introducing additional parameters. Experiments on robustness tests, language modeling benchmarks, and the MAD synthetic benchmark show that EFLA improves stability under corrupted and high-energy inputs, reduces perplexity, and achieves stronger downstream performance compared to SSM and Euler-style baselines. These results establish exact-flow integration as a principled and scalable update mechanism for delta-rule linear attention.

Abstract

Explore similar work

Exact Flow Linear Attention: Exact Solution from Continuous-Time Dynamics

Selective Synchronization Attention

Exact Linear Attention