SympFormer: Accelerated attention blocks via Inertial Dynamics on Density Manifolds
Authors: Viktor Stein, Wuchen Li, Gabriele Steidl
Organizations: Institute of Mathematics, Technische Universität Berlin, Straße des 17. Juni 136, 10623 Berlin, Germany · Department of Mathematics, University of South Carolina, Columbia. 1523 Greene St, Columbia, SC 29208, USA
Transformers owe much of their empirical success in natural language processing to the self-attention blocks. Recent perspectives interpret attention blocks as interacting particle systems, whose mean-field limits correspond to gradient flows of interaction energy functionals on probability density spaces equipped with Wasserstein-2-type metrics. We extend this viewpoint by introducing accelerated attention blocks derived from inertial Nesterov-type dynamics on density spaces. In our proposed architecture, tokens carry both spatial (feature) and velocity variables. The time discretization and the approximation of accelerated density dynamics yield Hamiltonian momentum attention blocks, which constitute the proposed accelerated attention architectures. In particular, for linear self-attention, we show that the attention blocks approximate a Stein variational gradient flow, using a bilinear kernel, of a potential energy. In this setting, we prove that elliptically contoured probability distributions are preserved by the accelerated attention blocks. We present implementable particle-based algorithms and demonstrate that the proposed accelerated attention blocks converge faster than the classical attention blocks while preserving the number of oracle calls.
Figures & tables
Figure 1: The accelerated attention block followed by an accelerated MLP block as described in Algorithm 1 .
model
last ( ↓ )
best ( ↓ )
wall clock time (seconds) ( ↓ )
Baseline
2.4687
2.4473
2409.4
YuriiFormer-Lie-Trotter
2.4041
2.3872
2835.2
Plain Euler
2.3247
2.3234
5036.1
Presymp Euler
2.4728
2.4592
4979.0
Presymp ExpEuler
2.3579
2.2523
5623.4
Presymp AB2
2.6653
2.6546
5249.4
Table 1: Validation loss on the tinystories data set after 10000 optimization steps.
Figure 2: Validation loss (circles) and training loss (lines) on the tinystories data set after 10000 optimization steps, illustrating the results from Table 1 .
model
last ( ↓ )
best ( ↓ )
wall clock time (seconds) ( ↓ )
Baseline
3.0830
2.961427
486.7
YuriiFormer
2.8783
2.765975
749.6
plain Euler
3.0399
2.920602
809.3
Presymp Euler
2.8467
2.731363
1172.8
Table 2: Validation loss of the SympFormer with linear attention (smaller configuration) on the TinyStories data set after 10000 optimization steps.
model
last ( ↓ )
best ( ↓ )
wall clock time (seconds) ( ↓ )
Baseline
3.0118
2.917168
8956.8
YuriiFormer
2.8486
2.765023
9601.3
plain Euler
2.7483
2.656692
10806.5
Presymp Euler
2.7862
2.690885
12620.0
Table 3: Validation loss of the SympFormer with linear attention (larger configuration) on the TinyStories data set after 10000 optimization steps.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
architecture
ζ1
ζ2
plain Euler
α
1
Presymp Euler
1−α(tk)hY
1
Presymp ExpEuler
e−Δηk
αeff1−e−Δηk
Appendix
Table 4: Scalar prefactors ζi for the different first-order time discretizations. Here, α∈(0,1) is a freely learned scalar.
In this paper, we introduce Exact Flow Linear Attention~(EFLA), an exact-flow formulation of delta-rule linear attention. We show that the delta-rule update can be interpreted as an explicit Euler discretization of an underlying continuous-time system. EFLA replaces this first-order update with the exact closed-form flow. By exploiting the rank-1 structure of the dynamics matrix, both the matrix exponential and the input integral collapse to a simple update that preserves delta-rule linear attention's algebraic structure, parameter count, linear-time complexity, and chunkwise parallelism. This attention mechanism removes the Euler discretization error of the delta-rule dynamics without introducing additional parameters. Experiments on robustness tests, language modeling benchmarks, and the MAD synthetic benchmark show that EFLA improves stability under corrupted and high-energy inputs, reduces perplexity, and achieves stronger downstream performance compared to SSM and Euler-style baselines. These results establish exact-flow integration as a principled and scalable update mechanism for delta-rule linear attention.
Jingdi Lei, Di Zhang, Soujanya Poria
Nanyang Technological University · Fudan University
The Transformer architecture has become the foundation of modern deep learning, yet its core self-attention mechanism suffers from quadratic computational complexity and lacks grounding in biological neural computation. We propose Selective Synchronization Attention (SSA), a novel attention mechanism that replaces the standard dot-product self-attention with a closed-form operator derived from the steady-state solution of the Kuramoto model of coupled oscillators. In SSA, each token is represented as an oscillator characterized by a learnable natural frequency and phase; the synchronization strength between token pairs, determined by a frequency-dependent coupling and phase-locking condition, serves as the attention weight. This formulation provides three key advantages: (i) natural sparsity arising from the phase-locking threshold, whereby tokens with incompatible frequencies automatically receive zero attention weight without explicit masking; (ii) unified positional-semantic encoding through the natural frequency spectrum, eliminating the need for separate positional encodings; and (iii) a single-pass, closed-form computation that avoids iterative ODE integration, with all components (coupling, order parameter, synchronization) derived from the oscillatory framework. We instantiate SSA within the Oscillatory Synchronization Network (OSN), a drop-in replacement for the Transformer block. Analysis of the synchronization matrices reveals non-uniform, head-diverse coupling patterns even at initialization, demonstrating a stronger architectural inductive bias than the approximately uniform attention produced by randomly initialized Transformers.
Hasi Hays
Department of Chemical Engineering, University of Arkansas, Fayetteville, AR 72701, USA
This paper introduces Exact Linear Attention (ELA), a mechanism that achieves linear computational complexity for Transformer attention by exploiting the exact decomposition property of kernel functions, thereby eliminating approximation error. We identify and address two key limitations of prior linear attention -- gradient explosion and token attention dilution -- by imposing kernel constraints that ensure non-negativity, discriminability, and geometric interpretability. Several kernel functions are proposed, including the Hadamard Exp Kernel, Summation Squared Euclidean Distance Kernel, and Subtraction Squared Euclidean Distance Kernel, each tailored for specific attention behaviors. Beyond the core attention formulation, the paper presents three engineering innovations: (1) a Hyper-Link structure that replaces traditional residual connections to mitigate gradient degradation; (2) a Memory Lobe module based on bidirectional linear attention, which captures "transformation flow" across layers to implement qualitative memory and an implicit reinforcement learning paradigm; and (3) a routing-score-based bias mechanism for Mixture-of-Experts (MoE) to improve interpretability and semantic alignment. Experimental results demonstrate that ELA achieves up to 6x faster decoding speed and 75% reduction in KV cache memory usage compared to full attention, while maintaining comparable or superior training performance. The proposed memory module accelerates convergence and enhances generalization. Furthermore, we extend the linear attention principle to vision models, yielding YOLO-LAT, which attains up to 4.3x GPU inference speedup and 7.9x parameter reduction with competitive detection accuracy. These results underline the broad applicability of exact linear attention for scaling Transformer models to ultra-long sequences and efficient visual tasks.