Exact Flow Linear Attention: Exact Solution from Continuous-Time Dynamics

Jingdi Lei; Di Zhang; Soujanya Poria

cs.LGDec 14, 2025

Exact Flow Linear Attention: Exact Solution from Continuous-Time Dynamics

Authors: Jingdi Lei, Di Zhang, Soujanya Poria

Organizations: Nanyang Technological University · Fudan University

Abstract

In this paper, we introduce Exact Flow Linear Attention~(EFLA), an exact-flow formulation of delta-rule linear attention. We show that the delta-rule update can be interpreted as an explicit Euler discretization of an underlying continuous-time system. EFLA replaces this first-order update with the exact closed-form flow. By exploiting the rank-1 structure of the dynamics matrix, both the matrix exponential and the input integral collapse to a simple update that preserves delta-rule linear attention's algebraic structure, parameter count, linear-time complexity, and chunkwise parallelism. This attention mechanism removes the Euler discretization error of the delta-rule dynamics without introducing additional parameters. Experiments on robustness tests, language modeling benchmarks, and the MAD synthetic benchmark show that EFLA improves stability under corrupted and high-energy inputs, reduces perplexity, and achieves stronger downstream performance compared to SSM and Euler-style baselines. These results establish exact-flow integration as a principled and scalable update mechanism for delta-rule linear attention.

Figures & tables

340M Parameters
	Perplexity ( $\downarrow$ )		Accuracy ( $\uparrow$ )
Model	Wiki.	LMB.	LMB.	PIQA	Hella.	Wino.	ARC-e	ARC-c	BoolQ	OBQA	SciQ	Avg.
	ppl	ppl	acc	acc	acc_n	acc	acc	acc_n	acc	acc_n	acc
Vanilla
Mamba-2	36.90	105.69	20.7	61.9	30.7	50.2	40.4	22.1	53.0	27.0	71.9	42.0
DeltaNet	38.09	96.26	22.5	60.7	30.1	51.9	41.7	21.6	60.1	29.0	70.4	42.2

Table 1 : Main language modeling results. The 340M models are trained for 8B tokens, whereas the 1.3B models are trained for 50B tokens on the same subset of the SlimPajama dataset ( Soboleva et al., 2023 ) with the Mistral ( Jiang et al., 2023 ) tokenizer. Perplexity: Lower (

\downarrow

) is better. Accuracy: Higher (

\uparrow

) is better. Best results are bolded.

Figure 1 : EFLA outperforms DeltaNet in both convergence speed and robustness on sMNIST. The plots illustrate training dynamics and robustness against dropout, intensity scaling, and additive Gaussian noise. EFLA maintains higher accuracy as perturbation intensity increases, especially under the larger learning-rate setting

3\times 10^{-3}

Panel of Figure 2 — Figure 2 : sMNIST results under different learning rates. EFLA maintains lower training loss and higher accuracy across the tested learning-rate range, indicating stronger robustness to learning-rate selection.

Explore similar work

MDN: Parallelizing Stepwise Momentum for Delta Linear Attention
May 7, 2026Yulong Huang, Xiang Liu, Hongxiang Huang +5Kimi Delta Attention Linear Attention
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
DeltaFlow: Noise-Adaptive Bidirectional Gated Delta Networks for Embedded Language Flows
Aug 2, 2026Guangfu Guo, Xiaoqian Lu, Linsey Pang +4Diffusion Language Models Kimi Delta Attention
1Clemson University · University of Ottawa · 3PayPal +1
Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
May 21, 2026Ali Hatamizadeh, Yejin Choi, Jan KautzKimi Delta Attention Gated Deltanet
NVIDIA

May 7, 2026, cs.LGJ/K move · Enter open · S save

MDN: Parallelizing Stepwise Momentum for Delta Linear Attention

Yulong Huang, Xiang Liu, Hongxiang Huang, Xiaopeng Lin+4

The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China

Kimi Delta Attention Linear Attention Large Language Model Training Momentum Stochastic Gradient Descent Gated Deltanet Momentum+1

Open paper AskPDF arXiv

Linear Attention (LA) offers a promising paradigm for scaling large language models (LLMs) to long sequences by avoiding the quadratic complexity of self-attention. Recent LA models such as Mamba2 and GDN interpret linear recurrences as closed-form online stochastic gradient descent (SGD), but naive SGD updates suffer from rapid information decay and suboptimal convergence in optimization. While momentum-based optimizers provide a natural remedy, they pose challenges in simultaneously achieving training efficiency and effectiveness. To address this, we develop a chunkwise parallel algorithm for LA with a stepwise momentum rule by geometrically reordering the update coefficients. Further, from a dynamical systems perspective, we analyze the momentum-based recurrence as a second-order system that introduces complex conjugate eigenvalues. This analysis guides the design of stable gating constraints. The resulting model, Momentum DeltaNet (MDN), leverages Triton kernels to achieve comparable training throughput with competitive linear models such as Mamba2 and KDA. Extensive experiments on the 400M and 1.3B parameter models demonstrate consistent performance improvements over strong baselines, including Transformers, Mamba2 and GDN, across diverse downstream evaluation benchmarks. Code: https://github.com/HuuYuLong/MomentumDeltaNet .

Abstract

Explore similar work

MDN: Parallelizing Stepwise Momentum for Delta Linear Attention

DeltaFlow: Noise-Adaptive Bidirectional Gated Delta Networks for Embedded Language Flows

Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention