Efficient Linear Attention for Multivariate Time Series Modeling via Entropy Calibration

Mingtao Zhang; Guoli Yang; Zhanxing Zhu; Mengzhu Wang; Xiaoying Bai

cs.LGNov 5, 2025

Efficient Linear Attention for Multivariate Time Series Modeling via Entropy Calibration

Authors: Mingtao Zhang, Guoli Yang, Zhanxing Zhu, Mengzhu Wang, Xiaoying Bai

Organizations: Advanced Institute of Big Data, Beijing · University of Southampton

Abstract

Attention mechanisms are effective for multivariate time series modeling because they adaptively aggregate information across timestamps and variables. Their quadratic cost, however, limits their use in multivariate and spatio-temporal systems. Linear attention reduces this cost by replacing nonlinear attention mappings with forms that support associative computation. Replacing exponential weighting may weaken the amplification of high-score positions and further bias weakly selective distributions toward uniform aggregation. We propose an entropy-calibrated linear attention module and instantiate it as the ELinFormer forecaster. Our operator decomposes aggregation into a uniform component and a query-dependent correction, whose magnitude is calibrated by a query-specific temperature. We derive a second-order KL proxy from centered query-key scores. The Taylor expansion reduces the proxy to the squared-score energy divided by the squared temperature. This score energy can be computed from shared products of the centered keys without forming the full pairwise score matrix, and the same products support the linear attention output. We design a smooth temperature rule to calibrate the KL proxy: it keeps low-energy residuals vanishing, approximately calibrates moderate-energy queries to a shared KL level, and allows the proxy to continue growing for high-energy queries. Experiments on five PEMS benchmarks show competitive forecasting accuracy. Controlled comparisons with vanilla attention and Linformer demonstrate a favorable accuracy--memory trade-off, while analysis on PEMS08 shows that the proposed rational temperature rule and query-specific calibration achieve lower MAE than a linear rule or shared-temperature alternatives. These results support entropy calibration as an effective mechanism for preserving selective deviations in linear attention.

Figures & tables

Fig. 1: Motivation and overview. (a) Illustrative diagnostic: a representative PEMS08 histogram of $\operatorname{KL}(a_{i}\|u)$ from the vanilla STAEformer backbone is concentrated in the low-value region, where $a_{i}$ is a softmax attention vector over $M$ positions and $u=M^{-1}\mathbf{1}$ is the uniform attention. The result indicates high entropy and weak selectivity for many sampled queries. (b) Backbone integration: the embedding, feed-forward, normalization, and prediction components of STAEformer are retained, while its temporal and spatial attention modules are replaced by the proposed operator. (c) Proposed module: centered query-key scores yield the squared-score energy $x_{i}$ . The normalized energy $z_{i}$ is mapped to a query-specific temperature $\tau_{i}$ , and $b(z_{i})=z_{i}/\tau_{i}^{2}$ is proportional to the KL proxy used for entropy calibration. Shared products of the centered keys then produce the large-axis linear attention output without forming the $M\times M$ pairwise attention matrix. The curve is smooth, and its labels describe the approximate behavior in low-, moderate-, and high-energy ranges.

Module	Time	Shared matrix state
Softmax attention	$O(M^{2}d)$	$O(M^{2})$
Proposed ( $M>d$ )	$O(Md^{2})$	$O(d^{2})$

TABLE I: Large-axis complexity per head. Shared matrix state excludes

O(Md)

token activations/intermediates and the proposed

O(M)

temperatures; it is not total training memory.

Dataset	#Nodes	#Edges	#Timesteps	Interval
PEMS-BAY	325	2369	52128	5 min
PEMS03	358	–	26208	5 min
PEMS04	307	340	16992	5 min
PEMS07	883	866	28224	5 min
PEMS08	170	295	17856	5 min

TABLE II: Dataset statistics.

Model	PEMS-BAY		PEMS03		PEMS04		PEMS07		PEMS08
Model	MAE	RMSE	MAE	RMSE	MAE	RMSE	MAE	RMSE	MAE	RMSE
STID [ 23 ]	–	–	15.37	25.10	18.68	30.61	20.07	33.39	14.55	23.85
M3-Net [ 24 ]	–	–	14.88	25.49	18.30	30.15	19.42	32.85	13.89	23.33
TSNN † [ 25 ]	–	–	15.21	24.44	19.28	31.69	20.37	34.56	14.57	24.80
AGCRN [ 14 ]	1.60	3.67	15.24	26.65	19.28	31.02	20.68	34.45	15.78	24.76
DCRNN [ 5 ]	1.59	3.69	15.54	27.18	19.66	31.18	21.16	34.15	15.23	24.17

TABLE III: Overall forecasting results on highway traffic datasets. Best results are in bold and second-best results are underlined. Some unavailable entries are omitted. † TSNN results are taken directly from the original paper and may not be fully comparable because they are not obtained from the common benchmark implementation.

Dataset	Model	MAE	RMSE	MAPE	Memory (MiB)	Time (s/epoch)
PEMS03	STAEformer	15.35	27.55	15.18%	8336	186
	Linformer	15.12	26.06	15.60%	5611	140
	NaLaFormer	14.75	26.13	15.00%	7155	157
	ELinFormer	14.81 ( $\downarrow$ 3.5%)	24.86 ( $\downarrow$ 9.8%)	15.21% ( $\uparrow$ 0.2%)	5278 ( $\downarrow$ 36.7%)	151 ( $\downarrow$ 18.8%)
PEMS04	STAEformer	18.22	30.18	11.98%	7096	101
	Linformer	18.39	30.15	12.10%	5217	79

TABLE IV: Controlled comparison in the same STAEformer backbone. Parenthesized values report relative changes from STAEformer. OOM denotes an out-of-memory failure, and “–” denotes an unavailable result.

Fig. 2: PEMS08 single-run temperature analyses (lower MAE is better). (a) Rational and matched linear mappings over the same five

\alpha

candidates under a common shortened schedule. (b) Five aggregation/adaptation variants, including dense softmax with the proposed temperature. “Shared

\tau

” is a single global mean computed on the validation split at the best

\alpha

. In the learnable-

\alpha

ablation, gradients are stopped through

z_{i}

and retained with respect to

\alpha

. The MAE axes display restricted ranges; exact values are listed in Supplement Tables S4 and S5.

Fig. 3: PEMS08 score-energy diagnostic. (a) Distributions of

z_{i}=x_{i}/(M\log M)

from

10{,}000

archived samples per layer: temporal

M=12

, spatial

M=170

. Thin spans indicate the 5th–95th percentiles, thick spans the interquartile range, and white circles the medians. (b) Analytical coefficient-deviation scale

b(z)=z/\tau^{2}(z)

on logarithmic axes; colored circles repeat the six layer medians from (a), showing where the sampled density is centered on the calibration curve. Shading and dashed guides indicate descriptive ranges separated at

\lambda\alpha/(1+\lambda)

and

\alpha

. The diagnostic uses

\alpha=1

\lambda=10^{-2}

, and

\beta=1

; the selected forecasting configuration remains

\alpha=1.33

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Specification
Operating system	Linux
CPU	12 vCPU Intel Xeon Platinum 8255C, 2.50 GHz
System memory	40 GB
GPU	NVIDIA RTX 3080, 10 GB VRAM
Python	3.8
PyTorch	1.10.0

Appendix

TABLE S1: Computing environment used for the locally reported experiments.

Dataset	Nodes	Edges	Time steps	Interval
PEMS-BAY	325	2369	52128	5 min
PEMS03	358	–	26208	5 min
PEMS04	307	340	16992	5 min
PEMS07	883	866	28224	5 min
PEMS08	170	295	17856	5 min

Appendix

TABLE S2: Dataset statistics, matching the main paper. A dash denotes unavailable graph-edge information.

Setting	Final value or selection procedure
Temperature floor	$\lambda=10^{-2}$ , fixed
Global scale	$\beta=1$ , fixed
$\alpha$ candidates	$\{0.50,1.00,1.33,1.50,2.00\}$
Selected $\alpha$	$1.33$ , selected by PEMS08 validation MAE
Cross-dataset use	The selected $\alpha$ is reused for the remaining PEMS datasets, heads, and queries
Backbone settings	Follow the corresponding STAEformer configurations unless explicitly changed

Appendix

TABLE S3: Method-specific final configurations.

$\alpha$	Rational $z/(z+\alpha)$	Linear $z/\alpha$
0.50	13.41	13.45
1.00	13.35	13.41
1.33	13.34	13.49
1.50	13.37	13.48
2.00	13.38	13.47

Appendix

TABLE S4: PEMS08 MAE for the rational and linear temperature mappings. Lower is better.

Variant	MAE
Proposed linear operator, rational temperature	13.34
Softmax attention with the proposed $\tau_{i}$	13.36
Learnable $\alpha$	13.44
Shared global $\tau$	13.57
Fixed $\tau=1$	13.51

Appendix

TABLE S5: Temperature-related PEMS08 ablations.

Fig. S1: Additional PEMS08 diagnostic at

\lambda=10^{-2}

\beta=1

, and reference

\alpha=1

. (a) Bounded analytical temperature rule. (b) Descriptive occupancy of the floor-dominated, calibration, and upper-tail ranges; the right column reports upper-tail percentages. Each layer contains

10{,}000

archived samples. The boundaries

z_{\mathrm{floor}}=\lambda\alpha/(1+\lambda)

and

z=\alpha

are visualization guides rather than hard model gates. The selected forecasting configuration remains

\alpha=1.33

Layer	Floor (%)	Transition (%)	Upper tail (%)	Median $\tau$
Temporal 1	79.38	20.62	0.00	0.1064
Temporal 2	21.99	77.93	0.08	0.1977
Temporal 3	5.47	93.45	1.08	0.3074
Spatial 1	55.59	44.41	0.00	0.1364
Spatial 2	56.10	43.90	0.00	0.1236
Spatial 3	76.08	23.92	0.00	0.1038

Appendix

TABLE S6: Descriptive occupancy of the PEMS08 diagnostic ranges. Each layer is a 10,000-value sample with replacement.

Linear Attention

Linear attention reduces sequence-length costs by replacing pairwise token interactions with kernel features, recurrent states, or compact memory.

See all 131 papers

May 11, 2026

Variational Linear Attention: Stable Associative Memory for Long-Context Transformers

Linear attention reduces the quadratic cost of softmax attention to

\mathcal{O}(T)

, but its memory state grows as

\mathcal{O}(T)

in Frobenius norm, causing progressive interference between stored associations. We introduce Variational Linear Attention (VLA), which reframes the memory update as an online regularised least-squares problem with an adaptive penalty matrix maintained via the Sherman-Morrison rank-1 formula. We prove that normalising the write direction to unit length gives the recurrence Jacobian spectral norm exactly

1

for all sequence lengths and head dimensions (Proposition 2), and that the state norm is self-limiting under bounded inputs (Proposition 1). Empirically, VLA reduces

\|S_t\|_F

109\times

relative to standard linear attention at

T{=}1{,}000

, achieves near-perfect exact-match accuracy on multi-query associative recall within the effective per-head memory regime (

n_\text{pairs} < d_h

), maintaining substantially higher retrieval performance than DeltaNet and standard linear attention under increasing memory load, and maintains 62% accuracy at the per-head capacity boundary. A Triton-fused kernel achieves

14\times

speedup over sequential Python and

\mathcal{O}(T)

scaling, crossing below softmax attention latency at approximately 43,000 tokens.

Vishal Pandey, Gopal Singh

Independent Researcher London, UK · Metriqual Athens, GR

Efficient Attention Linear Attention Associative Memory Memory-Efficient Attention Attention Approximation

Ask PDF

Jul 2, 2026

Self-Gating Attention for Efficient Time Series Forecasting

Transformer architectures have shown strong potential in time series forecasting, where multi-head self-attention is widely used to capture temporal dependencies across historical timestamps. However, standard self-attention has quadratic time and memory complexity with respect to the look-back length. This cost may limit its use in resource-constrained or high-throughput forecasting systems, where fast and memory-efficient inference is important. Through qualitative and quantitative analyses, we observe that self-attention maps in time series forecasting often contain redundant patterns across different timestamps. This phenomenon can be related to the repeated temporal patterns and relatively stable temporal correlations in many real-world time series. Motivated by this observation, we propose Self-Gating Attention (SGA), a plug-and-play attention mechanism that represents the attention score with a shared learnable matrix and an input-dependent residual component. The shared matrix captures common attention patterns, while the residual component captures input-dependent variations. In this way, SGA avoids the query and key projections used in standard attention score computation, leading to linear time and score-matrix memory complexity with respect to the look-back length. We integrate SGA into several forecasting backbones and compare it with standard self-attention and lightweight attention variants on nine publicly available real-world datasets covering electricity, finance, weather, medical monitoring, human activity, and climate records. The results show that SGA improves inference efficiency on public benchmarks while maintaining competitive forecasting performance against state-of-the-art attention mechanisms. These benchmark results provide deployment-oriented evidence.

Dezheng Wang, Tong Chen, Wei Yuan +3

School of Automation, Southeast University, 210096, China · School of Electrical Engineering and Computer Science, The University of Queensland, QLD 4072, Australia

Time Series Forecasting Linear Attention Attention Mechanisms Time Series Transformers

Ask PDF

Dec 14, 2025

Exact Flow Linear Attention: Exact Solution from Continuous-Time Dynamics

In this paper, we introduce Exact Flow Linear Attention~(EFLA), an exact-flow formulation of delta-rule linear attention. We show that the delta-rule update can be interpreted as an explicit Euler discretization of an underlying continuous-time system. EFLA replaces this first-order update with the exact closed-form flow. By exploiting the rank-1 structure of the dynamics matrix, both the matrix exponential and the input integral collapse to a simple update that preserves delta-rule linear attention's algebraic structure, parameter count, linear-time complexity, and chunkwise parallelism. This attention mechanism removes the Euler discretization error of the delta-rule dynamics without introducing additional parameters. Experiments on robustness tests, language modeling benchmarks, and the MAD synthetic benchmark show that EFLA improves stability under corrupted and high-energy inputs, reduces perplexity, and achieves stronger downstream performance compared to SSM and Euler-style baselines. These results establish exact-flow integration as a principled and scalable update mechanism for delta-rule linear attention.

Jingdi Lei, Di Zhang, Soujanya Poria

Nanyang Technological University · Fudan University

Dynamical Systems Linear Attention Delta-Rule Linear Attention Attention Mechanisms

Ask PDF

Abstract

Linear Attention

Explore similar work

Variational Linear Attention: Stable Associative Memory for Long-Context Transformers

Self-Gating Attention for Efficient Time Series Forecasting

Exact Flow Linear Attention: Exact Solution from Continuous-Time Dynamics