Beyond Static Positional Encoding: Bayesian Time-aware Attention for Non-Stationary Long-term Time Series Forecasting

Xinjie Lan; Yunbo Mei; Leheng Zhou; Hao Qi

cs.LGOct 14, 2026

Beyond Static Positional Encoding: Bayesian Time-aware Attention for Non-Stationary Long-term Time Series Forecasting

Authors: Xinjie Lan, Yunbo Mei, Leheng Zhou, Hao Qi

Organizations: Rice University · University of Hong Kong · University of North Carolina at Charlotte

Abstract

Transformers have demonstrated strong performance in long-term time series forecasting (LTSF), yet modeling non-stationary temporal dependencies remains challenging. In this paper, we interpret self-attention as a Bayesian posterior over attended positions, combining a content-based prior induced by scaled dot-product similarity with a temporal-evidence likelihood induced by positional encoding. This formulation provides a probabilistic interpretation of how positional encoding shapes the temporal dependencies captured by self-attention. Building on this perspective, we derive a population-level evidence lower bound (ELBO) that characterizes the role of positional encoding in forecasting weakly stationary time series. Our analysis further highlights a limitation of static positional encoding: when temporal dependence structures vary across input windows, a fixed temporal bias must accommodate heterogeneous dynamics and can therefore be suboptimal for non-stationary forecasting. Motivated by this analysis, we propose Bayesian Time-aware Attention (BTA), which employs a mixture-of-experts (MoE) router to adaptively combine temporal kernels and generate input-conditioned, head-specific positional biases. Extensive experiments on standard LTSF benchmarks demonstrate that BTA consistently improves forecasting performance across multiple Transformer backbones and achieves competitive overall performance with modest computational overhead and stable training.

Figures & tables

Figure 1: BTA consists of two components: a probabilistic mixer and a MoE router. The probabilistic mixer defines an over-complete set of candidate temporal patterns using time-aware kernels

\psi^{(c)}\!\left(\tau^{(c)}_{ij};\bm{\theta}^{(c)}\right)

. The MoE router learns the kernel weights

\{\omega_{c}\}_{c=1}^{M}

with an MLP that processes the 1D temporal-evidence (TE) profile, obtained by aggregating the original 2D attention scores over relative lags, together with its Fourier spectrum. This plug-and-play design can be seamlessly integrated into standard multi-head self-attention.

Model		Informer ( Zhou et al., 2021 )						PatchTST ( Nie et al., 2023 )						Timer-XL ( Liu et al., 2025c )
Self-Attention		Vanilla		BTA		Gain(%)		Vanilla		BTA		Gain(%)		Vanilla		BTA		Gain(%)
Data	ADF	MSE	MAE	MSE	MAE	$\Delta$ MSE	$\Delta$ MAE	MSE	MAE	MSE	MAE	$\Delta$ MSE	$\Delta$ MAE	MSE	MAE	MSE	MAE	$\Delta$ MSE	$\Delta$ MAE
ETTh1	-5.91	1.096	0.835	0.981	0.769	10.50	7.88	0.516	0.477	0.425	0.429	17.51	10.02	0.409	0.430	0.379	0.407	7.28	5.35
ETTm1	-14.98	0.826	0.690	0.756	0.652	8.47	5.47	0.406	0.408	0.371	0.395	8.80	3.07	0.360	0.383	0.353	0.380	1.74	0.65
Weather	-26.68	0.463	0.459	0.398	0.424	14.05	7.73	0.265	0.286	0.254	0.278	4.34	2.80	0.240	0.273	0.235	0.266	1.98	2.48
Solar	-37.23	0.233	0.268	0.220	0.261	5.79	2.89	0.270	0.307	0.241	0.272	10.74	11.48	0.218	0.247	0.193	0.242	11.80	2.22

Table 1: BTA generalizes across three Transformer backbones . Input lengths are 96 for Informer and PatchTST and 672 for Timer-XL; backbone-specific settings are in Table 9 . Each dataset entry is averaged over

H\in\{96,192,336,720\}

. Gain(%) denotes the MSE and MAE decrease from Vanilla attention to BTA. Full results in Table 10 .

Model	Ours		PULSE		DTAF		DUET		TimeBridge		iTransformer		TimeMixer		PatchTST		TimesNet		DLinear		Stationary
Model	Ours		Liu et al. (2026)		Lu et al. (2026)		Qiu et al. (2025)		Liu et al. (2025a)		Liu et al. (2024)		Wang et al. (2024)		Nie et al. (2023)		Wu et al. (2023)		Zeng et al. (2023)		Liu et al. (2022)
Metric	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETTh1	0.394	0.423	0.412	0.422	0.416	0.435	0.404	0.427	0.399	0.426	0.439	0.448	0.427	0.441	0.413	0.434	0.488	0.452	0.423	0.437	0.644	0.554
ETTh2	0.338	0.382	0.361	0.391	0.363	0.396	0.344	0.390	0.344	0.384	0.374	0.406	0.349	0.397	0.351	0.395	0.414	0.427	0.431	0.447	0.521	0.487
ETTm1	0.337	0.371	0.363	0.388	0.338	0.368	0.336	0.370	0.348	0.379	0.362	0.391	0.356	0.380	0.353	0.381	0.399	0.406	0.357	0.379	0.527	0.472
ETTm2	0.246	0.305	0.262	0.314	0.249	0.307	0.255	0.316	0.250	0.307	0.269	0.329	0.257	0.318	0.256	0.317	0.314	0.333	0.267	0.332	0.567	0.462

Table 2: Comparison with reference methods for non-stationary LTSF. Ours denotes TimeBridge+BTA, and ELC denotes Electricity. Sequence length

T=720

for all models. Each dataset entry is averaged over

H\in\{96,192,336,720\}

. Best and second-best results are shown in bold and underlined, respectively. The 1st Count row counts dataset-level wins, including ties. Full results are available in Table 12 .

Figure 2: Input-adaptive temporal kernel selection in BTA. Two ETTm1 windows with a 720-step input and 96-step forecasting horizon. Top: TimeBridge and TimeBridge+BTA forecasts, together with the three highest-weight temporal kernels and their weighted combination. Bottom: MoE router weights over candidate kernels; for periodic kernels,

\ell

denotes the peak width

\ell^{\prime}_{r,e}p_{e}

. The left window emphasizes weekly periodicity, assigning

0.75

of the total weight to periodic kernels, whereas the right window emphasizes local dependence, assigning

0.66

to RBF kernels. BTA reduces the OT-channel MSE by approximately

37\%

in both cases (

0.219\rightarrow 0.139

and

0.970\rightarrow 0.608

Figure 3: BTA reveals horizon-dependent temporal dependencies. Mean kernel weights across ETTm1 test samples and attention heads for

H\in\{96,192,336,720\}

. The dashed line separates local RBF and periodic kernels. As the forecasting horizon changes, BTA shifts weight across temporal scales, with longer horizons emphasizing broader local and periodic patterns.

Figure 4: Hyperparameter sensitivity on ETTm1: Top-

k

(left) and router hidden dimension (right). Each panel shows the percentage MSE increase relative to its lowest MSE.

Attention		Self-attention		Bayesian Time-aware Attention
Optimization		Static		Static				MoE (TE)		MoE (TE + Fourier)
Kernels		None		Local		Local + periodic		Local + periodic		Local + periodic
Dataset	$H$	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Solar	96	0.161	0.224	0.158	0.219	0.157	0.213	0.152	0.201	0.150	0.198
	192	0.177	0.237	0.176	0.225	0.176	0.219	0.173	0.214	0.171	0.211
	336	0.196	0.232	0.192	0.230	0.191	0.227	0.187	0.227	0.184	0.226

Table 3: Ablation study of BTA on Solar (TimeBridge as the backbone). The full BTA variant is also reported as Ours in Table 12 .

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

PE method	Domain	$p(Z_{ij}=z_{ij}\mid S_{i}=j)$	$\log p(Z_{ij}=z_{ij}\mid S_{i}=j)$
Gaussian/RBF	NLP	$\displaystyle\propto\exp\!\left(-\frac{\Delta_{ij}^{2}}{2\sigma^{2}}\right)$	$\displaystyle-\frac{\Delta_{ij}^{2}}{2\sigma^{2}}$
ALiBi	NLP	$\displaystyle\propto\exp\!\left(-m_{h}\|\Delta_{ij}\|\right)$	$\displaystyle-m_{h}\|\Delta_{ij}\|$
KERPLE	NLP	$\displaystyle\propto\left(1+r_{2}\|\Delta_{ij}\|\right)^{-r_{1}}$	$\displaystyle-r_{1}\log\!\left(1+r_{2}\|\Delta_{ij}\|\right)$
Powerformer	TSF	$\displaystyle\propto\left(1+\|\Delta_{ij}\|\right)^{-\alpha}$	$\displaystyle-\alpha\log\!\left(1+\|\Delta_{ij}\|\right)$
RoPE	NLP & TSF	—	$\displaystyle\{\bm{q}_{i}^{\top}\left(\bm{R}_{\Delta_{ij}}-\bm{I}\right)\bm{k}_{j}\}/\sqrt{d_{k}}$

Appendix

Table 4: Probabilistic interpretations of representative PE methods, where

\Delta_{ij}=i-j

. Here,

\sigma^{2}>0

denotes the variance of the Gaussian/RBF kernel;

m_{h}>0

is the head-specific ALiBi slope;

r_{1},r_{2}>0

are KERPLE parameters controlling the magnitude and scale of its logarithmic decay, respectively; and

\alpha>0

is the power-law decay exponent in Powerformer. Log-likelihood terms are shown up to additive normalization constants that do not affect the softmax over

j

Figure 5: Synthetic binary-regime GP time series of 2,048 observations. Each 64-step window is generated from either a smooth RBF GP (unshaded) or a periodic GP (shaded). The periodic regime has period 24.

PE Method	Best epoch	Val. MSE	Test MSE	Smooth MSE	Periodic MSE
RPE	30	0.2578	0.2337	0.1792	0.2882
RoPE	30	0.2364	0.2124	0.1354	0.2893
ALiBi	30	0.2573	0.2336	0.1801	0.2871
BTA	30	0.2049	0.1788	0.0993	0.2583

Appendix

Table 5: The forecasting performance on the binary-regime GP dataset. MSE is reported on standardized observations; lower is better. Smooth and periodic denote the two subsets of the test set.

Figure 6: Forecasting results from the smooth RBF GP (top row) and periodic GP (bottom row). The left panels show all 64 historical observations and the four future values; the right panels magnify the forecast boundary. Gray lines denote the observed history, black markers the true future, and colored markers the predictions from the simplified Transformer with the four PE methods: RPE, ALiBi, RoPE, and BTA. The dashed line marks the end of the input window.

Figure 7: The figure compares positional biases under smooth and periodic Gaussian process regimes. RPE and ALiBi retain the same bias pattern across both regimes, while BTA produces different biases for the two inputs, illustrating its ability to adapt temporal evidence to the regime.

PE	Test MSE	Expected log likelihood	Mean KL	ELBO
RPE	0.2337	$-4.1431$	348.0968	$-352.2399$
ALiBi	0.2336	$-4.1429$	271.2189	$-275.3619$
RoPE	0.2124	$-4.1105$	51.4227	$-55.5333$
BTA	0.1788	$\mathbf{-4.0492}$	46.6581	$\mathbf{-50.7073}$

Appendix

Table 6: Forecasting performance and estimated ELBO on the binary-regime synthetic test set with

\epsilon_{q}=10^{-4}

. The Test MSE column repeats the point-forecast results in Table 5 . The remaining columns report the Monte Carlo diagnostics under sampled temporal evidence defined in Equation 65 ; expected log likelihood is not computed from the point-forecast MSE. Lower MSE and higher ELBO are better.

Dataset	$C$	Training	Validation	Testing	Interval	ADF stat.
ETTh1	7	8,545	2,881	2,881	1 hour	$-5.91$
ETTh2	7	8,545	2,881	2,881	1 hour	$-4.13$
ETTm1	7	34,465	11,521	11,521	15 min	$-14.98$
ETTm2	7	34,465	11,521	11,521	15 min	$-5.66$
Electricity	321	18,317	2,633	5,261	1 hour	$-8.44$
Traffic	862	12,185	1,757	3,509	1 hour	$-15.02$

Appendix

Table 7: Dataset characteristics and benchmark split-size summary, following Table 7 of TimeBridge ( Liu et al., 2025a ) .

C

denotes the number of variables. Training, Validation, Testing, and ADF entries reproduce that source. Split sizes are provided as benchmark metadata; the number of usable forecasting windows depends on the input length, prediction horizon, and handling of historical overlap. Interval denotes the sampling period.

Backbone	Informer ( Zhou et al., 2021 )	PatchTST ( Nie et al., 2023 )	TimeBridge ( Liu et al., 2025a )	Timer-XL ( Liu et al., 2025c )
Representative role	Early LTSF Transformer	Patch-based Transformer	Non-stationary Transformer	Pretrained TSFM
Architecture	Encoder–decoder	Encoder-only	Encoder-only	Decoder-only
Attention design	ProbSparse	Self-attention	Integrated + PatchDownsampling + Cointegrated	Causal TimeAttention
Tokenization	Point-wise	Patch-wise	Patch-wise	Patch-wise
Positional mechanism	Sinusoidal absolute PE	Learnable absolute PE	None	Rotary PE + variable bias
Training paradigm	Task-specific	Task-specific	Task-specific	Pretraining / task-specific

Appendix

Table 8: Comparison of the Transformer backbones used in our experiments.

Configuration	Informer	PatchTST	TimeBridge	Timer-XL
Token level	Point-wise	Patch-wise	Patch-wise	Patch-wise
Sequence length	96	96	720	672
Patch length	1	16	24 or 48	96
Input tokens	96	12	30 or 15	7
Lag unit	Time step	Patch	Patch	Patch
Kernel families	RBF + Periodic	RBF + Periodic	RBF + Periodic	RBF + Periodic

Appendix

Table 9: BTA configuration across the four Transformer backbones. Router hidden-dimension search ranges are reported as

(d_{\mathrm{start}},d_{\mathrm{ub}},\Delta d)

, where

d_{\mathrm{ub}}

is the search upper bound and

\Delta d

is the step size. For PatchTST, patch length

P=16

, stride

S=8

, and end padding give

N=\lfloor(T-P)/S\rfloor+2=12

input patches for

T=96

, following the patching convention of Nie et al. (2023) .

Model		Informer ( Zhou et al., 2021 )						PatchTST ( Nie et al., 2023 )						Timer-XL ( Liu et al., 2025c )
Self-Attention		Vanilla		BTA		Gain (%)		Vanilla		BTA		Gain (%)		Vanilla		BTA		Gain (%)
Dataset		MSE	MAE	MSE	MAE	$\Delta$ MSE	$\Delta$ MAE	MSE	MAE	MSE	MAE	$\Delta$ MSE	$\Delta$ MAE	MSE	MAE	MSE	MAE	$\Delta$ MSE	$\Delta$ MAE
ETTh1	96	0.992	0.785	0.878	0.719	11.49	8.41	0.460	0.447	0.367	0.394	20.22	11.86	0.364	0.397	0.343	0.379	5.77	4.53
	192	1.008	0.801	0.895	0.726	11.21	9.36	0.512	0.447	0.423	0.425	17.38	4.92	0.405	0.424	0.376	0.402	7.16	5.19
	336	1.179	0.870	1.038	0.793	11.96	8.85	0.546	0.496	0.441	0.436	19.23	12.10	0.427	0.439	0.391	0.413	8.43	5.92
	720	1.204	0.882	1.112	0.837	7.64	5.10	0.544	0.517	0.470	0.461	13.60	10.83	0.439	0.459	0.406	0.433	7.52	5.66

Appendix

Table 10: Per-horizon forecasting results of BTA across three Transformer backbones. Results are reported for

H\in\{96,192,336,720\}

. Within each backbone and row, the lower MSE and MAE between Vanilla and BTA are shown in bold . Avg. denotes the mean over the four horizons, and Gain (%) is the relative error reduction

100\times(\mathrm{Vanilla}-\mathrm{BTA})/\mathrm{Vanilla}

, computed from unrounded values for the Avg. rows. Backbone configurations are summarized in Table 9 .

Model		Informer ( Zhou et al., 2021 )		PatchTST ( Nie et al., 2023 )		Timer-XL ( Liu et al., 2025c )
Dataset	Horizon	Vanilla MSE	BTA MSE	Vanilla MSE	BTA MSE	Vanilla MSE	BTA MSE
ETTh1	96	$0.994\pm 0.003$	$\mathbf{0.876}\pm\mathbf{0.004}$	$0.462\pm 0.005$	$\mathbf{0.366}\pm\mathbf{0.004}$	$0.366\pm 0.005$	$\mathbf{0.345}\pm\mathbf{0.003}$
	192	$1.012\pm 0.012$	$\mathbf{0.895}\pm\mathbf{0.003}$	$0.511\pm 0.004$	$\mathbf{0.422}\pm\mathbf{0.006}$	$0.404\pm 0.011$	$\mathbf{0.379}\pm\mathbf{0.006}$
	336	$1.174\pm 0.009$	$\mathbf{1.036}\pm\mathbf{0.020}$	$0.548\pm 0.016$	$\mathbf{0.447}\pm\mathbf{0.012}$	$0.422\pm 0.018$	$\mathbf{0.397}\pm\mathbf{0.010}$
	720	$1.204\pm 0.001$	$\mathbf{1.111}\pm\mathbf{0.006}$	$0.543\pm 0.005$	$\mathbf{0.470}\pm\mathbf{0.002}$	$0.437\pm 0.006$	$\mathbf{0.406}\pm\mathbf{0.003}$
ETTm1	96	$0.672\pm 0.001$	$\mathbf{0.586}\pm\mathbf{0.001}$	$0.352\pm 0.001$	$\mathbf{0.315}\pm\mathbf{0.001}$	$0.290\pm 0.001$	$\mathbf{0.283}\pm\mathbf{0.001}$

Appendix

Table 11: Forecasting MSE across three Transformer backbones and three random seeds. Results are reported as mean

\pm

sample standard deviation for

H\in\{96,192,336,720\}

. Within each backbone, dataset, and horizon, the method with the lower mean MSE has both its mean and standard deviation shown in bold.

Model		Ours		PULSE		DTAF		DUET		TimeBridge		iTransformer		TimeMixer		PatchTST		TimesNet		DLinear		Stationary
Model		Ours		( Liu et al., 2026 )		( Lu et al., 2026 )		( Qiu et al., 2025 )		( Liu et al., 2025a )		( Liu et al., 2024 )		( Wang et al., 2024 )		( Nie et al., 2023 )		( Wu et al., 2023 )		( Zeng et al., 2023 )		( Liu et al., 2022 )
Metric		MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETTm1	96	0.275	0.330	0.303	0.347	0.277	0.329	0.279	0.336	0.284	0.337	0.300	0.353	0.293	0.345	0.293	0.346	0.338	0.375	0.299	0.343	0.393	0.404
	192	0.315	0.359	0.349	0.379	0.329	0.361	0.318	0.359	0.323	0.365	0.345	0.382	0.335	0.372	0.333	0.370	0.371	0.387	0.335	0.365	0.481	0.447
	336	0.351	0.385	0.375	0.398	0.345	0.374	0.345	0.378	0.370	0.394	0.374	0.398	0.368	0.386	0.369	0.390	0.410	0.411	0.369	0.386	0.555	0.489
	720	0.408	0.411	0.424	0.426	0.399	0.407	0.400	0.408	0.415	0.418	0.429	0.430	0.426	0.417	0.416	0.420	0.478	0.450	0.425	0.421	0.677	0.548

Appendix

Table 12: Per-horizon comparison for non-stationary LTSF. Ours denotes TimeBridge+BTA, enabling direct comparison with the original TimeBridge to isolate the contribution of BTA. All methods use input length

T=720

and prediction horizons

H\in\{96,192,336,720\}

. Avg. denotes the mean over the four horizons. The best and second-best results for each dataset, horizon, and metric are shown in bold and underlined , respectively. 1st Count reports the number of horizon-level settings in which each method achieves the best result, excluding Avg. rows and counting ties for all tied methods.

Figure 8: Input-adaptive positional bias on Solar. Two Solar windows with a 720-step input and 96-step forecasting horizon; panels follow Figure 2 . Both windows are dominated by periodic kernels, but BTA adapts their temporal scales to the input dynamics. The left window, with more irregular peak amplitudes, combines a

3

-day periodic kernel with

\ell=0.5

days (

w=0.31

), a long-range local kernel with

\ell=1.25

days (

w=0.16

), and a broader

3

-day periodic kernel with

\ell=2

days (

w=0.16

). In contrast, the right window concentrates its weight on narrower periodic kernels, led by

3

-day kernels with

\ell=0.33

and

0.5

days (

w=0.41

and

0.21

) and a

7

-day kernel with

\ell=0.33

days (

w=0.22

). The resulting mixtures induce distinct temporal-evidence profiles and positional biases. BTA reduces MSE by

58\%

and

34\%

, respectively (

22.48\rightarrow 9.41

and

1.46\rightarrow 0.96

Figure 9: Input-adaptive positional bias on Weather. Two Weather windows with a 720-step input and 720-step forecasting horizon; panels follow Figure 2 . The left window exhibits a recent level shift without a clear recurring pattern, for which BTA assigns

0.69

of the total weight to local RBF kernels, dominated by the long-range

\ell=1.25

-day kernel (

w=0.56

). In contrast, the right window exhibits stronger recurring structure and is dominated by a

3

-day periodic kernel with

\ell=2

days (

w=0.68

), while the long-range local kernel receives

w=0.16

. The contrasting kernel mixtures produce different temporal-evidence profiles and positional biases, illustrating how BTA adapts between local and periodic dependencies across input windows.

Figure 10: Input-adaptive positional bias in Timer-XL on ETTm1. Two 30-day input windows are used to forecast the following day. The top panels compare Timer-XL and Timer-XL+BTA predictions and visualize the dominant temporal kernels selected by BTA; the bottom panels show the corresponding router weights for layer 2, head 2. For the left window, which exhibits an abrupt recent change, BTA emphasizes short-range local kernels, with its three largest weights assigned to local scales of $0.234$ , $0.117$ , and $0.469$ days. For the more stable right window, BTA shifts toward substantially broader local dependencies together with a periodic component. These input-dependent mixtures induce different positional biases and reduce the forecasting MSE from $0.019$ to $0.012$ and from $0.031$ to $0.019$ , respectively, demonstrating that BTA can adapt the temporal dependencies of a pretrained Transformer to non-stationary input dynamics.

Figure 11: BTA reveals horizon-dependent temporal dependencies on Solar. Mean kernel weights across Solar test samples and attention heads for

H\in\{96,192,336,720\}

. The dashed line separates local RBF and periodic kernels. As the forecasting horizon increases, BTA shifts weight toward broader local dependencies and redistributes weight among periodic kernels, with the broadest local RBF kernel becoming dominant at

H=720

Figure 12: Additional hyperparameter sensitivity analysis on ETTh1 and Solar. The top and bottom rows report results on ETTh1 and Solar, respectively, with Top-

k

varied in the left column and router hidden dimension

d_{r}

varied in the right column. Each panel reports the relative MSE increase with respect to the lowest MSE within that panel and uses an independent vertical scale; filled markers indicate the minimum. Dashed curves denote the Top-

k

sweeps, for which auxiliary configuration differences prevent a strictly controlled comparison of

k

alone.

Model		Ours		TimeBridge		DUET		DTAF
Model		Ours		Liu et al. (2025a)		Qiu et al. (2025)		Lu et al. (2026)
Metric		FLOPs	Size	FLOPs	Size	FLOPs	Size	FLOPs	Size
ETTh1	96	274.9M	0.786M	274.6M	0.772M	35.7M	3.605M	19.0M	0.582M
	192	280.0M	1.155M	279.7M	1.141M	46.7M	5.132M	26.8M	1.121M
	336	287.8M	1.708M	287.5M	1.693M	65.4M	5.730M	38.4M	1.964M
	720	308.4M	3.182M	308.1M	3.168M	72.6M	6.981M	69.4M	4.176M

Appendix

Table 13: Computational efficiency on ETTh1, ETTm1, and Weather. Ours denotes TimeBridge+BTA. FLOPs are measured per input window, and Size denotes the number of trainable parameters. Results are reported for forecasting horizons

H\in\{96,192,336,720\}

Attention		Self-attention		Bayesian Time-aware Attention
Optimization		Static		Static				MoE (TE only)		MoE (TE+Fourier)
Kernels		None		Local		Local+Periodic		Local+Periodic		Local+Periodic
Metric		MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Weather	96	0.313	0.375	0.274	0.353	0.270	0.348	0.266	0.339	0.253	0.335
	192	0.377	0.408	0.333	0.394	0.320	0.384	0.313	0.377	0.299	0.369
	336	0.448	0.449	0.467	0.458	0.460	0.457	0.412	0.428	0.389	0.413

Appendix

Table 14: Ablation study of BTA on Weather with Informer as the backbone. Starting from vanilla self-attention, we progressively introduce local kernels, periodic kernels, input-adaptive MoE routing based on the temporal-evidence (TE) profile, and its Fourier spectrum. Gain (%) denotes the relative reduction in MSE and MAE with respect to vanilla self-attention.

Explore similar work

Jul 2, 2026

Self-Gating Attention for Efficient Time Series Forecasting

Transformer architectures have shown strong potential in time series forecasting, where multi-head self-attention is widely used to capture temporal dependencies across historical timestamps. However, standard self-attention has quadratic time and memory complexity with respect to the look-back length. This cost may limit its use in resource-constrained or high-throughput forecasting systems, where fast and memory-efficient inference is important. Through qualitative and quantitative analyses, we observe that self-attention maps in time series forecasting often contain redundant patterns across different timestamps. This phenomenon can be related to the repeated temporal patterns and relatively stable temporal correlations in many real-world time series. Motivated by this observation, we propose Self-Gating Attention (SGA), a plug-and-play attention mechanism that represents the attention score with a shared learnable matrix and an input-dependent residual component. The shared matrix captures common attention patterns, while the residual component captures input-dependent variations. In this way, SGA avoids the query and key projections used in standard attention score computation, leading to linear time and score-matrix memory complexity with respect to the look-back length. We integrate SGA into several forecasting backbones and compare it with standard self-attention and lightweight attention variants on nine publicly available real-world datasets covering electricity, finance, weather, medical monitoring, human activity, and climate records. The results show that SGA improves inference efficiency on public benchmarks while maintaining competitive forecasting performance against state-of-the-art attention mechanisms. These benchmark results provide deployment-oriented evidence.

Dezheng Wang, Tong Chen, Wei Yuan +3

School of Automation, Southeast University, 210096, China · School of Electrical Engineering and Computer Science, The University of Queensland, QLD 4072, Australia

Time Series Forecasting Linear Attention Attention Mechanisms Time Series Transformers

Ask PDF

May 11, 2026

Beyond Similarity: Temporal Operator Attention for Time Series Analysis

A persistent paradox in time-series forecasting is that structurally simple MLP and linear models often outperform high-capacity Transformers. We argue that this gap arises from a mismatch in the sequence-modeling primitive: while many time-series dynamics are governed by global temporal operators (e.g., filtering and harmonic structure), standard attention forms each output as a convex combination of inputs. This restricts its ability to represent signed and oscillatory transformations that are fundamental to temporal signal processing. We formalize this limitation as a simplex-constrained mixing bottleneck in softmax attention, which becomes especially restrictive for operator-driven time-series tasks. To address this, we propose

\textbf{Temporal Operator Attention (TOA)}

, a framework that augments attention with explicit, learnable sequence-space operators, enabling direct signed mixing across time while preserving input-dependent adaptivity. To make dense

N \times N

operators practical, we introduce Stochastic Operator Regularization, a high-variance dropout mechanism that stabilizes training and prevents trivial memorization. Across forecasting, anomaly detection, and classification benchmarks, TOA consistently improves performance when integrated into standard backbones such as PatchTST and iTransformer, with particularly strong gains in reconstruction-heavy tasks. These results suggest that explicit operator learning is a key ingredient for effective time-series modeling.

Jevon Twitty, Vinh Pham, Nitiwith Rotchanarak +4

Georgia Institute of Technology1

Time Series Classification Time Series Forecasting Temporal Attention

Ask PDF

Jun 3, 2026

Signed Dual Attention: Capturing Signed Dependencies in Time Series Forecasting

Initially developed for natural language processing, Transformer architectures and attention mechanisms are now central to a wide range of deep learning models, including applications in time series forecasting. A standard attention mechanism, however, implicitly assumes homophilic interactions, limiting its ability to model data with positive and negative dependencies, such as time series. In this work, we introduce the Signed Dual Attention, a novel attention formulation that captures both positive and negative relational patterns without additional parameters. By leveraging a dual message-passing scheme inspired by correlation structures, Signed Dual Attention propagates both supportive and contrastive information within a single shared block, effectively achieving the expressiveness of two head attention without additional parameters. This module can be seamlessly integrated into existing architectures and can yield performance gains in certain situations, requiring signed relational modeling. This approach opens a pathway toward more expressive and parameter-efficient transformers.

Balthazar Courvoisier, Tristan Cazenave

Queensfield AI Technologies, Paris, France · LAMSADE, Universit´e Paris Dauphine - PSL, Paris, France

Time Series Forecasting Transformer Attention Time Series Transformers

Ask PDF