Standard attention mechanisms in transformers employ static token representations that remain unchanged across all pair-wise computations in each layer. This limits their representational alignment with the potentially diverse relational dynamics of each token-pair interaction. While they excel in domains with relatively homogeneous relationships, standard attention's static relational learning struggles to capture the diverse, heterogeneous inter-channel dependencies of multivariate time series (MTS) data--where different channel-pair interactions within a single system may be governed by entirely different physical laws or temporal dynamics. To better align the attention mechanism for such domain phenomena, we propose attention with dynamic relational priming (prime attention). Unlike standard attention where each token presents an identical representation across all of its pair-wise interactions, prime attention tailors each token dynamically (or per interaction) through learnable modulations to best capture the unique relational dynamics of each token pair, optimizing each pair-wise interaction for that specific relationship. This representational plasticity of prime attention enables effective extraction of relationship-specific information in MTS while maintaining the same asymptotic computational complexity as standard attention. Our results demonstrate that prime attention consistently outperforms standard attention across benchmarks, achieving up to 6.5% improvement in forecasting accuracy. In addition, we find that prime attention achieves comparable or superior performance using up to 40% less sequence length compared to standard attention, further demonstrating its superior relational modeling capabilities.
Figures & tables
Figure 1: Comparing performance between standard (blue) and prime (yellow) attention at various input sequence length (or look-back window L ). The horizontal target threshold line (green, dotted) represents standard attention’s performance at L=96 . Lower values indicate better performance.
Figure 2: Attention map comparison between standard and prime attention on the Solar dataset. Standard attention (left) shows an over-reliance on a single channel (displayed as a yellow vertical line) which may indicate that the model is prematurely converging to an easy solution, unable to capture more diverse and complex patterns. Prime attention (right) on the other hand shows its ability to act as a soft CI model, learning to prioritize self-signals along the diagonal instead of developing an over-reliance on a single channel.
Figure 3: Attention map comparison between standard (left) and prime (right) attention on the ECL dataset. Despite being comparable in performance on this dataset, prime attention exhibits a more diverse attention pattern than standard attention, learning to focus on a larger number of channels compared to standard attention. This can be seen from the larger number of vertical lines present in prime attention’s attention map.
Dataset
Std. Attn.
Std. Attn.
Prime Attn.
(8 heads)
(9 heads)
(8 heads)
ETTh1
0.386
0.385
0.378
Weather
0.176
0.176
0.170
Solar
0.207
0.209
0.199
Table 3: Parameter effectiveness analysis. Best result (in MSE) in each row is shown in bold .
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Dim
Prediction Length
Dataset Size
Frequency
Information
ETTh1
7
{96, 192, 336, 720}
(8545, 2881, 2881)
Hourly
Electricity
ETTh2
7
{96, 192, 336, 720}
(8545, 2881, 2881)
Hourly
Electricity
ETTm1
7
{96, 192, 336, 720}
(34465, 11521, 11521)
15min
Electricity
ETTm2
7
{96, 192, 336, 720}
(34465, 11521, 11521)
15min
Electricity
Exchange
8
{96, 192, 336, 720}
(5120, 665, 1422)
Daily
Economy
Weather
21
{96, 192, 336, 720}
(36792, 5271, 10540)
10min
Weather
Appendix
Table 4: Details of the datasets used in our experiments. The ETT datasets used training, validation, and testing split of 6:2:2 while the rest of the datasets used a 7:1:2 splits, in line with the experimental setups from ( Liu et al., 2024 ; Zhang and Yan, 2023 ) and other baselines.
Dataset
Model Hyper-Parameter
Training Process
Model Dim
Layers
Dropout
Learning Rate
Loss
Batch Size
Epochs
ETTh1
256
2
0.1
0.0001
MSE
128
10
ETTh2
128
2
0.1
0.0001
MSE
128
10
ETTm1
128
2
0.2
0.0001
MSE
128
10
ETTm2
128
2
0.8
0.0001
MSE
128
10
Weather
128
3
0.4
0.0005
MSE
128
10
Appendix
Table 5: Model configurations across different datasets with iTransformer ( Liu et al., 2024 ) backbone. We keep the hyperparameters used with standard attention (with dropout being the only exception) in prime attention to ensure fair comparison.
Horizon
ETTh1
ETTh2
Weather
ECL
Traffic
96
0.378±0.001
0.297±0.001
0.170±0.001
0.146±0.0005
0.395±0.0001
192
0.432±0.001
0.376±0.001
0.218±0.001
0.164±0.001
0.417±0.0002
336
0.473±0.001
0.413±0.001
0.275±0.001
0.179±0.001
0.431±0.0001
720
0.479±0.001
0.420±0.0008
0.354±0.0005
0.214±0.002
0.463±0.001
Avg.
0.441±0.001
0.377±0.001
0.254±0.0008
0.176±0.001
0.427±0.0003
Appendix
Table 6: Averaged forecasting results from 5 independent runs with standard deviation.
Figure 4: Ablation study on sparsity of pair-wise modulator F . Sparsity refers to the amount (in percentage) of token pairs getting modulated (from left to right, 0% sparsity is full pair-wise modulation and 100% sparsity is equivalent to standard attention).
Figure 5: Effect that different initialization strategies for pair-wise modulation primer F has on performance (y-axis, lower is better) across prediction lengths (x-axis).
Figure 6: Analysis of memory (left) and computation time (right) for standard (blue) and prime (yellow) attention on the Weather dataset. The red star indicates observed equivalent performance (loss in MSE) between standard and prime attention.
Figure 7: Attention maps of standard attention (left), prime attention (middle), and their difference (right) on the Weather dataset. Warmer colors (yellow/green) reflect higher attention coefficient and colder colors (blue/purple) reflect lower attention coefficient.
Figure 8: Attention maps of standard attention (left), prime attention (middle), and their difference (right) on the Solar dataset. Warmer colors (yellow/green) reflect higher attention coefficient and colder colors (blue/purple) reflect lower attention coefficient.
Figure 9: Attention maps of standard attention (left), prime attention (middle), and their difference (right) on the Exchange dataset. Warmer colors (yellow/green) reflect higher attention coefficient and colder colors (blue/purple) reflect lower attention coefficient.
Figure 10: Attention maps of standard attention (left), prime attention (middle), and their difference (right) on the ECL dataset. Warmer colors (yellow/green) reflect higher attention coefficient and colder colors (blue/purple) reflect lower attention coefficient.
Figure 11: Comparing performance between standard (blue) and prime (yellow) attention at various input sequence length (or look-back window L ). The horizontal target threshold line (green, dotted) represents standard attention’s performance at L=96 .
Std. Attn.
K only
V only
K and V
ETTh1
0.387
0.384
0.385
0.379
ETTh2
0.315
0.304
0.310
0.296
Weather
0.178
0.174
0.174
0.170
Appendix
Table 7: Effect of applying filter modulation to the keys (K), values (V), or both, compared to standard attention. Values are MSE (lower is better); the best result per dataset is in bold .
Figure 12: Forecasting visualization on ETTh1 dataset using predictions (in blue) made from standard attention (left) and prime attention (right) against ground-truth labels (in yellow). Each row represents a random batch.
Figure 13: Forecasting visualization on Solar dataset using predictions (in blue) made from standard attention (left) and prime attention (right) against ground-truth labels (in yellow). Each row represents a random batch.
Transformer attention lets models relate tokens by weighting information from other positions, enabling context-dependent sequence and multimodal processing.
Initially developed for natural language processing, Transformer architectures and attention mechanisms are now central to a wide range of deep learning models, including applications in time series forecasting. A standard attention mechanism, however, implicitly assumes homophilic interactions, limiting its ability to model data with positive and negative dependencies, such as time series. In this work, we introduce the Signed Dual Attention, a novel attention formulation that captures both positive and negative relational patterns without additional parameters. By leveraging a dual message-passing scheme inspired by correlation structures, Signed Dual Attention propagates both supportive and contrastive information within a single shared block, effectively achieving the expressiveness of two head attention without additional parameters. This module can be seamlessly integrated into existing architectures and can yield performance gains in certain situations, requiring signed relational modeling. This approach opens a pathway toward more expressive and parameter-efficient transformers.
Balthazar Courvoisier, Tristan Cazenave
Queensfield AI Technologies, Paris, France · LAMSADE, Universit´e Paris Dauphine - PSL, Paris, France
A persistent paradox in time-series forecasting is that structurally simple MLP and linear models often outperform high-capacity Transformers. We argue that this gap arises from a mismatch in the sequence-modeling primitive: while many time-series dynamics are governed by global temporal operators (e.g., filtering and harmonic structure), standard attention forms each output as a convex combination of inputs. This restricts its ability to represent signed and oscillatory transformations that are fundamental to temporal signal processing. We formalize this limitation as a simplex-constrained mixing bottleneck in softmax attention, which becomes especially restrictive for operator-driven time-series tasks. To address this, we propose Temporal Operator Attention (TOA), a framework that augments attention with explicit, learnable sequence-space operators, enabling direct signed mixing across time while preserving input-dependent adaptivity. To make dense N×N operators practical, we introduce Stochastic Operator Regularization, a high-variance dropout mechanism that stabilizes training and prevents trivial memorization. Across forecasting, anomaly detection, and classification benchmarks, TOA consistently improves performance when integrated into standard backbones such as PatchTST and iTransformer, with particularly strong gains in reconstruction-heavy tasks. These results suggest that explicit operator learning is a key ingredient for effective time-series modeling.
Transformers have demonstrated strong performance in long-term time series forecasting (LTSF), yet modeling non-stationary temporal dependencies remains challenging. In this paper, we interpret self-attention as a Bayesian posterior over attended positions, combining a content-based prior induced by scaled dot-product similarity with a temporal-evidence likelihood induced by positional encoding. This formulation provides a probabilistic interpretation of how positional encoding shapes the temporal dependencies captured by self-attention. Building on this perspective, we derive a population-level evidence lower bound (ELBO) that characterizes the role of positional encoding in forecasting weakly stationary time series. Our analysis further highlights a limitation of static positional encoding: when temporal dependence structures vary across input windows, a fixed temporal bias must accommodate heterogeneous dynamics and can therefore be suboptimal for non-stationary forecasting. Motivated by this analysis, we propose Bayesian Time-aware Attention (BTA), which employs a mixture-of-experts (MoE) router to adaptively combine temporal kernels and generate input-conditioned, head-specific positional biases. Extensive experiments on standard LTSF benchmarks demonstrate that BTA consistently improves forecasting performance across multiple Transformer backbones and achieves competitive overall performance with modest computational overhead and stable training.
Xinjie Lan, Yunbo Mei, Leheng Zhou +1
Rice University · University of Hong Kong · University of North Carolina at Charlotte