Dynamic Relational Priming Improves Transformer in Multivariate Time Series

Hunjae Lee; Corey Clark

cs.LGSep 15, 2025

Dynamic Relational Priming Improves Transformer in Multivariate Time Series

Authors: Hunjae Lee, Corey Clark

Organizations: Department of Computer Science, Southern Methodist University, Dallas TX USA · SMU Guildhall

Abstract

Standard attention mechanisms in transformers employ static token representations that remain unchanged across all pair-wise computations in each layer. This limits their representational alignment with the potentially diverse relational dynamics of each token-pair interaction. While they excel in domains with relatively homogeneous relationships, standard attention's static relational learning struggles to capture the diverse, heterogeneous inter-channel dependencies of multivariate time series (MTS) data--where different channel-pair interactions within a single system may be governed by entirely different physical laws or temporal dynamics. To better align the attention mechanism for such domain phenomena, we propose attention with dynamic relational priming (prime attention). Unlike standard attention where each token presents an identical representation across all of its pair-wise interactions, prime attention tailors each token dynamically (or per interaction) through learnable modulations to best capture the unique relational dynamics of each token pair, optimizing each pair-wise interaction for that specific relationship. This representational plasticity of prime attention enables effective extraction of relationship-specific information in MTS while maintaining the same asymptotic computational complexity as standard attention. Our results demonstrate that prime attention consistently outperforms standard attention across benchmarks, achieving up to 6.5% improvement in forecasting accuracy. In addition, we find that prime attention achieves comparable or superior performance using up to 40% less sequence length compared to standard attention, further demonstrating its superior relational modeling capabilities.

Figures & tables

Panel of Figure 1 — Figure 1: Comparing performance between standard (blue) and prime (yellow) attention at various input sequence length (or look-back window $L$ ). The horizontal target threshold line (green, dotted) represents standard attention’s performance at $L=96$ . Lower values indicate better performance.

Figure 2: Attention map comparison between standard and prime attention on the Solar dataset. Standard attention (left) shows an over-reliance on a single channel (displayed as a yellow vertical line) which may indicate that the model is prematurely converging to an easy solution, unable to capture more diverse and complex patterns. Prime attention (right) on the other hand shows its ability to act as a soft CI model, learning to prioritize self-signals along the diagonal instead of developing an over-reliance on a single channel.

Figure 3: Attention map comparison between standard (left) and prime (right) attention on the ECL dataset. Despite being comparable in performance on this dataset, prime attention exhibits a more diverse attention pattern than standard attention, learning to focus on a larger number of channels compared to standard attention. This can be seen from the larger number of vertical lines present in prime attention’s attention map.

Dataset	Std. Attn.	Std. Attn.	Prime Attn.
	(8 heads)	(9 heads)	(8 heads)
ETTh1	0.386	0.385	0.378
Weather	0.176	0.176	0.170
Solar	0.207	0.209	0.199

Table 3: Parameter effectiveness analysis. Best result (in MSE) in each row is shown in bold .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Dim	Prediction Length	Dataset Size	Frequency	Information
ETTh1	7	{96, 192, 336, 720}	(8545, 2881, 2881)	Hourly	Electricity
ETTh2	7	{96, 192, 336, 720}	(8545, 2881, 2881)	Hourly	Electricity
ETTm1	7	{96, 192, 336, 720}	(34465, 11521, 11521)	15min	Electricity
ETTm2	7	{96, 192, 336, 720}	(34465, 11521, 11521)	15min	Electricity
Exchange	8	{96, 192, 336, 720}	(5120, 665, 1422)	Daily	Economy
Weather	21	{96, 192, 336, 720}	(36792, 5271, 10540)	10min	Weather

Appendix

Table 4: Details of the datasets used in our experiments. The ETT datasets used training, validation, and testing split of 6:2:2 while the rest of the datasets used a 7:1:2 splits, in line with the experimental setups from ( Liu et al., 2024 ; Zhang and Yan, 2023 ) and other baselines.

Dataset	Model Hyper-Parameter				Training Process
	Model Dim	Layers	Dropout	Learning Rate	Loss	Batch Size	Epochs
ETTh1	256	2	0.1	0.0001	MSE	128	10
ETTh2	128	2	0.1	0.0001	MSE	128	10
ETTm1	128	2	0.2	0.0001	MSE	128	10
ETTm2	128	2	0.8	0.0001	MSE	128	10
Weather	128	3	0.4	0.0005	MSE	128	10

Appendix

Table 5: Model configurations across different datasets with iTransformer ( Liu et al., 2024 ) backbone. We keep the hyperparameters used with standard attention (with dropout being the only exception) in prime attention to ensure fair comparison.

Horizon	ETTh1	ETTh2	Weather	ECL	Traffic
96	$0.378\pm 0.001$	$0.297\pm 0.001$	$0.170\pm 0.001$	$0.146\pm 0.0005$	$0.395\pm 0.0001$
192	$0.432\pm 0.001$	$0.376\pm 0.001$	$0.218\pm 0.001$	$0.164\pm 0.001$	$0.417\pm 0.0002$
336	$0.473\pm 0.001$	$0.413\pm 0.001$	$0.275\pm 0.001$	$0.179\pm 0.001$	$0.431\pm 0.0001$
720	$0.479\pm 0.001$	$0.420\pm 0.0008$	$0.354\pm 0.0005$	$0.214\pm 0.002$	$0.463\pm 0.001$
Avg.	$0.441\pm 0.001$	$0.377\pm 0.001$	$0.254\pm 0.0008$	$0.176\pm 0.001$	$0.427\pm 0.0003$

Appendix

Table 6: Averaged forecasting results from 5 independent runs with standard deviation.

Panel of Figure 4 — Figure 4: Ablation study on sparsity of pair-wise modulator $\mathcal{F}$ . Sparsity refers to the amount (in percentage) of token pairs getting modulated (from left to right, 0% sparsity is full pair-wise modulation and 100% sparsity is equivalent to standard attention).

Figure 5: Effect that different initialization strategies for pair-wise modulation primer $\mathcal{F}$ has on performance (y-axis, lower is better) across prediction lengths (x-axis).

Figure 6: Analysis of memory (left) and computation time (right) for standard (blue) and prime (yellow) attention on the Weather dataset. The red star indicates observed equivalent performance (loss in MSE) between standard and prime attention.

Figure 7: Attention maps of standard attention (left), prime attention (middle), and their difference (right) on the Weather dataset. Warmer colors (yellow/green) reflect higher attention coefficient and colder colors (blue/purple) reflect lower attention coefficient.

Figure 8: Attention maps of standard attention (left), prime attention (middle), and their difference (right) on the Solar dataset. Warmer colors (yellow/green) reflect higher attention coefficient and colder colors (blue/purple) reflect lower attention coefficient.

Figure 9: Attention maps of standard attention (left), prime attention (middle), and their difference (right) on the Exchange dataset. Warmer colors (yellow/green) reflect higher attention coefficient and colder colors (blue/purple) reflect lower attention coefficient.

Figure 10: Attention maps of standard attention (left), prime attention (middle), and their difference (right) on the ECL dataset. Warmer colors (yellow/green) reflect higher attention coefficient and colder colors (blue/purple) reflect lower attention coefficient.

Panel of Figure 11 — Figure 11: Comparing performance between standard (blue) and prime (yellow) attention at various input sequence length (or look-back window $L$ ). The horizontal target threshold line (green, dotted) represents standard attention’s performance at $L=96$ .

	Std. Attn.	K only	V only	K and V
ETTh1	0.387	0.384	0.385	0.379
ETTh2	0.315	0.304	0.310	0.296
Weather	0.178	0.174	0.174	0.170

Appendix

Table 7: Effect of applying filter modulation to the keys (K), values (V), or both, compared to standard attention. Values are MSE (lower is better); the best result per dataset is in bold .

Figure 12: Forecasting visualization on ETTh1 dataset using predictions (in blue) made from standard attention (left) and prime attention (right) against ground-truth labels (in yellow). Each row represents a random batch.

Figure 13: Forecasting visualization on Solar dataset using predictions (in blue) made from standard attention (left) and prime attention (right) against ground-truth labels (in yellow). Each row represents a random batch.

Transformer Attention

Transformer attention lets models relate tokens by weighting information from other positions, enabling context-dependent sequence and multimodal processing.

See all 320 papers

Signed Dual Attention: Capturing Signed Dependencies in Time Series Forecasting
Jun 3, 2026Balthazar Courvoisier, Tristan CazenaveTime Series Forecasting Transformer Attention
Queensfield AI Technologies, Paris, France · LAMSADE, Universit´e Paris Dauphine - PSL, Paris, France
Beyond Similarity: Temporal Operator Attention for Time Series Analysis
May 11, 2026Jevon Twitty, Vinh Pham, Nitiwith Rotchanarak +4Time Series Classification Time Series Forecasting
Georgia Institute of Technology1
Beyond Static Positional Encoding: Bayesian Time-aware Attention for Non-Stationary Long-term Time Series Forecasting
Oct 14, 2026Xinjie Lan, Yunbo Mei, Leheng Zhou +1Long-Term Time Series Forecasting Non-Stationary Time Series Forecasting
Rice University · University of Hong Kong · University of North Carolina at Charlotte

Jun 3, 2026J/K move · Enter open · S save

Signed Dual Attention: Capturing Signed Dependencies in Time Series Forecasting

Balthazar Courvoisier, Tristan Cazenave

Queensfield AI Technologies, Paris, France · LAMSADE, Universit´e Paris Dauphine - PSL, Paris, France

Time Series Forecasting Transformer Attention Time Series Transformers

Open paper AskPDF arXiv

Initially developed for natural language processing, Transformer architectures and attention mechanisms are now central to a wide range of deep learning models, including applications in time series forecasting. A standard attention mechanism, however, implicitly assumes homophilic interactions, limiting its ability to model data with positive and negative dependencies, such as time series. In this work, we introduce the Signed Dual Attention, a novel attention formulation that captures both positive and negative relational patterns without additional parameters. By leveraging a dual message-passing scheme inspired by correlation structures, Signed Dual Attention propagates both supportive and contrastive information within a single shared block, effectively achieving the expressiveness of two head attention without additional parameters. This module can be seamlessly integrated into existing architectures and can yield performance gains in certain situations, requiring signed relational modeling. This approach opens a pathway toward more expressive and parameter-efficient transformers.

Abstract

Transformer Attention

Explore similar work

Signed Dual Attention: Capturing Signed Dependencies in Time Series Forecasting

Beyond Similarity: Temporal Operator Attention for Time Series Analysis

Beyond Static Positional Encoding: Bayesian Time-aware Attention for Non-Stationary Long-term Time Series Forecasting