Transformer architectures have established strong baselines in time series forecasting, yet they typically rely on positional encodings that assume uniform, index-based temporal progression. However, real-world systems, from shifting financial cycles to elastic biological rhythms, frequently exhibit "time-warped" dynamics where the effective flow of time decouples from the sampling index. In this work, we first formalize this misalignment and prove that rotary position embedding (RoPE) is mathematically incapable of representing non-affine temporal warping. To address this, we propose Symplectic Positional Embeddings (SyPE), a learnable encoding framework derived from Hamiltonian mechanics. SyPE strictly generalizes RoPE by extending the rotation group SO(2) to the symplectic group Sp(2,R), modulated by a novel input-dependent adaptive warp module. By allowing the attention mechanism to adaptively dilate or contract temporal coordinates end-to-end, our approach captures locally varying periodicities without requiring pre-defined warping functions. We implement this mechanism in StretchTime, a multivariate forecasting architecture that achieves state-of-the-art performance on standard benchmarks, demonstrating superior robustness on datasets exhibiting non-stationary temporal dynamics.
Figures & tables
Figure 1: Visualization of Temporal Stretching
Figure 2: Overview of the SyPE-Augmented Transformer architecture.
Horizon
StretchTime
RoPE
w/o MLP
MSE
MAE
MSE
MAE
MSE
MAE
96
0.053
0.180
0.058
0.186
0.078
0.218
192
0.073
0.207
0.084
0.219
0.105
0.250
336
0.120
0.255
0.184
0.311
0.161
0.298
720
0.331
0.416
0.411
0.480
0.358
0.439
Avg
0.144
0.265
0.184
0.299
0.176
0.301
Table 1: Results on the synthetic dataset with temporally warped seasonal dynamics (see Section 3.2 ). We report MSE and MAE for varying prediction lengths ( H ). Best results are in bold and second best are underlined .
Figure 3: Forecast visualization on warped seasonal dynamics. StretchTime (left) corrects the phase alignment errors observed in the static RoPE baseline (right).
Model
StretchTime
RoPE
OLinear
TimeMixer++
TimeMixer
iTransformer
PatchTST
TimesNet
DLinear
Weather
0.237
0.244
0.237
0.226
0.240
0.258
0.265
0.259
0.265
Solar
0.195
0.214
0.215
0.203
0.216
0.233
0.287
0.403
0.330
ECL
0.174
0.179
0.159
0.165
0.182
0.178
0.216
0.192
0.225
ETTh1
0.424
0.449
0.424
0.419
0.447
0.454
0.516
0.458
0.461
ETTh2
0.384
0.401
0.367
0.339
0.364
0.383
0.391
0.414
0.563
ETTm1
0.367
0.379
0.374
0.369
0.381
0.407
0.406
0.400
0.404
Table 2: Summary of Multivariate TSF Results. Averaged test set MSE are reported. Best results are in bold and second best are underlined .
Horizon
StretchTime
TimeMixer++
TimeMixer
iTransformer
PatchTST
TimesNet
DLinear
FLOPs
Params
FLOPs
Params
FLOPs
Params
FLOPs
Params
FLOPs
Params
FLOPs
Params
FLOPs
Params
96
1.24G
471K
6.24G
1.19M
2.77G
1.13M
210M
9.56M
2.51G
10.1M
2.26G
1.19M
259K
18.6K
192
1.86G
480K
6.25G
1.2M
2.81G
1.14M
211M
9.61M
2.52G
10.6M
3.38G
1.2M
517K
37.2K
336
2.79G
494K
6.25G
1.22M
2.85G
1.17M
213M
9.68M
2.54G
11.5M
5.06G
1.22M
905K
65.2K
720
5.27G
530K
6.26G
1.27M
2.98G
1.24M
217M
9.88M
2.59G
13.9M
9.57G
1.25M
1.94M
140K
Avg Cost
2.79G
493.75K
6.25G
1.22M
2.85G
1.17M
212.75M
9.68M
2.54G
11.53M
5.07G
1.22M
905.25K
65.25K
Table 3: Computational efficiency comparison (FLOPs and Parameters) with Average Rank. Models are ordered to match the main results. Best results are in bold and second best are underlined .
V1
V2
V3
V4
V5
Dataset (Len)
Warp Only
Fixed RoPE
Learnable Symplectic
Fixed RoPE + Warp
Full StretchTime
PEMS04 (12)
0.075
0.072
0.072
0.073
0.071
PEMS04 (24)
0.082
0.080
0.078
0.079
0.081
PEMS04 (48)
0.097
0.093
0.094
0.095
0.090
PEMS04 (96)
0.119
0.118
0.118
0.115
0.112
PEMS04 Avg
0.093
0.091
0.091
0.090
0.089
Table 4: Factorized ablation study on stable traffic forecasting benchmarks (MSE). We decouple the adaptive warp module from the underlying positional geometry. Best results are in bold and second best are underlined .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Model
StretchTime
RoPE
OLinear
TimeMixer++
TimeMixer
iTransformer
PatchTST
TimesNet
DLinear
Dataset
Len
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Weather
96
0.143
0.203
0.146
0.206
0.153
0.190
0.155
0.205
0.163
0.209
0.174
0.214
0.186
0.227
0.172
0.220
0.195
0.252
192
0.196
0.258
0.201
0.262
0.200
0.235
0.201
0.245
0.208
0.250
0.221
0.254
0.234
0.265
0.219
0.261
0.237
0.295
336
0.253
0.304
0.270
0.317
0.258
0.280
0.237
0.265
0.251
0.287
0.278
0.296
0.284
0.301
0.280
0.306
0.282
0.331
720
0.355
0.373
0.360
0.374
0.337
0.333
0.312
0.334
0.339
0.341
0.358
0.347
0.356
0.349
0.365
0.359
0.345
0.382
Solar
96
0.158
0.241
0.190
0.268
0.179
0.191
0.171
0.231
0.189
0.259
0.203
0.237
0.265
0.323
0.373
0.358
0.290
0.378
Appendix
Table 5: Full Multivariate TSF Results. We report MSE and MAE. Best results are in bold and second best are underlined . AvgRank and #Top1 are calculated based on MSE across all horizons.
Model
StretchTime
RoPE
Pure Softmax
LinAttn+RoPE
Dataset
Len
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Weather
96
0.143
0.203
0.146
0.206
0.146
0.209
0.152
0.211
192
0.196
0.258
0.201
0.262
0.198
0.260
0.210
0.270
336
0.253
0.304
0.270
0.317
0.269
0.317
0.280
0.329
720
0.355
0.373
0.360
0.374
0.359
0.378
0.369
0.378
Solar
96
0.158
0.241
0.190
0.268
0.183
0.249
0.187
0.245
Appendix
Table 6: Ablation Study Results. Comparison of StretchTime with component variants. Best results are in bold and second best are underlined .
Dataset
Horizon
Seed 2024
Seed 2025
Seed 2026
Seed 2027
Seed 2028
Mean
Std. Dev
PEMS04
12
0.069
0.069
0.071
0.070
0.070
0.0698
0.0008
24
0.079
0.081
0.081
0.081
0.080
0.0804
0.0009
48
0.096
0.097
0.090
0.095
0.097
0.0950
0.0029
96
0.115
0.110
0.112
0.111
0.112
0.1120
0.0019
Appendix
Table 7: Robustness analysis of StretchTime on the PEMS04 dataset (MSE) across five random seeds. The low standard deviation confirms the stability of the performance improvements.
Time series forecasting predicts future values from ordered observations using statistical, neural, and probabilistic models to support decisions under uncertainty.
Transformers have demonstrated strong performance in long-term time series forecasting (LTSF), yet modeling non-stationary temporal dependencies remains challenging. In this paper, we interpret self-attention as a Bayesian posterior over attended positions, combining a content-based prior induced by scaled dot-product similarity with a temporal-evidence likelihood induced by positional encoding. This formulation provides a probabilistic interpretation of how positional encoding shapes the temporal dependencies captured by self-attention. Building on this perspective, we derive a population-level evidence lower bound (ELBO) that characterizes the role of positional encoding in forecasting weakly stationary time series. Our analysis further highlights a limitation of static positional encoding: when temporal dependence structures vary across input windows, a fixed temporal bias must accommodate heterogeneous dynamics and can therefore be suboptimal for non-stationary forecasting. Motivated by this analysis, we propose Bayesian Time-aware Attention (BTA), which employs a mixture-of-experts (MoE) router to adaptively combine temporal kernels and generate input-conditioned, head-specific positional biases. Extensive experiments on standard LTSF benchmarks demonstrate that BTA consistently improves forecasting performance across multiple Transformer backbones and achieves competitive overall performance with modest computational overhead and stable training.
Xinjie Lan, Yunbo Mei, Leheng Zhou +1
Rice University · University of Hong Kong · University of North Carolina at Charlotte
Time series forecasting serves as an essential tool for many real-world applications, supporting tasks such as resource optimization and decision-making. Despite significant architectural advancements, most modern models still treat forecasting task as a fixed mapping from history to target horizons. This induces temporal decoupling across future time points and limits the model's ability to adapt to the evolving context as forecasting progresses. In this work, we present LeapTS, a novel framework that reformulates time series forecasting as a dynamic scheduling process over the prediction horizon. Specifically, LeapTS organizes the forecasting process into multi-level decisions using: (1) the hierarchical controller to dynamically select the optimal prediction scale and advancement length at each step, and (2) continuous-time state evolution driven by neural controlled differential equations. Within this process, the controlled update mechanism explicitly couples the irregular temporal dynamics with discrete scheduling feedback. Extensive evaluations on both real-world and synthetic datasets demonstrate that LeapTS improves overall forecasting performance by at least 7.4% while achieving a 2.6× to 5.3× inference speedup over representative Transformer-based models. Furthermore, by explicitly tracing the scheduling trajectories, we reveal how the model autonomously adapts its forecasting behavior to capture non-stationary dynamics.
Transformer-based time series foundation models face a fundamental trade-off in choice of tokenization: point-wise embeddings preserve temporal fidelity but scale poorly with sequence length, whereas fixed-length patching improves efficiency by imposing uniform boundaries that may disrupt natural transitions and blur informative local dynamics. In order to address these limitations, we introduce TimeSqueeze, a dynamic patching mechanism that adaptively selects patch boundaries within each sequence based on local signal complexity. TimeSqueeze first applies a lightweight state-space encoder to extract full-resolution point-wise features, then performs content-aware segmentation by allocating short patches to information-dense regions and long patches to smooth or redundant segments. This variable-resolution compression preserves critical temporal structure while substantially reducing the token sequence presented to the Transformer backbone. Specifically for large-scale pretraining, TimeSqueeze attains up to 20x faster convergence and 8x higher data efficiency compared to equivalent point-token baselines. Experiments across long-horizon forecasting benchmarks show that TimeSqueeze consistently outperforms comparable architectures that use either point-wise tokenization or fixed-size patching.
Sravan Kumar Ankireddy, Nikita Seleznev, Nam H. Nguyen +4
University of Texas at Austin, Austin, TX, USA · Capital One, USA · University of Maryland, College Park, MD, USA