This paper introduces Fuzzy Attention Network-Based Transformer (FANTF), an approach for time series forecasting, classification, and anomaly detection. FANTF combines fuzzy logic concepts with transformer architectures through a novel fuzzy attention mechanism. The proposed mechanism is inspired by Type-2 fuzzy set theory. It uses a learnable Gaussian uncertainty perturbation to model uncertainty in attention relevance scores. This makes it possible for the model to accurately capture uncertainty in temporal relationships better than normal deterministic attention. Transformers capacity to record complex temporal patterns and multivariate connections is enhanced by FANTF. The approach is both architecture-agnostic and task-agnostic. The proposed fuzzy attention module can be used as a direct replacement for the standard attention layer. To evaluate its effectiveness, FANTF is integrated into five transformer backbones: Transformer, Informer, PatchTST, Crossformer, and iTransformer. The fuzzy attention mechanism produces softer and more diverse attention distributions. It provides uncertainty-aware interpretations through attention heatmap visualizations. Experimental evaluations on some real-life datasets reveal that FANTF enhances the forecasting performance, classification, and anomaly detection tasks over traditional transformer-based models. FANTF achieves statistically significant and competitive performance improvements over standard transformer-based baselines on most benchmark datasets (p < 0.05, Wilcoxon signed-rank test), observed under noisy and non-stationary time series conditions.
Figures & tables
Fig. 1 : Overall approach of FANTF
Fig. 2 : Architecture of FANTF internal blocks
Q=SWQ,K=SWK,V=SWV
Algorithm 1 Learnable Fuzziness in Multi-Head Attention
Database
ETTh1
ETTh2
ETTm1
ETTm2
Weather
Electricity
Traffic
Exchange
Models
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
iTransformer
0.457
0.448
0.396
0.413
0.410
0.411
0.294
0.336
0.252
0.282
0.188
0.277
0.443
0.299
0.377
0.418
iTransformer+FAN
0.456
0.446
0.387
0.408
0.409
0.409
0.291
0.334
0.251
0.279
0.183
0.272
0.440
0.294
0.361
0.404
Difference (%)
0.22
0.45
2.27
1.21
0.24
0.49
1.02
0.60
0.40
1.06
2.66
1.81
1.81
1.67
0.68
3.35
Informer
1.058
0.808
4.665
1.772
0.890
0.702
1.716
0.903
0.628
0.548
0.362
0.439
0.862
0.487
1.622
1.005
Informer+FAN
0.964
0.780
3.249
1.440
0.854
0.702
1.553
0.873
0.633
0.551
0.278
0.373
0.659
0.355
1.438
0.931
TABLE I : An overview of the benchmark datasets’ outcomes. Before and after using FAN, average error coefficients (MSE, MAE) on multivariate long-term forecasting results are compared. The difference between the results is displayed in %, with the blue colors signifying that FANTF is superior to the current models. The negative growths are represented by hyphen.
Models
iTransformer+FAN
iTransformer
Crossformer+FAN
Crossformer
PatchTST+FAN
PatchTST
Database
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
PEMS03
0.249
0.350
0.238
0.343
0.175
0.286
0.157
0.273
0.300
0.382
0.309
0.383
PEMS04
0.218
0.319
0.218
0.319
0.189
0.302
0.149
0.270
0.312
0.386
0.313
0.387
PEMS07
0.231
0.334
0.244
0.349
0.175
0.272
0.179
0.273
0.292
0.376
0.298
0.379
PEMS08
0.230
0.316
0.237
0.323
0.220
0.274
0.223
0.276
0.272
0.349
0.268
0.350
TABLE II : Comparison of average error metrics (MSE, MAE) on the PEMS dataset for multivariate short-term forecasting, using a prediction length of 48 and a look-back window of 96. Values highlighted in red denote the improvements achieved by FANTF over existing results.
Metric
Category
iTransformer+FAN
iTransformer
PatchTST+FAN
PatchTST
Informer+FAN
Informer
Crossformer+FAN
Crossformer
Yearly
14.372
14.409
13.588
13.530
17.005
15.215
62.289
69.344
Quarterly
10.67
10.777
10.909
10.875
15.41
12.696
73.559
73.585
Monthly
14.296
16.650
14.149
14.431
19.075
15.210
69.795
69.80
Others
5.529
5.543
5.853
5.866
7.424
7.483
98.498
98.492
sMAPE
Average
13.005
14.170
12.827
14.942
17.137
14.306
71.017
72.038
Difference (%)
8.22
14.15
-
1.41
TABLE III : Summary of short-term forecasting results in the M4 dataset with a single variate. Every prediction length may be found in [6, 48]. The red-highlighted numbers show an improvement in FANTF above the current outcomes. The difference between the results is shown in % where the blue colours represent improvements of FANTF over existing models. The negative growths are represented by a hyphen.
Fig. 3 : Comparison of models efficiency in terms of forecasting
Fig. 4 : Prediction graphs on Exchange dataset (prediction length:96)
Fig. 5 : Prediction graphs on PEMS08 dataset for prediction length 48 for three FANTF approach
Fig. 6 : Models comparison with accuracy (%) in classification task on the 7 datasets of UEA.
Models
Informer+FAN
Informer
iTransformer+FAN
iTransformer
Crossformer+FAN
Crossformer
PatchTST+FAN
PatchTST
Transformer+FAN
Transformer
SMD
71.2
71.1
82.3
82.1
73.6
73.6
84.3
84.1
71.2
71.1
Difference (%)
0.14
0.24
0
0.24
0.14
MSL
81.8
81.2
72.8
72.4
81.3
80.8
78.5
79.1
81.6
80.8
Difference (%)
0.74
0.28
0.62
-
0.99
SMAP
73.3
68.9
67.1
66.9
67.2
67.5
67.2
67.2
72.8
73.0
Difference (%)
6.38
0.24
-
0
-
TABLE IV : Anomaly detection task. We calculate the F1-score (as %) for each dataset. A higher value of F1-score indicates better performance. The difference between the results is shown in % where the blue colours represent improvements of FANTF over existing models.
Transformers have achieved remarkable progress in time series forecasting, yet their reliance on deterministic dot-product attention limits their capacity to model uncertainty and nonlinear dependencies across multivariate temporal dimensions. To address this limitation, we propose FISFormer, a Fuzzy Inference System-driven Transformer that replaces conventional attention with a FIS Interaction mechanism. In this framework, each query-key pair undergoes a fuzzy inference process for every feature dimension, where learnable membership functions and rule-based reasoning estimate token-wise relational strengths. These FIS-derived interaction weights capture uncertainty and provide interpretable, continuous mappings between tokens. A softmax operation is applied along the token axis to normalize these weights, which are then combined with the corresponding value features through element-wise multiplication to yield the final context-enhanced token representations. This design fuses the interpretability and uncertainty modeling of fuzzy logic with the representational power of Transformers. Extensive experiments on multiple benchmark datasets demonstrate that FISFormer achieves superior forecasting accuracy, noise robustness, and interpretability compared to state-of-the-art Transformer variants, establishing fuzzy inference as an effective alternative to conventional attention mechanisms.
Bulent Haznedar, Levent Karacan
Computer Engineering Department, Gaziantep University, Gaziantep, Türkiye
A persistent paradox in time-series forecasting is that structurally simple MLP and linear models often outperform high-capacity Transformers. We argue that this gap arises from a mismatch in the sequence-modeling primitive: while many time-series dynamics are governed by global temporal operators (e.g., filtering and harmonic structure), standard attention forms each output as a convex combination of inputs. This restricts its ability to represent signed and oscillatory transformations that are fundamental to temporal signal processing. We formalize this limitation as a simplex-constrained mixing bottleneck in softmax attention, which becomes especially restrictive for operator-driven time-series tasks. To address this, we propose Temporal Operator Attention (TOA), a framework that augments attention with explicit, learnable sequence-space operators, enabling direct signed mixing across time while preserving input-dependent adaptivity. To make dense N×N operators practical, we introduce Stochastic Operator Regularization, a high-variance dropout mechanism that stabilizes training and prevents trivial memorization. Across forecasting, anomaly detection, and classification benchmarks, TOA consistently improves performance when integrated into standard backbones such as PatchTST and iTransformer, with particularly strong gains in reconstruction-heavy tasks. These results suggest that explicit operator learning is a key ingredient for effective time-series modeling.
Initially developed for natural language processing, Transformer architectures and attention mechanisms are now central to a wide range of deep learning models, including applications in time series forecasting. A standard attention mechanism, however, implicitly assumes homophilic interactions, limiting its ability to model data with positive and negative dependencies, such as time series. In this work, we introduce the Signed Dual Attention, a novel attention formulation that captures both positive and negative relational patterns without additional parameters. By leveraging a dual message-passing scheme inspired by correlation structures, Signed Dual Attention propagates both supportive and contrastive information within a single shared block, effectively achieving the expressiveness of two head attention without additional parameters. This module can be seamlessly integrated into existing architectures and can yield performance gains in certain situations, requiring signed relational modeling. This approach opens a pathway toward more expressive and parameter-efficient transformers.
Balthazar Courvoisier, Tristan Cazenave
Queensfield AI Technologies, Paris, France · LAMSADE, Universit´e Paris Dauphine - PSL, Paris, France