FISformer: Replacing Self-Attention with a Fuzzy Inference System in Transformer Models for Time Series Forecasting

Bulent Haznedar; Levent Karacan

cs.LGMar 23, 2026

FISformer: Replacing Self-Attention with a Fuzzy Inference System in Transformer Models for Time Series Forecasting

Authors: Bulent Haznedar, Levent Karacan

Organizations: Computer Engineering Department, Gaziantep University, Gaziantep, Türkiye

Abstract

Transformers have achieved remarkable progress in time series forecasting, yet their reliance on deterministic dot-product attention limits their capacity to model uncertainty and nonlinear dependencies across multivariate temporal dimensions. To address this limitation, we propose FISFormer, a Fuzzy Inference System-driven Transformer that replaces conventional attention with a FIS Interaction mechanism. In this framework, each query-key pair undergoes a fuzzy inference process for every feature dimension, where learnable membership functions and rule-based reasoning estimate token-wise relational strengths. These FIS-derived interaction weights capture uncertainty and provide interpretable, continuous mappings between tokens. A softmax operation is applied along the token axis to normalize these weights, which are then combined with the corresponding value features through element-wise multiplication to yield the final context-enhanced token representations. This design fuses the interpretability and uncertainty modeling of fuzzy logic with the representational power of Transformers. Extensive experiments on multiple benchmark datasets demonstrate that FISFormer achieves superior forecasting accuracy, noise robustness, and interpretability compared to state-of-the-art Transformer variants, establishing fuzzy inference as an effective alternative to conventional attention mechanisms.

Figures & tables

Fig. 1: Overall architecture of the proposed model, built upon a Transformer encoder. It comprises an embedding layer to convert multivariate input sequences into dense representations, a Transformer block to capture temporal and feature-wise dependencies via the proposed FIS-based Token Interaction module, and a projection module to map encoded features to the prediction output.

Fig. 2: FIS-based Token Interaction. Query and key tokens are fuzzified through learnable Gaussian membership functions and combined via rule-based fuzzy inference. The resulting fuzzy rules are defuzzified following a first-order Sugeno process to produce the Fuzzy Interaction Map.

Fig. 3: FIS Interaction Output. The Fuzzy Interaction Map encodes inferred relational strengths across all token dimensions. A softmax operation is applied along the token axis to obtain normalized interaction weights, which are then combined with value representations via element-wise multiplication to produce context-enhanced token embeddings.

Dataset	Dim	Prediction Length	Dataset Size	Frequency	Information
ETTh1, ETTh2	7	{96, 192, 336, 720}	(8545, 2881, 2881)	Hourly	Electricity
ETTm1, ETTm2	7	{96, 192, 336, 720}	(34465, 11521, 11521)	15min	Electricity
Exchange	8	{96, 192, 336, 720}	(5120, 665, 1422)	Daily	Economy
Weather	21	{96, 192, 336, 720}	(36792, 5271, 10540)	10min	Weather
ECL	321	{96, 192, 336, 720}	(18317, 2633, 5261)	Hourly	Electricity
Traffic	862	{96, 192, 336, 720}	(12185, 1757, 3509)	Hourly	Transportation

TABLE I: Detailed dataset descriptions including input dimensions, number of samples, prediction sequence length, and time frequency.

Models	FISformer		ITransformer		Crossformer		Fedformer		Autoformer		Informer
Models	(Ours)		(2024)		(2023)		(2022)		(2021)		(2021)
Metric	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ECL	0.172	0.262	0.178	0.270	0.244	0.334	0.214	0.327	0.227	0.338	0.392	0.448
ETT (Avg)	0.375	0.395	0.383	0.399	0.685	0.578	0.407	0.428	0.465	0.459	1.342	0.859
Exchange	0.356	0.402	0.360	0.403	0.940	0.707	0.519	0.429	0.613	0.539	-	-
Traffic	0.417	0.279	0.428	0.282	0.550	0.304	0.610	0.376	0.628	0.379	-	-

TABLE II: Average forecasting performance in terms of MSE and MAE across different prediction lengths. The results are reported for sequence lengths {12, 24, 48, 96} on the PEMS dataset, and {96, 192, 336, 720} on the other datasets. The best results are highlighted in red , and the second-best results are underlined blue .

Models		FISformer		ITransformer		Crossformer		Fedformer		Autoformer		Informer
Models		(Ours)		(2024)		(2023)		(2022)		(2021)		(2021)
Metric		MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETTh1	96	0.378	0.398	0.386	0.405	0.423	0.448	0.376	0.419	0.449	0.459	-	-
	192	0.427	0.427	0.441	0.436	0.471	0.474	0.420	0.448	0.500	0.482	-	-
	336	0.474	0.454	0.487	0.458	0.570	0.546	0.459	0.465	0.521	0.496	1.128	0.873
	720	0.478	0.475	0.503	0.491	0.653	0.621	0.506	0.507	0.514	0.512	1.215	0.896

TABLE III: Detailed forecasting comparison in terms of MSE and MAE across different prediction lengths. Results are reported for sequence lengths {96, 192, 336, 720} on the ETT, ECL, Exchange, Traffic, Weather, and Solar-Energy datasets. The best results are highlighted in red , and the second-best results are underlined blue .

Models		FISformer		ITransformer		Crossformer		Fedformer		Autoformer
Models		(Ours)		(2024)		(2023)		(2022)		(2021)
Metric		MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
PEMS03	12	0.068	0.174	0.071	0.174	0.090	0.203	0.126	0.251	0.272	0.385
	24	0.095	0.204	0.093	0.201	0.121	0.240	0.149	0.275	0.334	0.440
	48	0.153	0.262	0.125	0.236	0.202	0.317	0.227	0.348	1.032	0.782
	96	0.235	0.336	0.164	0.275	0.262	0.367	0.348	0.434	1.031	0.796

TABLE IV: Performance comparison of different models on the PEMS datasets in terms of MSE and MAE across various prediction lengths {12, 24, 48, 96}.

Fig. 4: Visualization of input-predict results on the Traffic, ECL and PEMS dataset.

Models	FISformer		FANTF		Promotion (%)
Models	(Ours)		(2025)		Promotion (%)
Metric	MSE	MAE	MSE	MAE	MSE	MAE
ETTh1	0.427	0.427	0.456	0.446	6.8%	4.4%
ETTh2	0.375	0.397	0.387	0.408	3.2%	2.8%
ETTm1	0.370	0.388	0.409	0.409	10.5%	5.4%
ETTm2	0.247	0.307	0.291	0.334	17.8%	8.8%

TABLE V: Quantitative comparison between FANTF [ 18 ] and FISformer on benchmark datasets in terms of MSE and MAE at a prediction length of 192. “Promotion” denotes the relative improvement of FISFormer over FANTF.

Datasets	ECL		ETT (Avg)		Exchange		Traffic		Weather		Solar-Energy		PEMS (Avg)
Metric	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
w/Self-Attention	0.178	0.270	0.383	0.399	0.360	0.403	0.428	0.282	0.258	0.278	0.233	0.262	0.119	0.218
w/FIS Interaction	0.172	0.262	0.375	0.395	0.356	0.402	0.417	0.279	0.256	0.277	0.232	0.261	0.122	0.219
Promotion	3.7%	2.8%	2.2%	1.1%	1.2%	0.2%	2.7%	1.3%	0.9%	0.2%	0.6%	0.3%	-2.3%	-0.7%

TABLE VI: Effect of FIS Interaction compared to standard Self-Attention in terms of MSE and MAE across different datasets.

Models		Informer		Informer		Promotion (%)
		ProbAttention		FIS Interaction
		(2021)		(Ours)
Metric		MSE	MAE	MSE	MAE	MSE	MAE
ETTh1	24	0.577	0.549	0.435	0.472	24.6%	14.1%
	48	0.685	0.625	0.623	0.593	9.1%	5.1%
	168	0.931	0.752	0.817	0.714	12.3%	5.1%

TABLE VII: The effect of integrating FIS Interaction into the Informer model by replacing the original ProbSparse Attention mechanism. Results are reported in terms of MSE and MAE across various datasets.

Fig. 5: Impact of learning rate on MSE scores for the ETT (Avg), ECL, Traffic, and Weather datasets.

MFs Types /	Gaussian		Triangular		Trapezoidal
Metric	MSE	MAE	MSE	MAE	MSE	MAE
ECL	0.172	0.262	0.176	0.267	0.175	0.267
ETT (Avg)	0.375	0.395	0.383	0.400	0.385	0.400
Exchange	0.356	0.402	0.356	0.406	0.362	0.408
Traffic	0.417	0.279	0.422	0.281	0.423	0.282
Weather	0.256	0.277	0.258	0.279	0.259	0.279

TABLE VIII: Analysis of different Membership Functions (MFs)—Gaussian, Triangular, and Trapezoidal—across various datasets.

Component	Self-Attention	FIS Interaction
Q/K/V projection	$\mathcal{O}(T\cdot d^{2})$	$\mathcal{O}(T\cdot d^{2})$
Self-Attention	$\mathcal{O}(T^{2}\cdot d)$	–
Fuzzification	–	$\mathcal{O}(T\cdot d\cdot R)$
Rule generation &
Defuzzification	–	$\mathcal{O}(T\cdot d\cdot R)$
Time Complexity	$\mathcal{O}(T^{2}\cdot d)$	$\mathcal{O}(T\cdot d\cdot R)$

TABLE IX: Comparison of time complexity for Self-Attention and FIS Interaction.

T

is sequence length,

d

is feature dimension, and

R

is the number of membership functions.

Explore similar work

IPatch: A Multi-Resolution Transformer Architecture for Robust Time-Series Forecasting
Mar 25, 2026Aymane Harkati, Moncef Garouani, Olivier Teste +2Time Series Forecasting Multivariate Time Series Forecasting
CCPS Laboratory, ENSAM, Hassan II University, Casablanca, 20670, Morocco · IRIT, UMR 5505 CNRS, Université Toulouse Capitole, Toulouse, 31000, France · IRIT, UMR 5505 CNRS, Université Toulouse, UT2J, Toulouse, 31000, France
Signed Dual Attention: Capturing Signed Dependencies in Time Series Forecasting
Jun 3, 2026Balthazar Courvoisier, Tristan CazenaveTime Series Forecasting Transformer Attention
Queensfield AI Technologies, Paris, France · LAMSADE, Universit´e Paris Dauphine - PSL, Paris, France
Beyond Static Positional Encoding: Bayesian Time-aware Attention for Non-Stationary Long-term Time Series Forecasting
Oct 14, 2026Xinjie Lan, Yunbo Mei, Leheng Zhou +1Long-Term Time Series Forecasting Non-Stationary Time Series Forecasting
Rice University · University of Hong Kong · University of North Carolina at Charlotte

Mar 25, 2026J/K move · Enter open · S save

IPatch: A Multi-Resolution Transformer Architecture for Robust Time-Series Forecasting

Aymane Harkati, Moncef Garouani, Olivier Teste, Julien Aligon+1

CCPS Laboratory, ENSAM, Hassan II University, Casablanca, 20670, Morocco · IRIT, UMR 5505 CNRS, Université Toulouse Capitole, Toulouse, 31000, France · IRIT, UMR 5505 CNRS, Université Toulouse, UT2J, Toulouse, 31000, France

Time Series Forecasting Multivariate Time Series Forecasting Time Series Transformers Multiscale Time-Series Modeling

Open paper AskPDF arXiv

Accurate forecasting of multivariate time series remains challenging due to the need to capture both short-term fluctuations and long-range temporal dependencies. Transformer-based models have emerged as a powerful approach, but their performance depends critically on the representation of temporal data. Traditional point-wise representations preserve individual time-step information, enabling fine-grained modeling, yet they tend to be computationally expensive and less effective at modeling broader contextual dependencies, limiting their scalability to long sequences. Patch-wise representations aggregate consecutive steps into compact tokens to improve efficiency and model local temporal dynamics, but they often discard fine-grained temporal details that are critical for accurate predictions in volatile or complex time series. We propose IPatch, a multi-resolution Transformer architecture that integrates both point-wise and patch-wise tokens, modeling temporal information at multiple resolutions. Experiments on 7 benchmark datasets demonstrate that IPatch consistently improves forecasting accuracy, robustness to noise, and generalization across various prediction horizons compared to single-representation baselines.

Abstract

Explore similar work

IPatch: A Multi-Resolution Transformer Architecture for Robust Time-Series Forecasting

Signed Dual Attention: Capturing Signed Dependencies in Time Series Forecasting

Beyond Static Positional Encoding: Bayesian Time-aware Attention for Non-Stationary Long-term Time Series Forecasting