Seeking SOTA: Time-Series Forecasting Must Adopt Taxonomy-Specific Evaluation to Dispel Illusory Gains

Raeid Saqur; Christoph Bergmeir; Blanka Horvath; Daniel Schmidt; Frank Rudzicz; Terry Lyons

cs.LGMar 16, 2026

Seeking SOTA: Time-Series Forecasting Must Adopt Taxonomy-Specific Evaluation to Dispel Illusory Gains

Authors: Raeid Saqur, Christoph Bergmeir, Blanka Horvath, Daniel Schmidt, Frank Rudzicz, Terry Lyons

Organizations: Dept. of Computer Science, University of Toronto · Dept. of Mathematics, University of Oxford · Vector Institute · Dept. of Data Science & AI, Monash University, Australia ·

Abstract

We argue that the current practice of evaluating AI/ML time-series forecasting models, predominantly on benchmarks characterized by strong, persistent periodicities and seasonalities, obscures real progress by overlooking the performance of efficient classical methods. We demonstrate that these "standard" datasets often exhibit dominant autocorrelation patterns and seasonal cycles that can be effectively captured by simpler linear or statistical models, rendering complex deep learning architectures frequently no more performant than their classical counterparts for these specific data characteristics, and raising questions as to whether any marginal improvements justify the significant increase in computational overhead and model complexity. We call on the community to (I) retire or substantially augment current benchmarks with datasets exhibiting a wider spectrum of non-stationarities, such as structural breaks, time-varying volatility, and concept drift, and less predictable dynamics drawn from diverse real-world domains, and (II) require every deep learning submission to include robust classical and simple baselines, appropriately chosen for the specific characteristics of the downstream tasks' time series. By doing so, we will help ensure that reported gains reflect genuine scientific methodological advances rather than artifacts of benchmark selection favoring models adept at learning repetitive patterns.

Figures & tables

Panel of Figure 1 — Figure 1: Comparative visualization of the target variable (consistently named ‘OT’ in these preprocessed versions) dynamics across five key LTSF benchmark datasets. Top row: Full time series plotted with daily frequency (weekly for ILI ) to highlight long-term trends and seasonalities. The vertical dashed line demarcates the train data (left) and test data (right) . Bottom row: Zoomed-in views ( $\pm$ largest typical forecast horizon: $\pm 30$ days for hourly ETT h2; $\pm 7.5$ days for 15-min ETT m2; $\pm 30$ days for hourly Traffic ; $\pm 30$ days for daily Exchange ; and $\pm 60$ weeks for weekly ILI ) around the train/test split, plotted at their original data frequency. These plots reveal the nature and persistence of inherent periodic patterns across the train / test boundary.

Figure 2: Schematic illustration of canonical non-stationarities commonly observed in time series data.

Model (Year)	ETTh1				ETTh2				ETTm1				ETTm2
Model (Year)	96	192	336	720	96	192	336	720	96	192	336	720	96	192	336	720
Informer ( 2021 )	0.865	1.008	1.107	1.181	3.755	5.602	4.721	3.647	-	-	-	-	-	-	-	-
Autoformer ( 2021 )	0.449	0.500	0.521	0.514	0.346	0.456	0.482	0.515	0.505	0.553	0.621	0.671	0.255	0.281	0.339	0.433
Stationary ( 2022b )	0.513	0.534	0.588	0.643	0.476	0.512	0.552	0.562	0.386	0.459	0.495	0.585	0.192	0.280	0.334	0.417
FEDformer ( 2022 )	0.395	0.469	0.530	0.598	0.358	0.429	0.496	0.463	0.379	0.426	0.445	0.543	0.203	0.269	0.325	0.421
SCINet ( 2022a )	0.654	0.719	0.778	0.836	0.707	0.860	1.000	1.249	0.418	0.439	0.490	0.595	0.286	0.399	0.637	0.960

Table 3: Long-term forecasting MSE (

\downarrow

) results on four ETT datasets for horizons

\{96,192,336,720\}

. Models are ordered by (descending) publication year. The specialist transformer variant model results are reproduced as is from Table 16 of [ 65 ] without reimplementation by the authors. Discrepancies with original publication values* are highlighted . The TSF foundation model (FM) results are obtained from Table 14 of Xu et al. [71] . Best results are in bold , closest to best are underlined for emphasis.

\spadesuit

denote reimplementation by the authors.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

	Running Times			Cricket (T20I)		Soccer (CF)		Chess	Weighted
Athlete	100m (s)	200m (s)	10K (m:s)	Bat. Avg.	Bowl. Avg.	Goals/G	Assists/G	Elo	Avg.
Usain Bolt (JAM)	9.58 (100%)	19.19 (100%)	38:00 (85%)	5 (10%)	150 (5%)	0.10 (50%)	0.05 (40%)	800 (10%)	0.93
Joshua Cheptegei (UGA)	11.50 (80%)	23.00 (75%)	26:24 (100%)	3 (5%)	180 (2%)	0.02 (20%)	0.01 (15%)	700 (5%)	0.92
Virat Kohli (IND)	11.80 (75%)	24.00 (70%)	45:00 (70%)	51.75 (99.9%)	90.00 (20%)	0.05 (40%)	0.02 (30%)	1100 (30%)	0.91
Rashid Khan (AFG)	12.50 (65%)	25.50 (60%)	50:00 (60%)	12.96 (40%)	14.50 (99.9%)	0.01 (15%)	0.01 (10%)	900 (15%)	0.90
Lionel Messi (ARG)	11.20 (85%)	23.50 (70%)	42:00 (78%)	2 (2%)	200 (1%)	0.79 (100%)	0.37 (100%)	1000 (20%)	0.94

Appendix

Table 4: Human evaluation results across diverse downstream tasks. The known best performances for specialists (world records or top ranks) are in green and bold . Hypothetical performances for specialists in non-primary disciplines are obtained by averaging over ChatGPT-4o [ 52 ] and Gemini-2.5 Pro’s [ 3 ] estimate (presented as plain text). The Author’s true performances are in green . The best value in each metric column is in bold . The global ranking percentile (estimated for a relevant population) is shown in parentheses adjacent to performance values.

Metric	Formula
Mean Squared Error (MSE)	$\displaystyle\text{MSE}=\frac{1}{H}\sum_{i=1}^{H}\left(y_{T+i}-\hat{y}_{T+i}\right)^{2}$
Mean Absolute Error (MAE)	$\displaystyle\text{MAE}=\frac{1}{H}\sum_{i=1}^{H}\left\|y_{T+i}-\hat{y}_{T+i}\right\|$
Mean Absolute Percentage Error (MAPE)	$\displaystyle\text{MAPE}=\frac{100}{H}\sum_{i=1}^{H}\frac{\|y_{T+i}-\hat{y}_{T+i}\|}{\|y_{T+i}\|}$
Symmetric Mean Absolute Percentage Error (sMAPE)	$\displaystyle\text{sMAPE}=\frac{200}{H}\sum_{i=1}^{H}\frac{\|y_{T+i}-\hat{y}_{T+i}\|}{\|y_{T+i}\|+\|\hat{y}_{T+i}\|}$
Mean Absolute Scaled Error (MASE)	$\displaystyle\text{MASE}=\frac{1}{H}\sum_{i=1}^{H}\frac{\|y_{T+i}-\hat{y}_{T+i}\|}{\frac{1}{T-H-m}\sum_{j=1}^{T-m}\|y_{j}-y_{j-m}\|}$
Overall Weighted Average (OWA)	$\displaystyle\text{OWA}=\frac{1}{2}\left[\frac{\text{sMAPE}}{\text{sMAPE}_{\text{Na\"{i}ve2}}}+\frac{\text{MASE}}{\text{MASE}_{\text{Na\"{i}ve2}}}\right]$

Appendix

Table 5: Common evaluation metrics used in time-series forecasting.

Metric	Elicited Functional	Scale Invariant	Sensitivity to Extreme Errors	Reported Form / Units	Built-in Baseline Normalization	Lead-time Aware
MSE	cond. mean	✗	Quadratic	squared units	✗	✗
MAE	cond. median	✗	Linear	original units	✗	✗
MAPE	$\|Y\|^{-1}$ -weighted median	✓	Linear; $\|Y\|^{-1}$ -weighted	percentage	✗	✗
sMAPE	nonstandard minimizer	✓	Bounded; saturates at $200\%$	bounded percentage $[0,200\%]$	✗	✗
MASE	cond. median (fixed scale)	✓	Linear (scaled)	ratio to in-sample naïve scale	✓	✗
OWA	nonstandard (composite)	✓	Component-dependent	index (Naïve2 $=1$ )	✓	✗

Appendix

Table 6: Comparison of common time-series forecasting evaluation metrics. The elicited functional maps the predictive distribution to the (possibly non-singleton) set of expected-loss minimizers [ 35 ] ; for strictly positive outcomes the

|Y|^{-1}

-weighted median coincides with the classical

(-1)

-median, and MAPE is undefined when

Y=0

. Scale invariant refers to multiplicative rescaling

y\mapsto cy

c>0

(not to shifts). Sensitivity to extreme errors and reported form describe loss behaviour and output units and do not rank the metrics. Lead-time aware refers to the metric’s construction; any metric can be reported per-horizon. MAPE is undefined when

Y=0

, while sMAPE requires a convention when

Y=\hat{Y}=0

Non- Stationarities	Statistical Variants	Example Problem Domains	Statistical Methodologies (e.g., Naive Baselines)
Trend	Mean	Global temperature rise	Detrending and differencing (ARIMA, ETS with trend)
Seasonal	Mean/Covariance	Quarterly sales, electricity demand	Seasonal adjustment (SARIMA, STL decomposition)
Structural Breaks	Mean/Variance	Economic shocks, policy changes	Regime-switching models (Markov Switching AR, piecewise models)
Volatility	Variance	Financial returns	Time-varying volatility models (GARCH, EGARCH)
Distributional	Distribution shape	Concept evolution in user behavior	Non-parametric/adaptive modeling (kernel density, quantile regression)
Concept Drift	Data relationships	User-item interaction in streaming	Online/continual learning (dynamic Bayesian models, online forests)

Appendix

Table 7: Non-Stationarity based task delineation in time series and corresponding statistical methodologies

Datasets	ETTh1 & ETTh2	ETTm1 &ETTm2	ECL	Weather	Traffic
Channels	7	7	322	21	862
Frequency	hourly	15-min	hourly	hourly	hourly
Timesteps	17,420	69,680	26,304	52,704	17,544
Train/Val/Test Split	12/4/4 months	12/4/4 months	15/3/4 months	28/10/10 months	7:1:

Appendix

Table 8: Summary of Time Series Forecasting Datasets.

Panel of Figure 3 — Figure 3: ETTh1 Dataset: (a) Autocorrelation functions showing persistent daily periodicity (e.g., lag 24) across train/test splits for all features. (b) Full daily average view of the ‘OT’ target. (c) Zoomed hourly view of ‘OT’ around the train/test split, showing continuity of patterns. Target plots share y-max of 60.

Panel of Figure 4 — Figure 4: ETTm1 Dataset: (a) Autocorrelation functions revealing strong daily periodicity (e.g., lag 96 for 15-min data) persisting across train/test splits. (b) Full daily average view of ‘OT’. (c) Zoomed 15-minute view of ‘OT’ ( $\pm 7.5$ days, i.e., $\pm 720$ steps) around the split. Target plots share y-max of 60.

Panel of Figure 5 — Figure 5: ETTh2 Dataset: (a) Persistent daily periodicity evident in ACF plots across train/test splits. (b) Full daily average view of ‘OT’. (c) Zoomed hourly view of ‘OT’ at the train/test boundary. Target plots share y-max of 60.

Panel of Figure 6 — Figure 6: ETTm2 Dataset: (a) Strong daily periodicity (e.g., lag 96) shown in ACF plots, consistent across train/test phases. (b) Full daily average of ‘OT’. (c) Zoomed 15-minute view of ‘OT’ ( $\pm 7.5$ days) around the split. Target plots share y-max of 60.

Panel of Figure 7 — Figure 7: Visualization of the Traffic dataset’s target feature ‘OT’ (an occupancy rate). (a) The full series is plotted as daily averages to reveal long-term trends and seasonalities. (b) A zoomed-in hourly view around the train/test split highlights the persistence of daily and weekly periodic patterns across the split. Both plots share a consistent y-axis maximum (0.15) for comparable scaling.

Panel of Figure 8 — Figure 8: Visualization of the Exchange Rate dataset’s target feature ‘OT’. (a) The full daily series reveals long-term trends and periods of varying volatility. (b) A zoomed-in daily view of $\pm 30$ days around the train/test split illustrates local dynamics. Both plots utilize a consistent (shared) y-axis maximum of 1.0.

Feature	Unit	Periodicity (ACF Peak)
		Train	Test
Temperature ( $T$ )	°C	141	140
Potential Temperature ( $T_{\text{pot}}$ )	K	141	140
Relative Humidity (RH)	%	142	141
Saturation Vapor Pressure (VP ${}_{\text{max}}$ )	mbar	141	140
Vapor Pressure Deficit (VP ${}_{\text{def}}$ )	mbar	141	142

Appendix

Table 9: Peak autocorrelation (ACF) periods for applicable features in the Weather [ 2 ] dataset.

Panel of Figure 9 — Figure 9: Visualization of the ILI dataset’s target feature ‘OT’. (a) The full weekly series clearly shows strong annual seasonality corresponding to flu seasons. (b) A zoomed-in weekly view of $\pm 60$ weeks around the train/test split highlights the consistency of these seasonal peaks and troughs across the split . Both plots are scaled appropriately from 0 to approximately 1.65 million, reflecting the observed range of the ‘OT’ feature (weekly counts) with minor padding.

	Reviewer 1	Reviewer 2	Reviewer 3	Area Chair	Mean (reviewers)
Rating	6 (Weak Accept)	7 (Accept)	4 (Borderline Reject)	4 (Borderline Reject)	5.67
Agreement	4: agree	4: agree	3: neither	3: neither	—
Confidence	4	4	2	4	3.33
Support	4 (excellent)	3 (good)	2 (fair)	—	3.00
Significance	2 (fair)	3 (good)	2 (fair)	—	2.33
Presentation	3 (good)	3 (good)	1 (poor)	—	2.33

Appendix

Table 10: NeurIPS 2025 Position Paper Track reviewer ratings for Submission #102. Ratings are on a 1–10 scale; confidence on a 1–5 scale. Sub-dimension scores (Support, Significance, Presentation, Context) are on a 1–4 scale. The mean column includes only the three reviewers and excludes the Area Chair (AC) meta-review.

Time Series Forecasting

Time series forecasting predicts future values from ordered observations using statistical, neural, and probabilistic models to support decisions under uncertainty.

See all 839 papers

QuitoBench: A High-Quality Open Time Series Forecasting Benchmark
Mar 27, 2026Siqiao Xue, Zhaoyang Zhu, Wei Zhang +7Time Series Forecasting Time Series Modeling
Ant Group
Are We Really Benchmarking Forecasting Models? The Impact of Preprocessing on Time Series Performance
Oct 6, 2026Guilherme Afonso Galindo Padilha, Paulo Salgado Gomes de Mattos Neto, Rafael Menelau Oliveira e CruzTime Series Forecasting Forecasting Benchmarks
LIVIA, Department of Software and IT Engineering École de technologie supérieure (ÉTS) Montreal, QC, Canada · Center for Informatics (CIn) Universidade Federal de Pernambuco (UFPE) Recife, PE, Brazil
TempusBench: An Evaluation Framework for Time-Series Forecasting
Apr 13, 2026Denizalp Goktas, Gerardo Riaño-Briceño, Omkar Tekawade +14Time Series Forecasting Forecasting Benchmarks
Simulacrum New York City, NY, USA

Mar 27, 2026J/K move · Enter open · S save

QuitoBench: A High-Quality Open Time Series Forecasting Benchmark

Siqiao Xue, Zhaoyang Zhu, Wei Zhang, Rongyao Cai+6

Ant Group

Time Series Forecasting Time Series Modeling Forecasting Benchmarks Time-Series Foundation Models

Open paper AskPDF arXiv

Time series forecasting is critical across finance, healthcare, and cloud computing, yet progress is constrained by a fundamental bottleneck: the scarcity of large-scale, high-quality benchmarks. To address this gap, we introduce QuitoBench, a regime-balanced benchmark for time series forecasting with coverage across eight trend

\times

seasonality

\times

forecastability (TSF) regimes, designed to capture forecasting-relevant properties rather than application-defined domain labels. The benchmark is built upon Quito, a billion-scale time series corpus of application traffic from Alipay spanning nine business domains. Benchmarking 10 models from deep learning, foundation models, and statistical baselines across 232,200 evaluation instances, we report four key findings: (i) a context-length crossover where deep learning models lead at short context (

L=96

) but foundation models dominate at long context (

L \ge 576

); (ii) forecastability is the dominant difficulty driver, producing a

3.64 \times

MAE gap across regimes; (iii) deep learning models match or surpass foundation models at

59 \times

fewer parameters; and (iv) scaling the amount of training data provides substantially greater benefit than scaling model size for both model families. These findings are validated by strong cross-benchmark and cross-metric consistency. Our open-source release enables reproducible, regime-aware evaluation for time series forecasting research.

Abstract

Time Series Forecasting

Explore similar work

QuitoBench: A High-Quality Open Time Series Forecasting Benchmark

Are We Really Benchmarking Forecasting Models? The Impact of Preprocessing on Time Series Performance

TempusBench: An Evaluation Framework for Time-Series Forecasting