QuitoBench: A High-Quality Open Time Series Forecasting Benchmark

Siqiao Xue; Zhaoyang Zhu; Wei Zhang; Rongyao Cai; Rui Wang; Yixiang Mu; Fan Zhou; Jianguo Li

cs.LGMar 27, 2026

QuitoBench: A High-Quality Open Time Series Forecasting Benchmark

Authors: Siqiao Xue, Zhaoyang Zhu, Wei Zhang, Rongyao Cai, Rui Wang, Yixiang Mu, Fan Zhou, Jianguo Li, +2 more

Organizations: Ant Group

Abstract

Time series forecasting is critical across finance, healthcare, and cloud computing, yet progress is constrained by a fundamental bottleneck: the scarcity of large-scale, high-quality benchmarks. To address this gap, we introduce QuitoBench, a regime-balanced benchmark for time series forecasting with coverage across eight trend $\times$ seasonality $\times$ forecastability (TSF) regimes, designed to capture forecasting-relevant properties rather than application-defined domain labels. The benchmark is built upon Quito, a billion-scale time series corpus of application traffic from Alipay spanning nine business domains. Benchmarking 10 models from deep learning, foundation models, and statistical baselines across 232,200 evaluation instances, we report four key findings: (i) a context-length crossover where deep learning models lead at short context ( $L=96$ ) but foundation models dominate at long context ( $L \ge 576$ ); (ii) forecastability is the dominant difficulty driver, producing a $3.64 \times$ MAE gap across regimes; (iii) deep learning models match or surpass foundation models at $59 \times$ fewer parameters; and (iv) scaling the amount of training data provides substantially greater benefit than scaling model size for both model families. These findings are validated by strong cross-benchmark and cross-metric consistency. Our open-source release enables reproducible, regime-aware evaluation for time series forecasting research.

Figures & tables

Figure 1: Benchmark contribution rate across domains. Time series has the lowest share (4.2%) vs. NLP (9.9%), speech (7.0%), and vision (6.8%). See Appendix I .

Panel of Figure 2 — Figure 2: 8-grid TSF regime classification across three benchmarks. 2(a) . GIFT-Eval is highly imbalanced, with 50.7% of series in a single low-structure regime. 2(b) . Timer concentrates 65.8% in the high-seasonality, high-forecastability regime. 2(c) . Quito distributes series near-uniformly ( ${\sim}12\%$ per regime).

Figure 3: Overall pipeline of Quito and QuitoBench . It contains five key stages: (1) Raw collection, (2) sanitization and standardization, (3) leakage-free temporal splitting, (4) trend/seasonality/forecastability computation and regime labeling, and (5) balanced QuitoBench construction and evaluation.

		Mean rank			Mean MAE
Model	Category	MV	UV	Overall	MV	UV	Overall
CrossFormer	Deep learning	3.05	2.67	2.86	0.282	0.275	0.279
Chronos-2	Foundation	3.21	3.51	3.36	0.310	0.317	0.314
TimesFM-2.5	Foundation	4.21	4.21	4.21	0.319	0.319	0.319
PatchTST	Deep learning	4.37	4.34	4.35	0.299	0.298	0.299
TiRex	Foundation	4.36	4.36	4.36	0.322	0.322	0.322

Table 3: Overall performance of all ten models sorted by overall mean rank (lower is better). Mean rank and mean MAE are each broken down by MV/UV mode and averaged overall.

Panel of Figure 4 — Figure 4: Scaling behavior on Quito for CrossFormer (deep learning) and TimesFM-2.5 (foundation model). More data yields far larger gains than more parameters for both models.

Table 6

Model	$H$ =48	$H$ =288	$H$ =512	$\Delta$ (48→288)	$\Delta$ (48→512)
CrossFormer	0.237	0.283	0.317	$+$ 19.3%	$+$ 33.9%
PatchTST	0.252	0.300	0.344	$+$ 19.0%	$+$ 36.7%
iTransformer	0.260	0.306	0.335	$+$ 17.6%	$+$ 28.5%
TSMixer	0.273	0.316	0.345	$+$ 15.8%	$+$ 26.3%
DLinear	0.345	0.367	0.396	$+$ 6.5%	$+$ 14.8%
Chronos-2	0.262	0.321	0.358	$+$ 22.8%	$+$ 37.0%

Table 6: Mean MAE at each forecast horizon and percentage degradation relative to

H=48

. Blue : best MAE (CrossFormer). Green : smallest degradation (DLinear,

+

14.8%). Bold = best in column.

Panel of Figure 6 — Figure 6: Ranking robustness. (a) Cross-metric: MAE and MSE produce highly consistent model orderings. (b) Cross-benchmark: model rankings on QuitoBench vs. Timer are strongly correlated. Color indicates ordinal-shift tier. CrossFormer ranks first in all cases.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Distribution of Quito items across nine coarse-grained business verticals. Each item corresponds to the traffic workload of a distinct application service, spanning finance, commerce, advertising, infrastructure, and more, reflecting the diversity of real-world application traffic.

Figure 10: Representative time series from four maximally contrasting tsf regimes (last 500 time steps, min–max normalized). Panel titles use the canonical trend_seasonality_forecastability notation; inset boxes report the exact diagnostic values. (a) high_high_high displays a pronounced downward drift with moderate periodicity, whereas (b) high_low_low is dominated by irregular spikes. (c) low_high_high shows regular bar-like seasonality, and (d) low_low_low exhibits unstructured noise.

Diagnostic	Std dev		Quito/Timer	JSD	KDE overlap
	Quito	Timer	(diversity ratio)	(bits 2 )
$F$	0.205	0.131	1.57 $\times$	0.603	0.40
$S$	0.182	0.134	1.36 $\times$	0.210	0.62
$T$	0.260	0.210	1.24 $\times$	0.437	0.35
Regime entropy (bits; max = 3.00 for 8 cells)
	1.99	1.19	1.67 $\times$	Quito: 66% / Timer: 40% of max

Appendix

Table 9: TSF diversity comparison: Quito (single provenance, 9 verticals, 1,290 items) vs. Timer (11 public datasets, 5 domains, 2,950 series). Quito achieves 1.24–1.57

\times

higher standard deviation on every TSF axis and 1.67

\times

higher regime entropy, demonstrating that single-provenance data can be more diverse in TSF space than multi-provenance benchmarks. JSD: Jensen–Shannon divergence (lower = more similar); KDE overlap: area under the minimum of two kernel density estimates (higher = more similar).

Panel of Figure 11 — Figure 11: Kernel density estimates of each TSF diagnostic for Quito (single provenance, blue) vs. Timer (multi-provenance, red). Dashed lines mark the $\tau{=}0.4$ binarization threshold. Quito’s distributions are broader on all three axes, with particularly greater spread in forecastability and trend, confirming that single-provenance data is not restricted to a narrow TSF sub-region.

Freq	$H=48$	$H=288$	$H=512$	Total (3 $L$ values, 3 $H$ values)
Hour ( $T_{\text{test}}\approx 1{,}536$ )	1,489	1,249	1,025	11,289
10-min ( $T_{\text{test}}\approx 590$ )	543	303	79	2,775

Appendix

Table 10: Number of rolling evaluation windows per series for each forecast horizon

H

and granularity. The rightmost column sums

W(H)

over all three context lengths

L\in\{96,576,1024\}

and all three horizons.

Figure 12: Illustration of the rolling window evaluation strategy. (a) Each series is split 70%/20%/10% into train, validation, and test portions. (b) QuitoBench slides a context+forecast window one step at a time across the test region, generating

W(H)=T_{\text{test}}-H+1

windows per series per

(L,H)

configuration. (c) In contrast, gift-eval -style evaluation uses non-overlapping windows with stride

H

(here shown with 3 windows for illustration), yielding far fewer evaluation samples per series.

Model	Category	Size range	Mean size
Crossformer	Deep learning	0.5M–3M	$\sim$ 1M
DLinear	Deep learning	10K–1M	$\sim$ 300K
iTransformer	Deep learning	0.5M–5M	$\sim$ 2M
PatchTST	Deep learning	1M–15M	$\sim$ 5M
TSMixer	Deep learning	100K–3M	$\sim$ 1M
TiRex	Foundation	30M	30M

Appendix

Table 11: Parameter counts for benchmarked models. Deep learning model sizes vary across task configurations; foundation model sizes are fixed.

Model	Category	MV MAE	UV MAE	Preferred mode
CrossFormer	Deep learning	0.282	0.275	UV ( $+$ 2.5%)
PatchTST	Deep learning	0.299	0.298	UV ( $+$ 0.4%)
TSMixer	Deep learning	0.313	0.309	UV ( $+$ 1.3%)
iTransformer	Deep learning	0.299	0.302	MV ( $+$ 0.7%)
DLinear	Deep learning	0.368	0.371	MV ( $+$ 0.7%)
Chronos-2	Foundation	0.310	0.317	MV ( $+$ 2.3%)

Appendix

Table 12: Mean MAE by MV and UV mode. “Preferred mode” shows the better option and relative improvement; differences below 0.1% are marked Neutral.

Rank	TSF Regime	Trend	Season	Forecast	Mean MAE
1 (Easiest)	high_high_high	HIGH	HIGH	HIGH	0.205
2	low_low_high	LOW	LOW	HIGH	0.220
3	low_high_high	LOW	HIGH	HIGH	0.299
4	low_high_low	LOW	HIGH	LOW	0.359
5	high_low_high	HIGH	LOW	HIGH	0.376
6	low_low_low	LOW	LOW	LOW	0.456

Appendix

Table 13: TSF regime difficulty ranking by mean MAE across all models. high_low_low is 3.64

\times

harder than the easiest regime, demonstrating forecastability’s dominant effect.

Model	HIGH Forecast MAE	LOW Forecast MAE	Ratio
PatchTST	0.185	0.420	2.28 $\times$
iTransformer	0.186	0.422	2.26 $\times$
TSMixer	0.194	0.436	2.25 $\times$
CrossFormer	0.174	0.390	2.24 $\times$
DLinear	0.245	0.501	2.04 $\times$
SNaive	0.517	0.843	1.63 $\times$

Appendix

Table 14: Model sensitivity to forecastability. Deep learning models (top) show higher sensitivity (2.0–2.3

\times

) than foundation models (bottom, 1.7–1.8

\times

), indicating greater robustness of pre-trained representations for unpredictable series.

Rank	Model	MAE	Gap from Best
1	CrossFormer	0.600	—
2	Chronos-2	0.628	+4.6%
3	TimesFM-2.5	0.633	+5.5%
4	TiRex	0.652	+8.6%
5	iTransformer	0.656	+9.3%
6	PatchTST	0.669	+11.5%

Appendix

Table 15: Model performance on the pathological high_low_low regime. Even the best model (CrossFormer) performs 3

\times

worse than on easy series, and statistical baselines fail catastrophically.

$\tau_{\mathrm{TSF}}$	High- $F$ MAE	Low- $F$ MAE	$F$ -gap	FM / DL wins	Kendall $\tau$	Jaccard
0.30	—	—	—	3/4 vs. 1/4	—	0.17
0.35	0.317	0.238	0.75 $\times$	5/6 vs. 1/6	$-$ 0.67	0.29
0.40 †	0.211	0.424	2.01 $\times$	4/6 vs. 2/6	1.00	1.00
0.45	0.209	0.400	1.92 $\times$	4/7 vs. 3/7	0.73	0.60
0.50	0.210	0.391	1.86 $\times$	4/7 vs. 3/7	0.60	0.60

Appendix

Table 16: TSF threshold sensitivity analysis.

F

-gap: ratio of low-

F

to high-

F

mean MAE; FM/DL wins: occupied regimes won by each model family; Kendall

\tau

and Jaccard are computed against the reference threshold

\tau_{\mathrm{TSF}}{=}0.40

(†). At

\tau{=}0.30

nearly all items are classified as high-

F

, so the

F

-gap and ranking metrics are undefined (—).

Panel of Figure 13 — Figure 13: TSF threshold sensitivity. 13(a) The forecastability gap (low- $F$ /high- $F$ MAE) is large and stable for $\tau_{\mathrm{TSF}}\geq 0.40$ . 13(b) Foundation models consistently win more regimes than deep learning across all thresholds. 13(c) Regime difficulty ranking (Kendall $\tau$ ) and FM-winning-regime overlap (Jaccard) remain high for $\tau_{\mathrm{TSF}}\in[0.40,0.50]$ .

Metric	Foundation	Deep learning	ROI
Avg. params (M)	110	1.9	$59\times$ fewer
Best MAE $\downarrow$	0.3138	0.2789	deep learning $-11\%$
Mean MAE $\downarrow$	0.3185	0.3117	deep learning $-2\%$
MAE at $L{=}96$	0.4551	0.3432	deep learning $-25\%$
MAE at $L{\geq}576$	0.2502	0.2960	foundation model $-15\%$
MAE / M params	0.0029	0.1676	$57.9\times$

Appendix

Table 17: Full parameter efficiency comparison: foundation models (avg. 110 M params) vs. deep learning models (avg. 1.9 M params), covering overall performance, context-conditional performance, and per-parameter efficiency.

Figure 14: Efficiency frontier stratified by context length

L

. At

L=96

the deep learning cluster dominates; at

L\geq 576

foundation models (Chronos-2) rise to the top, illustrating the context-conditional nature of the efficiency advantage.

Comparison	$p$ -value	Sig.	Cohen’s $d$
CrossFormer vs. Chronos-2	$2.19\times 10^{-66}$		$-0.067$
CrossFormer vs. TiRex	$3.05\times 10^{-86}$		$-0.083$
CrossFormer vs. TimesFM-2.5	$5.25\times 10^{-90}$		$-0.078$
CrossFormer vs. PatchTST	$7.89\times 10^{-197}$		$-0.028$
CrossFormer vs. iTransformer	$8.85\times 10^{-202}$		$-0.034$
CrossFormer vs. TSMixer	$3.73\times 10^{-208}$		$-0.047$

Appendix

Table 18: Pairwise Wilcoxon signed-rank tests between CrossFormer and each other model. All

p

-values are two-sided; Cohen’s

d

is computed on paired MAE differences. Significance: ***

p<0.001

. With

n{=}232{,}200

instances, all comparisons reach statistical significance; however, effect sizes for the top models (Chronos-2, PatchTST, iTransformer) are negligible (

|d|<0.1

), indicating practically indistinguishable performance. Only baselines (SNaive, ES) show large effects (

|d|>0.5

Model	Category	Timer Rank	Quito Rank	Change	Consistency Tier
TiRex	Foundation	1	5	$-4$	Tier 3 (Inconsistent)
Chronos-2	Foundation	2	2	$0$	Tier 1 (Highly Consistent)
TimesFM-2.5	Foundation	3	3	$0$	Tier 1 (Highly Consistent)
CrossFormer	Deep learning	4	1	$+3$	Tier 3 (Inconsistent)
TSMixer	Deep learning	5	7	$-2$	Tier 2 (Moderately Consistent)
iTransformer	Deep learning	6	6	$0$	Tier 1 (Highly Consistent)

Appendix

Table 19: Cross-benchmark consistency: Timer (zero-shot univariate) vs. QuitoBench (trained univariate). Correlation

=0.865

. Rank changes of

\leq 1

are Tier 1,

2

are Tier 2, and

\geq 3

are Tier 3 (highlighted).

Model	MAE Rank	MSE Rank	MAE Ord.	MSE Ord.	$\|\Delta\|$
CrossFormer	2.859	2.556	1	1	0
Chronos-2	3.360	4.836	2	5	3
TimesFM-2.5	4.211	5.002	3	6	3
PatchTST	4.354	3.328	4	2	2
TiRex	4.356	5.448	5	7	2
iTransformer	4.667	4.065	6	3	3

Appendix

Table 20: Mean MAE rank and MSE rank per model across all evaluation instances. “MAE/MSE Ord.” denotes the ordinal position when models are sorted by mean MAE/MSE rank.

|\Delta|

is the absolute ordinal shift. CrossFormer ranks first under both metrics; the bottom three models are unchanged.

Ctx $L$	Horizon $H$	Mode	$\rho$	$p$ -value
96	48	MV	0.879	$8.1\times 10^{-4}$
96	48	UV	0.939	$5.5\times 10^{-5}$
96	288	MV	0.891	$5.4\times 10^{-4}$
96	288	UV	0.939	$5.5\times 10^{-5}$
96	512	MV	0.927	$1.1\times 10^{-4}$
96	512	UV	0.952	$2.3\times 10^{-5}$

Appendix

Table 21: Spearman

\rho

between MSE-rank and MAE-rank model orderings per evaluation configuration. All correlations are significant (

p{<}0.03

), with a mean

\rho{=}0.847

TSF Regime	$\rho$	$p$ -value
high_low_high	0.976	$<0.001$
low_low_high	0.952	$<0.001$
high_low_low	0.806	0.005
high_high_high	0.794	0.006
low_high_high	0.745	0.013
low_low_low	0.721	0.019

Appendix

Table 22: Spearman

\rho

between MSE-rank and MAE-rank model orderings per TSF regime (mean rank per model within each regime). High-forecastability regimes show near-perfect agreement; low-forecastability regimes show weaker but still positive correlations.

TSF Regime	Count	Percentage	MAE Mean	MAE Std	MAE Min	MAE Max	Rank Mean	Rank Std
high_high_high	29,880	12.87	0.205	0.426	0.005	6.722	5.500	2.872
high_high_low	24,480	10.54	0.478	0.730	0.015	7.058	5.500	2.873
high_low_high	30,600	13.18	0.376	0.360	0.048	4.187	5.500	2.873
high_low_low	28,260	12.17	0.748	1.512	0.023	40.388	5.500	2.874
low_high_high	28,620	12.33	0.299	0.236	0.021	1.566	5.500	2.874
low_high_low	29,880	12.87	0.359	0.275	0.019	3.158	5.500	2.874

Appendix

Table 23: QuitoBench regime statistics. The benchmark enforces uniform coverage (

\sim

12.5% per regime), contrasting sharply with Timer’s 76.2% concentration in high_high_high .

TSF Regime	Chronos-2	CrossFormer	DLinear	ES	iTrans.	PatchTST	SNaive	TimesFM-2.5	TiRex	TSMixer
high_high_high	0.163	0.165	0.189	0.878	0.171	0.173	0.898	0.165	0.165	0.173
high_high_low	0.353	0.356	0.418	0.672	0.369	0.367	0.674	0.357	0.350	0.374
high_low_high	0.349	0.180	0.318	0.424	0.193	0.191	0.432	0.352	0.356	0.204
high_low_low	0.628	0.600	0.805	0.390	0.656	0.669	0.417	0.633	0.652	0.691
low_high_high	0.197	0.199	0.260	1.089	0.219	0.213	1.103	0.208	0.212	0.230
low_high_low	0.235	0.239	0.318	0.882	0.264	0.256	0.890	0.249	0.240	0.275

Appendix

Table 24: QuitoBench mean MAE by TSF regime and model (trained). Best performance in each regime shown in bold . CrossFormer wins 4/8 regimes overall and 8/8 regimes among deep learning models; Chronos-2 wins 3 regimes overall.

TSF Regime	Best Model	MAE	Category
high_high_high	Chronos-2	0.163	Foundation
high_high_low	TiRex	0.350	Foundation
high_low_high	CrossFormer	0.180	Deep learning
high_low_low	CrossFormer	0.600	Deep learning
low_high_high	Chronos-2	0.197	Foundation
low_high_low	Chronos-2	0.235	Foundation

Appendix

Table 25: Best performing model by TSF regime on QuitoBench (trained). CrossFormer wins

4/8

regimes; Chronos-2 wins

3/8

regimes; TiRex wins

1/8

regimes. Deep learning models dominate trend-driven regimes (highlighted).

TSF Regime	QuitoBench (Trained)			Timer Benchmark
TSF Regime	Rank	Model	MAE	Best Model	MAE	Dominance
H_H_H	1	CrossFormer	0.165	CrossFormer	0.252	0.166
	2	iTrans.	0.171	TSMixer	0.255
	3	PatchTST	0.173	iTrans.	0.256
	4	TSMixer	0.173	PatchTST	0.259
	5	DLinear	0.188	DLinear	0.260

Appendix

Table 26: Deep learning model rankings for selected TSF regimes on QuitoBench vs. Timer. CrossFormer is the best deep learning model in all

8/8

QuitoBench regimes and

7/8

Timer regimes.

TSF Regime	Chronos-2	CrossFormer	DLinear	ES	iTrans.	PatchTST	SNaive	TimesFM-2.5	TiRex	TSMixer
high_high_high	3.53	3.29	6.94	9.41	4.47	5.32	8.97	4.16	3.54	5.38
high_high_low	3.30	3.65	7.44	9.23	5.00	4.69	9.19	3.64	3.21	5.65
high_low_high	3.92	2.11	6.92	9.01	3.68	3.35	9.83	5.53	5.77	4.89
high_low_low	3.63	2.64	7.03	8.87	4.55	4.10	9.66	4.23	5.11	5.18
low_high_high	2.59	3.05	7.58	9.85	5.12	4.58	8.39	3.49	4.09	6.24
low_high_low	2.91	3.01	7.72	9.73	5.16	4.38	8.94	3.73	3.41	6.00

Appendix

Table 27: QuitoBench mean MAE Rank by TSF regime and model (lower = better). Bold indicates best mean rank per regime. CrossFormer achieves best rank in 5/8 regimes; Chronos-2 in 2/8; TiRex in 1/8.

TSF Regime	Best Model (by Rank)	Mean MAE Rank	Category
high_high_high	CrossFormer	3.29	Deep learning
high_high_low	TiRex	3.21	Foundation
high_low_high	CrossFormer	2.11	Deep learning
high_low_low	CrossFormer	2.64	Deep learning
low_high_high	Chronos-2	2.59	Foundation
low_high_low	Chronos-2	2.91	Foundation

Appendix

Table 28: Best performing model by TSF regime on QuitoBench using MAE Rank. CrossFormer wins 5/8 regimes; Chronos-2 wins 2/8; TiRex wins 1/8. Deep learning models dominate trend-driven regimes (highlighted).

TSF Regime	QuitoBench (MAE Rank)			Deep Learning Only Ranking
TSF Regime	Rank	Model	Mean MAE Rank	Rank	Model	Mean MAE Rank
H_H_H	1	CrossFormer	3.29	1	CrossFormer	3.29
	2	Chronos-2	3.53	2	iTrans.	4.47
	3	TiRex	3.54	3	PatchTST	5.32
	4	TimesFM-2.5	4.16	4	TSMixer	5.38
	5	iTrans.	4.47	5	DLinear	6.94

Appendix

Table 29: Complete model rankings by MAE Rank for all TSF regimes on QuitoBench . Left: all models; Right: deep learning only comparison. CrossFormer is the best deep learning model in all 8/8 groups.

TSF Regime	Chronos-2	CrossFormer	DLinear	ES	iTrans.	PatchTST	SNaive	TimesFM-2.5	TiRex	TSMixer
high_high_high	1.443	1.477	1.569	1.875	1.504	1.492	1.873	1.500	1.516	1.491
high_high_low	7.624	6.028	6.570	12.633	6.056	6.013	14.371	6.380	6.670	6.090
high_low_high	5.170	4.499	4.806	6.498	4.471	4.450	6.683	5.367	5.512	4.481
high_low_low	596.734	586.511	665.550	862.769	625.424	603.711	1207.710	565.137	569.852	630.411
low_high_high	0.652	0.468	0.561	1.625	0.493	0.482	1.647	0.518	0.526	0.502
low_high_low	1.011	0.973	1.161	3.438	1.079	1.031	4.795	1.085	1.025	1.068

Appendix

Table 30: QuitoBench mean MSE by TSF regime and model (trained). Best performance in each regime shown in bold . PatchTST wins 3/8 regimes; CrossFormer wins 3/8 regimes; Chronos-2 wins 1/8; TimesFM-2.5 wins 1/8.

TSF Regime	Chronos-2	CrossFormer	DLinear	ES	iTrans.	PatchTST	SNaive	TimesFM-2.5	TiRex	TSMixer
high_high_high	4.44	3.34	6.37	9.31	4.14	4.06	9.15	4.56	4.70	4.93
high_high_low	4.89	3.22	6.63	8.83	4.13	3.68	9.62	4.61	4.83	4.57
high_low_high	5.33	1.97	6.61	8.94	3.38	2.92	9.84	5.89	5.76	4.37
high_low_low	5.23	2.29	6.46	8.81	3.86	3.22	9.90	5.06	5.99	4.17
low_high_high	4.56	2.35	6.80	9.42	4.36	3.38	9.49	4.35	5.38	4.89
low_high_low	4.79	2.39	6.83	9.30	4.25	3.19	9.65	4.91	5.19	4.49

Appendix

Table 31: QuitoBench mean MSE Rank by TSF regime and model (lower = better). Bold indicates best mean rank per regime. CrossFormer achieves best rank in all 8/8 regimes.

TSF Regime	QuitoBench (MSE Rank)			Deep Learning Only Ranking
TSF Regime	Rank	Model	Mean MSE Rank	Rank	Model	Mean MSE Rank
H_H_H	1	CrossFormer	3.34	1	CrossFormer	3.34
	2	PatchTST	4.06	2	PatchTST	4.06
	3	iTrans.	4.14	3	iTrans.	4.14
	4	Chronos-2	4.44	4	TSMixer	4.93
	5	TimesFM-2.5	4.56	5	DLinear	6.37

Appendix

Table 32: Complete model rankings by MSE Rank for all TSF regimes on QuitoBench . Left: all models; Right: deep learning only comparison. CrossFormer is the best deep learning model in all 8/8 groups .

Explore similar work

TempusBench: An Evaluation Framework for Time-Series Forecasting
Apr 13, 2026Denizalp Goktas, Gerardo Riaño-Briceño, Omkar Tekawade +14Time Series Forecasting Forecasting Benchmarks
Simulacrum New York City, NY, USA
fev-bench: A Realistic Benchmark for Time Series Forecasting
Sep 30, 2025Oleksandr Shchur, Abdul Fatir Ansari, Caner Turkmen +5Benchmark Design Time Series Forecasting
AWS · Keystone AI · Prior Labs +2
TFRBench: A Reasoning Benchmark for Evaluating Forecasting Systems
Apr 7, 2026Md Atik Ahamed, Mihir Parmar, Palash Goyal +7Forecasting Benchmarks Time Series Reasoning
Google · University of Kentucky

Apr 13, 2026J/K move · Enter open · S save

TempusBench: An Evaluation Framework for Time-Series Forecasting

Denizalp Goktas, Gerardo Riaño-Briceño, Omkar Tekawade, Alif Abdullah+13

Simulacrum New York City, NY, USA

Time Series Forecasting Forecasting Benchmarks Time-Series Foundation Models

Open paper AskPDF arXiv

Foundation models have transformed natural language processing and computer vision, and a rapidly growing literature on time-series foundation models (TSFMs) seeks to replicate this success in forecasting. While recent open-source models demonstrate the promise of TSFMs, the field lacks a comprehensive and community-accepted model evaluation framework. We see at least four major issues impeding progress on the development of such a framework. First, existing evaluation frameworks comprise benchmark forecasting tasks derived from often outdated datasets (e.g., M3), many of which lack clear metadata and overlap with the corpora used to pre-train TSFMs. Second, these frameworks evaluate models along a narrowly defined set of benchmark forecasting tasks, such as forecast horizon length or domain, but overlook core statistical properties such as non-stationarity and seasonality. Third, domain-specific models (e.g., XGBoost) are often compared unfairly, as existing frameworks do not enforce a systematic and consistent hyperparameter tuning convention for all models. Fourth, visualization tools for interpreting comparative performance are lacking. To address these issues, we introduce TempusBench, an open-source evaluation framework for TSFMs. TempusBench consists of 1) new datasets which are not included in existing TSFM pretraining corpora, 2) a set of novel benchmark tasks that go beyond existing ones, 3) a model evaluation pipeline with a standardized hyperparameter tuning protocol, and 4) a tensorboard-based visualization interface. We provide access to our code on GitHub: https://github.com/Smlcrm/TempusBench and maintain a live leaderboard at https://smlcrm.com/tempusbench.

Abstract

Explore similar work

TempusBench: An Evaluation Framework for Time-Series Forecasting

fev-bench: A Realistic Benchmark for Time Series Forecasting

TFRBench: A Reasoning Benchmark for Evaluating Forecasting Systems