Time series forecasting is critical across finance, healthcare, and cloud computing, yet progress is constrained by a fundamental bottleneck: the scarcity of large-scale, high-quality benchmarks. To address this gap, we introduce QuitoBench, a regime-balanced benchmark for time series forecasting with coverage across eight trend×seasonality×forecastability (TSF) regimes, designed to capture forecasting-relevant properties rather than application-defined domain labels. The benchmark is built upon Quito, a billion-scale time series corpus of application traffic from Alipay spanning nine business domains. Benchmarking 10 models from deep learning, foundation models, and statistical baselines across 232,200 evaluation instances, we report four key findings: (i) a context-length crossover where deep learning models lead at short context (L=96) but foundation models dominate at long context (L≥576); (ii) forecastability is the dominant difficulty driver, producing a 3.64× MAE gap across regimes; (iii) deep learning models match or surpass foundation models at 59× fewer parameters; and (iv) scaling the amount of training data provides substantially greater benefit than scaling model size for both model families. These findings are validated by strong cross-benchmark and cross-metric consistency. Our open-source release enables reproducible, regime-aware evaluation for time series forecasting research.
Figures & tables
Figure 1: Benchmark contribution rate across domains. Time series has the lowest share (4.2%) vs. NLP (9.9%), speech (7.0%), and vision (6.8%). See Appendix I .
Figure 2: 8-grid TSF regime classification across three benchmarks. 2(a) . GIFT-Eval is highly imbalanced, with 50.7% of series in a single low-structure regime. 2(b) . Timer concentrates 65.8% in the high-seasonality, high-forecastability regime. 2(c) . Quito distributes series near-uniformly ( ∼12% per regime).
Figure 3: Overall pipeline of Quito and QuitoBench . It contains five key stages: (1) Raw collection, (2) sanitization and standardization, (3) leakage-free temporal splitting, (4) trend/seasonality/forecastability computation and regime labeling, and (5) balanced QuitoBench construction and evaluation.
Mean rank
Mean MAE
Model
Category
MV
UV
Overall
MV
UV
Overall
CrossFormer
Deep learning
3.05
2.67
2.86
0.282
0.275
0.279
Chronos-2
Foundation
3.21
3.51
3.36
0.310
0.317
0.314
TimesFM-2.5
Foundation
4.21
4.21
4.21
0.319
0.319
0.319
PatchTST
Deep learning
4.37
4.34
4.35
0.299
0.298
0.299
TiRex
Foundation
4.36
4.36
4.36
0.322
0.322
0.322
Table 3: Overall performance of all ten models sorted by overall mean rank (lower is better). Mean rank and mean MAE are each broken down by MV/UV mode and averaged overall.
Figure 4: Scaling behavior on Quito for CrossFormer (deep learning) and TimesFM-2.5 (foundation model). More data yields far larger gains than more parameters for both models.
Table 6
Model
H =48
H =288
H =512
Δ (48→288)
Δ (48→512)
CrossFormer
0.237
0.283
0.317
+ 19.3%
+ 33.9%
PatchTST
0.252
0.300
0.344
+ 19.0%
+ 36.7%
iTransformer
0.260
0.306
0.335
+ 17.6%
+ 28.5%
TSMixer
0.273
0.316
0.345
+ 15.8%
+ 26.3%
DLinear
0.345
0.367
0.396
+ 6.5%
+ 14.8%
Chronos-2
0.262
0.321
0.358
+ 22.8%
+ 37.0%
Table 6: Mean MAE at each forecast horizon and percentage degradation relative to H=48 . Blue : best MAE (CrossFormer). Green : smallest degradation (DLinear, + 14.8%). Bold = best in column.
Figure 8
Figure 6: Ranking robustness. (a) Cross-metric: MAE and MSE produce highly consistent model orderings. (b) Cross-benchmark: model rankings on QuitoBench vs. Timer are strongly correlated. Color indicates ordinal-shift tier. CrossFormer ranks first in all cases.
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Distribution of Quito items across nine coarse-grained business verticals. Each item corresponds to the traffic workload of a distinct application service, spanning finance, commerce, advertising, infrastructure, and more, reflecting the diversity of real-world application traffic.
Figure 10: Representative time series from four maximally contrasting tsf regimes (last 500 time steps, min–max normalized). Panel titles use the canonical trend_seasonality_forecastability notation; inset boxes report the exact diagnostic values. (a) high_high_high displays a pronounced downward drift with moderate periodicity, whereas (b) high_low_low is dominated by irregular spikes. (c) low_high_high shows regular bar-like seasonality, and (d) low_low_low exhibits unstructured noise.
Diagnostic
Std dev
Quito/Timer
JSD
KDE overlap
Quito
Timer
(diversity ratio)
(bits 2 )
F
0.205
0.131
1.57 ×
0.603
0.40
S
0.182
0.134
1.36 ×
0.210
0.62
T
0.260
0.210
1.24 ×
0.437
0.35
Regime entropy (bits; max = 3.00 for 8 cells)
1.99
1.19
1.67 ×
Quito: 66% / Timer: 40% of max
Appendix
Table 9: TSF diversity comparison: Quito (single provenance, 9 verticals, 1,290 items) vs. Timer (11 public datasets, 5 domains, 2,950 series). Quito achieves 1.24–1.57 × higher standard deviation on every TSF axis and 1.67 × higher regime entropy, demonstrating that single-provenance data can be more diverse in TSF space than multi-provenance benchmarks. JSD: Jensen–Shannon divergence (lower = more similar); KDE overlap: area under the minimum of two kernel density estimates (higher = more similar).
Figure 11: Kernel density estimates of each TSF diagnostic for Quito (single provenance, blue) vs. Timer (multi-provenance, red). Dashed lines mark the τ=0.4 binarization threshold. Quito’s distributions are broader on all three axes, with particularly greater spread in forecastability and trend, confirming that single-provenance data is not restricted to a narrow TSF sub-region.
Freq
H=48
H=288
H=512
Total (3 L values, 3 H values)
Hour ( Ttest≈1,536 )
1,489
1,249
1,025
11,289
10-min ( Ttest≈590 )
543
303
79
2,775
Appendix
Table 10: Number of rolling evaluation windows per series for each forecast horizon H and granularity. The rightmost column sums W(H) over all three context lengths L∈{96,576,1024} and all three horizons.
Figure 12: Illustration of the rolling window evaluation strategy. (a) Each series is split 70%/20%/10% into train, validation, and test portions. (b) QuitoBench slides a context+forecast window one step at a time across the test region, generating W(H)=Ttest−H+1 windows per series per (L,H) configuration. (c) In contrast, gift-eval -style evaluation uses non-overlapping windows with stride H (here shown with 3 windows for illustration), yielding far fewer evaluation samples per series.
Model
Category
Size range
Mean size
Crossformer
Deep learning
0.5M–3M
∼ 1M
DLinear
Deep learning
10K–1M
∼ 300K
iTransformer
Deep learning
0.5M–5M
∼ 2M
PatchTST
Deep learning
1M–15M
∼ 5M
TSMixer
Deep learning
100K–3M
∼ 1M
TiRex
Foundation
30M
30M
Appendix
Table 11: Parameter counts for benchmarked models. Deep learning model sizes vary across task configurations; foundation model sizes are fixed.
Model
Category
MV MAE
UV MAE
Preferred mode
CrossFormer
Deep learning
0.282
0.275
UV ( + 2.5%)
PatchTST
Deep learning
0.299
0.298
UV ( + 0.4%)
TSMixer
Deep learning
0.313
0.309
UV ( + 1.3%)
iTransformer
Deep learning
0.299
0.302
MV ( + 0.7%)
DLinear
Deep learning
0.368
0.371
MV ( + 0.7%)
Chronos-2
Foundation
0.310
0.317
MV ( + 2.3%)
Appendix
Table 12: Mean MAE by MV and UV mode. “Preferred mode” shows the better option and relative improvement; differences below 0.1% are marked Neutral.
Rank
TSF Regime
Trend
Season
Forecast
Mean MAE
1 (Easiest)
high_high_high
HIGH
HIGH
HIGH
0.205
2
low_low_high
LOW
LOW
HIGH
0.220
3
low_high_high
LOW
HIGH
HIGH
0.299
4
low_high_low
LOW
HIGH
LOW
0.359
5
high_low_high
HIGH
LOW
HIGH
0.376
6
low_low_low
LOW
LOW
LOW
0.456
Appendix
Table 13: TSF regime difficulty ranking by mean MAE across all models. high_low_low is 3.64 × harder than the easiest regime, demonstrating forecastability’s dominant effect.
Model
HIGH Forecast MAE
LOW Forecast MAE
Ratio
PatchTST
0.185
0.420
2.28 ×
iTransformer
0.186
0.422
2.26 ×
TSMixer
0.194
0.436
2.25 ×
CrossFormer
0.174
0.390
2.24 ×
DLinear
0.245
0.501
2.04 ×
SNaive
0.517
0.843
1.63 ×
Appendix
Table 14: Model sensitivity to forecastability. Deep learning models (top) show higher sensitivity (2.0–2.3 × ) than foundation models (bottom, 1.7–1.8 × ), indicating greater robustness of pre-trained representations for unpredictable series.
Rank
Model
MAE
Gap from Best
1
CrossFormer
0.600
—
2
Chronos-2
0.628
+4.6%
3
TimesFM-2.5
0.633
+5.5%
4
TiRex
0.652
+8.6%
5
iTransformer
0.656
+9.3%
6
PatchTST
0.669
+11.5%
Appendix
Table 15: Model performance on the pathological high_low_low regime. Even the best model (CrossFormer) performs 3 × worse than on easy series, and statistical baselines fail catastrophically.
τTSF
High- F MAE
Low- F MAE
F -gap
FM / DL wins
Kendall τ
Jaccard
0.30
—
—
—
3/4 vs. 1/4
—
0.17
0.35
0.317
0.238
0.75 ×
5/6 vs. 1/6
− 0.67
0.29
0.40 †
0.211
0.424
2.01 ×
4/6 vs. 2/6
1.00
1.00
0.45
0.209
0.400
1.92 ×
4/7 vs. 3/7
0.73
0.60
0.50
0.210
0.391
1.86 ×
4/7 vs. 3/7
0.60
0.60
Appendix
Table 16: TSF threshold sensitivity analysis. F -gap: ratio of low- F to high- F mean MAE; FM/DL wins: occupied regimes won by each model family; Kendall τ and Jaccard are computed against the reference threshold τTSF=0.40 (†). At τ=0.30 nearly all items are classified as high- F , so the F -gap and ranking metrics are undefined (—).
Figure 13: TSF threshold sensitivity. 13(a) The forecastability gap (low- F /high- F MAE) is large and stable for τTSF≥0.40 . 13(b) Foundation models consistently win more regimes than deep learning across all thresholds. 13(c) Regime difficulty ranking (Kendall τ ) and FM-winning-regime overlap (Jaccard) remain high for τTSF∈[0.40,0.50] .
Metric
Foundation
Deep learning
ROI
Avg. params (M)
110
1.9
59× fewer
Best MAE ↓
0.3138
0.2789
deep learning −11%
Mean MAE ↓
0.3185
0.3117
deep learning −2%
MAE at L=96
0.4551
0.3432
deep learning −25%
MAE at L≥576
0.2502
0.2960
foundation model −15%
MAE / M params
0.0029
0.1676
57.9×
Appendix
Table 17: Full parameter efficiency comparison: foundation models (avg. 110 M params) vs. deep learning models (avg. 1.9 M params), covering overall performance, context-conditional performance, and per-parameter efficiency.
Figure 14: Efficiency frontier stratified by context length L . At L=96 the deep learning cluster dominates; at L≥576 foundation models (Chronos-2) rise to the top, illustrating the context-conditional nature of the efficiency advantage.
Comparison
p -value
Sig.
Cohen’s d
CrossFormer vs. Chronos-2
2.19×10−66
−0.067
CrossFormer vs. TiRex
3.05×10−86
−0.083
CrossFormer vs. TimesFM-2.5
5.25×10−90
−0.078
CrossFormer vs. PatchTST
7.89×10−197
−0.028
CrossFormer vs. iTransformer
8.85×10−202
−0.034
CrossFormer vs. TSMixer
3.73×10−208
−0.047
Appendix
Table 18: Pairwise Wilcoxon signed-rank tests between CrossFormer and each other model. All p -values are two-sided; Cohen’s d is computed on paired MAE differences. Significance: *** p<0.001 . With n=232,200 instances, all comparisons reach statistical significance; however, effect sizes for the top models (Chronos-2, PatchTST, iTransformer) are negligible ( ∣d∣<0.1 ), indicating practically indistinguishable performance. Only baselines (SNaive, ES) show large effects ( ∣d∣>0.5 ).
Model
Category
Timer Rank
Quito Rank
Change
Consistency Tier
TiRex
Foundation
1
5
−4
Tier 3 (Inconsistent)
Chronos-2
Foundation
2
2
0
Tier 1 (Highly Consistent)
TimesFM-2.5
Foundation
3
3
0
Tier 1 (Highly Consistent)
CrossFormer
Deep learning
4
1
+3
Tier 3 (Inconsistent)
TSMixer
Deep learning
5
7
−2
Tier 2 (Moderately Consistent)
iTransformer
Deep learning
6
6
0
Tier 1 (Highly Consistent)
Appendix
Table 19: Cross-benchmark consistency: Timer (zero-shot univariate) vs. QuitoBench (trained univariate). Correlation =0.865 . Rank changes of ≤1 are Tier 1, 2 are Tier 2, and ≥3 are Tier 3 (highlighted).
Model
MAE Rank
MSE Rank
MAE Ord.
MSE Ord.
∣Δ∣
CrossFormer
2.859
2.556
1
1
0
Chronos-2
3.360
4.836
2
5
3
TimesFM-2.5
4.211
5.002
3
6
3
PatchTST
4.354
3.328
4
2
2
TiRex
4.356
5.448
5
7
2
iTransformer
4.667
4.065
6
3
3
Appendix
Table 20: Mean MAE rank and MSE rank per model across all evaluation instances. “MAE/MSE Ord.” denotes the ordinal position when models are sorted by mean MAE/MSE rank. ∣Δ∣ is the absolute ordinal shift. CrossFormer ranks first under both metrics; the bottom three models are unchanged.
Ctx L
Horizon H
Mode
ρ
p -value
96
48
MV
0.879
8.1×10−4
96
48
UV
0.939
5.5×10−5
96
288
MV
0.891
5.4×10−4
96
288
UV
0.939
5.5×10−5
96
512
MV
0.927
1.1×10−4
96
512
UV
0.952
2.3×10−5
Appendix
Table 21: Spearman ρ between MSE-rank and MAE-rank model orderings per evaluation configuration. All correlations are significant ( p<0.03 ), with a mean ρ=0.847 .
TSF Regime
ρ
p -value
high_low_high
0.976
<0.001
low_low_high
0.952
<0.001
high_low_low
0.806
0.005
high_high_high
0.794
0.006
low_high_high
0.745
0.013
low_low_low
0.721
0.019
Appendix
Table 22: Spearman ρ between MSE-rank and MAE-rank model orderings per TSF regime (mean rank per model within each regime). High-forecastability regimes show near-perfect agreement; low-forecastability regimes show weaker but still positive correlations.
TSF Regime
Count
Percentage
MAE Mean
MAE Std
MAE Min
MAE Max
Rank Mean
Rank Std
high_high_high
29,880
12.87
0.205
0.426
0.005
6.722
5.500
2.872
high_high_low
24,480
10.54
0.478
0.730
0.015
7.058
5.500
2.873
high_low_high
30,600
13.18
0.376
0.360
0.048
4.187
5.500
2.873
high_low_low
28,260
12.17
0.748
1.512
0.023
40.388
5.500
2.874
low_high_high
28,620
12.33
0.299
0.236
0.021
1.566
5.500
2.874
low_high_low
29,880
12.87
0.359
0.275
0.019
3.158
5.500
2.874
Appendix
Table 23: QuitoBench regime statistics. The benchmark enforces uniform coverage ( ∼ 12.5% per regime), contrasting sharply with Timer’s 76.2% concentration in high_high_high .
TSF Regime
Chronos-2
CrossFormer
DLinear
ES
iTrans.
PatchTST
SNaive
TimesFM-2.5
TiRex
TSMixer
high_high_high
0.163
0.165
0.189
0.878
0.171
0.173
0.898
0.165
0.165
0.173
high_high_low
0.353
0.356
0.418
0.672
0.369
0.367
0.674
0.357
0.350
0.374
high_low_high
0.349
0.180
0.318
0.424
0.193
0.191
0.432
0.352
0.356
0.204
high_low_low
0.628
0.600
0.805
0.390
0.656
0.669
0.417
0.633
0.652
0.691
low_high_high
0.197
0.199
0.260
1.089
0.219
0.213
1.103
0.208
0.212
0.230
low_high_low
0.235
0.239
0.318
0.882
0.264
0.256
0.890
0.249
0.240
0.275
Appendix
Table 24: QuitoBench mean MAE by TSF regime and model (trained). Best performance in each regime shown in bold . CrossFormer wins 4/8 regimes overall and 8/8 regimes among deep learning models; Chronos-2 wins 3 regimes overall.
TSF Regime
Best Model
MAE
Category
high_high_high
Chronos-2
0.163
Foundation
high_high_low
TiRex
0.350
Foundation
high_low_high
CrossFormer
0.180
Deep learning
high_low_low
CrossFormer
0.600
Deep learning
low_high_high
Chronos-2
0.197
Foundation
low_high_low
Chronos-2
0.235
Foundation
Appendix
Table 25: Best performing model by TSF regime on QuitoBench (trained). CrossFormer wins 4/8 regimes; Chronos-2 wins 3/8 regimes; TiRex wins 1/8 regimes. Deep learning models dominate trend-driven regimes (highlighted).
TSF Regime
QuitoBench (Trained)
Timer Benchmark
Rank
Model
MAE
Best Model
MAE
Dominance
H_H_H
1
CrossFormer
0.165
CrossFormer
0.252
0.166
2
iTrans.
0.171
TSMixer
0.255
3
PatchTST
0.173
iTrans.
0.256
4
TSMixer
0.173
PatchTST
0.259
5
DLinear
0.188
DLinear
0.260
Appendix
Table 26: Deep learning model rankings for selected TSF regimes on QuitoBench vs. Timer. CrossFormer is the best deep learning model in all 8/8 QuitoBench regimes and 7/8 Timer regimes.
TSF Regime
Chronos-2
CrossFormer
DLinear
ES
iTrans.
PatchTST
SNaive
TimesFM-2.5
TiRex
TSMixer
high_high_high
3.53
3.29
6.94
9.41
4.47
5.32
8.97
4.16
3.54
5.38
high_high_low
3.30
3.65
7.44
9.23
5.00
4.69
9.19
3.64
3.21
5.65
high_low_high
3.92
2.11
6.92
9.01
3.68
3.35
9.83
5.53
5.77
4.89
high_low_low
3.63
2.64
7.03
8.87
4.55
4.10
9.66
4.23
5.11
5.18
low_high_high
2.59
3.05
7.58
9.85
5.12
4.58
8.39
3.49
4.09
6.24
low_high_low
2.91
3.01
7.72
9.73
5.16
4.38
8.94
3.73
3.41
6.00
Appendix
Table 27: QuitoBench mean MAE Rank by TSF regime and model (lower = better). Bold indicates best mean rank per regime. CrossFormer achieves best rank in 5/8 regimes; Chronos-2 in 2/8; TiRex in 1/8.
TSF Regime
Best Model (by Rank)
Mean MAE Rank
Category
high_high_high
CrossFormer
3.29
Deep learning
high_high_low
TiRex
3.21
Foundation
high_low_high
CrossFormer
2.11
Deep learning
high_low_low
CrossFormer
2.64
Deep learning
low_high_high
Chronos-2
2.59
Foundation
low_high_low
Chronos-2
2.91
Foundation
Appendix
Table 28: Best performing model by TSF regime on QuitoBench using MAE Rank. CrossFormer wins 5/8 regimes; Chronos-2 wins 2/8; TiRex wins 1/8. Deep learning models dominate trend-driven regimes (highlighted).
TSF Regime
QuitoBench (MAE Rank)
Deep Learning Only Ranking
Rank
Model
Mean MAE Rank
Rank
Model
Mean MAE Rank
H_H_H
1
CrossFormer
3.29
1
CrossFormer
3.29
2
Chronos-2
3.53
2
iTrans.
4.47
3
TiRex
3.54
3
PatchTST
5.32
4
TimesFM-2.5
4.16
4
TSMixer
5.38
5
iTrans.
4.47
5
DLinear
6.94
Appendix
Table 29: Complete model rankings by MAE Rank for all TSF regimes on QuitoBench . Left: all models; Right: deep learning only comparison. CrossFormer is the best deep learning model in all 8/8 groups.
TSF Regime
Chronos-2
CrossFormer
DLinear
ES
iTrans.
PatchTST
SNaive
TimesFM-2.5
TiRex
TSMixer
high_high_high
1.443
1.477
1.569
1.875
1.504
1.492
1.873
1.500
1.516
1.491
high_high_low
7.624
6.028
6.570
12.633
6.056
6.013
14.371
6.380
6.670
6.090
high_low_high
5.170
4.499
4.806
6.498
4.471
4.450
6.683
5.367
5.512
4.481
high_low_low
596.734
586.511
665.550
862.769
625.424
603.711
1207.710
565.137
569.852
630.411
low_high_high
0.652
0.468
0.561
1.625
0.493
0.482
1.647
0.518
0.526
0.502
low_high_low
1.011
0.973
1.161
3.438
1.079
1.031
4.795
1.085
1.025
1.068
Appendix
Table 30: QuitoBench mean MSE by TSF regime and model (trained). Best performance in each regime shown in bold . PatchTST wins 3/8 regimes; CrossFormer wins 3/8 regimes; Chronos-2 wins 1/8; TimesFM-2.5 wins 1/8.
TSF Regime
Chronos-2
CrossFormer
DLinear
ES
iTrans.
PatchTST
SNaive
TimesFM-2.5
TiRex
TSMixer
high_high_high
4.44
3.34
6.37
9.31
4.14
4.06
9.15
4.56
4.70
4.93
high_high_low
4.89
3.22
6.63
8.83
4.13
3.68
9.62
4.61
4.83
4.57
high_low_high
5.33
1.97
6.61
8.94
3.38
2.92
9.84
5.89
5.76
4.37
high_low_low
5.23
2.29
6.46
8.81
3.86
3.22
9.90
5.06
5.99
4.17
low_high_high
4.56
2.35
6.80
9.42
4.36
3.38
9.49
4.35
5.38
4.89
low_high_low
4.79
2.39
6.83
9.30
4.25
3.19
9.65
4.91
5.19
4.49
Appendix
Table 31: QuitoBench mean MSE Rank by TSF regime and model (lower = better). Bold indicates best mean rank per regime. CrossFormer achieves best rank in all 8/8 regimes.
TSF Regime
QuitoBench (MSE Rank)
Deep Learning Only Ranking
Rank
Model
Mean MSE Rank
Rank
Model
Mean MSE Rank
H_H_H
1
CrossFormer
3.34
1
CrossFormer
3.34
2
PatchTST
4.06
2
PatchTST
4.06
3
iTrans.
4.14
3
iTrans.
4.14
4
Chronos-2
4.44
4
TSMixer
4.93
5
TimesFM-2.5
4.56
5
DLinear
6.37
Appendix
Table 32: Complete model rankings by MSE Rank for all TSF regimes on QuitoBench . Left: all models; Right: deep learning only comparison. CrossFormer is the best deep learning model in all 8/8 groups .
Foundation models have transformed natural language processing and computer vision, and a rapidly growing literature on time-series foundation models (TSFMs) seeks to replicate this success in forecasting. While recent open-source models demonstrate the promise of TSFMs, the field lacks a comprehensive and community-accepted model evaluation framework. We see at least four major issues impeding progress on the development of such a framework. First, existing evaluation frameworks comprise benchmark forecasting tasks derived from often outdated datasets (e.g., M3), many of which lack clear metadata and overlap with the corpora used to pre-train TSFMs. Second, these frameworks evaluate models along a narrowly defined set of benchmark forecasting tasks, such as forecast horizon length or domain, but overlook core statistical properties such as non-stationarity and seasonality. Third, domain-specific models (e.g., XGBoost) are often compared unfairly, as existing frameworks do not enforce a systematic and consistent hyperparameter tuning convention for all models. Fourth, visualization tools for interpreting comparative performance are lacking. To address these issues, we introduce TempusBench, an open-source evaluation framework for TSFMs. TempusBench consists of 1) new datasets which are not included in existing TSFM pretraining corpora, 2) a set of novel benchmark tasks that go beyond existing ones, 3) a model evaluation pipeline with a standardized hyperparameter tuning protocol, and 4) a tensorboard-based visualization interface. We provide access to our code on GitHub: https://github.com/Smlcrm/TempusBench and maintain a live leaderboard at https://smlcrm.com/tempusbench.
Benchmark quality is critical for meaningful evaluation and sustained progress in time series forecasting, particularly with the rise of pretrained models. Existing benchmarks often have limited domain coverage or overlook real-world settings such as tasks with covariates. Their aggregation procedures frequently lack statistical rigor, making it unclear whether observed performance differences reflect true improvements or random variation. Many benchmarks lack consistent evaluation infrastructure or are too rigid for integration into existing pipelines. To address these gaps, we propose fev-bench, a benchmark of 100 forecasting tasks across seven domains, including 46 with covariates. Supporting the benchmark, we introduce fev, a lightweight Python library for forecasting evaluation emphasizing reproducibility and integration with existing workflows. Using fev, fev-bench employs principled aggregation with bootstrapped confidence intervals to report performance along two dimensions: win rates and skill scores. We report results on fev-bench for pretrained, statistical, and baseline models and identify promising future research directions.
Oleksandr Shchur, Abdul Fatir Ansari, Caner Turkmen +5
We introduce TFRBench, the first benchmark designed to evaluate the reasoning capabilities of forecasting systems. Traditionally, time-series forecasting has been evaluated solely on numerical accuracy, treating foundation models as ``black boxes.'' Unlike existing benchmarks, TFRBench provides a protocol for evaluating the reasoning generated by forecasting systems--specifically their analysis of cross-channel dependencies, trends, and external events. To enable this, we propose a systematic multi-agent framework that utilizes an iterative verification loop to synthesize numerically grounded reasoning traces. Spanning ten datasets across five domains, our evaluation confirms that this reasoning is causally effective; useful for evaluation; and prompting LLMs with our generated traces significantly improves forecasting accuracy compared to direct numerical prediction (e.g., avg. ∼40.2%→56.6%), validating the quality of our reasoning. Conversely, benchmarking experiments reveal that off-the-shelf LLMs consistently struggle with both reasoning (lower LLM-as-a-Judge scores) and numerical forecasting, frequently failing to capture domain-specific dynamics. TFRBench thus establishes a new standard for interpretable, reasoning-based evaluation in time-series forecasting. Our benchmark is available at: https://tfrbench.github.io