Beyond Static Leaderboards: Capability-Focused Evaluation with Structural Diversity for Time Series Foundation Models

Hongyin Zhang; Mingyi Xiang; Rongzhao Chen; Yankai Wang; Jialin Qiao; Xiangdong Huang; Xiaoou Ding; Chen Wang

cs.AIOct 19, 2026

Beyond Static Leaderboards: Capability-Focused Evaluation with Structural Diversity for Time Series Foundation Models

Authors: Hongyin Zhang, Mingyi Xiang, Rongzhao Chen, Yankai Wang, Jialin Qiao, Xiangdong Huang, Xiaoou Ding, Chen Wang,

Organizations: School of Software, BNRist, Tsinghua University · Timecho Ltd, Beijing, China · Harbin Institute of Technology

Abstract

Static public benchmarks for time series foundation models (TSFMs) typically have fixed and narrow test distributions. Their repeated reuse encourages benchmark-specific tuning, making reported performance reflect adaptation to the fixed test set rather than genuine forecasting generalization. In addition, a single aggregate score cannot fully characterize model capabilities across distinct temporal structures and input relationships. We propose CaFE (Capability-Focused Evaluation), a refreshable evaluation extension grounded in original benchmark data that enriches static benchmarks with structural diversity and enriches single-score black-box evaluation with multi-dimensional, capability-oriented assessment. CaFE first profiles the original benchmark and defines eight Structural Features (SFs) that characterize its temporal structure. For each SF, it resamples the original data according to its structural support, removes edge-of-distribution samples, and augments the retained samples by varying the SF's salience, producing samples with different structural profiles. After filtering out out-of-distribution samples while preserving diversity, CaFE yields a refreshed evaluation set. TSFMs are then evaluated on this set using both conventional forecast-error metrics and intervention-response metrics, giving each model a per-feature capability profile alongside endpoint accuracy. Experiments on GIFT-Eval and FEV-Mini20 show that CaFE yields stable evaluation results and reveals per-feature strengths and weaknesses that a single aggregate score hides. Moreover, on genuine data filtered by structural condition, models selected by CaFE outperform those chosen by the aggregate leaderboard. CaFE therefore makes model selection conditional on the structural response of interest.

Figures & tables

Figure 1: A static test set provides one finite snapshot of structural conditions, while a single aggregate score obscures how forecasts respond across those conditions.

Family	Original instances	Explicit structure	Paired change	Refresh
Retrospective suites	$\checkmark$	$\times$	$\times$	$\times$
TSFM suites	$\checkmark$	$\times$	$\times$	$\times$
Covariate-aware suites	$\checkmark$	$\times$	$\times$	$\times$
Observed-structure suites	$\checkmark$	$\checkmark$	$\times$	$\times$
Live evaluation	$\checkmark$	$\times$	$\times$	$\checkmark$
Synthetic mechanisms	$\times$	$\checkmark$	$\times$	$\times$

Table 1: Benchmark families compared on four design properties: anchored to original instances, explicit structure, paired change, and refresh. CaFE is the only family that combines all four.

Figure 2: The CaFE workflow. Profiling maps source forecast windows to unchanged References; Augmentation generates, for each Reference, a family of samples at varying salience; Inspection and Sifting admit only complete families that realize the intended change; Evaluation scores each model on forecast-error and intervention-response metrics.

Figure 3: The Structural Features of CaFE : intrinsic structure for within-series patterns and relational structure for cross-series and covariate dependencies.

Figure 4: Probe MASE rank by suite, model, and Structural Feature for the four suites in one horizontal strip. Within each feature, rank 1 is the most accurate model (lowest Probe MASE), and the underlying values are reported in Appendix C .

Figure 5: Forecast-Change NRMSE rank by suite, model, and Structural Feature for the four suites in one horizontal strip. Within each feature, rank 1 is the most faithful response to the induced change; NRMSE = 1 is the zero-response reference, and the underlying values are reported in Appendix C .

Figure 6: Feature-level Forecast-Change NRMSE values (lower is better) across four suites. Each SF uses an independent range over the six shared models, with larger values toward the center and smaller values on extended outer spokes; endpoint labels show both bounds.

Figure 7: Top-3 coverage of the Reference-MASE-best model on genuine strong-structure tasks from held-out source families (higher is better). SF1–SF5 use 8, 4, 7, 7, and 3 eligible families, respectively.

Figure 8: Model-level response fidelity and input utility for SF6–SF8. Each point is an equal-weight model mean across datasets; lower NRMSE and positive Gap are preferred.

Figure 9: MASE on the References and on the Augmented Reference Families for five small models, ordered by Reference MASE. Points are three-seed means with

\pm 1

sample SD; near-identical pairs within a model indicate that augmentation preserves that model’s accuracy ordering.

Model	Probe MASE	Forecast-Change NRMSE
TimesFM 3.0	1.21 $\pm$ 0.003	0.580 $\pm$ 0.005
TiREx-2	1.20 $\pm$ 0.003	0.637 $\pm$ 0.007
Chronos-2	1.22 $\pm$ 0.004	0.646 $\pm$ 0.007
Toto 2.0	1.24 $\pm$ 0.004	0.671 $\pm$ 0.004
TimesFM 2.5	1.25 $\pm$ 0.004	0.684 $\pm$ 0.008
Timer-3.5	1.28 $\pm$ 0.005	0.788 $\pm$ 0.009

Figure 10: Five-seed GIFT-Eval Short stability on the generated cohorts. Values are mean

\pm

seed SD of Probe MASE and Forecast-Change NRMSE; ordering is unchanged across five seeds.

Figure 11: Chronos-2 fine-tuned on 11,269 CaFE -generated samples under two objectives. The two panels track MASE and Forecast-Change NRMSE over 0–10k steps; official quantile-loss training improves MASE, while the Forecast-Change-NRMSE objective improves response fidelity at the cost of endpoint accuracy.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Structural feature	Controlled change	Control parameter	Applicability
SF1 Trend	Persistent directional continuation	Normalized trend magnitude	Uni./multi.
SF2 Multi-Seasonality	Concurrent stable periods	Number of active periods	Uni./multi.
SF3 Time-Varying Seasonality	Evolving amplitude or phase	Normalized evolution magnitude	Uni./multi.
SF4 Regime Switching	Bounded transition to a persistent state	Transition location	Uni./multi.
SF5 Intermittency	Sparse recurring events	Mean event gap	Uni./multi.
SF6 Common Factor	Shared trajectory across targets	Normalized shared-profile magnitude	Multi., $\geq 3$ targets

Appendix

Table 2: Structural features, controlled changes, control parameters, and input requirements. Full constructions appear in this appendix.

Figure 12: Illustrative interventions for SF1–SF8. Each panel joins the modified history to its analytic continuation over the forecast horizon and preserves the task's target and auxiliary-input roles.

Suite	Horizon	Configurations	Models
GIFT-Eval Short	30–60	55	11
GIFT-Eval Medium	480–600	21	10
GIFT-Eval Long	720–900	21	10
fev-bench Mini-20	5–288	20	11

Appendix

Table 3: Displayed model support of the four latest result packages. Short and FEV use 11 models; Medium and Long use 10 because TiREx-2 is outside the supported horizon.

Suite	Equal-weight diagnostic ranking
GIFT-Eval Short	1. TimesFM 3.0 (0.925); 2. Toto 2.0 (0.625); 3. Chronos-2 (0.600); 4. TimesFM 2.5 (0.550); 5. Timer-3.5 (0.175); 6. Moirai 2 (0.125)
GIFT-Eval Medium	1. TimesFM 3.0 (0.925); 2. Timer-3.5 (0.650); 3. Chronos-2 (0.600); 4. TimesFM 2.5 (0.375); 5. Toto 2.0 (0.300); 6. Moirai 2 (0.150)
GIFT-Eval Long	1. TimesFM 3.0 (0.950); 2. Chronos-2 (0.600); 3. Timer-3.5 (0.600); 4. TimesFM 2.5 (0.400); 5. Toto 2.0 (0.375); 6. Moirai 2 (0.075)
fev-bench Mini-20	1. Chronos-2 (0.750); 2. TimesFM 3.0 (0.700); 3. TimesFM 2.5 (0.500); 4. Toto 2.0 (0.375); 5. Timer-3.5 (0.350); 6. Moirai 2 (0.325)

Appendix

Table 4: Secondary equal-weight structural-response ranking on common model support. Each SF score is converted to a direction-corrected feature-local rank score before averaging equally across SF1–SF8; raw NRMSE values are not pooled.

Suite	Configs	Models shown	Panel predictions
GIFT-Eval Short	55	11	399,740
GIFT-Eval Medium	21	10	127,380
GIFT-Eval Long	21	10	121,810
fev-bench Mini-20	20	11	121,902

Appendix

Table 5: Inventory of the four latest result panels after restricting each view to the comparison roster.

Suite	Lowest Probe MASE	Lowest Forecast-Change NRMSE
GIFT-Eval Short	TimesFM 3.0 (5/8); Toto 2.0 (2/8); Chronos-2 (1/8)	TimesFM 3.0 (4/8); Toto 2.0 (2/8); Linear Trend (1/8); TimesFM 2.5 (1/8)
GIFT-Eval Medium	TimesFM 3.0 (7/8); Chronos-2 (1/8)	TimesFM 3.0 (6/8); Chronos-2 (1/8); Linear Trend (1/8)
GIFT-Eval Long	TimesFM 3.0 (5/8); Timer-3.5 (2/8); Chronos-2 (1/8)	TimesFM 3.0 (6/8); Linear Trend (1/8); Timer-3.5 (1/8)
fev-bench Mini-20	TimesFM 3.0 (7/8); TiREx-2 (1/8)	TimesFM 3.0 (3/8); Chronos-2 (1/8); Linear Trend (1/8); Naive (1/8); TimesFM 2.5 (1/8); Toto 2.0 (1/8)

Appendix

Table 6: Winners for each structural feature in the four latest result panels. Entries give the model and the number of feature tracks won; ties would be reported jointly.

Recommendation	Top-1	Top-3	Top-5
Generic original MASE	1/7	3/7	7/7
Qualified original MASE	1/7	4/7	7/7
Capability-augmented MASE	1/7	6/7	7/7

Appendix

Table 7: Regime-switching candidate coverage in the leave-source-family-out analysis. A hit means that the B-side family-balanced original-MASE champion is present in the frozen Top-

K

list.

Recommendation	Mean gain vs generic	Wins–ties–losses
Qualified original MASE	11.02%	–
Capability-augmented MASE	9.06%	8–1–1

Appendix

Table 8: Bitbrains exploratory case under Q75 family-Borda selection. Values are direction-equal descriptive means over ten disjoint source-group directions; the B-side support has at least 16 A groups and 14 B groups per direction.

Group or structural feature	Spearman $\rho$
Multivariate	1.00
Univariate	0.30
Common factor	1.00
Cross-series dependence	1.00
Intermittency	1.00
Regime switching	1.00

Appendix

Table 9: Small-model B-to-C rank agreement by input group and structural feature. Correlations use five architectures; the multivariate group has only three native items.

Configuration–model	B	C
M4 Hourly–Transformer	1.8348	9.3486
M4 Hourly–DLinear	3.0342	1.8950
M4 Hourly–PatchTST	1.4468	2.1678
SZ Taxi–PatchTST	0.6568	1.4263
SZ Taxi–TSMixer	0.9936	0.6860

Appendix

Table 10: Trend-treatment sensitivity retained in the small-model package. B is the paired original MASE and C is the valid treatment MASE; lower is better.

SF	$N$	Chronos-2	Timer-3.5	TimesFM 2.5	TiREx-2	Moirai 2	Toto 2.0
SF1	18	2.132 / 0.082	1.966 / 0.243	1.898 / 0.216	1.843 / 0.256	2.152 / 0.215	1.876 / 0.243
SF2	20	2.370 / 0.779	2.758 / 0.946	2.499 / 0.802	2.532 / 0.844	4.171 / 1.010	1.934 / 0.637
SF3	19	1.526 / 0.529	1.738 / 0.674	1.462 / 0.679	1.484 / 0.580	1.853 / 0.740	1.572 / 0.571
SF4	20	1.703 / 0.096	1.717 / 0.133	1.627 / 0.091	1.597 / 0.071	1.910 / 0.134	1.574 / 0.144
SF5	20	2.929 / 0.905	2.872 / 0.914	2.833 / 0.837	2.745 / 0.972	4.211 / 0.965	2.699 / 0.955
SF6	4	0.618 / 0.257	0.644 / 0.301	0.648 / 0.333	0.658 / 0.259	0.695 / 0.353	0.625 / 0.291

Appendix

Table 11: Probe MASE / Forecast-Change NRMSE pairs (P/N) for the structural features on GIFT-Eval Short. Lower is better for each component; NRMSE

=1

is the zero-response reference.

Model	Official	SF1	SF2	SF3	SF4	SF5	SF6	SF7	SF8
Chronos-2	1.567	2.208/2.216	0.716/0.730	1.232/1.241	1.575/1.859	0.871/0.896	0.828/1.194	0.773/0.802	0.734/0.765
TimesFM 2.5	1.601	2.140/2.220	0.713/0.722	1.244/1.246	1.911/2.318	0.969/1.052	0.828/1.232	0.782/0.813	0.745/0.782
TimesFM 3.0	1.464	2.003/2.050	0.667/0.686	1.166/1.165	1.628/1.916	0.885/0.923	0.807/1.188	0.748/0.772	0.697/0.732
TiREx-2	1.610	2.239/2.216	0.706/0.732	1.241/1.246	1.722/1.914	0.876/0.921	0.838/1.233	0.788/0.817	0.751/0.801
Moirai 2	1.701	2.258/2.260	0.766/0.797	1.353/1.372	1.688/1.998	0.904/1.030	0.922/1.327	0.824/0.863	0.809/0.877

Appendix

Table 12: Expanded GIFT-Eval Short full55 package: Official MASE and matched original / augmented MASE for the first five model configurations.

Model	Official	SF1	SF2	SF3	SF4	SF5	SF6	SF7	SF8
Timer-3.5	1.757	2.534/2.644	0.711/0.741	1.185/1.182	1.644/1.931	0.865/0.899	0.855/1.260	0.797/0.828	0.723/0.772
Toto 2.0	1.452	1.971/2.065	0.673/0.700	1.186/1.177	1.580/1.848	0.934/0.963	0.837/1.220	0.771/0.793	0.690/0.726
Naive	2.579	3.531/4.370	1.543/1.891	2.041/2.015	2.210/2.595	1.064/1.237	1.215/1.779	1.171/1.239	1.678/2.001
Seasonal Naive	2.193	2.945/3.920	1.276/1.549	1.722/1.728	1.953/2.313	1.045/1.254	1.143/1.659	1.086/1.137	1.250/1.335
Drift	2.565	3.431/3.431	1.542/1.897	2.060/2.036	2.631/3.269	1.100/1.283	1.248/1.821	1.191/1.258	1.686/2.012
Linear Trend	4.806	8.074/8.074	2.211/2.449	2.566/2.585	5.219/7.105	1.599/1.867	2.155/3.016	1.905/2.028	1.841/2.168

Appendix

Table 13: Expanded GIFT-Eval Short full55 panel: Official MASE and matched original / augmented MASE for the remaining six model configurations.

Model	SF1	SF2	SF3	SF4	SF5	SF6	SF7	SF8
Chronos-2	1.371	0.437	0.655	0.401	0.991	0.392	0.427	0.420
TimesFM 2.5	1.506	0.472	0.677	0.396	0.901	0.419	0.537	0.475
TimesFM 3.0	1.179	0.340	0.624	0.387	0.928	0.392	0.403	0.389
TiREx-2	1.119	0.534	0.650	0.403	0.941	0.418	0.497	0.504
Moirai 2	2.109	0.609	0.642	0.419	1.024	0.449	0.609	0.617

Appendix

Table 14: Expanded GIFT-Eval Short full55 package: Forecast-Change NRMSE by structural feature for the first five model configurations.

Model	SF1	SF2	SF3	SF4	SF5	SF6	SF7	SF8
Timer-3.5	2.229	0.498	0.678	0.423	0.965	0.442	0.545	0.454
Toto 2.0	1.611	0.479	0.624	0.397	0.951	0.422	0.433	0.386
Naive	1.000	1.156	0.750	0.560	1.091	0.554	0.650	0.789
Seasonal Naive	2.213	1.020	0.747	0.532	1.068	0.563	0.628	0.760
Drift	$3.42\!\times\!10^{-6}$	1.175	0.753	0.578	1.112	0.560	0.657	0.794
Linear Trend	$3.24\!\times\!10^{-6}$	1.079	1.002	0.711	1.174	0.920	1.354	0.880

Appendix

Table 15: Expanded GIFT-Eval Short full55 panel: Forecast-Change NRMSE by structural feature for the remaining six model configurations.

Model	SF	Tasks	$\Delta\mathrm{MASE}$	95% interval
Chronos-2	SF6	13	-0.020	[-0.113, 0.052]
Chronos-2	SF7	12	0.000	[-0.119, 0.113]
Chronos-2	SF8	8	0.006	[0.000, 0.013]
TimesFM 3.0	SF6	13	0.157	[0.064, 0.259]
TimesFM 3.0	SF7	11	0.051	[-0.020, 0.160]
TimesFM 3.0	SF8	7	0.003	[-0.000, 0.009]

Appendix

Table 16: Expanded GIFT-Eval Short full55 input-removal results. Positive

\Delta\mathrm{MASE}

favors retaining the auxiliary input; intervals are descriptive task-bootstrap 95% intervals.

SF	$N$	Chronos-2	Timer-3.5	TimesFM 2.5	TiREx-2	Moirai 2	Toto 2.0
SF1	9	2.982 / 0.051	2.808 / 0.264	2.880 / 0.259	–	3.469 / 0.293	12.785 / 0.590
SF2	11	5.309 / 1.102	5.533 / 1.206	4.016 / 0.961	–	10.671 / 1.729	10.488 / 1.131
SF3	10	7.051 / 0.787	7.292 / 1.130	3.507 / 0.980	–	10.616 / 1.565	9.955 / 1.296
SF4	11	2.378 / 0.053	2.581 / 0.191	2.453 / 0.131	–	4.026 / 0.276	7.597 / 0.176
SF5	11	2.851 / 1.402	3.039 / 1.738	2.936 / 1.322	–	3.592 / 1.115	6.459 / 1.556
SF6	4	0.919 / 0.257	0.952 / 0.474	0.966 / 0.524	–	1.324 / 0.571	1.163 / 0.377

Appendix

Table 17: Probe MASE / Forecast-Change NRMSE pairs (P/N) for the structural features on GIFT-Eval Medium. Lower is better for each component; NRMSE

=1

is the zero-response reference.

SF	$N$	Chronos-2	Timer-3.5	TimesFM 2.5	TiREx-2	Moirai 2	Toto 2.0
SF1	10	3.597 / 0.062	3.549 / 0.191	3.682 / 0.124	–	4.609 / 0.213	16.115 / 0.257
SF2	11	6.252 / 0.914	6.780 / 1.168	4.890 / 0.966	–	9.542 / 1.638	21.869 / 0.998
SF3	10	4.860 / 0.859	8.021 / 1.223	5.632 / 0.961	–	9.717 / 1.190	23.432 / 1.119
SF4	11	2.934 / 0.064	3.100 / 0.175	3.253 / 0.106	–	4.442 / 0.326	16.994 / 0.265
SF5	11	3.131 / 1.051	3.087 / 1.118	3.185 / 1.069	–	4.636 / 1.683	16.407 / 1.184
SF6	4	1.039 / 0.385	1.144 / 0.739	1.113 / 0.772	–	1.749 / 1.084	1.324 / 0.663

Appendix

Table 18: Probe MASE / Forecast-Change NRMSE pairs (P/N) for the structural features on GIFT-Eval Long. Lower is better for each component; NRMSE

=1

is the zero-response reference.

SF	$N$	Chronos-2	Timer-3.5	TimesFM 2.5	TiREx-2	Moirai 2	Toto 2.0
SF1	18	1.938 / 0.176	1.807 / 0.181	1.469 / 0.111	1.736 / 0.130	1.625 / 0.087	1.591 / 0.124
SF2	20	2.656 / 1.202	3.003 / 1.143	2.582 / 0.954	2.703 / 1.042	3.144 / 1.213	1.839 / 0.809
SF3	20	2.190 / 0.812	2.140 / 0.756	1.744 / 0.734	2.040 / 0.794	2.045 / 0.797	1.659 / 0.663
SF4	20	1.913 / 0.213	1.917 / 0.163	1.562 / 0.120	1.853 / 0.136	1.647 / 0.103	1.603 / 0.110
SF5	18	2.188 / 0.934	1.987 / 0.963	1.712 / 0.864	2.154 / 0.894	2.055 / 1.062	1.731 / 0.852
SF6	9	1.766 / 0.286	2.044 / 0.359	1.630 / 0.301	1.819 / 0.291	1.791 / 0.380	1.663 / 0.295

Appendix

Table 19: Probe MASE / Forecast-Change NRMSE pairs (P/N) for the structural features on fev-bench Mini-20. Lower is better for each component; NRMSE

=1

is the zero-response reference.

Suite	Models	$\rho_{R,P}$	$\rho_{R,N}$	$\rho_{P,N}$
GIFT-Eval Short	6	0.943	-0.086	0.029
GIFT-Eval Medium	5	0.700	0.800	0.800
GIFT-Eval Long	5	0.900	0.800	0.900
fev-bench Mini-20	6	0.657	0.543	0.771

Appendix

Table 20: Spearman rank correlations among Reference MASE (R), Probe MASE (P), and NRMSE (N) for the displayed model sets; the table reports point correlations, with interval estimation treated separately.

SF	Unanimous / 5	Mean modal	Min modal	Setting-modal models
SF1	5	1.00	1.00	Chronos-2 (5)
SF2	0	0.82	0.70	TimesFM 2.5 (1), Toto 2.0 (4)
SF3	5	1.00	1.00	Chronos-2 (5)
SF4	0	0.68	0.50	TiREx-2 (3), TimesFM 2.5 (1), Chronos-2 (1)
SF5	0	0.74	0.50	TimesFM 2.5 (4), Chronos-2 (1)
SF6	0	0.64	0.50	Chronos-2 (5)

Appendix

Table 21: Local-winner stability across ten GIFT-Eval Short seeds for the structural features. Unanimous settings retain one lowest-NRMSE model in all seeds; modal fractions are descriptive.

Suite	SF	Model	Tasks	$\Delta$ MASE [95% interval]	Task median	Positive tasks
GIFT-Eval Short	SF6	Chronos-2	4	0.038 [0.009, 0.068]	0.042	3/4
GIFT-Eval Short	SF6	TiREx-2	4	-0.004 [-0.006, -0.001]	-0.005	1/4
GIFT-Eval Short	SF6	Toto 2.0	4	0.005 [-0.012, 0.030]	-0.004	2/4
GIFT-Eval Short	SF7	Chronos-2	8	0.012 [-0.001, 0.030]	0.002	6/8
GIFT-Eval Short	SF7	TiREx-2	8	0.003 [-0.002, 0.008]	0.001	5/8
GIFT-Eval Short	SF7	Toto 2.0	8	0.031 [-0.002, 0.077]	-0.001	4/8

Appendix

Table 22: Complete Auxiliary-Input Gap results for the evaluated suite–model–feature cells. Each estimate is

\mathrm{MASE}_{\mathrm{ablated}}-\mathrm{MASE}_{\mathrm{full}}

with a descriptive task-bootstrap 95% interval; positive values favor the full auxiliary input.

Step	Official MASE	Official FC-NRMSE	FC-NRMSE-objective MASE	FC-NRMSE-objective
0	0.725	0.641	0.725	0.641
1k	0.681	0.593	0.778	0.533
2k	0.665	0.556	0.804	0.488
3k	0.655	0.582	0.816	0.473
4k	0.645	0.587	0.815	0.483
5k	0.641	0.606	0.814	0.477

Appendix

Table 23: Complete single-realization Chronos-2 trajectories for official quantile-loss and Forecast-Change NRMSE objectives. Values use the panel aggregation on training seed 2026092406 and the same 11,269 samples. This block reports one training seed, so repeated-seed uncertainty is a separate analysis.

Panel	Evaluation setting	Main support	Provenance
GIFT-Eval Short latest	Frozen latest Short panel	55 configurations; 3 unordered variants; 11 models	Analysis record
GIFT-Eval Medium latest	Frozen latest Medium panel	21 configurations; 10 models	Analysis record
GIFT-Eval Long latest	Frozen latest Long panel	21 configurations; 10 models	Analysis record
fev-bench Mini-20 latest	Frozen latest Mini-20 panel	20 configurations; 11 models	Analysis record
Input removal	Matched full-input and target-only views	Same treatment contracts; assessed target history and future fixed	Separate block
Stability (latest)	Five augmentation seeds on Short	55 configurations; 7 models	Analysis record

Appendix

Table 24: Evaluation panels and their declared support. Detailed source provenance and run identities are retained with the analysis record.

Claim family	Evaluation evidence	Result presentation
SF1–SF8 interventions	Structural constructions and their ordered control settings	Feature-level response profiles and heatmaps
Generated-sample accuracy and response	Four evaluation panels with suite-specific support	MASE and Forecast-Change NRMSE tables
Structural-response profiles	Feature-local comparisons on the eligible model support	Main and supplementary heatmaps
Refresh stability	Five-seed and earlier fixed-panel comparisons	Stability summary and figure
Auxiliary-input utility	Matched full-input and target-only comparisons	Auxiliary-Input Gap table and figure
Training-objective trade-off	Two objectives trained on the same sample set	Fine-tuning curves and objective comparison

Appendix

Table 25: Summary of evaluation evidence used for the current results.

Suite	Configs	Candidates/References	Generated samples	Models	Predictions	Horizon
GIFT-Eval Short	55	15,745	17,773	11	399,740	30–60
GIFT-Eval Medium	21	4,259	6,875	10	127,380	480–600
GIFT-Eval Long	21	4,179	6,552	10	121,810	720–900
fev-bench Mini-20	20	2,809	5,609	11	121,902	5–288

Appendix

Table 26: Evaluation inventory across the four main panels.

Level	Declared environment	Purpose
Summary consistency	Linux or macOS; CPU; less than 1 GB memory and disk	Checks row counts, dimensions, and numerical agreement for representative tables and figures
Table and figure reconstruction	Linux or macOS; Python 3.11 or later; 4 CPU cores, 8 GB RAM, and 5 GB free disk recommended	Recomputes the four analysis blocks reported in the paper from compact numerical summaries
End-to-end evaluation	Linux x86-64; Python 3.12; CUDA 12.8-compatible NVIDIA driver; 32 GB VRAM per worker; 16 CPU cores and 64 GB RAM recommended	Defines the environment for reconstructing treatments, evaluating the models, and recomputing the reported analyses

Appendix

Table 27: Reproduction levels and resource requirements.

Forecasting Benchmarks

Forecasting Benchmarks are datasets and evaluation protocols for comparing predictions of future time-series values, events, and outcomes.

See all 224 papers

TempusBench: An Evaluation Framework for Time-Series Forecasting
Apr 13, 2026Denizalp Goktas, Gerardo Riaño-Briceño, Omkar Tekawade +14Time Series Forecasting Forecasting Benchmarks
Simulacrum New York City, NY, USA
Impermanent: A Live Benchmark for Temporal Generalization in Time Series Forecasting
Mar 9, 2026Azul Garza, Renée Rosillo, Rodrigo Mendoza-Smith +5Forecasting Benchmarks Time Series Forecasting
TimeCopilot · Independent Researcher · ELLIS Institute Tübingen +3
LiveHouse-TS: An Open-world Living Benchmark for Time Series Foundation Models
Aug 18, 2026Haomin Wen, Ziyu Zhou, Qingxiang Liu +2Forecasting Benchmarks Time-Series Foundation Models
Shanghai Innovation Institute · The Hong Kong University of Science and Technology (Guangzhou)

Apr 13, 2026J/K move · Enter open · S save

TempusBench: An Evaluation Framework for Time-Series Forecasting

Denizalp Goktas, Gerardo Riaño-Briceño, Omkar Tekawade, Alif Abdullah+13

Simulacrum New York City, NY, USA

Time Series Forecasting Forecasting Benchmarks Time-Series Foundation Models Time Series Modeling

Open paper AskPDF arXiv

Foundation models have transformed natural language processing and computer vision, and a rapidly growing literature on time-series foundation models (TSFMs) seeks to replicate this success in forecasting. While recent open-source models demonstrate the promise of TSFMs, the field lacks a comprehensive and community-accepted model evaluation framework. We see at least four major issues impeding progress on the development of such a framework. First, existing evaluation frameworks comprise benchmark forecasting tasks derived from often outdated datasets (e.g., M3), many of which lack clear metadata and overlap with the corpora used to pre-train TSFMs. Second, these frameworks evaluate models along a narrowly defined set of benchmark forecasting tasks, such as forecast horizon length or domain, but overlook core statistical properties such as non-stationarity and seasonality. Third, domain-specific models (e.g., XGBoost) are often compared unfairly, as existing frameworks do not enforce a systematic and consistent hyperparameter tuning convention for all models. Fourth, visualization tools for interpreting comparative performance are lacking. To address these issues, we introduce TempusBench, an open-source evaluation framework for TSFMs. TempusBench consists of 1) new datasets which are not included in existing TSFM pretraining corpora, 2) a set of novel benchmark tasks that go beyond existing ones, 3) a model evaluation pipeline with a standardized hyperparameter tuning protocol, and 4) a tensorboard-based visualization interface. We provide access to our code on GitHub: https://github.com/Smlcrm/TempusBench and maintain a live leaderboard at https://smlcrm.com/tempusbench.

Abstract

Forecasting Benchmarks

Explore similar work

TempusBench: An Evaluation Framework for Time-Series Forecasting

Impermanent: A Live Benchmark for Temporal Generalization in Time Series Forecasting

LiveHouse-TS: An Open-world Living Benchmark for Time Series Foundation Models