Static public benchmarks for time series foundation models (TSFMs) typically have fixed and narrow test distributions. Their repeated reuse encourages benchmark-specific tuning, making reported performance reflect adaptation to the fixed test set rather than genuine forecasting generalization. In addition, a single aggregate score cannot fully characterize model capabilities across distinct temporal structures and input relationships. We propose CaFE (Capability-Focused Evaluation), a refreshable evaluation extension grounded in original benchmark data that enriches static benchmarks with structural diversity and enriches single-score black-box evaluation with multi-dimensional, capability-oriented assessment. CaFE first profiles the original benchmark and defines eight Structural Features (SFs) that characterize its temporal structure. For each SF, it resamples the original data according to its structural support, removes edge-of-distribution samples, and augments the retained samples by varying the SF's salience, producing samples with different structural profiles. After filtering out out-of-distribution samples while preserving diversity, CaFE yields a refreshed evaluation set. TSFMs are then evaluated on this set using both conventional forecast-error metrics and intervention-response metrics, giving each model a per-feature capability profile alongside endpoint accuracy. Experiments on GIFT-Eval and FEV-Mini20 show that CaFE yields stable evaluation results and reveals per-feature strengths and weaknesses that a single aggregate score hides. Moreover, on genuine data filtered by structural condition, models selected by CaFE outperform those chosen by the aggregate leaderboard. CaFE therefore makes model selection conditional on the structural response of interest.
Figures & tables
Figure 1: A static test set provides one finite snapshot of structural conditions, while a single aggregate score obscures how forecasts respond across those conditions.
Family
Original instances
Explicit structure
Paired change
Refresh
Retrospective suites
✓
×
×
×
TSFM suites
✓
×
×
×
Covariate-aware suites
✓
×
×
×
Observed-structure suites
✓
✓
×
×
Live evaluation
✓
×
×
✓
Synthetic mechanisms
×
✓
×
×
Table 1: Benchmark families compared on four design properties: anchored to original instances, explicit structure, paired change, and refresh. CaFE is the only family that combines all four.
Figure 2: The CaFE workflow. Profiling maps source forecast windows to unchanged References; Augmentation generates, for each Reference, a family of samples at varying salience; Inspection and Sifting admit only complete families that realize the intended change; Evaluation scores each model on forecast-error and intervention-response metrics.
Figure 3: The Structural Features of CaFE : intrinsic structure for within-series patterns and relational structure for cross-series and covariate dependencies.
Figure 4: Probe MASE rank by suite, model, and Structural Feature for the four suites in one horizontal strip. Within each feature, rank 1 is the most accurate model (lowest Probe MASE), and the underlying values are reported in Appendix C .
Figure 5: Forecast-Change NRMSE rank by suite, model, and Structural Feature for the four suites in one horizontal strip. Within each feature, rank 1 is the most faithful response to the induced change; NRMSE = 1 is the zero-response reference, and the underlying values are reported in Appendix C .
Figure 6: Feature-level Forecast-Change NRMSE values (lower is better) across four suites. Each SF uses an independent range over the six shared models, with larger values toward the center and smaller values on extended outer spokes; endpoint labels show both bounds.
Figure 7: Top-3 coverage of the Reference-MASE-best model on genuine strong-structure tasks from held-out source families (higher is better). SF1–SF5 use 8, 4, 7, 7, and 3 eligible families, respectively.
Figure 8: Model-level response fidelity and input utility for SF6–SF8. Each point is an equal-weight model mean across datasets; lower NRMSE and positive Gap are preferred.
Figure 9: MASE on the References and on the Augmented Reference Families for five small models, ordered by Reference MASE. Points are three-seed means with ±1 sample SD; near-identical pairs within a model indicate that augmentation preserves that model’s accuracy ordering.
Model
Probe MASE
Forecast-Change NRMSE
TimesFM 3.0
1.21 ± 0.003
0.580 ± 0.005
TiREx-2
1.20 ± 0.003
0.637 ± 0.007
Chronos-2
1.22 ± 0.004
0.646 ± 0.007
Toto 2.0
1.24 ± 0.004
0.671 ± 0.004
TimesFM 2.5
1.25 ± 0.004
0.684 ± 0.008
Timer-3.5
1.28 ± 0.005
0.788 ± 0.009
Figure 10: Five-seed GIFT-Eval Short stability on the generated cohorts. Values are mean ± seed SD of Probe MASE and Forecast-Change NRMSE; ordering is unchanged across five seeds.
Figure 11: Chronos-2 fine-tuned on 11,269 CaFE -generated samples under two objectives. The two panels track MASE and Forecast-Change NRMSE over 0–10k steps; official quantile-loss training improves MASE, while the Forecast-Change-NRMSE objective improves response fidelity at the cost of endpoint accuracy.
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Structural feature
Controlled change
Control parameter
Applicability
SF1 Trend
Persistent directional continuation
Normalized trend magnitude
Uni./multi.
SF2 Multi-Seasonality
Concurrent stable periods
Number of active periods
Uni./multi.
SF3 Time-Varying Seasonality
Evolving amplitude or phase
Normalized evolution magnitude
Uni./multi.
SF4 Regime Switching
Bounded transition to a persistent state
Transition location
Uni./multi.
SF5 Intermittency
Sparse recurring events
Mean event gap
Uni./multi.
SF6 Common Factor
Shared trajectory across targets
Normalized shared-profile magnitude
Multi., ≥3 targets
Appendix
Table 2: Structural features, controlled changes, control parameters, and input requirements. Full constructions appear in this appendix.
Figure 12: Illustrative interventions for SF1–SF8. Each panel joins the modified history to its analytic continuation over the forecast horizon and preserves the task's target and auxiliary-input roles.
Suite
Horizon
Configurations
Models
GIFT-Eval Short
30–60
55
11
GIFT-Eval Medium
480–600
21
10
GIFT-Eval Long
720–900
21
10
fev-bench Mini-20
5–288
20
11
Appendix
Table 3: Displayed model support of the four latest result packages. Short and FEV use 11 models; Medium and Long use 10 because TiREx-2 is outside the supported horizon.
Table 4: Secondary equal-weight structural-response ranking on common model support. Each SF score is converted to a direction-corrected feature-local rank score before averaging equally across SF1–SF8; raw NRMSE values are not pooled.
Suite
Configs
Models shown
Panel predictions
GIFT-Eval Short
55
11
399,740
GIFT-Eval Medium
21
10
127,380
GIFT-Eval Long
21
10
121,810
fev-bench Mini-20
20
11
121,902
Appendix
Table 5: Inventory of the four latest result panels after restricting each view to the comparison roster.
Suite
Lowest Probe MASE
Lowest Forecast-Change NRMSE
GIFT-Eval Short
TimesFM 3.0 (5/8); Toto 2.0 (2/8); Chronos-2 (1/8)
TimesFM 3.0 (4/8); Toto 2.0 (2/8); Linear Trend (1/8); TimesFM 2.5 (1/8)
GIFT-Eval Medium
TimesFM 3.0 (7/8); Chronos-2 (1/8)
TimesFM 3.0 (6/8); Chronos-2 (1/8); Linear Trend (1/8)
TimesFM 3.0 (6/8); Linear Trend (1/8); Timer-3.5 (1/8)
fev-bench Mini-20
TimesFM 3.0 (7/8); TiREx-2 (1/8)
TimesFM 3.0 (3/8); Chronos-2 (1/8); Linear Trend (1/8); Naive (1/8); TimesFM 2.5 (1/8); Toto 2.0 (1/8)
Appendix
Table 6: Winners for each structural feature in the four latest result panels. Entries give the model and the number of feature tracks won; ties would be reported jointly.
Recommendation
Top-1
Top-3
Top-5
Generic original MASE
1/7
3/7
7/7
Qualified original MASE
1/7
4/7
7/7
Capability-augmented MASE
1/7
6/7
7/7
Appendix
Table 7: Regime-switching candidate coverage in the leave-source-family-out analysis. A hit means that the B-side family-balanced original-MASE champion is present in the frozen Top- K list.
Recommendation
Mean gain vs generic
Wins–ties–losses
Qualified original MASE
11.02%
–
Capability-augmented MASE
9.06%
8–1–1
Appendix
Table 8: Bitbrains exploratory case under Q75 family-Borda selection. Values are direction-equal descriptive means over ten disjoint source-group directions; the B-side support has at least 16 A groups and 14 B groups per direction.
Group or structural feature
Spearman ρ
Multivariate
1.00
Univariate
0.30
Common factor
1.00
Cross-series dependence
1.00
Intermittency
1.00
Regime switching
1.00
Appendix
Table 9: Small-model B-to-C rank agreement by input group and structural feature. Correlations use five architectures; the multivariate group has only three native items.
Configuration–model
B
C
M4 Hourly–Transformer
1.8348
9.3486
M4 Hourly–DLinear
3.0342
1.8950
M4 Hourly–PatchTST
1.4468
2.1678
SZ Taxi–PatchTST
0.6568
1.4263
SZ Taxi–TSMixer
0.9936
0.6860
Appendix
Table 10: Trend-treatment sensitivity retained in the small-model package. B is the paired original MASE and C is the valid treatment MASE; lower is better.
SF
N
Chronos-2
Timer-3.5
TimesFM 2.5
TiREx-2
Moirai 2
Toto 2.0
SF1
18
2.132 / 0.082
1.966 / 0.243
1.898 / 0.216
1.843 / 0.256
2.152 / 0.215
1.876 / 0.243
SF2
20
2.370 / 0.779
2.758 / 0.946
2.499 / 0.802
2.532 / 0.844
4.171 / 1.010
1.934 / 0.637
SF3
19
1.526 / 0.529
1.738 / 0.674
1.462 / 0.679
1.484 / 0.580
1.853 / 0.740
1.572 / 0.571
SF4
20
1.703 / 0.096
1.717 / 0.133
1.627 / 0.091
1.597 / 0.071
1.910 / 0.134
1.574 / 0.144
SF5
20
2.929 / 0.905
2.872 / 0.914
2.833 / 0.837
2.745 / 0.972
4.211 / 0.965
2.699 / 0.955
SF6
4
0.618 / 0.257
0.644 / 0.301
0.648 / 0.333
0.658 / 0.259
0.695 / 0.353
0.625 / 0.291
Appendix
Table 11: Probe MASE / Forecast-Change NRMSE pairs (P/N) for the structural features on GIFT-Eval Short. Lower is better for each component; NRMSE =1 is the zero-response reference.
Model
Official
SF1
SF2
SF3
SF4
SF5
SF6
SF7
SF8
Chronos-2
1.567
2.208/2.216
0.716/0.730
1.232/1.241
1.575/1.859
0.871/0.896
0.828/1.194
0.773/0.802
0.734/0.765
TimesFM 2.5
1.601
2.140/2.220
0.713/0.722
1.244/1.246
1.911/2.318
0.969/1.052
0.828/1.232
0.782/0.813
0.745/0.782
TimesFM 3.0
1.464
2.003/2.050
0.667/0.686
1.166/1.165
1.628/1.916
0.885/0.923
0.807/1.188
0.748/0.772
0.697/0.732
TiREx-2
1.610
2.239/2.216
0.706/0.732
1.241/1.246
1.722/1.914
0.876/0.921
0.838/1.233
0.788/0.817
0.751/0.801
Moirai 2
1.701
2.258/2.260
0.766/0.797
1.353/1.372
1.688/1.998
0.904/1.030
0.922/1.327
0.824/0.863
0.809/0.877
Appendix
Table 12: Expanded GIFT-Eval Short full55 package: Official MASE and matched original / augmented MASE for the first five model configurations.
Model
Official
SF1
SF2
SF3
SF4
SF5
SF6
SF7
SF8
Timer-3.5
1.757
2.534/2.644
0.711/0.741
1.185/1.182
1.644/1.931
0.865/0.899
0.855/1.260
0.797/0.828
0.723/0.772
Toto 2.0
1.452
1.971/2.065
0.673/0.700
1.186/1.177
1.580/1.848
0.934/0.963
0.837/1.220
0.771/0.793
0.690/0.726
Naive
2.579
3.531/4.370
1.543/1.891
2.041/2.015
2.210/2.595
1.064/1.237
1.215/1.779
1.171/1.239
1.678/2.001
Seasonal Naive
2.193
2.945/3.920
1.276/1.549
1.722/1.728
1.953/2.313
1.045/1.254
1.143/1.659
1.086/1.137
1.250/1.335
Drift
2.565
3.431/3.431
1.542/1.897
2.060/2.036
2.631/3.269
1.100/1.283
1.248/1.821
1.191/1.258
1.686/2.012
Linear Trend
4.806
8.074/8.074
2.211/2.449
2.566/2.585
5.219/7.105
1.599/1.867
2.155/3.016
1.905/2.028
1.841/2.168
Appendix
Table 13: Expanded GIFT-Eval Short full55 panel: Official MASE and matched original / augmented MASE for the remaining six model configurations.
Model
SF1
SF2
SF3
SF4
SF5
SF6
SF7
SF8
Chronos-2
1.371
0.437
0.655
0.401
0.991
0.392
0.427
0.420
TimesFM 2.5
1.506
0.472
0.677
0.396
0.901
0.419
0.537
0.475
TimesFM 3.0
1.179
0.340
0.624
0.387
0.928
0.392
0.403
0.389
TiREx-2
1.119
0.534
0.650
0.403
0.941
0.418
0.497
0.504
Moirai 2
2.109
0.609
0.642
0.419
1.024
0.449
0.609
0.617
Appendix
Table 14: Expanded GIFT-Eval Short full55 package: Forecast-Change NRMSE by structural feature for the first five model configurations.
Model
SF1
SF2
SF3
SF4
SF5
SF6
SF7
SF8
Timer-3.5
2.229
0.498
0.678
0.423
0.965
0.442
0.545
0.454
Toto 2.0
1.611
0.479
0.624
0.397
0.951
0.422
0.433
0.386
Naive
1.000
1.156
0.750
0.560
1.091
0.554
0.650
0.789
Seasonal Naive
2.213
1.020
0.747
0.532
1.068
0.563
0.628
0.760
Drift
3.42×10−6
1.175
0.753
0.578
1.112
0.560
0.657
0.794
Linear Trend
3.24×10−6
1.079
1.002
0.711
1.174
0.920
1.354
0.880
Appendix
Table 15: Expanded GIFT-Eval Short full55 panel: Forecast-Change NRMSE by structural feature for the remaining six model configurations.
Model
SF
Tasks
ΔMASE
95% interval
Chronos-2
SF6
13
-0.020
[-0.113, 0.052]
Chronos-2
SF7
12
0.000
[-0.119, 0.113]
Chronos-2
SF8
8
0.006
[0.000, 0.013]
TimesFM 3.0
SF6
13
0.157
[0.064, 0.259]
TimesFM 3.0
SF7
11
0.051
[-0.020, 0.160]
TimesFM 3.0
SF8
7
0.003
[-0.000, 0.009]
Appendix
Table 16: Expanded GIFT-Eval Short full55 input-removal results. Positive ΔMASE favors retaining the auxiliary input; intervals are descriptive task-bootstrap 95% intervals.
SF
N
Chronos-2
Timer-3.5
TimesFM 2.5
TiREx-2
Moirai 2
Toto 2.0
SF1
9
2.982 / 0.051
2.808 / 0.264
2.880 / 0.259
–
3.469 / 0.293
12.785 / 0.590
SF2
11
5.309 / 1.102
5.533 / 1.206
4.016 / 0.961
–
10.671 / 1.729
10.488 / 1.131
SF3
10
7.051 / 0.787
7.292 / 1.130
3.507 / 0.980
–
10.616 / 1.565
9.955 / 1.296
SF4
11
2.378 / 0.053
2.581 / 0.191
2.453 / 0.131
–
4.026 / 0.276
7.597 / 0.176
SF5
11
2.851 / 1.402
3.039 / 1.738
2.936 / 1.322
–
3.592 / 1.115
6.459 / 1.556
SF6
4
0.919 / 0.257
0.952 / 0.474
0.966 / 0.524
–
1.324 / 0.571
1.163 / 0.377
Appendix
Table 17: Probe MASE / Forecast-Change NRMSE pairs (P/N) for the structural features on GIFT-Eval Medium. Lower is better for each component; NRMSE =1 is the zero-response reference.
SF
N
Chronos-2
Timer-3.5
TimesFM 2.5
TiREx-2
Moirai 2
Toto 2.0
SF1
10
3.597 / 0.062
3.549 / 0.191
3.682 / 0.124
–
4.609 / 0.213
16.115 / 0.257
SF2
11
6.252 / 0.914
6.780 / 1.168
4.890 / 0.966
–
9.542 / 1.638
21.869 / 0.998
SF3
10
4.860 / 0.859
8.021 / 1.223
5.632 / 0.961
–
9.717 / 1.190
23.432 / 1.119
SF4
11
2.934 / 0.064
3.100 / 0.175
3.253 / 0.106
–
4.442 / 0.326
16.994 / 0.265
SF5
11
3.131 / 1.051
3.087 / 1.118
3.185 / 1.069
–
4.636 / 1.683
16.407 / 1.184
SF6
4
1.039 / 0.385
1.144 / 0.739
1.113 / 0.772
–
1.749 / 1.084
1.324 / 0.663
Appendix
Table 18: Probe MASE / Forecast-Change NRMSE pairs (P/N) for the structural features on GIFT-Eval Long. Lower is better for each component; NRMSE =1 is the zero-response reference.
SF
N
Chronos-2
Timer-3.5
TimesFM 2.5
TiREx-2
Moirai 2
Toto 2.0
SF1
18
1.938 / 0.176
1.807 / 0.181
1.469 / 0.111
1.736 / 0.130
1.625 / 0.087
1.591 / 0.124
SF2
20
2.656 / 1.202
3.003 / 1.143
2.582 / 0.954
2.703 / 1.042
3.144 / 1.213
1.839 / 0.809
SF3
20
2.190 / 0.812
2.140 / 0.756
1.744 / 0.734
2.040 / 0.794
2.045 / 0.797
1.659 / 0.663
SF4
20
1.913 / 0.213
1.917 / 0.163
1.562 / 0.120
1.853 / 0.136
1.647 / 0.103
1.603 / 0.110
SF5
18
2.188 / 0.934
1.987 / 0.963
1.712 / 0.864
2.154 / 0.894
2.055 / 1.062
1.731 / 0.852
SF6
9
1.766 / 0.286
2.044 / 0.359
1.630 / 0.301
1.819 / 0.291
1.791 / 0.380
1.663 / 0.295
Appendix
Table 19: Probe MASE / Forecast-Change NRMSE pairs (P/N) for the structural features on fev-bench Mini-20. Lower is better for each component; NRMSE =1 is the zero-response reference.
Suite
Models
ρR,P
ρR,N
ρP,N
GIFT-Eval Short
6
0.943
-0.086
0.029
GIFT-Eval Medium
5
0.700
0.800
0.800
GIFT-Eval Long
5
0.900
0.800
0.900
fev-bench Mini-20
6
0.657
0.543
0.771
Appendix
Table 20: Spearman rank correlations among Reference MASE (R), Probe MASE (P), and NRMSE (N) for the displayed model sets; the table reports point correlations, with interval estimation treated separately.
SF
Unanimous / 5
Mean modal
Min modal
Setting-modal models
SF1
5
1.00
1.00
Chronos-2 (5)
SF2
0
0.82
0.70
TimesFM 2.5 (1), Toto 2.0 (4)
SF3
5
1.00
1.00
Chronos-2 (5)
SF4
0
0.68
0.50
TiREx-2 (3), TimesFM 2.5 (1), Chronos-2 (1)
SF5
0
0.74
0.50
TimesFM 2.5 (4), Chronos-2 (1)
SF6
0
0.64
0.50
Chronos-2 (5)
Appendix
Table 21: Local-winner stability across ten GIFT-Eval Short seeds for the structural features. Unanimous settings retain one lowest-NRMSE model in all seeds; modal fractions are descriptive.
Suite
SF
Model
Tasks
Δ MASE [95% interval]
Task median
Positive tasks
GIFT-Eval Short
SF6
Chronos-2
4
0.038 [0.009, 0.068]
0.042
3/4
GIFT-Eval Short
SF6
TiREx-2
4
-0.004 [-0.006, -0.001]
-0.005
1/4
GIFT-Eval Short
SF6
Toto 2.0
4
0.005 [-0.012, 0.030]
-0.004
2/4
GIFT-Eval Short
SF7
Chronos-2
8
0.012 [-0.001, 0.030]
0.002
6/8
GIFT-Eval Short
SF7
TiREx-2
8
0.003 [-0.002, 0.008]
0.001
5/8
GIFT-Eval Short
SF7
Toto 2.0
8
0.031 [-0.002, 0.077]
-0.001
4/8
Appendix
Table 22: Complete Auxiliary-Input Gap results for the evaluated suite–model–feature cells. Each estimate is MASEablated−MASEfull with a descriptive task-bootstrap 95% interval; positive values favor the full auxiliary input.
Step
Official MASE
Official FC-NRMSE
FC-NRMSE-objective MASE
FC-NRMSE-objective
0
0.725
0.641
0.725
0.641
1k
0.681
0.593
0.778
0.533
2k
0.665
0.556
0.804
0.488
3k
0.655
0.582
0.816
0.473
4k
0.645
0.587
0.815
0.483
5k
0.641
0.606
0.814
0.477
Appendix
Table 23: Complete single-realization Chronos-2 trajectories for official quantile-loss and Forecast-Change NRMSE objectives. Values use the panel aggregation on training seed 2026092406 and the same 11,269 samples. This block reports one training seed, so repeated-seed uncertainty is a separate analysis.
Foundation models have transformed natural language processing and computer vision, and a rapidly growing literature on time-series foundation models (TSFMs) seeks to replicate this success in forecasting. While recent open-source models demonstrate the promise of TSFMs, the field lacks a comprehensive and community-accepted model evaluation framework. We see at least four major issues impeding progress on the development of such a framework. First, existing evaluation frameworks comprise benchmark forecasting tasks derived from often outdated datasets (e.g., M3), many of which lack clear metadata and overlap with the corpora used to pre-train TSFMs. Second, these frameworks evaluate models along a narrowly defined set of benchmark forecasting tasks, such as forecast horizon length or domain, but overlook core statistical properties such as non-stationarity and seasonality. Third, domain-specific models (e.g., XGBoost) are often compared unfairly, as existing frameworks do not enforce a systematic and consistent hyperparameter tuning convention for all models. Fourth, visualization tools for interpreting comparative performance are lacking. To address these issues, we introduce TempusBench, an open-source evaluation framework for TSFMs. TempusBench consists of 1) new datasets which are not included in existing TSFM pretraining corpora, 2) a set of novel benchmark tasks that go beyond existing ones, 3) a model evaluation pipeline with a standardized hyperparameter tuning protocol, and 4) a tensorboard-based visualization interface. We provide access to our code on GitHub: https://github.com/Smlcrm/TempusBench and maintain a live leaderboard at https://smlcrm.com/tempusbench.
Recent advances in time-series forecasting increasingly rely on pre-trained foundation-style models. While these models often claim broad generalization, existing evaluation protocols provide limited evidence. Indeed, most current benchmarks use static train-test splits that can easily lead to contamination as foundation models can inadvertently train on test data or perform model selection using test scores, which can inflate performance. We introduce Impermanent, a live benchmark that evaluates forecasting models under open-world temporal change by scoring forecasts sequentially over time on continuously updated data streams, enabling the study of temporal robustness, distributional shift, and performance stability rather than one-off accuracy on a frozen test set. Impermanent is instantiated on GitHub open-source activity, providing a naturally live and highly non-stationary dataset shaped by releases, shifting contributor behavior, platform/tooling changes, and external events. We focus on the top 400 repositories by star count and construct time series from issues opened, pull requests opened, push events, and new stargazers, evaluated over a rolling window with daily updates, alongside standardized protocols and leaderboards for reproducible, ongoing comparison. By shifting evaluation from static accuracy to sustained performance, Impermanent takes a concrete step toward assessing when and whether foundation-level generalization in time-series forecasting can be meaningfully claimed. Code and a live dashboard are available at https://github.com/TimeCopilot/impermanent and https://impermanent.timecopilot.dev.
Time Series Foundation Models (TSFMs) have recently emerged as a highly promising paradigm for cross-domain zero-shot forecasting. However, existing evaluation protocols predominantly rely on static benchmarks with fixed historical test windows. While these benchmarks provide a valuable baseline snapshot, they evaluate an average performance on a fixed history, failing to capture how models behave in continuously evolving real-world environments characterized by seasonal variations, distribution shifts, and unexpected events. To bridge this gap, we introduce LiveHouse-TS, the first open-world living benchmark infrastructure for TSFMs. By evaluating models prequentially on real future data in open-world environments, LiveHouse-TS shifts time series benchmarking from snapshot accuracy to continuous temporal validity. Rather than acting as a one-off leaderboard, our infrastructure serves as a continuous time series infrastructure designed to explore vital, long-term scientific questions: Can model rankings be maintained over the long term? Which models remain genuinely robust under distribution shifts? Extensive streaming evaluations across 11 domains with 17 datasets demonstrate that static rankings undergo a dramatic reshuffling under a live protocol.
Haomin Wen, Ziyu Zhou, Qingxiang Liu +2
Shanghai Innovation Institute · The Hong Kong University of Science and Technology (Guangzhou)