A forecasting benchmark reports which method won. We show that the answer is set by the evaluator's choices before any model is fitted. We benchmark 24 forecasting methods and one textbook reference, including six 2025-era time series foundation models, on a production marketplace panel of 1,887 business customers over 67 months. We hold the data, the horizon and the period fixed, and vary only the evaluation design. Three choices each reverse or dissolve a headline conclusion. Changing the unit of analysis from the market total to the individual customer moves our production baseline from second of nineteen, beaten by nothing, to twenty-third of twenty-five. Nineteen of its twenty-four challengers beat it there. Changing how much error is pooled decides whether a Diebold-Mariano test finds anything at all. Scoring prediction intervals rather than point forecasts reorders the field almost completely, with a rank correlation of 0.02 on intermittent demand. We then measure what the deployed system gets from this. Its selection rule captures 55% of the distance between doing nothing and choosing with hindsight. The reversal is not a quirk of our data. We ran the released protocol, unchanged, on the public M5 retail panel. The same baseline shape places first at the market total and last per series, beaten by everything, and a replayed selection rule closes 64.7% of the same floor-to-ceiling distance there. Adding five zero-shot foundation models to that roster changes who wins at the total, not the shape. The bands' blind spot travels too: conformal bands under-cover most on the spikiest items. Splitting our own panel into ever smaller groups turns the contrast into a curve: the baseline's rank worsens at every level of disaggregation. We release the evaluation protocol and report an error of our own that inverted a result before we caught it.
Figures & tables
Aggregate
Per buyer
Series scored
1
1,887
Rolling origins
22
8
Forecasts compared
19 methods
25 methods
Blocks
22
11,516
Ranked on
mean WAPE
mean RMSSE
Rank of the production baseline
2 of 19
23 of 25
Table 1: The same panel, the same window, scored two ways. Changing the unit of analysis moves the production baseline from second place to near-last and turns ”no method beats it” into ”most methods beat it”; the flip survives on the shared roster and origins (Section 4 ). Significance is Diebold–Mariano against the production baseline, Harvey–Leybourne–Newbold corrected; direction is the sign of the DM statistic on the pooled per-step loss.
Method
RMSSE
MASE
Mean
Beats
p
mean
median
mean
rank
(BH)
1
DeepAR (global) †
0.806
0.523
0.932
10.44
7,753
< 0.0001
2
N-HiTS (global) †
0.809
0.536
0.929
10.64
7,662
< 0.0001
3
TimesFM 2.5
0.831
0.519
0.926
10.43
8,693
< 0.0001
4
Moirai 2.0
0.833
0.539
0.929
10.65
8,592
< 0.0001
5
IMAPA
0.838
0.557
0.975
10.33
8,666
< 0.0001
Table 2: Per-buyer accuracy over 8 rolling origins and 11,516 buyer–origin blocks. Ranked on mean RMSSE, the metric the deployed system selects on. ”Beats” counts blocks in which the method’s RMSSE is below the production baseline’s. p is Diebold–Mariano against that baseline, two-sided, Benjamini–Hochberg adjusted within this family of 24 tests. For LightGBM (global) and Seasonal trend the significant difference favors the baseline: a worse pooled per-step loss despite the lower mean RMSSE. Nemenyi critical difference 0.37 on 10,341 complete blocks. † scored on 10,341 blocks rather than 11,516: these methods require more history than some buyer–origin pairs carry.
Groups
Buyers
Blocks
Baseline rank of 19, by convention
Smooth
k
per group
median WAPE
mean RMSSE
Friedman
share
1
5,779
23
2
5
5
100%
5
1,156
103
14
14
12
57%
25
231
479
14
14
14
51%
125
46
2,336
16
17
16
50%
625
9
9,862
16
17
17
27%
Table 3: From a contrast to a curve. Every buyer in the panel is assigned to one of k size-sorted groups, using only data from before the first origin, and the shared roster of nineteen methods is scored on each group’s aggregate over the same origins. The baseline’s rank worsens monotonically as the aggregate is split: Spearman against logk is 1.00 on Friedman mean rank and 0.95 on the other two conventions. The last column shows why: splitting the total replaces smooth aggregates with spiky ones, so the aggregation level and the demand mix are one dial seen from two sides.
Figure 1: The same fall on two panels. The production baseline’s place in its roster slides from near the top toward the bottom as the aggregate is split, whether the groups are constructed by size on our panel or are M5’s own levels, from the market total down to the single item–store series. Rank is shown as position within each roster, first to last, because the rosters differ in size. Production points are Friedman mean-rank placements at k = 1 to 625 groups (Exhibit 3 ); M5 points are mean-RMSSE placements at its six native levels.
Demand class
Blocks
Best
Top three
Baseline
Baseline
RMSSE
rank
RMSSE
Smooth
4,074
0.803
N-HiTS, DeepAR, Moirai
22 of 25
1.054
Erratic
4,295
0.700
DeepAR, N-HiTS, TimesFM
23 of 25
1.029
Intermittent
579
1.199
IMAPA, AutoTheta, TSB
24 of 25
1.447
Lumpy
2,568
0.870
DeepAR, TimesFM, IMAPA
24 of 25
1.168
Table 4: Which method wins depends on the buyer’s demand pattern. Buyers are classified at every origin from that origin’s training window alone, on average demand interval and squared coefficient of variation. Three different methods lead the four classes; the pooled leaderboard’s single winner leads only two of them.
80% band
95% band
Point vs interval
Records
cover
error
cover
error
ρ
p
move
Overall
193,067
0.782
-1.8
0.864
-8.6
0.39
0.0509
16
Smooth
71,337
0.815
1.5
0.925
-2.6
0.61
0.0011
11
Erratic
73,358
0.796
-0.4
0.884
-6.6
0.55
0.0047
12
Intermittent
7,848
0.714
-8.6
0.741
-20.9
0.02
0.9099
16
Lumpy
40,524
0.710
-9.0
0.742
-20.8
0.31
0.1285
16
Table 5: Prediction-interval quality over 193,067 scored records, one record per method–buyer–origin triple. The coverage columns score the deployed system’s own conformal bands, calibrated at each origin on that buyer’s earlier origins only; the last three columns span all twenty-five methods. ρ is the Spearman correlation between ranking methods on point accuracy and ranking them on interval quality (mean scaled interval score); ”move” is the largest number of places any method shifts between the two rankings. The bands are calibrated where demand is regular and under-cover badly where it is not.
Blocks
Baseline
Served
Oracle
Gain
Wins
Gap
RMSSE
RMSSE
RMSSE
closed
Overall
5,832
0.946
0.724
0.540
23.5%
4,489
54.6%
Smooth
2,313
0.995
0.783
0.572
21.3%
1,738
50.1%
Erratic
2,207
0.883
0.667
0.489
24.5%
1,702
54.8%
Intermittent
229
1.245
0.985
0.829
20.9%
198
62.6%
Lumpy
1,083
0.907
0.659
0.512
27.4%
851
62.8%
Table 6: What the deployed selection rule is worth. At every origin the rule is replayed using only that buyer’s earlier origins, then scored against two references: doing nothing (always the production baseline) and choosing the best method with hindsight. The last column is the share of the distance between those two that the rule actually closes. Scored on RMSSE, the metric the rule ranks on; MASE gives 22.1% and 48.9%.
Market total
Store totals
Per item–store
Series scored
1
10
30,449
Rolling origins
20
20
8
Blocks
20
200
232,651
Rank of the baseline
1 of 7
1 of 7
7 of 7
Methods significantly better
0
0
6
Per-step losses pooled by DM
120
1,200
1,395,906
Table 7: The reversal is not ours alone: the released protocol, run end-to-end on the public M5 (Walmart) panel aggregated to monthly, with the protocol’s 7 reference methods and the seasonal-naive-with-momentum baseline. The verdict is computed by the released code, not asserted. Significance is Diebold–Mariano on the protocol’s pooled squared-error differentials, Benjamini–Hochberg adjusted. Under the non-overlapping-fold variant (origin spacing equal to the horizon) and re-ranked on MASE instead of RMSSE the placements are unchanged (166,809 blocks in that variant).
Market total
Store totals
Per item–store
Blocks
20
200
232,651
Rank of the baseline
2 of 12
4 of 12
12 of 12
Methods significantly better
1
3
11
Winner
FlowState
FlowState
Damped smoothing
Table 8: The M5 roster extended with the five zero-shot foundation models (Chronos T5-Large excluded on compute grounds), scored on the same grid as Exhibit 7 with the same tests. The winner at the total changes — FlowState, the only method that beats the baseline by a distinguishable margin there — and the shape does not: the baseline is second at the total and last per item–store, the production panel’s exact pattern.
Forecasting has become a natural benchmark for reasoning under uncertainty. Yet existing evaluations of large language models remain limited to judgmental tasks in simple formats, such as binary or multiple-choice questions. In practice, however, forecasting spans a far broader scope. Across domains such as economics, public health, and social demographics, decisions hinge on numerical estimates over continuous quantities, a capability that current benchmarks do not capture. Evaluating such estimates requires a format that makes uncertainty explicit and testable. We propose prediction intervals as a natural and rigorous interface for this purpose. They demand scale awareness, internal consistency across confidence levels, and calibration over a continuum of outcomes, making them a more suitable evaluation format than point estimates for numerical forecasting. To assess this capability, we introduce a new benchmark QuantSightBench, and evaluate frontier models under multiple settings, assessing both empirical coverage and interval sharpness. Our results show that none of the 11 evaluated frontier and open-weight models achieves the 90% coverage target, with the top performers Gemini 3.1 Pro (79.1%), Grok 4 (76.4%), and GPT-5.4 (75.3%) all falling at least 10 percentage points short. Calibration degrades sharply at extreme magnitudes, revealing systematic overconfidence across all evaluated models.
Jeremy Qin, Maksym Andriushchenko
ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems · Tübingen AI Center
Frontier-model leaderboards now rank systems on economic benchmarks, and those rankings inform what organisations buy and what regulators scrutinise. Whether such benchmarks measure a capability distinct from general test-taking is a question of construct validity that a structural test and a predictive test can answer in opposite ways. We show that they do on a hash-pinned snapshot of a frontier leaderboard with 421 model configurations across twelve benchmarks, four of them economic, of which 103 configurations carry all three sparsely scored economic benchmarks and 96 carry all twelve; four hypotheses and their thresholds were fixed before analysis, and every deviation from the plan is reported. The first factor of a three-factor extraction carries 74.5% of common variance and tracks release date (R^2 = 0.505), and date adjustment lowers its share by 14.9 points. Under the dimensionality rule fixed in advance the economic benchmarks form no factor of their own. A leave-one-benchmark-out test with factors re-estimated inside every fold nevertheless finds that a multi-factor representation predicts held-out economic scores better than a single general index (pooled Delta-MSE 0.037, 95% bootstrap interval [0.019, 0.055]) under the linear learners that fit best, an advantage that reverses for tree learners. Under the linear learners the same representation also predicts the eight other benchmarks better, so the battery carries predictive structure that one index misses and the economic benchmarks share it without forming a distinct factor. Construct validity should therefore be assessed by predictive tests alongside structural ones. We give a two-test protocol for benchmark builders and release the pinned data, the analysis plan and the code.
Forecasting leaderboards rank models by predictive quality, but their winners are often read as deployment-ready top-1 advice. That reading can fail when forecasts are passed through a fixed decision interface, such as an alert threshold, a top-k budget, or a switching-cost policy. We study when a forecast-side winner can be certified as deployment-actionable for a specified interface and deployed utility. We introduce a fail-closed certification protocol whose gates are sufficient evidential conditions for a strong claim: a friction-caused, non-tie, statistically supported, and recurrent deployment-side reversal. Traffic-Hourly provides a certified anchor: winners agree at zero friction, but positive switching friction makes the forecast winner deployed-suboptimal. A locked native audit tests overclaiming: across 22 verified candidates and 362 full-grid cells, 155 apparent forecast/deployment winner inversions are blocked before certification. The contribution is not a new forecaster, metric, or universal utility, but a conservative protocol for deciding when forecasting leaderboard winners should be read as deployment-actionable top-1 advice.