Global forecasting models pool many series and learn one shared function. Gradient-boosted trees are their most common form. We measure a failure of this design that has not, to our knowledge, been documented. Train a global tree on the individual series of a hierarchy, then ask it for the hierarchical aggregate. The aggregate sits far outside the model's training range, and the forecast collapses. The model under-predicts the total by 30-50x in our production deployment, and by up to 496x in a public M5 reconstruction. The mechanism is known: beyond its training range, a tree predicts a constant. It surfaces at the aggregate because the total dwarfs every training series. The cure is not new. Per-series scaling, the preprocessing step that Montero-Manso and Hyndman (2021) recommend, prevents the collapse. So do a weighted aggregate-level training row and seasonal differencing. Our contribution is the characterization. The collapse reproduces on five panels: a production business-to-business marketplace, a synthetic hierarchy, M5, Australian Tourism, and a public business-buyer panel. It holds on three tree libraries, is invariant across training seeds, and is statistically significant. Its onset is immediate and tracks a simple support bound: a scale gap of only 1.15x already costs a third of the total. No standard configuration change prevents it: pooling every hierarchy level into training fails at scale, and the one knob that fits linear models in the leaves softens it without curing it. Rolling the forecasts forward recursively separates the cures: the aggregate-row cure re-collapses, per-series scaling degrades but stays low, and only seasonal differencing keeps its one-step accuracy unchanged. We close with a three-step procedure for diagnosing and preventing the failure in deployed systems.
Figures & tables
Figure 1: Full marketplace aggregate monthly GTV (summed over all buyers, indexed to Jan 2021 =100 ), Jan 2021–May 2026, shown on a calibrated synthetic surrogate (Section 3 ; not the proprietary series). The aggregate trends upward with a recurring seasonal cycle; every calendar year posts higher GTV than the prior. The five-month held-out evaluation window (Jan–May 2026) is shaded. The per-buyer series feeding this total are lumpy, intermittent, heavy-tailed, and short—the gap this paper turns on.
Figure 2: Why a global tree collapses at the aggregate, and the two cures. (a) A regression tree extrapolates as a constant: it cannot emit a value above its largest leaf—the largest buyer seen in training (the ceiling). The marketplace aggregate sits ≈1,088× above the median buyer and outside the per-buyer training support, so the out-of-support input is routed to the ceiling and the forecast under-predicts the total by 30 – 50× . (b) Either axis returns the target to within support: restoring scale—per-series scaling, or a single cohort-weighted aggregate training row—or removing scale—seasonal differencing, the route least harmed by recursive multi-step forecasting. Conceptual schematic; the quantities are those reported in this section.
Figure 3: The onset of the collapse tracks the support bound. One-step aggregate MAPE of the baseline global tree against the measured scale gap g (aggregate over largest single training series), swept on the synthetic panel via a dominant series that holds shape fixed while scale varies; three data seeds and both losses per gap setting. Every point sits at or above the 1−1/g floor (solid line), the error rises monotonically with the gap, and the grey squares anchor the sweep to the paper’s measured panels. A 1.15× gap already produces 34% error: there is no safe band above g=1 .
Panel
Loss
Gap A/c^
Implied floor
Measured MAPE
Synthetic
MAE
14.5×
93.0%
93.3%
Synthetic
RMSE
11.6×
91.3%
92.2%
Australian Tourism
MAE
17.2×
94.0%
96.2%
Australian Tourism
RMSE
14.5×
92.9%
96.3%
M5
MAE
487.9×
99.8%
99.8%
M5
RMSE
271.6×
99.6%
99.7%
Table 1: The measured ceilings close the loop on every public panel. For each fitted baseline, the realized ceiling c^ is the largest value the model emits when queried at the aggregate; the implied floor is 1−c^/A , the bound of Eq. ( 3 ) at the measured gap. Under both losses the floor sits no more than 3.4 points below the measured one-step collapse: the support bound does not just permit the collapse, it predicts its size. One fit per cell (tree seed 42 ); no emission exceeded the largest training target (largest ratio 0.762 ).
Configuration
One-step aggregate MAPE
Baseline: absolute lags, MAE, per-buyer only
94.7%
+ (a) log(1+y) target/lags
94.5%
+ (b) RMSE loss on log scale
95.4%
+ (c) cohort-weighted aggregate row
17.5%
+ (d) 2× recursive guardrail (full fix)
17.5%
Table 2: Cumulative ablation of the four-component recipe on a synthetic hierarchical panel (one-step aggregate MAPE, mean over six data seeds; Section 8.1 ). Component (c)—the cohort-weighted aggregate-level row—is the load-bearing lever; (a) and (b) do nothing alone because the collapse is a training-support problem, not a loss or transform problem.
Configuration
Synthetic
Production
Role
Full fix (reference)
17.5%
15.1%
—
− (c) cohort-weighted aggregate row
95.4%
97.8%
load-bearing
− (b) RMSE loss (use MAE)
202%
99.6%
load-bearing
− cohort weight only (row kept)
31.1%
79.6%
decisive at extreme gap
− (a) log(1+y) transform
15.7%
18.6%
refinement
− (d) 2× recursive guardrail
17.5%
15.1%
dormant guardrail
Table 3: Leave-one-out ablation: one-step aggregate MAPE when each component is removed from the full recipe, on the synthetic panel (mean over six seeds) and the production panel (pinned Jan–May 2026). Two components are load-bearing on both panels and interact; the log transform and the recursive cap are defensive refinements, not individually necessary.
Figure 4: Recursive aggregate error, resolved by step (five-seed mean; shaded bands one standard deviation; log scale). The cohort row re-collapses immediately on the synthetic panel and climbs a staircase on M5; per-series scaling drifts on M5; seasonal differencing is the lowest-error cure at nearly every step. The window means of Table 4 conceal these dynamics.
Dataset
Baseline
Per-series
Cohort row
Seasonal diff.
One-step aggregate MAPE
Synthetic
93.2±0.0
11.7±0.7
14.9±0.9
6.9±0.1
Production
97.8±0.0
5.4±0.3
15.1±0.3
11.0±0.0
M5
99.8±0.0
8.2±0.1
14.7±1.5
10.0±0.0
Tourism
96.1±0.1
9.6±0.6
14.0±0.1
4.6±0.0
Recursive aggregate MAPE
Table 4: Three cures across two axes: aggregate MAPE (five-seed mean ± sd), one-step and recursive, identical recipe. The naive global tree collapses on every panel. Scale axis: per-series scaling and the single-model cohort-weighted row both restore scale (scaling the more accurate one-step). Trend axis: seasonal differencing makes the target stationary. Under recursion the cohort row re-collapses, per-series scaling degrades but stays low, and seasonal differencing alone keeps its one-step accuracy unchanged; its seed variance is the lowest of the cures, while the cohort row’s recursive error is seed-fragile (M5 ±30.8 ). Values are from the five-seed cures-comparison run; the synthetic baseline ( 93.2% ) is thus ≈1.5 points below the six-seed cumulative-ablation baseline ( 94.7% , Table 2 ), run-to-run variation immaterial to the collapse.
Figure 5: Synthetic reproduction of the collapse and fix. The baseline global tree (flat line) under-predicts the smooth synthetic aggregate; the cohort-weighted recipe tracks it. Mechanism confirmation, not MAPE replication—synthetic per-buyer series carry no learnable cross-buyer structure, so recovery stops at 17.5% .
Configuration / library
Baseline
Four-component fix
LightGBM (ablation)
99.8%
14.1%
XGBoost
99.9%
6.3%
CatBoost
99.8%
16.8%
Table 5: Public replication on M5 (30,490 monthly item-store series, 12-month holdout). One-step aggregate MAPE. The collapse and the four-component recipe reproduce; the cohort-weighted aggregate row is the load-bearing lever, exactly as on the synthetic and production panels.
Figure 6: Collapse and fix replicated on public M5. Left: the baseline global tree collapses on the M5 grand total while the four-component recipe tracks it. Right: the cumulative ablation, with the cohort-weighted aggregate row as the decisive lever—the same structure as the synthetic (Figure 5 ) and production panels.
Configuration / library
MAPE
MASE
Baseline (LightGBM, no fix)
96.2%
16.4
Full fix — LightGBM
14.0%
2.30
Full fix — XGBoost
5.2%
0.90
Full fix — CatBoost
11.0%
1.84
Table 6: Second public replication on Australian Tourism ( 304 monthly series, 12 -month holdout, ≈598× aggregate-to-median gap). One-step aggregate MAPE / seasonal MASE. The collapse and the library-agnostic recipe reproduce in a non-retail domain.
Panel
Domain
Scale gap
Baseline
Full fix
Synthetic
hierarchical sim.
11.5×
94.7%
17.5%
Production
B2B field service
43 – 52×
97.9%
9.5%
M5
retail (Walmart)
113×
99.8%
12.6%
Australian Tourism
visitor nights
598×
95.9%
11.1%
UCI Online Retail II
B2B wholesale
2,062×
98.7%
36.8%
Table 7: Generalization of the collapse and the four-component recipe across all five panels, ordered by scale gap. One-step aggregate MAPE, baseline (global tree, no fix) versus the recipe. The scale-gap column reports each panel’s measured out-of-support ratio: the aggregate over the largest single training series for Synthetic and M5; the aggregate over the median series level for Tourism and Retail II; and, for Production, the measured aggregate-to-leaf-ceiling gap g of Eq. ( 3 ) ( 42.7× LightGBM, 51.9× CatBoost; the aggregate-to-median-buyer ratio is the larger ≈1,088× ). The collapse (baseline ≳90% ) and its recovery hold across five domains: 8× to 113× against the largest training series where that ratio is measured, and up to 2,062× against the median. Production, M5, and Tourism are rolling-origin means (Table 8 )—so the production fix here ( 9.5% ) is the twelve-origin mean, distinct from the 15.1% single pinned-holdout value carried by the ablation tables (Table 3 ); Synthetic is a six-seed mean and UCI Online Retail II a single 25 -month holdout. The fitted models’ own ceilings are also measured on every public panel (Table 1 ): gaps 11.6 – 488× , with the ceiling-implied floor no more than 3.4 points below the measured collapse. The mechanism additionally reconfirms on the public Olist marketplace (Section 10 ).
Rolling
Diebold–Mariano
Wilcoxon
Mean MAPE
Dataset
origins
one-step
recursive
p
base → fix
Production
12
179.5
36.4
2×10−4
97.9%→9.5%
Tourism
24
250.5
199.2
6×10−8
95.9%→11.1%
M5
10
57.6
7.1
1×10−3
99.8%→12.6%
Table 8: The aggregate collapse–fix gap is statistically significant on every dataset. Diebold–Mariano statistic (Harvey–Leybourne–Newbold small-sample correction) for the four-component recipe versus the collapsed baseline, on the per-step absolute-percentage-error loss differential pooled across rolling origins; a positive value favors the fix (read as an effect size—see text). The Wilcoxon column is the distribution-free signed-rank test across origins at its one-sided all-same-sign floor; the text reads it as an effect-size confirmation.
Model
Mean ± sd
Range
Reading
CatBoost (no cohort wt.)
91.9±1.2
89.1–92.9
collapses every seed
XGBoost (cohort-wt.)
13.9±2.6
11.1–18.3
stable, never collapses
LightGBM (cohort-wt.)
31.9±12.0
15.7–49.8
forecasts agg. every seed
Table 9: Eight-seed variance band on the stochastic tree models: pinned Jan–May 2026 aggregate MAPE through the production recursive serving path at each model’s shipped configuration, varying only the training seed. The recursive regime and the shipped configuration are why LightGBM’s band sits above its one-step cures-comparison value ( 15.1±0.3 , Table 4 ). The mechanism (collapse versus not) is seed-invariant; LightGBM’s point MAPE is seed-sensitive.
Probe / intervention
Outcome
Training-window length ( 36 – 120 mo)
Not a cure: the baseline collapses at every length.
Entity id + hyperparameter sweep + richer features
Still collapsed ( 98 – 100% ): a training-support property, not a capacity one.
Direct (non-recursive) horizons; largest-decile segmentation
Remove recursive drift but still need a scale cure.
Pooling all hierarchy levels into training
Cures a small hierarchy (Tourism 96→14% ) but fails at M5 scale ( 99.7→69% ): the in-support rows are washed out.
Tweedie loss; self-contained post-hoc multiplier
Neither cures: the residual re-collapse is a slope, not a level, problem.
Linear-leaf trees ( linear_tree ), incl. a linear_lambda sweep
Partial, inconsistent mitigation at every regularization: 99.9% under MAE; 12 – 70% one-step under RMSE, seed-fragile.
Table 10: Robustness map: each probe and its outcome. No single-axis intervention substitutes for restoring aggregate scale. The paragraphs below expand each row in order.
This paper explores the "Granularity Paradox" in time-series forecasting, wherein finer temporal disaggregation (e.g., Monthly to Weekly/Daily) improves in-sample diagnostics and dataset size (N), but degrades out-of-sample accuracy due to recursive error compounding over longer horizons (H). Conversely, coarse aggregation (Annual) eliminates recursive error propagation but reduces data available to estimators. We formalize this trade-off and benchmark 10 models - spanning naïve, statistical, machine learning, and deep learning architectures - across six granularities using a 13-year public procurement dataset. The empirical results reveal a non-monotonic threshold structure: recursive autoregressive and seasonal models degrade substantially under high-frequency forecasting (e.g., Holt-Winters reaches a Test R-squared of -151 and TPFE of 425.85% at the Daily grain), while the LSTM traces a U-shaped error curve, worsening from Monthly (19.66%) through Bi-Weekly (35.94%) before overcoming the error propagation penalty at Daily (TPFE of 4.35%, R-squared of 0.66). Linear Regression remains stable across all granularities (16.3-17.0% TPFE), confirming that the paradox is driven by recursive feedback topology, not model complexity. The results demonstrate that standard pointwise metrics (RMSE, MAE) systematically mask cumulative error propagation, and that evaluating forecasts without goal-dependent cumulative metrics produces misleading assessments of model adequacy. We introduce a consensus-dissensus diagnostic comparing the directional behaviour of pointwise metrics against cumulative TPFE across granularities, enabling the identification of models whose standard diagnostics mask systematic error propagation.
Hugo Moreira
1CIES - Centro de Investigação e Estudos de Sociologia, Iscte - Instituto Universitário de Lisboa, Lisbon, Portugal.
When forecasting hourly returns for 1,000 US equities, we observe an unexpected phenomenon: predictions become nearly flat and show poor stock ranking, as measured by cross-sectional correlation. We call this forecast collapse. Surprisingly, the phenomenon largely disappears when forecasting trading volume under the same setting. We investigate forecast collapse across time-series foundation models (TSFMs), twelve deep-learning forecasting models, and 97 public benchmark configurations, and find that it is closely tied to target predictability. We identify two distinct reasons behind it: low predictability limits the amplitude of calibrated point forecasts, while per-series objectives leave cross-series structure unidentified. These findings reveal a calibration-ranking tradeoff: optimizing squared error leads to flat predictions, whereas directly optimizing cross-sectional correlation improves ranking but can inflate forecast amplitude by more than an order of magnitude. To address this tradeoff, we introduce CalibRank, a simple objective that balances calibration and ranking. On Finance1K, CalibRank nearly triples cross-sectional correlation while keeping amplitude close to the target, and improves correlation on all tested models. Our results reveal a blind spot in conventional time-series evaluation: per-series metrics can hide failures in cross-series structure needed by downstream decisions.
Shu Wan, Miles Ma, Hank Zhu +4
Abel AI Lab · Arizona State University · University of Oxford
Large language models (LLMs) are increasingly used to forecast real-world events, but access to the strongest individual forecaster may be costly or otherwise constrained. We study weak-to-strong forecast aggregation: can individually weaker LLM forecasters be aggregated to outperform a stronger forecaster? Using ForecastBench (Karger et al., 2025), we evaluate 70 LLM forecasters across 16 comparison groups, each with more than 1,000 shared subquestions, yielding 1,121 weaker-model pairs. Within each group, we identify the strongest individual by test Brier score and evaluate aggregates composed exclusively of weaker forecasters, with aggregation weights learned on separate training data. We find substantial evidence of weak-to-strong improvement. Learned linear pooling identifies a weaker pair that matches or outperforms the strongest individual in 11 of 16 groups and comes within 5% of its Brier score in all 16 groups. We also find that these improvements do not rely on having a near-best constituent and are generally accompanied by good calibration. Additional analyses show that adding more models does not consistently improve performance, and competitive weaker-model aggregates also remain available under practical constraints.