cs.LGAug 21, 2026
SaveGlobal tree forecasters collapse at the hierarchical aggregate: a five-panel failure characterization
Organizations: Field Nation LLC, Minneapolis, MN, USA
Abstract
Global forecasting models pool many series and learn one shared function. Gradient-boosted trees are their most common form. We measure a failure of this design that has not, to our knowledge, been documented. Train a global tree on the individual series of a hierarchy, then ask it for the hierarchical aggregate. The aggregate sits far outside the model's training range, and the forecast collapses. The model under-predicts the total by 30-50x in our production deployment, and by up to 496x in a public M5 reconstruction. The mechanism is known: beyond its training range, a tree predicts a constant. It surfaces at the aggregate because the total dwarfs every training series. The cure is not new. Per-series scaling, the preprocessing step that Montero-Manso and Hyndman (2021) recommend, prevents the collapse. So do a weighted aggregate-level training row and seasonal differencing. Our contribution is the characterization. The collapse reproduces on five panels: a production business-to-business marketplace, a synthetic hierarchy, M5, Australian Tourism, and a public business-buyer panel. It holds on three tree libraries, is invariant across training seeds, and is statistically significant. Its onset is immediate and tracks a simple support bound: a scale gap of only 1.15x already costs a third of the total. No standard configuration change prevents it: pooling every hierarchy level into training fails at scale, and the one knob that fits linear models in the leaves softens it without curing it. Rolling the forecasts forward recursively separates the cures: the aggregate-row cure re-collapses, per-series scaling degrades but stays low, and only seasonal differencing keeps its one-step accuracy unchanged. We close with a three-step procedure for diagnosing and preventing the failure in deployed systems.
Figures & tables
Figure 1: Full marketplace aggregate monthly GTV (summed over all buyers, indexed to Jan 2021 ), Jan 2021–May 2026, shown on a calibrated synthetic surrogate (Section 3 ; not the proprietary series). The aggregate trends upward with a recurring seasonal cycle; every calendar year posts higher GTV than the prior. The five-month held-out evaluation window (Jan–May 2026) is shaded. The per-buyer series feeding this total are lumpy, intermittent, heavy-tailed, and short—the gap this paper turns on.
Figure 2: Why a global tree collapses at the aggregate, and the two cures. (a) A regression tree extrapolates as a constant: it cannot emit a value above its largest leaf—the largest buyer seen in training (the ceiling). The marketplace aggregate sits above the median buyer and outside the per-buyer training support, so the out-of-support input is routed to the ceiling and the forecast under-predicts the total by – . (b) Either axis returns the target to within support: restoring scale—per-series scaling, or a single cohort-weighted aggregate training row—or removing scale—seasonal differencing, the route least harmed by recursive multi-step forecasting. Conceptual schematic; the quantities are those reported in this section.
Figure 3: The onset of the collapse tracks the support bound. One-step aggregate MAPE of the baseline global tree against the measured scale gap (aggregate over largest single training series), swept on the synthetic panel via a dominant series that holds shape fixed while scale varies; three data seeds and both losses per gap setting. Every point sits at or above the floor (solid line), the error rises monotonically with the gap, and the grey squares anchor the sweep to the paper’s measured panels. A gap already produces error: there is no safe band above .
| Panel | Loss | Gap | Implied floor | Measured MAPE |
|---|---|---|---|---|
| Synthetic | MAE | |||
| Synthetic | RMSE | |||
| Australian Tourism | MAE | |||
| Australian Tourism | RMSE | |||
| M5 | MAE | |||
| M5 | RMSE |
Table 1: The measured ceilings close the loop on every public panel. For each fitted baseline, the realized ceiling is the largest value the model emits when queried at the aggregate; the implied floor is , the bound of Eq. ( 3 ) at the measured gap. Under both losses the floor sits no more than points below the measured one-step collapse: the support bound does not just permit the collapse, it predicts its size. One fit per cell (tree seed ); no emission exceeded the largest training target (largest ratio ).
| Configuration | One-step aggregate MAPE |
|---|---|
| Baseline: absolute lags, MAE, per-buyer only | 94.7% |
| (a) target/lags | 94.5% |
| (b) RMSE loss on log scale | 95.4% |
| (c) cohort-weighted aggregate row | 17.5% |
| (d) recursive guardrail (full fix) | 17.5% |
Table 2: Cumulative ablation of the four-component recipe on a synthetic hierarchical panel (one-step aggregate MAPE, mean over six data seeds; Section 8.1 ). Component (c)—the cohort-weighted aggregate-level row—is the load-bearing lever; (a) and (b) do nothing alone because the collapse is a training-support problem, not a loss or transform problem.
| Configuration | Synthetic | Production | Role |
|---|---|---|---|
| Full fix (reference) | 17.5% | 15.1% | — |
| (c) cohort-weighted aggregate row | 95.4% | 97.8% | load-bearing |
| (b) RMSE loss (use MAE) | 202% | 99.6% | load-bearing |
| cohort weight only (row kept) | 31.1% | 79.6% | decisive at extreme gap |
| (a) transform | 15.7% | 18.6% | refinement |
| (d) recursive guardrail | 17.5% | 15.1% | dormant guardrail |
Table 3: Leave-one-out ablation: one-step aggregate MAPE when each component is removed from the full recipe, on the synthetic panel (mean over six seeds) and the production panel (pinned Jan–May 2026). Two components are load-bearing on both panels and interact; the log transform and the recursive cap are defensive refinements, not individually necessary.
Figure 4: Recursive aggregate error, resolved by step (five-seed mean; shaded bands one standard deviation; log scale). The cohort row re-collapses immediately on the synthetic panel and climbs a staircase on M5; per-series scaling drifts on M5; seasonal differencing is the lowest-error cure at nearly every step. The window means of Table 4 conceal these dynamics.
| Dataset | Baseline | Per-series | Cohort row | Seasonal diff. |
|---|---|---|---|---|
| One-step aggregate MAPE | ||||
| Synthetic | ||||
| Production | ||||
| M5 | ||||
| Tourism | ||||
| Recursive aggregate MAPE | ||||
Table 4: Three cures across two axes: aggregate MAPE (five-seed mean sd), one-step and recursive, identical recipe. The naive global tree collapses on every panel. Scale axis: per-series scaling and the single-model cohort-weighted row both restore scale (scaling the more accurate one-step). Trend axis: seasonal differencing makes the target stationary. Under recursion the cohort row re-collapses, per-series scaling degrades but stays low, and seasonal differencing alone keeps its one-step accuracy unchanged; its seed variance is the lowest of the cures, while the cohort row’s recursive error is seed-fragile (M5 ). Values are from the five-seed cures-comparison run; the synthetic baseline ( ) is thus points below the six-seed cumulative-ablation baseline ( , Table 2 ), run-to-run variation immaterial to the collapse.
Figure 5: Synthetic reproduction of the collapse and fix. The baseline global tree (flat line) under-predicts the smooth synthetic aggregate; the cohort-weighted recipe tracks it. Mechanism confirmation, not MAPE replication—synthetic per-buyer series carry no learnable cross-buyer structure, so recovery stops at .
| Configuration / library | Baseline | Four-component fix |
|---|---|---|
| LightGBM (ablation) | 99.8% | 14.1% |
| XGBoost | 99.9% | 6.3% |
| CatBoost | 99.8% | 16.8% |
Table 5: Public replication on M5 (30,490 monthly item-store series, 12-month holdout). One-step aggregate MAPE. The collapse and the four-component recipe reproduce; the cohort-weighted aggregate row is the load-bearing lever, exactly as on the synthetic and production panels.
Figure 6: Collapse and fix replicated on public M5. Left: the baseline global tree collapses on the M5 grand total while the four-component recipe tracks it. Right: the cumulative ablation, with the cohort-weighted aggregate row as the decisive lever—the same structure as the synthetic (Figure 5 ) and production panels.
| Configuration / library | MAPE | MASE |
|---|---|---|
| Baseline (LightGBM, no fix) | 96.2% | 16.4 |
| Full fix — LightGBM | 14.0% | 2.30 |
| Full fix — XGBoost | 5.2% | 0.90 |
| Full fix — CatBoost | 11.0% | 1.84 |
Table 6: Second public replication on Australian Tourism ( monthly series, -month holdout, aggregate-to-median gap). One-step aggregate MAPE / seasonal MASE. The collapse and the library-agnostic recipe reproduce in a non-retail domain.
| Panel | Domain | Scale gap | Baseline | Full fix |
|---|---|---|---|---|
| Synthetic | hierarchical sim. | |||
| Production | B2B field service | – | ||
| M5 | retail (Walmart) | |||
| Australian Tourism | visitor nights | |||
| UCI Online Retail II | B2B wholesale |
Table 7: Generalization of the collapse and the four-component recipe across all five panels, ordered by scale gap. One-step aggregate MAPE, baseline (global tree, no fix) versus the recipe. The scale-gap column reports each panel’s measured out-of-support ratio: the aggregate over the largest single training series for Synthetic and M5; the aggregate over the median series level for Tourism and Retail II; and, for Production, the measured aggregate-to-leaf-ceiling gap of Eq. ( 3 ) ( LightGBM, CatBoost; the aggregate-to-median-buyer ratio is the larger ). The collapse (baseline ) and its recovery hold across five domains: to against the largest training series where that ratio is measured, and up to against the median. Production, M5, and Tourism are rolling-origin means (Table 8 )—so the production fix here ( ) is the twelve-origin mean, distinct from the single pinned-holdout value carried by the ablation tables (Table 3 ); Synthetic is a six-seed mean and UCI Online Retail II a single -month holdout. The fitted models’ own ceilings are also measured on every public panel (Table 1 ): gaps – , with the ceiling-implied floor no more than points below the measured collapse. The mechanism additionally reconfirms on the public Olist marketplace (Section 10 ).
| Rolling | Diebold–Mariano | Wilcoxon | Mean MAPE | ||
|---|---|---|---|---|---|
| Dataset | origins | one-step | recursive | base fix | |
| Production | 12 | ||||
| Tourism | 24 | ||||
| M5 | 10 | ||||
Table 8: The aggregate collapse–fix gap is statistically significant on every dataset. Diebold–Mariano statistic (Harvey–Leybourne–Newbold small-sample correction) for the four-component recipe versus the collapsed baseline, on the per-step absolute-percentage-error loss differential pooled across rolling origins; a positive value favors the fix (read as an effect size—see text). The Wilcoxon column is the distribution-free signed-rank test across origins at its one-sided all-same-sign floor; the text reads it as an effect-size confirmation.
| Model | Mean sd | Range | Reading |
|---|---|---|---|
| CatBoost (no cohort wt.) | 89.1–92.9 | collapses every seed | |
| XGBoost (cohort-wt.) | 11.1–18.3 | stable, never collapses | |
| LightGBM (cohort-wt.) | 15.7–49.8 | forecasts agg. every seed |
Table 9: Eight-seed variance band on the stochastic tree models: pinned Jan–May 2026 aggregate MAPE through the production recursive serving path at each model’s shipped configuration, varying only the training seed. The recursive regime and the shipped configuration are why LightGBM’s band sits above its one-step cures-comparison value ( , Table 4 ). The mechanism (collapse versus not) is seed-invariant; LightGBM’s point MAPE is seed-sensitive.
| Probe / intervention | Outcome |
|---|---|
| Training-window length ( – mo) | Not a cure: the baseline collapses at every length. |
| Entity id hyperparameter sweep richer features | Still collapsed ( – ): a training-support property, not a capacity one. |
| Direct (non-recursive) horizons; largest-decile segmentation | Remove recursive drift but still need a scale cure. |
| Pooling all hierarchy levels into training | Cures a small hierarchy (Tourism ) but fails at M5 scale ( ): the in-support rows are washed out. |
| Tweedie loss; self-contained post-hoc multiplier | Neither cures: the residual re-collapse is a slope, not a level, problem. |
| Linear-leaf trees ( linear_tree ), incl. a linear_lambda sweep | Partial, inconsistent mitigation at every regularization: under MAE; – one-step under RMSE, seed-fragile. |
Table 10: Robustness map: each probe and its outcome. No single-axis intervention substitutes for restoring aggregate scale. The paragraphs below expand each row in order.