Conformal Expert Pools: Training-Free Probabilistic Time-Series Forecasting Beyond Seasonality
Organizations: Independent researcher
Abstract
Conformal Seasonal Pools (CSP) showed that a forecaster with no training at all, built only from past values at the same point in the season and from past forecast errors, significantly outperformed DeepNPTS, Amazon's learned non-parametric forecaster. We propose Conformal Expert Pools (CEP), a training-free successor. CEP replaces the single seasonal-naive forecast by a pool of simple forecasters, builds its predictive distribution from their past forecast errors, and adapts both the pool and the errors to the series being forecast. No model is trained and no parameter is learned across series. On the CSP benchmark of 232 datasets and 24,502 forecast windows, CEP is better than CSP on 78% of datasets by CRPS, on 84% by MSIS and on 83% by Winkler score; on a typical dataset its CRPS is 14% lower and its MSIS 23% lower (both p < 1e-22, paired Wilcoxon test), and coverage of 95% intervals rises from 90.1% to 92.2%. CEP has a lower CRPS than DeepNPTS on 88% of datasets, than NPTS on 89% and than SeasonalNPTS on 83%. The improvement persists on dataset families outside the development set and on 36 additional fev-bench files. We also test the published claims that Laplace, an online ensemble forecaster, "beats (almost) everything" and "beats" CSP. On 228 datasets at multi-step horizons the claims do not hold: CSP outperforms Laplace on 60% of datasets by CRPS and on 64% by interval score, and Laplace ranks sixth of seven methods, with the widest intervals of all (69% wider than CEP's). Its point forecast is biased low and on 54% of datasets is less accurate than seasonal naive, and from a cold start it takes about 7,700 times longer than CSP. CEP is better than Laplace on 82% of datasets by CRPS and on 80% by MSIS, with intervals about 40% narrower, and runs 271 times faster.
Figures & tables
| Method | CRPS rank | MSIS rank | CRPS | MSIS | MSIS (mean) | Width | Coverage |
|---|---|---|---|---|---|---|---|
| CEP | 1.81 | 1.77 | 0.749 | 7.98 | 24.4 | 5.59 | 0.922 |
| CSP | 3.06 | 3.22 | 0.841 | 10.23 | 29.4 | 6.40 | 0.901 |
| CSP-Nixtla | 3.36 | 3.51 | 0.842 | 10.17 | 36.3 | 6.10 | 0.872 |
| SeasonalNPTS | 3.99 | 4.06 | 0.900 | 11.38 | 43.3 | 4.79 | 0.763 |
| DeepNPTS | 4.08 | 4.91 | 0.960 | 18.40 | 48.1 | 2.76 | 0.578 |
| NPTS | 4.71 | 3.53 | 1.284 | 10.58 | 41.5 | 7.74 | 0.857 |
| CEP vs | CRPS wins | CRPS ratio (95% CI) | MSIS wins | MSIS ratio (95% CI) | Winkler wins | (CRPS) | (MSIS) |
|---|---|---|---|---|---|---|---|
| CSP | 78% | 0.86 (0.83, 0.88) | 84% | 0.77 (0.74, 0.80) | 83% | ||
| CSP-Nixtla | 81% | 0.84 (0.80, 0.87) | 87% | 0.71 (0.67, 0.75) | 86% | ||
| DeepNPTS | 88% | 0.76 (0.72, 0.80) | 95% | 0.46 (0.42, 0.50) | 92% | ||
| SeasonalNPTS | 83% | 0.67 (0.60, 0.73) | 81% | 0.62 (0.57, 0.67) | 78% | ||
| NPTS | 89% | 0.57 (0.52, 0.62) | 76% | 0.66 (0.60, 0.71) | 73% |
| Comparison | Wins | Ratio (95% CI) | Median | |
|---|---|---|---|---|
| Corpus: CEP against | ||||
| CSP | 78% | 0.87 (0.84, 0.89) | 0.194 | |
| CSP-Nixtla | 77% | 0.88 (0.85, 0.90) | 0.193 | |
| DeepNPTS | 81% | 0.87 (0.83, 0.91) | 0.209 | |
| SeasonalNPTS | 80% | 0.72 (0.67, 0.77) | 0.267 | |
| NPTS | 84% | 0.64 (0.60, 0.69) | 0.297 | |
| Split | Coverage | CRPS wins | CRPS ratio (95% CI) | MSIS wins | MSIS ratio (95% CI) | Winkler wins | |
|---|---|---|---|---|---|---|---|
| CEP | CSP | ||||||
| Development | 0.941 | 0.871 | 86% | 0.80 (0.73, 0.88) | 86% | 0.66 (0.55, 0.78) | 86% |
| Related families | 0.917 | 0.874 | 83% | 0.89 (0.86, 0.92) | 83% | 0.79 (0.73, 0.85) | 86% |
| Unseen families | 0.922 | 0.915 | 76% | 0.85 (0.81, 0.89) | 83% | 0.78 (0.74, 0.82) | 81% |
| Method | Coverage | CRPS | MSIS | Width | CRPS wins | MSIS wins | Winkler wins | (CRPS) | (MSIS) |
|---|---|---|---|---|---|---|---|---|---|
| CSP | 0.877 | 1.016 | 15.52 | 7.29 | – | – | – | – | – |
| CEP | 0.904 | 0.886 | 13.27 | 7.33 | 78% | 81% | 81% |
| Comparison published by Laplace’s author | This paper | |
|---|---|---|
| Forecast horizon | One step ahead only | Multi-step, to steps |
| Data | series in four collections: daily, weekly and monthly FRED economic change series, and M4-Hourly | datasets: energy, transport, retail, web traffic, finance, economics, health, weather, cloud workloads, M-competitions |
| Choice of data | By Laplace’s author | The CSP corpus, fixed beforehand |
| Probabilistic metrics | CRPS and log-likelihood | CRPS, MSIS, Winkler score, coverage, interval width |
| Interval width, interval score | Not reported | Reported |
| Point accuracy | Not reported | MASE, RMSSE, bias |
| Method | CRPS | MSIS | Width | CRPS wins | MSIS wins | Winkler wins | (CRPS) | (MSIS) |
|---|---|---|---|---|---|---|---|---|
| CEP | 0.756 | 7.52 | 5.57 | 82% | 80% | 82% | ||
| CSP | 0.824 | 10.12 | 6.54 | 60% | 64% | 64% | ||
| Laplace | 0.960 | 11.68 | 9.40 | – | – | – | – | – |
| Method | CRPS rank | MSIS rank | CRPS | MSIS | Width | CRPS wins | MSIS wins |
|---|---|---|---|---|---|---|---|
| CEP | 2.23 | 2.38 | 0.756 | 7.52 | 5.57 | 82% | 80% |
| CSP-Nixtla | 3.75 | 3.94 | 0.819 | 9.45 | 5.93 | 58% | 61% |
| CSP | 3.81 | 3.73 | 0.824 | 10.12 | 6.54 | 60% | 64% |
| DeepNPTS | 4.36 | 5.22 | 0.973 | 18.28 | 2.50 | 50% | 38% |
| SeasonalNPTS | 4.45 | 4.19 | 0.920 | 10.73 | 4.61 | 51% | 59% |
| Laplace | 4.50 | 4.67 | 0.960 | 11.68 | 9.40 | – | – |
| Method | CRPS | MSIS | Width | CRPS wins | MSIS wins | Winkler wins |
|---|---|---|---|---|---|---|
| CEP | 0.948 | 9.60 | 7.36 | 81% | 69% | 72% |
| CSP | 0.995 | 10.54 | 7.78 | 61% | 53% | 53% |
| Laplace | 1.046 | 19.26 | 12.58 | – | – | – |
| Test set | Method | MASE | RMSSE | WAPE | rel. MAE | Bias | MASE wins | RMSSE wins |
|---|---|---|---|---|---|---|---|---|
| Corpus, full history ( ) | CEP | 1.012 | 0.867 | 0.170 | 0.748 | -0.091 | 78% | 76% |
| CSP | 1.101 | 0.945 | 0.183 | 0.875 | -0.060 | – | – | |
| Corpus, Laplace subset ( ) | CEP | 1.021 | 0.867 | 0.150 | 0.772 | -0.022 | 71% | 70% |
| CSP | 1.128 | 0.960 | 0.180 | 0.870 | -0.018 | 56% | 53% | |
| Laplace | 1.213 | 1.052 | 0.260 | 1.222 | -0.238 | – | – | |
| Fresh, full history ( ) | CEP | 1.194 | 0.990 | 0.167 | 0.871 | -0.071 | 75% | 69% |
| Horizon | CSP (ms) | CEP (ms) | Laplace (ms) | Laplace / CEP | Laplace / CSP |
|---|---|---|---|---|---|
| 1–12 | 0.3 | 9.8 | 750 | 77 | 2,500 |
| 13–24 | 0.9 | 21.4 | 5,030 | 235 | 5,600 |
| 25–48 | 1.2 | 38.8 | 11,755 | 303 | 9,800 |
| 49–96 | 2.4 | 59.0 | 47,304 | 802 | 19,700 |
| 96 | 4.5 | 82.4 | 107,024 | 1299 | 23,800 |
| Total, 2,534 windows (s) | 1.9 | 54.1 | 14,664 | 271 | 7,700 |
Time Series Forecasting
Time series forecasting predicts future values from ordered observations using statistical, neural, and probabilistic models to support decisions under uncertainty.
See all 941 papers