Ensemble Modeling for Time Series Forecasting: an Adaptive Robust Optimization Approach
Organizations: Operations Research Center, Massachusetts Institute of Technology, Cambridge, MA 02139, USA · Sloan School of Management and Operations Research Center, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
Abstract
Accurate time-series forecasting is critical for many problems involving temporal data. Ensemble modeling is a well-established technique that leverages multiple predictive models to improve accuracy and robustness, since a single predictor's performance can vary widely due to shifts in the underlying data distribution. This paper proposes a new methodology for building robust ensembles of time series forecasting models. Our approach uses Adaptive Robust Optimization (ARO) to construct a linear regression ensemble whose model weights adapt over time. We demonstrate the effectiveness of our method through synthetic experiments covering eight disturbance scenarios, from no drift to abrupt regime shifts, and real-world applications, including air pollution management, energy consumption forecasting, and tropical cyclone intensity forecasting. In the real-world applications, our adaptive ensembles outperform the best ensemble member in hindsight by 16-26% in root mean square error (RMSE) and 14-28% in conditional value at risk at the 15% level and improve over the other ensemble methods we compare on most metrics. In the synthetic experiments, they have lower mean RMSE than a rolling-ridge comparator in seven of the eight scenarios, significantly in four, while an abrupt regime shift that first appears after model selection remains a limitation.
Figures & tables
| Ensemble Method | MAE | RMSE | MAPE (%) | CVaR 5% | CVaR 15% |
|---|---|---|---|---|---|
| Best Model in Hindsight (oracle) | |||||
| Ensemble Mean | |||||
| Windowed EWA | |||||
| Passive-Aggressive | |||||
| Ridge | |||||
| Adaptive Ridge |
| Ensemble Method | MAE | RMSE | MAPE (%) | CVaR 5% | CVaR 15% |
|---|---|---|---|---|---|
| Best Model in Hindsight (oracle) | |||||
| Ensemble Mean | |||||
| Windowed EWA | |||||
| Passive-Aggressive | |||||
| Ridge | |||||
| Adaptive Ridge |
| North Atlantic Basin | Eastern Pacific Basin | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Ensemble Method | MAE | RMSE | MAPE (%) | CVaR 5% | CVaR 15% | MAE | RMSE | MAPE (%) | CVaR 5% | CVaR 15% |
| Best Model in Hindsight (oracle) | 9.2 | 12.2 | 14.2 | 31.4 | 24.6 | 11.3 | 15.3 | 14.5 | 41.1 | 30.8 |
| Ensemble Mean | 9.4 | 12.8 | 14.0 | 35.2 | 25.6 | 10.9 | 14.6 | 14.2 | 38.2 | 29.2 |
| Windowed EWA | 8.8 | 11.5 | 13.3 | 30.0 | 22.6 | 10.3 | 14.9 | 15.5 | 43.3 | 31.1 |
| Passive-Aggressive | 9.1 | 12.4 | 13.9 | 34.6 | 25.3 | 17.9 | 22.9 | 27.1 | 57.2 | 44.4 |
| FSSE | 8.8 | 11.6 | 13.8 | 31.0 | 23.2 | 10.0 | 14.4 | 15.5 | 41.3 | 28.9 |
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter | Notation | Fixed value | Range when varied |
| Total number of time steps | 4,000 | fixed | |
| Observations before testing | 3,000 | ||
| Test samples | 1,000 | fixed | |
| Seeds per configuration | 30 | fixed | |
| Period of the target signal | 500 | fixed | |
| Target noise mean and standard deviation | 0 and 0.1 | fixed |
| Setting | Adaptive Ridge | Rolling ridge | RLS | PA | Ridge | EWA | Mean | Oracle |
|---|---|---|---|---|---|---|---|---|
| Number of members | ||||||||
| 2 | ||||||||
| 5 | ||||||||
| 10 | ||||||||
| 15 | ||||||||
| 20 | ||||||||
| Method | Time-varying | Hyperparameters to tune |
|---|---|---|
| Best Model in Hindsight (oracle) | No | N/A |
| Ensemble Mean | No | N/A |
| Windowed EWA | Yes | Window of past data to use to compute regrets |
| Passive-Aggressive | Yes | Margin parameter |
| Ridge | No | Regularization factor |
| Adaptive Ridge | Yes | Regularization factor , window of past data in |
| Method | Validation candidates | Size |
|---|---|---|
| Adaptive Ridge | ; regularization | 20 |
| Validation-selected member | one candidate per member | 10 |
| Windowed EWA | 6 | |
| Hedge | learning rate | 5 |
| Fixed Share | learning rate ; share rate | 25 |
| PA-I | ; | 20 |
| Method | None | Grad. | Sudden | Per. | Bias | Rand. | Best | Corr. |
|---|---|---|---|---|---|---|---|---|
| Adaptive Ridge | ||||||||
| Ensemble mean | ||||||||
| Validation-selected member | ||||||||
| Windowed EWA | ||||||||
| Hedge | ||||||||
| Fixed Share |
| Method | None | Grad. | Sudden | Per. | Bias | Rand. | Best | Corr. |
|---|---|---|---|---|---|---|---|---|
| Adaptive Ridge | ||||||||
| Ensemble mean | ||||||||
| Validation-selected member | ||||||||
| Windowed EWA | ||||||||
| Hedge | ||||||||
| Fixed Share |
| Selected | ||||||
| 1 | 2/30 | |||||
| 2 | 7/30 | |||||
| 3 | 13/30 | |||||
| 4 | 5/30 | |||||
| 5 | 3/30 | |||||
| 10 | 0/30 |
| Pre-test history | Adaptive Ridge | Rolling ridge | RLS | PA-I | Ensemble mean |
|---|---|---|---|---|---|
| 50 | |||||
| 100 | |||||
| 200 | |||||
| 500 | |||||
| 1,000 | |||||
| 2,000 |
| Adaptive Ridge | Rolling ridge | RLS | PA-I | Ensemble mean | |
|---|---|---|---|---|---|
| 0 | |||||
| 0.25 | |||||
| 0.5 | |||||
| 0.75 | |||||
| 0.9 | |||||
| 0.99 |
| Method (grid size) | Validation search | Final refit or replay | 1,000-step test | Total |
|---|---|---|---|---|
| Adaptive Ridge (20) | 0.1046 | 0.0081 | 0.0003 | 0.1135 |
| PA-I (20) | 0.0047 | 0.0002 | 0.0001 | 0.0050 |
| RLS (5) | 0.0101 | 0.0016 | 0.0006 | 0.0124 |
| Rolling ridge (12) | 0.0483 | n/a | 0.0077 | 0.0560 |
| Windowed EWA (6) | 0.0007 | 0.0001 | 0.0001 | 0.0008 |
| Hedge (5) | 0.0009 | 0.0002 | 0.0001 | 0.0011 |
| Number of samples in training set | Average time (s) |
| 100 | |
| 200 | |
| 400 | |
| 750 | 64 |
| 1500 | 92 |
| 3000 | 239 |
| Number of time steps | Average time (s) |
| 2 | 54 |
| 3 | 100 |
| 5 | 239 |
| 10 | 603 |
| 15 | 1079 |
| 25 | 1361 |
| Member ID | Forecast identity | Used as |
|---|---|---|
| Safi-1 | Moroccan weather agency operational forecast | Ensemble member |
| Safi-2 | XGBoost | Ensemble member |
| Safi-3 | Decision Trees | Ensemble member |
| Safi-4 | Optimal Regression Trees | Ensemble member |
| Safi-5 | Lasso Regression | Ensemble member |
| Safi-6 | Ridge Regression | Ensemble member |
| Member ID | Forecast identity | Used as |
|---|---|---|
| Energy-1 | Ordinary Least Squares (OLS) | Ensemble member |
| Energy-2 | Ridge | Ensemble member |
| Energy-3 | Bayesian Ridge | Ensemble member |
| Energy-4 | ElasticNet | Ensemble member |
| Energy-5 | Huber Regressor | Ensemble member |
| Energy-6 | Least Angle Regression (LARS) | Ensemble member |
| Model ID | Model name | Model type | Used as |
|---|---|---|---|
| HUML-(stat, xgb) | Hurricast (tabular data) | Gradient-boosted trees | Ensemble member |
| HUML-(stat/viz, xgb/td) | Hurricast (tabular and vision data) | Feature extraction with tensor decomposition + Gradient-boosted trees | Ensemble member |
| HUML-(stat/viz, xgb/cnn/gru) | Hurricast (tabular and vision data) | Feature extraction with deep learning (CNN, GRU) + Gradient-boosted trees | Ensemble member |
| HUML-(stat/viz, xgb/cnn/transfo) | Hurricast (tabular and vision data) | Feature extraction with deep learning (CNN, Transformers) + Gradient-boosted trees | Ensemble member |
| Decay-SHIPS | Decay Statistical Hurricane Intensity Prediction Scheme | Statistical-dynamical | Ensemble member |
| GFSO | Global Forecast System model | Multi-layer global dynamical | Ensemble member |
Time Series Forecasting
Time series forecasting predicts future values from ordered observations using statistical, neural, and probabilistic models to support decisions under uncertainty.
See all 939 papers