Series-wise Cross-validation for Global Time Series Forecasting

Vitor Cerqueira; Mariana Oliveira; Luis Roque

stat.MLNov 5, 2026

Series-wise Cross-validation for Global Time Series Forecasting

Authors: Vitor Cerqueira, Mariana Oliveira, Luis Roque

Organizations: University of Coimbra, CISUC/LASI, DEI Coimbra, Portugal · Faculty of Engineering, University of Porto Porto, Portugal

Abstract

We consider the problem of cross-validation (CV) in time series forecasting, with a focus on global forecasting models based on deep learning. The prevailing CV strategy in forecasting is to partition data across time with the last part of each series used for testing. With global models trained on collections of time series, however, a fundamentally different approach becomes possible: partitioning across series and time. We formalize series-wise CV for global forecasting models and benchmark several instantiations, including K-fold CV, bootstrapping, and Monte Carlo CV, on 11 datasets with 10 neural network architectures. We evaluate CV procedures along five dimensions: performance estimation, overoptimism rate, model selection accuracy, regret, and downstream forecasting performance. We withhold a random fraction of series from training so that they appear only in the test set. This protocol simulates the emergence of new series not seen during training, a common scenario in global forecasting. Under this protocol, our results reveal a nuanced trade-off: series-wise CV yields better downstream forecasting accuracy by guiding model selection toward configurations that generalize across both time and series, while time-wise CV retains a slight advantage for estimating out-of-sample error.

Figures & tables

Panel of Figure 1 — Figure 1: Illustration of time-wise CV (a) and series-wise CV (b). In time-wise CV, all series contribute to both training and testing, with only the temporal dimension used for splitting. In series-wise CV, the split is across series and time: one subset is used for training, a disjoint subset for testing, and both are further split in time to preserve temporal order.

Panel of Figure 2 — Figure 2: Illustration of series-wise CV partitioning schemes. Each horizontal bar represents a time series; blue indicates training data and red indicates test data. (a) $K$ -fold CV with $K{=}5$ : the series are split into 5 disjoint folds, and each fold serves as the test set exactly once. (b) Holdout: a single random split assigns most series to training and the remainder to testing. (c) Monte Carlo CV: a random subset of series is used for training and a disjoint subset for testing; gray indicates unused series in that iteration. (d) Bootstrap: training series are sampled with replacement (darker shades indicate that a series may appear multiple times in the training set) and out-of-bag series form the test set.

Figure 3: Nested time-wise CV. The outer loop partitions data across time for performance estimation; the inner loop further splits the outer training portion across time to select hyperparameters. All series contribute to both training and testing at every level.

Figure 4: Nested series-wise CV. The outer loop partitions data across series and time for performance estimation: training series and test series are disjoint. The inner loop applies a time-wise holdout on the outer training series to select hyperparameters. This ensures that outer-test series are never seen during training or tuning.

Figure 5: Evaluation protocol and flow of information for scoring a CV procedure.

Dataset	Frequency	# time series	# observations	Horizon ( $h$ )
Tourism	Monthly	366	109,280	12
Tourism	Quarterly	427	42,544	4
M1	Monthly	617	55,998	12
M1	Quarterly	203	9,944	4
M3	Monthly	1,428	167,562	12
M3	Quarterly	756	37,004	4

Table 1: Summary of the datasets used in the experimental setup, including the number of time series, number of observations, forecast horizon, and frequency.

Series-wise							Time-wise
Method	Bootstrap	Holdout	Kf-CV	K-MC-CV	K-Bootstrap	K-Holdout	Time-CV
MAPEE	0.120	0.141	0.124	0.140	0.126	0.132	0.109
Overoptimism	0.315	0.373	0.258	0.267	0.255	0.261	0.324
Accuracy	0.242	0.364	0.242	0.303	0.152	0.212	0.273
Regret	0.023	0.019	0.044	0.016	0.019	0.020	0.034

Table 2: Results of performance estimation (MAPEE), overoptimism rate, selection accuracy, and regret for each CV approach. Numbers in bold (underlined) denote the best (second-best) approach.

Series-wise							Time-wise
Methods	Bootstrap	Holdout	Kf-CV	K-MC-CV	K-Bootstrap	K-Holdout	Time-CV
ECL	0.596	0.592	0.591	0.595	0.600	0.604	0.577
Exchange	0.151	0.153	0.147	0.145	0.145	0.147	0.155
Traffic	0.702	0.671	0.688	0.691	0.691	0.677	0.692
Weather	0.976	0.920	0.939	0.928	0.940	0.934	1.115
Hospital	0.811	0.814	0.797	0.796	0.804	0.804	0.804

Table 3: MASE score of each CV approach on each dataset, and respective average rank, average MASE, and top 2 counts (averaged across seeds).

Series-wise							Time-wise
Method	Bootstrap	Holdout	Kf-CV	K-MC-CV	K-Bootstrap	K-Holdout	Time-CV
MAPEE	0.119	0.109	0.108	0.127	14.321	0.105	0.092
Overoptimism	0.306	0.424	0.255	0.252	0.242	0.261	0.306
Accuracy	0.455	0.273	0.273	0.273	0.273	0.364	0.333
Regret	0.025	0.024	0.022	0.020	0.019	0.020	0.012

Table 4: Results of performance estimation (MAPEE), overoptimism rate, selection accuracy, and regret for each CV approach with a complete in-set.

Series-wise							Time-wise
Methods	Bootstrap	Holdout	Kf-CV	K-MC-CV	K-Bootstrap	K-Holdout	Time-CV
ECL	0.592	0.583	0.587	0.598	0.594	0.596	0.594
Exchange	0.147	0.142	0.148	0.148	0.143	0.144	0.156
Traffic	0.686	0.701	0.686	0.684	0.687	0.683	0.686
Weather	0.922	0.936	0.930	0.926	0.926	0.933	0.941
Hospital	0.813	0.797	0.801	0.797	0.808	0.797	0.800

Table 5: MASE score of each CV approach on each dataset, and respective average rank, average MASE, and top 2 counts with a complete in-set.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Series-wise							Time-wise
Methods	Bootstrap	Holdout	Kf-CV	K-MC-CV	K-Bootstrap	K-Holdout	Time-CV
ECL	0.586	0.595	0.585	0.586	0.586	0.606	0.574
Exchange	0.152	0.168	0.141	0.146	0.143	0.145	0.159
Traffic	0.700	0.675	0.690	0.698	0.690	0.690	0.695
Weather	1.063	0.921	0.939	0.939	0.939	0.939	1.148
Hospital	0.806	0.807	0.800	0.796	0.806	0.821	0.808

Appendix

Table 6: MASE score of each CV approach on each dataset, and respective average rank, average MASE, and top 2 counts (seed 123).

Series-wise							Time-wise
Methods	Bootstrap	Holdout	Kf-CV	K-MC-CV	K-Bootstrap	K-Holdout	Time-CV
ECL	0.598	0.595	0.602	0.583	0.598	0.598	0.583
Exchange	0.151	0.142	0.145	0.149	0.141	0.141	0.147
Traffic	0.686	0.684	0.684	0.684	0.684	0.686	0.687
Weather	0.945	0.898	0.940	0.920	0.942	0.920	1.050
Hospital	0.840	0.813	0.795	0.797	0.801	0.796	0.803

Appendix

Table 7: MASE score of each CV approach on each dataset, and respective average rank, average MASE, and top 2 counts (seed 456).

Series-wise							Time-wise
Methods	Bootstrap	Holdout	Kf-CV	K-MC-CV	K-Bootstrap	K-Holdout	Time-CV
ECL	0.604	0.586	0.585	0.615	0.615	0.606	0.574
Exchange	0.152	0.148	0.154	0.142	0.152	0.154	0.159
Traffic	0.721	0.654	0.690	0.690	0.700	0.654	0.695
Weather	0.920	0.939	0.939	0.926	0.939	0.943	1.148
Hospital	0.787	0.821	0.796	0.796	0.803	0.796	0.802

Appendix

Table 8: MASE score of each CV approach on each dataset, and respective average rank, average MASE, and top 2 counts (seed 789).

Series-wise							Time-wise
Method	Bootstrap	Holdout	Kf-CV	K-MC-CV	K-Bootstrap	K-Holdout	Time-CV
MAPEE	0.105	0.166	0.133	0.135	0.136	0.129	0.112
Overoptimism	0.282	0.336	0.245	0.336	0.227	0.255	0.327
Accuracy	0.091	0.455	0.364	0.091	0.182	0.273	0.182
Regret	0.031	0.008	0.073	0.021	0.022	0.018	0.043

Appendix

Table 9: Results of performance estimation (MAPEE), overoptimism rate, selection accuracy, and regret for each CV approach (seed 123).

Series-wise							Time-wise
Method	Bootstrap	Holdout	Kf-CV	K-MC-CV	K-Bootstrap	K-Holdout	Time-CV
MAPEE	0.134	0.118	0.120	0.157	0.121	0.133	0.103
Overoptimism	0.336	0.4	0.291	0.236	0.264	0.264	0.309
Accuracy	0.364	0.273	0.182	0.364	0.182	0.273	0.455
Regret	0.019	0.035	0.028	0.014	0.015	0.017	0.014

Appendix

Table 10: Results of performance estimation (MAPEE), overoptimism rate, selection accuracy, and regret for each CV approach (seed 456).

Series-wise							Time-wise
Method	Bootstrap	Holdout	Kf-CV	K-MC-CV	K-Bootstrap	K-Holdout	Time-CV
MAPEE	0.122	0.140	0.119	0.127	0.122	0.132	0.114
Overoptimism	0.327	0.382	0.236	0.227	0.273	0.264	0.336
Accuracy	0.273	0.364	0.182	0.455	0.091	0.091	0.182
Regret	0.019	0.016	0.032	0.012	0.020	0.026	0.043

Appendix

Table 11: Results of performance estimation (MAPEE), overoptimism rate, selection accuracy, and regret for each CV approach (seed 789).

Cross-Validation

See all 17 papers

Beyond Holistic Models: Systematic Component-level Benchmarking of Deep Multivariate Time-Series Forecasting
May 26, 2026Shuang Liang, Chaochuan Hou, Xu Yao +4Multivariate Time Series Forecasting Forecasting Benchmarks
Shanghai University of Finance and Economics Shanghai, China · Ant Group Shanghai, China · Key Laboratory of Interdisciplinary Research of Computation and Economics Shanghai University of Finance and Economics Shanghai, China
Seeking SOTA: Time-Series Forecasting Must Adopt Taxonomy-Specific Evaluation to Dispel Illusory Gains
Mar 16, 2026Raeid Saqur, Christoph Bergmeir, Blanka Horvath +3Time Series Forecasting Time Series Modeling
Dept. of Computer Science, University of Toronto · Dept. of Mathematics, University of Oxford · Vector Institute +5
Central limit theorems for the outputs of fully convolutional neural networks with time series input
Mar 31, 2026Annika Betken, Giorgio Micali, Johannes Schmidt-HieberConvolutional Neural Networks
University of Twente, Faculty of Electrical Engineering, Mathematics, and Computer Science (EEMCS), Drienerlolaan 5, 7522 NB Enschede, Netherlands

May 26, 2026J/K move · Enter open · S save

Beyond Holistic Models: Systematic Component-level Benchmarking of Deep Multivariate Time-Series Forecasting

Shuang Liang, Chaochuan Hou, Xu Yao, Shiping Wang+3

Shanghai University of Finance and Economics Shanghai, China · Ant Group Shanghai, China · Key Laboratory of Interdisciplinary Research of Computation and Economics Shanghai University of Finance and Economics Shanghai, China

Multivariate Time Series Forecasting Forecasting Benchmarks Automated Machine Learning

Open paper AskPDF arXiv

While previous research in multivariate time series forecasting has focused on developing complex holistic models, this work advocates for a shift toward a granular, component-level understanding of their impacts. We propose TSCOMP, the first large-scale benchmark that systematically deconstructs deep forecasting methods into their core, fine-grained components--spanning series preprocessing, encoding strategies, network architectures including specific and large time-series models, and optimization methods. Using constrained orthogonal experimental design and extensive evaluations, we conduct multi-view analyses that reveal component effectiveness across different backbones, data characteristics, and their interactions. Beyond providing insights, this benchmark establishes a fine-grained performance corpus comprising over 20,000 model-dataset evaluations, which supports the learning of automated component selection, enabling zero-shot model construction on new datasets. Our experiments demonstrate that the corpus-driven approach, despite its simplicity, consistently outperforms state-of-the-art methods, validating the soundness of our evaluation design and confirming that systematic component selection surpasses manually designed complex architectures. All code and the performance corpus are publicly available at https://github.com/SUFE-AILAB/TSCOMP.

Abstract

Cross-Validation

Explore similar work

Beyond Holistic Models: Systematic Component-level Benchmarking of Deep Multivariate Time-Series Forecasting

Seeking SOTA: Time-Series Forecasting Must Adopt Taxonomy-Specific Evaluation to Dispel Illusory Gains

Central limit theorems for the outputs of fully convolutional neural networks with time series input