Model-agnostic predictability quantification of multivariable time series

Yuxuan Shu; Vasileios Lampos

cs.LGOct 23, 2026

Model-agnostic predictability quantification of multivariable time series

Authors: Yuxuan Shu, Vasileios Lampos

Organizations: Centre for Artificial Intelligence, Department of Computer Science, University College London

Abstract

Multivariable time series forecasting plays an important role in various domains, where predictions depend on complex interactions among covariates. Forecasting models are commonly evaluated across numerous benchmark datasets. However, forecasting accuracy reflects both model capability and task predictability, making it difficult to characterise the difficulty of a task independently of the model used. Existing predictability measures often fail to capture nonlinear dependencies or are limited to univariate series. In this work, we propose Multivariable Predictability Score (MAPS), a model-agnostic metric for measuring the intrinsic predictability of multivariable time series forecasting tasks. MAPS uses covariate information to assess predictability across different lookback windows and forecasting horizons, and provides a quantitative assessment of the predictive information available for forecasting. We evaluate MAPS across synthetic and real-world forecasting datasets against 6 baseline predictability measures. The former confirms that our method closely aligns with the expected reference predictability (r=0.963). Empirical results further corroborate that MAPS yields more reliable projections in the majority of experimental configurations.

Figures & tables

Figure 1: Overview of the proposed Multivariable Predictability Score ( MAPS ).

Figure 2: Basic synthetic time series used to validate MAPS . The exogenous variables are shown in the top row, the corresponding target series in the middle row, and the predictability w.r.t. $H$ in the bottom row, each is a combination of the exogenous inputs. Predictability scores are computed using a rolling window where the input includes all covariates, i.e. both exogenous variables and the target.

Figure 3: Reference predictability compared to MAPS , ForeCA, Spectral, Persistence, and the empirical performance (using linear correlation) of a ridge regression model for complex synthetic time series tasks across increasing forecasting horizons (

H

) for a fixed lookback window (

L=16

Figure 4: Predictability scores of MAPS and $5$ baselines for the WEA task compared against the forecasting performance of Sonnet across $20$ locations (lookback $L{=}56$ , horizon $H{=}12$ ) for two configurations ( $5$ or $45$ covariates). Location order reflects Sonnet’s correlation performance.

	Method	WEA , $L=28$						WEA , $L=56$						WEA , $L=84$						Across $(L,H)$ pairs
		$H\!=\!4$		$H\!=\!12$		$H\!=\!28$		$H\!=\!4$		$H\!=\!12$		$H\!=\!28$		$H\!=\!4$		$H\!=\!12$		$H\!=\!28$		WEA	ILI
		$C\!=\!45$	$C\!=\!5$	$C\!=\!45$	$C\!=\!5$	$C\!=\!45$	$C\!=\!5$	$C\!=\!45$	$C\!=\!5$	$C\!=\!45$	$C\!=\!5$	$C\!=\!45$	$C\!=\!5$	$C\!=\!45$	$C\!=\!5$	$C\!=\!45$	$C\!=\!5$	$C\!=\!45$	$C\!=\!5$	WEA	ILI
$\rho$	MAPS	.843	.896	.975	.971	.967	.972	.812	.872	.977	.975	.972	.968	.797	.856	.975	.973	.973	.976	.961	.827
	Epiplexity	.991	.955	.951	.951	.968	.976	.987	.956	.922	.953	.940	.967	.961	.959	.853	.949	.859	.974	.833	.455
	ForeCA	.801	.836	.972	.974	.961	.972	.804	.845	.976	.974	.970	.963	.805	.850	.974	.972	.973	.973	–	–
	Spectral	.459	.514	.844	.768	.851	.779	.471	.525	.836	.763	.840	.780	.467	.526	.836	.779	.848	.788	–	–

Table 1: Average linear correlation (across all test periods) between predictability scores and Sonnet’s forecasting performance, which is measured by correlation (

\rho

) or scaled MAE (

s

). Correlation breakdowns for lookback windows (

L

), forecasting horizons (

H

), and covariate number (

C

) are provided for the WEA task, over the

20

considered locations. The right-most two columns enumerate the average correlation among all combinations of

(L,H)

across regions for WEA and ILI .

Table 7

Figure 6: Effect of window stride on MAPS across nine

(L,H)

settings across

20

locations ( WEA ). (A) Averaged predictability score; (B) Correlation across

20

locations. The axis breaks at stride 100, which separates the equally spaced stride values from a linear scale over 100–500.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: Relationship between SMAPE and linear correlation across weather forecasting tasks for Sonnet against target correlation

\rho

(left) and scaled MAE score

s

(right). Each point represents one task for one of 20 locations, with colours indicating different locations.

Figure S2: Generated time series for the six synthetic processes with known predictability. Each series contains

T=4{,}000

time steps, of which the first

600

are shown.

Figure S3: Distribution of the 20 locations used in the weather forecasting tasks.

Method	Exponential	Exponential + Noise	Damped Sine	Logarithm	Log + Seasonal	Log + Noise	Power Law	Logistic	Chirp	Sawtooth	Constant + Outliers	Gaussian Function
MAPS	0.998	0.888	0.965	0.998	0.990	0.864	0.998	0.999	0.975	0.996	0.965	0.997
ForeCA	0.453	0.100	0.463	0.453	0.455	0.101	0.453	0.453	0.461	0.453	0.789	0.453
Spectral	0.919	0.520	0.918	0.919	0.919	0.521	0.919	0.919	0.918	0.916	0.898	0.919
MI	1.000	0.580	0.268	1.000	0.756	0.562	1.000	1.000	0.556	1.000	0.003	1.000

Appendix

Table S1: Predictability scores of bounded baseline methods on additional synthetic time series. All scores lie in

[0,1]

, with higher values indicating greater estimated predictability.

Table 14

Figure S5: Relationship between different predictability measures and the target SMAPE across all evaluated locations under the Sonnet forecasting model.

	Method	$L=28$						$L=56$						$L=84$
		$H=4$		$H=12$		$H=28$		$H=4$		$H=12$		$H=28$		$H=4$		$H=12$		$H=28$
		$C=45$	$C=5$	$C=45$	$C=5$	$C=45$	$C=5$	$C=45$	$C=5$	$C=45$	$C=5$	$C=45$	$C=5$	$C=45$	$C=5$	$C=45$	$C=5$	$C=45$	$C=5$
$\rho$	MAPS	.847	.898	.973	.966	.970	.965	.811	.858	.977	.972	.971	.970	.800	.847	.974	.975	.970	.969
	Epiplexity	.989	.959	.957	.947	.968	.970	.988	.953	.926	.945	.943	.971	.958	.956	.860	.951	.858	.971
	ForeCA	.805	.838	.972	.969	.967	.968	.803	.829	.975	.970	.969	.967	.810	.841	.972	.973	.964	.966
	Spectral	.467	.513	.831	.764	.850	.774	.463	.506	.837	.775	.853	.785	.477	.516	.843	.774	.869	.786

Appendix

Table S4: Linear correlation

r

between predictability scores and Crossformer’s forecasting performance, measured by target correlation

\rho

or scaled MAE

s

, across the

20

WEA locations. Predictability scores and model performance are averaged over the 2016–2018 test periods before computing

r

. Results are reported across lookback windows (

L

), forecasting horizons (

H

), and covariate numbers (

C

). The best results are bolded.

	Method	$L=28$						$L=56$						$L=84$
		$H\!=\!4$		$H\!=\!12$		$H\!=\!28$		$H\!=\!4$		$H\!=\!12$		$H\!=\!28$		$H\!=\!4$		$H\!=\!12$		$H\!=\!28$
		$C\!=\!45$	$C\!=\!5$	$C\!=\!45$	$C\!=\!5$	$C\!=\!45$	$C\!=\!5$	$C\!=\!45$	$C\!=\!5$	$C\!=\!45$	$C\!=\!5$	$C\!=\!45$	$C\!=\!5$	$C\!=\!45$	$C\!=\!5$	$C\!=\!45$	$C\!=\!5$	$C\!=\!45$	$C\!=\!5$
$\rho$	MAPS	.822 $\downarrow$	.854 $\downarrow$	.974 $\downarrow$	.964 $\downarrow$	.960 $\downarrow$	.954 $\downarrow$	.787 $\downarrow$	.827 $\downarrow$	.974 $\downarrow$	.963 $\downarrow$	.965 $\downarrow$	.952 $\downarrow$	.764 $\downarrow$	.812 $\downarrow$	.974 $\downarrow$	.965 $\downarrow$	.967 $\downarrow$	.962 $\downarrow$
	ForeCA	.801 $\uparrow$	.836 $\uparrow$	.972 $\uparrow$	.974 $\uparrow$	.961 $\downarrow$	.972 $\uparrow$	.804 $\uparrow$	.845 $\uparrow$	.976 $\uparrow$	.974 $\uparrow$	.970 $\uparrow$	.963 $\uparrow$	.805 $\uparrow$	.850 $\uparrow$	.974 $\uparrow$	.972 $\uparrow$	.973 $\downarrow$	.973 $\uparrow$
	Spectral	.806 $\uparrow$	.840 $\uparrow$	.968 $\uparrow$	.970 $\uparrow$	.954 $\uparrow$	.967 $\uparrow$	.808 $\uparrow$	.850 $\uparrow$	.973 $\uparrow$	.970 $\uparrow$	.963 $\uparrow$	.958 $\uparrow$	.810 $\uparrow$	.854 $\uparrow$	.970 $\uparrow$	.969 $\uparrow$	.968 $\uparrow$	.969 $\uparrow$
	MI	.713 $\downarrow$	.759 $\downarrow$	.940 $\uparrow$	.940 $\uparrow$	.944 $\uparrow$	.953 $\uparrow$	.703 $\downarrow$	.757 $\downarrow$	.934 $\uparrow$	.936 $\uparrow$	.948 $\uparrow$	.948 $\uparrow$	.701 $\downarrow$	.755 $\downarrow$	.931 $\uparrow$	.935 $\uparrow$	.951 $\uparrow$	.956 $\uparrow$

Appendix

Table S5: Linear correlation

r

between target-only predictability scores and Sonnet’s forecasting performance. Arrows indicate whether

r

increases or decreases relative to the corresponding result in Table 1 and

=

indicates no change.

Figure S6: Runtime analysis of predictability estimators. We measure the median per-window runtime (ms) versus the number of variables, lookback window length, and forecast horizon.

Multivariate Time Series Forecasting

Multivariate time series forecasting predicts future values across related variables by learning their temporal and cross-variable dependencies.

See all 307 papers

Temporal Predictive Multiplicity: Equally Accurate Time Series Models Yield Different Forecast Trajectories
Oct 7, 2026Emanuele Albini, Francesca Toni, Saumitra Mishra +1Time Series Forecasting Time Series Modeling
J.P. Morgan AI Research Imperial College London London, UK · Imperial College London London, UK · J.P. Morgan AI Research London, UK
Multivariate Time Series Forecasting needs Cross Variable Loss
Aug 6, 2026Kuiye Ding, Yifan Hu, Hao Xue +1Multivariate Time Series Forecasting Time Series Forecasting
University of Technology Sydney · Tsinghua University · The Hong Kong University of Science and Technology (Guangzhou)
SDMixer: Sparse Dual-Mixer for Time Series Forecasting
Feb 27, 2026Xiang AoMultivariate Time Series Forecasting Time Series Forecasting
School of Software Engineering, Beijing Jiaotong University, Beijing, China

Oct 7, 2026J/K move · Enter open · S save

Temporal Predictive Multiplicity: Equally Accurate Time Series Models Yield Different Forecast Trajectories

Emanuele Albini, Francesca Toni, Saumitra Mishra, Francesco Leofante

J.P. Morgan AI Research Imperial College London London, UK · Imperial College London London, UK · J.P. Morgan AI Research London, UK

Time Series Forecasting Time Series Modeling

Open paper AskPDF arXiv

Models with near-identical predictive performance can yield substantially different predictions, a phenomenon known as predictive multiplicity. Prior work has mostly studied this at the level of individual scalar outputs. In time-series forecasting, however, predictions across horizons jointly define a trajectory, and horizon-wise comparisons can hide important differences in predictive behavior. To address this problem, we introduce temporal predictive multiplicity, a framework that characterizes disagreement over complete forecast trajectories among models with near-identical predictive performance. We show that constraining predictive performance alone can still admit a broad range of different trajectories. We further show that constraining multiplicity at individual horizons partially reduces, but does not eliminate, trajectory-level multiplicity. Experiments with 19 neural forecasting architectures on 11 datasets confirm that near-optimal models can exhibit substantial variability in the forecast trajectories they produce, and trajectory-level disagreement is largely unrelated to horizon-wise disagreement. Our framework, therefore, exposes a gap in existing multiplicity studies: models with indistinguishable predictive performance imply fundamentally different temporal trajectories, with consequential downstream effects.

Abstract

Multivariate Time Series Forecasting

Explore similar work

Temporal Predictive Multiplicity: Equally Accurate Time Series Models Yield Different Forecast Trajectories

Multivariate Time Series Forecasting needs Cross Variable Loss

SDMixer: Sparse Dual-Mixer for Time Series Forecasting