Time Series Forecasting Benchmarks Need Scenario-Grounded Stress Testing

Yuyang Zhao; Lian Xu; Hao Xue

cs.AIOct 2, 2026

Time Series Forecasting Benchmarks Need Scenario-Grounded Stress Testing

Authors: Yuyang Zhao, Lian Xu, Hao Xue

Organizations: The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · The University of Western Australia, Perth, Australia

Abstract

Time series forecasting (TSF) increasingly drives decisions in transportation, energy, finance, healthcare, and infrastructure, yet current evaluation remains overly narrow: standard benchmarks reward low held-out error, while robustness studies typically reduce failure to Gaussian noise, random masking, or bounded adversarial perturbations. This obscures the real failure modes of deployed forecasting systems. Input-side anomalies are not merely noisier inputs: they often reflect structured events that alter temporal dynamics, break cross-variable dependencies, induce regime shifts, or propagate from faulty sensors to downstream decisions. These semantic, causal, and system-level failures cannot be faithfully captured by i.i.d. perturbations alone. The rise of TSF foundation models makes this evaluation gap more urgent, as unauditable pretraining corpora make held-out generalization increasingly unreliable. We therefore advocate scenario-grounded stress testing. Each test instance should include historical inputs and future targets, together with a semantic scenario, an explicit failure operator, and a measurable difficulty level. This shift makes evaluation interpretable, attributable, and deployment-relevant and friendly, enabling the community to ask not only which model is accurate, but under what conditions it fails and why.

Figures & tables

Figure 1: The misalignment between current TSF benchmarking and real deployment failure. The left side evaluates models on clean held-out data with i.i.d. noise, masking, or $\epsilon$ -bounded perturbations and reports aggregate error. The right side fails through structured events (e.g., local shocks, dependency fractures, regime transitions, cascading failures)demanding scenario-aware reporting.

Generation	Representative Benchmarks	Treatment of Distribution Shift
G1 (2010s)	M3, M4	Implicit, via domain diversity
G2 (2018–2022)	Monash, LTSF	Explicit, as a domain-shift experiment
G3 (2023–2024)	GIFT-Eval, TFB, BasicTS+	Controlled by held-out discipline
G4 (2024–2025)	fev-bench, TSFM-Bench, ProbTS	Acknowledged but unmitigated
G5 (this work)	Scenario-grounded	Treated as the primary design variable

Table 1: Generational evolution of TSF benchmarks. G1–G4 share a common logic of optimizing average performance on a chosen distribution; G5 (this work) reorients evaluation around parameterized failure scenarios.

	Sampled	Constructed
Unstructured	Noise & Missingness (Gaussian, uniform, random masking)	Adversarial Attacks (FGSM, PGD, C&W within $\epsilon$ -balls)
Structured	Distribution Shift (Wild-Time, temporal drift)	Scenario-Grounded (this work)

Table 2: Four quadrants of TSF robustness research, organized by how deviations are generated (sampled vs. constructed) and whether they carry temporal, cross-variable, or causal structure.

ID	Ontological Property	Blind Spot in Current Evaluation
B1	Temporal shape	No semantically shaped local events
B2	Cross-variable structure	No cross-variable dependency disturbances
B3	Mechanism coupling	No coupling of regime change and missingness
B4	Causal propagation	No cascading failure propagation
B5	Attributability	No exposure of scenario parameters

Table 3: Five ontological properties of real TSF failures and the corresponding structural blind spots in the current evaluation paradigm.

Figure 2: Workflow of scenario-grounded stress testing. This workflow (i) defines the semantic content of a target scenario, (ii) specifies parameter vector $\Theta$ , (iii) designs failure operator $\phi$ , (iv) generates evaluation instances $(\tilde{X},\tilde{Y},\Theta)$ by applying $\phi$ to clean base pairs $(X,Y)$ , and (v) evaluates the model with $\Theta$ exposed as conditioning information rather than discarded as latent metadata.

Principles P1 – P3

Principles P4 – P6

P1. Expose scenario parameters Every test instance must be released with its scenario parameters

\Theta

and difficulty level

\delta

, so that performance can be analyzed against specific failure conditions rather than only as an aggregate. P2. Cover structural failure families A benchmark must include instances from each of F1–F4, avoiding the implicit omission of failure classes such as dependency fracture or cascading propagation. P3. Vary difficulty systematically Each family must be instantiated across an ordered range of difficulty levels, supporting robustness curves and threshold analyses rather than single-point comparisons.

P4. Separate mechanism from observation Mechanism change and observation degradation must be parameterized independently, so their individual and joint contributions to forecast error are decomposable. P5. Preserve base-sequence fidelity Failure operators must be applied to real or simulated base sequences

X

, ensuring that the difficulty introduced remains anchored to plausible deployment conditions. P6. Report results attributably Results must be stratified by failure family, parameter, and difficulty level rather than reported as a single aggregate, so that a benchmark indicates not only how a model performs but where and why it fails.

Table 4: Six design principles for next-generation TSF benchmarks. The principles operationalize the formal framework into a contract that any fifth-generation benchmark should satisfy.

Robust Time Series Forecasting

See all 17 papers

TS-Fault: Benchmarking Time Series Forecasters Against Structural Faults
Jun 16, 2026Yuyang Zhao, Lian Xu, Hao Miao +2Time Series Forecasting Forecasting Benchmarks
Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · The University of Western Australia, Perth, Australia · Hong Kong Polytechnic University, Hong Kong, China +1
Do Time Series Foundation Model Benchmarks Hide Regime-Dependent Failures? Evidence from Traffic Speed Forecasting
Jun 16, 2026Yingshuo Wang, Xian Sun, Lingdong Kong +4Time Series Forecasting Forecasting Benchmarks
University of California, Berkeley, CA, USA · Duke University, Durham, NC, USA · National University of Singapore, Singapore +3
TempusBench: An Evaluation Framework for Time-Series Forecasting
Apr 13, 2026Denizalp Goktas, Gerardo Riaño-Briceño, Omkar Tekawade +14Time Series Forecasting Forecasting Benchmarks
Simulacrum New York City, NY, USA

Jun 16, 2026J/K move · Enter open · S save

TS-Fault: Benchmarking Time Series Forecasters Against Structural Faults

Yuyang Zhao, Lian Xu, Hao Miao, Chenxi Liu+1

Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · The University of Western Australia, Perth, Australia · Hong Kong Polytechnic University, Hong Kong, China+1

Time Series Forecasting Forecasting Benchmarks Robust Time Series Forecasting

Open paper AskPDF arXiv

Time series forecasting (TSF) underpins consequential decisions in energy, transportation, finance, and healthcare, yet TSF models are almost universally ranked by a single number (e.g., average error) on clean held-out data, under the implicit assumption that it predicts deployed reliability. However, real faults are not i.i.d noise but structured events with temporal shape, broken cross-variable dependencies, regime change coupled with missingness, and causal propagation across a sensing pipeline. Treating TSF robustness as a data-quality problem, we present TS-Fault, a benchmark that evaluates forecasting models under explicit, parameterized fault scenarios with controllable semantic difficulty. TS-Fault organizes recurring failures into four modes along two orthogonal axes (observation- vs mechanism-level; univariate vs multivariate) and injects each fault into the most prediction-critical window via a unified importance score. This design enables robustness to be tested against the structures models actually rely on, rather than reduced to generic noise sensitivity. We evaluate 21 models across 6 datasets, 4 modes, and 5 difficulty levels under a paired clean/corrupt protocol. The results reveal three findings that contradict common leaderboard intuition: (i) clean-data accuracy anti-correlates with robustness; (ii) clean rankings are preserved under observation-level faults but reshuffled under mechanism-level faults; and (iii) all catastrophic failures occur under mechanism-level faults, with foundation models achieving the highest clean-data accuracy yet exhibiting the greatest fragility. The code is publicly available at https://github.com/Ray-zyy/TS-Fault.

Abstract

Robust Time Series Forecasting

Explore similar work

TS-Fault: Benchmarking Time Series Forecasters Against Structural Faults

Do Time Series Foundation Model Benchmarks Hide Regime-Dependent Failures? Evidence from Traffic Speed Forecasting

TempusBench: An Evaluation Framework for Time-Series Forecasting