Beyond Average Error through Oracle-Informed Stress Tests for Time-Series Forecasting

Xu Lin; Runheng Zuo; Shengxuan Xu; Qitai Tan; Hongyu Lin; Xiao-Ping Zhang

cs.LGSep 19, 2026

Beyond Average Error through Oracle-Informed Stress Tests for Time-Series Forecasting

Authors: Xu Lin, Runheng Zuo, Shengxuan Xu, Qitai Tan, Hongyu Lin, Xiao-Ping Zhang

Organizations: Tsinghua University

Abstract

Average squared error cannot reveal whether forecasting performance degrades because the future becomes less predictable or because forecasts move farther from the conditional mean. We introduce paired, mechanism-controlled stress tests that decompose changes in expected squared error at each lead time into environmental risk and forecast-oracle distance, using an origin-conditioned predictive oracle unavailable to the evaluated models. Three end-to-end controls have known attribution. Specifically, the null, environmental-only, and information-gap controls verify that the pipeline assigns changes to the correct component. We then apply the benchmark to 24 deployable forecasters. Under frequent switching, 14 methods have higher realized MSE but lower oracle distance; under outlier-variance feedback, 19 have higher MSE but lower scale-standardized MSE. Short- and long-lead stress-response rankings have Spearman correlation 0.624, revealing substantial horizon-dependent reordering. We then study multivariate relation shifts. Across six models and three coupling severities, oracle distance accounts for only 0.7-3.9% of the decomposed expected-risk increase, and environmental-risk majority persists in an eight-channel system and a matched-difficulty audit of Ring, Block, and Hub relations. Finally, prespecified contrasts on independent data-generating process (DGP) realizations show that several visually compelling discovery profiles, including trend accumulation and the hypothesized switching reversal, do not replicate. The benchmark thus combines component-wise diagnosis with a held-out stability audit. It complements real-data out-of-distribution evaluation, which measures performance under realistic shifts when exact oracle attribution is unavailable.

Figures & tables

Figure 1: Overview of our oracle-informed, mechanism-controlled stress-test framework, which uses matched pairs of Base and Shift environments across six mechanism families to separate environmental-risk changes from changes in oracle distance and report aggregate and lead-resolved diagnostics.

Figure 2: The complete discovery atlas retains model, mechanism, and individual lead. Rows are the 24 deployable forecasters, grouped by protocol; columns within each panel are the 192 individual leads at $H=192$ . Brown indicates increased oracle distance under Shift , green indicates a reduction. Each mechanism uses an explicitly labelled robust symmetric range, so color intensity is compared within rather than across panels. These ten-seed profiles support descriptive comparison and hypothesis generation; the stability of selected patterns is evaluated on independent batches below.

Figure 3: Aggregate error conceals diagnostic disagreement and lead-dependent ordering. (A) compares the change in realized raw MSE with the change in oracle distance under frequent switching. (B) compares the change in realized raw MSE with the change in SMSE under outlier-variance feedback. Shaded lower-right quadrants mark increased raw loss accompanied by a lower diagnostic quantity; black rings mark methods whose two unadjusted seed-bootstrap intervals exclude zero. (C) compares descriptive stress-response ranks for short leads 1 through 24 and long leads 97 through 192; rank 1 denotes the lower weighted response score and the dashed diagonal denotes no reordering. Colors identify forecasting protocol.

Figure 4: Coupling degradation is dominated by environmental risk across severity and structure. (A) Environmental

\Delta B

(black squares) and model-specific

\Delta D

(colored circles) on a log scale as four-channel post-change coupling increases from the common 0.18 Base. (B) Added-distance fractions and simultaneous one-sided 95% upper bounds over 20 new seeds. (C) The same attribution in a second eight-channel VAR for three prespecified models. The dashed line marks the prespecified 10% practical-significance criterion.

Figure 5: Held-out batches separate stable from visually compelling discovery patterns. Violin shapes and small points show seed-level distributions for the prespecified contrasts H1 through H5 in independent batches A, B, and C; larger circles and whiskers show batch means and two-sided 95% Student-

t

intervals, and dark diamonds show the All-30 precision analysis. Green and red regions mark the locked and opposite directions for one-sided tests. The vertical axis is the seed-level value of each locked scalar contrast; zero denotes no contrast, and panel scales differ. Only H4 is supported in every batch.

Figure 6: Held-out lead profiles reveal batch heterogeneity. Rolling means compare batches A–C with pooled seeds; shading marks locked blocks. Decisions follow Table 3 .

A. End-to-end controls with known attribution
Evidence	Scope	Reference quantity	Observed model term	Audit decision
Identical-data null	400 profiles	$\Delta B=0$	$\Delta D=0$	all 400 exactly zero
Innovation scale $.2\!\to\!.4$	100 seeds	$\Delta B=.120000$	$f=.20\%$	$U_{.95}=.22\%<10\%$
Delete latest AR input	100 seeds	$\Delta B=0$	$\Delta D=.000315$ ; $f=100\%$	95% CI $[.000262,.000372]$
B. Coupling attribution across severity and structure
Evidence	Scope	Environmental $\Delta B$	Added-distance fraction $f$	Upper-bound audit

Table 1: Main results for calibration, coupling generalization, and held-out stability.

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Record	Frozen	SHA-256 prefix	Purpose
Stage-2 execution v3	Sep 10, 20:58	7951e4b02fa39a86	Documents uniform OOM fallback; science unchanged
Stage-2.5/3 plan	Sep 10, 23:05	27c3500e69930ebb	Locks new seeds, tasks, and All-30 analysis
Stage-3 clarification	Sep 10, 23:50	fb56eb02cf0f58bd	Fixes estimands before model results
Stage-4 manifest	Sep 11, 13:32	50fadb5ec398f18f	Locks controls, severity, and second system
Topology manifest	Sep 11, 15:27	bb0346f88d410ad5	Locks matched Ring/Block/Hub audit

Appendix

Table 2: Prospective protocol locks and file hashes. SHA-256 fingerprints identify the exact bytes of each original prespecified record. Readable prefixes are shown here; the anonymous artifact records the complete original and release-copy digests for recomputation. Times use Asia/Hong Kong (UTC

+8

Sign-flip sensitivity: Holm-adjusted $p_{\rm SF,H}$ (H5 unadjusted as a one-test family)
Hyp.	Alternative	A: effect [95% CI]; $p_{t,H}$	B: effect [95% CI]; $p_{t,H}$	C: effect [95% CI]; $p_{t,H}$
H1	two-sided	$0.123$ [ $-0.074$ , $0.321$ ] $0.574$	$-0.063$ [ $-0.322$ , $0.195$ ] $1.000$	$0.013$ [ $-0.210$ , $0.235$ ] $1.000$
H2	$>0$	$0.002$ [ $-0.021$ , $0.026$ ] $0.844$	$-0.037$ [ $-0.085$ , $0.010$ ] $1.000$	$-0.012$ [ $-0.075$ , $0.052$ ] $1.000$
H3	$>0$	$-0.128$ [ $-0.230$ , $-0.026$ ] $0.990$	$-0.163$ [ $-0.215$ , $-0.112$ ] $1.000$	$-0.195$ [ $-0.280$ , $-0.110$ ] $1.000$
H4	$<0$	$-0.379$ [ $-0.509$ , $-0.248$ ] $2.05\times 10^{-4}$	$-0.350$ [ $-0.474$ , $-0.227$ ] $2.44\times 10^{-4}$	$-0.379$ [ $-0.491$ , $-0.268$ ] $6.08\times 10^{-5}$
H5	two-sided	$-0.061$ [ $-0.133$ , $0.011$ ] $0.087$	$-0.082$ [ $-0.163$ , $-2.13\times 10^{-4}$ ] $0.050$	$-0.231$ [ $-0.520$ , $0.058$ ] $0.105$

Appendix

Table 3: Held-out results from three independent ten-seed batches. Two-sided 95% Student-

t

intervals quantify effect size.

p_{t,H}

is the direction-matched Student-

t

p

-value after Holm correction;

p_{\rm SF,H}

is the corresponding seed-level sign-flip sensitivity result. H1–H4 form one family within each batch and pooled analysis; H5 is its separately locked one-test family.

Figure 7: Lead-resolved and block-level held-out evidence for H2–H5. (A) overlays the three lead profiles on a common scale. (B) reports the four locked block contrasts with 95% confidence intervals. (C) shows their batch-level means; confirmatory decisions still follow the prespecified batch-level rules.

Model	$\Delta$ MSE real	$\Delta R_{\rm oracle}$	$\Delta B$	$\Delta D$	$f_{m}$ [95% CI]	$U_{.95}$
Chronos-2	0.01242	0.01263	0.01235	0.00028	0.022 [0.013, 0.033]	0.031
TTM	0.01258	0.01264	0.01235	0.00029	0.023 [0.013, 0.035]	0.033
DLinear	0.01250	0.01263	0.01235	0.00028	0.022 [0.017, 0.028]	0.027
PatchTST	0.01275	0.01289	0.01235	0.00054	0.042 [0.034, 0.050]	0.049
TSMixer	0.01227	0.01251	0.01235	0.00016	0.013 [0.008, 0.018]	0.017
iTransformer	0.01223	0.01259	0.01235	0.00024	0.019 [0.013, 0.027]	0.025

Appendix

Table 4: Attribution for a multivariate coupling-strength shift.

f_{m}=\Delta D_{m}/(\Delta B+\Delta D_{m})

uses analytic expected risk, rather than realized raw MSE, in its denominator. Intervals and

U_{.95}

are percentiles from 50,000 paired DGP-seed bootstrap draws;

U_{.95}

is the one-sided 95% upper bound.

Model	Origin 3,712	Origin 3,904
Chronos-2	0.006 (0.009)	0.038 (0.055)
TTM	0.005 (0.008)	0.040 (0.060)
DLinear	0.009 (0.013)	0.035 (0.043)
PatchTST	0.005 (0.008)	0.075 (0.088)
TSMixer	0.010 (0.016)	0.015 (0.024)
iTransformer	0.009 (0.014)	0.029 (0.042)

Appendix

Table 5: Attribution by forecast origin for the multivariate coupling-strength shift. Origin 3,712 uses an all-pre-change context; origin 3,904 includes 192 post-change observations without a parameter update. Entries are

f_{m}

(one-sided 95% upper bound).

Model	$\Delta R_{oracle}$	95% CI	$p_{H}$
Chronos-2	0.0020	[-0.0000, 0.0041]	0.324
TimesFM	0.0032	[-0.0004, 0.0067]	0.378
AR	-0.0002	[-0.0004, 0.0001]	0.638
NST	0.0007	[-0.0003, 0.0017]	0.592
SegRNN	0.0002	[-0.0001, 0.0004]	0.638
TSMixer	-0.0161	[-0.1040, 0.0719]	0.707

Appendix

Table 6: Standardized Student-

t_{5}

Shift versus Gaussian Base at

H=192

Model	0.26	0.34	0.42
Chronos-2	1.05 (1.65)	1.25 (2.43)	2.33 (6.52)
TTM	1.46 (2.37)	1.64 (2.60)	2.01 (3.41)
DLinear	1.68 (2.11)	1.74 (2.18)	1.94 (2.54)
PatchTST	3.17 (4.30)	3.39 (4.45)	3.93 (5.15)
TSMixer	0.68 (1.03)	0.71 (1.04)	0.85 (1.41)
iTransformer	1.14 (1.77)	1.20 (1.90)	1.46 (2.37)

Appendix

Table 7: Coupling attribution at three shift severities. Cells are added oracle-distance fraction in percent, with the locked family-wise one-sided 95% upper bound in parentheses. Bounds use 50,000 paired DGP-seed bootstrap draws and Bonferroni correction over the 18 cells. The common Base coupling is 0.18; 20 new DGP seeds are used.

Model	$\Delta B$	$\Delta D$	$f$ (%)	$U_{.95}$ (%)
DLinear	0.003110	0.000041	1.29	1.61
PatchTST	0.003110	0.000095	2.96	3.92
Chronos-2	0.003110	0.000026	0.84	2.19

Appendix

Table 8: Attribution in the eight-channel two-community VAR. Cross-community coupling changes from 0.02 to 0.16 over 20 new DGP seeds.

U_{.95}

uses 50,000 paired DGP-seed bootstrap draws and Bonferroni correction over the three models.

System	Channels	Graph	Coupling Base $\to$ Shift	Spectral radius Base $\to$ Shift	$\Delta B$	Models	Seeds
Ring VAR	4	directed cycle	$.18\to\{.26,.34,.42\}$	$.63\to\{.71,.79,.87\}$	$.00438,.01235,.02917$	6	20
Two-community VAR	8	two dense blocks	$.02\to.16$	$.54\to.68$	$.00311$	3	20

Appendix

Table 9: Configurations of the two multivariate systems. Coupling coefficients are structure-specific controls rather than a common severity scale. The eight-channel panel was fixed before execution to represent a linear target-trained model (DLinear), a deep target-trained model (PatchTST), and a frozen TSFM (Chronos-2).

	Ring		Block		Hub
Model	$\tau=0$	$\tau=192$	$\tau=0$	$\tau=192$	$\tau=0$	$\tau=192$
Chronos-2	0.44 (0.72)	2.04 (4.45)	0.30 (0.43)	2.92 (5.04)	0.84 (1.42)	0.69 (3.56)
TTM	0.55 (0.81)	2.71 (4.63)	0.23 (0.46)	5.78 (9.84)	0.86 (1.51)	3.59 (8.28)
DLinear	0.61 (1.07)	2.85 (3.72)	0.44 (0.70)	4.10 (5.61)	0.97 (1.67)	2.80 (3.61)
PatchTST	0.61 (1.04)	6.02 (8.04)	0.55 (0.92)	9.05 (11.97)	1.11 (2.02)	6.11 (8.17)
TSMixer	0.61 (0.95)	0.80 (1.54)	0.62 (0.92)	1.79 (3.12)	1.11 (2.20)	1.17 (2.21)

Appendix

Table 10: Attribution across coupling topologies with matched environmental difficulty. Cells report the added-distance fraction in percent and the simultaneous one-sided 95% upper bound in parentheses. Ring is the previously locked reference; Block and Hub use independent 20-seed batches with environmental-risk changes matched to the Ring value. Bold marks Block–PatchTST at

\tau=192

, the only bound above the prespecified 10% margin (11.97%).

Figure 8: Origin-resolved coupling-severity responses for all six models. Lines show the added-distance fraction across the three Shift couplings; shaded bands are two-sided 95% paired-seed bootstrap intervals. The dashed line is the prespecified 10% diagnostic margin, not a theoretical constant.

Lead block	Base	Shift	Shift $-$ Base
1–24	0.550 [0.410, 0.690]	0.261 [0.248, 0.274]	-0.289 [-0.425, -0.153]
169–192	0.250 [0.250, 0.250]	0.250 [0.250, 0.250]	-0.000 [-0.000, 0.000]

Appendix

Table 11: Oracle-target amplitude for the locked switching contrast. The analysis uses 30 held-out DGP seeds, with origins averaged within each seed. Entries are

\mathbb{E}[(\mu^{F})^{2}]

with two-sided 95% Student-

t

intervals.

Benchmark	Real-data breadth	Controlled mechanisms	Strict pairing	Origin oracle	Risk split	$k\!\times\!h$ profile	Adaptation
TFB / GIFT-Eval ( Qiu et al., 2024 ; Aksu et al., 2024 )	Yes	–	–	–	–	–	–
SynTSBench ( Tan et al., 2025 )	–	Yes	–	Partial	–	Partial	–
FinStressTS ( Sun et al., 2026 )	–	Yes	Partial	Partial	–	Partial	–
Ours	Partial	Yes	Yes	Yes	Yes	Yes	Yes

Appendix

Table 12: Evaluation scope of representative forecasting benchmarks. “Partial” denotes a related capability with a different target; “–” means outside the work’s primary scope, not a quality judgment. Entries describe the cited papers’ documented primary evaluations.

Protocol	Base	Shift	What the contrast supports
Statistical anchors	Refit from visible Base context	Refit from visible Shift context	Stress response of a fixed fitting rule
Univariate target-trained (15)	Train/validate/test on Base (70/10/20)	Separately train/validate/test on Shift (70/10/20)	Mechanism-controlled response of an environment-specific fitting procedure
Multivariate target-trained	Fit on common pre-change prefix	Identical pre-change data/seed; no post-change update	Immediate OOD at the change point; passive contextual response at the later origin
Frozen TSFM (5)	Fixed pretrained checkpoint	The identical checkpoint; only context changes	Fixed-model cross-environment stress
Matched adaptation (5)	Random/zero/adapt arms on Base support/test	Separately matched arms on Shift support/test	Whether environment-specific limited-data intervention repairs the diagnosed gap

Appendix

Table 13: Training and evaluation protocols. The multivariate relation shift is separated from the univariate environment-specific training protocol.

Entry	Protocol	Raw MSE	Oracle distance	SMSE
DGP oracle mean	Evaluator reference	0.2558	0.0000	0.9948
AR	Statistical	0.2700	0.0159	1.0148
ETS	Statistical	0.3339	0.0912	1.1516
Naive	Statistical	0.4464	0.2185	1.8908
Seasonal naive	Statistical	0.5994	0.3509	2.4151
SegRNN	Target-trained	0.2672	0.0063	1.0213

Appendix

Table 14: Aggregate results on the shared discovery grid. The table contains 24 deployable forecasters plus the evaluator-only DGP oracle-mean reference. Lower is better within each metric, but raw MSE, oracle distance, and SMSE answer different questions. Values are descriptive averages over the common 38 case–horizon cells; DGP seed remains the independent unit for inference.

Term	Definition	Interpretation	Interpretive scope
Raw risk	$\mathbb{E}(Y-\hat{Y})^{2}$	Operational squared loss	Jointly reflects environmental risk and oracle distance
Environmental risk	$\mathbb{E}\operatorname{Var}(Y\mid\mathcal{F}_{t}^{\mathrm{eval}})$	Evaluator-conditioned uncertainty	Uses declared pre-origin information only
Forecast–oracle distance / $\operatorname{MSE}_{\mu}$	$\mathbb{E}(\hat{Y}-\mu^{F})^{2}$	Information restriction plus model approximation/estimation	Combined distance, not architecture error alone
SMSE	$\mathbb{E}[(Y-\hat{Y})^{2}/\max(\sigma^{F},\epsilon)^{2}]$	Mean pointwise error relative to local predictive scale	Secondary diagnostic, not MASE or probabilistic calibration
Stress profile	$\{A_{m}(k,h)\}_{k,h}$	Paired response over mechanism and individual lead	Aggregation requires deployment weights over $(k,h)$

Appendix

Table 15: Terms used throughout the paper.

Figure 9: Complete lead-resolved forecast–oracle-distance atlas. Rows are the 24 deployable forecasters, columns are mechanism contrasts, and each microplot runs from lead 1 to 192; red/blue fill denotes farther from/closer to the oracle under Shift. Column-specific symmetric limits are stated above the panels, and protocol groups are separated by dark rules.

Figure 10: Early-to-late diagnostic phase portraits. Circles average leads 1–24, triangles average leads 169–192, and line segments connect the same forecaster; the horizontal and vertical axes are paired changes in realized raw squared risk and forecast–oracle distance.

Stress	Model-resolved evidence	Structure-linked interpretation	Evidential status
Strong trend	AR is numerically unchanged. TSMixer increases by $.305$ $[.281,.330]$ , TTM by $.077$ $[.071,.085]$ , and Time-MoE by $.086$ $[.073,.100]$ ; PatchTST changes by $-.003$ $[-.004,-.002]$ .	AR explicitly fits a linear trend in each environment. Persistence and fixed-window mixing do not enforce slope extrapolation, which can produce a level-dependent direct forecast. PatchTST shows that lacking an explicit trend term is not sufficient for failure.	The TSMixer failure is clear in discovery, but the locked H2 claim about stronger late-lead accumulation does not replicate.
Frequent switching	Descriptively, Naive increases by $.258$ $[.127,.376]$ , whereas AR, ETS, TSMixer, and Chronos-2 change by $-.071$ , $-.172$ , $-.421$ , and $-.187$ ; ETS is not in the locked panel. The formal H3/H4 panel is AR, Chronos-2, TimesFM, and TSMixer, whose late effect is negative in all three held-out batches.	A last-value predictor persists the current regime and becomes stale after rapid switches. Other forecasting procedures may approach the common long-run predictive center more closely under faster switching.	The panel-level late reduction is confirmed. Its mechanism and individual-model differences remain descriptive.
Outlier-variance feedback	Nineteen of 24 methods combine higher raw MSE with lower SMSE. Seasonal naive and naive increase their oracle distance by $.182$ and $.104$ , while AR changes by only $.003$ .	Copying a recent or seasonal observation can propagate an event-contaminated value. A fitted low-order dynamic can dampen that transient. Much of the raw-loss increase still reflects predictive scale rather than model distance.	The cross-protocol direction is broad, but the proposed propagation route has not been isolated by an architectural ablation.
Relation shift	At $\tau=0$ , every Base and Shift forecast is bitwise identical. At $\tau=192$ , PatchTST fractions are $6.02\%$ , $9.05\%$ , and $6.11\%$ for Ring, Block, and Hub, compared with $0.80\%$ , $1.79\%$ , and $1.17\%$ for TSMixer.	Because weights are fixed, the later separation reflects how each architecture converts 192 post-shift observations into a forecast. Patch tokenization and temporal mixing are associated with different passive context responses, not different online adaptation.	The fractions replicate on independent topology seeds; which internal operation causes the difference remains open.

Appendix

Table 16: Observed failure patterns and architecture-linked hypotheses. Numerical effects are changes in forecast–oracle distance unless a fraction is stated. The evidence column reports observations, while the interpretation column states the corresponding structural hypothesis.

Control	Prespecified pass rule	Observed attribution	Result
Identical-data null	all paired effects are numerically zero	400/400 profiles: $\Delta R=\Delta B=\Delta D=0$	pass
Innovation $\sigma:.2\to.4$	$\Delta B>0$ and $U_{.95}(f)<10\%$	$\Delta B=0.120000$ , $f=0.20\%$ , $U_{.95}=0.22\%$	pass
Delete last AR input	$\Delta B=0$ and $\Delta D>0$	$\Delta B=0.000000$ , $\Delta D=0.000315$ , $f=100\%$ , CI [0.000262, 0.000372]	pass

Appendix

Table 17: End-to-end evaluator calibration. The pass rules were locked before execution. Positive-control intervals and the fraction bound use 50,000 paired seed-bootstrap draws.

Figure 11: Deployment weighting changes stress-response rankings. Each panel contains all 24 deployable forecasters; lower rank means a smaller weighted forecast–oracle-distance response. Scores are standardized within each scenario, the dashed vertical line is that scenario’s mean, and the top three methods are labeled.

Pair	Case role(s)	Mechanism	Exact parameters	Formal horizons
Trend strength	Base; moderate; strong	linear mean	$b=0;.001;.002$ , $a=.5,\sigma=.2$	1,24,96,192
Frequency acceleration	Base; Shift	sinusoidal mean	$A=1$ , $f:1/48\!\to\!1/48;1/48\!\to\!1/12$ , $\sigma=.2$	1,24,96,192
Innovation shape	Base; $t_{3}$ ; skew; mixture	standardized $Z_{t}$	Gaussian; $\nu=3$ ; shape $=5$ ; $w=.5,m_{c}=2,\sigma_{c}=1$	1,24,96,192
GARCH persistence	Base; Shift	GARCH(1,1)	$(\omega,\alpha,\beta)=(.0272,.08,.75);(.0048,.12,.85)$	1,24,96,192
Markov rate	Base; Shift	two-state Gaussian	$\mu=(-.5,1.5)$ , $\sigma=(.2,.2)$ , stay $=.97;.75$	1,24,96,192
Additive outliers	Base; strong	symmetric events	$(p,A)=(0,0);(.02,2)$	1,24,96,192

Appendix

Table 18: Univariate mechanism pairs and parameter settings. Shared defaults are

a=.5

b=.002

, and

\sigma=.2

unless replaced below; parameters not named within a pair are identical.

Figure 12: Environmental majority persists across matched-difficulty Ring, Block, and Hub relations. (A) Added-distance fractions for six models immediately after the relation shift ( $\tau=0$ ) and after 192 post-shift observations ( $\tau=192$ ); the systems match $\Delta B\approx 0.01235$ . (B) The largest simultaneous one-sided 95% upper confidence bound across models for each topology–origin combination.

Figure 13: Model-resolved attribution across matched-difficulty coupling topologies. Points show the added-distance fraction at the immediate and later origins; error bars are two-sided 95% paired-seed bootstrap intervals. The green band lies below the prespecified 10% margin; formal simultaneous one-sided upper bounds are reported in Table 10 .

Implementation	lr / bs	Main dimensions	Other fixed settings
TimeKAN	$.01/128$	$d=16,f=32,L=2$	downsample layers/window $=2/2$ , order 0
TimeMixer	$.01/128$	$d=16,f=32,L=2$	average downsample layers/window $=2/2$
PaiFilter	$.01/32$	hidden 256	label length 0
TexFilter	$.001/32$	embed/hidden $=512/512$	dropout 0
TimesNet	$.001/32$	$d=512,f=2048,L=2$	top- $k=5$ , factor 3
SegRNN	$.0001/32$	$d=512$	dropout .5; segment $1$ for $H=1$ , else $2$

Appendix

Table 19: Implementation settings for target-trained models. “lr/bs” denotes learning rate and batch size;

d

and

f

denote model and feed-forward width.

Model	Repository	Revision prefix	Context/horizon and output
Chronos-2	amazon/chronos-2	29ec3766d36d	$C=512,H\leq 192$ ; point + 9 quantiles
TimesFM 2.5	google/timesfm-2.5-200m-pytorch	1d952420fba8	$C=512,H\leq 192$ ; point + 9 quantiles
Moirai 2.0 R-small	Salesforce/moirai-2.0-R-small	30f43ff08c84	$C=512,H\leq 192$ ; point + 9 quantiles
TTM-R2	ibm-granite/granite-timeseries-ttm-r2	d6a79570cac0 / 25f4a00a25e1	$H=1,24,96$ : native 512–96; $H=192$ : native 512–192; point only
Time-MoE-200M	Maple728/TimeMoE-200M	794591bfeb12	$C=512,H\leq 192$ autoregressive; point only

Appendix

Table 20: Frozen TSFM checkpoints and forecasting settings. Revision entries are 12-character prefixes of immutable repository commits; the released machine-readable manifests retain every full hash.

Model	Zero	Adapt	$\Delta$ [95% CI]	Repair [95% CI]	$p_{H}$
TimesFM	.279	.253	$-.026$ [ $-.038,-.010$ ]	9.3% [3.9,13.5]	.031
Chronos-2	.271	.271	$-.0001$ [ $-.0013,.0009$ ]	0.04% [ $-0.3,0.5$ ]	.891
Moirai	.300	.270	$-.030$ [ $-.042,-.017$ ]	9.9% [6.0,14.3]	.010
TTM	.344	.256	$-.088$ [ $-.106,-.071$ ]	25.5% [21.5,30.1]	.010
Time-MoE	.354	.255	$-.098$ [ $-.108,-.089$ ]	27.8% [25.2,31.1]	.010

Appendix

Table 21: Matched adaptation results on the discovery grid. Effects average origins and three training seeds inside each of 10 DGP seeds. Repair is

(\text{zero}-\text{adapt})/\text{zero}

; intervals bootstrap DGP seeds and

p_{H}

is from the paired sign-flip test (finite-sample exact under sign exchangeability) with Holm correction across five models. Repair values are computed from unrounded seed-level quantities rather than the displayed means.

Donor (DGP seeds)	Frozen model	Reference MSE	Donor MSE	$\Delta$ raw MSE	$\Delta$ center distance	Paired windows
UCI Air Quality (10)	Chronos-2	0.0405	0.0369	-0.0036	+0.0055	80
	TimesFM	0.0423	0.0764	+0.0341	+0.0389	80
	Moirai	0.0505	0.0612	+0.0108	+0.0174	80
	TTM	0.1169	0.1040	-0.0129	-0.0081	80
	Time-MoE	0.1230	0.1384	+0.0154	+0.0211	80
UCI Electricity (10)	Chronos-2	0.0405	0.0378	-0.0027	+0.0078	80

Appendix

Table 22: Sensitivity to semisynthetic residual noise. Each donor residual pool is extracted by robust STL on the named real series, centered and standardized, then injected at scale

.2

around the same linear center as the Gaussian reference. Values pool four horizons and two origins within each DGP seed, preserving the seed as the unit of description. The reported center distance reflects the absence of a validated conditional residual resampler.

	V only		F/M + V		F + M + V
$H$	$\mathrm{MSE}_{true}$	DED	$\mathrm{MSE}_{true}$	DED	$\mathrm{MSE}_{true}$	DED
24	0.0196	0.0073	0.3584	0.2699	0.4709	0.5390
48	0.0204	0.0078	0.3340	0.2913	0.4587	0.5749
96	0.0179	0.0069	0.3385	0.3196	0.4523	0.6143
192	0.0170	0.0065	0.3484	0.3499	0.4494	0.6516

Appendix

Table 23: Exploratory FMV factorial results. Results average 12 implementation labels under one DGP/training seed and serve as a hypothesis-generating factorial analysis.

\mathrm{MSE}_{true}

is the deterministic predecessor of oracle distance and DED is the fraction outside the known scale band.

Explore similar work

FinStressTS: A Parametric Synthetic Benchmark for Time-Series Forecasting in Finance
Jun 2, 2026Jiaze Sun, Kelvin J. L. Koa, Ruiyang Ni +3Time Series Forecasting Finance Benchmarks
National University of Singapore Singapore · National University of Singapore Asian Institute of Digital Finance Singapore · Nanyang Technological University Singapore
OracleProto: A Reproducible Framework for Benchmarking LLM Native Forecasting via Knowledge Cutoff and Temporal Masking
May 5, 2026Yiding Ma, Chengyun Ruan, Kaibo Huang +2Large Language Model Forecasting Data-Driven Forecasting
Beijing University of Posts and Telecommunications
OccStress: Stress-Testing the 4D Occupancy Forecasting Chain
Dec 17, 2025Yu Zheng, Jie Hu, Jiaqi Xiong +4
Hunan University · ETH Zurich

Jun 2, 2026, q-fin.CPJ/K move · Enter open · S save

FinStressTS: A Parametric Synthetic Benchmark for Time-Series Forecasting in Finance

Jiaze Sun, Kelvin J. L. Koa, Ruiyang Ni, Yize Liu+2

National University of Singapore Singapore · National University of Singapore Asian Institute of Digital Finance Singapore · Nanyang Technological University Singapore

Time Series Forecasting Finance Benchmarks Volatility Synthetic Benchmark Conformal Risk Control Probabilistic Model

Open paper AskPDF arXiv

Financial forecasting is difficult due to low signal-to-noise ratios, latent factors, heavy tails, regime shifts, and jumps. Real-world benchmarks offer limited failure attribution: researchers can observe underperformance, but often cannot isolate why because mechanisms are unobservable and entangled. Real financial data reveal only one realized path, making it difficult to assess tail-risk calibration or data efficiency. We introduce FinStressTS, a mechanism-aware synthetic benchmark that links model behavior to controlled structural causes. FinStressTS comprises 30 diagnostic environments around six mechanism families: volatility clustering, multi-scale persistence, heavy-tailed shocks, regime switching, self-exciting jumps, and zero-inflated processes. We evaluate two tasks: point forecasting, using NMAE across five settings, and probabilistic forecasting, using CRPS under known data-generating mechanisms. We benchmark 15 models, from classical methods (HAR, VAR) to Transformer forecasters (PatchTST, iTransformer) and deep probabilistic architectures (DeepAR, TSFlow), and use learning curves to measure sample efficiency. Our evaluation reveals three insights. First, performance is mechanism-dependent: autoregressive and linear models are highly competitive, and often outperform Transformer-based models, in several volatility-, tail-, and jump-driven environments. Second, distributional alignment matters: parametric probabilistic models such as DeepAR calibrate well in stationary settings, while flexible models can help when distributions become multimodal or sparse. Third, neural models often require more data to match simple baselines, with larger gains mainly when learning latent regimes or complex distributions. FinStressTS provides an open framework for diagnosing failure modes and advancing risk-aware forecasting.

Abstract

Explore similar work

FinStressTS: A Parametric Synthetic Benchmark for Time-Series Forecasting in Finance

OracleProto: A Reproducible Framework for Benchmarking LLM Native Forecasting via Knowledge Cutoff and Temporal Masking

OccStress: Stress-Testing the 4D Occupancy Forecasting Chain