A Controlled Comparison of Deep Learning Architectures for Multi-Horizon Financial Forecasting: Evidence from 918 Experiments

Nabeel Ahmad Saidd

q-fin.STFeb 27, 2026

A Controlled Comparison of Deep Learning Architectures for Multi-Horizon Financial Forecasting: Evidence from 918 Experiments

Authors: Nabeel Ahmad Saidd

Organizations: Dr. A.P.J. Abdul Kalam Technical University (AKTU)

Abstract

Multi-horizon price forecasting is central to portfolio allocation, risk management, and algorithmic trading, yet deep learning architectures have proliferated faster than rigorous financial benchmarks can evaluate them. This study provides a controlled comparison of nine architectures (Autoformer, DLinear, iTransformer, LSTM, ModernTCN, N-HiTS, PatchTST, TimesNet, and TimeXer) spanning Transformer, MLP, CNN, and RNN families across cryptocurrency, forex, and equity index markets at 4-hour and 24-hour horizons. A total of 918 experiments were conducted under a strict five-stage protocol including fixed-seed Bayesian hyperparameter optimization, configuration freezing per asset class, multi-seed retraining, uncertainty aggregation, and statistical validation. ModernTCN achieves the best mean rank (1.333) with a 75 percent first-place rate, followed by PatchTST (2.000). Results reveal a clear three-tier ranking structure and show that architecture explains nearly all performance variance, while seed randomness is negligible. Rankings remain stable across horizons despite 2 to 2.5 times error amplification. Directional accuracy remains near 50 percent across all configurations, indicating that MSE-trained models lack directional skill at hourly resolution. The findings highlight the importance of architectural inductive bias over raw parameter count and provide reproducible guidance for multi-step financial forecasting.

Figures & tables

Study	Models	Datasets	Horizons	Multi-Seed	Pairwise Tests	Open Code
( Makridakis et al., 2018 ) (M4)	Many	100K series	Multiple	No	No	Partial
( Hewamalage et al., 2021 )	RNN only	6	Multiple	No	No	Partial
( Zeng et al., 2023 )	6	9	4	No	No	Yes
( Wu et al., 2023 )	8	8	4	No	No	Yes
( Nie et al., 2023 )	7	8	4	No	No	Yes
( Liu et al., 2024 )	8	7	4	No	No	Yes

Table 1: Summary of prior comparative studies in time-series forecasting. Columns indicate the number of models evaluated, number of datasets or asset classes, horizons tested, whether multi-seed evaluation was performed, and whether post-hoc pairwise statistical tests were applied.

Category	Asset	Date Range	Train	Val	Test	Total
Crypto	BTC/USDT	2021-03 – 2026-02	21,000	4,500	4,500	30,000
	ETH/USDT	2021-03 – 2026-02	21,000	4,500	4,500	30,000
	BNB/USDT	2021-03 – 2026-02	21,000	4,500	4,500	30,000
	ADA/USDT	2021-05 – 2026-02	21,000	4,500	4,500	30,000
Forex	EUR/USD	2017-12 – 2026-02	21,000	4,500	4,500	30,000
	USD/JPY	2017-12 – 2026-02	21,000	4,500	4,500	30,000

Table 2: Dataset summary. All assets use H1 (hourly) frequency. The most recent 30,000 windowed samples are retained per (asset, horizon) pair, split chronologically into 70%/15%/15% train/val/test partitions. Window lengths:

w=24

for

h=4

and

w=96

for

h=24

. Features: OHLCV (5 channels); target: close price.

Cat.	Asset	$\bar{\mu}_{r}$ ( ${\times}10^{-4}$ )	$\sigma_{r}$ (%)	Skew	Kurt	ACF(1)	ADF $p$
Crypto	BTC/USDT	$+0.4$	0.783	$-0.47$	$+35.8$	$-0.043$	$<$ 0.001
	ETH/USDT	$+0.3$	0.973	$-0.56$	$+27.3$	$-0.026$	$<$ 0.001
	BNB/USDT	$+0.8$	1.107	$+0.08$	$+35.0$	$-0.077$	$<$ 0.001
	ADA/USDT	$\phantom{+}0.0$	1.187	$-0.05$	$+19.2$	$-0.085$	$<$ 0.001
Forex	EUR/USD	$\phantom{+}0.0$	0.109	$-0.01$	$+15.2$	$-0.010$	$<$ 0.001
	USD/JPY	$+0.1$	0.119	$-0.98$	$+46.6$	$-0.002$	$<$ 0.001

Table 3: Distributional statistics of hourly log returns for all twelve assets. All series reject the Augmented Dickey–Fuller unit-root null at

p<0.001

, confirming return stationarity.

Figure 1: Representative hourly Close-price time series for one asset per class: BTC/USDT (cryptocurrency), EUR/USD (forex), and Dow Jones (equity indices). Vertical dashed lines indicate chronological train/validation/test boundaries (70/15/15 split). The three classes exhibit qualitatively different dynamics: high-volatility trending behaviour (cryptocurrency), low-volatility mean-reversion around a narrow range (forex), and moderate-volatility upward drift (equity indices). All series comprise the most recent 30,000 hourly observations.

Model	Family	Key Mechanism	Reference
Autoformer	TF	Auto-correlation & series decomposition	( Wu et al., 2021 )
DLinear	MLP	Linear layers with trend–seasonal decomposition	( Zeng et al., 2023 )
iTransformer	TF	Inverted attention on variate tokens	( Liu et al., 2024 )
LSTM	RNN	Gated recurrent cells + dense MLP projection head	( Hochreiter and Schmidhuber, 1997 )
ModernTCN	TCN	Large-kernel depthwise convolutions with patching	( Luo and Wang, 2024 )
N-HiTS	MLP	Hierarchical interpolation with multi-rate pooling	( Challu et al., 2023 )

Table 4: Summary of the nine deep learning architectures evaluated. All models receive OHLCV input of shape

(B,\,w,\,5)

and produce direct multi-step forecasts of shape

(B,\,h)

. Family abbreviations: RNN = recurrent, MLP = multi-layer perceptron, TCN = temporal convolutional network, TF = Transformer.

Model	Hyperparameter	Range / Choices
Autoformer	Model dimension	$\{64,128\}$
	Attention heads	$\{4,8\}$
	Enc./dec. layers	$\{1,2\}$ each
	Feedforward dimension	$\{64,128\}$
	Dropout rate	$[0.0,0.2]$ , step $0.05$
DLinear	Moving-average kernel	$[3,51]$ , step $2$

Table 5: Hyperparameter search spaces per model. All models share learning rate

\in[5\times 10^{-4},\,5\times 10^{-3}]

(log-uniform) and batch size

\in\{64,128\}

. Only architecture-specific parameters are shown. HPO uses Optuna TPE with 5 trials per (model

\times

horizon

\times

asset class) on representative assets (BTC/USDT, EUR/USD, Dow Jones) only.

Model	Category	$h$	Key Parameters
Autoformer	Crypto/Forex	4/24	$d_{\mathrm{model}}{=}128$ , heads ${=}4$ , enc ${=}1$ , dec ${=}2$ , $d_{\mathrm{ff}}{=}128$ , drop ${=}0.20$
Autoformer	Indices	4/24	$d_{\mathrm{model}}{=}128$ , heads ${=}4$ , enc ${=}1$ , dec ${=}2$ , $d_{\mathrm{ff}}{=}128$ , drop ${=}0.10$
DLinear	All	4	kernel ${=}21$ , individual ${=}\text{true}$
DLinear	Crypto/Forex	24	kernel ${=}43$ , individual ${=}\text{true}$
iTransformer	Crypto	24	$d_{\mathrm{model}}{=}112$ , $d_{\mathrm{ff}}{=}128$ , layers ${=}2$ , heads ${=}16$ , drop ${=}0.15$
iTransformer	Other	4	$d_{\mathrm{model}}{=}96$ , $d_{\mathrm{ff}}{=}128$ , layers ${=}4$ , heads ${=}16$ , drop ${=}0.00$

Table 6: Frozen best hyperparameters selected via Optuna TPE (5 trials, seed 42, objective: minimum validation mse ). Only key architecture-specific parameters are shown; all configurations also include learning rate and batch size. Shared entries across categories indicate that the representative-asset optimum was identical.

Figure 2: Five-stage experimental pipeline. Stage 1: Fixed-seed Bayesian HPO on representative assets (BTC/USDT, EUR/USD, Dow Jones; seed 42; 5 Optuna TPE trials; 50 epochs per trial). Stage 2: Best configuration frozen per (model, category, horizon) triple. Stage 3: Multi-seed final training (seeds 123, 456, 789; 100 epochs maximum; early stopping with patience 15). Stage 4: Test-set metric aggregation with inverse scaling (mean

\pm

std across seeds). Stage 5: Benchmarking with rank-based leaderboard analysis, visualisation, and variance decomposition. All 918 experimental runs—270 HPO trials plus 648 final training runs—are conducted under identical conditions.

Model	Family	Mean Rank	Median Rank	Wins (of 24)
ModernTCN	CNN	1.333	1.0	18 (75.0%)
PatchTST	Transformer	$2.000$	$2.0$	3 (12.5%)
iTransformer	Transformer	$3.667$	$3.0$	0
TimeXer	Transformer	$4.292$	$4.0$	0
DLinear	MLP / Linear	$4.958$	$5.0$	0
N-HiTS	MLP / Linear	$5.250$	$6.0$	3 (12.5%)

Table 7: Global model ranking aggregated across all 12 assets and both horizons (

h\in\{4,24\}

), categorised by architectural family. Each (asset, horizon) pair contributes one rank based on mean rmse over three seeds. Mean and median ranks are computed over 24 evaluation slots. Win Count indicates the number of slots in which a model achieved rank 1. Bold marks the best value per column.

Category	Asset	Best at $h=4$	Best at $h=24$
Crypto	BTC/USDT	PatchTST	ModernTCN
	ETH/USDT	N-HiTS	N-HiTS
	BNB/USDT	ModernTCN	ModernTCN
	ADA/USDT	N-HiTS	PatchTST
Forex	EUR/USD	ModernTCN	ModernTCN
	USD/JPY	ModernTCN	ModernTCN

Table 8: Best-performing model per asset and horizon, determined by lowest mean rmse across three seeds. ModernTCN achieves the lowest rmse on 18 of 24 evaluation points (75%); N-HiTS wins on 3 points (all in cryptocurrency); PatchTST wins on 3 points (one crypto, one crypto, one forex). Bold highlights non-ModernTCN winners, revealing niche asset-specific advantages.

Figure 3: Global rmse heatmap across eight modern architectures and 24 evaluation points (12 assets $\times$ 2 horizons). Lighter cells indicate lower error. ModernTCN and PatchTST consistently achieve the lowest rmse values across all asset–horizon combinations. LSTM is excluded for visual clarity; the full nine-model variant is provided in Appendix A.5 , Figure 31 . Values represent mean rmse across three seeds.

Figure 4: Global mean rank comparison across 24 evaluation points (12 assets

\times

2 horizons). Lower values indicate better performance. Three distinct tiers are visible: ModernTCN and PatchTST (ranks 1–2), a middle group of four models (ranks 3–6), and a bottom group comprising TimesNet, Autoformer, and LSTM (ranks 7–9). Error bars represent rank standard deviation across evaluation points.

	Crypto		Forex		Indices
Model	rmse	mae	rmse	mae	rmse	mae
Autoformer	$532.21$	$393.83$	$0.2068$	$0.1551$	$179.66$	$136.24$
DLinear	$323.48$	$220.87$	$0.1279$	$0.0940$	$123.80$	$90.67$
iTransformer	$320.83$	$217.99$	$0.1136$	$0.0785$	$113.72$	$79.13$
LSTM	$2398.94$	$2041.58$	$3.6679$	$3.5858$	$1548.56$	$1478.21$
ModernTCN	314.66	211.14	0.1098	0.0750	110.89	76.09

Table 9: Category-level mean rmse and mae averaged across all assets within each category and across both horizons (

h\in\{4,24\}

). Values are aggregated over 4 assets

\times

2 horizons

\times

3 seeds. Bold marks the best (lowest) value per category and metric. LSTM is included for completeness but excluded from ranking discussions due to convergence failures.

Figure 5: Category-level rank distributions across assets within each category, excluding LSTM for visual clarity. ModernTCN exhibits the tightest rank distribution (consistently rank 1 across all categories), indicating stable cross-asset performance. The full nine-model variant is provided in Appendix A.5 . Boxes show interquartile range; whiskers extend to the most extreme rank observed.

Figure 6: Category-level rmse vs. mae scatter plot for eight modern architectures. Each point represents one model’s mean error within a category. The near-perfect linear correlation (

R^{2}>0.99

) confirms that model rankings are consistent across the two error metrics, indicating that findings based on rmse generalise to mae . The full nine-model variant is provided in Appendix A.5 .

Model	rmse , $h{=}4$	rmse , $h{=}24$	$\Delta\%$
Autoformer	$1284.9$	$2670.0$	$107.8$
DLinear	$772.0$	$1644.1$	$113.0$
iTransformer	$743.5$	$1651.3$	$122.1$
LSTM	$8029.9$	$10\,878.7$	$35.5$
ModernTCN	$731.6$	1617.4	$121.1$
N-HiTS	$930.5$	$1724.5$	$85.3$

Table 10: Horizon degradation for BTC/USDT (top), EUR/USD (middle), and Dow Jones (bottom). rmse values are three-seed means.

\Delta\%

denotes the relative rmse increase from

h{=}4

h{=}24

\Delta\%=100\times(\mathrm{RMSE}_{24}-\mathrm{RMSE}_{4})/\mathrm{RMSE}_{4}

. Bold marks the model with the lowest rmse at

h{=}24

across all nine architectures.

Model	rmse , $h{=}4$	rmse , $h{=}24$	$\Delta\%$
Autoformer	$0.002\,84$	$0.006\,26$	$120.3$
DLinear	$0.001\,59$	$0.003\,47$	$118.5$
iTransformer	$0.001\,56$	$0.003\,45$	$121.0$
LSTM	$0.001\,79$	$0.003\,64$	$102.6$
ModernTCN	$0.001\,52$	0.00338	$121.4$
N-HiTS	$0.001\,70$	$0.003\,53$	$107.4$

Table 11: Horizon degradation for EUR/USD . rmse values are three-seed means.

\Delta\%

defined as in Table 10 .

Model	rmse , $h{=}4$	rmse , $h{=}24$	$\Delta\%$
Autoformer	$160.7$	$410.7$	$155.6$
DLinear	$121.3$	$279.1$	$130.1$
iTransformer	$115.2$	$258.4$	$124.3$
LSTM	$1261.7$	$1688.5$	$33.8$
ModernTCN	$112.5$	249.5	$121.7$
N-HiTS	$160.2$	$255.9$	$59.7$

Table 12: Horizon degradation for Dow Jones . rmse values are three-seed means.

\Delta\%

defined as in Table 10 .

	BTC/USDT			EUR/USD			Dow Jones
Model	$r_{4}$	$r_{24}$	$\Delta$	$r_{4}$	$r_{24}$	$\Delta$	$r_{4}$	$r_{24}$	$\Delta$
Autoformer	$8$	$8$	$0$	$8$	$8$	$0$	$8$	$7$	$-1$
DLinear	$5$	$4$	$-1$	$4$	$4$	$0$	$5$	$6$	$1$
iTransformer	$3$	$5$	+2	$3$	$3$	$0$	$3$	$5$	+2
LSTM	$9$	$9$	$0$	$7$	$7$	$0$	$9$	$9$	$0$
ModernTCN	$2$	$1$	$-1$	$1$	$1$	$0$	$1$	$1$	$0$

Table 13: Horizon ranking shift for the three representative assets.

r_{4}

and

r_{24}

: model rank at

h=4

and

h=24

respectively (lower is better; 1 = best).

\Delta=r_{24}-r_{4}

: positive values indicate rank degradation (the model performs relatively worse at longer horizons); negative values indicate rank improvement. Values in bold denote

|\Delta|\geq 2

. Rankings are based on mean rmse across three seeds.

Figure 7: Cross-horizon rmse comparison for eight modern architectures across twelve assets. Each asset group displays rmse at

h=4

and

h=24

. The top-tier architectures (ModernTCN, PatchTST) maintain superior rankings across both horizons, while middle-tier models exhibit varying sensitivity to the forecast window length.

Figure 8: Cross-horizon rmse degradation for eight modern architectures across representative assets. Lines connect each model’s rmse at

h=4

and

h=24

. All architectures exhibit absolute error growth, but degradation magnitudes are architecture-dependent: N-HiTS degrades least, while TimesNet and Autoformer exhibit the steepest increase. ModernTCN and PatchTST maintain top-tier performance at both horizons. The full nine-model variant is provided in Appendix A.5 .

Factor	Raw (%)	$z$ -norm, all (%)	$z$ -norm, no LSTM (%)
Model (architecture)	99.90	48.32	68.33
Seed (initialisation)	$0.01$	$0.04$	$0.02$
Residual (model $\times$ slot)	$0.09$	$51.64$	$31.66$

Table 14: Two-factor variance decomposition of forecast rmse . Raw (untransformed) and z-normalised (within each asset–horizon slot) panels are shown with and without LSTM. Seed variance is negligible (

<\!0.1\,\%

) in all cases.

Panel of Figure 9 — Figure 9: Seed robustness analysis. (a) Violin plot of seed-to-seed rmse variation. Inter-seed variation is negligible relative to inter-model differences across all nine architectures. (b) Two-factor variance decomposition on raw price-scale rmse : architecture explains 99.90%, seed 0.01%, residual 0.09%. After z-score normalisation within each (asset, horizon) slot, the architecture share falls to 48.3% (68.3% excluding LSTM); see Table 14 for the full dual-panel breakdown.

	Crypto		Forex		Indices
Model	$h{=}4$	$h{=}24$	$h{=}4$	$h{=}24$	$h{=}4$	$h{=}24$
Autoformer	$50.41$	$49.96$	$50.06$	$50.11$	$50.01$	$49.98$
DLinear	$49.70$	$49.94$	$50.13$	$50.12$	$49.97$	$49.96$
iTransformer	$49.58$	$50.22$	$50.04$	$49.99$	$49.90$	$50.18$
LSTM	$49.73$	$50.07$	$50.08$	$49.95$	$49.88$	$49.95$
ModernTCN	$50.04$	$49.96$	$49.96$	$50.03$	$50.34$	$50.03$

Table 15: Mean directional accuracy ( da , %) per model, asset class, and horizon, averaged across assets within each category and over three seeds. Values close to 50% indicate no systematic directional bias.

Scope	$\chi^{2}$ (8)	$F$	df	$p$	$n$
Global (all 12 assets, $h\in\{4,24\}$ )	$156.49$	$101.36$	(8, 184)	$<\!10^{-15}$	24
Crypto (4 assets, $h\in\{4,24\}$ )	$49.23$	$23.34$	(8, 56)	$<\!10^{-15}$	8
Forex (4 assets, $h\in\{4,24\}$ )	$56.00$	$49.00$	(8, 56)	$<\!10^{-15}$	8
Indices (4 assets, $h\in\{4,24\}$ )	$60.40$	$117.44$	(8, 56)	$<\!10^{-15}$	8

Table 16: Statistical significance tests — Panel A: Friedman-Iman-Davenport omnibus tests.

\chi^{2}

: Friedman

\chi^{2}

statistic (8 df);

F

: Iman-Davenport

F

-statistic;

n

: number of blocks (evaluation points). All tests reject

H_{0}

\alpha=0.001

Category	Asset	Spearman $\rho$	$p$ -value
Crypto	ADA/USDT	$0.683$	0.0424
Crypto	BNB/USDT	$0.917$	0.0005
Crypto	BTC/USDT	$0.917$	0.0005
Crypto	ETH/USDT	$0.783$	0.0125
Forex	AUD/USD	$0.867$	0.0025
Forex	EUR/USD	$1.000$	$<\!10^{-9}$

Table 17: Statistical significance tests — Panel B: Spearman rank correlations between model rankings at

h=4

and

h=24

, plus Stouffer combined test. Rankings are based on mean rmse across three seeds for each of the nine architectures (

n=9

per asset). All 12 per-asset correlations are significant at

\alpha=0.05

Asset ( $h{=}24$ )	icc	$F$ -statistic	$p$ -value
BTC/USDT	$0.9982$	$1650.2$	$<\!10^{-15}$
EUR/USD	$0.9904$	$309.6$	$<\!10^{-15}$
S&P 500	$0.9987$	$2255.6$	$<\!10^{-15}$

Table 18: Statistical significance tests — Panel C: Intraclass Correlation Coefficient ( icc ; two-way mixed, absolute agreement) for three representative assets at

h=24

. High icc values confirm negligible seed-to-seed variation relative to inter-model differences.

Category	Horizon	jt $z$	$p$ -value	Monotonic?
Crypto	4	$0.348$	$0.364$	No
Crypto	24	$-0.348$	$0.636$	No
Forex	4	$0.377$	$0.353$	No
Forex	24	$-0.290$	$0.614$	No
Indices	4	$-1.248$	$0.894$	No
Indices	24	$-1.480$	$0.931$	No

Table 19: Statistical significance tests — Panel D: Jonckheere-Terpstra ( jt ) test for a monotonic relationship between model complexity (parameter count) and rmse rank. A significant positive result would indicate that more parameters reliably yield better ranks. Three complexity groups:

\leq\!30\mathrm{K}

30\mathrm{K}

–

200\mathrm{K}

>\!200\mathrm{K}

parameters.

Panel of Figure 10 — Figure 10: Critical difference diagram for nine forecasting architectures across $N=24$ evaluation blocks (12 assets $\times$ 2 horizons). Models are placed at their exact mean rmse rank; lower rank is better. Thick horizontal bars connect models that are not statistically distinguishable at $\alpha=0.05$ under Holm-corrected Wilcoxon tests ( holm_wilcoxon.csv ): iTransformer–TimeXer ( $p_{\text{Holm}}=0.480$ ), DLinear–N-HiTS ( $p_{\text{Holm}}=0.529$ ), and TimesNet–Autoformer ( $p_{\text{Holm}}=0.529$ ). All other 33 pairwise comparisons are significant. The bracketed interval (upper left) shows the Nemenyi critical difference $\mathrm{CD}_{0.05}=2.451$ ( $k=9$ , $N=24$ , $q_{0.05}=3.102$ ) for reference only.

Panel of Figure 11 — Figure 11: Complexity–performance trade-off (excluding LSTM). The horizontal axis represents the number of trainable parameters (log scale); the vertical axis represents the mean rmse rank across all assets and seeds. The Pareto frontier is clearly defined by DLinear, PatchTST, and ModernTCN.

Panel of Figure 12 — Figure 12: Actual versus predicted close price for ModernTCN on BTC/USDT, $h=4$ (seed 123). (a) At step 1, the model tracks the actual price with high fidelity, capturing directional turns and amplitude fluctuations. (b) At step 4, trend structure is preserved but short-lived volatility spikes are modestly attenuated, consistent with MSE-induced shrinkage. No systematic phase shift is observed.

Panel of Figure 13 — Figure 13: Cross-architecture, cross-asset contrast at $h=4$ , step 1 (seed 123). (a) PatchTST on EUR/USD (rank 2) tracks the low-amplitude, mean-reverting forex dynamics with high accuracy. (b) TimeXer on Dow Jones (rank 4) captures the directional structure but with a wider deviation band, illustrating the qualitative signature of the rank gap between top and middle tiers.

Panel of Figure 14 — Figure 14: iTransformer (rank 3) on BTC/USDT at $h=4$ (seed 123). (a) At step 1, tracking fidelity is comparable to ModernTCN (Figure 12(a) ). (b) At step 4, slightly more amplitude attenuation is visible relative to ModernTCN (Figure 12(b) ), consistent with the 1.6% rmse gap. The inverted attention mechanism produces qualitatively similar but measurably weaker temporal representations for short-horizon cryptocurrency forecasting.

Panel of Figure 15 — Figure 15: Medium-horizon actual-versus-predicted overlays at step 12 of the $h=24$ forecast vector (seed 123). (a) ModernTCN on EUR/USD: directional content is preserved 12 hours ahead while high-frequency amplitude is dampened. (b) PatchTST on Dow Jones: directional integrity is maintained but the forecast envelope is wider than at $h=4$ , corroborating the $2$ – $2.5\times$ rmse amplification (Table 10 ). Both top architectures exhibit amplitude attenuation as the primary degradation mode.

Figure 16: Long-horizon overlay: ModernTCN on BTC/USDT,

h=24

, step 24 (seed 123). The model retains directional integrity and captures low-frequency trend components, but high-amplitude intra-day reversals are under-predicted. This pattern—directional fidelity without amplitude precision—is the signature of MSE-optimised direct multi-step forecasting at the maximum horizon. The rmse at

h=24

(1,617.4) represents a 121.1% degradation relative to

h=4

(731.6; Table 10 ).

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 17: rmse comparison for BTC/USDT at

h=4

(left) and

h=24

(right), excluding LSTM for visual clarity. PatchTST and ModernTCN achieve the two lowest errors at

h=4

; ModernTCN leads marginally at

h=24

. Mean

\pm

std across three seeds.

Figure 18: rmse comparison for EUR/USD at

h=4

(left) and

h=24

(right), excluding LSTM. ModernTCN and PatchTST rank highest across both horizons; the remaining modern architectures cluster within a narrow error band. Mean

\pm

std across three seeds.

Figure 19: rmse comparison for Dow Jones (USA30IDXUSD) at

h=4

(left) and

h=24

(right), excluding LSTM. ModernTCN ranks first at both horizons; PatchTST follows closely. Mean

\pm

std across three seeds.

Figure 20: rmse comparison for ETH/USDT at

h=4

(left) and

h=24

(right), excluding LSTM. N-HiTS achieves the lowest rmse at both horizons, one of only two assets where ModernTCN does not rank first. Mean

\pm

std across three seeds.

Figure 21: rmse comparison for BNB/USDT at

h=4

(left) and

h=24

(right), excluding LSTM. ModernTCN leads at both horizons with a clear margin. Mean

\pm

std across three seeds.

Figure 22: rmse comparison for ADA/USDT at

h=4

(left) and

h=24

(right), excluding LSTM. N-HiTS leads at

h=4

; PatchTST leads at

h=24

. This asset exhibits the lowest cross-horizon Spearman

\rho

(

0.683

; Table 17 ), consistent with more variable rankings between horizons. Mean

\pm

std across three seeds.

Figure 23: rmse comparison for USD/JPY at

h=4

(left) and

h=24

(right), excluding LSTM. ModernTCN ranks first; the modern architecture cluster is tightly packed. Mean

\pm

std across three seeds.

Figure 24: rmse comparison for GBP/USD at

h=4

(left) and

h=24

(right), excluding LSTM. PatchTST leads at

h=4

; ModernTCN leads at

h=24

. Mean

\pm

std across three seeds.

Figure 25: rmse comparison for AUD/USD at

h=4

(left) and

h=24

(right), excluding LSTM. ModernTCN ranks first at both horizons. Mean

\pm

std across three seeds.

Figure 26: rmse comparison for DAX (DEUIDXEUR) at

h=4

(left) and

h=24

(right), excluding LSTM. ModernTCN leads at both horizons; the top-four cluster (ModernTCN, PatchTST, iTransformer, TimeXer) is tightly grouped. Mean

\pm

std across three seeds.

Figure 27: rmse comparison for S&P 500 (USA500IDXUSD) at

h=4

(left) and

h=24

(right), excluding LSTM. ModernTCN leads; the cross-horizon Spearman

\rho=0.967

(Table 17 ) indicates near-perfect rank preservation. Mean

\pm

std across three seeds.

Figure 28: rmse comparison for NASDAQ 100 (USATECHIDXUSD) at

h=4

(left) and

h=24

(right), excluding LSTM. ModernTCN leads; the cross-horizon Spearman

\rho=0.983

(Table 17 ) is the highest among all assets. Mean

\pm

std across three seeds.

Figure 29: Cross-horizon rmse degradation for all nine models. Compare with Figure 8 (no-LSTM variant) for finer inter-model discrimination.

Figure 30: Horizon sensitivity heatmap: percentage rmse degradation ( $\Delta\%=100\times(\text{RMSE}_{24}-\text{RMSE}_{4})/\text{RMSE}_{4}$ ) for eight modern architectures across all twelve assets. LSTM is excluded for visual clarity. Values below 90% (low degradation) appear in cooler colours; values above 150% appear in warmer shades. N-HiTS on Dow Jones achieves the lowest degradation (59.7%); TimesNet on Dow Jones the highest (169.1%).

Figure 31: Global rmse heatmap for all nine architectures (including LSTM) across 24 evaluation points. LSTM’s extreme errors dominate the colour scale, which is why the no-LSTM variant (Figure 3 ) is used as the primary figure in the main body.

Figure 32: Cross-horizon rmse comparison for all nine architectures across twelve assets, including LSTM. The recurrent baseline illustrates the generational performance gap relative to modern time-series architectures.

Panel of Figure 33 — Figure 33: Extended complexity–performance relationship including LSTM. The recurrent baseline lies far from the Pareto frontier defined by modern architectures.

Panel of Figure 34 — Figure 34: Category hierarchical ranking dendrograms for the eight modern architectures (LSTM excluded). Branch lengths encode performance dissimilarity within each asset class. ModernTCN and PatchTST are consistently co-clustered at the top of each dendrogram, confirming their joint dominance is stable across all three asset classes.

Panel of Figure 35 — Figure 35: Category performance matrices for the eight modern architectures (LSTM excluded). Cell intensity encodes normalised rmse relative to the best model per asset (darker = worse). ModernTCN and PatchTST occupy the lightest cells consistently; Autoformer and TimesNet occupy the darkest, confirming the three-tier structure observed in the global leaderboard.

Panel of Figure 36 — Figure 36: Seed-variance box plots at $h=4$ for the three representative assets (LSTM excluded). Each box spans the interquartile range of rmse values across seeds 123, 456, and 789. Negligible box widths relative to inter-model differences confirm that seed accounts for $<0.1\%$ of total forecast variance.

Panel of Figure 37 — Figure 37: Seed-variance box plots at $h=24$ for the three representative assets (LSTM excluded). The pattern mirrors $h=4$ : box widths remain negligible at the longer horizon, confirming H3 holds at both forecast depths.

Panel of Figure 38 — Figure 38: Mean rmse vs. seed variance scatter at $h=4$ for eight modern architectures. Each point represents one model; the horizontal axis shows mean rmse across seeds, and the vertical axis shows the across-seed variance. High-performing models (low mean rmse ) also exhibit low seed variance, confirming that architectural quality and robustness co-vary positively.

Panel of Figure 39 — Figure 39: Mean rmse vs. seed variance scatter at $h=24$ . The positive co-variation between performance and seed stability is maintained at the longer horizon.

Time Series Forecasting

Time series forecasting predicts future values from ordered observations using statistical, neural, and probabilistic models to support decisions under uncertainty.

See all 858 papers

How do standard time series forecasting benchmarks translate to finance? A broad study of risk adjusted performance
Mar 2, 2026Adir Saly-Kaufmann, Kieran Wood, Jan-Peter Calliess +1Financial Time Series Time Series Forecasting
Machine Learning Research Group, Department of Engineering Science, University of Oxford · Oxford-Man Institute of Quantitative Finance, University of Oxford
Forecast Accuracy Is Not Trading Profit: Evolving Small Recurrent Networks for Stock Return Prediction
Oct 6, 2026Jonathan Chang, Zimeng LyuTime Series Forecasting Recurrent Neural Networks
Union County Magnet High School Scotch Plains, New Jersey, USA · Kean University Union, New Jersey, USA
VertiFuseX: Generalizable Financial Forecasting via Multi-Stream Temporal Fusion
Sep 11, 2026Aashish Bohra, Vivek VijayFinancial Forecasting Non-Stationary Time Series Forecasting
Department of Computer Science and Engineering, Indian Institute of Technology Jodhpur, Jodhpur, Rajasthan, India · Department of Mathematics, Indian Institute of Technology Jodhpur, Jodhour, Inida

Mar 2, 2026J/K move · Enter open · S save

How do standard time series forecasting benchmarks translate to finance? A broad study of risk adjusted performance

Adir Saly-Kaufmann, Kieran Wood, Jan-Peter Calliess, Stefan Zohren

Machine Learning Research Group, Department of Engineering Science, University of Oxford · Oxford-Man Institute of Quantitative Finance, University of Oxford

Financial Time Series Time Series Forecasting Forecasting Benchmarks Algorithmic Trading Time Series Representation Learning

Open paper AskPDF arXiv

Forecasting financial asset returns is challenging due to low signal-to-noise ratios, non-stationarity and heavy-tailed distributions. We benchmark 16 deep learning architectures for time series forecasting, including Transformers, State Space Models (SSMs), xLSTMs and recurrent hybrids, on a 15-year daily futures dataset using an end-to-end volatility-targeted Sharpe ratio optimisation task. Our results challenge prevailing assumptions in the literature. Several widely used 'state-of-the-art' architectures, including generic Transformers and SSMs, underperformed a simple AR(1) benchmark in a realistic financial forecasting task. The strongest results came from comparatively simple hybrid approaches using recurrent structures: a Variable Selection Network combined with an LSTM achieved the highest Sharpe ratio, while xLSTM showed the greatest robustness to transaction costs. These findings suggest that success on standard time-series benchmarks does not reliably transfer to finance, and that architectures tailored to the unique properties of asset returns may remain more effective than generic attention-based approaches for financial forecasting.

Abstract

Time Series Forecasting

Explore similar work

How do standard time series forecasting benchmarks translate to finance? A broad study of risk adjusted performance

Forecast Accuracy Is Not Trading Profit: Evolving Small Recurrent Networks for Stock Return Prediction

VertiFuseX: Generalizable Financial Forecasting via Multi-Stream Temporal Fusion