Under temporal nonstationarity, model performance rankings reverse across training horizons due to heterogeneous sensitivities to memory. While existing literature addresses adaptive lookbacks, effective model complexity, and empirical forecast evaluation in isolation, no unified framework explains the structural mechanics driving these rank reversals. We bridge this gap by deriving a model-agnostic decomposition of out-of-sample R2 into target alignment, miscalibrated aggressiveness, and predictive instability. Within this framework, we mathematically characterize complexity phase boundaries, establish closed-form optimal-memory equations for ridge regression under local drift, and extend these guarantees to complex model architectures via surrogate bounds. We validate these theoretical dynamics by benchmarking 14 linear, tree-based, and neural architectures across large-scale financial panels. In doing so, our framework demonstrates that empirical model rank reversals are driven by predictable surges in miscalibrated aggressiveness, providing a rigorous mechanistic diagnosis of generalization under temporal shift.
Figures & tables
Figure 1: Theoretical leading-order dynamics with normalized parameters. (A) The complex class achieves lower risk when its history-independent advantage Δ0 exceeds the complexity cost ϕ(h) ; bands denote remainder bounds. (B) Variance V/h decays and the drift penalty Wh2 grows with memory h , creating an interior risk minimum at h∗ .
Tabular
Temporal, 60-day context
Ridge, α=1
Tuned ridge
Random forest
XGBoost
Small MLP
Medium MLP
Flattened ridge
Flattened XGBoost
Fixed flattened MLP
Tuned flattened MLP
Vanilla RNN
LSTM
Small Transformer
Mean R2↑
16m
− 0.0909
0.3515
0.3549
0.3316
0.2869
0.2786
− 0.0772
0.2812
− 0.8216
− 0.2223
0.2776
0.1913
0.2712
64m
− 0.3256
0.3668
0.3374
0.3419
0.3231
0.3189
− 0.1078
0.3223
− 0.0124
− 0.0002
0.2989
0.2472
0.2969
Exp.
− 0.3386
0.3602
0.2953
0.3232
0.3301
0.3270
− 0.2835
0.3148
− 0.0881
− 0.1158
0.3169
0.2885
0.3195
Ensemble R2↑
Table 1: Memory-induced rank reversals and the empirical recovery of predictive instability. Expanding the training horizon (16m to Exp.) systematically inverts model rankings, favoring heavily regularized or temporally structured architectures over static, complex baselines at longer histories. Additionally, replicate ensembling strictly improves upon mean predictive performance, quantifying the exact out-of-sample cost of finite-sample instability. Shading identifies the optimal architecture within the tabular and 60-day temporal branches; RMSE is reported in percentage points of daily excess returns.
Context
252d
756d
Exp.
Tabular
−0.00189
−0.00145
−0.00129
Temporal 20d
−0.00198
−0.00132
−0.00076
Temporal 60d
−0.00199
−0.00135
−0.00096
Table 2: Mean pooled out-of-sample R2 against a zero-return baseline (January 2021–December 2024).
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
Window
Months
Window days
Subsampled days
Network fit days
Validation days
16 months
16
328–340
256–265
192–199
79–87
64 months
28–64
590–1,352
460–1,052
263–856
246–255
Expanding
28–350
590–7,353
460–5,718
263–5,522
246–255
Appendix
Table 3: Trading days per fit in the tabular branch, from the minimum to the maximum over the 323 forecast months. Window days are the days of the window before subsampling. Network fit days are the subsampled days before the validation block, which train the networks. The counts do not depend on the seed. A temporal fit has L−1 fewer window days.
Model
Input
Fixed settings
Candidate grid (eight candidates)
Tabular branch
Ridge, fixed alpha
xt
α=1
none
Tuned ridge
xt
none
α∈{10−4,10−2,0.1,1,10,100,104,106}
XGBoost
xt
150 rounds; min. child weight 5; L2 penalty 1; no subsampling
D∈{2,3,5,7} and η∈{0.03,0.1}
Random forest
xt
cuML; 200 bootstrap trees; depth 5; min. leaf 1; all features; 128 bins
none
Small MLP
xt
width (32); 4,193 parameters; batch 256; at most 30 epochs; patience 5
Table 4: Model specifications. The input is the predictor vector xt , the flattened context (flat), or the ordered context. Each tuned model has eight candidates, which are all combinations of the listed values. Parameter counts are trainable parameters per industry. When a cell lists two counts, they belong to the two grid values in the listed order. All networks use AdamW and clip the gradient norm at 1.0.
Model
16 months
64 months
Expanding
Ridge, fixed alpha
Ensemble
0.0706
0.3481
0.3477
Mean [95% CI]
− 0.0909 [ − 0.1033, − 0.0786]
0.3256 [0.3235, 0.3277]
0.3386 [0.3374, 0.3397]
Tuned ridge
Ensemble
0.3671
0.3767
0.3644
Mean [95% CI]
0.3515 [0.3487, 0.3543]
0.3668 [0.3658, 0.3678]
0.3602 [0.3593, 0.3611]
XGBoost
Ensemble
0.3621
0.3619
0.3392
Mean [95% CI]
0.3316 [0.3284, 0.3348]
0.3419 [0.3380, 0.3457]
0.3232 [0.3202, 0.3262]
Appendix
Table 5: Tabular: pooled out-of-sample R2 for every reported model and training window. In each cell, the upper line gives the R2 of the six-seed ensemble, which scores the mean forecast. The lower line gives the mean of the six seed scores, with its seed-only 95% t interval in brackets.
Figure 10: Tabular, Ridge, fixed alpha: monthly pooled R2 for the 17 industries, one curve each, under the 16-month, 64-month, and expanding windows. Values below −2.0 fall outside the frame: 393, 83, and 90 of 5,491 in the three panels.
Figure 11: Tabular, Tuned ridge: monthly pooled R2 for the 17 industries, one curve each, under the 16-month, 64-month, and expanding windows. Values below −2.0 fall outside the frame: 2, 51, and 52 of 5,491 in the three panels.
Figure 12: Tabular, XGBoost: monthly pooled R2 for the 17 industries, one curve each, under the 16-month, 64-month, and expanding windows. Values below −2.0 fall outside the frame: 0, 36, and 38 of 5,491 in the three panels.
Figure 13: Tabular, Small MLP: monthly pooled R2 for the 17 industries, one curve each, under the 16-month, 64-month, and expanding windows. Values below −2.0 fall outside the frame: 1, 2, and 10 of 5,491 in the three panels.
Figure 14: Tabular, Medium MLP: monthly pooled R2 for the 17 industries, one curve each, under the 16-month, 64-month, and expanding windows. Values below −2.0 fall outside the frame: 6, 1, and 7 of 5,491 in the three panels.
Figure 15: Tabular, Random forest: monthly pooled R2 for the 17 industries, one curve each, under the 16-month, 64-month, and expanding windows. Values below −2.0 fall outside the frame: 2, 55, and 63 of 5,491 in the three panels.
Model
16 months
64 months
Expanding
Flattened ridge
Ensemble
0.1915
− 0.0139
− 0.3308
Mean [95% CI]
0.1397 [0.1346, 0.1447]
− 0.2128 [ − 0.2208, − 0.2049]
− 1.1533 [ − 1.1871, − 1.1194]
Flattened XGBoost
Ensemble
0.3341
0.3485
0.3379
Mean [95% CI]
0.3026 [0.2992, 0.3061]
0.3287 [0.3258, 0.3315]
0.3208 [0.3170, 0.3247]
Small flattened MLP
Ensemble
0.0200
0.1544
0.2485
Mean [95% CI]
− 0.0841 [ − 0.0926, − 0.0756]
0.0574 [0.0532, 0.0617]
0.1623 [0.1552, 0.1694]
Appendix
Table 6: Temporal, 20-day context: pooled out-of-sample R2 for every completed model and training window. In each cell, the upper line gives the R2 of the six-seed ensemble, which scores the mean forecast. The lower line gives the mean of the six seed scores, with its seed-only 95% t interval in brackets.
Figure 16: Temporal, 20-day context: monthly pooled R2 of every completed model, one curve each, under the 16-month, 64-month, and expanding windows. Each month pools all 17 industries and the trading days of that month. Values below −2.0 fall outside the frame: 3, 11, and 112 of 2,261 model-months in the three panels.
Figure 17: Temporal, 20-day context, Flattened ridge: monthly pooled R2 for the 17 industries, one curve each, under the 16-month, 64-month, and expanding windows. Values below −2.0 fall outside the frame: 14, 552, and 1,962 of 5,491 in the three panels.
Figure 18: Temporal, 20-day context, Flattened XGBoost: monthly pooled R2 for the 17 industries, one curve each, under the 16-month, 64-month, and expanding windows. Values below −2.0 fall outside the frame: 0, 29, and 29 of 5,491 in the three panels.
Figure 19: Temporal, 20-day context, Small flattened MLP: monthly pooled R2 for the 17 industries, one curve each, under the 16-month, 64-month, and expanding windows. Values below −2.0 fall outside the frame: 31, 46, and 42 of 5,491 in the three panels.
Figure 20: Temporal, 20-day context, Tuned flattened MLP: monthly pooled R2 for the 17 industries, one curve each, under the 16-month, 64-month, and expanding windows. Values below −2.0 fall outside the frame: 23, 2, and 9 of 5,491 in the three panels.
Figure 21: Temporal, 20-day context, Vanilla RNN: monthly pooled R2 for the 17 industries, one curve each, under the 16-month, 64-month, and expanding windows. Values below −2.0 fall outside the frame: 0, 9, and 9 of 5,491 in the three panels.
Figure 22: Temporal, 20-day context, LSTM: monthly pooled R2 for the 17 industries, one curve each, under the 16-month, 64-month, and expanding windows. Values below −2.0 fall outside the frame: 2, 6, and 4 of 5,491 in the three panels.
Figure 23: Temporal, 20-day context, Small Transformer: monthly pooled R2 for the 17 industries, one curve each, under the 16-month, 64-month, and expanding windows. Values below −2.0 fall outside the frame: 2, 21, and 22 of 5,491 in the three panels.
Model
16 months
64 months
Expanding
Flattened ridge
Ensemble
0.1092
0.1654
− 0.0295
Mean [95% CI]
0.0772 [0.0712, 0.0831]
0.1078 [0.1018, 0.1138]
− 0.2835 [ − 0.2946, − 0.2724]
Flattened XGBoost
Ensemble
0.3161
0.3402
0.3253
Mean [95% CI]
0.2812 [0.2779, 0.2845]
0.3223 [0.3188, 0.3257]
0.3148 [0.3132, 0.3164]
Small flattened MLP
Ensemble
− 0.3891
0.0765
0.1690
Mean [95% CI]
− 0.8216 [ − 0.8845, − 0.7588]
− 0.0124 [ − 0.0180, − 0.0068]
0.0881 [0.0846, 0.0916]
Appendix
Table 7: Temporal, 60-day context: pooled out-of-sample R2 for every completed model and training window. In each cell, the upper line gives the R2 of the six-seed ensemble, which scores the mean forecast. The lower line gives the mean of the six seed scores, with its seed-only 95% t interval in brackets.
Figure 24: Temporal, 60-day context: monthly pooled R2 of every completed model, one curve each, under the 16-month, 64-month, and expanding windows. Each month pools all 17 industries and the trading days of that month. Values below −2.0 fall outside the frame: 32, 0, and 27 of 2,261 model-months in the three panels.
Figure 25: Temporal, 60-day context, Flattened ridge: monthly pooled R2 for the 17 industries, one curve each, under the 16-month, 64-month, and expanding windows. Values below −2.0 fall outside the frame: 1, 48, and 685 of 5,491 in the three panels.
Figure 26: Temporal, 60-day context, Flattened XGBoost: monthly pooled R2 for the 17 industries, one curve each, under the 16-month, 64-month, and expanding windows. Values below −2.0 fall outside the frame: 0, 5, and 5 of 5,491 in the three panels.
Figure 27: Temporal, 60-day context, Small flattened MLP: monthly pooled R2 for the 17 industries, one curve each, under the 16-month, 64-month, and expanding windows. Values below −2.0 fall outside the frame: 673, 55, and 53 of 5,491 in the three panels.
Figure 28: Temporal, 60-day context, Tuned flattened MLP: monthly pooled R2 for the 17 industries, one curve each, under the 16-month, 64-month, and expanding windows. Values below −2.0 fall outside the frame: 88, 29, and 32 of 5,491 in the three panels.
Figure 29: Temporal, 60-day context, Vanilla RNN: monthly pooled R2 for the 17 industries, one curve each, under the 16-month, 64-month, and expanding windows. Values below −2.0 fall outside the frame: 2, 4, and 5 of 5,491 in the three panels.
Figure 30: Temporal, 60-day context, LSTM: monthly pooled R2 for the 17 industries, one curve each, under the 16-month, 64-month, and expanding windows. Values below −2.0 fall outside the frame: 1, 4, and 3 of 5,491 in the three panels.
Figure 31: Temporal, 60-day context, Small Transformer: monthly pooled R2 for the 17 industries, one curve each, under the 16-month, 64-month, and expanding windows. Values below −2.0 fall outside the frame: 1, 6, and 6 of 5,491 in the three panels.
Model
Window
Mean R2
SD
CI low
CI high
Ensemble
2ag
g2
u
Tabular
Ridge, fixed alpha
16m
− 0.0909
0.0118
− 0.1033
− 0.0786
0.0706
0.9726
0.9019
0.1615
Ridge, fixed alpha
64m
0.3256
0.0020
0.3235
0.3277
0.3481
0.9373
0.5892
0.0225
Ridge, fixed alpha
Exp.
0.3386
0.0011
0.3374
0.3397
0.3477
0.7620
0.4143
0.0091
Tuned ridge
16m
0.3515
0.0027
0.3487
0.3543
0.3671
0.8296
0.4625
0.0156
Tuned ridge
64m
0.3668
0.0010
0.3658
0.3678
0.3767
0.8571
0.4804
0.0099
Appendix
Table 8: All 60 reported Stage 5 model-window-context cells. R2 is pooled against zero over 6,784 dates and 17 industries. The interval uses six seed scores; ensemble R2 scores their mean forecast. The final three columns obey R2=2ag−g2−u up to rounding.
When forecasting hourly returns for 1,000 US equities, we observe an unexpected phenomenon: predictions become nearly flat and show poor stock ranking, as measured by cross-sectional correlation. We call this forecast collapse. Surprisingly, the phenomenon largely disappears when forecasting trading volume under the same setting. We investigate forecast collapse across time-series foundation models (TSFMs), twelve deep-learning forecasting models, and 97 public benchmark configurations, and find that it is closely tied to target predictability. We identify two distinct reasons behind it: low predictability limits the amplitude of calibrated point forecasts, while per-series objectives leave cross-series structure unidentified. These findings reveal a calibration-ranking tradeoff: optimizing squared error leads to flat predictions, whereas directly optimizing cross-sectional correlation improves ranking but can inflate forecast amplitude by more than an order of magnitude. To address this tradeoff, we introduce CalibRank, a simple objective that balances calibration and ranking. On Finance1K, CalibRank nearly triples cross-sectional correlation while keeping amplitude close to the target, and improves correlation on all tested models. Our results reveal a blind spot in conventional time-series evaluation: per-series metrics can hide failures in cross-series structure needed by downstream decisions.
Shu Wan, Miles Ma, Hank Zhu +4
Abel AI Lab · Arizona State University · University of Oxford
Real-world data distributions evolve over time, inducing temporal distribution shift that can substantially degrade the reliability of deployed machine learning systems. However, the extent to which architectural choices and their associated inductive biases affect temporal robustness remains insufficiently understood. We present a systematic empirical comparison of temporal robustness across three heterogeneous, time-indexed domains encompassing image classification, multi-label text classification, and text regression tasks. Using a unified evaluation framework based on temporal drift matrices, we train models on cumulative historical data and evaluate their performance on both earlier and later time periods, thereby quantifying cross-temporal generalization. Our study spans model families ranging from simple multilayer perceptrons and convolutional networks to recurrent networks and pretrained Transformer-based encoders. Collectively, the results show that architectural inductive biases systematically shape temporal robustness: models whose inductive biases lead them to exploit localized, highly discriminative features attain the highest in-distribution accuracy, yet those features are often the ones that change most over time, so these models degrade fastest, while pretrained encoders that draw on coarser, more stable representations drift more gradually. These observations offer practical guidance for selecting architectures for real-world systems subject to temporal drift.
Robin Holzinger, Riccardo Colletti
Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, USA
Evaluating robustness under temporal distribution shift remains an open challenge. Existing metrics quantify the average decline in performance, but fail to capture how models adapt to evolving data. As a result, temporal degradation is often misinterpreted: when accuracy declines, it is unclear whether the model is failing to adapt or whether the data itself has become inherently more challenging to learn. In this work, we propose three complementary metrics to distinguish adaptation from intrinsic difficulty in the data. Together, these metrics provide a dynamic and interpretable view of model behavior under temporal distribution shift. Results show that our metrics uncover adaptation patterns hidden by existing analysis, offering a richer understanding of temporal robustness in evolving environments.
Lorenzo Iovine, Giacomo Ziffer, Emanuele Della Valle
Politecnico di Milano, DEIB, Via Giuseppe Ponzio, 34, 20133 Milan, Italy