From Fixed Defaults to Calibrated Activation Policies for Long-Term Time-Series Forecasting
Authors: Shijie Xu, Boyan Tang, Kai He, Jianghua Wu
Organizations: School of Automation, Nanjing University of Information Science and Technology · Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen, Guangdong, China · School of Engineering, Hong Kong University of Science and Technology · School of Aerospace, Mechanical and Mechatronic Engineering, The University of Sydney · Shenzhen Loop Area Institute, Shenzhen, Guangdong, China
Long-term time-series forecasting supports decisions in energy, transportation, and other applications where reliable predictions are needed beyond the observed data. Although forecasting research has advanced model architectures, fixed activation configurations provide limited guidance on how activation choices should vary across datasets and prediction horizons. This study aims to establish when task-specific activation choices improve forecasting accuracy and how previous experiments can guide the selection of suitable configurations for new tasks. We develop a calibrated selection policy that learns a reliable default from prior experiments, adapts the activation function and its location to each task, and retains the default when evidence for a change is insufficient. In an evaluation supported by 1,225 Informer training runs across 17 dataset variants and 51 forecasting tasks, the policy reduces mean squared error by 7.86% on average relative to the conventional fixed-activation baseline and improves 72.55% of tasks. These findings establish activation choice and placement as practical opportunities for improving existing forecasting models and show how accumulated experimental evidence can guide configuration decisions without repeating an exhaustive search for every task.
Figures & tables
Figure 1: Raw paired gain distribution for non-GELU activation configurations. Dots are matched seed-run comparisons against gelu_all ; each configuration has 108 raw points and total comparisons are 864; boxes show median/IQR, whiskers use 1.5 IQR, and diamonds mark means
Dataset
L/H
Config.
S1
S2
S3
MSE
MAE
MSE
MAE
MSE
MAE
Informer
ETTh1
48/24
GELU
0.5368
0.5253
0.6383
0.5824
0.5189
0.5164
ETTh1
48/24
ReLU
0.5449
0.5218
0.6683
0.5963
0.5355
0.5223
ETTh1
48/24
tanh
0.4975
0.4986
0.5593
0.5376
0.5423
0.5337
ETTh1
336/720
GELU
1.3769
0.9390
1.4686
0.9883
1.3924
0.9710
Table 1: Representative activation comparisons: individual-seed MSE and MAE (lower is better). Cases illustrate horizon, placement, baseline retention, and backbone effects. Seeds S1–S3 are 2024–2026; a dash denotes an unobserved run. Complete results are reported in Appendix A , Table 7 .
Configuration
Encoder FFNs
Decoder FFNs
Output head
gelu_all
GELU
GELU
linear
relu_all
ReLU
ReLU
linear
swish_all
SiLU/Swish
SiLU/Swish
linear
tanh_all
tanh
tanh
linear
softsign_all
softsign
softsign
linear
tanh_sin001_all
tanh(x)+0.01sin(x)
tanh(x)+0.01sin(x)
linear
Table 2: Orthogonal definition of the nine Informer activation configurations. “Encoder” and “decoder” refer to the nonlinearities between the two convolutions in every corresponding FFN block. Attention projections, temporal embeddings, normalization, and all other modules are unchanged. Output activation is applied after the final linear projection
template
factor condition sketch
action
support / precision
extrap.
bounded-low-horizon-period
low H/L, high dominant-period-to-horizon ratio
bounded regime
28 / 0.89
no
tanh-sin-short-bursty
low H/L, high time-frequency burstiness
tanh-sin
24 / 0.83
no
decoder-tanh-low-volatility
low input volatility, moderate or long H/L
decoder-side tanh
15–17 / 0.60–0.73
no
short-high-dim-GELU-guard
Traffic-like short/mid horizon, high autocorr., low HF noise
abstain to GELU
2 / 1.00
yes
long-horizon-bounded-static
Traffic-like long horizon, high autocorr., low HF noise
bounded static
1 / 0.65
yes
Table 3: Examples of frozen offline-assisted templates
Block
variants
cells
configs
seeds
runs
Complete activation grid
12
36
9
3
972/972
Oracle extension
3
9
7
3
189/189
Held-out stress test
2
6
7
1
42/42
Targeted router check
2
5
targeted
3
33/33
Unique Informer total
17
51
–
–
1,225
Table 4: Evidence scope for calibrated activation-policy evaluation
Figure 2: Forecasting improvements from activation selection. (a) Mean relative MSE reduction for each policy layer. (b) Mean absolute MSE and MAE reductions relative to matched GELU results across the same 51 tasks; higher and rightward indicate larger improvements. The axes are zoomed to separate the five selectors; the GELU reference at (0,0) is outside the displayed range. (c) Cell-bootstrap 95% intervals for relative MSE reduction across four evidence scopes
Table 7: Complete activation results: individual-seed MSE and MAE (lower is better). Seeds S1, S2, and S3 are 2024, 2025, and 2026; a dash denotes an unobserved run.
Dataset
Tasks
Gain vs GELU
Gain vs CF-SBC
Regret
Positive
Config. align.
Family align.
ETTh1
5
+3.35%
-3.57%
6.89%
60.0%
0.0%
80.0%
ETTh2
5
+32.73%
+0.00%
3.74%
100.0%
40.0%
60.0%
ETTm1
5
+6.43%
+0.00%
4.23%
80.0%
20.0%
100.0%
ETTm2
5
+37.12%
+0.00%
6.60%
100.0%
20.0%
60.0%
Electricity
3
+0.24%
+3.62%
0.58%
33.3%
66.7%
100.0%
Exchange
3
-15.50%
+0.00%
17.68%
0.0%
0.0%
100.0%
Appendix
Table 8: Dataset-level results for the balanced policy with CF-SBC fallback.
Table 12: iTransformer mixed-configuration probe: hidden activations and output placement.
Dataset
Rows
Channels
Train
Validation
Test
ETTh1
17,420
7
8,640
2,880
2,880
ETTh2
17,420
7
8,640
2,880
2,880
ETTm1
69,680
7
34,560
11,520
11,520
ETTm2
69,680
7
34,560
11,520
11,520
Solar1
70,176
6
49,123
7,018
14,035
Solar2
70,176
6
49,123
7,018
14,035
Appendix
Table 13: Experimental dataset snapshots. Channels exclude the timestamp. Split sizes count target-region rows; validation and test windows also use the preceding input context.
Dataset
H
L
K
Encoder layers
Decoder layers
Factor
ETTh1
24
48
48
2
1
3
ETTh1
48
96
48
2
1
5
ETTh1
168
168
168
2
1
5
ETTh1
336
168
168
2
1
5
ETTh1
720
336
336
2
1
5
ETTh2
24
48
48
2
1
5
Appendix
Table 14: Informer task-specific settings. H is horizon, L is input length, and K is decoder context length. Other settings are given in Appendix E.3 .
Time series forecasting predicts future values from ordered observations using statistical, neural, and probabilistic models to support decisions under uncertainty.
Financial forecasting models are typically developed in full precision, yet production deployment often requires low-precision inference to reduce memory and computational cost. Post-training quantization (PTQ) enables such deployment without retraining. However, reliable activation quantization requires calibration: activation ranges are estimated from historical data before deployment and then remain fixed during future inference. The importance of this deployment choice for financial forecasting remains poorly understood. We present a systematic study of activation calibration for PTQ in cross-sectional volatility forecasting on the S&P 500. Our evaluation covers seven representative neural architectures, eight walk-forward test years (2018-2025), and 560 trained models. We find that activation calibration has little effect at 8 bits but becomes the primary determinant of predictive performance at 4 bits. Under default absolute-maximum (abs-max) calibration, static 4-bit quantization of both weights and activations removes 11-62% of the full-precision mean information coefficient in affected architectures. Replacing abs-max with percentile calibration recovers 53-94% of this degradation in the four most affected architectures. The preferred activation range also varies across market periods. Narrow ranges improve resolution under typical market conditions but lose part of their advantage when test-period market dispersion exceeds the calibration history. These findings show that activation calibration is a first-class deployment decision for reliable 4-bit PTQ in financial forecasting. When substantial degradation remains, 8-bit activations or weight-only 4-bit quantization provide more robust deployment choices.
Junyi Ye, Ivy Gateri Wanjiku
School of Computing, Montclair State University · Montclair, New Jersey, USA
Time-series foundation models (TSFMs) provide forecasts for operational decisions, but accuracy alone does not determine their value. Evaluating agents that use these models requires measuring decision quality and forecast cost. FWBench evaluates this capability on 1,251 electricity and cycle-hire cases using fixed forecast tools and simulated capacity contracts. Agents select models, histories and horizons, then submit capacities to minimize a stated loss-cost objective. We evaluated two hosted and eight local configurations, including small language models, and tested local models with and without TSFMs. GPT-6 Astra bought inexpensive short-horizon forecasts selectively, using 2.5% of the budget, and outperformed fixed policies when the saved decisions were scored with three loss-cost weightings. FWBench enables reproducible evaluation of how language models select and use time-series forecasts to make decisions under cost constraints.
Time-series models increasingly condition on known future inputs. In controlled systems, some of these inputs are actions that can be changed, compared, and optimized before execution. A model must therefore do more than predict the realized future: it must predict how outcomes change under different actions from the same observed history and support reliable comparison of candidate plans. We introduce ActTime, an action-conditioned world model for time series. Over the observed history, ActTime updates variable-specific latent states through an action-conditioned transition and corrects them using observations. It then rolls the same transition forward under candidate actions to generate future trajectories. Training uses only factual trajectories, without paired alternative-action outcomes or planning supervision. We construct a simulator-grounded evaluation suite across building energy, traffic, greenhouse climate, and inventory management, covering factual prediction, intervention-effect prediction, offline decision making, and closed-loop planning. ActTime ranks first in 12 of 16 domain-task settings. Across eight methods, the mean pairwise model-ranking reversal rate is 17.8% between factual prediction and intervention fidelity, and rises to 28.4% between intervention fidelity and decision ranking. These results show that factual accuracy, intervention fidelity, and decision quality are related but not interchangeable.