From Fixed Defaults to Calibrated Activation Policies for Long-Term Time-Series Forecasting
Authors: Shijie Xu, Boyan Tang, Kai He, Jianghua Wu
Organizations: School of Automation, Nanjing University of Information Science and Technology · Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen, Guangdong, China · School of Engineering, Hong Kong University of Science and Technology · School of Aerospace, Mechanical and Mechatronic Engineering, The University of Sydney · Shenzhen Loop Area Institute, Shenzhen, Guangdong, China
Long-term time-series forecasting supports decisions in energy, transportation, and other applications where reliable predictions are needed beyond the observed data. Although forecasting research has advanced model architectures, fixed activation configurations provide limited guidance on how activation choices should vary across datasets and prediction horizons. This study aims to establish when task-specific activation choices improve forecasting accuracy and how previous experiments can guide the selection of suitable configurations for new tasks. We develop a calibrated selection policy that learns a reliable default from prior experiments, adapts the activation function and its location to each task, and retains the default when evidence for a change is insufficient. In an evaluation supported by 1,225 Informer training runs across 17 dataset variants and 51 forecasting tasks, the policy reduces mean squared error by 7.86% on average relative to the conventional fixed-activation baseline and improves 72.55% of tasks. These findings establish activation choice and placement as practical opportunities for improving existing forecasting models and show how accumulated experimental evidence can guide configuration decisions without repeating an exhaustive search for every task.
Figures & tables
Figure 1: Raw paired gain distribution for non-GELU activation configurations. Dots are matched seed-run comparisons against gelu_all ; each configuration has 108 raw points and total comparisons are 864; boxes show median/IQR, whiskers use 1.5 IQR, and diamonds mark means
Dataset
L/H
Config.
S1
S2
S3
MSE
MAE
MSE
MAE
MSE
MAE
Informer
ETTh1
48/24
GELU
0.5368
0.5253
0.6383
0.5824
0.5189
0.5164
ETTh1
48/24
ReLU
0.5449
0.5218
0.6683
0.5963
0.5355
0.5223
ETTh1
48/24
tanh
0.4975
0.4986
0.5593
0.5376
0.5423
0.5337
ETTh1
336/720
GELU
1.3769
0.9390
1.4686
0.9883
1.3924
0.9710
Table 1: Representative activation comparisons: individual-seed MSE and MAE (lower is better). Cases illustrate horizon, placement, baseline retention, and backbone effects. Seeds S1–S3 are 2024–2026; a dash denotes an unobserved run. Complete results are reported in Appendix A , Table 7 .
Configuration
Encoder FFNs
Decoder FFNs
Output head
gelu_all
GELU
GELU
linear
relu_all
ReLU
ReLU
linear
swish_all
SiLU/Swish
SiLU/Swish
linear
tanh_all
tanh
tanh
linear
softsign_all
softsign
softsign
linear
tanh_sin001_all
tanh(x)+0.01sin(x)
tanh(x)+0.01sin(x)
linear
Table 2: Orthogonal definition of the nine Informer activation configurations. “Encoder” and “decoder” refer to the nonlinearities between the two convolutions in every corresponding FFN block. Attention projections, temporal embeddings, normalization, and all other modules are unchanged. Output activation is applied after the final linear projection
template
factor condition sketch
action
support / precision
extrap.
bounded-low-horizon-period
low H/L, high dominant-period-to-horizon ratio
bounded regime
28 / 0.89
no
tanh-sin-short-bursty
low H/L, high time-frequency burstiness
tanh-sin
24 / 0.83
no
decoder-tanh-low-volatility
low input volatility, moderate or long H/L
decoder-side tanh
15–17 / 0.60–0.73
no
short-high-dim-GELU-guard
Traffic-like short/mid horizon, high autocorr., low HF noise
abstain to GELU
2 / 1.00
yes
long-horizon-bounded-static
Traffic-like long horizon, high autocorr., low HF noise
bounded static
1 / 0.65
yes
Table 3: Examples of frozen offline-assisted templates
Block
variants
cells
configs
seeds
runs
Complete activation grid
12
36
9
3
972/972
Oracle extension
3
9
7
3
189/189
Held-out stress test
2
6
7
1
42/42
Targeted router check
2
5
targeted
3
33/33
Unique Informer total
17
51
–
–
1,225
Table 4: Evidence scope for calibrated activation-policy evaluation
Figure 2: Forecasting improvements from activation selection. (a) Mean relative MSE reduction for each policy layer. (b) Mean absolute MSE and MAE reductions relative to matched GELU results across the same 51 tasks; higher and rightward indicate larger improvements. The axes are zoomed to separate the five selectors; the GELU reference at (0,0) is outside the displayed range. (c) Cell-bootstrap 95% intervals for relative MSE reduction across four evidence scopes
Table 7: Complete activation results: individual-seed MSE and MAE (lower is better). Seeds S1, S2, and S3 are 2024, 2025, and 2026; a dash denotes an unobserved run.
Dataset
Tasks
Gain vs GELU
Gain vs CF-SBC
Regret
Positive
Config. align.
Family align.
ETTh1
5
+3.35%
-3.57%
6.89%
60.0%
0.0%
80.0%
ETTh2
5
+32.73%
+0.00%
3.74%
100.0%
40.0%
60.0%
ETTm1
5
+6.43%
+0.00%
4.23%
80.0%
20.0%
100.0%
ETTm2
5
+37.12%
+0.00%
6.60%
100.0%
20.0%
60.0%
Electricity
3
+0.24%
+3.62%
0.58%
33.3%
66.7%
100.0%
Exchange
3
-15.50%
+0.00%
17.68%
0.0%
0.0%
100.0%
Appendix
Table 8: Dataset-level results for the balanced policy with CF-SBC fallback.
Table 12: iTransformer mixed-configuration probe: hidden activations and output placement.
Dataset
Rows
Channels
Train
Validation
Test
ETTh1
17,420
7
8,640
2,880
2,880
ETTh2
17,420
7
8,640
2,880
2,880
ETTm1
69,680
7
34,560
11,520
11,520
ETTm2
69,680
7
34,560
11,520
11,520
Solar1
70,176
6
49,123
7,018
14,035
Solar2
70,176
6
49,123
7,018
14,035
Appendix
Table 13: Experimental dataset snapshots. Channels exclude the timestamp. Split sizes count target-region rows; validation and test windows also use the preceding input context.
Dataset
H
L
K
Encoder layers
Decoder layers
Factor
ETTh1
24
48
48
2
1
3
ETTh1
48
96
48
2
1
5
ETTh1
168
168
168
2
1
5
ETTh1
336
168
168
2
1
5
ETTh1
720
336
336
2
1
5
ETTh2
24
48
48
2
1
5
Appendix
Table 14: Informer task-specific settings. H is horizon, L is input length, and K is decoder context length. Other settings are given in Appendix E.3 .
Time series forecasting predicts future values from ordered observations using statistical, neural, and probabilistic models to support decisions under uncertainty.