Time-series forecasting informs critical decisions in energy dispatch, industrial operations, and environmental monitoring; understanding the patterns models rely on is essential for assessing reliability and identifying failures. Input attribution identifies important variables and time segments but offers limited insight into internal features, while standard sparse autoencoder (SAE) objectives do not directly constrain cross-variable structure or temporal continuity. We introduce TSAE, a structured sparse autoencoder for forecasting representations that decomposes hidden states into individually inspectable features. TSAE organizes cross-variable structure through shared and variable-routed private dictionaries, separates feature detection from magnitude estimation with gated encoding, and constrains neighboring sparse-code changes according to raw-segment similarity. These mechanisms support analysis of variable context, activation strength, and temporal evolution. Forecast-consistency fine-tuning further improves preservation of the frozen forecaster's outputs. The accompanying TSEVAL protocol separately audits fidelity, feature coherence, and physical calibration to ground feature interpretation. In three-seed experiments with frozen PatchTST on ETTh1, ETTh2, and ETTm1, TSAE achieves the lowest hidden-state reconstruction error, normalized forecast-reconstruction error (NFRE), and feature transition rate among five SAEs at comparable per-token activity. NFRE decreases by 4.3-26.4% relative to the next-best mean. Dataset-dependent tradeoffs in selectivity, physical correlation, and calibration show that fidelity and temporal-stability gains require independent semantic validation to support feature interpretation.
Figures & tables
Figure 1: Three time-series properties motivate TSAE : (a) shared motifs and variable-specific structure motivate shared/private routing; (b) motif strength motivates separating detection from magnitude; (c) similar adjacent patches of one variable motivate code continuity, with weaker coupling across real changes. Traces are schematic.
Figure 2: TSAE model framework. Raw time-series inputs are first encoded by a forecasting backbone into hidden tokens. For each token, the TSAE encoder produces shared and variable-routed private gated sparse codes, and the decoder reconstructs the hidden token from the corresponding shared and private dictionaries. Conditional temporal continuity is applied to adjacent sparse codes, with the regularization strength determined by the similarity of their raw temporal segments.
Diagnostic
Question
Fidelity
Does the sparse reconstruction preserve the original hidden state?
Coherence
Are activations concentrated in coherent variable and temporal contexts?
Calibration
Do feature activations track independently measurable signal motifs?
Forecast sensitivity
How sensitive is the forecast to a specified sparse-feature intervention?
Hidden control
Is the effect specific to the SAE decoder path rather than a generic hidden-space perturbation?
Input baselines
Do hidden sparse features provide information beyond saliency and occlusion?
Table 1: The TSEval diagnostic checklist. Each axis tests a claim that reconstruction-only evaluation cannot adjudicate. In TSEval , a feature is interpreted only with respect to the claims supported by the corresponding diagnostics.
Setting
Operating point
Evaluation metrics
Data
Method
Active
Recon. ↓
NFRE ↓
Dead ↓
Trans. ↓
Sel. ↑
Phys. ↑
Calib. ↓
ETTh1
TSAE
231.4 ± 2.7
13.44 ± 0.57
5.10 ± 0.25
0.080 ± 0.025
0.073 ± 0.005
0.639 ± 0.007
0.541 ± 0.106
0.624 ± 0.049
L1
252.8 ± 10.3
13.59 ± 0.45
5.33 ± 0.27
0.000 ± 0.000
0.181 ± 0.006
0.570 ± 0.023
0.653 ± 0.047
0.567 ± 0.022
Gated
244.4 ± 7.4
16.39 ± 0.75
6.39 ± 0.22
0.000 ± 0.000
0.144 ± 0.005
0.573 ± 0.020
0.523 ± 0.060
0.681 ± 0.024
TopK
264.9 ± 5.7
30.54 ± 2.00
10.84 ± 0.75
0.001 ± 0.001
0.143 ± 0.002
0.592 ± 0.007
0.622 ± 0.047
0.587 ± 0.043
JumpReLU
269.8 ± 11.4
20.69 ± 0.88
8.09 ± 0.37
0.000 ± 0.000
0.156 ± 0.007
0.612 ± 0.066
0.587 ± 0.017
0.657 ± 0.002
Table 2: Validation-selected PatchTST results on four datasets. Each entry is the mean ± standard deviation over three independently trained SAE seeds. Operating points are selected only on validation data to target 256 active features per token (admissible range: 192–320); all reported metrics are test-split measurements. Bold and underline mark the best and second-best mean, respectively, within a dataset and metric. Recon. and NFRE are multiplied by 103 .
Figure 3: PatchTST–ETTh1 validation activity–fidelity curves. Each point is a validation calibration run; the horizontal axis reports measured active features per token. Table 2 reports separate test-split metrics.
Backbone
Method
Active
Recon. ↓
NFRE ↓
PatchTST
TSAE
231.4 ± 2.7
13.44 ± 0.57
5.10 ± 0.25
Gated
244.4 ± 7.4
16.39 ± 0.75
6.39 ± 0.22
L1
252.8 ± 10.3
13.59 ± 0.45
5.33 ± 0.27
TopK
264.9 ± 5.7
30.54 ± 2.00
10.84 ± 0.75
JumpReLU
269.8 ± 11.4
20.69 ± 0.88
8.09 ± 0.37
iTransformer
TSAE
253.1 ± 2.5
14.48 ± 0.23
3.01 ± 0.06
Table 3: Cross-backbone active-matched comparison (ETTh1, three-seed means). Bold and underline mark the best and second-best means, respectively, within each backbone and metric. Recon. and NFRE are ×103 . Full tables and validation sweeps are in Appendix C.4 .
Data
Variant
Active
Recon. ↓
NFRE ↓
Dead
Trans.
ETTh1
Full
268
11.7
4.9
6.3
0.086
No routing
252
13.4
5.5
0.0
0.145
No gating
108
18.1
7.9
0.0
0.083
No temporal
279
12.3
5.1
6.3
0.091
ETTh2
Full
203
2.2
3.4
10.9
0.036
No routing
178
2.4
3.6
13.1
0.050
Table 4: PatchTST component ablation (three-seed means). Bold marks the lowest reconstruction and NFRE values within each dataset. Dead-feature ratio and transition rate are reported as diagnostic tradeoffs. Recon. and NFRE use the report scale ( ×103 ).
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Multi-feature case atlas from the formal PatchTST–ETTh1 run. Blue curves are non-overlapping high-activation patches and red curves are non-overlapping low-activation controls. The first row shows shared atoms that recur across variables; the second row shows private atoms, each restricted to its routed variable. The panel is illustrative; the full pre-recorded 12-atom atlas and selection manifest are stored in the artifact. In-panel annotations are small and are best viewed zoomed in the electronic PDF.
Figure 5: Feature-level evidence beyond reconstruction. (a) ETTh1 activation as a function of the associated raw physical metric; bars show 95% confidence intervals within equal-frequency bins. (b) Mean forecast shift after deleting a selected physical-pattern feature (blue) or an activation-rate-matched, lower-correlation control (gray), while the PatchTST backbone remains frozen. The intervention measures sensitivity within the model pathway, not a causal effect in the data-generating system.
Dataset
L1 SAE Jaccard
TSAE Jaccard
Difference
ETTh1
0.184
0.257
+0.073
ETTh2
0.456
0.676
+0.220
ETTm1
0.424
0.540
+0.116
Appendix
Table 5: Adjacent-patch support stability at an identical Top- 24 feature budget per token. Values are descriptive means over adjacent test-patch pairs for the seed-2024 formal checkpoints.
Dataset
Method
Selected setting
Test Active
ETTh1
TSAE
shared λ=6.0 , private λ=0.30
231.4 ± 2.7
L1
λ=2.0
252.8 ± 10.3
Gated
λ=6.0
244.4 ± 7.4
TopK
k=320
264.9 ± 5.7
JumpReLU
λ=10−4 , threshold =0.10
269.8 ± 11.4
ETTh2
TSAE
shared λ=3.0 , private λ=0.15
250.9 ± 13.6
Appendix
Table 6: Validation-selected operating points with test-split active counts. TSAE runs use λtemp=0.5 , λfc=0.1 , and λbalance=0.003 .
λtemp
Active
Recon. ↓
NFRE ↓
Trans. ↓
0
280
12.3
5.1
0.0908
0.0001
280
12.3
5.1
0.0909
0.001
280
12.3
5.1
0.0908
0.01
278
12.3
5.1
0.0901
0.05
276
12.2
5.1
0.0903
0.1
273
12.2
5.1
0.0895
Appendix
Table 7: Temporal coefficient sweep (single seed unless noted). The bold row is the validation-selected formal coefficient. Recon. and NFRE are ×103 .
λpriv
Active
Recon. ↓
NFRE ↓
Dead%
0.02
500
10.7
4.4
6.2
0.05
489
9.4
3.8
2.3
0.1
456
9.5
3.9
0.6
0.2
416
10.2
4.2
1.0
0.5
392
11.0
4.4
7.0
1.0
386
11.3
4.5
13.8
Appendix
Table 8: Private sparsity sensitivity at a fixed shared coefficient (single seed). Bold marks the illustrative low-dead-feature range; formal comparisons use dataset-specific validation-selected values.
Figure 6: Validation calibration curves on PatchTST–ETTh1. Horizontal axis: sparsity parameter; vertical axis: measured active features per token. Gray band: admissible [192,320] window.
Method
Active
In 192–320?
Recon. ↓
NFRE ↓
Dead%
Trans. ↓
TSAE
231 ± 3
✓
13.4
5.10
8.0
0.073
L1
253 ± 10
✓
13.6
5.33
0.0
0.181
Gated
244 ± 7
✓
16.4
6.39
0.0
0.144
TopK
265 ± 6
✓
30.5
10.84
0.1
0.143
JumpReLU
270 ± 11
✓
20.7
8.09
0.0
0.156
Appendix
Table 9: ETTh1: all five methods within the [192,320] window. TSAE has the best Recon, NFRE, and Transition rate.
Method
Active
In 192–320?
Recon. ↓
NFRE ↓
Dead%
Trans. ↓
TSAE
251 ± 14
✓
6.04
2.70
4.3
0.050
L1
195 ± 11
✓
7.99
3.66
2.5
0.106
Gated
213 ± 6
✓
8.19
3.76
10.3
0.055
TopK
218 ± 1
✓
9.88
4.56
14.3
0.063
JumpReLU
311 ± 40
✓
10.9
4.79
4.1
0.064
Appendix
Table 10: ETTh2: all five methods are within the [192,320] window. TSAE is the best on Recon, NFRE, and Transition.
Method
Active
In 192–320?
Recon. ↓
NFRE ↓
Dead%
Trans. ↓
TSAE
205 ± 12
✓
10.4
5.00
3.1
0.047
L1
258 ± 12
✓
11.4
5.50
0.0
0.135
Gated
247 ± 6
✓
13.5
6.58
0.0
0.089
TopK
224 ± 2
✓
19.9
8.02
0.6
0.083
JumpReLU
270 ± 13
✓
17.1
8.63
0.0
0.092
Appendix
Table 11: ETTm1: all five methods within the [192,320] window. TSAE has the best Recon, NFRE, and Transition rate.
Method
Active
In 192–320?
Recon. ↓
NFRE ↓
Dead%
Trans. ↓
TSAE
217 ± 3
✓
1.70
2.52
19.5
0.019
L1
188 ± 16
×
1.85
3.37
0.0
0.055
Gated
285 ± 9
✓
2.10
2.84
0.0
0.046
TopK
261 ± 14
✓
3.84
4.64
7.3
0.038
JumpReLU
236 ± 1
✓
4.49
6.21
0.1
0.031
Appendix
Table 12: Weather: the validation-selected settings yield in-window test means for TSAE, Gated, TopK, and JumpReLU. L1 is slightly below the boundary at 188, despite its validation point lying in-window. TSAE has the best Recon, NFRE, and Transition.
Method
Active
Recon. ↓
NFRE ↓
TSAE
253 ± 3
14.5
3.01
L1
282 ± 1
18.1
4.14
Gated
316 ± 1
17.3
3.20
TopK
212 ± 0
30.1
5.24
JumpReLU
269 ± 3
18.8
3.44
Appendix
Table 13: iTransformer–ETTh1 active-matched results (three-seed means). Recon. and NFRE are ×103 .
Method
Active
Recon. ↓
NFRE ↓
TSAE
215 ± 10
9.09
7.91
L1
250 ± 1
9.17
8.74
Gated
314 ± 1
7.71
6.88
TopK
256 ± 0
9.90
8.27
JumpReLU
266 ± 4
7.99
7.02
Appendix
Table 14: DLinear–ETTh1 active-matched results (three-seed means). Recon. and NFRE are ×103 .
Data
Variant
Active
Recon. ↓
NFRE ↓
Dead%
Trans. ↓
ETTh1
Full
268
11.7
4.9
6.3
0.086
No routing
252
13.4
5.5
0.0
0.145
No gating
108
18.1
7.9
0.0
0.083
No temporal
279
12.3
5.1
6.3
0.091
ETTh2
Full
203
2.2
3.4
10.9
0.036
No routing
178
2.4
3.6
13.1
0.050
Appendix
Table 15: PatchTST component ablation (three-seed means). Bold marks the best per column per dataset.
Data
Variant
Active
Recon. ↓
NFRE ↓
Dead%
ETTh1
Full
399
13.7
3.3
0.0
No routing
309
23.3
4.3
0.0
No gating
188
34.7
7.8
0.0
ETTh2
Full
360
22.9
1.1
0.0
No routing
284
31.6
1.4
0.0
No gating
187
45.7
1.9
0.0
Appendix
Table 16: iTransformer component ablation. Transition rate not applicable.
Explainable time series forecasting makes predictions over time interpretable through attributions, structured models, causal reasoning, and counterfactuals.
Time series foundation models (TSFMs) are increasingly deployed in high-stakes domains, yet their internal representations remain opaque. We present the first application of sparse autoencoders (SAEs) to a TSFM, training TopK SAEs on activations of Chronos-T5-Large (710M parameters) across six layers. Through 392 single-feature ablation experiments, we establish that every ablated feature produces a positive CRPS degradation, confirming causal relevance. Our analysis reveals a depth-dependent hierarchy: early encoder layers encode low-level frequency features, the mid-encoder concentrates causally critical change-detection features, and the final encoder compresses a rich but less causally important taxonomy of temporal concepts. The most critical features reside in the mid-encoder (max single-feature Delta CRPS = 38.61), not in the semantically richest final encoder layer, where progressive ablation paradoxically improves forecast quality. These findings demonstrate that mechanistic interpretability transfers effectively to TSFMs and that Chronos-T5 relies on abrupt-dynamics detection rather than periodic pattern recognition.
Transformer architectures have been widely adopted for time series forecasting, yet whether the representational mechanisms that make them powerful in NLP actually engage on time series data remains unexplored. The persistent competitiveness of simple linear models such as DLinear has fueled ongoing debate, but no mechanistic explanation for this phenomenon has been offered. We address this gap by applying sparse autoencoders (SAEs), a tool from mechanistic interpretability, to probe the internal representations of PatchTST. We first establish that a single-layer, narrow-dimensional transformer matches the forecasting performance of deeper configurations across commonly used benchmarks. We then train SAEs on the post-GELU intermediate FFN activations with dictionary sizes ranging from 0.5x to 4.0x the native dimensionality. Expanding the dictionary yields negligible downstream performance change (average 0.214%), with large portions of overcomplete dictionaries remaining inactive. Targeted causal interventions on dominant latent features produce minimal forecast perturbation. Across all evaluated settings, we observe no empirical evidence that the analyzed FFN representations rely on strong superposition. Instead, the representations remain sparse, stable under aggressive dictionary expansion, and largely insensitive to latent interventions. These results demonstrate that superposition is not necessary for competitive performance on standard forecasting benchmarks, suggesting they may not demand the rich compositional representations that drive transformer success in language modeling, and helping explain the persistent competitiveness of simple linear models
Recent progress in time-series forecasting has led to rapidly increasing architectural complexity, yet many reported State-of-the-Art gains are statistically fragile or misattributed. We argue that progress requires a shift from model selection to modular attribution, identifying which components truly drive performance. We propose CombinationTS, a self-contained probabilistic evaluation framework that decomposes forecasting models into orthogonal modules--Input Transformation, Embedding, Encoder, Decoder, and Output Transformation--and evaluates them under a shared evaluation condition space. By quantifying each component via marginalized performance (μ) and stability (σ), CombinationTS enables robust attribution beyond fragile point estimates. Through large-scale paired evaluation, we uncover the Identity Paradox: once the data view (Embedding) is well-designed, a parameter-free Identity Encoder often matches or outperforms complex backbones. We further show that explicit structural priors introduced via Input Transformations yield a more favorable performance-stability trade-off than increasing Encoder complexity, establishing a principled baseline for architectural necessity.
Xiaorui Wang, Fanda Fan, Chenxi Wang +9
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Beijing Normal University - Hong Kong Baptist University United International College, Zhuhai, China +1