TSAE: Structured Sparse Autoencoders for Interpreting Time-Series Forecasting Models

Baoxi Liu; Yi Xie

cs.LGOct 4, 2026

TSAE: Structured Sparse Autoencoders for Interpreting Time-Series Forecasting Models

Authors: Baoxi Liu, Yi Xie

Abstract

Time-series forecasting informs critical decisions in energy dispatch, industrial operations, and environmental monitoring; understanding the patterns models rely on is essential for assessing reliability and identifying failures. Input attribution identifies important variables and time segments but offers limited insight into internal features, while standard sparse autoencoder (SAE) objectives do not directly constrain cross-variable structure or temporal continuity. We introduce TSAE, a structured sparse autoencoder for forecasting representations that decomposes hidden states into individually inspectable features. TSAE organizes cross-variable structure through shared and variable-routed private dictionaries, separates feature detection from magnitude estimation with gated encoding, and constrains neighboring sparse-code changes according to raw-segment similarity. These mechanisms support analysis of variable context, activation strength, and temporal evolution. Forecast-consistency fine-tuning further improves preservation of the frozen forecaster's outputs. The accompanying TSEVAL protocol separately audits fidelity, feature coherence, and physical calibration to ground feature interpretation. In three-seed experiments with frozen PatchTST on ETTh1, ETTh2, and ETTm1, TSAE achieves the lowest hidden-state reconstruction error, normalized forecast-reconstruction error (NFRE), and feature transition rate among five SAEs at comparable per-token activity. NFRE decreases by 4.3-26.4% relative to the next-best mean. Dataset-dependent tradeoffs in selectivity, physical correlation, and calibration show that fidelity and temporal-stability gains require independent semantic validation to support feature interpretation.

Figures & tables

Figure 1: Three time-series properties motivate TSAE : (a) shared motifs and variable-specific structure motivate shared/private routing; (b) motif strength motivates separating detection from magnitude; (c) similar adjacent patches of one variable motivate code continuity, with weaker coupling across real changes. Traces are schematic.

Figure 2: TSAE model framework. Raw time-series inputs are first encoded by a forecasting backbone into hidden tokens. For each token, the TSAE encoder produces shared and variable-routed private gated sparse codes, and the decoder reconstructs the hidden token from the corresponding shared and private dictionaries. Conditional temporal continuity is applied to adjacent sparse codes, with the regularization strength determined by the similarity of their raw temporal segments.

Diagnostic	Question
Fidelity	Does the sparse reconstruction preserve the original hidden state?
Coherence	Are activations concentrated in coherent variable and temporal contexts?
Calibration	Do feature activations track independently measurable signal motifs?
Forecast sensitivity	How sensitive is the forecast to a specified sparse-feature intervention?
Hidden control	Is the effect specific to the SAE decoder path rather than a generic hidden-space perturbation?
Input baselines	Do hidden sparse features provide information beyond saliency and occlusion?

Table 1: The TSEval diagnostic checklist. Each axis tests a claim that reconstruction-only evaluation cannot adjudicate. In TSEval , a feature is interpreted only with respect to the claims supported by the corresponding diagnostics.

Setting		Operating point	Evaluation metrics
Data	Method	Active	Recon. $\downarrow$	NFRE $\downarrow$	Dead $\downarrow$	Trans. $\downarrow$	Sel. $\uparrow$	Phys. $\uparrow$	Calib. $\downarrow$
ETTh1	TSAE	231.4 $\pm$ 2.7	13.44 $\pm$ 0.57	5.10 $\pm$ 0.25	0.080 $\pm$ 0.025	0.073 $\pm$ 0.005	0.639 $\pm$ 0.007	0.541 $\pm$ 0.106	0.624 $\pm$ 0.049
	L1	252.8 $\pm$ 10.3	13.59 $\pm$ 0.45	5.33 $\pm$ 0.27	0.000 $\pm$ 0.000	0.181 $\pm$ 0.006	0.570 $\pm$ 0.023	0.653 $\pm$ 0.047	0.567 $\pm$ 0.022
	Gated	244.4 $\pm$ 7.4	16.39 $\pm$ 0.75	6.39 $\pm$ 0.22	0.000 $\pm$ 0.000	0.144 $\pm$ 0.005	0.573 $\pm$ 0.020	0.523 $\pm$ 0.060	0.681 $\pm$ 0.024
	TopK	264.9 $\pm$ 5.7	30.54 $\pm$ 2.00	10.84 $\pm$ 0.75	0.001 $\pm$ 0.001	0.143 $\pm$ 0.002	0.592 $\pm$ 0.007	0.622 $\pm$ 0.047	0.587 $\pm$ 0.043
	JumpReLU	269.8 $\pm$ 11.4	20.69 $\pm$ 0.88	8.09 $\pm$ 0.37	0.000 $\pm$ 0.000	0.156 $\pm$ 0.007	0.612 $\pm$ 0.066	0.587 $\pm$ 0.017	0.657 $\pm$ 0.002

Table 2: Validation-selected PatchTST results on four datasets. Each entry is the mean

\pm

standard deviation over three independently trained SAE seeds. Operating points are selected only on validation data to target 256 active features per token (admissible range: 192–320); all reported metrics are test-split measurements. Bold and underline mark the best and second-best mean, respectively, within a dataset and metric. Recon. and NFRE are multiplied by

10^{3}

Figure 3: PatchTST–ETTh1 validation activity–fidelity curves. Each point is a validation calibration run; the horizontal axis reports measured active features per token. Table 2 reports separate test-split metrics.

Backbone	Method	Active	Recon. $\downarrow$	NFRE $\downarrow$
PatchTST	TSAE	231.4 $\pm$ 2.7	13.44 $\pm$ 0.57	5.10 $\pm$ 0.25
	Gated	244.4 $\pm$ 7.4	16.39 $\pm$ 0.75	6.39 $\pm$ 0.22
	L1	252.8 $\pm$ 10.3	13.59 $\pm$ 0.45	5.33 $\pm$ 0.27
	TopK	264.9 $\pm$ 5.7	30.54 $\pm$ 2.00	10.84 $\pm$ 0.75
	JumpReLU	269.8 $\pm$ 11.4	20.69 $\pm$ 0.88	8.09 $\pm$ 0.37
iTransformer	TSAE	253.1 $\pm$ 2.5	14.48 $\pm$ 0.23	3.01 $\pm$ 0.06

Table 3: Cross-backbone active-matched comparison (ETTh1, three-seed means). Bold and underline mark the best and second-best means, respectively, within each backbone and metric. Recon. and NFRE are

\times 10^{3}

. Full tables and validation sweeps are in Appendix C.4 .

Data	Variant	Active	Recon. $\downarrow$	NFRE $\downarrow$	Dead	Trans.
ETTh1	Full	268	11.7	4.9	6.3	0.086
	No routing	252	13.4	5.5	0.0	0.145
	No gating	108	18.1	7.9	0.0	0.083
	No temporal	279	12.3	5.1	6.3	0.091
ETTh2	Full	203	2.2	3.4	10.9	0.036
	No routing	178	2.4	3.6	13.1	0.050

Table 4: PatchTST component ablation (three-seed means). Bold marks the lowest reconstruction and NFRE values within each dataset. Dead-feature ratio and transition rate are reported as diagnostic tradeoffs. Recon. and NFRE use the report scale (

\times 10^{3}

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Multi-feature case atlas from the formal PatchTST–ETTh1 run. Blue curves are non-overlapping high-activation patches and red curves are non-overlapping low-activation controls. The first row shows shared atoms that recur across variables; the second row shows private atoms, each restricted to its routed variable. The panel is illustrative; the full pre-recorded 12-atom atlas and selection manifest are stored in the artifact. In-panel annotations are small and are best viewed zoomed in the electronic PDF.

Panel of Figure 5 — Figure 5: Feature-level evidence beyond reconstruction. (a) ETTh1 activation as a function of the associated raw physical metric; bars show 95% confidence intervals within equal-frequency bins. (b) Mean forecast shift after deleting a selected physical-pattern feature (blue) or an activation-rate-matched, lower-correlation control (gray), while the PatchTST backbone remains frozen. The intervention measures sensitivity within the model pathway, not a causal effect in the data-generating system.

Dataset	L1 SAE Jaccard	TSAE Jaccard	Difference
ETTh1	0.184	0.257	+0.073
ETTh2	0.456	0.676	+0.220
ETTm1	0.424	0.540	+0.116

Appendix

Table 5: Adjacent-patch support stability at an identical Top-

24

feature budget per token. Values are descriptive means over adjacent test-patch pairs for the seed-2024 formal checkpoints.

Dataset	Method	Selected setting	Test Active
ETTh1	TSAE	shared $\lambda=6.0$ , private $\lambda=0.30$	231.4 $\pm$ 2.7
	L1	$\lambda=2.0$	252.8 $\pm$ 10.3
	Gated	$\lambda=6.0$	244.4 $\pm$ 7.4
	TopK	$k=320$	264.9 $\pm$ 5.7
	JumpReLU	$\lambda{=}10^{-4}$ , threshold ${=}0.10$	269.8 $\pm$ 11.4
ETTh2	TSAE	shared $\lambda=3.0$ , private $\lambda=0.15$	250.9 $\pm$ 13.6

Appendix

Table 6: Validation-selected operating points with test-split active counts. TSAE runs use

\lambda_{\mathrm{temp}}=0.5

\lambda_{\mathrm{fc}}=0.1

, and

\lambda_{\mathrm{balance}}=0.003

$\lambda_{\mathrm{temp}}$	Active	Recon. $\downarrow$	NFRE $\downarrow$	Trans. $\downarrow$
0	280	12.3	5.1	0.0908
0.0001	280	12.3	5.1	0.0909
0.001	280	12.3	5.1	0.0908
0.01	278	12.3	5.1	0.0901
0.05	276	12.2	5.1	0.0903
0.1	273	12.2	5.1	0.0895

Appendix

Table 7: Temporal coefficient sweep (single seed unless noted). The bold row is the validation-selected formal coefficient. Recon. and NFRE are

\times 10^{3}

$\lambda_{\mathrm{priv}}$	Active	Recon. $\downarrow$	NFRE $\downarrow$	Dead%
0.02	500	10.7	4.4	6.2
0.05	489	9.4	3.8	2.3
0.1	456	9.5	3.9	0.6
0.2	416	10.2	4.2	1.0
0.5	392	11.0	4.4	7.0
1.0	386	11.3	4.5	13.8

Appendix

Table 8: Private sparsity sensitivity at a fixed shared coefficient (single seed). Bold marks the illustrative low-dead-feature range; formal comparisons use dataset-specific validation-selected values.

Figure 6: Validation calibration curves on PatchTST–ETTh1. Horizontal axis: sparsity parameter; vertical axis: measured active features per token. Gray band: admissible [192,320] window.

Method	Active	In 192–320?	Recon. $\downarrow$	NFRE $\downarrow$	Dead%	Trans. $\downarrow$
TSAE	231 $\pm$ 3	✓	13.4	5.10	8.0	0.073
L1	253 $\pm$ 10	✓	13.6	5.33	0.0	0.181
Gated	244 $\pm$ 7	✓	16.4	6.39	0.0	0.144
TopK	265 $\pm$ 6	✓	30.5	10.84	0.1	0.143
JumpReLU	270 $\pm$ 11	✓	20.7	8.09	0.0	0.156

Appendix

Table 9: ETTh1: all five methods within the [192,320] window. TSAE has the best Recon, NFRE, and Transition rate.

Method	Active	In 192–320?	Recon. $\downarrow$	NFRE $\downarrow$	Dead%	Trans. $\downarrow$
TSAE	251 $\pm$ 14	✓	6.04	2.70	4.3	0.050
L1	195 $\pm$ 11	✓	7.99	3.66	2.5	0.106
Gated	213 $\pm$ 6	✓	8.19	3.76	10.3	0.055
TopK	218 $\pm$ 1	✓	9.88	4.56	14.3	0.063
JumpReLU	311 $\pm$ 40	✓	10.9	4.79	4.1	0.064

Appendix

Table 10: ETTh2: all five methods are within the [192,320] window. TSAE is the best on Recon, NFRE, and Transition.

Method	Active	In 192–320?	Recon. $\downarrow$	NFRE $\downarrow$	Dead%	Trans. $\downarrow$
TSAE	205 $\pm$ 12	✓	10.4	5.00	3.1	0.047
L1	258 $\pm$ 12	✓	11.4	5.50	0.0	0.135
Gated	247 $\pm$ 6	✓	13.5	6.58	0.0	0.089
TopK	224 $\pm$ 2	✓	19.9	8.02	0.6	0.083
JumpReLU	270 $\pm$ 13	✓	17.1	8.63	0.0	0.092

Appendix

Table 11: ETTm1: all five methods within the [192,320] window. TSAE has the best Recon, NFRE, and Transition rate.

Method	Active	In 192–320?	Recon. $\downarrow$	NFRE $\downarrow$	Dead%	Trans. $\downarrow$
TSAE	217 $\pm$ 3	✓	1.70	2.52	19.5	0.019
L1	188 $\pm$ 16	$\times$	1.85	3.37	0.0	0.055
Gated	285 $\pm$ 9	✓	2.10	2.84	0.0	0.046
TopK	261 $\pm$ 14	✓	3.84	4.64	7.3	0.038
JumpReLU	236 $\pm$ 1	✓	4.49	6.21	0.1	0.031

Appendix

Table 12: Weather: the validation-selected settings yield in-window test means for TSAE, Gated, TopK, and JumpReLU. L1 is slightly below the boundary at 188, despite its validation point lying in-window. TSAE has the best Recon, NFRE, and Transition.

Method	Active	Recon. $\downarrow$	NFRE $\downarrow$
TSAE	253 $\pm$ 3	14.5	3.01
L1	282 $\pm$ 1	18.1	4.14
Gated	316 $\pm$ 1	17.3	3.20
TopK	212 $\pm$ 0	30.1	5.24
JumpReLU	269 $\pm$ 3	18.8	3.44

Appendix

Table 13: iTransformer–ETTh1 active-matched results (three-seed means). Recon. and NFRE are

\times 10^{3}

Method	Active	Recon. $\downarrow$	NFRE $\downarrow$
TSAE	215 $\pm$ 10	9.09	7.91
L1	250 $\pm$ 1	9.17	8.74
Gated	314 $\pm$ 1	7.71	6.88
TopK	256 $\pm$ 0	9.90	8.27
JumpReLU	266 $\pm$ 4	7.99	7.02

Appendix

Table 14: DLinear–ETTh1 active-matched results (three-seed means). Recon. and NFRE are

\times 10^{3}

Data	Variant	Active	Recon. $\downarrow$	NFRE $\downarrow$	Dead%	Trans. $\downarrow$
ETTh1	Full	268	11.7	4.9	6.3	0.086
	No routing	252	13.4	5.5	0.0	0.145
	No gating	108	18.1	7.9	0.0	0.083
	No temporal	279	12.3	5.1	6.3	0.091
ETTh2	Full	203	2.2	3.4	10.9	0.036
	No routing	178	2.4	3.6	13.1	0.050

Appendix

Table 15: PatchTST component ablation (three-seed means). Bold marks the best per column per dataset.

Data	Variant	Active	Recon. $\downarrow$	NFRE $\downarrow$
ETTh1	Full	399	13.7	3.3
	No routing	309	23.3	4.3
	No gating	188	34.7	7.8
ETTh2	Full	360	22.9	1.1
	No routing	284	31.6	1.4
	No gating	187	45.7	1.9

Appendix

Table 16: iTransformer component ablation. Transition rate not applicable.

Data	Variant	Active	Recon. $\downarrow$	NFRE $\downarrow$	Dead%
ETTh1	Full	219	21.3	9.8	2.4
	No routing	165	25.7	11.6	0.0
	No gating	121	43.5	21.9	0.0
ETTh2	Full	153	7.9	7.3	5.9
	No routing	117	8.0	7.2	0.0
	No gating	87	15.0	13.2	0.0

Appendix

Table 17: DLinear component ablation.

Data	Variant	Active	Recon. $\downarrow$	NFRE $\downarrow$	Dead%
ETTh1	Full	322	13.6	3.5	3.2
	No routing	281	18.7	5.0	0.0
	No gating	126	38.1	12.8	0.0
ETTh2	Full	287	4.2	4.6	8.3
	No routing	221	5.6	6.1	8.7
	No gating	74	12.7	15.8	0.0

Appendix

Table 18: Transformer component ablation.

Explainable Time Series Forecasting

Explainable time series forecasting makes predictions over time interpretable through attributions, structured models, causal reasoning, and counterfactuals.

See all 61 papers

Dissecting Chronos: Sparse Autoencoders Reveal Causal Feature Hierarchies in Time Series Foundation Models
Mar 10, 2026Anurag MishraTime-Series Foundation Models Transformer Interpretability
Rochester Institute of Technology
Superposition Is Not Necessary: A Mechanistic Interpretability Analysis of Transformer Representations for Time Series Forecasting
May 6, 2026Alper YıldırımTransformer Interpretability Time Series Forecasting
CombinationTS: A Modular Framework for Understanding Time-Series Forecasting Models
May 2, 2026Xiaorui Wang, Fanda Fan, Chenxi Wang +9Time Series Forecasting Forecasting Benchmarks
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Beijing Normal University - Hong Kong Baptist University United International College, Zhuhai, China +1

Mar 10, 2026J/K move · Enter open · S save

Dissecting Chronos: Sparse Autoencoders Reveal Causal Feature Hierarchies in Time Series Foundation Models

Anurag Mishra

Rochester Institute of Technology

Time-Series Foundation Models Transformer Interpretability Mechanistic Interpretability Time Series Forecasting Sparse Autoencoders

Open paper AskPDF arXiv

Time series foundation models (TSFMs) are increasingly deployed in high-stakes domains, yet their internal representations remain opaque. We present the first application of sparse autoencoders (SAEs) to a TSFM, training TopK SAEs on activations of Chronos-T5-Large (710M parameters) across six layers. Through 392 single-feature ablation experiments, we establish that every ablated feature produces a positive CRPS degradation, confirming causal relevance. Our analysis reveals a depth-dependent hierarchy: early encoder layers encode low-level frequency features, the mid-encoder concentrates causally critical change-detection features, and the final encoder compresses a rich but less causally important taxonomy of temporal concepts. The most critical features reside in the mid-encoder (max single-feature Delta CRPS = 38.61), not in the semantically richest final encoder layer, where progressive ablation paradoxically improves forecast quality. These findings demonstrate that mechanistic interpretability transfers effectively to TSFMs and that Chronos-T5 relies on abrupt-dynamics detection rather than periodic pattern recognition.

Abstract

Explainable Time Series Forecasting

Explore similar work

Dissecting Chronos: Sparse Autoencoders Reveal Causal Feature Hierarchies in Time Series Foundation Models

Superposition Is Not Necessary: A Mechanistic Interpretability Analysis of Transformer Representations for Time Series Forecasting

CombinationTS: A Modular Framework for Understanding Time-Series Forecasting Models