TempShap: When and How Much a Feature Matters in Time-Series Forecasting

Chi Zhao; Anastasiia Chelnokova; Jing Liu; Elena Parilina

cs.AIOct 23, 2026

TempShap: When and How Much a Feature Matters in Time-Series Forecasting

Authors: Chi Zhao, Anastasiia Chelnokova, Jing Liu, Elena Parilina

Organizations: Saint Petersburg State University, 7/9 Universitetskaya nab., Saint Petersburg 199034, Russia

Abstract

Attribution for a time-series forecaster should answer two questions that a feature-level scalar cannot: when in the input window a feature acts, and how much its influence changes across the forecast horizon. Most existing Shapley explainers treat each feature as one player and therefore collapse both axes. We propose TempShap, a retrospective, loss-based Shapley framework whose players are the individual input cells $(t,i)$ . Its attribution tensor yields temporal-localization and horizon-impact profiles. We validate these against planted temporal structures, a tractable exact cell game, and a closed-form attribution oracle for a real linear forecaster. Permutation sharing estimates the cell-level game at localized perturbation cost. Under redundant inputs, one-at-a-time scores under-credit substitutable information, whereas the signed Shapley decomposition preserves total value and yields more reliable held-out deletion rankings under mean masking. A planted counterexample isolates the redundancy mechanism. Experiments show a deletion-ranking advantage across Shapley estimators, forecasting benchmarks, and model families. TempShap therefore makes fine-grained Shapley attribution both practical and operationally useful.

Figures & tables

Panel of Figure 1 — Figure 1: TempShap at a glance. (left) Players are the input cells $(t,i)$ , not whole feature columns (which mask all $T$ lags at once and lose which lag matters); a per-output-cell loss game $v(S)=-(Y-\hat{Y}|_{S})^{2}$ yields (right) the tensor $\Phi\in\mathbb{R}^{N\times H\times K\times T\times M}$ , whose profiles (averages of $|\Phi|$ over the other axes) give $g_{i}(t)$ ( when ) and $D_{i}(h)$ ( how much ).

Construction	Estimator	when centroid	agreement
C1 ( $T{=}24$ , $M{=}8$ )	planted	$(9.5,17.5)$	—
	cell-pShapley	$(9.7,17.0)$	both top-2
	cell-ShapG	$(9.3,16.8)$	both top-2
	TimeSHAP-c	$(9.7,16.9)$	both top-2
Cell-Exact ( $T{=}6$ , $M{=}2$ )	exact oracle	$2.54$	—
	cell-pShapley	$2.54$	$\tau=0.91$

Table 1: Recovery of the planted signal. Centroids (in lags):

(\bar{t}_{0},\bar{t}_{4})

for C1;

\bar{t}_{0}

for Cell-Exact.

Figure 2: Synthetic ground-truth validation. (a) C1: localization profiles

g_{i}(t)

peak at the planted active lags (shaded). (b) C2: the horizon-impact profile

D_{0}(h)

tracks the planted

\gamma^{h}

. (c) C3: the ranking settles immediately (

\tau_{h}\approx 1

) while the magnitude

D(h)

decays; the ranking and the magnitude are distinct.

Method	Shapley	Attribution granularity	Top-2 recovery	Centroid $\bar{t}_{0}$ (true: 9)	Centroid $\bar{t}_{4}$ (true: 17)	windows	wall-clock
feature-ShapG	yes	feature	Yes	—	—	$5.9$ k	$1.3$ s
WindowSHAP	yes	time chunk	N/A	—	—	$25$ k	$0.2$ s
GS-SHAP	yes	group $\times$ segment	No g	$13.0^{g}$	$11.5^{g}$	$0.8$ k	$64.2$ s
PAX-TS	no	cell $(t,i)$	Yes	$9.4$	$15.2$	$111$ k	$0.7$ s
cell-ShapG (graph)	yes	cell	Yes	$9.7$	$16.3$	$706$ k	$23.6$ s
cell-pShapley	yes	cell	Yes	$9.4$	$16.4$	$111$ k	$\mathbf{0.5}$ s

Table 2: Localization and cost comparison. Top-2 recovery: features 0 and 4 occupy the top two ranks; N/A: no feature ranking; g : derived from group-level attributions. Centroids are in lags; windows count model-forward evaluations; times use one GPU. Budgets:

m_{\pi}=3

(cell-pShapley),

m_{G}=5

(cell-ShapG). All Shapley methods use the same mean baseline.

corr $(x_{0},x_{1})$	cell-pShapley	LOO	PAX-TS mass
$0.98$	$104\pm 8$	$50\pm 3$	$53\pm 5$
$0.80$	$109\pm 10$	$42\pm 8$	$43\pm 5$
$0.20$	$72\pm 47$	$84\pm 32$	$46\pm 5$

Table 3: Share of the redundant pair’s joint value

\Delta_{\mathrm{pair}}

credited by each method (synthetic;

10

seeds, mean

\pm

std, %; cell-pShapley at

m_{\pi}{=}8

). The PAX-TS column is its (non-additive) fraction of total attribution mass.

Dataset	Family	cell-pShapley	$\|\phi\|$	cell-ShapG	TimeSHAP-c	PAX-TS	cell-LOO
Jena	LSTM	$1.12\pm 0.19$	$0.68\pm 0.08$	$\mathbf{1.16}\pm 0.25$	$1.12\pm 0.19$	$0.55\pm 0.02$	$0.63\pm 0.07$
Jena	Transformer	$0.80\pm 0.03$	$0.62\pm 0.03$	$\mathbf{0.80}\pm 0.03$	$0.80\pm 0.03$	$0.52\pm 0.02$	$0.60\pm 0.06$
Jena	DLinear	$\mathbf{1.95}\pm 0.20$	$0.85\pm 0.07$	$1.93\pm 0.20$	$1.95\pm 0.20$	$0.81\pm 0.09$	$0.90\pm 0.06$
ETTh1	LSTM	$\mathbf{0.93}\pm 0.01$	$0.92\pm 0.00$	$0.92\pm 0.01$	$0.92\pm 0.01$	$0.91\pm 0.01$	$0.92\pm 0.00$
ETTh1	Transformer	$\mathbf{0.92}\pm 0.00$	$0.91\pm 0.01$	$0.92\pm 0.00$	$0.91\pm 0.01$	$0.90\pm 0.01$	$0.90\pm 0.00$
ETTh1	DLinear	$\mathbf{1.00}\pm 0.02$	$0.93\pm 0.00$	$0.99\pm 0.02$	$1.00\pm 0.02$	$0.94\pm 0.00$	$0.93\pm 0.00$

Table 4: Weighted slope

S

(mean

\pm

std over three model-training seeds) of the deletion curve under mean masking; bold = best mean per row, and

|\phi|

is the unsigned cell-pShapley ranking. The last two rows count how often each ranking beats a baseline over the 27 (case, seed) runs; exact sign tests give

p<10^{-6}

(cell-pShapley vs cell-LOO) and

p<10^{-4}

(vs PAX-TS).

Figure 3: Cell-level deletion curves for one of the three training seeds averaged in Table 4 (held-out windows;

U{=}1

full model,

U{=}0

all-masked; legend gives

S

; cell-pShapley at

m_{\pi}{=}16

). Deleting in cell-pShapley order drives

U

below the all-masked baseline on the redundant Jena panels while PAX-TS and cell-LOO plateau; the methods nearly tie on ETTh1.

Dataset	Family	test/naive	top feature	when-centroid	settle $\tau{\geq}0.9$
Jena	LSTM	$0.051$	Tpot (K)	$12.5$	$16$
Jena	Transformer	$0.051$	Tpot (K)	$11.9$	$17$
Jena	DLinear	$0.012$	rho (g/m**3)	$11.4$	$2$
ETTh1	LSTM	$0.048$	OT	$12.6$	$5$
ETTh1	Transformer	$0.046$	OT	$13.0$	$1$
ETTh1	DLinear	$0.019$	OT	$12.2$	$1$

Table 5: Real-data when /horizon-impact across families (

K=1

, cell-pShapley

m_{\pi}{=}32

, mean baseline). when-centroid = mean input lag weighted by

g(t)

(

0

=oldest); settle = horizon step at which the ranking reaches

\tau_{h}\geq 0.9

(single training runs).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Per-feature when recovery on the planted band (dotted = planted lags

9,17

). The cell estimators peak sharply at both lags; PAX-TS recovers them with a more multi-peaked feat4 profile; GS-SHAP is flat, since its groups cannot separate the features.

Figure 5: Cumulative per-feature importance up to step

h

; the order in which the curves settle is what

\tau_{h}

measures. Synthetic C3 settles at

h=1

while its magnitudes decay; on Jena the settling horizon differs by architecture.

Figure 6: Running-mean importance of the top-

4

features against the number of input windows averaged (cell-pShapley, real data,

N{=}256

). The top-

4

set settles within a few dozen windows, well before the individual magnitudes stop drifting.

Dataset	Estimator	signed Pearson	per-cell $\tau$	Spearman	centroid $\Delta$ (lags)
Jena	cell-pShapley	$1.000$	$1.000$	$1.000$	$0.00$
	cell-ShapG	$0.986$	$0.969$	$0.980$	$0.11$
ETTh1	cell-pShapley	$1.000$	$1.000$	$1.000$	$0.00$
	cell-ShapG	$0.916$	$0.904$	$0.954$	$0.50$
Exchange-Rate	cell-pShapley	$1.000$	$1.000$	$1.000$	$0.00$
	cell-ShapG	$0.871$	$0.841$	$0.872$	$0.05$

Appendix

Table 6: Agreement with the closed-form DLinear oracle: signed Pearson, per-cell

\tau

and Spearman, and the max-abs centroid difference in lags.

Figure 7: Real-data profiles per architecture family, one panel per dataset. (top) when : temporal influence

g(t)

, with solid curves for cell-pShapley, dashed for cell-ShapG, and vertical dotted lines at the centroid of each cell-pShapley profile. (bottom) how much : horizon impact

D(h)

of the top feature, which is not monotone, so a feature’s influence fluctuates across the horizon rather than decaying. Both are normalized.

		reference (vs mean)		payoff (vs error-reduction)
Dataset	Family	marginal	conditional	prediction	$R^{2}$
Jena	LSTM	$+1.4$	$-0.6$	$+0.2$	$+1.2$
Jena	Transformer	$-0.1$	$-0.7$	$-0.1$	$+0.7$
Jena	DLinear	$+0.6$	$+0.0$	$-0.0$	$-0.0$
ETTh1	LSTM	$+0.4$	$-1.0$	$+0.2$	$-0.5$
ETTh1	Transformer	$+0.2$	$-0.4$	$-0.3$	$-0.9$

Appendix

Table 7: Shift of the when-centroid, in lags, from the deployed setting of each control run; the payoff columns keep the mean reference. † marks a variant that also changes which feature ranks first.

Figure 8: Normalised top-feature profiles for two of the nine cases: Jena LSTM and the Exchange-Rate Transformer. Coloured bands show the pointwise min–max range across eight permutation seeds at

m_{\pi}{=}1

and

32

. The black solid and orange dashed lines show two independent

m_{\pi}{=}128

references; distances are computed against the black curve.

		relative RMSE of $\phi$ , paired			relative RMSE of $\phi$ , unpaired			when-centroid error (lags)
Dataset	Family	$1$	$32$	$64$	$1$	$32$	$64$	$1$	$32$	$64$
Jena	LSTM	$0.224$	$0.042$	$0.034$	$0.601$	$0.107$	$0.074$	$0.18$	$0.01$	$0.03$
Jena	Transformer	$0.194$	$0.035$	$0.025$	$0.566$	$0.094$	$0.071$	$0.09$	$0.00$	$0.01$
Jena	DLinear	$0.000$	$0.000$	$0.000$	$0.673$	$0.120$	$0.083$	$0.00$	$0.00$	$0.00$
ETTh1	LSTM	$0.064$	$0.013$	$0.010$	$0.361$	$0.052$	$0.038$	$0.10$	$0.04$	$0.02$
ETTh1	Transformer	$0.132$	$0.026$	$0.022$	$0.387$	$0.054$	$0.043$	$0.14$	$0.02$	$0.02$

Appendix

Table 8: Convergence of cell-pShapley against a high-budget reference of 128 antithetic pairs, averaged over 3 sampling seeds; column groups are the estimation budget

m_{\pi}

. The unpaired group spends the same budget on independent permutations instead of antithetic pairs: DLinear is exact only when paired.

Evaluator		cell-pShapley	$\|\phi\|$	cell-ShapG	TimeSHAP-c	PAX-TS	cell-LOO
mean mask	median $S$	$0.931$	$0.901$	$\mathbf{0.941}$	$0.929$	$0.902$	$0.892$
	beats PAX-TS	$24$	$16$	$24$	$24$	—	$10$
	beats cell-LOO	$26$	$16$	$24$	$26$	$16$	—
conditional	median $S$	$0.097$	$\mathbf{0.269}$	$0.054$	$0.053$	$0.212$	$0.216$
	beats PAX-TS	$3$	$17$	$3$	$4$	—	$11$
	beats cell-LOO	$11$	$21$	$9$	$10$	$16$	—

Appendix

Table 9: Mean-mask versus Gaussian conditional-mean deletion. Entries report median

S

and wins against each baseline over 27 (case, seed) runs; bold marks the highest median within each evaluator.

Explainable Time Series Forecasting

Explainable time series forecasting makes predictions over time interpretable through attributions, structured models, causal reasoning, and counterfactuals.

See all 61 papers

Sep 11, 2026

Explaining Time Series Forecasting with Horizon-Resolved Attribution

Recent advances in explaining time series (TS) models have produced methods that identify which past values a prediction depends on. However, most existing methods return a single importance vector, assuming that every predicted step depends on the same past values. In this paper, we show that this assumption does not hold, as different forecast steps depend on different past values. Motivated by this observation, we propose Horizon-Resolved eXplanation (HRX), which adds a horizon axis to the explanation, so that every forecast step receives its own importance map. HRX is a simple yet effective plug-in framework with three components: 1) an estimator that reads these maps out of any differentiable forecaster without modifying the TS backbone, 2) an evaluation protocol that validates the horizon axis by measuring how much a single forecast step changes when the inputs an importance map ranks highest are removed, and 3) a rank criterion that predicts in advance whether the axis is worth resolving on a given TS. We further show that this step-wise dependence is low-dimensional, as the explanations of all steps are built from a few shared maps whose number does not grow with the forecast length. Extensive experiments across various backbones and datasets show that the improvement comes from the horizon axis and holds for estimators of previous explanation methods. Code is available at https://github.com/seunghan96/HRX.

Seunghan Lee, Jun Seo, Jaehoon Lee +9

LG AI Research

Explainable Time Series Forecasting Feature Attribution Explainable Artificial Intelligence Time Series Forecasting Time Series Modeling

Ask PDF

Sep 4, 2026

How Faithful Is Attribution for Sales Forecasting? A Counterfactual Study

Deep models for sales forecasting, such as WaveNet-style dilated convolutional networks, are accurate but opaque: when a single model predicts sales for one of many series, it offers no account of why. We add a post-hoc, architecture-agnostic counterfactual interpretability layer to a multi-series WaveNet forecaster trained on the full Corporacion Favorita grocery dataset (174,685 series over 1,688 days). The method decomposes each forecast into contributions that sum exactly to the predicted value, avoiding the allocation artifacts we observed with additive SHAP-style attribution. We evaluate faithfulness with a deletion/insertion protocol and find a statistically significant effect on both tests (deletion gap 0.22, p<0.001; insertion gap 0.27, p<0.01; robust across five background-sampling seeds), establishing that the attributions reflect genuine model behavior rather than plausible-looking artifacts. We then characterize, honestly, where attribution is and is not informative: reliance on the promotion signal is heterogeneous across series (median ratio approximately 1.0, with roughly 20% of series showing a strong effect), and the model captures the shape of the weekly sales cycle (day-of-week r=0.78) while systematically under-predicting its amplitude. Our contribution is not improved accuracy but an interpretability layer with a rigorous faithfulness evaluation and a candid account of its limits.

Glib Kechyn

Explainable Time Series Forecasting Feature Attribution Perturbation-Based Feature Attribution Time Series Forecasting

Ask PDF

Mar 14, 2026

Aumann-SHAP: The Geometry of Counterfactual Interaction Explanations in Machine Learning

We introduce Aumann-SHAP, an interaction-aware framework that decomposes counterfactual transitions by restricting the model to a local hypercube connecting baseline and counterfactual features. Each hypercube is discretized into a grid to construct an induced micro-player cooperative game in which elementary grid-step moves become players. Shapley and LES values on this TU-micro-game yield geometry-aware within-pot attributions that converge to the diagonal Aumann--Shapley / Integrated Gradients limit under grid refinement, and recover equal-split Shapley as the degenerate

m=1

special case. An exact grid-state closed form gives polynomial-time computation for fixed interaction order. On a synthetic benchmark with known ground truth, equal-split Shapley carries an irreducible bias while Aumann-SHAP converges to the correct decomposition. On German Credit, interaction geometry changes feature priority rankings in

12.3\%

of instances. On UCI Heart Disease, equal-split misattributes a cholesterol suppressor as a positive contributor, which is a sign error Aumann-SHAP corrects. On MNIST, game-theoretic attribution reaches target confidence with

3.5\times

fewer edits than magnitude-based ordering, with micro-game Shapley achieving the best efficiency across all budgets.

Adam Belahcen, Stéphane Mussard

AIRESS & UM6P

Feature Attribution Shapley Value Attribution

Ask PDF

Abstract

Explainable Time Series Forecasting

Explore similar work

Explaining Time Series Forecasting with Horizon-Resolved Attribution

How Faithful Is Attribution for Sales Forecasting? A Counterfactual Study

Aumann-SHAP: The Geometry of Counterfactual Interaction Explanations in Machine Learning