When Attention Does Not Explain the Peak: Temporal Reference vs. Forecast Output in Attention-Based Time-Series Forecasting

Yuji Akamatsu; Takao Yamanaka

cs.LGOct 5, 2026

When Attention Does Not Explain the Peak: Temporal Reference vs. Forecast Output in Attention-Based Time-Series Forecasting

Authors: Yuji Akamatsu, Takao Yamanaka

Organizations: Sophia University Tokyo, Japan

Abstract

Attention maps are often interpreted as evidence of what a forecasting model uses when making predictions. In our load-forecasting model, a CLS representation of historical demand queries 24 future exogenous horizon tokens through cross-attention, inviting a temporal interpretation in which highly attended horizons may appear to explain forecast peak timing. We test this interpretation using a horizon-level attention descriptor, $Ψ_{\mathrm{out}}$ . Across 31 day-aligned windows of the Panama load dataset, the forecast achieves a median peak-time error of 0 h and a 51.6% exact-match rate, whereas the argmax of $Ψ_{\mathrm{out}}$ has a median error of 5 h and 0% exact match. The forecast peak is closer to the observed peak in 27 of 31 windows. This dissociation is not merely an argmax artifact: within $\pm1$ h, attention reaches only $1.16\times$ , $1.11\times$ , and $1.14\times$ the uniform baseline around observed, predicted, and weekly-naive peaks, respectively, indicating weak and non-selective concentration. Yet the attention profile is structured, with cross-window consistency of 0.83. Replacing 12 future weather features with their training-set means makes the profile nearly uniform, showing sensitivity to future weather variation rather than fixed horizon position alone. The dissociation is also reproduced across three random-seed runs. These results show that structured, input-sensitive, and reproducible horizon-level cross-attention need not provide a valid peak-selective explanation of forecast behavior. The observed behavior is instead consistent with an internal horizon-reference role for integrating future exogenous information, although this functional role is not causally established.

Figures & tables

Figure 1: Dissociation between forecast peak error and attention-derived peak error over 31 day-aligned windows. Points above the diagonal indicate windows in which the forecast peak is closer to the observed peak than the $\Psi_{\mathrm{out}}$ argmax. This occurs in 27 of 31 windows.

Signal	Median peak error (h)	Exact match
Forecast (ViT, GAF)	0	51.6%
Weekly naive (lag 168)	0	51.6%
$\Psi_{\mathrm{out}}$ argmax	5	0.0%

Table 1: Peak-time comparison on the Panama test period (31 day-aligned windows).

Figure 2: Horizon-level attention profiles under the original inference setting (left) and the weather-mean sensitivity condition (right). Rows correspond to test windows and columns to the 24 future horizons. Under the original inputs,

\Psi_{\mathrm{out}}

exhibits a structured and broadly consistent temporal profile that does not selectively coincide with the observed or forecast peaks. Replacing future weather features with their training-set means makes the profile nearly uniform, showing that the structure is sensitive to future weather variation and cannot be explained by fixed horizon position alone.

Seed	$e_{\Psi}$	Forecast exact	$\Psi_{\mathrm{out}}$ exact	$\Psi_{\mathrm{out}}$ /F	Consistency
0	4.5	53.3%	0.0%	2/26	0.82
1	6.0	53.3%	10.0%	2/25	0.79
2	1.0	66.7%	16.7%	2/22	0.89

Table 2: Random-seed robustness on the 30 day-aligned origins common to all three runs.

\Psi_{\mathrm{out}}

/F denotes the numbers of windows in which the attention-derived peak and the forecast peak, respectively, are closer to the observed peak.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Representative day-aligned windows selected by forecast peak-time error: best, median, and worst. The left column shows the observed demand and forecast, with vertical lines marking their peak horizons. The right column shows the corresponding horizon-level attention profile

\Psi_{\mathrm{out}}(h)

and the uniform reference level. The examples illustrate the dissociation between forecast peak timing and the attention-derived horizon reference.

Explore similar work

Sep 14, 2026cs.LG

Explaining Time Series Forecasting with Horizon-Resolved Attribution

Recent advances in explaining time series (TS) models have produced methods that identify which past values a prediction depends on. However, most existing methods return a single importance vector, assuming that every predicted step depends on the same past values. In this paper, we show that this assumption does not hold, as different forecast steps depend on different past values. Motivated by this observation, we propose Horizon-Resolved eXplanation (HRX), which adds a horizon axis to the explanation, so that every forecast step receives its own importance map. HRX is a simple yet effective plug-in framework with three components: 1) an estimator that reads these maps out of any differentiable forecaster without modifying the TS backbone, 2) an evaluation protocol that validates the horizon axis by measuring how much a single forecast step changes when the inputs an importance map ranks highest are removed, and 3) a rank criterion that predicts in advance whether the axis is worth resolving on a given TS. We further show that this step-wise dependence is low-dimensional, as the explanations of all steps are built from a few shared maps whose number does not grow with the forecast length. Extensive experiments across various backbones and datasets show that the improvement comes from the horizon axis and holds for estimators of previous explanation methods. Code is available at https://github.com/seunghan96/HRX.

Seunghan Lee, Jun Seo, Jaehoon Lee +9

LG AI Research

Time Series Forecasting Time Series Forecast Horizon

Ask PDF

Mar 16, 2026cs.LG

Not All Retrievals are Useful: Cross-Attention for Input-Aware RAG in Time Series Forecasting

Retrieval-augmented generation (RAG) enhances zero-shot time series (TS) forecasting by leveraging external knowledge bases, yet existing approaches overlook input-level relevance when fusing retrieved samples with the query. We argue that not all retrievals are equally useful, and irrelevant ones can degrade performance. To this end, we propose Cross-RAG, a zero-shot RAG-based forecasting framework that selectively attends to query-relevant retrieved samples via query--retrieval cross-attention. By modeling input-level relevance between the query and retrieved samples, Cross-RAG jointly incorporates three sources of information: 1) the query itself, 2) the retrieved samples, and 3) their relational interactions. In particular, this input-aware design enables Cross-RAG to remain stable as the number of retrieved samples

k

grows, whereas prior methods without cross-attention require careful

k

tuning to avoid degradation from irrelevant retrievals. Extensive experiments demonstrate that Cross-RAG consistently improves zero-shot forecasting performance across multiple TSFM backbones and various RAG methods, with additional analyses confirming its effectiveness across various retrieval scenarios. Code is available at https://github.com/seunghan96/cross-rag/.

Seunghan Lee, Jaehoon Lee, Jun Seo +7

LG AI Research Seoul, South Korea

Zero-Shot Forecasting Time Series Forecasting Cross-Attention

Ask PDF

May 20, 2026cs.LG

PeakFocus: Bridging Peak Localization and Intensity Regression via a Unified Multi-Scale Framework for Electricity Load Forecasting

Electricity load peak forecasting (ELPF), simultaneously predicting peak timing and intensity, is a prerequisite for effective grid scheduling and risk management. However, existing methods face three limitations. First, they adopt a two-stage predict-then-locate paradigm, which severs the link between temporal localization and intensity regression. Second, they still struggle with the multi-scale representation conflict, leading to peak misjudgment and timing misalignment. Third, the lack of explicit peak timing context during intensity regression causes intensity smoothing because predictions are dominated by global smoothing trends. To address these limitations, we propose PeakFocus, a unified framework for ELPF. (i) A Unified Peak-Aware Pipeline (UPAP) utilizes a triple hybrid loss to jointly supervise temporal localization and intensity regression, alongside a tolerance-based evaluation protocol. (ii) A Multi-Scale Mixing Peak Locator (MSM-PL) exploits coarse-grained features to mitigate peak misjudgment caused by local fluctuations, and injects them into fine-grained features via a cascade mechanism to resolve timing misalignment. (iii) A Location-Aware Decoder (LAD) injects peak timing context into the intensity regression process, providing explicit guidance to counteract intensity smoothing and improve peak intensity estimation. Extensive experiments on the public Electricity (ELC) dataset and our industrial-scale World Large-scale Electricity Load (WLEL) dataset show that PeakFocus outperforms baselines in both timing precision and intensity estimation.

Wangzhi Yu, Peng Zhu, Qing Zhao +2

School of Computer Science and Technology, Tongji University, Shanghai, China · Big Data Center, State Grid Corporation of China, Beijing, China

Accurate Short-Term Electricity Load Forecasting Intensity Multiscale

Ask PDF