cs.AIOct 27, 2026

ATEMPO: Amortized Transfer Entropy Model Per-lag Over-Time

Authors: Emma Andrews, Konstantinos Kontras, Keane Ong, Maarten De Vos, Gianmarco Mengaldo

Organizations: National University of Singapore · KU Leuven · MIT

Abstract

When one time series informs another, the delay at which that information arrives determines if models can utilize this relationship effectively. A source may be strongly predictive at one delay but uninformative at neighboring lags. Therefore, fixed input windows may obscure this structure by bundling informative and irrelevant history together. Transfer entropy (TE) can recover these delays by measuring how much a source's past tells us about a target beyond the target's own history. However, existing TE estimators require separate fits across source and conditioning depths, with further estimation on local windows to track delays over time. We introduce ATEMPO, a diffusion-based transfer entropy estimator amortized over lags and time. ATEMPO samples source and conditioning depths during training and conditions explicitly on temporal position, allowing a single score network to estimate the full TE surface without refitting. Differencing this surface recovers when information arrives. Across synthetic datasets with ground-truth TE, ATEMPO achieves the lowest mean relative error among baselines, and on datasets with known lags, ATEMPO achieves 0.98±0.040.98\pm0.04 lag-detection accuracy. ATEMPO requires a single fit across source depths K=6K=6 and target depths L=3L=3, whereas existing estimators require 18 separate fits to evaluate the same K×LK \times L grid. On three systems with time-varying lag, ATEMPO is the only estimator to maintain perfect lag-detection accuracy across all evaluated window lengths. These recovered delays improve downstream forecasting: ATEMPO-selected inputs reduce MSE on WEATHER-5K by 7.25%, while 120 selected inputs outperform the benchmark's 240-feature default by 4.6%. Across three brain-computer interface tasks, using ATEMPO-selected lags achieves the highest or equal-best R2R^2, improving mean R2R^2 by 4.1% relative to the strongest competing baseline.

Figures & tables

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Nov 5, 2025

Efficient Linear Attention for Multivariate Time Series Modeling via Entropy Calibration

Attention mechanisms are effective for multivariate time series modeling because they adaptively aggregate information across timestamps and variables. Their quadratic cost, however, limits their use in multivariate and spatio-temporal systems. Linear attention reduces this cost by replacing nonlinear attention mappings with forms that support associative computation. Replacing exponential weighting may weaken the amplification of high-score positions and further bias weakly selective distributions toward uniform aggregation. We propose an entropy-calibrated linear attention module and instantiate it as the ELinFormer forecaster. Our operator decomposes aggregation into a uniform component and a query-dependent correction, whose magnitude is calibrated by a query-specific temperature. We derive a second-order KL proxy from centered query-key scores. The Taylor expansion reduces the proxy to the squared-score energy divided by the squared temperature. This score energy can be computed from shared products of the centered keys without forming the full pairwise score matrix, and the same products support the linear attention output. We design a smooth temperature rule to calibrate the KL proxy: it keeps low-energy residuals vanishing, approximately calibrates moderate-energy queries to a shared KL level, and allows the proxy to continue growing for high-energy queries. Experiments on five PEMS benchmarks show competitive forecasting accuracy. Controlled comparisons with vanilla attention and Linformer demonstrate a favorable accuracy--memory trade-off, while analysis on PEMS08 shows that the proposed rational temperature rule and query-specific calibration achieve lower MAE than a linear rule or shared-temperature alternatives. These results support entropy calibration as an effective mechanism for preserving selective deviations in linear attention.
May 24, 2026

AME-TS: Anchored Mixture-of-Experts for Time Series Forecasting

Time series forecasting models are increasingly scaled through large Transformer backbones, yet most existing approaches process all series through a shared dense computation path despite substantial heterogeneity in temporal structure. Mixture-of-Experts (MoE) offers a natural alternative by enabling conditional computation, but standard MoE routing leaves expert specialization weakly identified and often unstable during downstream adaptation. We propose AME-TS, a structure-guided sparse time series foundation model that aligns expert routing with interpretable temporal structure. AME-TS first uses a lightweight regime predictor to estimate series-level descriptors, including forecastability, seasonality, trend, and sparsity, and maps them to a soft structural prior over experts. This series-level prior guides token-level routing during training, encouraging structure-aligned specialization. On the GIFT-Eval benchmark, AME-TS delivers a strong accuracy-efficiency tradeoff across model scales: it substantially outperforms existing time series foundation models at small model scales and remains competitive with the strongest models at larger scales, while activating substantially fewer parameters through sparse routing. We further show that AME-TS learns more interpretable routing geometry and substantially more stable expert specialization than standard MoE during fine-tuning on the M5 dataset. These results suggest that structure-aware routing is an effective and reliable way to realize the benefits of sparse expert models for time series forecasting.
Apr 25, 2026

TEMPO: Transformers for Temporal Disease Progression from Cross-Sectional Data

Event-Based Models (EBMs) infer biomarker progression from cross-sectional data but typically only as ordinal sequences and rely on rigid model assumptions. We propose Tempo, a Transformer architecture that learns both ordinal and continuous event sequences through simulation-based supervised learning. Tempo uses two Transformer modules: one treats biomarkers as tokens to infer event sequencing; the other treats patients as tokens, representing each by their per-biomarker abnormality profile, to infer patients' disease stages. On synthetic benchmarks, Tempo reduces normalized Kendall's Tau distance by 52.89% and staging MAE by 25.33% compared to state-of-the-art SA-EBM, with larger reductions in high-dimensional settings (58.88% and 61.10%). Applied to ADNI, Tempo recovers a biologically plausible Alzheimer's progression: early medial temporal atrophy, followed by amyloid accumulation and cognitive decline, and late-stage tau pathology with terminal acceleration of global neurodegeneration -- broadly consistent with established disease models. Tempo also eliminates the need to derive custom inference algorithms and enables rapid empirical comparison of generative hypotheses.