Overview

Autoregressive models represent a sequence as successive predictions conditioned on earlier elements, enabling both forecasting and generation. Common approaches include Transformer and state-space models, order-agnostic or masked prediction, coarse-to-fine generation, and continuous-latent models. They are used for tasks ranging from language and speech to weather, motion, and physical simulation, with evaluations including long-horizon rollouts and benchmarks such as ERA5, M5, and LibriSpeech. A central challenge is balancing accurate, efficient stepwise prediction with error accumulation, stability, and reliable behavior over long horizons or under distribution shift.

Latest papers

91 papers8 papers in the last four weeks, up 60% on the four weeks before. 0.1% of all new papers.

Jun 4, 2025

Temporal horizons in forecasting: a performance-learnability trade-off

When training autoregressive models to forecast dynamical systems, a critical question arises: how far into the future should the model be trained to predict for optimal performance? In this work, we address this question by analyzing the relationship between the geometry of the loss landscape and the training time horizon. Using dynamical systems theory, we prove that loss minima for long horizons generalize well to short-term forecasts, whereas minima found on short horizons result in worse long-term predictions. However, we also prove that the loss landscape becomes rougher as the training horizon grows, making long-horizon training inherently challenging. We validate our theory through numerical experiments and discuss practical implications for selecting training horizons. Our results provide a principled foundation for hyperparameter optimization in autoregressive forecasting models.