Time Flies When You Parallelise: Pareto-efficient Time Series Foundation Models with Linear Recurrences
Organizations: Continuous Intelligence Limited · Heidelberg Institute for Theoretical Studies · University of Nottingham
Abstract
Time series foundation models have demonstrated strong zero-shot forecasting performance, but computational efficiency remains a central architectural challenge. Temporal self-attention incurs quadratic cost in the number of temporal tokens, while nonlinear recurrent state transitions restrict parallel computation along the temporal dimension. We introduce ci-proto, a family of time series foundation models based on Mamba-3, whose linear state recurrences support chunk-wise parallel-in-time computation during training, and inference costs that scale linearly with sequence length. The architecture alternates recurrent temporal processing with cross-variate attention, supporting multivariate probabilistic forecasting with past-only and future-known covariates. Across fev-bench's 100 forecasting tasks, ci-proto XL sets a new state of the art among time series foundation models. The five ci-proto variants also make up the entire Pareto frontier for mean scaled quantile loss and mean absolute scaled error against both parameter count and median runtime.
Figures & tables
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Configuration | XS | S | M | L | XL |
| Number of blocks | 6 | 8 | 10 | 12 | 16 |
| Hidden dimension | 128 | 192 | 320 | 512 | 640 |
| Patch length | 32 | 32 | 32 | 32 | 32 |
| Mamba-3 state dimension | 128 | 128 | 128 | 128 | 128 |
| Mamba-3 head dimension | 64 | 64 | 64 | 64 | 64 |
| Variate-attention heads | 4 | 4 | 4 | 4 | 4 |
| Setting | Value |
|---|---|
| Optimiser and coefficients | AdamW; cycled between and by the one-cycle schedule ( at the peak learning rate), , |
| Learning rate and schedule | Cosine one-cycle schedule with warm-up steps; peak LR (3.5M–28.6M) and (82.5M, 167.2M); initial LR , final LR |
| Effective batch size | sampled time series groups per optimiser step |
| Training steps per variant | |
| Weight decay and gradient clipping | Weight decay , excluding Mamba parameters marked for no decay; elementwise gradient clipping to |
| Numerical precision | BF16 mixed precision with FP32 parameters |
Time-Series Foundation Models
Time-series foundation models are pretrained sequence models designed to transfer across forecasting, classification, anomaly detection, and other temporal tasks.
See all 344 papers