The Unused Input: The Initial State as a Conditioning Interface for Recurrent TSFMs
Organizations: NXAI GmbH, Linz, Austria · Institute for Machine Learning, Johannes Kepler University, Linz, Austria
Abstract
Time series foundation models (TSFMs) forecast unseen series without task-specific training. Increasingly they are built on recurrent backbones, whose constant per-step cost makes long-context inference cheap, and which, unlike transformers, carry a state. A recurrent backbone computes its prediction from two inputs: the observed context window and the initial recurrent state. In practice only the first is utilised and the initial state is unused by convention. We ask whether the initial state can be utilised in recurrent TSFMs. Using the xLSTM-based zero-shot forecaster TiRex as a case study, we freeze the backbone and train only a small head that writes a context vector into the initial state, so that any change in accuracy is attributable to the injected state rather than to adapted weights. A spectral summary of the input window, learned once and applied unchanged to every series, lowers mean GiftEval MASE by 0.175%, with every seed improving, which neither a parameter-matched LoRA adapter nor a learnable static state reproduces. An embedding of text attached to each series, evaluated on six MoTime datasets, reveals two roles for the state. As a per-dataset adaptation surface, a static state lowers MASE by 0.04-6.96% of the zero-shot baseline (8-70% of the gain of full fine-tuning). As a channel for exogenous information, the text-conditioned state lowers it by a further 0.15-0.41% of the zero-shot baseline on four of six datasets and, combined with a LoRA adapter, exceeds full fine-tuning, which has no access to the text, on one of them. In TiRex, the effect concentrates in the mLSTM blocks and is strongest at short context. The results are preliminary and the improvements small, but they point to the initial state as a direction for further research.
Figures & tables
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | Value |
|---|---|
| Blocks | , alternating; sLSTM at indices , , , , , |
| Embedding dimension | |
| mLSTM heads | ; query/key dimension , value dimension |
| sLSTM heads | ; head dimension |
| Input and output patch size | , stride |
| Maximum context | steps |
| Variant | sLSTM part | mLSTM part | Total | |
|---|---|---|---|---|
| Spectral head (GiftEval) | ||||
| Text head (MoTime) | ||||
| Static state | – |
| Rank | Parameters | Matched against |
|---|---|---|
| text head, | ||
| spectral head, | ||
| LoRA plus text head, | ||
| doubled-budget control for rank |
| Dataset | Series length | Series kept | Train | Validation | Test | Windows | Window step | Seasonality |
|---|---|---|---|---|---|---|---|---|
| MovieLens | ||||||||
| News-FO | variable | |||||||
| PixelRec Full | ||||||||
| PixelRec Sub | ||||||||
| Tianchi | ||||||||
| Wiki People |
| Dataset | Initial | Best | Reduction |
|---|---|---|---|
| MovieLens | |||
| News-FO | |||
| PixelRec Full | |||
| PixelRec Sub | |||
| Tianchi | |||
| Wiki People |
| Setting | Value |
|---|---|
| Optimiser | AdamW, weight decay |
| Schedule | one-cycle, peak learning rate , warm-up fraction , no momentum cycling |
| Precision | bfloat16 mixed |
| Batch size | , no gradient accumulation |
| Loss | quantile loss over the nine levels to , label offset of one output patch, no per-sample impact cap |
| Hardware | one H100 per run |
| Arm | Trainable parameters | Backbone |
|---|---|---|
| State-init head, static state | state initialiser only | frozen |
| LoRA | adapter factors only | frozen |
| LoRA and state init | adapter factors and state initialiser | frozen |
| Full fine-tuning | all parameters | updated |
| Zero-shot | none, evaluation only | frozen |
| Dataset | Train series | Steps | Samples drawn | Passes |
|---|---|---|---|---|
| MovieLens | ||||
| News-FO | ||||
| PixelRec Full | ||||
| PixelRec Sub | ||||
| Tianchi | ||||
| Wiki People |
| Method | MASE | CRPS | |||
|---|---|---|---|---|---|
| FFT init | 6 | ||||
| static prior | 6 | ||||
| LoRA r23 | 6 | ||||
| LoRA r23 + FFT | 6 | ||||
| LoRA r46 | 6 | ||||
| Zero-shot baseline ( ) | 1 | – | – |
| Injection locus | MASE | CRPS | |||
|---|---|---|---|---|---|
| both | 6 | ||||
| mLSTM only | 6 | ||||
| sLSTM only | 6 | ||||
| Zero-shot baseline ( ) | 1 | – | – |
| Injection locus | mean | std | ||||||
|---|---|---|---|---|---|---|---|---|
| MASE | ||||||||
| both | ||||||||
| mLSTM only | ||||||||
| sLSTM only | ||||||||
| Zero-shot baseline | – | – | – | – | – | – | – | |
| CRPS | ||||||||
| Method | MovieLens | News-FO | PixelRec Full | PixelRec Sub | Tianchi | Wiki People | |
|---|---|---|---|---|---|---|---|
| static | 6 | ||||||
| text | 6 | ||||||
| LoRA r12 | 6 | ||||||
| LoRA+text | 6 | ||||||
| LoRA r24 | 6 | ||||||
| full FT | 6 |
| Dataset | Grid points | Context range | Windows | Instances at the shortest point | Instances at the longest point |
|---|---|---|---|---|---|
| MovieLens | to | ||||
| News-FO | to | ||||
| PixelRec Full | to | ||||
| PixelRec Sub | to | ||||
| Tianchi | to | / | |||
| Wiki People | to |
| Dataset | Arm | Control | MASE | (%) | seeds worse | |
|---|---|---|---|---|---|---|
| Tianchi | text | shuffled | 6/6 | |||
| Tianchi | text | mean vector | 6/6 | |||
| Tianchi | LoRA+text | shuffled | 6/6 | |||
| Tianchi | LoRA+text | mean vector | 6/6 | |||
| Wiki People | text | shuffled | 3/6 | |||
| Wiki People | text | mean vector | 3/6 |
| Dataset | Embedding | Pairs | Min | Median | Mean | Max |
|---|---|---|---|---|---|---|
| Tianchi | Qwen3 ( -d) | |||||
| Tianchi | CLIP ( -d) | |||||
| Wiki People | Qwen3 ( -d) | |||||
| Wiki People | CLIP ( -d) | |||||
| ∗ Some Tianchi texts occur more than once ( distinct texts for series). | ||||||
Time-Series Foundation Models
Time-series foundation models are pretrained sequence models designed to transfer across forecasting, classification, anomaly detection, and other temporal tasks.
See all 340 papers