Time-series language models typically align numerical signals with language using labels, captions, or question-answer pairs, without explicitly requiring their representations to preserve how the underlying signals evolve. We introduce forecast grounding, a pretraining principle that requires language-aligned time-series representations to remain predictive of the future evolution before they are adapted to downstream tasks. We instantiate this principle in Lotus, which divides each time-series channel into non-overlapping patches and maps them into the LLM embedding space. It then predicts the next patch from the LLM hidden states of the time-series tokens, encouraging the learned representation to retain temporal information. To capture both temporal changes and periodic patterns, we also propose a temporal-spectral patch representation that combines each raw time-series patch with its spectral representation before mapping it into the LLM embedding space. Across three language model backbones and six downstream evaluation datasets, Lotus achieves the best average performance with minimal trainable parameters, reaching 59.91% accuracy and outperforming the strongest matched baseline by 3.36%. Our representation analyses further show that Lotus uses the numerical input and preserves temporal structure in its learned patch representations. These results establish forecast grounding as a promising pretraining principle, enabling compact time-series language models to outperform models 25-50 times their size on TSRBench. Our code is publicly available at https://github.com/DA2I2-SLM/Lotus
Figures & tables
Figure 1: Six-task average exact match accuracy versus trainable parameters (LoRA). G and Q denote Gemma3 and Qwen3 backbones, respectively.
Figure 2: Overview of Lotus . (a) Architecture: The architecture maps time-series patches to soft tokens that are processed jointly with text by the pretrained language model. (b) Pretraining: During forecast grounding pretraining, language supervision trains textual outputs while next-patch prediction constrains LLM’s hidden states to retain future-relevant information. (c) Downstream Adaptation: During downstream adaptation, the next-patch decoder is removed and the model is trained with language supervision only.
Baseline
# Params
HAR
Sleep
ECG
PAMAP2
MCQ
T/F
Avg.
Reference
Chance
—
12.50
14.29
0.98
8.33
25.00
50.00
18.52
Llama-3.1-8B-Instruct, zero-shot
0
9.26
24.52
18.36
9.83
8.74
9.42
13.35
Qwen2.5-7B-Instruct, zero-shot
0
15.97
31.40
31.76
9.40
10.02
3.34
16.98
Mistral-7B-Instruct-v0.3, zero-shot
0
7.07
21.40
21.31
6.84
24.31
24.62
17.59
Gemma3-4B-pt (image), zero-shot
0
1.97
0.00
0.00
0.00
17.48
31.61
8.51
Gemma3-4B-pt (image), FT
14.9M
51.31
61.18
30.94
51.71
46.10
70.85
52.02
Table 1: Accuracy on downstream tasks. Avg. is the mean over the six splits; # Params counts trainable parameters in downstream adaptation. Per backbone block, bold is best and underline second best per column; † marks OpenTSLM’s original evaluation protocol ( Langer et al., 2026 ) .
Figure 3: TSRBench overall accuracy . Per-subset results are in Appendix’s Table 4 .
Figure 4: Component and input analyses on Gemma3-270M. (a) Six-task average under ablations of pretraining and the frequency branch. (b) Sleep-EDF accuracy after removing individual frequency bands from the test signal. In-bar labels show percentage-point drops.
Figure 5: Representation analysis. (a) Pairwise patch similarities. Green boxes highlight patterns shared by Lotus and the raw signal; red boxes mark corresponding regions where these patterns are less apparent with cross-entropy (CE)-only training. (b) Patch colors indicate cosine distance from the window-average representation. Boxes highlight spans selected for comparison.
Table 2: Configuration of the two phases. Settings shared by both phases are listed first.
Variant
HAR
Sleep
ECG
PAMAP2
MCQ
T/F
Avg.
Lotus
58.38
70.54
33.16
55.15
39.41
74.47
55.19
w/o frequency branch (Phase 2 only)
61.55
69.46
25.80
41.45
37.10
72.95
51.39
w/o frequency branch (both phases)
55.89
67.54
25.24
44.44
37.31
72.64
50.51
Language-only Phase 1 (same budget)
54.38
66.45
30.84
48.72
39.23
71.16
51.79
Appendix
Table 3: Ablations on Gemma3-270M, per task. Each variant changes one ingredient and is otherwise trained and evaluated as in Table 1 . Bold is best per column.
Figure 6: Input diagnostics. Accuracy after corrupting the numerical input, on the Gemma3-270M checkpoint and the full test splits (HAR n=8,222 , Sleep-EDF n=930 ).
Perception
Reasoning
Prediction
Decision
Model
PR
NU
AD
CA
ER
CD
AR
TR
NR
IR
TSF
EP
QualDM
QuantDM
Overall
Qwen2.5-7B
40.3
50.0
43.3
64.0
17.7
28.2
32.0
20.0
32.1
34.3
31.8
42.0
32.6
19.4
32.9
Qwen3-1.7B
39.1
51.2
33.9
48.5
16.0
35.3
42.7
28.1
28.0
32.0
40.5
69.4
32.5
17.7
36.3
Qwen2.5-3B
40.7
40.7
33.1
46.9
25.4
20.1
24.0
28.0
33.3
20.2
32.4
66.9
34.0
20.0
34.1
Qwen2.5-VL-3B
56.1
56.0
45.0
53.1
28.0
17.1
26.7
28.8
30.0
32.0
26.5
66.7
33.4
23.7
34.7
InternVL3.5-1B
47.0
40.2
34.9
48.2
23.1
27.7
24.7
27.5
27.0
25.0
39.6
27.6
32.2
20.3
32.3
Appendix
Table 4: Per-task TSRBench accuracy (%) over 14 subsets . Columns are grouped into perception, reasoning, prediction, and decision-making.
Figure 7: Qualitative example, Sleep-EDF . All three baselines read the epoch as wakefulness; Lotus places it in a lighter sleep stage, consistent with its spectral centroid.
Figure 8: Qualitative example, HAR . Lotus and OpenTSLM-SP share their first 360 characters verbatim and differ only in the final label; OpenTSLM-Flamingo describes a stepping motion that the Z-axis spectrum does not show.
Figure 9: Qualitative example, ECG-QA . The q wave and ST depression cited by two baselines are present in the recording but are not the queried finding; Lotus identifies the reduced T-wave amplitude in lead I.
Figure 10: Qualitative example, PAMAP2 . OpenTSLM-SP misses the three shaded excursions and answers sitting. Per-channel normalization removes the gravity reference, so ascent and descent cannot be distinguished from the input; the example illustrates the missed motion rather than the direction.
Generator family
Series
Share
Variates
Length
Chronos-2 multivariatizer
310,000
36.7%
3 – 10
512 – 8,192
KernelSynth (univariate)
299,999
35.5%
1
1,024 – 4,096
TimePFN LMC
215,000
25.5%
3 – 10
512 – 2,048
SymTime S2Generator
19,276
2.3%
3 – 10
1,024 – 4,096
Total
844,275
100%
1 – 10
512 – 8,192
Appendix
Table 5: Composition of the synthetic training corpus. Variate and length ranges are observed after collection and filtering.
Spectral
Seasonal
Trend
Cross-
Source
n
entropy
ACF (1)
peak
R2
channel
Chronos-2 multivariatizer
1,204
0.51±0.14
0.63±0.27
0.78±0.14
0.22±0.16
0.36±0.14
KernelSynth (univariate)
1,206
0.40±0.22
0.77±0.34
0.85±0.27
0.10±0.24
–
TimePFN LMC
1,202
0.45±0.14
0.66±0.28
0.80±0.16
0.07±0.12
0.59±0.20
SymTime S2Generator
1,204
0.75±0.16
0.37±0.22
0.41±0.21
0.10±0.13
0.24±0.19
Australian electricity (real)
1,200
0.40±0.02
0.98±0.00
0.93±0.02
0.03±0.02
0.80±0.07
Appendix
Table 6: Phase 1 dataset statistics , by generator family. Mean ± standard deviation over sampled series; real series are cut into 2,048 -point windows. Spectral entropy is spectral flatness ( 0 a pure tone, 1 white noise); seasonal peak the largest ∣ACF∣ over lags [2,L/4] ; trend R2 the variance share of a least-squares line; cross-channel the mean ∣ correlation ∣ over variate pairs, undefined when univariate.
Corpus
Train
Val
Test
# Train Per Epoch
Multiplicity
HAR
68,542
8,718
8,222
40,000
0.58× (fixed subset)
Sleep-EDF
7,434
930
930
40,000
5 – 6×
ECG-QA
159,313
31,137
41,093
40,000
0.25× (fixed subset)
PAMAP2
4,207
234
234
40,000
9 – 10×
TimeMQA
33,764
1,876
1,876
40,000
1 – 2×
Total
273,260
200,000
Appendix
Table 7: Phase-2 corpora. Natural split sizes and the balanced per-epoch count; multiplicity is how often an example is seen per epoch, with corpora larger than the target truncated to a fixed subset.
Figure 11: Inference scaling with series length , all baselines on Gemma3-270m (single series, one 10 -token generation per point; peak allocated memory). Left: Lotus is cheapest at every length; OpenTSLM-SP’s line terminates at its architectural length ceiling. Right: context positions consumed, L/32 against L/4 .
Sleep-EDF ( 1×1500 )
ECG-QA ( 12×1000 )
Interface
tok
s/ex
GB
tok
s/ex
GB
Lotus (ours)
47
0.26
0.63
384
0.28
0.65
OpenTSLM-SP
375
0.19
2.44
3000
0.85
16.88
OpenTSLM-Flamingo
const.
0.22
9.14
const.
0.23
9.11
Tokenized
7239
3.48
22.78
7582
3.76
23.82
Appendix
Table 8: Phase-2 cost per example on one GPU (Gemma3-270m, batch 1 ). tok is the context the series occupies, constant for Flamingo, whose perceiver compresses to 64 latents; s/ex is one training step and GB peak training memory.
Backbone
Encoder
GPUs
Batch
Peak
Wall clock
GPU-h
Epochs
/device
eff.
VRAM
Gemma3-270M
Transformer Decoder
2×4
16
128
31.3 GB
26 h
210
0.7
Qwen3-0.6B
Transformer Decoder
2×4
8
64
9.1 GB
5.0 d
960
2.1
Gemma3-1B
Transformer Decoder
4×4
8
128
15.2 GB
10.0 d
3,840
6.2
Total
5,010
Appendix
Table 9: Phase 1 training runs . Wall-clock time, allocation, and GPU-hours are from scheduler records. An epoch is one pass through the weighted sample drawn from the four corpora, not through every available example. Peak VRAM is the per-device maximum from a single-process V100 run at the listed per-device batch size.
Figure 12: Phase-1 forecasting prompt, shown for a 1,024 -step window of electricity demand. A multivariate item from the synthetic corpus takes the same form with multivariate , its channel count, and one block per channel. The answer field is the end-of-sequence token; the training signal is the next-patch regression rather than any token in this prompt.
Figure 13: Prompt template for HAR.
Figure 14: Prompt template for PAMAP2. The template matches HAR except for the eighteen-way label set.
Figure 15: Prompt template for Sleep-EDF. Unlike the accelerometer corpora, the class list is placed in the post-prompt.
Figure 16: Prompt template for ECG-QA. The clinical-context paragraph is assembled from the record’s metadata; the answer set in the post-prompt is that of the question template.
Figure 17: Prompt template for TimeMQA. The corpus is open-ended, so the post-prompt is a bare Answer: . In comparison questions the first segment enters through the time-series interface while the second is inlined as digits, so the model compares a soft-token series against a textual one.
Figure 18: Standard TSRBench template. Options are labelled A – G in the order the benchmark stores them.
Figure 19: Temporal relation reasoning. The subset ships an explicit timestamp channel, which would otherwise be encoded as though it were a signal.
Figure 20: Abductive reasoning. A window of ten steps either side of the critical moment is kept. The win-probability channels are given twice, once serialised in the table and once through the encoder, which is the benchmark’s own design and the one setting where our interface receives the signal redundantly.
Can language-pretrained transformers become effective time-series forecasters, and why? In this paper, we show that cross-modal transfer arises because language pretraining preconditions time series training with a reusable manifold. A linear probe on frozen LLM states decodes realistic time-series trajectories without paired supervision, and retrieval in this projected space yields competitive forecasts, showing that structure and dynamics exist before finetuning. Pretrained initialization also improves optimization, producing coherent gradients and a highly anisotropic loss landscape unlike random initialization. Finetuning then acts as low-dimensional alignment, reusing existing directions rather than learning temporal primitives from scratch, as evidenced by low-rank updates, subspace alignment, and shared features for periodicity, trend, and repetition. Together, these results support a geometric account of LLM-to-time-series transfer: language pretraining builds the manifold, and finetuning projects numerical dynamics onto task-relevant directions.
Adapting pretrained Large Language Models (LLMs) for time series forecasting primarily relies on token-level linguistic-temporal alignment, leading to the stacking of logically disjointed tokens as input. While empirically effective, these methods overlook a fundamental capability of LLMs: modeling linguistic logic and structure, rather than merely processing token features. To address this limitation, we propose the Markovian-Guided Structure-Aware Alignment (MGSAA). Our core contribution is a framework that transcends pointwise feature matching to achieve global structural isomorphism between the linguistic and temporal domains. Specifically, MGSAA distills latent evolutionary patterns of language within LLMs into a Markovian state transition graph, which is transferred as a structural prior to the time series domain. Under this prior, time series patches are decoded into latent states and then aligned via state-constrained cross-attention. Ultimately, MGSAA generates a token sequence topologically isomorphic to the LLM's inherent mental structure, reactivating its reasoning capabilities for forecasting. Comprehensive evaluations across multiple benchmarks demonstrate that MGSAA achieves state-of-the-art performance, providing an innovative solution for cross-modal alignment in LLM for time series forecasting. Code is available at https://github.com/sunzju/MGSAA.
While Next-Token Prediction (NTP) has unified LLM pretraining, its adaptation to unbounded, continuous time series (TS) remains open. To bridge the gap, we introduce UniTok, a universal tokenizer that transforms TS into discrete tokens, and UniTok-FM, a foundation model pretrained via NTP on these tokens. UniTok-FM is a general-purpose foundation model that supports zero-shot and prompt-boosted forecasting, as well as few-shot generation and classification via training-free in-context inference--a capability not achieved by prior works. Technically, UniTok is a vector-quantized autoencoder incorporating prefix normalization for scale stabilization, a progressive-resolution causal architecture for encoding and decoding, and a structure-preserving reconstruction loss for training. UniTok-FM adopts an off-the-shelf LLM architecture without TS-specific modifications. Instead of pretraining on isolated TS, it performs NTP on context windows formed by multiple series with similar patterns, aiming to capture their shared dynamics. Experiments on forecasting, generation, and classification show that a single unified UniTok-FM consistently outperforms statistical and supervised baselines, achieves competitive performance with task-specific foundation models, and uniquely enables training-free in-context inference across tasks.
Yunhao Zhang, Ruiying Qi, Jiale Zheng +3
Shanghai Jiao Tong University · Huawei Noah’s Ark Lab