OliO: ODE-based Linear Transition Operator for Self-Supervised Time Series Forecasting

Kwangryeol Park; Seulki Lee

cs.AIOct 16, 2026

OliO: ODE-based Linear Transition Operator for Self-Supervised Time Series Forecasting

Authors: Kwangryeol Park, Seulki Lee

Organizations: Graduate School of Artificial Intelligence, Ulsan National Institute of Science and Technology, Ulsan, South Korea · School of Electrical Engineering, Korea Advanced Institute of Science and Technology, Daejeon, South Korea

Abstract

Self-supervised representation learning has advanced time series forecasting by capturing robust features from unlabeled data. However, existing contrastive and masking-based methods often struggle to explicitly model the underlying temporal flow, either by treating time lag as noise or by disrupting the inherent temporal dynamics essential for forecasting. To address these limitations, we propose OliO (ODE-based Linear Transition Operator), a plug-and-play self-supervised learning method that learns bidirectional temporal flows as a continuous and structurally consistent dynamical system. OliO introduces a transition operator derived from linear Ordinary Differential Equations (ODEs) to align latent representations across arbitrary time shifts. By imposing a strictly upper triangular constraint on the transition matrix, we mathematically ensure numerical stability and enforce a robust polynomial boundary that effectively captures long-term dependencies while preventing the exponential divergence typically found in unconstrained ODE-based models. This structural constraint induces a temporally coherent representation space that preserves the underlying flow essential for accurate forecasting. Extensive experiments across various backbone architectures and benchmarks demonstrate that OliO achieves significant performance gains, providing Mean Squared Error (MSE) reductions of up to 5.75% over the most competitive grid-searched SOTA baselines. The code is available at https://github.com/eai-lab/OliO

Figures & tables

Panel of Figure 1 — Figure 1: Visual comparison of the pseudo output $h$ from the pretrained iTransformer [ 15 ] backbone using UMAP [ 17 ] (top row) and PHATE [ 21 ] (bottom row) on the Weather dataset [ 31 ] . The color gradient represents the progression of time. While baseline methods (SimMTM [ 6 ] , TS2Vec [ 32 ] , SoftCLT [ 12 ] ) yield scattered or tightly clustered results that collapse temporal order, OliO maintains a continuous trajectory, effectively capturing the underlying temporal flow.

Figure 2: Overall architecture of the OliO framework. (a) Input windows

x

and

x_{\tau}

are encoded by a shared backbone into pseudo-outputs. (b) The forward shift operator aligns

h

h_{\tau}

via the matrix exponential

e^{\tau K}

, modeling the transition from past to future. (c) Conversely, the backward shift operator restores the temporal flow from future to past, ensuring bidirectional transition consistency.

Figure 3: Relative performance improvement of our proposed method, OliO, compared to state-of-the-art SSL baselines. The performance index is defined as

MSE_{\text{Rand}}/MSE_{\text{Meth.}}

across varying prediction lengths, where a value of 1.0 represents the random initialization baseline, and higher values indicate better performance . Solid lines indicate the mean index aggregated across three backbone architectures (PatchTST, iTransformer, TimeVLM) and all datasets, with shaded regions representing the min-max performance envelope. Note the truncated Y-axis

[0.96,1.08]

for focused visualization of performance gains.

Dataset	TimesURL	Mask Recon.	OliO
ETTh1	0.4096	0.3958	0.3937
ETTh2	0.3501	0.3474	0.3420
ETTm1	0.3461	0.3335	0.3329
ETTm2	0.2566	0.2461	0.2461
Weather	0.2357	0.2134	0.2182
ECL	0.2556	0.2476	0.2325

Table 4: Zero-shot forecasting results with the LightGTS-4M time-series foundation model [ 29 ] . Mask Recon. denotes its default mask-reconstruction pre-training. Values average four forecasting horizons. Best results are in bold , and tied best values are underlined . Lower MSE is better.

Table 5

Figure 4: Approximation speedup. Latency improvement relative to full matrix exponential.

Table 10: Relative approximation error between the complete transition and its first-order approximation.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Latent space visualizations of the PatchTST backbone on ETTh1 and ETTh2 datasets using UMAP (top) and PHATE (bottom). For ETTh2, we utilize the causal pretraining configuration. The continuous trajectories in the PHATE plots demonstrate that OliO effectively preserves the temporal flow.

Panel of Figure 6 — Figure 6: Latent space visualization of PatchTST on the ETTm1 dataset, comparing UMAP and PHATE embeddings.

Panel of Figure 7 — Figure 7: Latent space visualizations of iTransformer on ETTm1 and ETTm2 datasets.

Panel of Figure 8 — Figure 8: Latent space visualizations of TimeVLM on ETTh1 and ETTh2 datasets.

Panel of Figure 9 — Figure 9: Latent space visualizations of TimeVLM on ETTm1 and ETTm2 datasets.

Panel of Figure 10 — Figure 10: Latent space visualization of TimeVLM on the Weather dataset, comparing UMAP and PHATE embeddings.

Figure 11: Speedup of the OliO module’s

p

-th order approximation relative to the full matrix exponential (computed via torch.matrix_exp ). The benchmark evaluates the computational efficiency of the TriangularCausalShift module using 4,096 samples per operation, with values averaged over 5 independent trials on an NVIDIA RTX 3090 GPU. The first-order approximation (

p=1

) provides an acceleration of approximately 3.5

\times

to 4.5

\times

compared to the exact computation, highlighting the high-throughput capability of OliO in processing high-dimensional temporal representations.

Backbone	Dataset	TimesURL	OliO
PatchTST	Electricity	0.1846 $\pm$ 0.0012	0.1833 $\pm$ 0.0004
	ETTh1	0.4147 $\pm$ 0.0021	0.4107 $\pm$ 0.0009
	ETTh2	0.3409 $\pm$ 0.0006	0.3461 $\pm$ 0.0011
	ETTm1	0.3546 $\pm$ 0.0009	0.3616 $\pm$ 0.0012
	ETTm2	0.2550 $\pm$ 0.0006	0.2528 $\pm$ 0.0005
	Weather	0.2469 $\pm$ 0.0015	0.2439 $\pm$ 0.0005

Appendix

Table 13: Frozen-backbone linear probing results. Each dataset value averages four forecasting horizons. Lower MSE is better.

Figure 12: Summary of relative few-shot forecasting performance for PatchTST backbone across 1%, 5%, and 10% settings. Performance is normalized relative to OliO (

=1.0

), where higher values indicate better forecasting quality (lower MSE).

Figure 13: Summary of relative few-shot forecasting performance for iTransformer backbone across 1%, 5%, and 10% settings. Performance is normalized relative to OliO (

=1.0

), where higher values indicate better forecasting quality (lower MSE).

Figure 14: Summary of relative few-shot forecasting performance for TimeVLM backbone across 1%, 5%, and 10% settings. Performance is normalized relative to OliO (

=1.0

), where higher values indicate better forecasting quality (lower MSE).

Panel of Figure 19 — Appendix figure: detailed performance improvement heatmaps for each backbone architecture. The values represent the mean performance index (Random MSE / Method MSE) aggregated across six benchmark datasets. Dark blue cells indicate higher performance indices.

Panel of Figure 20 — Appendix figure: performance improvement heatmaps aggregated over all backbone architectures for each benchmark dataset. The values represent the mean performance index (Random MSE / Method MSE). These visualizations summarize how relative forecasting performance varies across datasets and prediction horizons.

Matrix Type	Order ( $p$ )	ETTh1	ETTh2	ETTm1	ETTm2	Weather	ECL
Unconstrained (Full)	Exact	NaN	NaN	NaN	NaN	NaN	NaN
Nilpotent (OliO)	Exact	.4260	.3562	.3616	.2653	.2378	.1664
Nilpotent (OliO)	$p=1$	.4434	.3661	.3752	.2674	.2470	.2137
Nilpotent (OliO)	$p=2$	.4315	.3624	.3695	.2661	.2480	.2063
Nilpotent (OliO)	$p=3$	.4315	.3606	.3686	.2678	.2382	.2029
Nilpotent (OliO)	$p=4$	.4325	.3629	.3720	.2675	.2457	.2051

Appendix

Table 40: Full computation analysis: evaluation of transition matrix constraints and approximation order

p=1\dots 5

. Values represent the average MSE across all backbones.

Setting	90%	80%	60%	20%
PatchTST–ETTh1	0.421	0.418	0.417	0.417
PatchTST–ETTh2	0.343	0.343	0.343	0.344
iTransformer–ETTm1	0.404	0.406	0.403	0.400
iTransformer–ETTm2	0.299	0.305	0.306	0.306

Appendix

Table 41: Forecasting MSE under different overlap ratios. Lower is better.

Setting	w/o OliO	100% Overlap	0% Overlap
PatchTST–ETTh1	0.4189	0.4178	0.4179
PatchTST–ETTh2	0.3594	0.3416	0.3437
PatchTST–Weather	0.2297	0.2299	0.2287
iTransformer–ETTm1	0.4097	0.4068	0.4060
iTransformer–ETTm2	0.2917	0.2889	0.2908
iTransformer–Weather	0.2615	0.2871	0.2668

Appendix

Table 43: Comparison of models without OliO, OliO with full-overlap pairs, and OliO with zero-overlap pairs. Results are averaged over three seeds. Lower MSE is better.

Backbone	Overlap Mask	Unique Mask	Gap	Wins
PatchTST	0.29818	0.29767	+0.17%	9 / 15
iTransformer	0.35022	0.34932	+0.26%	14 / 10
Overall	0.32420	0.32349	+0.22%	23 / 25

Appendix

Table 44: Matched masking control at 50% overlap. MSE is averaged over ETTh1, ETTh2, ETTm1, ETTm2, Weather, and ECL and horizons

\{96,192,336,720\}

(24 settings per backbone; 48 overall). “Wins” reports Overlap-Mask / Unique-Mask wins. Lower MSE is better.

Lag $\tau$	ETTh1	ETTh2
1	0.018	0.000
10	0.617	0.034
20	0.857	0.104
30	0.931	0.182
40	0.965	0.258
50	0.986	0.331

Appendix

Table 46: Relative approximation error between the complete transition and its first-order approximation across temporal lags.

Time Series Forecasting

Time series forecasting predicts future values from ordered observations using statistical, neural, and probabilistic models to support decisions under uncertainty.

See all 873 papers

WinoTS: Wavelet-based Self-Distillation for Time Series Models
Sep 30, 2026Noam Major, Kathy Razmadze, Yoli ShavitSelf-Supervised Time Series Representation Learning Time Series Data Augmentation
Bar-Ilan University, Ramat-Gan, Israel
Time-o1: Time-Series Forecasting Needs Transformed Label Alignment
May 23, 2025Hao Wang, Licheng Pan, Zhichao Chen +5Long-Term Time Series Forecasting Time Series Forecasting
Xiaohongshu Inc. · State Key Lab of General AI, School of Intelligence Science and Technology, Peking University · Gaoling School of Artificial Intelligence, Renmin University of China +4
CF-JEPA: Mask-free forward prediction with asymmetric encoder utilization for time-series representation learning
Jun 5, 2026Jaehoon Lee, Sunghyun SimMultivariate Time Series Forecasting Time Series Classification
aGraduate School of Artificial Intelligence Convergence Engineering, Changwon National University, Changwon, South Korea · bDepartment of Artificial Intelligence Engineering, Changwon National University, Changwon, South Korea

Sep 30, 2026J/K move · Enter open · S save

WinoTS: Wavelet-based Self-Distillation for Time Series Models

Noam Major, Kathy Razmadze, Yoli Shavit

Bar-Ilan University, Ramat-Gan, Israel

Self-Supervised Time Series Representation Learning Time Series Data Augmentation Self-Distillation

Open paper AskPDF arXiv

Self-supervised pre-training of time series models is currently dominated by next-token prediction and reconstruction objectives. In continuous-valued domains, these paradigms often waste model capacity on high-frequency, point-wise noise at the expense of learning invariant structure. While invariance-based self-distillation has proven highly effective in computer vision, its application to temporal data remains largely underexplored. Effectively adapting such methods to time series requires carefully designed augmentations: spatial operations like cropping can shift the timing of repeating cycles or distort the signal, while basic jittering may provide limited variation. We introduce Wavelet-based self-distillation for time series (WinoTS), an invariance-based pre-training paradigm designed specifically for temporal signals. At its core, WinoTS leverages time-frequency augmentations to construct multi-scale structural views without distorting underlying signal dynamics. Across extensive evaluations, WinoTS outperforms state-of-the-art baselines in long-term forecasting, cross-domain zero-shot transfer, and unsupervised anomaly detection. Notably, linear probing on frozen WinoTS representations frequently surpasses fully supervised models trained from scratch. Systematic ablations demonstrate that WinoTS is a flexible, architecture-agnostic framework yielding gains across time series backbones, and establish that time-frequency transformations provide a principled alternative to vision-style spatial augmentations.

Abstract

Time Series Forecasting

Explore similar work

WinoTS: Wavelet-based Self-Distillation for Time Series Models

Time-o1: Time-Series Forecasting Needs Transformed Label Alignment

CF-JEPA: Mask-free forward prediction with asymmetric encoder utilization for time-series representation learning