OliO: ODE-based Linear Transition Operator for Self-Supervised Time Series Forecasting

Kwangryeol Park; Seulki Lee

cs.AIOct 16, 2026

OliO: ODE-based Linear Transition Operator for Self-Supervised Time Series Forecasting

Authors: Kwangryeol Park, Seulki Lee

Organizations: Graduate School of Artificial Intelligence, Ulsan National Institute of Science and Technology, Ulsan, South Korea · School of Electrical Engineering, Korea Advanced Institute of Science and Technology, Daejeon, South Korea

Abstract

Self-supervised representation learning has advanced time series forecasting by capturing robust features from unlabeled data. However, existing contrastive and masking-based methods often struggle to explicitly model the underlying temporal flow, either by treating time lag as noise or by disrupting the inherent temporal dynamics essential for forecasting. To address these limitations, we propose OliO (ODE-based Linear Transition Operator), a plug-and-play self-supervised learning method that learns bidirectional temporal flows as a continuous and structurally consistent dynamical system. OliO introduces a transition operator derived from linear Ordinary Differential Equations (ODEs) to align latent representations across arbitrary time shifts. By imposing a strictly upper triangular constraint on the transition matrix, we mathematically ensure numerical stability and enforce a robust polynomial boundary that effectively captures long-term dependencies while preventing the exponential divergence typically found in unconstrained ODE-based models. This structural constraint induces a temporally coherent representation space that preserves the underlying flow essential for accurate forecasting. Extensive experiments across various backbone architectures and benchmarks demonstrate that OliO achieves significant performance gains, providing Mean Squared Error (MSE) reductions of up to 5.75% over the most competitive grid-searched SOTA baselines. The code is available at https://github.com/eai-lab/OliO

Figures & tables

Panel of Figure 1 — Figure 1: Visual comparison of the pseudo output $h$ from the pretrained iTransformer [ 15 ] backbone using UMAP [ 17 ] (top row) and PHATE [ 21 ] (bottom row) on the Weather dataset [ 31 ] . The color gradient represents the progression of time. While baseline methods (SimMTM [ 6 ] , TS2Vec [ 32 ] , SoftCLT [ 12 ] ) yield scattered or tightly clustered results that collapse temporal order, OliO maintains a continuous trajectory, effectively capturing the underlying temporal flow.

Figure 2: Overall architecture of the OliO framework. (a) Input windows

x

and

x_{\tau}

are encoded by a shared backbone into pseudo-outputs. (b) The forward shift operator aligns

h

h_{\tau}

via the matrix exponential

e^{\tau K}

, modeling the transition from past to future. (c) Conversely, the backward shift operator restores the temporal flow from future to past, ensuring bidirectional transition consistency.

Figure 3: Relative performance improvement of our proposed method, OliO, compared to state-of-the-art SSL baselines. The performance index is defined as

MSE_{\text{Rand}}/MSE_{\text{Meth.}}

across varying prediction lengths, where a value of 1.0 represents the random initialization baseline, and higher values indicate better performance . Solid lines indicate the mean index aggregated across three backbone architectures (PatchTST, iTransformer, TimeVLM) and all datasets, with shaded regions representing the min-max performance envelope. Note the truncated Y-axis

[0.96,1.08]

for focused visualization of performance gains.

Dataset	TimesURL	Mask Recon.	OliO
ETTh1	0.4096	0.3958	0.3937
ETTh2	0.3501	0.3474	0.3420
ETTm1	0.3461	0.3335	0.3329
ETTm2	0.2566	0.2461	0.2461
Weather	0.2357	0.2134	0.2182
ECL	0.2556	0.2476	0.2325

Table 4: Zero-shot forecasting results with the LightGTS-4M time-series foundation model [ 29 ] . Mask Recon. denotes its default mask-reconstruction pre-training. Values average four forecasting horizons. Best results are in bold , and tied best values are underlined . Lower MSE is better.

Table 5

Figure 4: Approximation speedup. Latency improvement relative to full matrix exponential.

Table 10: Relative approximation error between the complete transition and its first-order approximation.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Latent space visualizations of the PatchTST backbone on ETTh1 and ETTh2 datasets using UMAP (top) and PHATE (bottom). For ETTh2, we utilize the causal pretraining configuration. The continuous trajectories in the PHATE plots demonstrate that OliO effectively preserves the temporal flow.

Panel of Figure 6 — Figure 6: Latent space visualization of PatchTST on the ETTm1 dataset, comparing UMAP and PHATE embeddings.

Panel of Figure 7 — Figure 7: Latent space visualizations of iTransformer on ETTm1 and ETTm2 datasets.

Panel of Figure 8 — Figure 8: Latent space visualizations of TimeVLM on ETTh1 and ETTh2 datasets.

Panel of Figure 9 — Figure 9: Latent space visualizations of TimeVLM on ETTm1 and ETTm2 datasets.

Panel of Figure 10 — Figure 10: Latent space visualization of TimeVLM on the Weather dataset, comparing UMAP and PHATE embeddings.

Figure 11: Speedup of the OliO module’s

p

-th order approximation relative to the full matrix exponential (computed via torch.matrix_exp ). The benchmark evaluates the computational efficiency of the TriangularCausalShift module using 4,096 samples per operation, with values averaged over 5 independent trials on an NVIDIA RTX 3090 GPU. The first-order approximation (

p=1

) provides an acceleration of approximately 3.5

\times

to 4.5

\times

compared to the exact computation, highlighting the high-throughput capability of OliO in processing high-dimensional temporal representations.

Backbone	Dataset	TimesURL	OliO
PatchTST	Electricity	0.1846 $\pm$ 0.0012	0.1833 $\pm$ 0.0004
	ETTh1	0.4147 $\pm$ 0.0021	0.4107 $\pm$ 0.0009
	ETTh2	0.3409 $\pm$ 0.0006	0.3461 $\pm$ 0.0011
	ETTm1	0.3546 $\pm$ 0.0009	0.3616 $\pm$ 0.0012
	ETTm2	0.2550 $\pm$ 0.0006	0.2528 $\pm$ 0.0005
	Weather	0.2469 $\pm$ 0.0015	0.2439 $\pm$ 0.0005

Appendix

Table 13: Frozen-backbone linear probing results. Each dataset value averages four forecasting horizons. Lower MSE is better.

Figure 12: Summary of relative few-shot forecasting performance for PatchTST backbone across 1%, 5%, and 10% settings. Performance is normalized relative to OliO (

=1.0

), where higher values indicate better forecasting quality (lower MSE).

Figure 13: Summary of relative few-shot forecasting performance for iTransformer backbone across 1%, 5%, and 10% settings. Performance is normalized relative to OliO (

=1.0

), where higher values indicate better forecasting quality (lower MSE).

Figure 14: Summary of relative few-shot forecasting performance for TimeVLM backbone across 1%, 5%, and 10% settings. Performance is normalized relative to OliO (

=1.0

), where higher values indicate better forecasting quality (lower MSE).

Panel of Figure 19 — Appendix figure: detailed performance improvement heatmaps for each backbone architecture. The values represent the mean performance index (Random MSE / Method MSE) aggregated across six benchmark datasets. Dark blue cells indicate higher performance indices.

Panel of Figure 20 — Appendix figure: performance improvement heatmaps aggregated over all backbone architectures for each benchmark dataset. The values represent the mean performance index (Random MSE / Method MSE). These visualizations summarize how relative forecasting performance varies across datasets and prediction horizons.

Matrix Type	Order ( $p$ )	ETTh1	ETTh2	ETTm1	ETTm2	Weather	ECL
Unconstrained (Full)	Exact	NaN	NaN	NaN	NaN	NaN	NaN
Nilpotent (OliO)	Exact	.4260	.3562	.3616	.2653	.2378	.1664
Nilpotent (OliO)	$p=1$	.4434	.3661	.3752	.2674	.2470	.2137
Nilpotent (OliO)	$p=2$	.4315	.3624	.3695	.2661	.2480	.2063
Nilpotent (OliO)	$p=3$	.4315	.3606	.3686	.2678	.2382	.2029
Nilpotent (OliO)	$p=4$	.4325	.3629	.3720	.2675	.2457	.2051

Appendix

Table 40: Full computation analysis: evaluation of transition matrix constraints and approximation order

p=1\dots 5

. Values represent the average MSE across all backbones.

Setting	90%	80%	60%	20%
PatchTST–ETTh1	0.421	0.418	0.417	0.417
PatchTST–ETTh2	0.343	0.343	0.343	0.344
iTransformer–ETTm1	0.404	0.406	0.403	0.400
iTransformer–ETTm2	0.299	0.305	0.306	0.306

Appendix

Table 41: Forecasting MSE under different overlap ratios. Lower is better.

Setting	w/o OliO	100% Overlap	0% Overlap
PatchTST–ETTh1	0.4189	0.4178	0.4179
PatchTST–ETTh2	0.3594	0.3416	0.3437
PatchTST–Weather	0.2297	0.2299	0.2287
iTransformer–ETTm1	0.4097	0.4068	0.4060
iTransformer–ETTm2	0.2917	0.2889	0.2908
iTransformer–Weather	0.2615	0.2871	0.2668

Appendix

Table 43: Comparison of models without OliO, OliO with full-overlap pairs, and OliO with zero-overlap pairs. Results are averaged over three seeds. Lower MSE is better.

Backbone	Overlap Mask	Unique Mask	Gap	Wins
PatchTST	0.29818	0.29767	+0.17%	9 / 15
iTransformer	0.35022	0.34932	+0.26%	14 / 10
Overall	0.32420	0.32349	+0.22%	23 / 25

Appendix

Table 44: Matched masking control at 50% overlap. MSE is averaged over ETTh1, ETTh2, ETTm1, ETTm2, Weather, and ECL and horizons

\{96,192,336,720\}

(24 settings per backbone; 48 overall). “Wins” reports Overlap-Mask / Unique-Mask wins. Lower MSE is better.

Lag $\tau$	ETTh1	ETTh2
1	0.018	0.000
10	0.617	0.034
20	0.857	0.104
30	0.931	0.182
40	0.965	0.258
50	0.986	0.331

Appendix

Table 46: Relative approximation error between the complete transition and its first-order approximation across temporal lags.

Time Series Forecasting

Time series forecasting predicts future values from ordered observations using statistical, neural, and probabilistic models to support decisions under uncertainty.

See all 873 papers

Sep 30, 2026

WinoTS: Wavelet-based Self-Distillation for Time Series Models

Self-supervised pre-training of time series models is currently dominated by next-token prediction and reconstruction objectives. In continuous-valued domains, these paradigms often waste model capacity on high-frequency, point-wise noise at the expense of learning invariant structure. While invariance-based self-distillation has proven highly effective in computer vision, its application to temporal data remains largely underexplored. Effectively adapting such methods to time series requires carefully designed augmentations: spatial operations like cropping can shift the timing of repeating cycles or distort the signal, while basic jittering may provide limited variation. We introduce Wavelet-based self-distillation for time series (WinoTS), an invariance-based pre-training paradigm designed specifically for temporal signals. At its core, WinoTS leverages time-frequency augmentations to construct multi-scale structural views without distorting underlying signal dynamics. Across extensive evaluations, WinoTS outperforms state-of-the-art baselines in long-term forecasting, cross-domain zero-shot transfer, and unsupervised anomaly detection. Notably, linear probing on frozen WinoTS representations frequently surpasses fully supervised models trained from scratch. Systematic ablations demonstrate that WinoTS is a flexible, architecture-agnostic framework yielding gains across time series backbones, and establish that time-frequency transformations provide a principled alternative to vision-style spatial augmentations.

Noam Major, Kathy Razmadze, Yoli Shavit

Bar-Ilan University, Ramat-Gan, Israel

Self-Supervised Time Series Representation Learning Time Series Data Augmentation Self-Distillation

Ask PDF

May 23, 2025

Time-o1: Time-Series Forecasting Needs Transformed Label Alignment

Training time-series forecasting models poses unique challenges in loss function design. Most existing approaches adopt temporal mean squared error, but this study reveals two critical limitations: (1) it ignores the presence of label autocorrelation, which biases it from the true label sequence likelihood; (2) it involves excessive number of tasks, which complicates optimization, especially for long-term forecasting. To address these issues, we introduce Time-o1, a transform-enhanced loss function for time-series forecasting. The central idea is to transform the label sequence into decorrelated components with discriminated significance. Models are then trained to align the most significant components, thereby effectively mitigating label autocorrelation and reducing task amount. Experiments demonstrate that Time-o1 achieves state-of-the-art performance and is compatible with various forecast models. Code is available at https://github.com/Master-PLC/Time-o1.

Hao Wang, Licheng Pan, Zhichao Chen +5

Xiaohongshu Inc. · State Key Lab of General AI, School of Intelligence Science and Technology, Peking University · Gaoling School of Artificial Intelligence, Renmin University of China +4

Long-Term Time Series Forecasting Time Series Forecasting

Ask PDF

Jun 5, 2026

CF-JEPA: Mask-free forward prediction with asymmetric encoder utilization for time-series representation learning

Self-supervised learning (SSL) for time-series representation learning is dominated by two paradigms: contrastive methods, which face challenges in constructing positive or negative pairs, and masking-based methods, which disrupt the temporal continuity of time-series signals. Joint-Embedding Predictive Architecture (JEPA) offers a promising alternative by predicting in representation space rather than reconstructing raw inputs. However, existing time-series JEPA variants still rely on masking and therefore inherit its continuity problem. Crop-based Forward JEPA (CF-JEPA) is proposed as an innovative mask-free framework that replaces masking with multi-horizon forward prediction: random crops serve as context views, and short-, mid-, and long-horizon future representations are predicted in the forward temporal direction, directly leveraging the inherent temporal ordering of time-series data as a learning signal. A strong asymmetry is also identified between the online encoder and the exponential moving average (EMA) target encoder, both produced from a single training run: the online encoder develops higher-rank discriminative features, while the EMA target encoder develops smoother, lower-rank temporal features. Exploiting this asymmetry, classification is routed to the online encoder and forecasting or anomaly detection to the EMA target encoder, achieving a 27% reduction in multivariate forecasting mean squared error (MSE) at no additional training cost. Across 126 University of California, Riverside (UCR) and 26 University of East Anglia (UEA) classification datasets, eight electricity transformer temperature forecasting benchmarks, and Key Performance Indicator /Yahoo anomaly detection, CF-JEPA achieves the highest average accuracy and rank on UCR and UEA among self-supervised baselines and ranks second on univariate forecasting and k-nearest neighbors-scored anomaly detection.

Jaehoon Lee, Sunghyun Sim

aGraduate School of Artificial Intelligence Convergence Engineering, Changwon National University, Changwon, South Korea · bDepartment of Artificial Intelligence Engineering, Changwon National University, Changwon, South Korea

Multivariate Time Series Forecasting Time Series Classification Self-Supervised Time Series Representation Learning Joint-Embedding Predictive Architecture Time-Series Anomaly Detection

Ask PDF

Abstract

Time Series Forecasting

Explore similar work

WinoTS: Wavelet-based Self-Distillation for Time Series Models

Time-o1: Time-Series Forecasting Needs Transformed Label Alignment

CF-JEPA: Mask-free forward prediction with asymmetric encoder utilization for time-series representation learning