Breaking feature collapse in self-supervised time series encoders
Authors: Özgün Turgut, Daniel Rueckert
Organizations: TaktAI · Chair for AI in Healthcare and Medicine, Technical University of Munich, Germany · Munich Center for Machine Learning, Munich, Germany · Department of Computing, Imperial College London, United Kingdom
General-purpose time series encoders promise a single backbone that generalises across tasks and domains; however, they currently do not scale. We trace this down to a critical point of failure: under the predominant masked data modelling supervision, the encoder's output tokens partially collapse. This is the natural consequence of supervising encoders only in the input space, which leaves no constraint against redundant token dimensions in the feature space. We introduce OTISv2, which suppresses the collapse by complementing the reconstruction objective in input space with two objectives in the feature space: a self-distillation loss against an exponential-moving-average teacher, and a KoLeo regulariser. Under matched training, the resulting encoder reduces the cosine similarity and lifts the effective rank of its output tokens relative to its predecessor, and it scales monotonically from 7.1 M to 116 M parameters where its predecessor declines. It matches or surpasses existing general-purpose time series encoders across 143 uni- and multi-variate datasets. To foster further research, we release our code and pre-trained weights.
Figures & tables
Figure 1: General-purpose time series encoders saturate at scale. Average accuracy over 23 UEA and 120 UCR datasets. Across three general-purpose encoder families, gains through parameter scaling saturate. OTIS-7.1M nearly matches the 43× larger MOMENT-308M .
Figure 2: Feature quality and its consequences for downstream performance. Pre-training of OTIS and OTISv2 with the same data, width, depth, optimiser, and schedule. Both use C=6ND=6.82×1016 FLOPs at N=7.1 M parameters, D=1.60 B training tokens, and DC=32 M unique tokens. (Left) Patch-token cosine similarity ( ↓ ). (Centre) RankMe ( ↑ ) normalised by the encoder width d=192 . (Right) UEA 5 -shot balanced accuracy on 23 datasets ( ↑ ). Token similarity drops in both encoders within the first 200 M tokens, then stays at 0.46 in OTIS but declines to 0.32 in OTISv2 , whose 5 -shot lead grows to 5 to 7 pp by 1.60 B tokens.
Feature Quality
Downstream Performance
Encoder
Encoder width d
Compute C=6ND [FLOPs] ‡
Tokens/param TPP=D/N
Patch-token cosine sim ( ↓ )
RankMe ( ↑ )
UEA 5-shot ( ↑ )
UEA k-NN ( ↑ )
UEA linear ( ↑ )
UCR 5-shot ( ↑ )
UCR k-NN ( ↑ )
UCR linear ( ↑ )
Avg weighted ( ↑ )
UniTS-0.4M
32
4.3×1016 †
45,000 †
0.23
16
38.48
46.02
50.92
49.71
57.36
61.29
54.35
UniTS-1.1M §
64
6.0×1016 †
8,200 †
0.09
32
46.46
52.78
56.45
53.97
62.93
66.03
59.52
UniTS-3.5M
128
3.8×1017 †
5,200 †
0.27
46
44.82
51.77
58.17
52.92
61.43
68.10
59.33
MOMENT-19M
512
3.5×1016 †
16 †
0.84
25
48.13
54.77
63.60
60.87
69.41
77.41
67.02
MOMENT-85M
768
1.6×1017 †
3.6 †
0.83
49
49.46
56.43
63.63
62.28
70.24
77.75
67.91
Table 1: Feature quality and downstream performance of frozen general-purpose time series encoders. Balanced accuracy on 23 UEA and 120 UCR datasets, with best and second-best scores highlighted. D counts patch tokens seen during pre-training, including repetition. OTISv2-7.1M matches or surpasses encoders up to 43× larger on average.
Beff / η
Sim ( ↓ )
RankMe ( ↑ )
5 -shot ( ↑ )
Beff / η
Sim ( ↓ )
RankMe ( ↑ )
5 -shot ( ↑ )
384 / 3×10−4
0.528
51.2
52.18
384 / 1.2×10−3
0.519
62.1
53.22
192 / 3×10−4
0.487
58.8
53.26
96 / 3×10−4
0.481
62.6
54.29
384 / 2.4×10−3
0.556
59.3
52.99
192 / 1.2×10−3
0.516
62.9
53.69
384 / 6×10−4
0.505
59.8
53.00
192 / 6×10−4
0.489
64.3
54.27
96 / 1.2×10−3
0.538
59.8
54.09
96 / 6×10−4
0.494
64.8
55.01
Table 2: Feature quality and downstream performance under a fixed objective. OTISv2-7.1M pre-trained at ten combinations of effective batch size Beff and peak learning rate η , ordered by RankMe. RankMe, patch-token cosine similarity (Sim), and UEA 5 -shot accuracy ( 5 -shot) are means over three seeds. RankMe correlates with 5 -shot ( r=+0.83 ), but Sim does not ( r=−0.48 ).
Feature Quality
Downstream Performance
Encoder
Patch-token cosine sim ( ↓ )
RankMe ( ↑ )
UEA 5-shot ( ↑ )
UEA k-NN ( ↑ )
UEA linear ( ↑ )
UCR 5-shot ( ↑ )
UCR k-NN ( ↑ )
UCR linear ( ↑ )
Avg weighted ( ↑ )
OTIS-7.1M
0.56
52
52.48
56.56
64.38
58.62
61.75
74.11
63.69
+ domain token
0.59 ↑ 0.03
51 ↓ 1
51.08 ↓ 1.40
56.78 ↑ 0.22
64.31 ↓ 0.07
57.01 ↓ 1.61
60.53 ↓ 1.22
72.56 ↓ 1.55
62.40 ↓ 1.29
+ register tokens
0.59 ↓ 0.00
50 ↓ 1
52.78 ↑ 1.70
57.60 ↑ 0.82
65.51 ↑ 1.20
58.37 ↑ 1.36
62.04 ↑ 1.51
73.58 ↑ 1.02
63.69 ↑ 1.29
+ Axial RoPE
0.66 ↑ 0.07
56 ↑ 6
53.12 ↑ 0.34
57.97 ↑ 0.37
65.98 ↑ 0.47
58.75 ↑ 0.38
62.26 ↑ 0.22
74.83 ↑ 1.25
64.27 ↑ 0.58
+ zero-init residuals
0.64 ↓ 0.02
54 ↓ 2
53.44 ↑ 0.32
57.58 ↓ 0.39
66.10 ↑ 0.12
56.47 ↓ 2.28
58.77 ↓ 3.49
73.49 ↓ 1.34
62.29 ↓ 1.98
Table 3: Cumulative ablation from OTIS to OTISv2 . One component added per row, starting from the OTIS pre-training recipe towards the OTISv2 recipe. All rows use C=6ND=6.82×1016 FLOPs at N=7.1 M parameters, D=1.60 B training tokens, and DC=32 M unique tokens. Mean scores over three pre-training seeds are reported. Arrows give the change from the previous row.
Figure 3: Scaling encoder size under data constraints. Columns. The four IsoFLOP (N,D) groups listed in the table. Rows. (Top) Patch-token cosine similarity ( ↓ ). (Centre) RankMe ( ↑ ) normalised by the encoder width d . (Bottom) UEA 5 -shot balanced accuracy on 23 datasets, using mean-pooled patch tokens ( ↑ ). Within each column, the eight data budgets DC correspond to {1,2,3,4,5,7,14,32} epochs of repetition (colour). OTIS is dotted, OTISv2 is solid. At every size and data budget, OTISv2 has lower token similarity, higher RankMe, and higher accuracy than OTIS .
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Domain
Source
Samples
Variates V
Time points T
V⋅T
Frequency
ECG
MIMIC-IV-ECG [ 12 ]
400,000
12
5,000
24.0 B
500 Hz
EEG
TDBRAIN [ 35 ]
2,692
19
60,000
3.07 B
500 Hz
SEED [ 42 ]
675
19
37,000
475 M
200 Hz
Audio
AudioSet- 20 K [ 10 ]
19,614
1 – 2
24,576 – 480,000
15.6 B
44.1 kHz
Electromechanics
FD-A [ 21 ]
8,184
1
5,120
41.9 M
64 kHz
Temperature
DWD [ 6 ]
202,712
1
720
146 M
278μ Hz
Appendix
Table 4: Pre-training corpus. Eleven domains from healthcare, audio, engineering, natural sciences, infrastructure, and finance, with 756,267 samples and 45.2 B time points in total. V⋅T counts the time points of all variates, summed over samples, and Frequency denotes the sampling rate.
Encoder
Parameters
GPU
Count
Hours †
Type
OTISv2-7.1M
7.1 M
2
∼61
NVIDIA A 100 - 80 GB
OTISv2-14.8M ‡
14.8 M
1
∼12
NVIDIA A 100 - 80 GB
OTISv2-40M ‡
40 M
1
∼18
NVIDIA A 100 - 80 GB
OTISv2-116M ‡
116 M
1
∼27
NVIDIA A 100 - 80 GB
† GPU hours, summed over all GPUs.
Appendix
Table 5: Computational resources used for pre-training OTISv2 .
Table 6: Pre-training hyperparameters of OTISv2-7.1M .
Encoder
Strategy
Encoder width d
Parameters
Pre-training time points
UniTS-0.4M [ 8 ]
MDM + task prompts
32
0.4 M
0.13 B †
UniTS-1.1M [ 8 ]
Supervised multi-task ‡
64
1.1 M
0.13 B †
UniTS-3.5M [ 8 ]
MDM + task prompts
128
3.5 M
0.13 B †
MOMENT-19M [ 11 ]
MDM
512
19 M
1.2 B
MOMENT-85M [ 11 ]
MDM
768
85 M
1.2 B
MOMENT-308M [ 11 ]
MDM
1,024
308 M
1.2 B
Appendix
Table 7: General-purpose time series encoders. Pre-training strategy (MDM, masked data modelling; SD, self-distillation), encoder width, parameter count, and pre-training corpus size.
Figure 4: The advantage of OTISv2 builds up over depth. Token similarity, RankMe, and 5 -shot accuracy at every block of the matched runs of OTIS and OTISv2 ( N=7.1 M, D=1.60 B, DC=32 M, 12 blocks, d=192 ). Lines show the mean over three pre-training seeds, and shading one standard deviation. Similarity and RankMe follow the protocol of Table 3 (App. E ), and at block 12 they equal its first and last rows. (Left) Patch-token cosine similarity. (Centre) RankMe normalised by d . (Right) UEA 5 -shot balanced accuracy on the [CLS] token of each block.
Figure 5: Neither encoder develops high-norm artefact tokens during pre-training. Patch-token ℓ2 norms of the matched runs of OTIS and OTISv2 ( N=7.1 M, D=1.60 B, DC=32 M), one pre-training seed each. Colour gives the density of tokens. (Left two) Last encoder block over training. (Right two) All encoder blocks at the end of pre-training.
OTIS
OTISv2
Token ℓ2 norm range
[6.65,10.34]
[11.50,20.47]
Token ℓ2 norm spread (max / min)
1.56×
1.78×
Spearman ρ (token norm, raw patch energy)
−0.18
+0.39
max∣ρ∣ (PC1–PC3, raw patch energy)
0.62 (PC2)
0.45 (PC2)
PCA explained variance, top-3 sum
0.35
0.38
Attention max received (uniform =0.00198 )
0.0132 ( 6.7× )
0.0187 ( 9.4× )
Appendix
Table 8: Last-block properties on four 12 -lead ECG samples ( 504 patches per sample). Norms are taken after the final normalisation layer, and attention is averaged over the patch queries of each head. Patch energy is the standard deviation of the raw signal in a patch, and the effective keys of a query are the exponential of the entropy of its attention. The two checkpoints are not matched, so these contrasts are illustrative only (App. H ).
Figure 6: Both encoders keep token norms in a narrow band without outliers. Per-patch token ℓ2 norms after the final normalisation layer on four 12 -lead ECGs. In each subfigure, the top row shows the raw signal, and the bottom row shows the norm of each patch with the signal overlaid, on a colour scale per encoder shared across the four samples.
Figure 7: The leading patch-token components of OTIS vary smoothly across neighbouring patches. Joint PCA of the last-block patch tokens of OTIS over the four samples. Row 1 shows the raw signal, and rows 2 to 4 show PC1 to PC3 with the signal overlaid, on a symmetric colour scale shared across samples.
Figure 8: The leading patch-token components of OTISv2 vary more from patch to patch. Joint PCA of the last-block patch tokens of OTISv2 , with the same layout as Figure 7 .
Figure 9: OTIS attends to QRS-aligned patches in one head. Mean last-block attention that each patch receives from all query patches, per head, with the signal overlaid. Row 1 shows the raw signal, and rows 2 to 4 show the three heads.
Figure 10: OTISv2 attends to QRS-aligned patches in one head and to a few leads in the other two. Per-head last-block attention of OTISv2 , with the same layout as Figure 9 .
Figure 11: Each OTISv2 query attends to its own set of patches. Head-averaged last-block attention of the [CLS] token and four random patch queries on one 12 -lead ECG. Column 1 shows the signal with the query positions marked by coloured crosses, column 2 the attention of [CLS] , and columns 3 to 6 the attention of each patch query.
Encoder
Blocks
Encoder width d
MLP size
Heads
dh
Parameters
OTISv2-7.1M
12
192
768
3
64
7.1 M
OTISv2-14.8M
14
256
1,024
4
64
14.8 M
OTISv2-40M
24
320
1,280
5
64
40 M
OTISv2-116M
36
448
1,792
7
64
116 M
Appendix
Table 9: Encoder configurations of OTISv2 .
Encoder
ETTh1
ETTh2
ETTm1
ETTm2
Weather
Electricity
ILI
Avg
OTIS-7.1M
1.916
1.611
2.230
1.500
0.565
0.797
7.295
2.273
OTIS-43M
1.161
0.479
1.409
0.479
0.394
0.823
5.414
1.451
OTIS-128M
1.071
0.317
1.068
0.381
0.383
0.866
5.868
1.422
OTISv2-7.1M
1.048
0.427
1.506
0.546
0.257
0.617
5.074
1.354
Appendix
Table 10: Without any training, OTISv2 has the lowest average forecasting error. Zero-shot MSE ( ↓ ) of the frozen released encoders and their pre-training decoders, averaged over the horizons 96 , 192 , and 336 ( 24 , 48 , and 72 for ILI). Bold marks the best encoder per dataset.
Encoder
Clean
Noise
Spikes
Missing
Dropout †
5 -shot prototypical classification
MOMENT-308M
52.84
52.20
50.04
52.83
52.82
OTIS-7.1M
55.53
55.49
53.43
55.02
53.75
OTIS-43M
57.53
57.48
55.49
57.28
55.41
OTIS-128M
58.70
58.63
56.97
58.19
56.61
OTISv2-7.1M
60.00
56.82
56.98
58.39
56.04
Appendix
Table 11: OTISv2 loses more accuracy under corruptions, mostly through its [CLS] embedding. Balanced accuracy ( ↑ ) on the 23 UEA datasets with clean training splits and corrupted test splits. Bold marks the best encoder per protocol and condition. UniTS and the two smaller MOMENT encoders, which lead in no setting, are omitted, and the clean accuracies reproduce Table 1 within 0.1 pp.
Self-supervised time series representation learning uses unlabeled sequences to learn features that transfer to forecasting, classification, and anomaly detection.
TS2Vec is one of the strongest self-supervised methods for time series classification, introduced in 2022. It is designed to be trained anew on every dataset: self-supervised pre-training followed by linear probing. Since then, the field has shifted towards foundation models that reuse one encoder across datasets. Yet TS2Vec offers a compact convolutional architecture that operates directly on temporal samples and accepts variable-length inputs without resampling. We ask whether these properties can be retained when the encoder is pre-trained only once, and show that they can: TS2Vec can essentially be turned into a foundation model. Three changes suffice: pre-train on a synthetic corpus, feed the encoder a raw amplitude view and an instance-normalized shape view of the input through two small stems, and distill one specialist encoder per view into a single encoder. The result, TS2Vec-FM, is a 1.6M-parameter dilated convolutional network that, kept frozen, reaches 83.3% on UCR-128 and 75.3% on UEA-27 under linear probing, within 0.3 pp of MantisV2, the current state of the art, and above TS2Vec trained separately on each target dataset. What we find most interesting is how different TS2Vec-FM is from other foundation models in design: near-parity is reached without attention, without a patch tokenizer, and without intermediate-layer extraction, using an "old-school" convolutional encoder and a two-layer MLP stem.
Vasilii Feofanov, Zhenghan Tai, Andrei Ermakov +1
NL · University of Toronto · Université de Montréal Mila - Quebec AI Institute
Self-supervised foundation models have achieved remarkable success across domains, including time series. However, the potential of non-contrastive methods, a paradigm that has driven significant advances in computer vision, remains underexplored for time series. In this work, we adapt DINOv2-style self-distillation to pretrain a time series foundation model, building on the Mantis tokenizer and transformer encoder architecture as our backbone. Through a student-teacher framework, our method Utica learns representations that capture both temporal invariance via augmented crops and fine-grained local structure via patch masking. Our approach achieves state-of-the-art classification performance on both UCR and UEA benchmarks. These results suggest that non-contrastive methods are a promising and complementary pretraining strategy for time series foundation models.
Yessin Moakher, Youssef Attia El Hili, Vasilii Feofanov
Ecole Polytechnique, France · Huawei Noah’s Ark Lab
Self-supervised representation learning has advanced time series forecasting by capturing robust features from unlabeled data. However, existing contrastive and masking-based methods often struggle to explicitly model the underlying temporal flow, either by treating time lag as noise or by disrupting the inherent temporal dynamics essential for forecasting. To address these limitations, we propose OliO (ODE-based Linear Transition Operator), a plug-and-play self-supervised learning method that learns bidirectional temporal flows as a continuous and structurally consistent dynamical system. OliO introduces a transition operator derived from linear Ordinary Differential Equations (ODEs) to align latent representations across arbitrary time shifts. By imposing a strictly upper triangular constraint on the transition matrix, we mathematically ensure numerical stability and enforce a robust polynomial boundary that effectively captures long-term dependencies while preventing the exponential divergence typically found in unconstrained ODE-based models. This structural constraint induces a temporally coherent representation space that preserves the underlying flow essential for accurate forecasting. Extensive experiments across various backbone architectures and benchmarks demonstrate that OliO achieves significant performance gains, providing Mean Squared Error (MSE) reductions of up to 5.75% over the most competitive grid-searched SOTA baselines. The code is available at https://github.com/eai-lab/OliO
Kwangryeol Park, Seulki Lee
Graduate School of Artificial Intelligence, Ulsan National Institute of Science and Technology, Ulsan, South Korea · School of Electrical Engineering, Korea Advanced Institute of Science and Technology, Daejeon, South Korea