cs.LGNov 16, 2026

Breaking feature collapse in self-supervised time series encoders

Authors: Özgün Turgut, Daniel Rueckert

Organizations: TaktAI · Chair for AI in Healthcare and Medicine, Technical University of Munich, Germany · Munich Center for Machine Learning, Munich, Germany · Department of Computing, Imperial College London, United Kingdom

Abstract

General-purpose time series encoders promise a single backbone that generalises across tasks and domains; however, they currently do not scale. We trace this down to a critical point of failure: under the predominant masked data modelling supervision, the encoder's output tokens partially collapse. This is the natural consequence of supervising encoders only in the input space, which leaves no constraint against redundant token dimensions in the feature space. We introduce OTISv2, which suppresses the collapse by complementing the reconstruction objective in input space with two objectives in the feature space: a self-distillation loss against an exponential-moving-average teacher, and a KoLeo regulariser. Under matched training, the resulting encoder reduces the cosine similarity and lifts the effective rank of its output tokens relative to its predecessor, and it scales monotonically from 7.1 M to 116 M parameters where its predecessor declines. It matches or surpasses existing general-purpose time series encoders across 143 uni- and multi-variate datasets. To foster further research, we release our code and pre-trained weights.

Figures & tables

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Oct 20, 2026

Turning TS2Vec into a Foundation Model for Time Series Classification

TS2Vec is one of the strongest self-supervised methods for time series classification, introduced in 2022. It is designed to be trained anew on every dataset: self-supervised pre-training followed by linear probing. Since then, the field has shifted towards foundation models that reuse one encoder across datasets. Yet TS2Vec offers a compact convolutional architecture that operates directly on temporal samples and accepts variable-length inputs without resampling. We ask whether these properties can be retained when the encoder is pre-trained only once, and show that they can: TS2Vec can essentially be turned into a foundation model. Three changes suffice: pre-train on a synthetic corpus, feed the encoder a raw amplitude view and an instance-normalized shape view of the input through two small stems, and distill one specialist encoder per view into a single encoder. The result, TS2Vec-FM, is a 1.6M-parameter dilated convolutional network that, kept frozen, reaches 83.3% on UCR-128 and 75.3% on UEA-27 under linear probing, within 0.3 pp of MantisV2, the current state of the art, and above TS2Vec trained separately on each target dataset. What we find most interesting is how different TS2Vec-FM is from other foundation models in design: near-parity is reached without attention, without a patch tokenizer, and without intermediate-layer extraction, using an "old-school" convolutional encoder and a two-layer MLP stem.
Mar 2, 2026

UTICA: Multi-Objective Self-Distllation Foundation Model Pretraining for Time Series Classification

Self-supervised foundation models have achieved remarkable success across domains, including time series. However, the potential of non-contrastive methods, a paradigm that has driven significant advances in computer vision, remains underexplored for time series. In this work, we adapt DINOv2-style self-distillation to pretrain a time series foundation model, building on the Mantis tokenizer and transformer encoder architecture as our backbone. Through a student-teacher framework, our method Utica learns representations that capture both temporal invariance via augmented crops and fine-grained local structure via patch masking. Our approach achieves state-of-the-art classification performance on both UCR and UEA benchmarks. These results suggest that non-contrastive methods are a promising and complementary pretraining strategy for time series foundation models.
Oct 16, 2026

OliO: ODE-based Linear Transition Operator for Self-Supervised Time Series Forecasting

Self-supervised representation learning has advanced time series forecasting by capturing robust features from unlabeled data. However, existing contrastive and masking-based methods often struggle to explicitly model the underlying temporal flow, either by treating time lag as noise or by disrupting the inherent temporal dynamics essential for forecasting. To address these limitations, we propose OliO (ODE-based Linear Transition Operator), a plug-and-play self-supervised learning method that learns bidirectional temporal flows as a continuous and structurally consistent dynamical system. OliO introduces a transition operator derived from linear Ordinary Differential Equations (ODEs) to align latent representations across arbitrary time shifts. By imposing a strictly upper triangular constraint on the transition matrix, we mathematically ensure numerical stability and enforce a robust polynomial boundary that effectively captures long-term dependencies while preventing the exponential divergence typically found in unconstrained ODE-based models. This structural constraint induces a temporally coherent representation space that preserves the underlying flow essential for accurate forecasting. Extensive experiments across various backbone architectures and benchmarks demonstrate that OliO achieves significant performance gains, providing Mean Squared Error (MSE) reductions of up to 5.75% over the most competitive grid-searched SOTA baselines. The code is available at https://github.com/eai-lab/OliO