cs.LGNov 16, 2026

Breaking feature collapse in self-supervised time series encoders

Authors: Özgün Turgut, Daniel Rueckert

Organizations: TaktAI · Chair for AI in Healthcare and Medicine, Technical University of Munich, Germany · Munich Center for Machine Learning, Munich, Germany · Department of Computing, Imperial College London, United Kingdom

Abstract

General-purpose time series encoders promise a single backbone that generalises across tasks and domains; however, they currently do not scale. We trace this down to a critical point of failure: under the predominant masked data modelling supervision, the encoder's output tokens partially collapse. This is the natural consequence of supervising encoders only in the input space, which leaves no constraint against redundant token dimensions in the feature space. We introduce OTISv2, which suppresses the collapse by complementing the reconstruction objective in input space with two objectives in the feature space: a self-distillation loss against an exponential-moving-average teacher, and a KoLeo regulariser. Under matched training, the resulting encoder reduces the cosine similarity and lifts the effective rank of its output tokens relative to its predecessor, and it scales monotonically from 7.1 M to 116 M parameters where its predecessor declines. It matches or surpasses existing general-purpose time series encoders across 143 uni- and multi-variate datasets. To foster further research, we release our code and pre-trained weights.

Figures & tables

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Turning TS2Vec into a Foundation Model for Time Series Classification

    Oct 20, 2026Vasilii Feofanov, Zhenghan Tai, Andrei Ermakov +1Time Series ClassificationTime-Series Foundation Models

  2. UTICA: Multi-Objective Self-Distllation Foundation Model Pretraining for Time Series Classification

    Mar 2, 2026Yessin Moakher, Youssef Attia El Hili, Vasilii FeofanovSelf-Supervised Time Series Representation LearningTime-Series Foundation Models

  3. OliO: ODE-based Linear Transition Operator for Self-Supervised Time Series Forecasting

    Oct 16, 2026Kwangryeol Park, Seulki LeeTime Series ForecastingSelf-Supervised Time Series Representation Learning