UTICA: Multi-Objective Self-Distllation Foundation Model Pretraining for Time Series Classification

Yessin Moakher; Youssef Attia El Hili; Vasilii Feofanov

cs.LGMar 2, 2026

UTICA: Multi-Objective Self-Distllation Foundation Model Pretraining for Time Series Classification

Authors: Yessin Moakher, Youssef Attia El Hili, Vasilii Feofanov

Organizations: Ecole Polytechnique, France · Huawei Noah’s Ark Lab

Abstract

Self-supervised foundation models have achieved remarkable success across domains, including time series. However, the potential of non-contrastive methods, a paradigm that has driven significant advances in computer vision, remains underexplored for time series. In this work, we adapt DINOv2-style self-distillation to pretrain a time series foundation model, building on the Mantis tokenizer and transformer encoder architecture as our backbone. Through a student-teacher framework, our method Utica learns representations that capture both temporal invariance via augmented crops and fine-grained local structure via patch masking. Our approach achieves state-of-the-art classification performance on both UCR and UEA benchmarks. These results suggest that non-contrastive methods are a promising and complementary pretraining strategy for time series foundation models.

Figures & tables

Figure 1: Example of a time series sample generation via causal DAG.

Figure 2: Architecture.

Figure 3: U nlabeled Ti me-series C rop A ugmented framework. The self-supervised objective aims to match the features produced by the teacher with those produced by the student. Full details about pretraining can be found in Appendix B .

Figure 4: Average accuracy comparison across linear probing and fine-tuning on both UCR and UEA benchmarks. Number of wins shown in parentheses.

Loss Type	data2vec	iBOT+KoLeo	DINO+KoLeo	UTICA
Accuracy	0.7802	0.735	0.747	0.794

Table 1: Ablation on the loss type used for pretraining.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Example of synthetic time series from the DAG-based generator.

Parameter	Value
Depth (number of layers)	6
Attention heads	8
Key/query/value dimension per head	128
MLP hidden dimension	512
Hidden dimension $d$	256
Dropout	0.1

Appendix

Table 2: Transformer hyper-parameters of the Transformer Encoder Unit.

Parameter	Value
Input dimension	256
Hidden dimension	2 048
Bottleneck dimension	256
Output dimension (prototypes) $K$	65 536
Number of MLP layers	3
Activation	GELU

Appendix

Table 3: Projection head hyper-parameters (shared by DINO and iBOT heads).

Panel of Figure 8 — Figure 8: Four local views before and after eventual jittering.

Parameter	Value
Optimiser	AdamW
$(\beta_{1},\;\beta_{2})$	$(0.9,\;0.999)$
Base learning rate	$1\times 10^{-3}$
Minimum learning rate	$1\times 10^{-7}$
Learning rate schedule	Cosine decay
Warmup epochs	0.7

Appendix

Table 4: Optimisation hyper-parameters.

Dataset	NuTime	Moment	Mantis	Utica
ArticularyWordRecognition	0.9933 ±0.0000	0.9800 ±0.0000	0.9930 ±0.0030	0.9900 ±0.0027
BasicMotions	1.0000 ±0.0000	0.9920 ±0.0140	1.0000 ±0.0000	1.0000 ±0.0000
CharacterTrajectories	0.9649 ±0.0038	0.9700 ±0.0020	0.9400 ±0.0010	0.9847 ±0.0006
Cricket	0.9907 ±0.0080	0.9770 ±0.0080	1.0000 ±0.0000	0.9907 ±0.0066
ERing	0.9716 ±0.0057	0.9690 ±0.0080	0.9410 ±0.0100	0.9605 ±0.0017
Eigenworms	0.7786 ±0.0202	0.7350 ±0.0160	0.7530 ±0.0160	0.7837 ±0.0190

Appendix

Table 5: Comparison under linear probing regime on UEA. The best epoch performance is averaged over 3 random seeds and reported with the standard deviation. Bold indicates the best result per row. NuTime, Moment and Mantis results are from [ Feofanov et al., 2025 ] . We exclude datasets whose memory requirements exceed our available GPU memory (24 GB).

Dataset	GPT4TS	NuTime	Moment	Mantis	Utica
ArticularyWordRecognition	$0.9633_{\pm 0.0058}$	$0.9833_{\pm 0.0033}$	$0.9733_{\pm 0.0033}$	0.9944 ±0.0019	$0.9900_{\pm 0.0033}$
BasicMotions	$0.8833_{\pm 0.0144}$	1.0000 ±0.0000	1.0000 ±0.0000	1.0000 ±0.0000	1.0000 ±0.0000
CharacterTrajectories	$0.9731_{\pm 0.0056}$	$0.9800_{\pm 0.0036}$	$0.9791_{\pm 0.0028}$	$0.9893_{\pm 0.0008}$	0.9937 ±0.0007
Cricket	$0.9213_{\pm 0.0080}$	$0.9907_{\pm 0.0080}$	$0.9491_{\pm 0.0160}$	$0.9954_{\pm 0.0080}$	1.0000 ±0.0000
ERing	$0.9074_{\pm 0.0098}$	$0.9506_{\pm 0.0077}$	$0.9358_{\pm 0.0077}$	$0.9827_{\pm 0.0043}$	0.9864 ±0.0021
Epilepsy	$0.8092_{\pm 0.0233}$	$0.9952_{\pm 0.0042}$	$0.9976_{\pm 0.0042}$	1.0000 ±0.0000	$0.9952_{\pm 0.0042}$

Appendix

Table 6: Comparison under the fine tuning regime on UEA. The best epoch performance is averaged over 3 random seeds and reported with the standard deviation. Bold indicates the best result per row. GPT4S, NuTime, Moment and Mantis results are from [ Feofanov et al., 2025 ] . We exclude datasets whose memory requirements exceed our available GPU memory (24 GB).

Dataset	NuTime	Moment	Mantis	Utica
ACSF1	0.7367 ±0.0351	0.75 ±0.0173	0.6133 ±0.0208	0.6200 ±0.0000
Adiac	0.7255 ±0.0115	0.7886 ±0.0074	0.7332 ±0.0039	0.5090 ±0.0142
AllGestureWiimoteX	0.661 ±0.003	0.6105 ±0.0136	0.6705 ±0.0044	0.6824 ±0.0068
AllGestureWiimoteY	0.6362 ±0.0079	0.6576 ±0.0079	0.6671 ±0.0057	0.7029 ±0.0071
AllGestureWiimoteZ	0.6024 ±0.0179	0.5767 ±0.0103	0.6695 ±0.0033	0.6448 ±0.0016
ArrowHead	0.76 ±0.0198	0.8076 ±0.0144	0.7105 ±0.0175	0.6514 ±0.0618

Appendix

Table 7: Comparison under linear probing regime on UCR. The best epoch performance is averaged over 3 random seeds and reported with the standard deviation. Bold indicates the best result per row. NuTime, Moment and Mantis results are from [ Feofanov et al., 2025 ] .

Dataset	GPT4TS	NuTime	Moment	Mantis	Utica
ACSF1	0.4900 ±0.0265	0.6733 ±0.0321	0.6033 ±0.0379	0.7433 ±0.0115	0.6733 ±0.0419
Adiac	0.3572 ±0.0059	0.7349 ±0.0141	0.5823 ±0.0296	0.7766 ±0.0030	0.8159 ±0.0130
AllGestureWiimoteX	0.4895 ±0.0105	0.6386 ±0.0038	0.7000 ±0.0103	0.7619 ±0.0050	0.7395 ±0.0115
AllGestureWiimoteY	0.5062 ±0.0116	0.7152 ±0.0103	0.7081 ±0.0343	0.7948 ±0.0097	0.7390 ±0.0106
AllGestureWiimoteZ	0.4719 ±0.0119	0.6419 ±0.0058	0.6976 ±0.0218	0.7300 ±0.0100	0.6967 ±0.0141
ArrowHead	0.7638 ±0.0033	0.8305 ±0.0033	0.7790 ±0.0682	0.8210 ±0.0389	0.8457 ±0.0123

Appendix

Table 8: Comparison under the fine tuning regime on UCR. The best epoch performance is averaged over 3 random seeds and reported with the standard deviation. Bold indicates the best result per row. GPT4S, NuTime, Moment and Mantis results are from [ Feofanov et al., 2025 ] .

Self-Supervised Time Series Representation Learning

Self-supervised time series representation learning uses unlabeled sequences to learn features that transfer to forecasting, classification, and anomaly detection.

See all 142 papers

Mantis: Lightweight Foundation Model for Time Series Classification
Feb 21, 2025Vasilii Feofanov, Songkang Wen, Shifeng Xie +10Time Series Classification Self-Supervised Time Series Representation Learning
1. Huawei Noah’s Ark Lab · 3. Paris Cit´e University · 4. Helmholtz Munich +3
Turning TS2Vec into a Foundation Model for Time Series Classification
Oct 20, 2026Vasilii Feofanov, Zhenghan Tai, Andrei Ermakov +1Time Series Classification Time-Series Foundation Models
NL · University of Toronto · Université de Montréal Mila - Quebec AI Institute
WinoTS: Wavelet-based Self-Distillation for Time Series Models
Sep 30, 2026Noam Major, Kathy Razmadze, Yoli ShavitSelf-Supervised Time Series Representation Learning Time Series Data Augmentation
Bar-Ilan University, Ramat-Gan, Israel

Feb 21, 2025J/K move · Enter open · S save

Mantis: Lightweight Foundation Model for Time Series Classification

Vasilii Feofanov, Songkang Wen, Shifeng Xie, Simon Roschmann+9

1. Huawei Noah’s Ark Lab · 3. Paris Cit´e University · 4. Helmholtz Munich+3

Time Series Classification Self-Supervised Time Series Representation Learning Time Series Tokenization Time-Series Foundation Models

Open paper AskPDF arXiv

While foundation models have revolutionized various domains, their application to time series classification remains rather under-explored, with existing literature predominantly focused on forecasting. To bridge this gap, we introduce Mantis, a transformer-based foundation model pre-trained exclusively on synthetic data via self-supervised contrastive learning. We demonstrate that effective tokenization is critical to unlocking the full potential of transformers, proposing a novel token generator unit. Furthermore, we introduce an enhanced test-time methodology that bridges the performance gap between Mantis and strong specialized approaches by leveraging intermediate-layer representations, self-ensembling, and cross-model embedding fusion. Extensive experiments demonstrate that Mantis establishes a new state-of-the-art, outperforming existing foundation models across four diverse dataset collections covering various application domains.

Abstract

Self-Supervised Time Series Representation Learning

Explore similar work

Mantis: Lightweight Foundation Model for Time Series Classification

Turning TS2Vec into a Foundation Model for Time Series Classification

WinoTS: Wavelet-based Self-Distillation for Time Series Models