Random Initialization as a Baseline for Time Series Foundation Models

Pinar Sungu Isiacik; Georgiana Ifrim

cs.LGOct 22, 2026

Random Initialization as a Baseline for Time Series Foundation Models

Authors: Pinar Sungu Isiacik, Georgiana Ifrim

Organizations: University College Dublin, Dublin, Ireland

Abstract

Time series foundation models (TSFMs) use self-supervised pretraining to learn reusable representations for downstream tasks such as multivariate time series classification (MTSC). We examine how much pretrained weights contribute beyond the architecture itself by comparing seven publicly released instances from four TSFM families with architecture-matched randomly initialized counterparts on the UEA MTSC archive. For each instance, five random initializations are evaluated using four classifiers on frozen representations: ridge classification, logistic regression, RBF SVM, and random forest. After model-specific exclusions due to computational constraints, the study comprises 183 paired dataset-level comparisons. Pretraining improves mean accuracy for all seven instances, but the gains are modest, ranging from 1.43 to 5.19 percentage points, with an overall average improvement of 2.58 points. Three of seven per-instance Wilcoxon signed-rank tests have raw $p$ -values below 0.05, but only NuTime remains significant after Holm correction. A secondary pooled test across all 183 pairs gives $p < 0.001$ , while the corrected per-instance tests remain the primary evidence. We also compare with ROCKET, Hydra, and Quant, which remain competitive without requiring large-scale pretraining. These findings suggest that MTSC evaluations should distinguish the effect of pretrained weights from that of the underlying architecture and should include both architecture-matched random initialization and strong non-pretrained baselines.

Figures & tables

Instance	$n$	Pretrained	Random init	Mean $\Delta$ [95% CI]	Median $\Delta$	W/T/L	$p_{\text{raw}}$	$p_{\text{Holm}}$
MOMENT-Small	26	0.703	$0.680\pm 0.002$	$+2.29\,[-1.20,\,5.46]$	+2.40	21/1/4	0.011	0.066
MOMENT-Base	26	0.698	$0.684\pm 0.003$	$+1.43\,[-2.02,\,4.67]$	+1.20	17/0/9	0.143	0.233
MOMENT-Large	26	0.702	$0.684\pm 0.005$	$+1.76\,[-1.73,\,4.82]$	+2.35	16/0/10	0.075	0.225
Mantis	26	0.728	$0.702\pm 0.002$	$+2.56\,[-0.29,\,5.45]$	+2.60	18/1/7	0.053	0.210
MantisV2	26	0.721	$0.695\pm 0.003$	$+2.58\,[0.16,\,5.08]$	+0.80	16/0/10	0.116	0.233
UniTS	27	0.651	$0.628\pm 0.003$	$+2.24\,[0.80,\,3.83]$	+0.40	18/1/8	0.021	0.107

Table 1: Architecture-matched comparison of pretrained and randomly initialized foundation models. Accuracy is averaged across four downstream classifiers. Random-init values are mean

\pm

standard deviation across five encoder seeds. Mean

\Delta

includes an unadjusted 95% paired-bootstrap confidence interval; W/T/L denotes pretrained wins/ties/losses.

Instance	Ridge	LogReg	RBF	RF	Linear	Nonlinear
MOMENT-Small	2.59	2.93	1.09	2.53	2.76	1.81
MOMENT-Base	1.69	1.49	1.47	1.09	1.59	1.28
MOMENT-Large	2.84	2.35	1.24	0.62	2.59	0.93
Mantis	0.38	1.92	4.71	3.23	1.15	3.97
MantisV2	1.33	1.55	4.26	3.19	1.44	3.73
UniTS	1.53	2.00	1.82	3.62	1.76	2.72

Table 2: Pretrained minus random-init accuracy (

\Delta

, percentage points) for each downstream classifier. Linear and nonlinear columns report the average difference within each classifier family.

Method	Mean accuracy	Avg. rank	Wins
ROCKET	0.754	3.48	8
Hydra	0.752	3.58	10
Quant	0.752	4.25	3
Mantis	0.731	3.94	6
MantisV2	0.720	4.77	4
MOMENT-Small	0.703	5.73	3

Table 3: Comparison on the common 24 dataset subset. For each foundation model, accuracy is averaged across the four downstream classifiers; ROCKET, Hydra, and Quant are reported using their respective end-to-end classifier outputs. Average rank is computed per dataset across all ten methods, with tied accuracies assigned their average rank. Wins count datasets on which a method attains the highest accuracy; tied methods each receive a win, so the total can exceed 24.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Coverage in the four classifier aggregate
DuckDuckGeese	Missing for UniTS; complete for the other six model instances.
FaceDetection	Complete only for UniTS; missing for all MOMENT instances, Mantis, MantisV2, and NuTime.
InsectWingbeat	Not evaluated for any of the seven foundation model instances.
PenDigits	Missing for NuTime; complete for the other six model instances.
PEMS-SF	Missing for all seven foundation model instances.
StandWalkJump	Complete for UniTS and NuTime. For the three MOMENT instances, Mantis, and MantisV2, the logistic-regression result is unavailable, so no four-classifier aggregate is reported.

Appendix

Table 4: Foundation model dataset coverage. “Missing” indicates that a complete four classifier result is unavailable under at least one of the pretrained or randomly initialized conditions.

Classifier	Fixed configuration	Selected hyperparameters
Ridge	StandardScaler ; RidgeClassifier	$\alpha\in\{10^{-3},10^{-2},10^{-1},1,10,10^{2},10^{3}\}$
Logistic regression	StandardScaler ; max_iter=5000 ; solver=lbfgs	$C\in\{10^{-3},10^{-2},10^{-1},1,10,10^{2},10^{3}\}$
RBF SVM	StandardScaler ; kernel=rbf	$C\in\{10^{-2},10^{-1},1,10,10^{2}\}$ ; gamma $\in\{\texttt{scale},\texttt{auto}\}$
Random forest	200 trees; max_features=sqrt ; random_state=42	max_depth $\in\{\texttt{None},10\}$ ; min_samples_leaf $\in\{1,2\}$

Appendix

Table 5: Downstream-classifier configurations. The same grids and cross-validation procedure are used for every dataset, model instance, pretraining condition, and random encoder seed.

Dataset	MOMENT-S	MOMENT-B	MOMENT-L	Mantis	MantisV2	UniTS	NuTime
Articulary WordRecognition	0.983	0.983	0.983	0.988	0.984	0.899	0.903
Atrial Fibrillation	0.310	0.330	0.387	0.290	0.227	0.320	0.240
BasicMotions	0.982	0.986	0.981	0.993	0.994	0.781	0.482
Character Trajectories	0.977	0.980	0.979	0.944	0.938	0.950	0.828
Cricket	0.938	0.928	0.940	0.974	0.979	0.848	0.923
DuckDuckGeese	0.266	0.302	0.312	0.550	0.524	–	0.249

Appendix

Table 6: Per-dataset classification accuracy of randomly initialized foundation models, averaged across all four downstream classifiers and five initialization seeds. Entries with any missing classifier result are shown as a dash.

Dataset	MOMENT-S	MOMENT-B	MOMENT-L	Mantis	MantisV2	UniTS	NuTime
Articulary WordRecognition	0.992	0.989	0.990	0.993	0.993	0.903	0.983
Atrial Fibrillation	0.117	0.150	0.183	0.200	0.133	0.383	0.317
BasicMotions	1.000	1.000	0.988	1.000	1.000	0.888	0.731
Character Trajectories	0.977	0.975	0.975	0.971	0.962	0.940	0.934
Cricket	0.983	0.986	0.983	0.993	0.986	0.830	0.972
DuckDuckGeese	0.455	0.440	0.475	0.430	0.495	–	0.195

Appendix

Table 7: Per-dataset classification accuracy of pretrained foundation models, averaged across all four downstream classifiers. Entries with any missing classifier result are shown as a dash.

Dataset	MOMENT-S	MOMENT-B	MOMENT-L	Mantis	MantisV2	UniTS	NuTime
Articulary WordRecognition	0.983	0.983	0.982	0.997	0.995	0.954	0.911
Atrial Fibrillation	0.293	0.360	0.413	0.267	0.240	0.307	0.267
BasicMotions	0.995	0.995	0.985	1.000	1.000	0.820	0.455
Character Trajectories	0.981	0.983	0.981	0.982	0.978	0.967	0.865
Cricket	0.942	0.928	0.939	0.992	0.997	0.822	0.975
DuckDuckGeese	0.304	0.348	0.340	0.616	0.580	–	0.264

Appendix

Table 8: Per-dataset top-one accuracy of randomly initialized foundation models using ridge classification. Values are averaged across five initialization seeds.

Dataset	MOMENT-S	MOMENT-B	MOMENT-L	Mantis	MantisV2	UniTS	NuTime
Articulary WordRecognition	0.987	0.985	0.985	0.988	0.987	0.916	0.897
Atrial Fibrillation	0.293	0.373	0.400	0.293	0.227	0.333	0.200
BasicMotions	0.990	0.995	0.970	1.000	1.000	0.790	0.495
Character Trajectories	0.980	0.983	0.982	0.973	0.968	0.952	0.848
Cricket	0.939	0.936	0.944	0.972	0.975	0.856	0.936
DuckDuckGeese	0.260	0.272	0.276	0.596	0.548	–	0.272

Appendix

Table 9: Per-dataset top-one accuracy of randomly initialized foundation models using logistic regression. Values are averaged across five initialization seeds.

Dataset	MOMENT-S	MOMENT-B	MOMENT-L	Mantis	MantisV2	UniTS	NuTime
Articulary WordRecognition	0.985	0.983	0.984	0.979	0.976	0.894	0.908
Atrial Fibrillation	0.307	0.293	0.307	0.373	0.240	0.267	0.253
BasicMotions	0.980	0.995	0.995	0.990	0.975	0.780	0.510
Character Trajectories	0.974	0.979	0.977	0.911	0.907	0.956	0.832
Cricket	0.939	0.933	0.950	0.972	0.978	0.856	0.942
DuckDuckGeese	0.272	0.276	0.304	0.412	0.428	–	0.256

Appendix

Table 10: Per-dataset top-one accuracy of randomly initialized foundation models using RBF SVM. Values are averaged across five initialization seeds.

Dataset	MOMENT-S	MOMENT-B	MOMENT-L	Mantis	MantisV2	UniTS	NuTime
Articulary WordRecognition	0.977	0.982	0.980	0.989	0.976	0.831	0.895
Atrial Fibrillation	0.347	0.293	0.427	0.227	0.200	0.373	0.240
BasicMotions	0.965	0.960	0.975	0.980	1.000	0.735	0.470
Character Trajectories	0.971	0.975	0.976	0.910	0.898	0.923	0.765
Cricket	0.933	0.914	0.928	0.961	0.967	0.858	0.839
DuckDuckGeese	0.228	0.312	0.328	0.576	0.540	–	0.204

Appendix

Table 11: Per-dataset top-one accuracy of randomly initialized foundation models using random forest. Values are averaged across five initialization seeds.

Dataset	MOMENT-S	MOMENT-B	MOMENT-L	Mantis	MantisV2	UniTS	NuTime
Articulary WordRecognition	0.997	0.990	0.997	0.993	0.993	0.933	0.990
Atrial Fibrillation	0.133	0.133	0.200	0.133	0.200	0.400	0.333
BasicMotions	1.000	1.000	1.000	1.000	1.000	0.900	0.750
Character Trajectories	0.981	0.983	0.986	0.988	0.975	0.948	0.934
Cricket	1.000	0.986	1.000	0.986	0.986	0.833	0.986
DuckDuckGeese	0.480	0.520	0.580	0.440	0.560	–	0.180

Appendix

Table 12: Per-dataset top-one accuracy of pretrained foundation models using ridge classification.

Dataset	MOMENT-S	MOMENT-B	MOMENT-L	Mantis	MantisV2	UniTS	NuTime
Articulary WordRecognition	0.993	0.990	0.983	0.993	0.993	0.910	0.980
Atrial Fibrillation	0.133	0.133	0.267	0.200	0.133	0.467	0.267
BasicMotions	1.000	1.000	0.975	1.000	1.000	0.925	0.750
Character Trajectories	0.980	0.976	0.977	0.974	0.968	0.950	0.950
Cricket	0.986	0.986	0.986	0.986	0.986	0.861	0.986
DuckDuckGeese	0.480	0.500	0.540	0.420	0.500	–	0.240

Appendix

Table 13: Per-dataset top-one accuracy of pretrained foundation models using logistic regression.

Dataset	MOMENT-S	MOMENT-B	MOMENT-L	Mantis	MantisV2	UniTS	NuTime
Articulary WordRecognition	0.990	0.983	0.987	0.993	0.993	0.903	0.987
Atrial Fibrillation	0.067	0.200	0.133	0.267	0.133	0.333	0.400
BasicMotions	1.000	1.000	1.000	1.000	1.000	0.825	0.700
Character Trajectories	0.976	0.977	0.973	0.969	0.965	0.948	0.949
Cricket	0.958	0.972	0.958	1.000	0.986	0.833	0.972
DuckDuckGeese	0.380	0.380	0.400	0.440	0.420	–	0.180

Appendix

Table 14: Per-dataset top-one accuracy of pretrained foundation models using RBF SVM.

Dataset	MOMENT-S	MOMENT-B	MOMENT-L	Mantis	MantisV2	UniTS	NuTime
Articulary WordRecognition	0.990	0.993	0.993	0.993	0.993	0.867	0.973
Atrial Fibrillation	0.133	0.133	0.133	0.200	0.067	0.333	0.267
BasicMotions	1.000	1.000	0.975	1.000	1.000	0.900	0.725
Character Trajectories	0.972	0.965	0.966	0.953	0.942	0.916	0.901
Cricket	0.986	1.000	0.986	1.000	0.986	0.792	0.944
DuckDuckGeese	0.480	0.360	0.380	0.420	0.500	–	0.180

Appendix

Table 15: Per-dataset top-one accuracy of pretrained foundation models using random forest.

Dataset	ROCKET	Quant	Hydra
Articulary WordRecognition	0.9933	0.9800	0.9933
Atrial Fibrillation	0.0667	0.2000	0.1333
BasicMotions	1.0000	1.0000	1.0000
Character Trajectories	0.9923	0.9833	0.9861
Cricket	1.0000	0.9722	0.9861
DuckDuckGeese	0.5400	0.4600	0.7000

Appendix

Table 16: Per-dataset classification accuracy of the time-series-specific reference methods on all 30 UEA datasets.

Method	Median latency (ms)	$n$	aeon fit (s) †
Frozen TSFM encoder and aggregation
MantisV2	15.88	100	–
Mantis	18.61	100	–
MOMENT-Small	82.48	100	–
MOMENT-Base ‡	354.91	50	–
MOMENT-Large ∗	1354.18	5	–

Appendix

Table 17: Controlled single-thread CPU diagnostic on synthetic float32

[1,3,512]

input.

n

denotes the number of timed calls. The aeon fit time is measured separately on 100 synthetic training cases and is not comparable to TSFM pretraining.

Neural encoder	Registered operations (GFLOPs)
MantisV2	0.482
Mantis	0.874
MOMENT-Small	7.400
MOMENT-Base	33.070
MOMENT-Large	119.591

Appendix

Table 18: Registered matrix-multiplication and convolution operations for one frozen neural-encoder forward pass on synthetic

[1,3,512]

input. Values are partial instrumented counts rather than total FLOPs.

Time-Series Foundation Models

Time-series foundation models are pretrained sequence models designed to transfer across forecasting, classification, anomaly detection, and other temporal tasks.

See all 301 papers

Sep 29, 2026

Loss-Guided Pretraining Data Selection for Time-Series Foundation Models

Time series foundation models (TSFMs) are pretrained on heterogeneous collections containing billions of observations, yet their training windows are typically sampled without estimating whether they provide useful learning signal. We introduce a static data-selection framework that scores each window with a reference forecaster and retains an intermediate interval within every source dataset. Specifically, we connect forecasting loss to optimization difficulty by showing that normalized squared loss controls the per-sample gradient norm under a local Jacobian condition. We then define a reference loss score and apply dataset-stratified selection to preserve the diversity of samples. Across various TSFM architectures, our method outperforms random selection by an absolute margin and even improves both relative MASE and CRPS over full-data pretraining by retaining fewer candidate pretraining windows. Further analyses show strong cross-scale and cross-architecture score correlations, indicating that a small reference model can often select data for larger targets, provided that the reference and target share compatible difficulty orderings.

Yike Li, Shaoxu Song, Jianmin Wang

Tsinghua University

Time Series Forecasting Training Data Selection Time-Series Foundation Models

Ask PDF

Mar 2, 2026

UTICA: Multi-Objective Self-Distllation Foundation Model Pretraining for Time Series Classification

Self-supervised foundation models have achieved remarkable success across domains, including time series. However, the potential of non-contrastive methods, a paradigm that has driven significant advances in computer vision, remains underexplored for time series. In this work, we adapt DINOv2-style self-distillation to pretrain a time series foundation model, building on the Mantis tokenizer and transformer encoder architecture as our backbone. Through a student-teacher framework, our method Utica learns representations that capture both temporal invariance via augmented crops and fine-grained local structure via patch masking. Our approach achieves state-of-the-art classification performance on both UCR and UEA benchmarks. These results suggest that non-contrastive methods are a promising and complementary pretraining strategy for time series foundation models.

Yessin Moakher, Youssef Attia El Hili, Vasilii Feofanov

Ecole Polytechnique, France · Huawei Noah’s Ark Lab

Self-Supervised Time Series Representation Learning Time-Series Foundation Models Time Series Classification Self-Supervised Learning Self-Distillation

Ask PDF

Jul 22, 2026

Post-Training in Time Series Foundation Models: A Unifying Framework

Time series foundation models (TSFMs) have emerged as general-purpose models for time series analysis, but pretraining alone is often insufficient for reliable downstream deployment. Bridging this gap requires further intervention to handle domain shift, task heterogeneity, limited supervision, and computational constraints, which motivates post-training as a broad class of methods to adapt, augment, compose, calibrate, or specialize pretrained TSFMs for downstream tasks. In this work, we analyze TSFM post-training methods based on their locus of intervention in the prediction pipeline, yielding five categories: parameter adaptation, context augmentation, model composition, output processing and uncertainty control, and compression and specialization. Within each category, we study main representative methods and discuss their current limitations. We further identify future directions toward controlled adaptation, reliable context construction, uncertainty-aware model composition, calibrated output processing, and deployment-aware specialization. Overall, by providing a unifying framework for the emerging TSFM post-training landscape, this work aims to support future research to navigate the design space between a pretrained TSFM and its reliable downstream deployment.

Shifeng Xie, Ambroise Odonnat, Zehao Xiao +7

Noah’s Ark Lab · Université Paris Cité · Inria +2

Model Compression Time-Series Foundation Models

Ask PDF

Abstract

Time-Series Foundation Models

Explore similar work

Loss-Guided Pretraining Data Selection for Time-Series Foundation Models

UTICA: Multi-Objective Self-Distllation Foundation Model Pretraining for Time Series Classification

Post-Training in Time Series Foundation Models: A Unifying Framework