The pigs were bleeding. That was the point of the experiment: could airway pressure reveal hemorrhage before it became critical? In the UCR Time Series Classification Archive, the de facto benchmark for Time Series Classification (TSC), the same recordings ask a different question: "which of 52 pigs produced this trace?". The dataset PigAirwayPressure is not an outlier. Although the archive is the standard benchmark, whether its tasks resemble the problems TSC is meant to solve has rarely been examined. We find that several datasets are repurposed from their original task, built from toy setups, or have flawed test splits. We further question how representative the archive's ordered non-temporal sequences (e.g. spectra, shape outlines) are of real TSC, and the potential downsides from the decomposition of multivariate problems. Our findings call for more careful interpretation of archive-wide scores and greater attention to deployment-relevant evaluation. The audit is at https://github.com/automl/ucr-provenance-audit.
Figures & tables
Figure 1: We show the archive datasets distribution over the different realism criterion. Our audit identifies only 38/128 UCR datasets as real temporal tasks; about a quarter of these are prone to split leakage. We indicate the proportion of datasets failing our leakage criteria (white boxes) and with no leakage found (black boxes) within each category, although most splits cannot be checked (53 of the 76 real-task datasets are Unclear ).
Figure 2: MiniRocket accuracy with the original training set (with duplicates, grey ) and after removing the duplicates ( blue ). On several datasets, duplication inflates test accuracy.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: The six audit fields, two per stage of a dataset’s life. Green marks the value that passes, red a value that fails or flags leakage, grey a value the verdict does not read.
Dataset
Benchmark target
Source target or purpose
Label by
ItalyPowerDemand
Cold season (Oct–Mar) vs warm season (Apr–Sep)
None; utility demand records
Curators
PowerCons
Warm vs cold season
None; EDF consumption records
Curators
Chinatown
Weekend vs weekday
None; Melbourne pedestrian counts
Curators
DodgerLoopDay
Day of week
Unsupervised event detection; no class target
Curators
DodgerLoopGame
Whether a Dodgers home game took place that day
Unsupervised detection of game traffic events; schedule used as validation ground truth
Curators
DodgerLoopWeekend
Weekend vs weekday
As DodgerLoopDay
Curators
Appendix
Table 1: Task-construction cases in the current audit: metadata. None means no classification target is documented, not that the source data had no scientific purpose. † also listed under another mechanism.
Dataset
Benchmark target
Source target or purpose
Label by
PigAirwayPressure
Which of 52 pigs
Before vs during induced bleeding
Curators
PigArtPressure
Which of 52 pigs
As PigAirwayPressure
Curators
PigCVP
Which of 52 pigs
As PigAirwayPressure
Curators
CinCECGTorso
Which of four people
Infarct extent and location
Curators
MelbournePedestrian
Which of ten sensor locations
Pedestrian monitoring; no class target
Curators
SemgHandSubjectCh2
Which of five subjects
Six hand-grasp types
Curators
Appendix
Table 2: Task-construction cases in the current audit: identity. None means no classification target is documented, not that the source data had no scientific purpose. † also listed under another mechanism.
Dataset
Benchmark target
Source target or purpose
Label by
AllGestureWiimoteX
Which of ten gestures
User identity; gesture was a protocol factor
Curators
AllGestureWiimoteY
As AllGestureWiimoteX
As AllGestureWiimoteX
Curators
AllGestureWiimoteZ
As AllGestureWiimoteX
As AllGestureWiimoteX
Curators
Earthquakes
Whether a large event follows a window; curator thresholds
Continuous seismic monitoring; no class target
Curators
HouseTwenty
Whole-house aggregate vs summed dryer/washer channels
REFIT load monitoring; no class target
Curators
ECG5000
Five automatically assigned beat types
No clinician beat labels used for this record
Curators
Appendix
Table 3: Task-construction cases in the current audit: replacement. None means no classification target is documented, not that the source data had no scientific purpose.
Dataset
Benchmark target
Source target or purpose
Label by
BME
Bell at begin, none, or at end of a central plate
None; unpublished generator (Grenoble LIG/AMA)
Generator
UMD
Bell pointing up, absent, or pointing down
As BME
Generator
CBF
Cylinder, bell or funnel shape in noise
None; demo signal for local feature extraction
Generator
SyntheticControl
Six control-chart patterns
None; no process measured, generated to test a similarity method
Generator
TwoPatterns
Ordered pair of up/down steps in noise
None; designed for thesis experiments
Generator
Mallat
Eight generated signal variants; classes undocumented
None; wavelet test signal
Generator
Appendix
Table 4: Task-construction cases in the current audit: toy source task. None means no classification target is documented, not that the source data had no scientific purpose. † also listed under another mechanism.
Dataset
Archive
Test = train
%
Acc. dup.
Acc. rest
Mixed
PEMS-SF
UEA
79 / 173
45.7
1.000
0.681
–
PowerCons
UCR
67 / 180
37.2
1.000
0.991
–
Crop
UCR
2,753 / 16,800
16.4
0.992
0.722
–
WormsTwoClass
UCR
10 / 77
13.0
1.000
0.746
–
Worms
UCR
10 / 77
13.0
1.000
0.701
–
Libras
UEA
19 / 180
10.6
1.000
0.907
–
Appendix
Table 5: Exact cross-split matches and MiniRocket accuracy, retained from the original generated results. Counts refer to test series, not pairs. Mixed counts duplicate groups with conflicting labels; – denotes none.
Dataset
Relevant new case
Verdict
What the split tests and why a concern can rise
ACSF1
New appliances
Unclear
The 100/100 split matches the source’s Test Protocol 1.0, which trains on the first measurement session and tests on the second session of the same appliances. The archive does not say which protocol was used, and the series length does not match the documented one-hour sessions. Whether the test presents familiar appliances again or appliances the model has never seen therefore cannot be established. A concern arises only if the result is presented as performance on new appliances.
Adiac
New independently collected samples
Unclear
The cases are outlines of tiny biological specimens. We do not know whether samples grown from the same biological source, or photographed on the same microscope slide, occur in both sets. They could share growth or imaging conditions. The records do not let us check whether the test samples were collected independently.
AllGestureWiimoteX
New people
Unclear
The data contain gestures from ten people. The archive shuffled and sampled recordings, but does not say that all recordings from a person were kept in one set. We therefore do not know whether test users are new. This matters if the intended application is to recognise gestures from people the model has never seen.
AllGestureWiimoteY
New people
Unclear
Same as AllGestureWiimoteX .
AllGestureWiimoteZ
New people
Unclear
Same as AllGestureWiimoteX .
ArrowHead
New physical objects
Unclear
Each case is an outline of an arrowhead. The source assigned outlines to a 36/175 split at random. We cannot check whether multiple views or representations of the same object were kept together.
Appendix
Table 6: UCR split-audit categorization (128 datasets): Concern 32.03%; Unclear 50.78%; Separated 5.47%; Related periods 5.47%; Generated 6.25%.
Time series machine learning (TSML) is a growing research field that spans a wide range of tasks. The popularity of established tasks such as classification, clustering, and extrinsic regression has, in part, been driven by the availability of benchmark datasets. An archive of 30 multivariate time series classification datasets, introduced in 2018 and commonly known as the UEA archive, has since become an essential resource cited in hundreds of publications. We present a substantial expansion of this archive that more than quadruples its size, from 30 to 133 classification problems. We also release preprocessed versions of datasets containing missing values or unequal length series, bringing the total number of datasets to 147. Reflecting the growth of the archive and the broader community, we rebrand it as the Multiverse archive to capture its diversity of domains. The Multiverse archive includes datasets from multiple sources, consolidating other collections and standalone datasets into a single, unified repository. Recognising that running experiments across the full archive is computationally demanding, we recommend a subset of the full archive called Multiverse-core (MV-core) for initial exploration. To support researchers in using the new archive, we provide detailed guidance and a baseline evaluation of established and recent classification algorithms, establishing performance benchmarks for future research. We have created a dedicated repository for the Multiverse archive that provides a common aeon and scikit-learn compatible framework for reproducibility, an extensive record of published results, and an interactive interface to explore the results.
Matthew Middlehurst, Aiden Rushbrooke, Ali Ismail-Fawaz +6
University of Bradford · University of East Anglia · Université de Haute-Alsace +2
We introduce RocketPFN, a training-free pipeline for time series classification that combines random convolutional feature extraction (Rocket) with in-context classification via a pretrained tabular foundation model (TabPFN v2.5). On 92 UCR datasets (30-resample protocol), RocketPFN matches HC2, the strongest published method on the archive, in mean accuracy (both 0.900, Wilcoxon p=0.50), with no training on the target data and a median inference time of 30 seconds per fold. It also significantly outperforms every individual classifier in the HC2 ensemble. On UEA (20 datasets) the difference is likewise not statistically significant. A separate comparison concerns TSC foundation models: when paired with the same downstream classifier, MOMENT, Mantis, and MantisV2 are all significantly outperformed by RocketPFN using fewer extracted features and no learned parameters (p<0.001 in each case). This holds even when the encoders were pretrained on corpora that include the UCR training samples. We propose this two-stage pipeline as a reference point for evaluating zero-shot TSC foundation models.
Franco Martino O'Rourke, Ana Trisovic, Dimitris Bertsimas
Time series classification (TSC) is dominated by a two-stage paradigm: train a feature encoder -- either from scratch on the target dataset or via pretraining on large corpora -- and then fit a task-specific classifier on top. While effective, this decoupling optimizes representation learning independently of the classification objective, requires per-dataset training, and prevents the model from exploiting label information during inference. We introduce TimEE, a 4.5M-parameter foundation model for end-to-end TSC via in-context learning. Given a labeled support set and a query time series, TimEE directly outputs a predicted class distribution in a single forward pass with no per-dataset training required. Following the prior-data fitted network (PFN) framework, TimEE is meta-trained exclusively on synthetic TSC tasks, where each task contains time series with distinct class identities arising from structured distributional shifts in the generative process. Despite seeing no real time series during pre-training, TimEE ranks first in ROC AUC (and third on accuracy) on the UCR benchmark among all compared methods, which include both foundation models and supervised deep learning baselines. To our knowledge, TimEE is the first purely synthetic-pretrained model to reach state-of-the-art performance on the UCR benchmark. These results establish end-to-end ICL with synthetic priors as a compelling, largely unexplored direction for TSC, with scaling, prior design, and richer generation mechanisms as natural avenues for improvement. Code is publicly available at http://github.com/automl/timee.
Jaris Küken, Shi Bin Hoo, Martin Mráz +2
University of Freiburg · Zuse School ELIZA Darmstadt · Prior Labs +1