Fifty-Two Shades of Pig: What the UCR Archive Really Asks Classifiers to Do
Organizations: University of Freiburg · Prior Labs · ELLIS Institute Tübingen
Abstract
The pigs were bleeding. That was the point of the experiment: could airway pressure reveal hemorrhage before it became critical? In the UCR Time Series Classification Archive, the de facto benchmark for Time Series Classification (TSC), the same recordings ask a different question: "which of 52 pigs produced this trace?". The dataset PigAirwayPressure is not an outlier. Although the archive is the standard benchmark, whether its tasks resemble the problems TSC is meant to solve has rarely been examined. We find that several datasets are repurposed from their original task, built from toy setups, or have flawed test splits. We further question how representative the archive's ordered non-temporal sequences (e.g. spectra, shape outlines) are of real TSC, and the potential downsides from the decomposition of multivariate problems. Our findings call for more careful interpretation of archive-wide scores and greater attention to deployment-relevant evaluation. The audit is at https://github.com/automl/ucr-provenance-audit.
Figures & tables
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Benchmark target | Source target or purpose | Label by |
|---|---|---|---|
| ItalyPowerDemand | Cold season (Oct–Mar) vs warm season (Apr–Sep) | None; utility demand records | Curators |
| PowerCons | Warm vs cold season | None; EDF consumption records | Curators |
| Chinatown | Weekend vs weekday | None; Melbourne pedestrian counts | Curators |
| DodgerLoopDay | Day of week | Unsupervised event detection; no class target | Curators |
| DodgerLoopGame | Whether a Dodgers home game took place that day | Unsupervised detection of game traffic events; schedule used as validation ground truth | Curators |
| DodgerLoopWeekend | Weekend vs weekday | As DodgerLoopDay | Curators |
| Dataset | Benchmark target | Source target or purpose | Label by |
|---|---|---|---|
| PigAirwayPressure | Which of 52 pigs | Before vs during induced bleeding | Curators |
| PigArtPressure | Which of 52 pigs | As PigAirwayPressure | Curators |
| PigCVP | Which of 52 pigs | As PigAirwayPressure | Curators |
| CinCECGTorso | Which of four people | Infarct extent and location | Curators |
| MelbournePedestrian | Which of ten sensor locations | Pedestrian monitoring; no class target | Curators |
| SemgHandSubjectCh2 | Which of five subjects | Six hand-grasp types | Curators |
| Dataset | Benchmark target | Source target or purpose | Label by |
|---|---|---|---|
| AllGestureWiimoteX | Which of ten gestures | User identity; gesture was a protocol factor | Curators |
| AllGestureWiimoteY | As AllGestureWiimoteX | As AllGestureWiimoteX | Curators |
| AllGestureWiimoteZ | As AllGestureWiimoteX | As AllGestureWiimoteX | Curators |
| Earthquakes | Whether a large event follows a window; curator thresholds | Continuous seismic monitoring; no class target | Curators |
| HouseTwenty | Whole-house aggregate vs summed dryer/washer channels | REFIT load monitoring; no class target | Curators |
| ECG5000 | Five automatically assigned beat types | No clinician beat labels used for this record | Curators |
| Dataset | Benchmark target | Source target or purpose | Label by |
|---|---|---|---|
| BME | Bell at begin, none, or at end of a central plate | None; unpublished generator (Grenoble LIG/AMA) | Generator |
| UMD | Bell pointing up, absent, or pointing down | As BME | Generator |
| CBF | Cylinder, bell or funnel shape in noise | None; demo signal for local feature extraction | Generator |
| SyntheticControl | Six control-chart patterns | None; no process measured, generated to test a similarity method | Generator |
| TwoPatterns | Ordered pair of up/down steps in noise | None; designed for thesis experiments | Generator |
| Mallat | Eight generated signal variants; classes undocumented | None; wavelet test signal | Generator |
| Dataset | Archive | Test = train | % | Acc. dup. | Acc. rest | Mixed |
|---|---|---|---|---|---|---|
| PEMS-SF | UEA | 79 / 173 | 45.7 | 1.000 | 0.681 | – |
| PowerCons | UCR | 67 / 180 | 37.2 | 1.000 | 0.991 | – |
| Crop | UCR | 2,753 / 16,800 | 16.4 | 0.992 | 0.722 | – |
| WormsTwoClass | UCR | 10 / 77 | 13.0 | 1.000 | 0.746 | – |
| Worms | UCR | 10 / 77 | 13.0 | 1.000 | 0.701 | – |
| Libras | UEA | 19 / 180 | 10.6 | 1.000 | 0.907 | – |
| Dataset | Relevant new case | Verdict | What the split tests and why a concern can rise |
|---|---|---|---|
| ACSF1 | New appliances | Unclear | The 100/100 split matches the source’s Test Protocol 1.0, which trains on the first measurement session and tests on the second session of the same appliances. The archive does not say which protocol was used, and the series length does not match the documented one-hour sessions. Whether the test presents familiar appliances again or appliances the model has never seen therefore cannot be established. A concern arises only if the result is presented as performance on new appliances. |
| Adiac | New independently collected samples | Unclear | The cases are outlines of tiny biological specimens. We do not know whether samples grown from the same biological source, or photographed on the same microscope slide, occur in both sets. They could share growth or imaging conditions. The records do not let us check whether the test samples were collected independently. |
| AllGestureWiimoteX | New people | Unclear | The data contain gestures from ten people. The archive shuffled and sampled recordings, but does not say that all recordings from a person were kept in one set. We therefore do not know whether test users are new. This matters if the intended application is to recognise gestures from people the model has never seen. |
| AllGestureWiimoteY | New people | Unclear | Same as AllGestureWiimoteX . |
| AllGestureWiimoteZ | New people | Unclear | Same as AllGestureWiimoteX . |
| ArrowHead | New physical objects | Unclear | Each case is an outline of an arrowhead. The source assigned outlines to a 36/175 split at random. We cannot check whether multiple views or representations of the same object were kept together. |
Time Series Classification
Time series classification assigns labels to sequential data, using temporal patterns to support decisions in health, sensing, science, and industry.
See all 192 papers