MarathonTS: a benchmark of long synthetic time series from window assembly and long-series generators
Abstract
Many applications of synthetic time series require a series that is much longer than the windows on which a generative model is trained. Such a series can be assembled from a pool of short generated windows, which requires deciding which windows are taken, in what order, and how their seams are processed, or it can be produced directly by a long-series generator. This study presents MarathonTS, to our knowledge the first open benchmark that compares both families under one protocol. Every method produces a series of the same length on 9 open hourly datasets with 10 seeds. The series is evaluated by its fidelity to the real data (83 metrics in 12 groups, against three real references: train, validation and an out-of-time split), by its utility (models are trained on the synthetic series only and tested on real data, TSTR against TRTR, in 14 classification and regression tasks), by its memorisation of the training data and by its computational cost. The statistical analysis is carried out as a separate stage that reads the published tables and gives the rankings of every method in every slice, the answers to the research questions with tests and uncertainty, and the results of a pre-specified confirmatory family. In the main run, the methods differ in both confirmatory cells; the choice of windows matters more than the processing of their seams, which does not improve the series, and no method trains the downstream models as well as the real series. The top tier of the leaderboard is reproduced on other hardware, whereas the exact places within it are not. For the reproducibility of the study, the code, the data hashes and all results are publicly available (links below).
Figures & tables
| Dataset | Target | Source step | Hours | |
|---|---|---|---|---|
| ETTh1 | OT | 1 h | 17 420 | 62 |
| Electricity Load Diagrams | sum over 370 clients | 15 min | 35 065 | 125 |
| Household Electric Power | Global_active_power | 1 min | 124 | |
| PJM Hourly Energy (PJME) | PJME_MW | 1 h | 145 392 | 519 |
| Beijing PM2.5 | pm2.5 | 1 h | 43 800 | 156 |
| Jena Climate 2009–2016 | T (degC) | 10 min | 250 |
| Group | What it measures | Examples | |
|---|---|---|---|
| F1 | 15 | Marginal distribution of values | KS, Wasserstein-1, quantile errors, Hill tail index |
| F2 | 9 | Increments | KS of , ; volatility ratio |
| F3 | 10 | Temporal dependence | ACF within and beyond a window, PACF, spectrum, DFA |
| F4 | 5 | Seasonality and calendar | weekly and daily profiles, calendar continuity |
| F5 | 8 | Seams | jumps at seams, change points at seams, seam detector |
| F6 | 7 | Distribution of windows | discriminative score, C2ST, MMD, feature FID |
| Domain | Criterion | Content |
| Fidelity | distribution | F1 values, F2 increments, F12 outliers and extremes |
| dependence | F3 temporal dependence, F4 seasonality | |
| windows | F6 distribution of windows, F9 diversity and coverage | |
| long structure | F7 | |
| target | F8 tail events of U2.1, F11 class balance | |
| seams | F5 |
| Place | Method | Family | Mean rank | 95 % interval | P(first) | Top tier | Datasets |
|---|---|---|---|---|---|---|---|
| 1 | zedge_quilting ∗ | assembly | 8.27 | [7.35, 9.20] | 0.85 | yes | 9 |
| 2 | zedge_zseries ∗ | assembly | 8.79 | [8.28, 9.47] | 0.08 | yes | 9 |
| 3 | zedge_greedy ∗ | assembly | 8.95 | [7.97, 10.00] | 0.01 | yes | 9 |
| 4 | zedge_pctsp ∗ | assembly | 9.18 | [8.13, 10.25] | 0.00 | yes | 9 |
| 5 | zedge_knn ∗ | assembly | 9.33 | [8.32, 10.48] | 0.00 | yes | 9 |
| 6 | zedge_atsp ∗ | assembly | 9.39 | [7.83, 11.10] | 0.03 | yes | 9 |
| 95 % interval | ||||||||
|---|---|---|---|---|---|---|---|---|
| Test | Comparison | Effect | mean | median | Better | Worse | Verdict | |
| Classification, U2.1 | ||||||||
| C1 | all methods (Friedman) | 0.012 | differ | |||||
| C2 | Diffusion-TS vs assembly median | 1.89 | [ , 4.85] | [ , 4.68] | 5 | 2 | 0.49 | inconclusive |
| C2 | Chronos vs assembly median | 0.08 | [ , 0.95] | [ , 1.18] | 2 | 2 | 1.00 | inconclusive |
| C3 | assemblers vs random path | 0.06 | [ , 1.01] | [ , 1.41] | 2 | 0 | 1.00 | inconclusive |
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Source | Licence | Reference |
|---|---|---|---|
| ETTh1 | GitHub, ETDataset | CC BY-ND 4.0 | Zhou et al. [2021] |
| Electricity | UCI #321 | CC BY 4.0 | Trindade [2015] |
| Household | UCI #235 | CC BY 4.0 | Hébrail and Bérard [2012] |
| PJME | Kaggle, Hourly Energy Consumption | CC0 1.0 | Mulla [2018] |
| PM2.5 | UCI #381 | CC BY 4.0 | Liang et al. [2015] |
| Jena | MPI-BGC weather station (Keras mirror) | CC BY 4.0 | Max Planck Institute for Biogeochemistry [2016] |
| Dataset | Chronos | TimesFM 3.0 | Timestral |
|---|---|---|---|
| ETTh1 | none (evaluation only) | none (a test set, not in pretraining) | unknown |
| Electricity | exact : Electricity at 15 min, hourly and weekly | none (a test set) | unknown |
| Household | none | exact : the same series under another name, 2007–2010 | unknown |
| PJME | none | none found | unknown |
| PM2.5 | related : KDD Cup 2018, Beijing stations, 2017–2018 | related : Beijing Air Quality, 12 stations, 2013–2017 | unknown |
| Jena | weak: gridded reanalysis only | weak: gridded reanalysis only | unknown |
| Method | Idea | Settings |
|---|---|---|
| A. Window selection and order ( of windows, without repetition) | ||
| <pool>_zseries | proprietary (zypl.ai), evaluated as a black box | not released |
| <pool>_rand | random windows in random order; control | — |
| <pool>_greedy | nearest neighbour by the jump at the seam, from a random window | — |
| <pool>_atsp | asymmetric TSP over the pool, Lin–Kernighan–Helsgaun [ Helsgaun, 2000 ] ; cheapest run of consecutive windows of the tour | LKH via elkai |
| <pool>_pctsp | prize-collecting TSP [ Balas, 1989 ] : which windows and their order at once | OR-Tools; at most 2 000 solutions or 60 s per unit |
| ID | How it is computed | |
|---|---|---|
| F1. Marginal distribution of values | ||
| F1.1 | 1 | Kolmogorov–Smirnov statistic |
| F1.2 | 1 | Wasserstein-1 distance divided by the standard deviation of the reference |
| F1.3 | 1 | Anderson–Darling -sample statistic |
| F1.4 | 1 | correlation of the quantiles 0.01–0.99 (higher is better) |
| F1.5 | 3 | quantile errors in reference standard deviations: mean over , and at 0.01 and 0.99 |
| Main run | Replication run | ||||||
| Method | Step | Error | 90 % | h | Step | h | |
| vae_rand | 0.68 | 1 | 4 | 18 % | 0.68 | 20 % | |
| vae_greedy | 0.72 | 1 | 3 | 15 % | 0.68 | 16 % | |
| vae_atsp | 0.75 | 1 | 2 | 10 % | 0.75 | 12 % | |
| vae_pctsp | 0.74 | 1 | 2.5 | 10 % | 0.75 | 9 % | |
| vae_mbb | 0.68 | 1 | 5 | 20 % | 0.69 | 19 % | |
| Group | Type | Target | Primary metric |
|---|---|---|---|
| U2.1 | binary | above the 0.90 quantile | ROC AUC |
| U2.2 | binary | above the 0.90 quantile, | ROC AUC |
| U2.3 | binary | above the 0.95 and the 0.99 quantile (rare events) | ROC AUC |
| U2.4 | binary | below the 0.10 quantile | ROC AUC |
| U2.5 | 3 classes | below the 0.10, between, or above the 0.90 quantile | log-loss |
| U2.6 | regression | value , | MASE |
| Component | What is fixed | Protocol |
|---|---|---|
| Core metrics | one or two canonical fidelity metrics per criterion for the core rankings; the primary metric of every utility task | RQ1, core rankings |
| Confirmatory family | cells U2.1 and U2.6 at , linear model, OOT, primary metric; tests C1–C4; the long-series generators of C2 (Diffusion-TS, Chronos) | 3.9 |
| Contrasts | RQ2: each long-series generator against the random path and the assembly median of one pool; RQ3: each assembler against the random path, each joiner against the same path without seam processing, and the main effects | RQ2, RQ3 |
| Multiplicity | C1–C4: Holm over the ten tests; secondary questions (RQ0, RQ2–RQ6, RQ8, RQ9): Benjamini–Hochberg within a question; RQ7: exploratory, no conclusions | 2, multiple comparisons |
| Noise and verdicts | noise: the median difference between seeds of the random path, or the larger between-seed spread for independently trained methods; “better” or “worse”: significant after correction and a median effect of more than one noise unit; “same”: TOST with the margin of one noise unit | 2, noise; 2, verdict |
| Exclusion rules | pairs of a pretrained model and a dataset in its training data; seam-processing variants not ranked; calendar-continuity metrics F4.4–F4.5 left out; a missing utility cell enters at the chance level or the worst value of its block | 2; RQ1, ranked methods |
| Metric | Chance level |
|---|---|
| ROC AUC (binary; macro for three classes) | 0.5 |
| PR AUC | |
| Log-loss | the entropy for a binary task; for classes |
| Brier score | |
| F1, ECE, lift at 10 %, macro F1 | none: the worst value of the block |
| Losses of regression, quantile, volatility and multi-step tasks | none: the worst value of the block (not needed in the published runs, in which no such cell failed) |
| Run | Variant | Methods datasets | Classification | Regression |
|---|---|---|---|---|
| main | pre-specified | 0.0014 / 0.0006 | / | |
| every method, incomplete (Skillings–Mack) | 0.0025 / 0.0012 | / | ||
| every method, complete datasets | 0.0015 / 0.0002 | 0.0018 / 0.0003 | ||
| methods of the replication run | 0.79 / 0.81 | 0.0002 / | ||
| replication | pre-specified | 0.29 / 0.29 | / | |
| every method, incomplete (Skillings–Mack) | 0.45 / 0.47 | / |
| Task | Model | Split | TRTR | TSTR, median | TSTR, best | Best method |
| U2.1 (AUC) | linear | val | 0.747 | 0.664 | 0.749 | zedge_pctsp ∗ |
| linear | OOT | 0.743 | 0.657 | 0.742 | zedge_ts_ar ∗ | |
| gbdt | val | 0.739 | 0.644 | 0.703 | diffusion_ts | |
| gbdt | OOT | 0.764 | 0.651 | 0.758 | zedge_ts_ar ∗ | |
| mlp | val | 0.757 | 0.652 | 0.727 | zedge_quilting ∗ | |
| mlp | OOT | 0.789 | 0.672 | 0.757 | zedge_quilting ∗ |
| Method | Seams | Near a seam, % | F5.6 | Chance level | Ratio |
|---|---|---|---|---|---|
| timestral | 4 | 0.6 | 0.000 | 0.005 | 0.00 |
| vae_zseries ∗ | 29 | 4.0 | 0.022 | 0.040 | 0.57 |
| zedge_zseries ∗ | 29 | 4.0 | 0.024 | 0.040 | 0.60 |
| vae_pctsp | 29 | 4.0 | 0.027 | 0.040 | 0.67 |
| vae_atsp | 29 | 4.0 | 0.028 | 0.040 | 0.67 |
| zedge_ts_ar ∗ | 58 | 8.1 | 0.029 | 0.081 | 0.38 |
| Method | Log ratio | 95 % interval | Near copies | 95 % interval | Exact copies |
| timestral | [ , ] | 0.61 | [0.00, 1.00] | 0 / 10 530 | |
| timesfm | [ , ] | 0.50 | [0.09, 0.86] | 0 / 10 530 | |
| diffusion_ts | [ , ] | 0.27 | [0.05, 0.69] | 0 / 10 530 | |
| vae_pctsp | [ , ] | 0.22 | [0.00, 0.75] | 0 / 10 530 | |
| vae_atsp | [ , ] | 0.24 | [0.00, 0.77] | 0 / 10 530 | |
| vae_zseries_xfade ∗ | [ , ] | 0.12 | [0.00, 0.40] | 0 / 10 530 |
| Method | Score | Noise-aware rank | Ordinary rank |
|---|---|---|---|
| A | 1.2 | 1 | |
| B | 0.6 | 2 | |
| C | 0.0 | 3 |
| Noise-aware | Ordinary | ||||
|---|---|---|---|---|---|
| Run | Method | Place | Top tier | Place | Top tier |
| Main | zedge_knn | 5 | yes | 8 | yes |
| zedge_ts_ar | 7 | yes | 5 | yes | |
| zedge_ts_long | 8 | yes | 7 | yes | |
| zedge_rand | 11 | yes | 11 | no | |
| Replication | vae_mbb | 8 | yes | 9 | yes |
| Main run | Replication run | ||||||
|---|---|---|---|---|---|---|---|
| Method | Datasets | Tier | Method | Datasets | Tier | ||
| zedge_quilting ∗ | 9 | – | yes | vae_herding | 9 | – | yes |
| zedge_zseries ∗ | 9 | 0.410 | yes | diffusion_ts | 9 | 0.500 | yes |
| zedge_greedy ∗ | 9 | 0.410 | yes | vae_quilting | 9 | 0.500 | yes |
| zedge_pctsp ∗ | 9 | 0.293 | yes | vae_zseries ∗ | 9 | 0.500 | yes |
| zedge_knn ∗ | 9 | 0.410 | yes | timesfm | 6 | 0.500 | yes |
| Main run | Replication run | |
|---|---|---|
| GPU | 1 NVIDIA H200 (143 GB) | 1 NVIDIA L4 (24 GB) |
| CPU cores | 192 | 12 |
| System | Ubuntu 22.04.5, driver 580.173.02 | Google Colab (Linux 6.6) |
| Software | Python 3.12.13, PyTorch 2.9.1, TensorFlow 2.20.0 (environment manifests published) | pinned requirements of the repository |
| Code | internal commit 365232d6 (23 October 2026) | completed before the freeze; results in commit 7fea7530 (21 October 2026) |
| Methods | 30 (24 ranked) | 16 of the VAE pool and the open generators (13 ranked) |
| Replication run (L4) | Main run (H200) | |||||||
|---|---|---|---|---|---|---|---|---|
| Method | Place | Mean rank | 95 % interval | Tier | Place | Mean rank | 95 % interval | Tier |
| vae_herding | 1 | 5.54 | [4.86, 6.18] | yes | 1 | 5.57 | [4.62, 6.49] | yes |
| diffusion_ts | 2 | 6.27 | [4.76, 7.59] | yes | 3 | 6.24 | [4.91, 7.43] | yes |
| vae_quilting | 3 | 6.36 | [5.82, 6.92] | yes | 4 | 6.51 | [5.98, 7.10] | yes |
| vae_zseries ∗ | 4 | 6.41 | [5.67, 7.20] | yes | 2 | 5.76 | [5.16, 6.35] | yes |
| timesfm | 5 | 6.72 | [5.29, 8.27] | yes | 5 | 6.77 | [5.43, 8.26] | yes |
Time Series Generation
Time series generation creates synthetic sequential data that preserve temporal patterns, cross-variable structure, and controllable conditions.
See all 99 papers