TSB-SEG: A Systematic Time-Series Segmentation Benchmark
Authors: Félix Chavelli, Arik Ermshaus, Fan Yang, Patrick Schäfer, John Paparrizos, Paul Boniol
Organizations: Inria, ENS, CNRS, PSL · Humboldt-Universität zu Berlin · The Ohio State University · The Ohio State University and Aristotle University of Thessaloniki
Time-series segmentation, studied as either Change Point Detection (CPD) or State Detection (SD), underpins a broad range of monitoring and diagnostic applications. However, the field suffers from a persistent divide between the statistical community (focusing on CPD) and the machine learning and data mining communities (focusing on SD). Consequently, the literature remains fragmented: evaluations typically cover narrow methodological families, datasets are often small or homogeneous, and CPD and SD are rarely evaluated together. We address these gaps with TSB-SEG, a unified benchmark comprising 592 univariate and multivariate time series from 8 heterogeneous datasets and surveying 26 algorithms that span five decades of progress. To ensure a fair comparison, we introduce a two-pool design: a scalable main pool of tractable detectors used for the aggregate ranking, and an extended-scope pool for computationally intensive methods. All methods are unified under tsseg, an open-source library to ensure full reproducibility. Our systematic evaluation yields three insights: (a) no single detector dominates across domains, and per-dataset hyperparameter tuning remains decisive for performance, (b) CPD accuracy is a useful, though imperfect, proxy for SD accuracy, and (c) data characteristics should guide method selection. We provide actionable guidance based on data properties, error tolerances, and tuning budgets, identify impactful hyperparameters, and characterize algorithm robustness to these choices.
Figures & tables
Table 1: Comparison with existing TSS benchmarks.
Figure 1: Time Series Segmentation: Change Point Detection (CPD) and State Detection (SD).
Figure 2: Time-series segmentation taxonomy: search procedure (columns; DP: dynamic programming) and homogeneity criterion (rows). Tinted: evaluated in TSB-SEG ; white: representative methods not evaluated (Appendix D ). K: needs (filled) or accepts (outlined) nCP or nS (Section 4.3 ).
Table 4
Figure 3: CPD performance - Non-Guided and Guided modes. Strip plots: mean over the 592 series of the tuned configuration (dot), the median and worst-case ones (thick body, thin stem) and the default one ( ◊ ); colour: CPD or SD method; dotted line: AMOC. CD diagrams: mean ranks; bars join rank-consecutive methods that a two-sided Wilcoxon signed-rank test with Holm correction does not separate ( α=0.1 ).
Figure 4: SD performance - Non-Guided and Guided modes. (a, b) SMS strip plots and (c) CD diagrams, as in Figure 3 ; in (c, top), HDP-HSMM is also tied with AutoPlait and E2USD.
Figure 5: CPD vs. SD on every run of the SD methods, both modes. (a) SMS against BiC; the dashed envelope holds 95% of the runs, the tails outside it in black. (b) Distribution of r=SMS−BiC . (c) Mean share of each SMS error type per zone (default configurations).
Figure 6: Runtime-accuracy trade-off and SD error breakdown. (a, b) Mean tuned BiC against median runtime of the tuned configuration (circles: CPD methods; squares: SD methods). (c) Share of each SMS error type in the penalty of each SD method (default configuration; NG: Non-Guided, G: Guided).
Appendix figures & tables32 assets
Supplementary material from the paper’s appendix.
Appendix
App.
Content
Supports
Part I - Benchmark design and protocol
B
Notations
Section 2
C
Datasets, provenance and uncertainty of the labels, licences
Section 4.2
D
Per-method descriptions
Section 4.1
E
Re-implementations and univariate/multivariate adaptations
Section 4
F
Formal definitions of the metrics
Section 4.4
Appendix
Table 4: Reader’s guide to the appendix.
Symbol
Definition
T
Time series T∈Rn×d
n
Time series length
d
Time series number of dimensions
Ts,e
Subsequence (ts,…,te) of T
ℓ
Subsequence / window length
C
Set of change points { CPi }
Appendix
Table 5: List of symbols and notations used in this paper.
Dataset
Licence / Usage Terms
PAMAP2 [ 123 ]
CC BY 4.0
USC-HAD [ 126 ]
Licence not explicit; presented as freely available in the original paper
TSSB [ 4 ]
BSD 3-Clause License
UTSA [ 94 ]
No explicit licence at the source; redistributed under BSD 3-Clause in the CLaP repository
ActRecTut [ 124 ]
GPL-3.0-or-later
MoCap [ 125 ]
Free for research; commercial use allowed in products, but resale (even converted) is prohibited
Appendix
Table 6: Licences of datasets used in our experiments.
Figure 7: Multivariate adaptations for univariate methods (FLUSS, Prophet, ChangeFinder, BOCD): L2 Norm Reduction collapses the multivariate signal into a univariate L2 norm series; Ensemble Aggregation runs the detector independently per channel and merges the nearby change points of the channels (the K best-supported ones when K is given; shown with K=3 ).
Figure 8: Evaluation measures in TSB-SEG .
Method
#cfg
Parameter
Default
Non-Guided values
Guided values
CPD, distribution-based
AMOC
3/3
min_size
5
{5, 15, 30}
same
BinSeg
16/4
model
ℓ2
{ ℓ1 , ℓ2 , RBF, cosine}
same
penalty
8
{1, 2, 8, 32}
-
n_cps
-
-
nCP
BottomUp
16/4
model
ℓ2
{ ℓ1 , ℓ2 , RBF, cosine}
same
Appendix
Table 7: Hyperparameter grids, as evaluated. For each method, the parameters swept in Parameter-search mode, their value in Default mode and the values tried in each mode; #cfg is the number of configurations per dataset (Non-Guided/Guided). In Guided mode the parameter marked with nCP , nCP+1 (number of segments) or nS receives the ground-truth number of events of each series, and “-” marks a Non-Guided stopping rule that it replaces; “same”: the Non-Guided values. The penalties of PELT, BinSeg, BottomUp and Window are multiples X of the BIC penalty ulogn (Appendix G ). Time2State and E2USD skip the window 64/step 100 combinations (step larger than the window). PELT, AMOC, iCID, EAgglo and tGLAD take no K : their Guided grid is their Non-Guided one (PELT and AMOC are not rerun). ClaSP distances: euclidean, z-normalised euclidean and complexity-invariant (CID) euclidean; window-size selection: SuSS, FFT or ACF; L2 norm and ensembling: the multivariate adaptations of Appendix E . tGLAD’s default runs used 100 epochs, 2000 on ActRecTut, MoCap and SKAB. #cfg “-”: default configuration only (BOCD, EAgglo, TS-CP 2 : main parameters shown); n/a : mode not evaluated.
Default, NG
Default, G
Tuned, NG
Tuned, G
Algorithm
Dataset
Series
Compl.
BiC
t~ [s]
Compl.
BiC
t~ [s]
Compl.
BiC
t~ [s]
Compl.
BiC
t~ [s]
tGLAD
actrectut
2
2/2
0.04
36706
2/2
0.04
33986
2/2
0.49
9885
2/2
0.46
9736
has
250
206/250
0.46
288
171/250
0.45
343
40/250
0.53
1462
43/250
0.53
1377
mocap
9
9/9
0.05
7304
9/9
0.06
5256
9/9
0.46
1841
9/9
0.49
1795
pump
120
57/120
0.07
1367
55/120
0.06
1575
13/120
0.52
6820
12/120
0.51
6800
skab
34
34/34
0.10
1326
34/34
0.10
1301
34/34
0.57
309
34/34
0.60
329
Appendix
Table 8: Per-dataset coverage and score of the five extended-scope algorithms with a CPD output. Compl. = completed series / dataset cardinality. Default columns report the mean Bidirectional Covering (BiC) under the author-recommended configuration; Tuned columns report the mean BiC of the configuration maximising the dataset-level mean over the parameter sweep. t~ is the median wall-clock per finished trial (in seconds). NG = Non-Guided, G = Guided. “-” = (mode, dataset) pair not evaluated.
Default, NG
Default, G
Tuned, NG
Tuned, G
Algorithm
Dataset
Series
Compl.
SMS
t~ [s]
Compl.
SMS
t~ [s]
Compl.
SMS
t~ [s]
Compl.
SMS
t~ [s]
Hidalgo
actrectut
2
-
-
-
2/2
0.25
10846
-
-
-
2/2
0.32
16506
has
250
-
-
-
8/250
0.39
757
-
-
-
8/250
0.39
776
mocap
9
-
-
-
9/9
0.25
1855
-
-
-
9/9
0.32
2746
pump
120
-
-
-
2/120
0.17
41896
-
-
-
1/120
0.17
72877
skab
34
-
-
-
34/34
0.59
151
-
-
-
34/34
0.59
159
Appendix
Table 9: Per-dataset coverage and score of the SD-side extended-scope algorithm (Hidalgo). Same column convention as Table 8 , with the SD primary metric (State Matching Score, SMS) replacing BiC. Hidalgo is multivariate-only by design, hence the absent (TSSB, UTSA) cells, and intrinsically Guided so the Non-Guided columns are uniformly empty.
Figure 9: Pairwise Pearson correlation between evaluation metrics over all grid runs ( Non-Guided and Guided ) of every algorithm on the 8 main datasets ( 438,176 runs with all CPD metrics, 105,316 with all SD metrics). Left: CPD metrics; right: SD metrics.
Figure 10: Per-domain spider plots of the six best algorithms (tuned score) for CPD and SD, in both supervision modes. Within a task, a method keeps its colour and marker across the Non-Guided and Guided panels.
Figure 11: Per-dataset spider plots of the six best algorithms (tuned score) for CPD and SD, in both supervision modes. Same colours and markers as Figure 10 .
Figure 12: Per-dataset distribution of the per-series tuned BiC of the six best CPD methods of each mode, ordered by their best mean score over the two modes; number of series in parentheses. Light left half: Non-Guided ; dark right half: Guided ; a full-width box is drawn for a method run in one mode only (DynP) or that cannot use nCP (PELT). Boxes span the interquartile range, with the median as a line; whiskers extend to 1.5 times the interquartile range, and dots mark the series beyond.
Figure 13: Per-dataset distribution of the per-series tuned SMS of the SD methods, with the conventions of Figure 12 ; TICC runs in Guided mode only, and AutoPlait cannot use nS .
Task
Mode
#Methods
Single best
Score
Strict best on
Oracle, top 3
Oracle, all
CPD
Non-Guided
17
ClaSP
0.696
18%
0.805
0.858
CPD
Guided
20
DynP
0.778
4%
0.793
0.884
SD
Non-Guided
5
CLaP
0.633
40%
0.732
0.772
SD
Guided
6
CLaP
0.714
37%
0.796
0.821
Appendix
Table 10: Single best method against a per-series oracle (mean tuned BiC for CPD, SMS for SD, over the 592 series). Single best : highest mean tuned score; strict best on : share of the series on which it beats every other method (ties are frequent among the penalised detectors given nCP ); oracle : mean over the series of the best score among the three best methods or among all methods of the mode.
Figure 14: Critical-difference diagrams of the Default configurations over the 592 series, with the test of Figure 3 (two-sided Wilcoxon signed-rank test with Holm correction, α=0.1 ).
Pooled
Non-Guided
Guided
Runs
97,076
46,825
50,797
Pearson correlation of BiC and SMS
0.79
0.79
0.76
Runs with ∣r∣≤0.05
33.8%
33.2%
34.6%
95% envelope of r
[−0.27,+0.36]
[−0.26,+0.31]
[−0.28,+0.40]
SMS<0.6 among runs with BiC≥0.8
3.7%
2.7%
4.0%
Lower tail ( r<−0.27 ): Delay / Missing share
36% / 40%
39% / 37%
36% / 40%
Appendix
Table 11: RQ2 statistics in each supervision mode (SD-capable methods, 8 main datasets, all default and grid runs). The Guided runs of AutoPlait are its Non-Guided ones ( 546 runs), counted once in the pooled column.
Figure 15: Per-series runtime distribution on the CPD task (tuned configuration on BiC, 8 main datasets), for the CPD methods (blue) and the SD methods scored on CPD (orange), sorted by median runtime. The light left half of each box reports the Non-Guided runtimes, the dark right half the Guided ones; a method run in one mode only is drawn as a full-width box. Boxes span the interquartile range, whiskers 1.5 times it.
Figure 16: Runtime profile of SD methods on TSB-SEG . (a, b) Mean tuned SMS against the median runtime of the tuned configuration, as in Figure 6 ; (c) per-series runtime of the tuned configuration, with the conventions of Figure 15 .
Figure 17: Per-algorithm hyperparameter influence on CPD performance, Non-Guided mode. Score: BiC (top left). For each algorithm, the most influential parameter (largest cross-value spread of marginal means) is used as the x-axis; colour and marker shape encode up to two additional parameters (legend below each panel). Parameter and value names are abbreviated for readability (e.g. win = window_size , cost = segment cost, pen = multiple X of the BIC penalty, thr = threshold, cinv = cinvariant_euclidean_distance ); the full mapping is given in the source code released with the benchmark.
Figure 18: Per-algorithm hyperparameter influence on CPD performance, Guided mode. Same conventions as Figure 17 ; PELT, which cannot use nCP , is shown there only. Compared to the Non-Guided figure, knowing nCP generally lifts the response surface and shrinks its vertical spread; TICC, run in this mode only, remains strongly responsive to its grid.
Figure 19: Per-algorithm hyperparameter influence on SD performance, Non-Guided mode. Score: SMS (top left). Same conventions as Figure 17 .
Figure 20: Per-algorithm hyperparameter influence on SD performance, Guided mode. Same conventions as Figure 19 .
Figure 21: Sensitivity of the tuned score to input series features (Spearman ρ ). Top: CPD score (BiC); bottom: SD score (SMS). One row per algorithm, the four features in each supervision mode as columns. Red: the score decreases as the feature grows ( ρ<0 ); blue: it increases; grey: no monotonic dependence.
Figure 22: Analysis on the PAMAP2 dataset (up to 447 k samples per series) , for the 17 methods that process its 9 series. Top: tuned score of each series, (a) BiC for CPD, (b) SMS for SD; bottom: runtime of the tuned configuration on each series. Same conventions as Figure 15 ; methods sorted by median tuned score.
Non-Guided
Guided
Algo
Dataset
Def.
Worst
Med.
Tuned
t~ [s]
Def.
Worst
Med.
Tuned
t~ [s]
AMOC
ActRecTut
0.24
0.24
0.24
0.24
0.2
0.24
0.24
0.24
0.24
0.2
HAS
0.61
0.61
0.61
0.61
0.2
0.61
0.61
0.61
0.61
0.2
MoCap
0.30
0.30
0.30
0.30
0.2
0.30
0.30
0.30
0.30
0.2
PUMP
0.07
0.07
0.07
0.07
0.2
0.07
0.07
0.07
0.07
0.2
SKAB
0.71
0.71
0.72
0.72
0.3
0.71
0.71
0.72
0.72
0.3
Appendix
Table 12: Main-pool algorithms on CPD: per-(algorithm, dataset) default / worst-case / median / tuned BiC, in both Non-Guided (NG) and Guided (G) modes. The SD methods (last block) are scored on the change points induced by their labels, except CLaP, which labels the segments of ClaSP and is not scored separately. t~ is the median wall-clock of the tuned configuration in seconds. “-” = (mode, dataset) pair not evaluated for that algorithm (e.g. intrinsically Guided methods have no NG entries).
Non-Guided
Guided
Algo
Dataset
Def.
Worst
Med.
Tuned
t~ [s]
Def.
Worst
Med.
Tuned
t~ [s]
AutoPlait
ActRecTut
0.31
0.31
0.31
0.31
47.4
0.31
0.31
0.31
0.31
47.4
HAS
0.65
0.65
0.65
0.65
10.4
0.65
0.65
0.65
0.65
10.4
MoCap
0.86
0.86
0.86
0.86
25.0
0.86
0.86
0.86
0.86
25.0
PUMP
0.14
0.14
0.14
0.14
316
0.14
0.14
0.14
0.14
316
SKAB
0.73
0.73
0.73
0.73
1.5
0.73
0.73
0.73
0.73
1.5
Appendix
Table 13: Main-pool SD algorithms: per-(algorithm, dataset) default / worst-case / median / tuned SMS, in both Non-Guided (NG) and Guided (G) modes. Conventions identical to Table 12 .
Non-Guided
Guided
Algo
Dataset
Def.
Worst
Med.
Tuned
t~ [s]
Def.
Worst
Med.
Tuned
t~ [s]
AMOC
ActRecTut
0.05
0.05
0.05
0.05
0.2
0.05
0.05
0.05
0.05
0.2
HAS
0.19
0.19
0.19
0.19
0.2
0.19
0.19
0.19
0.19
0.2
MoCap
0.17
0.17
0.17
0.17
0.2
0.17
0.17
0.17
0.17
0.2
PUMP
0.03
0.03
0.03
0.03
0.2
0.03
0.03
0.03
0.03
0.2
SKAB
0.12
0.12
0.12
0.12
0.3
0.12
0.12
0.12
0.12
0.3
Appendix
Table 14: Main-pool algorithms on CPD: per-(algorithm, dataset) default / worst-case / median / tuned Gaussian F1 , in both Non-Guided (NG) and Guided (G) modes. Same conventions as Table 12 , with BiC replaced by Gaussian F1 . The tuned configuration in this table is selected on Gaussian F1 itself (not on BiC), so each table is self-contained.
Non-Guided
Guided
Algo
Dataset
Def.
Worst
Med.
Tuned
t~ [s]
Def.
Worst
Med.
Tuned
t~ [s]
AutoPlait
ActRecTut
0.19
0.19
0.19
0.19
47.4
0.19
0.19
0.19
0.19
47.4
HAS
0.47
0.47
0.47
0.47
10.4
0.47
0.47
0.47
0.47
10.4
MoCap
0.80
0.80
0.80
0.80
25.0
0.80
0.80
0.80
0.80
25.0
PUMP
0.24
0.24
0.24
0.24
316
0.24
0.24
0.24
0.24
316
SKAB
0.38
0.38
0.38
0.38
1.5
0.38
0.38
0.38
0.38
1.5
Appendix
Table 15: Main-pool SD algorithms: per-(algorithm, dataset) default / worst-case / median / tuned ARI, in both Non-Guided (NG) and Guided (G) modes. Same conventions as Table 13 , with SMS replaced by ARI. The tuned configuration in this table is selected on ARI itself (not on SMS), so each table is self-contained.
Non-Guided
Guided
Algo
Dataset
Def.
Worst
Med.
Tuned
t~ [s]
Def.
Worst
Med.
Tuned
t~ [s]
AutoPlait
ActRecTut
0.26
0.26
0.26
0.26
47.4
0.26
0.26
0.26
0.26
47.4
HAS
0.51
0.51
0.51
0.51
10.4
0.51
0.51
0.51
0.51
10.4
MoCap
0.89
0.89
0.89
0.89
25.0
0.89
0.89
0.89
0.89
25.0
PUMP
0.36
0.36
0.36
0.36
316
0.36
0.36
0.36
0.36
316
SKAB
0.57
0.57
0.57
0.57
1.5
0.57
0.57
0.57
0.57
1.5
Appendix
Table 16: Main-pool SD algorithms: per-(algorithm, dataset) default / worst-case / median / tuned WARI, in both Non-Guided (NG) and Guided (G) modes. Same conventions as Table 13 , with SMS replaced by WARI. The tuned configuration in this table is selected on WARI itself (not on SMS), so each table is self-contained.
Default, NG
Default, G
Tuned, NG
Tuned, G
Algorithm
Dataset
Series
Compl.
gF1
t~ [s]
Compl.
gF1
t~ [s]
Compl.
gF1
t~ [s]
Compl.
gF1
t~ [s]
tGLAD
actrectut
2
2/2
0.05
36706
2/2
0.05
33986
2/2
0.36
9885
2/2
0.39
8928
has
250
206/250
0.11
288
171/250
0.10
343
6/250
0.17
35.2
171/250
0.14
323
mocap
9
9/9
0.05
7304
9/9
0.05
5256
9/9
0.18
1752
9/9
0.19
1735
pump
120
57/120
0.03
1367
55/120
0.02
1575
10/120
0.49
7741
12/120
0.50
6669
skab
34
34/34
0.06
1326
34/34
0.06
1301
34/34
0.09
309
34/34
0.11
317
Appendix
Table 17: Per-dataset coverage and score (Gaussian F1 ) of the five extended-scope CPD algorithms. Tuned configurations are selected on Gaussian F1 .
Default, NG
Default, G
Tuned, NG
Tuned, G
Algorithm
Dataset
Series
Compl.
ARI
t~ [s]
Compl.
ARI
t~ [s]
Compl.
ARI
t~ [s]
Compl.
ARI
t~ [s]
Hidalgo
actrectut
2
-
-
-
2/2
0.07
10846
-
-
-
2/2
0.07
10487
has
250
-
-
-
8/250
0.15
757
-
-
-
8/250
0.15
731
mocap
9
-
-
-
9/9
0.05
1855
-
-
-
9/9
0.05
1716
pump
120
-
-
-
2/120
0.06
41896
-
-
-
1/120
0.06
72877
skab
34
-
-
-
34/34
0.13
151
-
-
-
34/34
0.14
221
Appendix
Table 18: Per-dataset coverage and score (ARI) of the SD-side extended-scope algorithm (Hidalgo). Tuned configurations are selected on ARI.
Default, NG
Default, G
Tuned, NG
Tuned, G
Algorithm
Dataset
Series
Compl.
WARI
t~ [s]
Compl.
WARI
t~ [s]
Compl.
WARI
t~ [s]
Compl.
WARI
t~ [s]
Hidalgo
actrectut
2
-
-
-
2/2
0.06
10846
-
-
-
2/2
0.06
10487
has
250
-
-
-
8/250
0.15
757
-
-
-
8/250
0.15
731
mocap
9
-
-
-
9/9
0.06
1855
-
-
-
9/9
0.06
1716
pump
120
-
-
-
2/120
0.05
41896
-
-
-
2/120
0.05
42438
skab
34
-
-
-
34/34
0.17
151
-
-
-
34/34
0.21
221
Appendix
Table 19: Per-dataset coverage and score (WARI) of the SD-side extended-scope algorithm (Hidalgo). Tuned configurations are selected on WARI.