costream: A Streaming Evaluation Protocol for Wearable Fall Detection
Authors: Timilehin B. Aderinola, Ilaria D'Ascanio, Luca Palmerini, Lorenzo Chiari, Jochen Klenk, Clemens Becker, Brian Caulfield, Georgiana Ifrim
Organizations: the Insight RI Research Centre for Data Analytics, University College Dublin, Dublin, Ireland · the Department of Electrical, Electronic and Information Engineering “Guglielmo Marconi”, University of Bologna, Bologna, Italy · the Department of Clinical Gerontology, Robert Bosch Hospital, Stuttgart, Germany · the Digital Geriatrics Unit, Heidelberg University Hospital, Heidelberg, Germany
Wearable fall detectors are commonly evaluated on presegmented windows, where a fall window is extracted at an annotated impact and non-fall windows are sampled from fall-free signal. However, a deployed detector processes a continuous sensor stream in which the location of a fall is unknown. We specify a streaming evaluation protocol in which a detector traverses an unsegmented recording, aggregates window-level scores into a temporal confidence map, and emits timestamped events that are evaluated against the annotated fall. Holding the dataset, participant-wise splits, window construction and fitted models fixed, we compare presegmented and streaming evaluation on clinically verified real-world falls. The two protocols identify the same best-performing method overall, but select different methods in 6 of 20 model-window configurations, with the divergence associated with differences in the temporal clustering of false alarms. Streaming evaluation also exposes deployment-relevant quantities, including false alarm rate over elapsed monitoring time, detection delay, and the effect of the decision threshold, that are not captured by per-window evaluation. These results show that evaluation protocol can materially affect conclusions about wearable fall detectors and should be considered when assessing their suitability for continuous monitoring.
Figures & tables
Figure 1: A multiphase fall from FARSEEING, showing the falling, impact, and post-fall phases. The post-fall phase covers the resting period during which the faller remains on the ground, followed by recovery.
Figure 2: Overlapping windows are scored independently and aggregated by pointwise maximum into a temporal confidence map P(t) . Thresholding at τ yields detection episodes, each emitting one detection. A detection overlapping the reference interval R counts as a true positive; one with no overlap is a false positive.
Model
Family
Implementation
Catch22 [ 8 ]
Feature-based
aeon
ExtraTrees [ 12 ]
Tree-based
scikit-learn
MiniRocket [ 4 ]
Convolutional
aeon
QUANT [ 5 ]
Interval-based
aeon
ResNet [ 20 ]
Deep CNN
aeon
Table 1: Classifiers evaluated.
Protocol A (per-window)
Protocol B (event-level)
Model
Mean F1
SD
Rank
Mean F1
SD
Rank
QUANT
0.822
0.087
1
0.811
0.074
1
ExtraTrees
0.683
0.097
2
0.668
0.095
3
MiniRocket
0.682
0.186
3
0.739
0.144
2
Catch22
0.541
0.225
4
0.563
0.211
5
ResNet
0.527
0.216
5
0.638
0.160
4
Table 2: Mean F1 per model under each protocol over the 20 model-window configurations, with the implied rank. The two columns are not commensurable; they are placed side by side to compare orderings, not values.
Model
Collapse ratio
95% CI
Gap (B − A)
Wilcoxon p
QUANT
1.09
[1.00, 1.24]
−0.011
0.152
Catch22
1.33
[1.05, 1.46]
+0.022
0.075
ExtraTrees
1.38
[1.00, 1.69]
−0.015
0.021
MiniRocket
2.86
[2.05, 3.85]
+0.057
0.048
ResNet
6.07
[2.50, 9.94]
+0.111
0.002
Table 3: False alarm clustering and the protocol gap. The collapse ratio is window-level false firings per emitted false alarm episode, with a 95% participant-cluster bootstrap interval over 4000 resamples. The ratio has a floor of 1.0 by construction. The gap is mean F1 under Protocol B minus Protocol A, reported to show how it varies across models rather than as a difference in performance.
Figure 3: Recall as a function of decision threshold (left) and resulting false alarm rate (right), averaged across the four window-size configurations within each model. The right-hand panel has no equivalent under presegmented evaluation because the sampled window pool has no elapsed-time denominator. MiniRocket is shown as a single operating point because its default estimator provides hard decisions rather than probabilities.
Model
c
τ
Precision
Recall
F1
FA/h
Catch22
default
0.50
0.782
0.456
0.576
0.38
1
0.40
0.718
0.595
0.650
0.70
2
0.33
0.657
0.675
0.666
1.06
5
0.20
0.487
0.782
0.600
2.47
10
0.14
0.423
0.881
0.572
3.60
ExtraTrees
default
0.50
0.847
0.536
0.657
0.29
Table 4: Threshold selected by cost-sensitive optimisation and the resulting operating point. c is the assumed cost of a missed fall relative to a false alarm; τ is the mean threshold selected across folds. FA/h is false alarm episodes per hour of recording. MiniRocket is excluded: its scores are binary, so the gain function is constant in τ and cost-based selection cannot act.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Model
w∗
Protocol
TP
FP
FN
Precision
Recall
F1
MiniRocket
10
A
17
2
4
0.895
0.810
0.850
B
17
2
4
0.895
0.810
0.850
ExtraTrees
3
A
15
0
6
1.000
0.714
0.833
B
14
0
7
1.000
0.667
0.800
QUANT
10
A
15
2
6
0.882
0.714
0.789
B
16
2
5
0.889
0.762
0.821
Appendix
Table A1: Held-out cohort, 9 participants and 21 recordings, each model at its own w∗ and τ=0.5 . Counts are given because the cohort contains 21 falls, so a single detection changes recall by approximately 0.05 .
Model
w∗
τ
TP
FP
FN
Precision
Recall
F1
QUANT
10
0.36
19
4
2
0.826
0.905
0.864
ExtraTrees
3
0.29
16
4
5
0.800
0.762
0.780
Catch22
10
0.33
17
7
4
0.708
0.810
0.756
ResNet
3
0.32
13
11
8
0.542
0.619
0.578
Appendix
Table A2: Protocol B on the held-out cohort at the development-selected threshold for c=2 , applied unchanged. MiniRocket is excluded: its scores are binary, so the gain function is constant in τ .
School of Computer Science, University College Dublin (UCD), Ireland · Department of Electrical, Electronic and Information Engineering “Guglielmo Marconi”, University of Bologna, Italy · Department of Clinical Gerontology, Robert Bosch Hospital, Stuttgart, Germany +2