OneShot: Label-Efficient Detector Weighting for Time Series Anomaly Detection
Organizations: LG AI Research
Abstract
Time series (TS) anomaly detection (AD) lacks a universal detector, with the best choice determined by the series rather than by the algorithm. Existing methods therefore either select one detector for each series or average a pool of them. However, both assume that no label ever arrives, while a deployed monitor does receive them, since an operator confirms the alarms it raises. In this paper, we propose ONESHOT, which uses a single confirmed anomaly to weight the pool rather than to choose one detector from it. Specifically, we score each detector on the confirmed anomaly, and those scores become the weights of a soft mixture over the whole pool. We show that a single confirmed anomaly is enough to come close to the best detector for the series. Across 152 univariate and 73 multivariate series, ONESHOT outperforms the strongest single detector, averaging the whole pool, and even picking the best detector on the confirmed event. Code is available at: https://github.com/seunghan96/one_shot.
Figures & tables
| Family | What it needs | What it returns | |||
|---|---|---|---|---|---|
| Frozen | No corpus | Label | Mixture | Threshold | |
| Single detector (w/o labels) | |||||
| One fixed detector ( Liu & Paparrizos, 2024 ) | ✓ | ✓ | ✗ | ✗ | ✗ |
| Internal measures ( Ma et al., 2023 ) | ✓ | ✓ | ✗ | ✗ | ✗ |
| Local selection ( Zhao et al., 2019 ) | ✓ | ✓ | ✗ | ✗ | ✗ |
| Meta-learned selection ( Zhao et al., 2021 ) | ✓ | ✗ | ✗ | ✗ | ✗ |
| Method | Wins | AUC-PR | AUC-ROC | VUS-PR | VUS-ROC | |
|---|---|---|---|---|---|---|
| Sub_KNN ( Ramaswamy et al., 2000 ) | 17.1 | 0.521 | 0.854 | 0.623 | 0.875 | 60.5% |
| PCA ( Shyu et al., 2003 ) | 23.7 | 0.494 | 0.786 | 0.537 | 0.815 | 38.9% |
| Sub_PCA ( Shyu et al., 2003 ) | 8.6 | 0.504 | 0.786 | 0.531 | 0.819 | 37.2% |
| IForest ( Liu et al., 2008 ) | 9.2 | 0.468 | 0.785 | 0.501 | 0.829 | 29.8% |
| Sub_IForest ( Liu et al., 2008 ) | 0.0 | 0.460 | 0.787 | 0.491 | 0.833 | 27.1% |
| POLY ( Liu & Paparrizos, 2024 ) | 5.3 | 0.406 | 0.762 | 0.482 | 0.797 | 24.8% |
| Event | ||
|---|---|---|
| Label | Confirmed window | covered |
| Exact | 100% | |
| Half of it | 50% | |
| Quarter of it | 25% | |
| Shifted by half | 50% | |
| Shifted by its length | 0% |
| Confirmed event | VUS-PR | Retention ∗ | |||||
|---|---|---|---|---|---|---|---|
| Best on window | OneShot | Imp (%) | Best on window | OneShot | Imp (%) | ||
| Exact | 0.568 | 0.699 | +23.0 | 100% | 100% | +0.0 | 0.001 |
| Half of it | 0.519 | 0.692 | +33.3 | 91% | 99% | +8.4 | 0.001 |
| Quarter of it | 0.476 | 0.672 | +41.0 | 84% | 96% | +14.7 | 0.001 |
| Shifted by half | 0.543 | 0.668 | +22.9 | 96% | 96% | +0.0 | 0.001 |
| Shifted by its length | 0.419 | 0.554 | +32.1 | 74% | 79% | +7.5 | 0.001 |
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
| Choice of | Values picked | VUS-PR | |
|---|---|---|---|
| Chosen on the other series | |||
| 2-fold | 0.01, 0.035 | 0.701 | 80.5% |
| 5-fold | 0.01, 0.02, 0.035 | 0.702 | 80.6% |
| 10-fold | 0.01, 0.02 | 0.707 | 81.9% |
| Leave-one-out (Ours) | 0.02 | 0.710 | 82.8% |
| Fixed in advance | |||
| Block | Pool | Detectors added |
|---|---|---|
| Statistical | 18 | CBLOF ( He et al., 2003 ) , COPOD ( Li et al., 2020 ) , EIF ( Hariri et al., 2021 ) , FFT ( Liu & Paparrizos, 2024 ) , HBOS ( Goldstein & Dengel, 2012 ) , IForest ( Liu et al., 2008 ) , KMeansAD_U ( Liu & Paparrizos, 2024 ) , KNN ( Ramaswamy et al., 2000 ) , LOF ( Breunig et al., 2000 ) , MatrixProfile ( Yeh et al., 2016 ) , PCA ( Shyu et al., 2003 ) , POLY ( Liu & Paparrizos, 2024 ) , SR ( Ren et al., 2019 ) , Sub_HBOS ( Liu & Paparrizos, 2024 ) , Sub_IForest ( Liu & Paparrizos, 2024 ) , Sub_KNN ( Liu & Paparrizos, 2024 ) , Sub_LOF ( Liu & Paparrizos, 2024 ) , Sub_PCA ( Liu & Paparrizos, 2024 ) |
| Deep and foundation (DL) | 29 | AnomalyTransformer ( Xu et al., 2022 ) , CNN ( Liu & Paparrizos, 2024 ) , FITS ( Xu et al., 2024 ) , LSTMAD ( Malhotra et al., 2015 ) , MOMENT_ZS ( Goswami et al., 2024 ) , OFA ( Zhou et al., 2023 ) , OmniAnomaly ( Su et al., 2019 ) , PatchTST ( Nie et al., 2023 ) , TimesNet ( Wu et al., 2023 ) , TranAD ( Tuli et al., 2022 ) , USAD ( Audibert et al., 2020 ) |
| Window lengths | 77 | HSF ( Liu & Paparrizos, 2024 ) (9 window lengths), HSF_Causal ( Liu & Paparrizos, 2024 ) (9 window lengths), HSF_U ( Liu & Paparrizos, 2024 ) (9 window lengths), PaAno_PAI ( Liu & Paparrizos, 2024 ) (9 window lengths), Left_STAMPi ( Liu & Paparrizos, 2024 ) , M2N2 ( Liu & Paparrizos, 2024 ) , MCD ( Rousseeuw & Driessen, 1999 ) , MMPAD ( Liu & Paparrizos, 2024 ) , RobustPCA ( Candès et al., 2011 ) , StreamVAE ( Liu & Paparrizos, 2024 ) , Sub_MCD ( Liu & Paparrizos, 2024 ) , Time_RCD ( Liu & Paparrizos, 2024 ) |
| Block | Pool | Detectors added |
|---|---|---|
| Statistical | 11 | CBLOF ( He et al., 2003 ) , COPOD ( Li et al., 2020 ) , EIF ( Hariri et al., 2021 ) , HBOS ( Goldstein & Dengel, 2012 ) , IForest ( Liu et al., 2008 ) , KMeansAD ( Liu & Paparrizos, 2024 ) , KNN ( Ramaswamy et al., 2000 ) , LOF ( Breunig et al., 2000 ) , OCSVM ( Schölkopf et al., 2001 ) , PCA ( Shyu et al., 2003 ) , RobustPCA ( Candès et al., 2011 ) |
| Deep and foundation (DL) | 22 | AnomalyTransformer ( Xu et al., 2022 ) , CNN ( Liu & Paparrizos, 2024 ) , FITS ( Xu et al., 2024 ) , LSTMAD ( Malhotra et al., 2015 ) , MOMENT_ZS ( Goswami et al., 2024 ) , OFA ( Zhou et al., 2023 ) , OmniAnomaly ( Su et al., 2019 ) , PatchTST ( Nie et al., 2023 ) , TimesNet ( Wu et al., 2023 ) , TranAD ( Tuli et al., 2022 ) , USAD ( Audibert et al., 2020 ) |
| Window lengths | 31 | PaAno_PAI ( Liu & Paparrizos, 2024 ) (3 window lengths), Left_STAMPi ( Liu & Paparrizos, 2024 ) , M2N2 ( Liu & Paparrizos, 2024 ) , MCD ( Rousseeuw & Driessen, 1999 ) , MMPAD ( Liu & Paparrizos, 2024 ) , StreamVAE ( Liu & Paparrizos, 2024 ) |
| Rank | |||||
|---|---|---|---|---|---|
| Detector | Wins (%) | Mean VUS-PR | Median | Best | Worst |
| PCA | 23.7 | 0.537 | 4 | 1 | 13 |
| Sub_KNN | 17.1 | 0.623 | 3 | 1 | 12 |
| Sub_LOF | 13.2 | 0.352 | 8 | 1 | 13 |
| IForest | 9.2 | 0.501 | 6 | 1 | 13 |
| Sub_PCA | 8.6 | 0.531 | 5 | 1 | 13 |
| Method | Median | Series won | |
|---|---|---|---|
| No labels | |||
| Random detector | +0.334 | 95.4% | 0.001 |
| Score average | +0.076 | 84.9% | 0.001 |
| Rank average | +0.258 | 91.4% | 0.001 |
| One confirmed event | |||
| Best on labeled window | +0.009 | 53.9% | 0.001 |
| Confirmed events | Series | Median |
| 1 | 153 | 6.1 |
| 2 | 117 | 6.5 |
| 3 | 72 | 6.3 |
| 4 | 63 | 6.4 |
| 5 | 51 | 6.9 |
| 6 | 48 | 7.0 |
| Rule | Series it is best on | Mean VUS-PR |
|---|---|---|
| Top-2 average | 36.2% | 0.702 |
| Top-3 average | 21.7% | 0.704 |
| Top-5 average | 23.7% | 0.703 |
| Top-10 average | 18.4% | 0.649 |
| OneShot | – | 0.710 |
| chosen for each series | – | 0.747 |
| Source | Threshold rule | F1 | Event recall |
|---|---|---|---|
| Practice | Top 1% of points | 0.243 | 0.598 |
| Top 5% of points | 0.385 | 0.763 | |
| Estimated contamination | 0.140 | 0.961 | |
| One confirmed event | Highest recall at a precision floor | 0.450 | 0.794 |
| Highest F1 on the window | 0.454 | 0.790 | |
| Oracle | Highest F1 on the hidden part | 0.546 | 0.830 |
| Pool | Series | Detectors | Score average | Best on window | OneShot |
|---|---|---|---|---|---|
| Statistical | 73 | 9 | 0.272 | 0.368 | 0.382 |
| + DL | 73 | 19 | 0.453 | 0.489 | 0.503 |
| + Window lengths | 73 | 28 | 0.482 | 0.511 | 0.539 |
| Confirmed event | Best | OneShot | Imp (%) |
|---|---|---|---|
| Exact | 0.433 | 0.459 | +5.9 |
| Half of it | 0.417 | 0.449 | +7.5 |
| Quarter of it | 0.392 | 0.437 | +11.6 |
| Shifted by half | 0.395 | 0.420 | +6.3 |
| Shifted by its length | 0.317 | 0.345 | +9.0 |
| Normal window | 0.296 | 0.326 | +10.2 |
| Threshold | F1 | Events caught |
|---|---|---|
| Top 1% of points | 0.194 | 59.1% |
| Top 5% of points | 0.324 | 77.9% |
| From the confirmed window | 0.360 | 87.3% |
| Best on the hidden part | 0.448 | 88.5% |
| Agreement on the window | Series | Best on window | OneShot |
|---|---|---|---|
| Below the median | 36 | 0.225 | 0.261 |
| Above the median | 37 | 0.507 | 0.510 |
| Method | AUC-PR | AUC-ROC | VUS-PR | VUS-ROC | ||
|---|---|---|---|---|---|---|
| No labels | ||||||
| Random detector | 0.211 | 0.680 | 0.211 | 0.712 | 0.0% | 0.001 |
| Score average | 0.262 | 0.783 | 0.272 | 0.812 | 25.0% | 0.001 |
| Rank average | 0.217 | 0.749 | 0.220 | 0.779 | 3.6% | 0.001 |
| One confirmed event | ||||||
| Best on labeled window | 0.356 | 0.804 | 0.368 | 0.832 | 64.7% | 0.209 |
| Block | Share of the pool | Share of the weight | Ratio |
|---|---|---|---|
| Statistical | 37.7% | 24.1% | 0.64 |
| Deep and foundation | 37.2% | 42.7% | 1.15 |
| Window-length variants | 25.1% | 33.2% | 1.32 |
| Where is drawn from | Series | Best on window | OneShot | |
|---|---|---|---|---|
| The whole series (ours) | 150 | 0.675 | 0.712 | 84.3% |
| The normal points (needs labels) | 150 | 0.653 | 0.710 | 83.6% |
| Difference | -0.022 | -0.003 | -0.7% | |
| vs. the first row | 0.030 | 0.432 |
| Size of | Anomalous in it | Best on window | OneShot | vs. | |
|---|---|---|---|---|---|
| (ours) | 6.38% | 0.667 | 0.713 | 83.5% | – |
| 6.41% | 0.684 | 0.716 | 84.4% | 0.233 | |
| 6.42% | 0.707 | 0.729 | 87.7% | 0.001 |
| Method | Drawn from the whole series | Drawn from the past | |
|---|---|---|---|
| Best on labeled window | 0.672 | 0.656 | -0.016 |
| OneShot | 0.710 | 0.704 | -0.006 |
| Oracle (Best detector for each) | 0.779 | 0.779 | +0.000 |
| Confirmed events | Series | Score average | Best on window | - Shot | ||
| Every series that carries | ||||||
| 1 | 154 | 0.599 | 0.674 | 0.711 | 82.5% | 0.001 |
| 2 | 122 | 0.584 | 0.667 | 0.700 | 85.9% | 0.001 |
| 3 | 75 | 0.476 | 0.568 | 0.620 | 86.2% | 0.001 |
| 4 | 67 | 0.499 | 0.582 | 0.622 | 82.6% | 0.001 |
| 5 | 54 | 0.520 | 0.590 | 0.647 | 88.9% | 0.001 |
| Confirmed events | Series | Best on window | - Shot | vs. one event | |
|---|---|---|---|---|---|
| Series that carry 2 events | |||||
| 1 | 105 | 0.715 | 0.781 | 86.4% | – |
| 2 | 105 | 0.727 | 0.789 | 88.4% | 0.615 |
| Series that carry 3 events | |||||
| 1 | 57 | 0.620 | 0.709 | 80.4% | – |
| 2 | 57 | 0.633 | 0.735 | 86.9% | 0.001 |
| Quarter | Anomaly rate | OneShot | Best on window | Gain |
|---|---|---|---|---|
| 1Q (Sparse) | 0.53% | 0.536 | 0.419 | +0.116 |
| 2Q | 3.69% | 0.779 | 0.740 | +0.040 |
| 3Q | 7.80% | 0.692 | 0.676 | +0.016 |
| 4Q (Dense) | 12.62% | 0.835 | 0.853 | -0.018 |
| Concern | Where it is answered |
|---|---|
| The method | |
| A softmax over accuracies is a standard idea | Appendix N , Propositions 1 and 2 |
| The margin over the top- averages is small | Table 3 , Table 20 |
| The temperature needs labeled series | Appendix A.2 |
| The label | |
| A real operator confirms imperfectly | Table 9 , Table 23 |