Did We Actually Fix It? An Independent Adversarial Stress-Test of Post-Point-Adjustment Evaluation Metrics for Time-Series Anomaly Detection

Zongye Lyu

stat.MLJul 12, 2026

Did We Actually Fix It? An Independent Adversarial Stress-Test of Post-Point-Adjustment Evaluation Metrics for Time-Series Anomaly Detection

Authors: Zongye Lyu

Organizations: Faculty of Information Technology, Monash University, Melbourne, VIC 3800, Australia

Abstract

Point-adjustment (PA), long the default scoring protocol in time-series anomaly detection (TSAD), was shown by Kim et al. (2022) to award near-perfect F1 to random scores. The field migrated to replacement metrics: PA%K, range-based precision/recall, affiliation precision/recall, and Volume-Under-the-Surface (VUS) ROC/PR. But did the fix work? Every robustness check to date is proposer-run or theoretical; no independent, adversarial, SOTA-relative audit on real benchmarks exists. We provide one. We stress-test 12 adopted metrics against trivial and adversarial no-skill score generators on the 250-series UCR Anomaly Archive (primary) plus five further benchmarks (SMD, SMAP, MSL, NAB, PSM), under a pre-registered SOTA-relative gameability criterion (a metric is gamed on a series if a no-skill generator reaches >=90% of the best real detector's score). The fix is only partial: affiliation-F1 (gamed on 99% of series) and every ROC-based metric tested (all roughly 62-64%) are gamed by no-skill detectors, genuinely so (real detectors score 0.97-1.00 on the very series they game), while the PR-based metrics and PA%K resist (14-18%). A decisive, one-directional McNemar split: VUS-ROC is gamed on 119 series where its sibling VUS-PR is not, and the reverse on none. The ROC-over-PR direction replicates across all five further benchmarks and never reverses, but the absolute gameability rate is benchmark-dependent and no single metric stays safe on all six (even VUS-PR is gamed on NAB). We release a pip-installable stress-test harness with all metric implementations vendored at frozen commits, plus a metric-selection decision protocol: prefer a PR-based metric or PA%K, treat affiliation-F1 and the ROC-AUC variants as gameable, and verify per benchmark rather than assuming robustness.

Explore similar work

Jun 8, 2026cs.LG

PAI: Preserving Amplitude Information in Representation-Based Time-Series Anomaly Detection

Representation-based time-series anomaly detection algorithms significantly outperform other methods on diverse anomaly detection tasks. However, we notice that they suffer from a major limitation in our evaluation - their learned embeddings are often amplitude-agnostic. Losing amplitude information can degrade performance on amplitude related anomalies, and this failure is prevalent across all existing representation-based methods. To address aforementioned issues, we propose a new anomaly scoring scheme named PAI. PAI consists of two complementary modules, a diagnostic module and a final score augmentation function. The diagnostic module compares cosine and Euclidean scoring on the same representation bank to test whether amplitude information is already captured in the learned representation. Then in final score augmentation function, PAI computes a point-wise median and MAD deviation score and a local mean-shift score-which are fused with the representation score to produce the final anomaly score. On the TSB-AD-U-Eva and TAB UV datasets, PAI improves all four evaluated representation-based methods across every reported metric, achieving average VUS-PR gains of 98.4% and 36.8%, respectively. Among all evaluated combinations, PaAno + PAI achieves the best performance, outperforming the state-of-the-art method by 15%. Further evaluation on bootstrap confidence intervals, anomaly-type breakdowns, and a TS2Vec input-normalization ablation further support the proposed scheme. These results suggest that explicitly retaining amplitude information is important for representation-based time-series anomaly detection, which has been underemphasized in existing scoring schemes. Code is available at: https://github.com/pantheon5100/PAI

Kang Zhang, Wei Jian Lau, Shoushou Ren +3

Time-Series Anomaly Detection Anomaly Score

Ask PDF

Apr 19, 2026cs.LG

Back to Repair: A Minimal Denoising Network for Time Series Anomaly Detection

We introduce JuRe (Just Repair), a minimal denoising network for time series anomaly detection that exposes a central finding: architectural complexity is unnecessary when the training objective correctly implements the manifold-projection principle. JuRe consists of a single depthwise-separable convolutional residual block with hidden dimension 128, trained to repair corrupted time series windows and scored at inference by a fixed, parameter-free structural discrepancy function. Despite using no attention, no latent variable, and no adversarial component, JuRe ranks second on the TSB-AD multivariate benchmark (AUC-PR 0.404, 180 series, 17 datasets) and second on the UCR univariate archive by AUC-PR (0.198, 250 series), leading all neural baselines on AUC-PR and VUS-PR. Component ablation on TSB-AD identifies training-time corruption as the dominant factor (

Δ

AUC-PR

= 0.047

on removal), confirming that the denoising objective, not network capacity, drives detection quality. Pairwise Wilcoxon signed-rank tests establish statistical significance against 21 of 25 baselines on TSB-AD. Code is available at the URL https://github.com/iis-esslingen/JuRe.

Kadir-Kaan Özer, René Ebeling, Markus Enzweiler

Time-Series Anomaly Detection Time Series Image Denoising

Ask PDF

May 27, 2026cs.LG

Benchmarking Inductive Biases for Multivariate Time-Series Anomaly Detection with a Robust Multi-View Channel-Graph Detector

We present a unified experiment, analysis, and benchmark study of multivariate time-series (MTS) anomaly detection. Ten family-representative detectors -- spanning statistical, reconstruction, association, frequency, and generic-transformer families -- are evaluated on five datasets (SMD, MSL, SMAP, PSM, and MSDS) under effectiveness, efficiency, robustness, and cross-dataset generalisation. All methods share the same windowing, scoring, hardware, and metric protocols. Effectiveness, ablation, and robustness use three random seeds; cross-dataset transfer uses seed~0 because each extra seed requires

250

source-target evaluations. The benchmark yields three method-independent findings: no single-bias baseline dominates; absolute perturbation VUS-ROC is more informative than retention ratios; and MSDS behaves as an event-dense deployment workload rather than a sparse point-anomaly benchmark. Under this protocol we also introduce \ours{}, an adaptive detector family combining a NOTEARS-constrained directed channel-graph view with optional patch-attention and temporal-association views. \ours{} achieves the best macro-average VUS-ROC (

0.675

+5.1

~pt over the second-best LSTM-AE), ranks first overall, and reaches the top-3 on all five datasets. Its wins on MSL and MSDS are narrow, while its average and robustness gains are larger: under the same three-seed robustness protocol for every method, it obtains the strongest absolute VUS-ROC across noise, channel dropout, and time-shift perturbations. We release the MSDS preprocessing protocol, configurations, scripts, and seed-level metric dumps.

Junhao Wei, Yanxiao Li, Bidong Chen +9

Time-Series Anomaly Detection Anomaly Detection Time-Series Classification Auc-Roc Multi-View Bidirectional Long Short-Term Memory+1

Ask PDF