Benchmarking Quantum Machine Learning for Power-System Attack Detection: Evaluation Choices Decide the Outcome Before the Models Do
Organizations: Dhaka, Bangladesh
Abstract
Machine-learning detectors for power-system cyberattacks are themselves attack surfaces, and quantum machine learning has been proposed for them. We benchmark fidelity-kernel SVMs and variational classifiers against six tuned classical models on public power-system attack data (Mississippi State/ORNL), across white-box, transfer, decision-based black-box, and poisoning attacks. Our headline finding is methodological: the benchmark's answers are set by the evaluator's choices before the models. Eight choices -- six in the evaluation protocol, two in the tuning the benchmark itself runs -- each reversed or moved a conclusion at fixed models. The largest is the split: the row-level protocol scores 0.905 macro-F1 where holding whole source files out leaves 0.594, and in the capped matched-dimensionality regime the quantum arm sits within noise of chance with the classical arm 0.024 above it. A fidelity kernel looks most robust until attacked directly (retention 0.886 to 0.064); a mis-fitted surrogate manufactures a 10x asymmetry; an unseeded black-box attack moves 75% between restarts. A positive control explains the accuracy null: the labels, not the pipeline. We give the control that catches each choice and release the seeded benchmark.
Figures & tables
| Row-level split | Whole files held out | |
| Full 128 feat., tuned LightGBM | 0.905 | 0.594 0.008 |
| PCA-8 (2k cap), tuned XGBoost | 0.596 | 0.523 0.013 |
| PCA-8 (2k cap), tuned ZZ-kernel SVM | 0.564 | 0.508 0.012 |
| contrast (classical quantum) | ||
| Stratified-random floor | 0.501 | 0.499 0.001 |
| Test rows w/ train neighbour | 45.8% | 5.0 0.4% |
| Model | macro_f1 | balanced_accuracy | auprc | roc_auc | brier | ece |
|---|---|---|---|---|---|---|
| logreg ∗ | 0.511 0.000 | 0.538 0.000 | 0.746 0.000 | 0.540 0.000 | 0.496 0.000 | 0.038 0.000 |
| rbf_svm ∗ | 0.578 0.000 | 0.576 0.000 | 0.762 0.000 | 0.605 0.000 | 0.466 0.000 | 0.084 0.000 |
| rf ∗ | 0.586 0.002 | 0.582 0.001 | 0.793 0.001 | 0.635 0.001 | 0.408 0.000 | 0.030 0.001 |
| xgb † | 0.597 0.002 | 0.595 0.002 | 0.784 0.001 | 0.630 0.001 | 0.438 0.001 | 0.069 0.002 |
| lgbm ∗ | 0.593 0.001 | 0.590 0.001 | 0.784 0.001 | 0.632 0.000 | 0.454 0.000 | 0.130 0.001 |
| mlp ∗ | 0.456 0.013 | 0.512 0.004 | 0.755 0.006 | 0.563 0.007 | 0.412 0.003 | 0.032 0.006 |
| Model | clean F1 | [email protected] | F1 floor | retention | WB reten. | rand. flip | HSJ med. ( ) |
|---|---|---|---|---|---|---|---|
| qsvm | 0.563 | 0.499 | -0.001 | 0.886 | 0.064 | 45% | 2.818 |
| logreg | 0.514 | 0.431 | -0.069 | 0.838 | — | 8% | 0.517 |
| rbf_svm | 0.587 | 0.415 | -0.085 | 0.707 | 0.685 | 24% | 0.000 |
| mlp | 0.465 | 0.289 | -0.211 | 0.621 | 0.621 | 3% | 0.867 |
| rf | 0.616 | 0.367 | -0.133 | 0.596 | — | 18% | 0.000 |
| lgbm | 0.634 | 0.373 | -0.127 | 0.589 | — | 24% | 0.548 |
| Choice | Control that catches it | Instance |
| Evaluation protocol | ||
| Split protocol | whole-group held-out arm + neighbour audit | Sec. IV-A |
| Surrogate specification | surrogate-competence dose-response | Sec. IV-E |
| Attack availability | matched white-box attack per family | Sec. IV-D |
| Attack objective | loss- and distance-seeking attackers | Sec. IV-F |
| Attack starting point | restarts, reported mean std | Sec. IV-F |
| partition | held-out files | full LGBM | XGB | ZZ QSVM | contrast | floor |
|---|---|---|---|---|---|---|
| row (shipped) | – | 0.9053 | 0.5957 | 0.5643 | 0.5005 | |
| p0 | 1,8,13 | 0.6032 | 0.5347 | 0.5070 | 0.4980 | |
| p1 | 3,7,9 | 0.5837 | 0.5134 | 0.5253 | 0.4979 | |
| p2 | 1,10,11 | 0.6033 | 0.5393 | 0.5093 | 0.5003 | |
| p3 | 5,9,14 | 0.5937 | 0.5028 | 0.4872 | 0.4995 | |
| p4 | 1,11,12 | 0.5884 | 0.5259 | 0.5105 | 0.4991 |
| Model | macro_f1 | balanced_accuracy | auprc | roc_auc | brier | ece |
|---|---|---|---|---|---|---|
| logreg ∗ | 0.598 0.000 | 0.630 0.000 | 0.849 0.000 | 0.692 0.000 | 0.454 0.000 | 0.045 0.000 |
| rf ∗ | 0.884 0.001 | 0.862 0.001 | 0.986 0.000 | 0.966 0.000 | 0.147 0.000 | 0.075 0.001 |
| xgb ∗ | 0.891 0.002 | 0.886 0.002 | 0.979 0.001 | 0.957 0.001 | 0.134 0.001 | 0.009 0.001 |
| lgbm † | 0.906 0.001 | 0.896 0.002 | 0.986 0.001 | 0.970 0.001 | 0.126 0.001 | 0.051 0.001 |
| rbf_svm ∗ | 0.705 0.000 | 0.686 0.000 | 0.881 0.000 | 0.791 0.000 | 0.336 0.000 | 0.109 0.000 |
| mlp ∗ | 0.766 0.015 | 0.760 0.017 | 0.921 0.003 | 0.849 0.008 | 0.285 0.009 | 0.077 0.007 |
| Cell | KTA (grid ) | KTA (eng. ) | quantum win | |
|---|---|---|---|---|
| angle_ry_q8_binary | 13.43 | 0.0023 | 0.183 | 0.027 0.007 |
| zz_q8_binary | 6.34 | 0.0035 | 0.230 | 0.016 0.007 |
| angle_ry_q12_binary | 17.21 | 0.0022 | 0.500 | 0.004 0.003 |
| zz_q12_binary | 73.04 | 0.0145 | 0.055 | 0.053 0.006 |
| angle_ry_q16_binary | 5.60 | 0.0031 | 0.233 | 0.026 0.009 |
| zz_q16_binary | 70.12 | 0.0143 | 0.046 | 0.092 0.010 |
| Model | macro_f1 | balanced_accuracy | auprc | roc_auc | brier | ece |
|---|---|---|---|---|---|---|
| logreg ∗ | 0.496 0.000 | 0.667 0.000 | 0.563 0.000 | 0.772 0.000 | 0.532 0.000 | 0.035 0.000 |
| rf ∗ | 0.887 0.001 | 0.865 0.001 | 0.952 0.000 | 0.973 0.000 | 0.162 0.000 | 0.091 0.001 |
| xgb ∗ | 0.910 0.002 | 0.904 0.002 | 0.959 0.001 | 0.975 0.000 | 0.124 0.001 | 0.014 0.001 |
| lgbm † | 0.917 0.001 | 0.908 0.001 | 0.967 0.000 | 0.980 0.000 | 0.114 0.001 | 0.024 0.001 |
| rbf_svm ∗ | 0.678 0.000 | 0.637 0.000 | 0.685 0.000 | 0.808 0.000 | 0.344 0.000 | 0.068 0.000 |
| mlp ∗ | 0.763 0.030 | 0.755 0.033 | 0.811 0.019 | 0.894 0.010 | 0.288 0.012 | 0.073 0.011 |
| Model | macro_f1 | balanced_accuracy | auprc | roc_auc | brier | ece |
|---|---|---|---|---|---|---|
| logreg ∗ | 0.343 0.000 | 0.513 0.000 | 0.393 0.000 | 0.635 0.000 | 0.648 0.000 | 0.073 0.000 |
| rbf_svm ∗ | 0.490 0.000 | 0.554 0.000 | 0.450 0.000 | 0.677 0.000 | 0.549 0.000 | 0.113 0.000 |
| rf ∗ | 0.498 0.003 | 0.485 0.003 | 0.497 0.001 | 0.700 0.001 | 0.455 0.001 | 0.075 0.002 |
| xgb † | 0.516 0.002 | 0.517 0.002 | 0.501 0.001 | 0.708 0.001 | 0.471 0.001 | 0.059 0.001 |
| lgbm ∗ | 0.510 0.002 | 0.504 0.002 | 0.509 0.001 | 0.707 0.000 | 0.467 0.001 | 0.081 0.001 |
| mlp ∗ | 0.305 0.010 | 0.344 0.004 | 0.373 0.007 | 0.575 0.013 | 0.440 0.003 | 0.046 0.012 |
| (a) all 128 features | |||||
|---|---|---|---|---|---|
| model | nat. atk | atk nat | F1 no-event | F1 natural | F1 attack |
| logreg | 0.253 | 0.376 | 0.432 | 0.433 | 0.622 |
| rf | 0.302 | 0.027 | 0.944 | 0.782 | 0.935 |
| xgb | 0.199 | 0.045 | 0.960 | 0.825 | 0.943 |
| lgbm | 0.197 | 0.035 | 0.963 | 0.840 | 0.948 |
| rbf_svm | 0.367 | 0.034 | 0.900 | 0.724 | 0.920 |
| Encoding | qubits | test macro-F1 | KTA q | KTA RBF | |
|---|---|---|---|---|---|
| angle_ry | 8 | 0.527 0.000 | 0.002 | 0.013 | 13.43 |
| zz | 8 | 0.564 0.000 | 0.003 | 0.013 | 6.34 |
| angle_ry | 12 | 0.541 0.000 | 0.002 | 0.019 | 17.21 |
| zz | 12 | 0.564 0.000 | 0.015 | 0.005 | 73.04 |
| angle_ry | 16 | 0.574 0.000 | 0.003 | 0.006 | 5.60 |
| zz | 16 | 0.567 0.000 | 0.014 | 0.006 | 70.12 |
| Model | train (s) | inference ( s/sample) | vs fastest |
|---|---|---|---|
| logreg | 0.01 | 0.07 | 1 |
| mlp | 1.85 | 2.37 | 33 |
| xgb | 0.78 | 5.40 | 76 |
| lgbm | 3.76 | 12.18 | 171 |
| rf | 2.46 | 26.03 | 365 |
| vqc ‡ | 92.81 | 136.72 | 1918 |
| kernel | length scale | clean F1 | crossings/ray | max | 1 crossing |
|---|---|---|---|---|---|
| fidelity (ZZ) | 0.499 | 0.19 | 1 | 0% | |
| fidelity (ZZ) | 0.578 | 0.41 | 1 | 0% | |
| fidelity (ZZ) | † | 0.579 | 1.01 | 3 | 26% |
| fidelity (ZZ) | 0.517 | 2.35 | 8 | 63% | |
| fidelity (ZZ) | 0.499 | 3.84 | 19 | 69% | |
| RBF (classical) | 0.546 | 0.42 | 2 | 2% |
| surrogate | clean F1 | floor | pos. rate | VQC dmg | class. dmg | fwd/rev |
|---|---|---|---|---|---|---|
| MLP, as tuned | 0.465 | -0.031 | 0.966 | 0.344 | 0.209 | 10.16 |
| MLP, balanced weights | 0.532 | +0.035 | 0.547 | 0.114 | 0.205 | 0.78 |
| RBF-SVM (other class) | 0.587 | +0.090 | — | 0.025 | 0.124 | 0.15 |
| bandwidth | clean F1 | std | F1 floor | end state | |
|---|---|---|---|---|---|
| 0.05 | 0.521 | 0.985 | 0.403 | -0.404 | inverted |
| 0.1 | 0.557 | 0.926 | 0.446 | -0.407 | inverted |
| 0.2 | 0.569 | 0.710 | 0.451 | -0.414 | inverted |
| 0.4 | 0.555 | 0.358 | 0.456 | -0.371 | inverted |
| 0.793 (tuned) | 0.583 | 0.164 | 0.447 | -0.199 | partial |
| 0.8 | 0.583 | 0.155 | 0.434 | -0.210 | inverted |
| (a) objective only: same budget, steps, seed and model | ||||
|---|---|---|---|---|
| bandwidth | loss obj. | mass obj. | factor | baseline |
| 0.05 | 0.985 | 0.971 | 1.0 | 1.000 |
| 0.1 | 0.932 | 0.894 | 1.0 | 1.000 |
| 0.2 | 0.712 | 0.559 | 1.3 | 0.989 |
| 0.4 | 0.335 | 0.073 | 4.6 | 0.693 |
| 0.793 (tuned) | 0.167 | 0.027 | 6.2 | 0.613 |
| configuration | clean F1 | F1 floor | end state | |
|---|---|---|---|---|
| RBF (tuned) | 0.606 | 0.000 | +0.000 | disabled |
| RBF ( 0.3) | 0.587 | 0.002 | +0.003 | disabled |
| RBF ( 0.1) | 0.578 | 0.023 | +0.036 | disabled |
| RBF ( 0.05) | 0.582 | 0.076 | -0.041 | partial |
| RBF ( 0.03) | 0.607 | 0.158 | -0.173 | partial |
| RBF ( 0.02) | 0.563 | 0.238 | -0.286 | inverted |
| quantum kernel | variational (VQC) | classical RBF | ||||
|---|---|---|---|---|---|---|
| ( ) | F1 floor | F1 floor | F1 floor | |||
| 0 | 1.000 | +0.168 | 1.000 | +0.080 | 1.000 | +0.191 |
| 0.01 | 0.997 | -0.151 | 0.982 | +0.073 | 0.990 | -0.067 |
| 0.05 | 0.964 | -0.372 | 0.925 | -0.017 | 0.860 | -0.245 |
| 0.1 | 0.897 | -0.391 | 0.887 | -0.084 | 0.763 | -0.252 |
| 0.2 | 0.769 | -0.405 | 0.981 | -0.195 | 0.640 | -0.256 |