Stop Early, Stay Simple: Evidence Acquisition for Power System Fault and Attack Diagnosis
Authors: S M Zia Ur Rashid, Sazidul Islam Hira, Rupesh Sah, Suman Rath
Organizations: Department of Electrical and Computer Engineering, The University of Tulsa, Tulsa, OK, USA · Department of Computer Science and Engineering, International Islamic University Chittagong, Chittagong, Bangladesh
Diagnostic models for cyber-physical power systems usually inspect every telemetry source for every event, although some events are clear from one source and others need several. We ask when a learned policy for choosing that evidence is worth its added complexity. To answer this, we present ActiGrid, a benchmark framework for active evidence acquisition that pairs a classifier trained to tolerate missing modalities with a Double Deep Q-Network (DQN), which either requests another source or stops and diagnoses but never operates grid actuators. On the public TU Delft digital substation and MSU/ORNL power system attack benchmarks, we compare the DQN with fixed subsets and orders, calibrated entropy stopping, a greedy value of information rule, and tree ensembles trained per subset. The learned policy does adapt to what it observes, yet it is statistically indistinguishable from entropy stopping on TU Delft and Pareto dominated by simple controls on MSU/ORNL, while compact subset classifiers outperform it while reading fewer sources on both benchmarks. Entropy stopping run over those subset classifiers reaches the highest mean Macro-F1 on TU Delft (0.928 while reading 57% of the sources) but brings no saving over a static three source subset on MSU/ORNL, where no method exceeds 0.45 Macro-F1. Selective acquisition can pay off, but on these benchmarks it does not need a learned policy.
Figures & tables
Fig. 1: ActiGrid acquisition loop, drawn for the three TU Delft sources.
Dataset
Method
Macro-F1
Cost
TU Delft
Full fusion
0.8158±0.0323
1.000
DQN ( λ=0.5 )
0.7917±0.0333
0.703
Order+entropy
0.7868±0.0347
0.690
Fixed subset, k=2
0.6933±0.0583
0.667
MSU/ORNL
Full fusion
0.3924±0.0077
1.000
DQN ( λ=0.5 )
0.3769±0.0118
0.754
TABLE I: Shared backbone comparison on the primary split over ten training seeds.
Dataset
Method
Macro-F1
Cost
TU Delft
DQN
0.8045±0.0165
0.706
Order+entropy
0.8095±0.0113
0.732
HGB, k=1
0.8673±0.0272
0.333
HGB, k=2
0.8779±0.0292
0.667
HGB, k=3
0.9077±0.0466
1.000
MSU/ORNL
DQN
0.3632±0.0268
0.728
TABLE II: Hardening results over partition seeds 42, 123, and 456.
Dataset
Method
Macro-F1
Cost
TU Delft
Static HGB
0.8972±0.0580
0.778
Sequential HGB, raw
0.9254±0.0443
0.520
Sequential HGB, calibrated
0.9282±0.0378
0.571
MSU/ORNL
Static HGB
0.4376±0.0077
0.933
Sequential HGB, raw
0.4395±0.0057
0.879
Sequential HGB, calibrated
0.4400±0.0066
0.861
TABLE III: Static VS sequential HGB over partition seeds 42, 123, and 456. Static HGB takes the subset with the best Macro-F1 on the second validation half, without a cost cap.
Fig. 2: Macro-F1 against normalized cost on the primary split, with the backbone of initialization seed 42 and the HGB models of partition 42.
Fig. 3: Mask-wise test ECE before and after temperature scaling (seed 42).
Dataset
ECE raw → cal.
Δ F1 zero
Δ F1 noise
TU Delft
0.162→0.100
−0.371
−0.107
MSU/ORNL
0.163→0.059
−0.033
−0.134
TABLE IV: Mask-wise calibration of the initialization seed 42 backbone and first modality corruption of its DQN.
Fig. 4: DQN decision after resource telemetry on TU Delft (seed 42).
Department of Electrical Engineering and Computer Science, University of Tennessee · Department of Computer Science, The University of Illinois at Springfield · Department of Electrical and Computer Engineering, Clemson University, South Carolina