Metamorphic testing of learned AC optimal power flow surrogates reveals representation dependence
Organizations: AI Science Institute, Boston Consulting Group
Abstract
Evaluating a learned AC optimal power flow (AC-OPF) surrogate on one grid serialization cannot show whether its accuracy depends on that encoding. We test this dependence with thirteen metamorphic relations, each specifying when a transformation is valid and how to compare its outputs. Our audit covers 17 model and pipeline configurations: two released trained checkpoints and 15 random-weight architecture or mechanism controls. Every configuration violates at least six relations, even at reporting tolerances ten times tighter or looser. For GridSFM-Open v1.1, source inspection and intermediate-tensor interventions point to an indexing bug and three noncanonical or numerically unstable preprocessing choices; a repaired checkpoint remains untested. In a separate case500 training experiment, augmentation lowers transformed-view error in three of four models and median discrepancy and constraint-violation P95 in all four. These benefits depend on the training protocol: some relations worsen, and every evaluated prediction remains AC infeasible. A GPU/CPU eigensolver mismatch accounts for most of the large raw-Laplacian-baseline gain. We recommend reporting representation consistency alongside accuracy, feasibility, and cost. These results establish neither deployment readiness nor universal invariance.
Figures & tables
| Tier | Relations | What is held fixed / restored |
|---|---|---|
| REP | component-relabel , bus-relabel , row-order | Physical identities; inverse permutations |
| REP | branch-reversal , tap-side-flip | Branch two-port equations; endpoint/attribute transport |
| REP | ref-bus-move , island-ref-mix | Per-island angle gauge after mapping back |
| REP | basekv-scale | Exported per-unit target; conditional on schema semantics |
| ELC | component-split , parallel-split | Aggregate injection, costs or admittance; sum split outputs |
| ELC | load-as-gen , null-pad | Net bus injection; undo device recoding/padding |
| Model | Canonical MAE | Transformed MAE | Median | Eq. violation P95 (p.u.) | Ineq. violation P95 ( tol.) |
|---|---|---|---|---|---|
| MPNN | |||||
| LapPE+MPNN | |||||
| RWSE+MPNN | |||||
| LUMINA |
| Record | Minimum contents |
|---|---|
| Data and labels | Release, split, topology membership, scenario identifier, units, angle-reference convention, and label-generation pipeline. |
| Relation | Name, tier, parameters, applicability decision, transformed-case hash, and output map. |
| Solver control | Status, objective difference, equality and inequality constraint violations, and admission decision for both views. |
| Model | Code and checkpoint version, checksums, preprocessing contract, feature order, normalization, and output semantics. |
| Sampling | Source-scenario count, sampled transformations, weight seeds, and the clustering unit used for uncertainty. |
| Metrics | Per-channel median, tail, and worst discrepancy plus accuracy, AC constraint violations, bounds, feasibility, and cost. |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Relation | Tier | Transformation | Validity condition | Output map |
|---|---|---|---|---|
| component-relabel | REP | Reorder the generator array (or load, or shunt) by a permutation , moving its bus-link rows with it; buses and branches untouched. | always (every ) | apply to the per-component outputs |
| bus-relabel | REP | Reorder the bus array by a permutation ; rewrite every stored bus index through (branch from/to endpoints and all component-to-bus links). | always (every ) | apply to the per-bus outputs |
| row-order | REP | Reorder the rows of one edge family ( ac_line , transformer, or a component-link table); every field value and endpoint unchanged (a pure serialisation reshuffle). | always | identity on physical quantities; re-sort through canonical_link_order before positional conversion |
| branch-reversal | REP | Reverse a branch: swap its from/to endpoints, swap the shunt halves , and replace with . | per row: , shift , (all ac_line rows; only untapped, unshifted transformers) | reverse that branch’s directed-flow channels, |
| ref-bus-move | REP | Move the angle reference: set a chosen generator bus’s type to reference and demote the former reference to PV; no other field changes. | the chosen bus is a generator bus in the same island (one reference per island) | identity on components; re-anchor both solutions’ angles to a common physical bus |
| basekv-scale | REP | Multiply the model-visible base_kv field at every bus by a common factor after OPFData export. | pipeline-specific: the standard per-unit AC-OPF target is unchanged | identity (solver quantities are unchanged) |
| Collapse in | Relations |
|---|---|
| Incidence is label-free | component-relabel , bus-relabel , row-order , island-ref-mix |
| Only the aggregate branch model enters the equations | branch-reversal , tap-side-flip |
| Bus balance sees aggregate injection instead of per-device | component-split , load-as-gen , null-pad |
| Parallel branches collapse to one effective admittance | parallel-split |
| Angle reference is a gauge convention | ref-bus-move , island-ref-mix |
| The post-export per-unit target does not use model-visible base_kv | basekv-scale |
| Row | Class | Scope |
|---|---|---|
| GridSFM-Open v1.1 | Open weight | Released 15,148,227-parameter GridSFM graph-transformer checkpoint and inference pipeline [ 23 , 24 , 25 ] . |
| LUMINA-2M v0.1.3 | Open weight | Released 2,375,548-parameter, six-layer heterogeneous model and inference pipeline [ 26 , 27 ] . |
| GridFM-GNS v0.8.1 | Public architecture | Released heterogeneous preprocessing, model, and decoder at random initialization; we found no compatible public AC-OPF checkpoint as of August 25, 2026 [ 43 ] . |
| DC3 (index partition) | Design reimplementation | Random index masks for partial outputs followed by covariant mean completion [ 7 ] . |
| DC3 (physical partition) | Matched control | Same completion stand-in with masks keyed by slack/PV/PQ roles rather than array position. |
| Graph transformer (LapPE-4) | Design reimplementation | Global bus self-attention with four raw, uncanonicalized Laplacian eigenvectors [ 12 ] . |
| Threshold | Failed cells / 221 | Failed relations per model | Models with a failure |
|---|---|---|---|
| 160 | 6–13 | 17/17 | |
| 157 | 6–13 | 17/17 | |
| 154 | 6–12 | 17/17 | |
| 144 | 4–12 | 17/17 | |
| 134 | 4–12 | 17/17 |
| GridSFM-Open v1.1 | LUMINA-2M v0.1.3 | |||
|---|---|---|---|---|
| Relation | Trained | Random seed range | Trained | Random seed range |
| component-relabel | ||||
| bus-relabel | ||||
| row-order | ||||
| branch-reversal | ||||
| ref-bus-move | ||||
| Relation | Trained weights | Random weights | ||
|---|---|---|---|---|
| Absolute | Relative | Absolute | Relative | |
| component-relabel | ||||
| bus-relabel | ||||
| row-order | ||||
| branch-reversal | ||||
| ref-bus-move | ||||
| Outcome | Presentation | Metric | MPNN | LapPE+MPNN | RWSE+MPNN | LUMINA | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Base | Aug. | Imp. (%) | Base | Aug. | Imp. (%) | Base | Aug. | Imp. (%) | Base | Aug. | Imp. (%) | |||
| Mean absolute error (MAE) | Canonical | Mean | ||||||||||||
| Model | Arm | Canonical MAE | Transformed MAE | Median | Equality P95 (p.u.) | Inequality P95 ( tol.) | Feasible (%) |
|---|---|---|---|---|---|---|---|
| MPNN | Base | ||||||
| Aug. | |||||||
| LapPE+MPNN | Base | ||||||
| Aug. | |||||||
| RWSE+MPNN | Base | ||||||
| Aug. |
| Model | Constraint family | Unit | Median | P95 | Maximum | |||
|---|---|---|---|---|---|---|---|---|
| Base | Aug. | Base | Aug. | Base | Aug. | |||
| MPNN | Active-power balance | p.u. | ||||||
| Reactive-power balance | p.u. | |||||||
| Voltage bounds | p.u. | |||||||
| Generator active-power bounds | p.u. | |||||||
| Generator reactive-power bounds | p.u. | |||||||
| Global-max nodal complex-power balance residual (% total apparent demand) | Global-max voltage-bound excess (% nominal) | Canonical mean maximum nodal complex-power balance residual (% total apparent demand) | |||||
| Model | Base | Aug. | Base | Aug. | Base | Aug. | Imp. (%) |
| MPNN | |||||||
| LapPE+MPNN | |||||||
| RWSE+MPNN | |||||||
| LUMINA | |||||||
| Grid | MPNN | LapPE+MPNN | RWSE+MPNN | LUMINA |
|---|---|---|---|---|
| case14 | ||||
| case118 | ||||
| case500 | ||||
| case2000 | ||||
| case4661 |
Optimal Power Flow
Optimal Power Flow (OPF) finds low-cost power-system operating plans that satisfy network, voltage, and equipment constraints.
See all 24 papers