A graph neural network (GNN) forecasting a collection of load series combines two mechanisms, sharing parameters across series and reading neighboring series, and a gain over a non-graph model may come from either; telling them apart decides whether a graph is worth maintaining. We propose a model diagnostic protocol that compares the GNN with three reference models: a per-node network, a network shared across nodes without neighbors, and the same GNN without edges. Degree-preserving rewiring tests whether neighbor identity matters. After a tuned benchmark on regional load in France and on low-voltage feeders in the UK, we apply the protocol to deliberately short node histories, as when non-stationary demand leaves only recent observations relevant: the feeders, within one week, and a public dataset of client consumption series. On the feeders, sharing parameters recovers most of the error gap between node-specific models and GNNs, about two thirds against a regularized one. Adding the edges to an otherwise identical backbone brings a smaller gain, half or more of which disappears under rewiring. On the client series, the neighbor mean degrades the per-node model on the shorter window and the edges' gain is not consistently distinguishable from zero; on the regions of France, where each node's own history is already informative, the ranking between per-node models and GNNs changes with tuning. A synthetic study points to noisy local measurements as one regime where neighbors help. In an exploratory online aggregation, graph and non-graph forecasters appear complementary, and the weight assigned to GNNs is not proportional to their contribution. In none of the three datasets do the edges account for most of a GNN's advantage over node-specific models, so a forecasting gain should be credited to the graph only after comparison with per-node and shared references.
Figures & tables
Figure 1: The four node-wise models on the sharing (rows) and access (columns) axes. Each cell shows the series reaching the forecast y^3 , the matrix producing it and its parameter count for N nodes, d features, and H hidden units; each segment is one d×H block.
Block
Datasets
Training window
Models
Fixed or distinctive settings
Architecture benchmark (§ 6.1 ; Q1)
regions-fr ; weave-uk
1090 d; 9 d
8 tuned GNNs, non-graph baselines, GNN and shared-MLP seed ensembles, and GNN mixtures
Split and horizon within each dataset; inputs as specified
Per-node reference (§ 6.2 ; Q1, Q4)
regions-fr
54 – 1090 d
Per-node MLP ( ± neighbor mean) and all-node MLP; identity-graph and SAGE GNN
Table 1: The experimental blocks presented in Section 6 that address the questions of Section 1 .
Table 3
Figure 2: regions-fr . Left: national RMSE of the 4 node-wise models at one fixed configuration (width 64 , untuned) as the training set contracts, 3 seeds (Table C.2 , Appendix). Right: at the full sample, each family at the fixed configuration (hollow) and tuned (filled, Table 3 ), beside the tuned pooled XGBoost and the bottom-up graph mixture.
Figure 3: RMSE over nodes and test snapshots, in standardized load units, of the 4 node-wise models on the synthetic denoising data, 3 seeds per point with the spread shaded; each plot varies one factor around the base configuration.
electricity , 319 clients (RMSE in MW)
weave-uk , 500 feeders (RMSE in kWh)
Model
RMSE
MAPE (%)
nRMSE
RMSE
MAPE (%)
nRMSE
Short window: 11 days ( electricity ), 30 hours ( weave-uk )
Per-node MLP
22.3 ( 0.7 )
2.34 ( 0.11 )
0.3030 ( 0.0002 )
346.8 ( 3.8 )
9.73 ( 0.14 )
1.0509 ( 0.0013 )
Per-node MLP + neighbors
24.1 ( 2.6 )
2.72 ( 0.34 )
0.3201 ( 0.0036 )
323.2 ( 2.0 )
8.89 ( 0.08 )
0.9995 ( 0.0010 )
Shared MLP
19.4 ( 1.3 )
2.10 ( 0.15 )
0.2804 ( 0.0014 )
329.7 ( 11.0 )
9.02 ( 0.31 )
0.9074 ( 0.0103 )
All-node MLP
26.0 ( 2.4 )
2.89 ( 0.23 )
0.4527 ( 0.0048 )
421.8 ( 82.3 )
12.45 ( 2.75 )
1.0609 ( 0.0583 )
Table 4: The node-wise models at hidden width 64 on the full datasets, 3 seeds, under the two system scores and the per-node error (nRMSE). The GNN uses ChebConv ( K=3 ) on electricity at 60 days and weave-uk at 30 hours, SAGE otherwise, as selected on validation. Equal width does not match depth or total parameter count. weave-uk window lengths include the validation portion; electricity lengths are fitting windows, followed by 2 validation days.
Figure 4: The two decompositions of the per-node gap on the 60 -day electricity window (blue) and the 30 -hour weave-uk window (green) of Table 4 . Vertices give per-node errors; each arrow points to the better model and gives the error decrease. Both routes reach the same total, with increments differing by 0.0248 on weave-uk ; arrows into the GNN also change the backbone, so this difference is not a pure factorial interaction.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1: Example of a message-passing layer in a GNN. Vℓ , Eℓ , and Uℓ respectively refer to node, edge, and global level at layer ℓ . ϕ and ρ are respectively update and propagation functions.
Figure B.1: Weights ML-Poly plays over the test year 2019 . The two left columns are regions-fr and the synthetic dataset whose residual covariance follows the spatial graph (unobservable by the experts); the third has none, its residuals being independent across nodes. Single-step forecasting is above and sequence-to-sequence forecasting below. The 6 GNN experts are summed into one plum band, the feedforward network appearing in blue and the additive model in orange. The dashed line marks 0.75 , the share the 6 hold among the 8 by count alone, so a band sitting on it reports an allocation that cannot separate its experts and not an accuracy the edges bought. The strip beneath each plot carries the departure from that share, plum where the mixture buys the GNN experts more than counting them would give and gray where it buys them less.
Dataset
Setting
Best expert
Uniform
ML-Poly
Without the GNN experts
Without the two others
regions-fr
single-step
1249
1144
970
+132 ( 14% )
+117 ( 12% )
seq-to-seq
1047
893
844
+162 ( 19% )
+90 ( 11% )
Synthetic structured
single-step
917
838
799
+79 ( 10% )
+43 ( 5% )
seq-to-seq
800
784
766
+34 ( 4% )
+1 ( 0% )
Synthetic independent
single-step
658
656
624
+23 ( 4% )
+24 ( 4% )
seq-to-seq
604
576
574
+19 ( 3% )
+4 ( 1% )
Appendix
Table B.1: National root mean squared error in MW over the test year, and what each family of experts is worth inside the mixture. The 4 cells above the rule have spatially correlated loads, the two below independent ones given their own covariates; ‘seq-to-seq’ abbreviates the sequence-to-sequence setting. The last two columns give the error ML-Poly loses when the 6 GNNs are withdrawn from the pool, and when the feedforward and the additive model are, in MW and as a share of the mixture’s own error. The datasets sit at different load levels, so the shares compare across rows where the MW values do not.
Figure B.2: Mean absolute percentage error of each region’s own forecast over the test year, single-step setting. Left, GatedGraph, the most accurate of the 6 GNN experts region by region; center, the additive model; right, ML-Poly run region by region, each region carrying its own mixture and its own weights. The color scale is shared. Regions carry their official codes: ARA Auvergne-Rhône-Alpes, BFC Bourgogne-Franche-Comté, BRE Bretagne, CVL Centre-Val de Loire, GES Grand Est, HDF Hauts-de-France, IDF Île-de-France, NAQ Nouvelle-Aquitaine, NOR Normandie, OCC Occitanie, PAC Provence-Alpes-Côte d’Azur, PDL Pays de la Loire.
Figure C.1: Sequence-to-sequence evaluation protocol ( 48 half-hourly steps, stride h=48 ). Input windows ik (blue) each feed a forecast window fk (orange), and consecutive forecast windows are contiguous, so each prediction spans w=48 lags into h=48 future steps. Training, validation and testing all use this stride, so every forecast origin falls on a day boundary and output k always denotes slot k of the day. A stride of one would place the origin at an arbitrary half-hour and force a single 48 -output head to represent 48 phase-shifted mappings, which is a different learning problem and a different test set; every model of the benchmark, graph or not, is trained and scored under the stride of this figure; Section 6.4 states its own protocol.
Figure C.2: Temporal splits for training, validation, and testing on regions-fr and weave-uk .
Features
Best
Best to worst
Calendar and weather
1141
586
+ temperatures of the target slots
1133
367
+ 48 load lags
865
120
Both blocks
768
119
Appendix
Table C.1: Validation RMSE in MW on regions-fr , by feature set. The middle column is the best of 18 GNN configurations ( 6 convolutions at three widths), and the right column is the gap between the best and the worst of those. It uses 1 seed, so the column reads as a screening pass and not as a ranking.
Training days
Per-node MLP
All-node MLP
Identity-graph GNN
SAGE GNN
54 ( 5% )
1948 ( 69 )
2432 ( 269 )
2038 ( 103 )
2080 ( 112 )
109 ( 10% )
1412 ( 28 )
1687 ( 105 )
1763 ( 101 )
1738 ( 55 )
272 ( 25% )
1090 ( 41 )
1179 ( 25 )
1292 ( 17 )
1206 ( 41 )
545 ( 50% )
958 ( 5 )
1014 ( 29 )
1113 ( 31 )
983 ( 27 )
1090 ( 100% )
876 ( 3 )
921 ( 3 )
918 ( 7 )
891 ( 19 )
Appendix
Table C.2: National RMSE (MW) on regions-fr as the training set contracts, SAGE convolution, days drawn at random, 3 seeds.
Figure C.3: Table C.2 read against the training window, on a logarithmic day axis. Left: the national error of the 4 node-wise models, the per-node MLP leading at every volume. Right: the same table as increments over that reference model, so a curve is the price of one restriction and a higher value is a costlier one. The access restriction falls from 484 to 45 MW as the window opens and the edge cost from 132 to 15 MW; weight sharing costs 42 – 351 MW with no monotone trend.
Window
Nodes
MLP
GNN
Per-node
Shared
All-node
SAGE
ChebConv
Per-node error, nRMSE
30 h
10
1.0790 ( 0.0194 )
0.9846 ( 0.0578 )
0.9333 ( 0.0135 )
0.9444 ( 0.0170 )
0.9632 ( 0.0049 )
30 h
40
1.0216 ( 0.0161 )
0.8746 ( 0.0298 )
0.8952 ( 0.0100 )
0.8719 ( 0.0405 )
0.8637 ( 0.0304 )
30 h
160
1.0733 ( 0.0190 )
0.9315 ( 0.0302 )
0.9402 ( 0.0244 )
0.8949 ( 0.0256 )
0.8994 ( 0.0164 )
30 h
500
1.0509 ( 0.0013 )
0.9074 ( 0.0103 )
1.0609 ( 0.0583 )
0.8760 ( 0.0015 )
0.8808 ( 0.0047 )
Appendix
Table E.1: weave-uk as the dataset is thinned, three draws of nodes and 1 seed each, the same 5 -nearest graph rebuilt inside every draw. The per-node error is computed after standardizing targets separately by node and horizon using training moments, then taking the root mean squared error over test origins and horizons and averaging over nodes. A value of 1 denotes an error on the training-standard-deviation scale, not a test-set mean-forecast baseline. This score and MAPE compare across node counts, whereas a raw RMSE grows with the number of feeders summed and is given at 500 of them in Table 4 , whose fixed width of 64 this table shares. The stds over the draws in parentheses; at 500 feeders the draw is the whole dataset, so the three repetitions of that row vary the fit alone. Gray shades mark the top three models of each row, darkest being the best. The GNN improves on the per-node reference model at every sampled size. Its gain over the shared MLP grows with node count on the 10 -day window and does not order with node count on the 30 -hour one; the limited repetitions do not identify a threshold on either.
Dataset and window
Forecast change: rewired − true
Edge gain
Graph diagnostic: true → rewired
RMSE
MAPE points
nRMSE
nRMSE
zresidual
Two-hop nodes
electricity , 11 d
+2.82 MW
+0.31
+0.0046
+0.0033
23.3→1.5
39.5→72.8
electricity , 60 d
−3.29 MW
−0.35
−0.0082
+0.0002
21.0→1.9
43.7→76.3
weave-uk , 30 h
−33.6 kWh
−0.40
+0.0036
+0.0069
13.1→1.6
135.1→144.0
weave-uk , 10 d
+24.3 kWh
+1.21
+0.0178
+0.0191
10.3→0.7
135.1→144.0
Appendix
Table F.1: Paired degree-preserving rewiring control, averaged over 3 seeds, against the true graph stored as a simple graph. Forecast columns report rewired minus true performance, so positive values are degradations; the last forecast column gives the edge increment at constant backbone on that simple graph. Structural columns show how the residual-alignment score and two-hop neighborhood size change when only the edge identities are randomized.
electricity
weave-uk
Model
11 d
60 d
30 h
10 d
Persistence
0.5573
0.5768
0.9776
0.9600
Per-node MLP
0.3030
0.2776
1.0509
0.8956
Per-node ridge
0.2909
0.2839
0.9519
0.8662
Shared ridge
0.2980
0.2942
0.8922
0.8286
Shared MLP
0.2804
0.2770
0.9074
0.8013
Appendix
Table F.2: Per-node error (nRMSE) of the decomposition’s models beside stronger node-specific references, on the cells of Table 4 . Ridge regressions are fitted in closed form on the same standardized inputs; the per-node ridge selects its penalty node by node on the validation window, the shared ridge on the pooled validation error. Persistence repeats the same slot of the previous day. The last two rows give the share of the gap between the stronger node-specific reference and the GNN that the shared MLP and the shared ridge recover.
electricity
weave-uk
Model
11 d
60 d
30 h
10 d
Per-node MLP
0.3030
0.2776
1.0509
0.8956
Per-node MLP + neighbors
0.3201
0.2752
0.9995
0.8762
Shared MLP
0.2804
0.2770
0.9074
0.8013
Shared MLP + neighbors
0.2933
0.2886
0.8941
0.7707
GNN
0.2775
0.2752
0.8808
0.7745
Appendix
Table F.3: Per-node error (nRMSE) of the MLP family crossed on sharing and neighbor access, beside the GNN, on the cells of Table 4 ; 3 seeds, same inputs, graph, width, and training routine. The neighbor mean is that of the per-node MLP with neighbors.
Window end
Total gap
Sharing (share)
Neighbors, unshared
Edges, same backbone
Jul 2019
+0.026
+0.023 ( 89% )
−0.017
+0.005[−0.001,+0.011]
Jun 2019
+0.017
+0.014 ( 80% )
−0.013
−0.001[−0.005,+0.002]
May 2019
+0.046
−0.023 ( −50% )
−0.011
+0.004[+0.000,+0.007]
Apr 2019
+0.028
+0.031 ( 111% )
−0.012
+0.001[−0.004,+0.006]
Mar 2019
+0.017
+0.004 ( 23% )
−0.022
−0.001[−0.005,+0.002]
Appendix
Table F.4: Increments of per-node error on the 11 -day electricity cell over 5 disjoint windows, 3 seeds; seed-and-node 95% intervals in brackets. The first row is the window of Table 4 .
Figure G.1: Explanation graphs for the GAT convolutional layer on regions-fr .
Figure G.2: t-SNE projections of the final 64 -dimensional latent representations of the matched shared MLP and SAGE GNN on the regions-fr test set. Every third test day and all 12 regions are retained, giving 1452 identical observations in each plot; the upper row colors them by region and the lower row by quarter of the test period. The projections are fitted separately with perplexity 40 and a fixed seed, so only within-plot neighborhoods are read.
Figure G.3: Geographic distance against mean same-day latent distance for all 66 pairs of regions, in the original standardized representation spaces. The dashed line is a linear visual guide; the reported statistic is Spearman’s rank correlation, with a two-sided p -value obtained from 9999 permutations of the region labels to preserve the dependence among pairwise distances. The 5 pairs separated by less than 200 km are labeled.
Figure G.4: t-SNE projections of the final 64 -dimensional states of the shared MLP and SAGE GNN on weave-uk at 500 feeders. All feeders and every 12th test origin are retained, giving 6500 identical observations per plot; the upper row colors them by 8 spectral clusters of the fixed graph and the lower row by quarter of the test window. The projections are fitted separately with perplexity 50 and a fixed seed, so only within-plot neighborhoods are read.
Figure G.5: Two projections of the GAT attention matrices on regions-fr , first layer and first head, one point per half-hour of the test year and colored by its position in that year. The partial separation of successive periods places the attention matrices along the seasonal cycle, which moves temperature, daylight, and activity together.
Figure G.6: PCA of the attention matrices on the synthetic dataset with two coupling modes A1 and A2 , colored by the mode in effect. In (a), the mode is handed to the model as a binary input and the two clusters separate at once. In (b) to (d), it is only implied by the joint value of two binary inputs, and the separation is built layer by layer: the first layer mixes the two modes, the second splits them partially, the third recovers them.
Figure G.7: Why pruning is performed separately for every target node. A global ranking mostly separates node degrees because attention is normalized over each incoming neighborhood. Within-node pruning fixes the retained degree and changes only the identity of the retained edges. Green edges are kept.
Edges kept
Count
Highest attention
Random
Lowest attention
All edges
5000
0.8763
0.8763
0.8763
One half
2412
0.8768 ( +0.06% )
0.8835 ( +0.82% )
0.8912 ( +1.70% )
One quarter
1235
0.8862 ( +1.07% )
0.8938 ( +1.17% )
0.9079 ( +1.87% )
One tenth
654
0.8931 ( +0.78% )
0.9031 ( +1.04% )
0.9218 ( +1.53% )
No neighbors
0
0.8956 ( +0.28% )
0.8956 ( −0.83% )
0.8956 ( −2.84% )
Appendix
Table G.1: Direct test nRMSE, lower is better, of the per-node MLP augmented by the mean features of its retained neighbors. Values are averaged over 3 seeds. Each value in parentheses gives the relative change from the row above to the row below, 100(Ebelow−Eabove)/Eabove . Red positive values denote an increase in error; green negative values denote an improvement. GATv2 ranks the edges on the full retained graph; it does not produce the errors in the table. The full-graph and no-neighbor values are repeated across the three columns because edge ranking is irrelevant when all or no edges are retained. Seed dispersion is 0.0003 – 0.0033 .
Dataset
Model
Graph
Batch size
Layers
Width
Learning rate
regions-fr
GCN
Space
8
3
68
2⋅10−3
SAGE
DTW
8
2
196
1⋅10−3
GAT
Space
8
2
75
2⋅10−3
GATv2
Space
8
3
155
6⋅10−4
Transformer
Correlation
8
2
88
2⋅10−3
TAG
Space
8
3
98
4⋅10−3
Appendix
Table H.1: Hyperparameters of the GNN models for regions-fr and weave-uk in the sequence-to-sequence forecasting setup. The Graph column gives the construction of Section 4.1 retained by validation; on regions-fr the dense constructions share one (near-)complete support, which SAGE, GAT, GATv2, and Transformer read identically (Section 4.1 ).
Electricity forecasting often involves spatially related signals observed over regions, substations, and feeders, and Graph Neural Networks (GNNs) provide a natural way to represent these relations. Building a complete GNN forecasting experiment is nonetheless laborious, because graph construction, model selection, training, aggregation, and interpretation sit in incompatible tools. We present GraphToolbox, an open-source Python framework that unifies these stages in one configurationdriven pipeline built on PyTorch Geometric. It offers data-driven graph construction, an adapter that instantiates and trains 51 of the 65 PyTorch Geometric convolutions together with the recurrent cells of PyTorch Geometric Temporal, online expert aggregation, forecasting interpretability, and significance testing on cached forecasts. We evaluate the pipeline in two case studies. On French regional load, the 48 convolutions included in the complete forecasting sweep fall in a band from 1.14% to 1.60% error, online aggregation lowers this to 0.98%, and the graph models improve on classical additive and boosting baselines. On net-load, direct graph models are less accurate than a classical additive model, while forecasting each physical component separately improves them without closing that gap. Both comparisons use the same experimental interface, illustrating the role of GraphToolbox in systematic architectural evaluation.
Graph neural networks (GNNs) are routinely employed for spatiotemporal forecasting, yet their performance across widely used benchmark datasets is inconsistent. Here, we perform an audit of dataset properties and baseline models to assess the quality of the benchmarks, and the robustness of the conclusions drawn from them. Using classical statistical tools, we characterise spatiotemporal lagged dependencies in benchmarks, and examine how temporal differencing changes these relationships and affects model rankings. Motivated by this, we re-evaluate temporal linear baselines, significantly reducing the apparent gains from GNNs on several benchmarks, and surpassing GNNs on others. Suspecting that GNNs struggle to extract linear, node-wise signals, we find that supplying them with autoregressive residuals improves their performance particularly on non-traffic benchmarks. Finally, controlled synthetic experiments reveal that GNNs are sensitive to heterogeneity in temporal dynamics and spatial graph interactions. Together, our findings demonstrate that baseline specification, data pre-processing and system heterogeneity shape the interpretations drawn from benchmark rankings, informing the design and robust evaluation of GNNs.
Kenneth Martin, Simon Heilig, Asja Fischer +3
Ruhr University Bochum Bochum, Germany · Ben-Gurion University of the Negev Beersheba, Israel
In this paper, we propose a proof-of-concept Graph Neural Network model that can successfully predict network flow-level traffic (NetFlow) by accurately modelling the graph structure and the connection features. We use sliding-windows to split the network traffic in equal-sized heterogeneous bidirectional graphs containing IP, Port, and Connection nodes. We then use the GNN to model the evolution of the graph structure and the connection features. Our approach shows superior results when identifying the Port and IP to which connections attach, while feature reconstruction remains competitive with strong forecasting baselines. Overall, our work showcases the use of GNNs for per-flow NetFlow prediction.
Georgios Anyfantis, Pere Barlet-Ros
Department of Computer Architecture Universitat Polit`ecnica de Catalunya Barcelona, Spain