Topology-Aware Graph Reinforcement Learning for Energy Storage Systems Optimal Dispatch in Distribution Networks

Shuyi Gao; Stavros Orfanoudakis; Shengren Hou; Peter Palensky; Pedro P. Vergara

cs.LGMar 27, 2026

Topology-Aware Graph Reinforcement Learning for Energy Storage Systems Optimal Dispatch in Distribution Networks

Authors: Shuyi Gao, Stavros Orfanoudakis, Shengren Hou, Peter Palensky, Pedro P. Vergara

Organizations: Department of Electrical Sustainable Energy, Delft University of Technology, Mekelweg 4, Delft, 2628 CD, The Netherlands · Beijing Energy Quant Ltd., Haidian, Beijing, 53100, China

Abstract

Optimal dispatch of energy storage systems (ESSs) in distribution networks involves jointly improving operating economy and voltage security under time-varying conditions and possible topology changes. To support fast online decision making, we develop a topology-aware Reinforcement Learning architecture based on Twin Delayed Deep Deterministic Policy Gradient (TD3), which integrates graph neural networks (GNNs) as graph feature encoders for ESS dispatch. We conduct a systematic investigation of three GNN variants: graph convolutional networks (GCNs), topology adaptive graph convolutional networks (TAGConv), and graph attention networks (GATs) on the 34-bus and 69-bus systems, and evaluate robustness under multiple topology reconfiguration cases as well as cross-system transfer between networks with different system sizes. Results show that GNN-based controllers consistently reduce the number and magnitude of voltage violations, with clearer benefits on the 69-bus system and under reconfiguration; on the 69-bus system, TD3-GCN and TD3-TAGConv also achieve lower saved cost relative to the NLP benchmark than the NN baseline. We also highlight that transfer gains are case-dependent, and zero-shot transfer between fundamentally different systems results in notable performance degradation and increased voltage magnitude violations. This work is available at: https://github.com/ShuyiGao/GNNs_RL_ESSs and https://github.com/distributionnetworksTUDelft/GNNs_RL_ESSs.

Figures & tables

Figure 1: Architecture of the proposed GNN-based RL framework for distribution-network ESS optimal dispatch, consisting of the graph feature encoder, the ESS dispatch environment, the experience replay buffer, and the actor–critic networks.

Figure 2: The average reward of RL algorithms during 1000 episodes of training on (a)34 buses system, (b)69 buses system.

Figure 3: Comparison of RL algorithms on 34 buses system, (a)electricity price and ESS nodes’ active power, (b)-(f) ESS nodes’ SOC and charging/discharging strategy of NLP, TD3-NN, TD3-GCN, TD3-TAGConv, TD3-GATConv on Node 34.

Figure 4: ESS nodes’ voltage magnitude on 34 buses system, (a)without ESS control, (b)NLP, (c)TD3-NN, (d)TD3-GCN, (e)TD3-TAGConv, (f)TD3-GATConv.

Algorithms	TD3-NN	TD3-GCN	TD3-TAGConv	TD3-GATConv	NLP
Training [ $h/m/s$ ]	2h36m6.6s	5h22m14.7s	5h58m55.8s	7h28m51.3s	-
Execution [ $s$ ]	0.0998 (0.0990, 0.1007)	0.1968 (0.1935, 0.2001)	0.1909 (0.1898, 0.1920)	0.2274 (0.2256, 0.2291)	19.77
Saved costs [$]	197.92 (195.71, 200.13)	179.74 (175.12, 184.38)	188.04 183.53, 192.54)	195.28 (191.48, 199.07)	260.40 (252.34, 268.47)
Accuracy vs NLP [ $\%$ ]	76.31 (75.46, 77.15)	69.51 (67.72, 71.31)	72.61 (70.95, 74.26)	75.46 (74.02, 76.90)	-
Accuracy vs TD3-NN [ $\%$ ]	-	91.92 (89.87, 93.96)	96 (93.58, 98.43)	99.57 (97.68, 101.46)	-
Average Voltage Violation Counts [-]	4	5	4	5	-

Table 1: RL algorithms’ average training time [

h/m/s

], mean and

95\%

confidence bounds of execution time [

s

] and performances on 34 buses testsets.

Algorithms	TD3-NN	TD3-GCN	TD3-TAGConv	TD3-GATConv	NLP
Training [ $h/m/s$ ]	2h47m42.6s	5h32m22.5s	6h11m58.2s	7h50m39.6s	-
Execution [ $s$ ]	0.1340 (0.1327, 0.1354)	0.2351 (0.2323, 0.2378)	0.2381 (0.2347, 0.2415)	0.2878 (0.2830, 0.2927)	144.37
Saved costs [$]	458.98 (450.91, 467.06)	498.83 (489.58, 508.07)	466.44 (453.93, 478.96)	439.42 (388.57, 490.281)	868.19 (830.15, 906.25)
Accuracy vs NLP [ $\%$ ]	54.56 (53.69, 55.44)	59.30 (58.21, 60.38)	55.54 (54.25, 56.82)	52.74 (47.19, 58.29)	-
Accuracy vs TD3-NN [ $\%$ ]	-	109.87 (106.65, 113.10)	102.54 (100.86, 104.21)	97.99 (87.13, 108.86)	-
Average Voltage Violation Counts [-]	77	54	33	151	-

Table 2: RL algorithms’ average training time [

h/m/s

], mean and

95\%

confidence bounds of execution time [

s

] and performances on 69 buses testsets.

ID	Reconfiguration	Connections
TP1	–	Baseline topology
TP2	1 within a branch	(25,26) $\Rightarrow$ (24,26)
TP3	1 within a branch	(32,33) $\Rightarrow$ (31,33)
TP4	2 within a branch	(11,12) $\Rightarrow$ (10,12); (29,30) $\Rightarrow$ (28,30)
TP5	2 within a branch	(15,16) $\Rightarrow$ (14,16); (33,34) $\Rightarrow$ (32,34)
TP6	1 whole branch	(10,31) $\Rightarrow$ (8,31)

Table 3: Topological Variation Scenarios or Evaluating the Generalization of GNN-based RL Algorithms on 34 Buses System.

ID	Reconfiguration	Connections
TP1	–	Baseline topology
TP2	1 within a branch	(67,68) $\Rightarrow$ (13,68)
TP3	1 within a branch	(45,46) $\Rightarrow$ (44,46)
TP4	2 within a branch	(35,36) $\Rightarrow$ (34,36); (13,69) $\Rightarrow$ (14,69)
TP5	2 within a branch	(50,51) $\Rightarrow$ (49,51); (52,53) $\Rightarrow$ (9,53)
TP6	1 whole branch	(12,13) $\Rightarrow$ (11,13)

Table 4: Topological Variation Scenarios or Evaluating the Generalization of GNN-based RL Algorithms on 69 Buses System.

Figure 5: 34 buses system RL algorithms’ operational accuracy [ $\%$ ] on topological variations

Figure 6: 69 buses system RL algorithms’ operational accuracy [ $\%$ ] on topological variations

Figure 7: 34 buses system RL algorithms’ mean $95\%$ confidence bounds of average voltage violation [p.u.] on topological variations.

Figure 8: 69 buses system RL algorithms’ mean $95\%$ confidence bounds of average voltage violation [p.u.] on topological variations.

Figure 9: RL algorithms’ (a)operational accuracy [ $\%$ ] and (b) mean $95\%$ confidence bounds of average voltage violation [p.u.] on 69 buses system with agents trained on 34 buses system

Figure 10: RL algorithms’ (a)operational accuracy [ $\%$ ] and (b) mean $95\%$ confidence bounds of average voltage violation [p.u.] on 34 buses system with agents trained on 69 buses system

Explore similar work

Topology-Aware Reinforcement Learning over Graphs for Resilient Power Distribution Networks
Mar 7, 2026Roshni Anna Jacob, Prithvi Poddar, Jaidev Goel +3Power Systems Reinforcement Learning
The University of Texas at Dallas, Richardson, TX, USA · University at Buffalo, Buffalo, NY, USA · Virginia Tech, Blacksburg, VA, USA
Knowledge Distillation for Efficient Transformer-Based Reinforcement Learning in Hardware-Constrained Energy Management Systems
Mar 27, 2026Pascal Henrich, Jonas Sievers, Maximilian Beichter +3Policy Distillation Transformer-Based RL
Karlsruhe Institute of Technology, Karlsruhe, Germany
Robustness of Spatio-temporal Graph Neural Networks for Fault Location in Partially Observable Distribution Grids
Apr 22, 2026Burak Karabulut, Carlo Manna, Chris DevelderGraph Attention Networks Graph Neural Networks
Ghent University – imec · VITO

Mar 7, 2026J/K move · Enter open · S save

Topology-Aware Reinforcement Learning over Graphs for Resilient Power Distribution Networks

Roshni Anna Jacob, Prithvi Poddar, Jaidev Goel, Souma Chowdhury+2

The University of Texas at Dallas, Richardson, TX, USA · University at Buffalo, Buffalo, NY, USA · Virginia Tech, Blacksburg, VA, USA

Power Systems Reinforcement Learning

Open paper AskPDF arXiv

Extreme weather events and cyberattacks can cause component failures and disrupt the operation of power distribution networks (DNs), during which reconfiguration and load shedding are often adopted for resilience enhancement. This study introduces a topology-aware graph reinforcement learning (RL) framework for outage management that embeds higher-order topological features of the DN into a graph-based RL model, enabling reconfiguration and load shedding to maximize energy supply while maintaining operational stability. Results on the modified IEEE 123-bus feeder across 300 diverse outage scenarios demonstrate that incorporating the topological data analysis (TDA) tool, persistence homology (PH), yields 9-18% higher cumulative rewards, up to 6% increase in power delivery, and 6-8% fewer voltage violations compared to a baseline graph-RL model. These findings highlight the potential of integrating RL with TDA to enable self-healing in DNs, facilitating fast, adaptive, and automated restoration.

Abstract

Explore similar work

Topology-Aware Reinforcement Learning over Graphs for Resilient Power Distribution Networks

Knowledge Distillation for Efficient Transformer-Based Reinforcement Learning in Hardware-Constrained Energy Management Systems

Robustness of Spatio-temporal Graph Neural Networks for Fault Location in Partially Observable Distribution Grids