High penetrations of distributed energy resources (DERs) and fast-varying electric vehicle (EV) charging intensify voltage and line-limit violations in distribution networks (DNs), making safe, low-cost operation increasingly challenging. This paper proposes a physics-informed safe deep reinforcement learning (DRL) framework for coordinating community battery energy storage systems to mitigate constraint violations while minimizing operating cost. To represent nonlocal spatiotemporal interactions, we develop a dual-level physics-informed graph representation in which line impedance and capacity shape spatial information propagation, while power-balance, voltage-profile, and flow-reconstruction losses regularize the latent state toward control-relevant physical consistency. Building on this representation, we develop Lagrangian Dedicated-Critic PPO (LDCPPO), which implements constraint-specific value decomposition: independent reward and cost critics preserve heterogeneous voltage- and line-loading-related learning signals and combine their advantages only during the Lagrangian policy update. Case studies on DNs with high rooftop photovoltaic penetration and EV charging demonstrate that the proposed method reduces the key voltage-violation degree by up to 80.5% compared with the tested baselines, while also achieving the lowest voltage-violation count and line-overload degree with competitive operating cost.
Figures & tables
Ref.
Problem setting (mainly about DERs)
Methodology
PV deployment (%)
PV penetration (%)
Residential nighttime EVC
Solution
Constraints
Feature correlation modeling
Physics-informed
[ 16 ]
27%
50%
No
Control theory
Separate
No
No
[ 17 , 18 , 19 , 20 , 21 ]
-, -, -, 40%,18%
40%,40%,70%, -, -
No
Optimization
Separate
No
No
[ 22 ]
44%
∼ 50%
No
DRL
Mixed
No
No
[ 23 , 24 ]
-, 24%
90%, 30%
No
DRL
Mixed
No
No
[ 25 ]
18%
70%
No
DRL
Mixed
No
No
TABLE I: Comparison of grid-level voltage regulation methods for medium-size BESS, including business BESS and CBESS.
Fig. 1: System model of CBESS coordination via a DSO’s centralized training in a rich-DER DN.
Fig. 2: Problem-driven coupling of the dual-level PISTG representation (Stage 1) and constraint-specific LDCPPO policy optimization (Stage 2) for DSO-coordinated multi-CBESS control.
Fig. 3: Problem-driven coupling of the dual-level PISTG representation (Stage 1) and constraint-specific LDCPPO policy optimization (Stage 2) for DSO-coordinated multi-CBESS control.
Fig. 4: CBESS deployment in the IEEE 33-bus system with high DER penetration.
Fig. 5: System setup of PV, EV, load, and CBESSs.
Category
Parameter
Value
PISTG encoder
PI adjacency constant, ϵ
10−6
Capacity-weight coefficient, ν
1.0
GCN layers, L
2
GCN hidden dimension
64
Spatial embedding dim. dS
64
DN-TCN layers
2
TABLE II: Hyperparameter and network architecture settings.
Fig. 6: Training performance on the four key metrics of the compared methods on the IEEE 33-bus system. The curves show the mean performance over episodes with shaded regions indicating variance across multiple runs.
Method
Operating cost
Volt. vio. degree
Volt. vio. count
Line-overload degree
AUD/day
p.u./ (5 min ⋅ bus )
times/ (5 min ⋅ bus )
%/ (5 min ⋅ line )
TD3
1583.7 ± 68.4
(1.63 ± 0.27) ×10−3
(1.02 ± 0.16) ×10−2
(1.93 ± 0.39) ×10−3
SAC
1546.2 ± 69.1
(1.52 ± 0.23) ×10−3
(0.94 ± 0.14) ×10−2
(1.82 ± 0.36) ×10−3
PPO
1424.5±53.7
(1.28±0.19)×10−3
(0.87±0.11)×10−2
(1.54±0.28)×10−3
LDCPPO
1453.1±49.8
(0.77±0.15)×10−3
(0.79±0.10)×10−2
(1.32±0.21)×10−3
STG-LDCPPO
1414.8±36.1
(0.53±0.09)×10−3
(0.63±0.06)×10−2
(1.15±0.14)×10−3
TABLE III: Test performance comparison over 100 episodes for 5 runs.
Fig. 7: Bus voltage distribution in test.
Fig. 8: Line loading rate distribution in test.
Fig. 9: Average daily evolution of system-level safety indicators.
Fig. 10: Daily SOC trajectories of four CBESS units under PISTG-LDCPPO and PPO during testing.
Fig. 11: CBESS deployment in the IEEE 69-bus system with high DER penetration.
Fig. 12: Training performance on the four key metrics of the compared methods on IEEE 69-bus system. The curves show the mean performance over episodes with shaded regions indicating variance across multiple runs.
Method
Operating Cost (AUD/day)
Volt. Vio. Degree [p.u./(5min ⋅ bus)]
Volt. Vio. Count [times/(5min ⋅ bus)]
Line-Overload Degree [%/(5min ⋅ line)]
TD3
3740.8 ± 190.4
(1.98 ± 0.42) ×10−3
(2.28 ± 0.35) ×10−2
(1.85 ± 0.42) ×10−3
SAC
3455.6 ± 210.7
(2.12 ± 0.48) ×10−3
(1.91 ± 0.43) ×10−2
(1.92 ± 0.35) ×10−3
PPO
2965.2 ± 245.6
(1.60 ± 0.22) ×10−3
(1.22 ± 0.20) ×10−2
(1.76 ± 0.25) ×10−3
LDCPPO
3265.4 ± 260.1
(1.03 ± 0.18) ×10−3
(0.84 ± 0.14) ×10−2
(1.43 ± 0.15) ×10−3
STG-LDCPPO
3135.5 ± 95.4
(0.82 ± 0.15) ×10−3
(0.76 ± 0.18) ×10−2
(1.22 ± 0.12) ×10−3
PI-GR-STG-LDCPPO
3122.1 ± 172.7
(0.67 ± 0.12) ×10−3
(0.67 ± 0.11) ×10−2
(1.23 ± 0.14) ×10−3
TABLE IV: Test performance comparison over 100 episodes for 5 runs on IEEE 69-bus system.
Method
33-bus
69-bus
Time
Params
Time
Params
(ms)
(M)
(ms)
(M)
PPO
0.34
0.10
0.42
0.12
LDCPPO
0.49
0.24
0.58
0.28
STG-LDCPPO
1.18
0.47
1.74
0.53
PISTG-LDCPPO
1.22
0.47
1.78
0.53
TABLE V: Inference time and parameter count on IEEE 33-bus and 69-bus systems.
Method
Case
Ecurt (kWh/day)
ρcurt (%)
ηHP (%)
TD3
33
142.6±19.8
3.57±0.50
56.8
69
386.4±42.7
4.55±0.50
55.3
SAC
33
134.1±18.5
3.35±0.46
59.4
69
338.7±39.6
3.98±0.47
59.1
PPO
33
111.3±15.6
2.78±0.39
63.5
69
226.9±28.4
2.67±0.33
65.8
TABLE VI: PV curtailment statistics during testing.
Encoder variant
Volt. MAE
Flow
Norm. KCL
Norm. V-drop
×10−3 p.u.
NRMSE %
residual ×10−2
residual ×10−3
STG encoder
7.41
6.82
4.86
7.53
PISTG w/o PI graph
5.12
5.41
2.67
4.96
PISTG w/o KCL
4.62
4.83
3.78
3.81
PISTG w/o VDrop
5.89
5.16
2.18
6.45
Full PISTG
3.68
4.12
1.54
2.91
TABLE VII: Ablation of PISTG encoder pre-training on the IEEE 33-bus test windows.
Method
Operating cost
Volt. vio. degree
Volt. vio. count
Line-overload degree
AUD/day
p.u./ (5 min ⋅ bus )
times/ (5 min ⋅ bus )
%/ (5 min ⋅ line )
STG-LDCPPO
1414.8±36.1
(0.53±0.09)×10−3
(0.63±0.06)×10−2
(1.15±0.14)×10−3
w/o PI graph
1396.7±37.2
(0.34±0.05)×10−3
(0.46±0.05)×10−2
(1.14±0.13)×10−3
w/o KCL
1401.2±38.5
(0.38±0.06)×10−3
(0.50±0.05)×10−2
(1.14±0.14)×10−3
w/o VDrop
1405.8±37.9
(0.42±0.07)×10−3
(0.55±0.06)×10−2
(1.15±0.15)×10−3
Full
1382.4±38.9
(0.25±0.04)×10−3
(0.37±0.05)×10−2
(1.13±0.14)×10−3
TABLE VIII: Impact of individual physics-informed ingredients on downstream LDCPPO testing performance.
Fig. 13: Training-curve comparison between PISTG-LSCPPO and PISTG-LDCPPO.
System
Method
Operating cost
Volt. vio. degree
Volt. vio. count
Line-overload degree
AUD/day
p.u./ (5 min ⋅ bus )
times/ (5 min ⋅ bus )
%/ (5 min ⋅ line )
33-bus
PISTG-LSCPPO
1408.6±41.5
(0.52±0.08)×10−3
(0.58±0.07)×10−2
(1.20±0.15)×10−3
33-bus
PISTG-LDCPPO
1382.4±38.9
(0.25±0.04)×10−3
(0.37±0.05)×10−2
(1.13±0.14)×10−3
69-bus
PISTG-LSCPPO
3118.4±148.7
(0.88±0.14)×10−3
(0.79±0.12)×10−2
(1.28±0.16)×10−3
69-bus
PISTG-LDCPPO
3060.7±135.2
(0.52±0.09)×10−3
(0.60±0.10)×10−2
(1.15±0.14)×10−3
TABLE IX: Comparison between PISTG-LSCPPO and PISTG-LDCPPO over 100 test episodes for 5 runs.
Fig. 14: Sensitivity analysis of different PIAM edge-weight constructions.
Fig. 15: Sensitivity analysis of the PIAM capacity-weight coefficient ν .
Encoder
Volt. vio. degree
Volt. vio. count
Line-overload degree
ρ
R2
ρ
R2
ρ
R2
STG encoder
0.71
0.49
0.67
0.44
0.54
0.29
PISTG encoder
0.89
0.78
0.84
0.69
0.58
0.32
TABLE X: Physical correlation analysis of learned representations.
As power systems advance toward net-zero targets, behind-the-meter renewables are driving rapid growth in distributed energy resources (DERs). Virtual power plants (VPPs) increasingly coordinate these resources to support power distribution network (PDN) operation, with EV charging stations (EVCSs) emerging as a key asset due to their strong impact on local voltages. However, in practice, VPPs must make operational decisions with only partial visibility of PDN states, relying on limited, aggregated information shared by the distribution system operator. This work proposes a safety-enhanced VPP framework for coordinating multiple EVCSs under such realistic information constraints to ensure voltage security while maintaining economic operation. We develop Transformer-assisted Lagrangian Multi-Agent Proximal Policy Optimization (TL-MAPPO), in which EVCS agents learn decentralized charging policies via centralized training with Lagrangian regularization to enforce voltage and demand-satisfaction constraints. A transformer-based embedding layer deployed on each EVCS agent captures temporal correlations among prices, loads, and charging demand to improve decision quality. Experiments on a realistic 33-bus PDN show that the proposed framework reduces voltage violations by approximately 45% and operational costs by approximately 10% compared to representative multi-agent DRL baselines, highlighting its potential for practical VPP deployment.
Chenghao Huang, Jiarong Fan, Weiqing Wang +1
Department of Data Science and AI, Faculty of IT, Monash University, Australia · Monash Energy Institute, Monash University, Australia
Optimal dispatch of energy storage systems (ESSs) in distribution networks involves jointly improving operating economy and voltage security under time-varying conditions and possible topology changes. To support fast online decision making, we develop a topology-aware Reinforcement Learning architecture based on Twin Delayed Deep Deterministic Policy Gradient (TD3), which integrates graph neural networks (GNNs) as graph feature encoders for ESS dispatch. We conduct a systematic investigation of three GNN variants: graph convolutional networks (GCNs), topology adaptive graph convolutional networks (TAGConv), and graph attention networks (GATs) on the 34-bus and 69-bus systems, and evaluate robustness under multiple topology reconfiguration cases as well as cross-system transfer between networks with different system sizes. Results show that GNN-based controllers consistently reduce the number and magnitude of voltage violations, with clearer benefits on the 69-bus system and under reconfiguration; on the 69-bus system, TD3-GCN and TD3-TAGConv also achieve lower saved cost relative to the NLP benchmark than the NN baseline. We also highlight that transfer gains are case-dependent, and zero-shot transfer between fundamentally different systems results in notable performance degradation and increased voltage magnitude violations. This work is available at: https://github.com/ShuyiGao/GNNs_RL_ESSs and https://github.com/distributionnetworksTUDelft/GNNs_RL_ESSs.
Shuyi Gao, Stavros Orfanoudakis, Shengren Hou +2
Department of Electrical Sustainable Energy, Delft University of Technology, Mekelweg 4, Delft, 2628 CD, The Netherlands · Beijing Energy Quant Ltd., Haidian, Beijing, 53100, China
Safety interventions for large populations of network-coupled agents must protect shared constraints without unnecessarily overriding task-oriented policy decisions. We present HetGPS, a hybrid graph-control framework synergizing learned graph risk with physics-anchored correction by separating intervention magnitude from corrective direction. An action-conditioned graph residual model schedules state-dependent intervention authority, while a physics model determines its direction. For electric vehicle (EV) charging, we couple this filter with a parameter-shared heterogeneous graph soft actor-critic policy, enabling topology-aware coordination with a learned model size independent of fleet size. Across five nested distribution networks with 200--3,218 EVs and 100 evaluation days, Adaptive Authority reduces bus--step voltage violations from 3.93--7.74% without filtering to 0.52--3.44%, while maintaining 99.06--100% departure success. Relative to the same physics-directed projection with fixed authority, it improves mean reward on all five networks and lowers the mean safety score on four. The deployed policy-and-risk model contains 383,702 learned parameters at every scale; at 3,218 EVs, a matched centralized SAC actor is about 170× larger. A policy trained on the eight-transformer system transfers zero-shot to the 16- and 32-transformer systems, attaining 0.57--0.75% violation rates and at least 99.99% departure success. These results show that learned graph risk can allocate intervention authority at scale while feeder physics anchors corrective action.
Xiangwei Wang, Nanduni Nimalsiri, Yu Xia +2
The University of Melbourne, Australia · CSIRO, Australia · Shanghai Jiao Tong University, China