Physics-Informed Graph-Based Safe RL with Constraint-Specific Critics for Community Battery Coordination in Distribution Network

Chenghao Huang; Yue Yang; Hao Wang

cs.AIOct 24, 2026

Physics-Informed Graph-Based Safe RL with Constraint-Specific Critics for Community Battery Coordination in Distribution Network

Authors: Chenghao Huang, Yue Yang, Hao Wang

Organizations: Department of Data Science and AI, Faculty of IT and Monash Energy Institute, Monash University, Melbourne, VIC 3800, Australia

Abstract

High penetrations of distributed energy resources (DERs) and fast-varying electric vehicle (EV) charging intensify voltage and line-limit violations in distribution networks (DNs), making safe, low-cost operation increasingly challenging. This paper proposes a physics-informed safe deep reinforcement learning (DRL) framework for coordinating community battery energy storage systems to mitigate constraint violations while minimizing operating cost. To represent nonlocal spatiotemporal interactions, we develop a dual-level physics-informed graph representation in which line impedance and capacity shape spatial information propagation, while power-balance, voltage-profile, and flow-reconstruction losses regularize the latent state toward control-relevant physical consistency. Building on this representation, we develop Lagrangian Dedicated-Critic PPO (LDCPPO), which implements constraint-specific value decomposition: independent reward and cost critics preserve heterogeneous voltage- and line-loading-related learning signals and combine their advantages only during the Lagrangian policy update. Case studies on DNs with high rooftop photovoltaic penetration and EV charging demonstrate that the proposed method reduces the key voltage-violation degree by up to 80.5% compared with the tested baselines, while also achieving the lowest voltage-violation count and line-overload degree with competitive operating cost.

Figures & tables

Ref.	Problem setting (mainly about DERs)			Methodology
	PV deployment (%)	PV penetration (%)	Residential nighttime EVC	Solution	Constraints	Feature correlation modeling	Physics-informed
[ 16 ]	27%	50%	No	Control theory	Separate	No	No
[ 17 , 18 , 19 , 20 , 21 ]	-, -, -, 40%,18%	40%,40%,70%, -, -	No	Optimization	Separate	No	No
[ 22 ]	44%	$\sim$ 50%	No	DRL	Mixed	No	No
[ 23 , 24 ]	-, 24%	90%, 30%	No	DRL	Mixed	No	No
[ 25 ]	18%	70%	No	DRL	Mixed	No	No

TABLE I: Comparison of grid-level voltage regulation methods for medium-size BESS, including business BESS and CBESS.

Fig. 1: System model of CBESS coordination via a DSO’s centralized training in a rich-DER DN.

Fig. 2: Problem-driven coupling of the dual-level PISTG representation (Stage 1) and constraint-specific LDCPPO policy optimization (Stage 2) for DSO-coordinated multi-CBESS control.

Fig. 3: Problem-driven coupling of the dual-level PISTG representation (Stage 1) and constraint-specific LDCPPO policy optimization (Stage 2) for DSO-coordinated multi-CBESS control.

Fig. 4: CBESS deployment in the IEEE 33-bus system with high DER penetration.

Panel of Figure 6 — Fig. 5: System setup of PV, EV, load, and CBESSs.

Category	Parameter	Value
PISTG encoder	PI adjacency constant, $\epsilon$	$10^{-6}$
	Capacity-weight coefficient, $\nu$	1.0
	GCN layers, $L$	2
	GCN hidden dimension	64
	Spatial embedding dim. $d^{S}$	64
	DN-TCN layers	2

TABLE II: Hyperparameter and network architecture settings.

Fig. 6: Training performance on the four key metrics of the compared methods on the IEEE 33-bus system. The curves show the mean performance over episodes with shaded regions indicating variance across multiple runs.

Method	Operating cost	Volt. vio. degree	Volt. vio. count	Line-overload degree
	AUD/day	p.u./ $(5$ min $\cdot$ bus $)$	times/ $(5$ min $\cdot$ bus $)$	%/ $(5$ min $\cdot$ line $)$
TD3	1583.7 $\pm$ 68.4	(1.63 $\pm$ 0.27) $\times 10^{-3}$	(1.02 $\pm$ 0.16) $\times 10^{-2}$	(1.93 $\pm$ 0.39) $\times 10^{-3}$
SAC	1546.2 $\pm$ 69.1	(1.52 $\pm$ 0.23) $\times 10^{-3}$	(0.94 $\pm$ 0.14) $\times 10^{-2}$	(1.82 $\pm$ 0.36) $\times 10^{-3}$
PPO	$1424.5\pm 53.7$	$(1.28\pm 0.19)\times 10^{-3}$	$(0.87\pm 0.11)\times 10^{-2}$	$(1.54\pm 0.28)\times 10^{-3}$
LDCPPO	$1453.1\pm 49.8$	$(0.77\pm 0.15)\times 10^{-3}$	$(0.79\pm 0.10)\times 10^{-2}$	$(1.32\pm 0.21)\times 10^{-3}$
STG-LDCPPO	$1414.8\pm 36.1$	$(0.53\pm 0.09)\times 10^{-3}$	$(0.63\pm 0.06)\times 10^{-2}$	$(1.15\pm 0.14)\times 10^{-3}$

TABLE III: Test performance comparison over 100 episodes for 5 runs.

Fig. 7: Bus voltage distribution in test.

Fig. 8: Line loading rate distribution in test.

Panel of Figure 12 — Fig. 9: Average daily evolution of system-level safety indicators.

Panel of Figure 13 — Fig. 10: Daily SOC trajectories of four CBESS units under PISTG-LDCPPO and PPO during testing.

Fig. 11: CBESS deployment in the IEEE 69-bus system with high DER penetration.

Fig. 12: Training performance on the four key metrics of the compared methods on IEEE 69-bus system. The curves show the mean performance over episodes with shaded regions indicating variance across multiple runs.

Method	Operating Cost (AUD/day)	Volt. Vio. Degree [p.u./(5min $\cdot$ bus)]	Volt. Vio. Count [times/(5min $\cdot$ bus)]	Line-Overload Degree [%/(5min $\cdot$ line)]
TD3	3740.8 $\pm$ 190.4	(1.98 $\pm$ 0.42) $\times 10^{-3}$	(2.28 $\pm$ 0.35) $\times 10^{-2}$	(1.85 $\pm$ 0.42) $\times 10^{-3}$
SAC	3455.6 $\pm$ 210.7	(2.12 $\pm$ 0.48) $\times 10^{-3}$	(1.91 $\pm$ 0.43) $\times 10^{-2}$	(1.92 $\pm$ 0.35) $\times 10^{-3}$
PPO	2965.2 $\pm$ 245.6	(1.60 $\pm$ 0.22) $\times 10^{-3}$	(1.22 $\pm$ 0.20) $\times 10^{-2}$	(1.76 $\pm$ 0.25) $\times 10^{-3}$
LDCPPO	3265.4 $\pm$ 260.1	(1.03 $\pm$ 0.18) $\times 10^{-3}$	(0.84 $\pm$ 0.14) $\times 10^{-2}$	(1.43 $\pm$ 0.15) $\times 10^{-3}$
STG-LDCPPO	3135.5 $\pm$ 95.4	(0.82 $\pm$ 0.15) $\times 10^{-3}$	(0.76 $\pm$ 0.18) $\times 10^{-2}$	(1.22 $\pm$ 0.12) $\times 10^{-3}$
PI-GR-STG-LDCPPO	3122.1 $\pm$ 172.7	(0.67 $\pm$ 0.12) $\times 10^{-3}$	(0.67 $\pm$ 0.11) $\times 10^{-2}$	(1.23 $\pm$ 0.14) $\times 10^{-3}$

TABLE IV: Test performance comparison over 100 episodes for 5 runs on IEEE 69-bus system.

Method	33-bus		69-bus
Method	Time	Params	Time	Params
	(ms)	(M)	(ms)	(M)
PPO	0.34	0.10	0.42	0.12
LDCPPO	0.49	0.24	0.58	0.28
STG-LDCPPO	1.18	0.47	1.74	0.53
PISTG-LDCPPO	1.22	0.47	1.78	0.53

TABLE V: Inference time and parameter count on IEEE 33-bus and 69-bus systems.

Method	Case	$E^{\mathrm{curt}}$ (kWh/day)	$\rho^{\mathrm{curt}}$ (%)	$\eta^{\mathrm{HP}}$ (%)
TD3	33	$142.6{\pm}19.8$	$3.57{\pm}0.50$	$56.8$
TD3	69	$386.4{\pm}42.7$	$4.55{\pm}0.50$	$55.3$
SAC	33	$134.1{\pm}18.5$	$3.35{\pm}0.46$	$59.4$
SAC	69	$338.7{\pm}39.6$	$3.98{\pm}0.47$	$59.1$
PPO	33	$111.3{\pm}15.6$	$2.78{\pm}0.39$	$63.5$
PPO	69	$226.9{\pm}28.4$	$2.67{\pm}0.33$	$65.8$

TABLE VI: PV curtailment statistics during testing.

Encoder variant	Volt. MAE	Flow	Norm. KCL	Norm. V-drop
	$\times 10^{-3}$ p.u.	NRMSE %	residual $\times 10^{-2}$	residual $\times 10^{-3}$
STG encoder	7.41	6.82	4.86	7.53
PISTG w/o PI graph	5.12	5.41	2.67	4.96
PISTG w/o KCL	4.62	4.83	3.78	3.81
PISTG w/o VDrop	5.89	5.16	2.18	6.45
Full PISTG	3.68	4.12	1.54	2.91

TABLE VII: Ablation of PISTG encoder pre-training on the IEEE 33-bus test windows.

Method	Operating cost	Volt. vio. degree	Volt. vio. count	Line-overload degree
	AUD/day	p.u./ $(5$ min $\cdot$ bus $)$	times/ $(5$ min $\cdot$ bus $)$	%/ $(5$ min $\cdot$ line $)$
STG-LDCPPO	$1414.8\pm 36.1$	$(0.53\pm 0.09)\times 10^{-3}$	$(0.63\pm 0.06)\times 10^{-2}$	$(1.15\pm 0.14)\times 10^{-3}$
w/o PI graph	$1396.7\pm 37.2$	$(0.34\pm 0.05)\times 10^{-3}$	$(0.46\pm 0.05)\times 10^{-2}$	$(1.14\pm 0.13)\times 10^{-3}$
w/o KCL	$1401.2\pm 38.5$	$(0.38\pm 0.06)\times 10^{-3}$	$(0.50\pm 0.05)\times 10^{-2}$	$(1.14\pm 0.14)\times 10^{-3}$
w/o VDrop	$1405.8\pm 37.9$	$(0.42\pm 0.07)\times 10^{-3}$	$(0.55\pm 0.06)\times 10^{-2}$	$(1.15\pm 0.15)\times 10^{-3}$
Full	$\textbf{1382.4}\pm\textbf{38.9}$	$\textbf{(0.25}\pm\textbf{0.04)}\times 10^{-3}$	$\textbf{(0.37}\pm\textbf{0.05)}\times 10^{-2}$	$\textbf{(1.13}\pm\textbf{0.14)}\times 10^{-3}$

TABLE VIII: Impact of individual physics-informed ingredients on downstream LDCPPO testing performance.

Panel of Figure 21 — Fig. 13: Training-curve comparison between PISTG-LSCPPO and PISTG-LDCPPO.

System	Method	Operating cost	Volt. vio. degree	Volt. vio. count	Line-overload degree
		AUD/day	p.u./ $(5$ min $\cdot$ bus $)$	times/ $(5$ min $\cdot$ bus $)$	%/ $(5$ min $\cdot$ line $)$
33-bus	PISTG-LSCPPO	$1408.6\pm 41.5$	$(0.52\pm 0.08)\times 10^{-3}$	$(0.58\pm 0.07)\times 10^{-2}$	$(1.20\pm 0.15)\times 10^{-3}$
33-bus	PISTG-LDCPPO	$\textbf{1382.4}\pm\textbf{38.9}$	$\textbf{(0.25}\pm\textbf{0.04)}\times 10^{-3}$	$\textbf{(0.37}\pm\textbf{0.05)}\times 10^{-2}$	$\textbf{(1.13}\pm\textbf{0.14)}\times 10^{-3}$
69-bus	PISTG-LSCPPO	$3118.4\pm 148.7$	$(0.88\pm 0.14)\times 10^{-3}$	$(0.79\pm 0.12)\times 10^{-2}$	$(1.28\pm 0.16)\times 10^{-3}$
69-bus	PISTG-LDCPPO	$\textbf{3060.7}\pm\textbf{135.2}$	$\textbf{(0.52}\pm\textbf{0.09)}\times 10^{-3}$	$\textbf{(0.60}\pm\textbf{0.10)}\times 10^{-2}$	$\textbf{(1.15}\pm\textbf{0.14)}\times 10^{-3}$

TABLE IX: Comparison between PISTG-LSCPPO and PISTG-LDCPPO over 100 test episodes for 5 runs.

Fig. 14: Sensitivity analysis of different PIAM edge-weight constructions.

Fig. 15: Sensitivity analysis of the PIAM capacity-weight coefficient

\nu

Encoder	Volt. vio. degree		Volt. vio. count		Line-overload degree
Encoder	$\rho$	$R^{2}$	$\rho$	$R^{2}$	$\rho$	$R^{2}$
STG encoder	0.71	0.49	0.67	0.44	0.54	0.29
PISTG encoder	0.89	0.78	0.84	0.69	0.58	0.32

TABLE X: Physical correlation analysis of learned representations.

Power Systems

Power systems research studies the planning, operation, control, and protection of electrical grids and their integration with energy resources.

See all 231 papers

Safe Decentralized Operation of EV Virtual Power Plant with Limited Network Visibility via Multi-Agent Reinforcement Learning
Mar 24, 2026Chenghao Huang, Jiarong Fan, Weiqing Wang +1Power Systems Decentralized MARL
Department of Data Science and AI, Faculty of IT, Monash University, Australia · Monash Energy Institute, Monash University, Australia
Topology-Aware Graph Reinforcement Learning for Energy Storage Systems Optimal Dispatch in Distribution Networks
Mar 27, 2026Shuyi Gao, Stavros Orfanoudakis, Shengren Hou +2Power Systems RL Control
Department of Electrical Sustainable Energy, Delft University of Technology, Mekelweg 4, Delft, 2628 CD, The Netherlands · Beijing Energy Quant Ltd., Haidian, Beijing, 53100, China
HetGPS: Scalable Graph Multi-Agent Reinforcement Learning with Physics-Anchored Adaptive Safety for EV Charging
Aug 1, 2026Xiangwei Wang, Nanduni Nimalsiri, Yu Xia +2Electric Vehicle Charging Safe RL
The University of Melbourne, Australia · CSIRO, Australia · Shanghai Jiao Tong University, China

Mar 24, 2026J/K move · Enter open · S save

Safe Decentralized Operation of EV Virtual Power Plant with Limited Network Visibility via Multi-Agent Reinforcement Learning

Chenghao Huang, Jiarong Fan, Weiqing Wang, Hao Wang

Department of Data Science and AI, Faculty of IT, Monash University, Australia · Monash Energy Institute, Monash University, Australia

Power Systems Decentralized MARL Electric Vehicle Charging Safe RL Partially Observable RL

Open paper AskPDF arXiv

As power systems advance toward net-zero targets, behind-the-meter renewables are driving rapid growth in distributed energy resources (DERs). Virtual power plants (VPPs) increasingly coordinate these resources to support power distribution network (PDN) operation, with EV charging stations (EVCSs) emerging as a key asset due to their strong impact on local voltages. However, in practice, VPPs must make operational decisions with only partial visibility of PDN states, relying on limited, aggregated information shared by the distribution system operator. This work proposes a safety-enhanced VPP framework for coordinating multiple EVCSs under such realistic information constraints to ensure voltage security while maintaining economic operation. We develop Transformer-assisted Lagrangian Multi-Agent Proximal Policy Optimization (TL-MAPPO), in which EVCS agents learn decentralized charging policies via centralized training with Lagrangian regularization to enforce voltage and demand-satisfaction constraints. A transformer-based embedding layer deployed on each EVCS agent captures temporal correlations among prices, loads, and charging demand to improve decision quality. Experiments on a realistic 33-bus PDN show that the proposed framework reduces voltage violations by approximately 45% and operational costs by approximately 10% compared to representative multi-agent DRL baselines, highlighting its potential for practical VPP deployment.

Abstract

Power Systems

Explore similar work

Safe Decentralized Operation of EV Virtual Power Plant with Limited Network Visibility via Multi-Agent Reinforcement Learning

Topology-Aware Graph Reinforcement Learning for Energy Storage Systems Optimal Dispatch in Distribution Networks

HetGPS: Scalable Graph Multi-Agent Reinforcement Learning with Physics-Anchored Adaptive Safety for EV Charging