Multi-agent Reinforcement Learning-based Joint Design of Low-Carbon P2P Market and Bidding Strategy in Microgrids
Authors: Junhao Ren, Honglin Gao, Sijie Wang, Lan Zhao, Qiyu Kang, Aniq Ashan, Yajuan Sun, Gaoxi Xiao
Organizations: School of Electrical and Electronic Engineering, Nanyang Technological University, 639798, Singapore · School of Information Science and Technology, University of Science and Technology of China, Hefei, 230026, China · MOYA Analytics, 051912, Singapore · Singapore Institute of Manufacturing Technology, Agency for Science, Technology and Research, 636732, Singapore
The challenges of the uncertainties in renewable energy generation and the instability of the real-time market limit the effective utilization of clean energy in microgrid communities. Existing peer-to-peer (P2P) and microgrid coordination approaches typically rely on certain centralized optimization or restrictive coordination rules which are difficult to be implemented in real-life applications. To address the challenge, we propose an intraday P2P trading framework that allows self-interested microgrids to pursue their economic benefits, while allowing the market operator to maximize the social welfare, namely the low carbon emission objective, of the entire community. Specifically, the decision-making processes of the microgrids are formulated as a Decentralized Partially Observable Markov Decision Process (DEC-POMDP) and solved using a Multi-Agent Reinforcement Learning (MARL) framework. Such an approach grants each microgrid a high degree of decision-making autonomy, while a novel market clearing mechanism is introduced to provide macro-regulation, incentivizing microgrids to prioritize local renewable energy consumption and hence reduce carbon emissions. Simulation results demonstrate that the combination of the self-interested bidding strategy and the P2P market design helps significantly improve renewable energy utilization and reduce reliance on external electricity with high carbon-emissions. The framework achieves a balanced integration of local autonomy, self-interest pursuit, and improved community-level economic and environmental benefits.
Figures & tables
Fig. 1: Structure of the power distribution network.
Fig. 2: The training process of LSTM-MAPPO algorithm for Intraday P2P market.
Fig. 3: 24 -hour Normalized Demand and PV Profiles of 4 microgrids.
Fig. 4: 24 -hour dynamical change of emergency price and FiT over times.
Parameter
Grid 1
Grid 2
Grid 3
Grid 4
Limax (kWh)
25
6
40
5
Gimax (kWh)
5
7
10
15
Emax (kWh)
8
15
15
30
TiES (kW)
4
5
8
10
−TiES (kW)
4
5
8
10
E0 (kWh)
0
2
0
20
TABLE I: Simulation Parameters for the 4 Microgrids
Fig. 5: Training performance of four market clearing mechanism with LSTM-MAPPO algorithm over 7000 episodes. Each point denotes the hourly average across the four microgrids in a simulation trajectory.
Clearing
Reward
Emergency Bought
Feed In Sold
Storage
Mechanism
($)
(kWh)
(kWh)
(kWh)
Greedy
-5.87
3.37
7.11
14.87
MRDA
-5.92
3.38
7.30
11.95
VVDA
-5.94
3.42
7.21
10.13
JPQ
-3.98
2.40
6.10
38.04
TABLE II: Average Hourly Performance Comparison Under Different Market Clearing Mechanisms
Grid
Mechanism
Reward
Emergency Bought
Feed In Sold
Storage
($)
(kWh)
(kWh)
(kWh)
Grid 1
JPQ
-2.02
0.91
0.33
2.59
Greedy
-3.30
0.63
1.56
1.84
MRDA
-2.78
1.24
0.80
0.99
VDA
-3.28
1.58
0.74
0.89
Grid 2
JPQ
0.45
0.03
0.89
8.04
TABLE III: Average Hourly Behavior of Individual Microgrids Under Different Market Clearing Mechanisms
Fig. 6: Training performance of six MARL algorithms over 7000 episodes. Each point denotes the hourly average across the four microgrids in a simulation trajectory.
Algorithm
Reward
Emergency Bought
Feed In Sold
Storage
($)
(kWh)
(kWh)
(kWh)
MAPPO
-6.26
3.49
7.19
9.59
IPPO
-6.54
3.63
7.66
3.22
MASAC
-7.81
4.22
8.26
0.10
MADDPG
-7.73
4.18
8.09
1.13
MADDPG-GCN
-7.52
4.06
7.97
1.02
TABLE IV: Average Hourly Performance Comparison of the Multi-microgrid System with Different MARL Algorithms
School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · School of Information Science and Technology, University of Science and Technology of China, Hefei, China · Singapore Institute of Manufacturing Technology, Agency for Science, Technology and Research, Singapore
aDepartment Electrical and Computer Engineering, Rowan University, 201 Mullica Hill Rd, Glassboro, New Jersey, 08028, United States · bSchool of Electrical Engineering, Sichuan University, 24 South Section 1, 1st Ring Road, Chengdu, 610065, China
Department of Mechanical Engineering, Universidad de los Andes, 111711, Bogotá D.C., Colombia · Department of Industrial Engineering, Universidad de los Andes, 111711, Bogotá D.C., Colombia