Multi-agent Reinforcement Learning-based Joint Design of Low-Carbon P2P Market and Bidding Strategy in Microgrids
Authors: Junhao Ren, Honglin Gao, Sijie Wang, Lan Zhao, Qiyu Kang, Aniq Ashan, Yajuan Sun, Gaoxi Xiao
Organizations: School of Electrical and Electronic Engineering, Nanyang Technological University, 639798, Singapore · School of Information Science and Technology, University of Science and Technology of China, Hefei, 230026, China · MOYA Analytics, 051912, Singapore · Singapore Institute of Manufacturing Technology, Agency for Science, Technology and Research, 636732, Singapore
The challenges of the uncertainties in renewable energy generation and the instability of the real-time market limit the effective utilization of clean energy in microgrid communities. Existing peer-to-peer (P2P) and microgrid coordination approaches typically rely on certain centralized optimization or restrictive coordination rules which are difficult to be implemented in real-life applications. To address the challenge, we propose an intraday P2P trading framework that allows self-interested microgrids to pursue their economic benefits, while allowing the market operator to maximize the social welfare, namely the low carbon emission objective, of the entire community. Specifically, the decision-making processes of the microgrids are formulated as a Decentralized Partially Observable Markov Decision Process (DEC-POMDP) and solved using a Multi-Agent Reinforcement Learning (MARL) framework. Such an approach grants each microgrid a high degree of decision-making autonomy, while a novel market clearing mechanism is introduced to provide macro-regulation, incentivizing microgrids to prioritize local renewable energy consumption and hence reduce carbon emissions. Simulation results demonstrate that the combination of the self-interested bidding strategy and the P2P market design helps significantly improve renewable energy utilization and reduce reliance on external electricity with high carbon-emissions. The framework achieves a balanced integration of local autonomy, self-interest pursuit, and improved community-level economic and environmental benefits.
Figures & tables
Fig. 1: Structure of the power distribution network.
Fig. 2: The training process of LSTM-MAPPO algorithm for Intraday P2P market.
Fig. 3: 24 -hour Normalized Demand and PV Profiles of 4 microgrids.
Fig. 4: 24 -hour dynamical change of emergency price and FiT over times.
Parameter
Grid 1
Grid 2
Grid 3
Grid 4
Limax (kWh)
25
6
40
5
Gimax (kWh)
5
7
10
15
Emax (kWh)
8
15
15
30
TiES (kW)
4
5
8
10
−TiES (kW)
4
5
8
10
E0 (kWh)
0
2
0
20
TABLE I: Simulation Parameters for the 4 Microgrids
Fig. 5: Training performance of four market clearing mechanism with LSTM-MAPPO algorithm over 7000 episodes. Each point denotes the hourly average across the four microgrids in a simulation trajectory.
Clearing
Reward
Emergency Bought
Feed In Sold
Storage
Mechanism
($)
(kWh)
(kWh)
(kWh)
Greedy
-5.87
3.37
7.11
14.87
MRDA
-5.92
3.38
7.30
11.95
VVDA
-5.94
3.42
7.21
10.13
JPQ
-3.98
2.40
6.10
38.04
TABLE II: Average Hourly Performance Comparison Under Different Market Clearing Mechanisms
Grid
Mechanism
Reward
Emergency Bought
Feed In Sold
Storage
($)
(kWh)
(kWh)
(kWh)
Grid 1
JPQ
-2.02
0.91
0.33
2.59
Greedy
-3.30
0.63
1.56
1.84
MRDA
-2.78
1.24
0.80
0.99
VDA
-3.28
1.58
0.74
0.89
Grid 2
JPQ
0.45
0.03
0.89
8.04
TABLE III: Average Hourly Behavior of Individual Microgrids Under Different Market Clearing Mechanisms
Fig. 6: Training performance of six MARL algorithms over 7000 episodes. Each point denotes the hourly average across the four microgrids in a simulation trajectory.
Algorithm
Reward
Emergency Bought
Feed In Sold
Storage
($)
(kWh)
(kWh)
(kWh)
MAPPO
-6.26
3.49
7.19
9.59
IPPO
-6.54
3.63
7.66
3.22
MASAC
-7.81
4.22
8.26
0.10
MADDPG
-7.73
4.18
8.09
1.13
MADDPG-GCN
-7.52
4.06
7.97
1.02
TABLE IV: Average Hourly Performance Comparison of the Multi-microgrid System with Different MARL Algorithms
Uncertainties in renewable generation and demand dynamics challenge day-ahead scheduling. To enhance renewable penetration and maintain intra-day balance, we develop a multi-agent reinforcement learning framework for self-interested microgrids participating in peer-to-peer (P2P) electricity trading. Each microgrid independently bids both price and quantity while optimizing its own profit via storage arbitrage under time-varying main-grid prices. A market-clearing mechanism coordinating trades and promoting incentive compatibility is proposed. Simulation results show that the learned bidding policy improves renewable utilization and reduces reliance on high-carbon electricity, while increasing community-level economic welfare, delivering a win-win situation in emission reduction and local prosperity.
Junhao Ren, Honglin Gao, Lan Zhao +3
School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · School of Information Science and Technology, University of Science and Technology of China, Hefei, China · Singapore Institute of Manufacturing Technology, Agency for Science, Technology and Research, Singapore
The ongoing shift towards decentralization of the electric energy sector, driven by the growing electrification across end-use sectors, and widespread adoption of distributed energy resources (DERs), necessitates their active participation in the electricity markets to support grid operations. Furthermore, with bi-directional energy and communication flows becoming standard, intelligent, easy-to-deploy, resource-conservative demand-side participation is expected to play a critical role in securing power grid operational flexibility and market efficiency. This work proposes a market engagement framework that leverages a hierarchical multi-agent deep reinforcement learning (MARL) approach to enable individual prosumers to participate in peer-to-peer retail auctions and further aggregate these intelligent prosumers to facilitate effective DER participation in wholesale markets. Ultimately, a Stackelberg game is proposed to coordinate this hierarchical MARL-based DER market participation framework toward enhanced market performance.
Patrick Wilk, Ethan Cantor, Yikui Liu +1
aDepartment Electrical and Computer Engineering, Rowan University, 201 Mullica Hill Rd, Glassboro, New Jersey, 08028, United States · bSchool of Electrical Engineering, Sichuan University, 24 South Section 1, 1st Ring Road, Chengdu, 610065, China
This paper proposes implicit cooperation, a framework enabling decentralized agents to approximate optimal coordination in local energy markets without explicit peer-to-peer communication. We formulate the problem as a decentralized partially observable Markov decision problem that is solved through a multi-agent reinforcement learning task in which agents use stigmergic signals (key performance indicators at the system level) to infer and react to global states. Through a 3x3 factorial design on an IEEE 34-node topology, we evaluated three training paradigms (CTCE, CTDE, DTDE) and three algorithms (PPO, APPO, SAC). Results identify APPO-DTDE as the optimal configuration, achieving a coordination score of 91.7% relative to the theoretical centralized benchmark (CTCE). However, a critical trade-off emerges between efficiency and stability: while the centralized benchmark maximizes allocative efficiency with a peer-to-peer trade ratio of 0.6, the fully decentralized approach (DTDE) demonstrates superior physical stability. Specifically, DTDE reduces the variance of grid balance by 31% compared to hybrid architectures, establishing a highly predictable, import-biased load profile that simplifies grid regulation. Furthermore, topological analysis reveals emergent spatial clustering, where decentralized agents self-organize into stable trading communities to minimize congestion penalties. While SAC excelled in hybrid settings, it failed in decentralized environments due to entropy-driven instability. This research proves that stigmergic signaling provides sufficient context for complex grid coordination, offering a robust, privacy-preserving alternative to expensive centralized communication infrastructure.
Nelson Salazar-Pena, Alejandra Tabares, Andres Gonzalez-Mancera
Department of Mechanical Engineering, Universidad de los Andes, 111711, Bogotá D.C., Colombia · Department of Industrial Engineering, Universidad de los Andes, 111711, Bogotá D.C., Colombia