Multi-agent Reinforcement Learning for Low-Carbon P2P Energy Trading among Self-Interested Microgrids

Junhao Ren; Honglin Gao; Lan Zhao; Qiyu Kang; Gaoxi Xiao; Yajuan Sun

cs.MAApr 10, 2026

Multi-agent Reinforcement Learning for Low-Carbon P2P Energy Trading among Self-Interested Microgrids

Authors: Junhao Ren, Honglin Gao, Lan Zhao, Qiyu Kang, Gaoxi Xiao, Yajuan Sun

Organizations: School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · School of Information Science and Technology, University of Science and Technology of China, Hefei, China · Singapore Institute of Manufacturing Technology, Agency for Science, Technology and Research, Singapore

Abstract

Uncertainties in renewable generation and demand dynamics challenge day-ahead scheduling. To enhance renewable penetration and maintain intra-day balance, we develop a multi-agent reinforcement learning framework for self-interested microgrids participating in peer-to-peer (P2P) electricity trading. Each microgrid independently bids both price and quantity while optimizing its own profit via storage arbitrage under time-varying main-grid prices. A market-clearing mechanism coordinating trades and promoting incentive compatibility is proposed. Simulation results show that the learned bidding policy improves renewable utilization and reduces reliance on high-carbon electricity, while increasing community-level economic welfare, delivering a win-win situation in emission reduction and local prosperity.

Figures & tables

Fig. 1: The two-stage electricity market framework.

Parameter	Grid 1	Grid 2	Grid 3	Grid 4
$L_{i}^{\text{max}}$ (kWh)	25	6	40	5
$G_{i}^{\text{max}}$ (kWh)	5	7	10	15
$E_{\text{max}}$ (kWh)	8	15	15	30
$\overline{T}^{\text{ES}}_{i}$ (kW)	4	5	8	10
$-\underline{T}^{\text{ES}}_{i}$ (kW)	4	5	8	10
$E_{0}$ (kWh)	0	2	0	20

TABLE I: Simulation Parameters for the 4 Microgrids

Fig. 2: Training curves of different MARL algorithm $+$ MRDA.

Metric	MRDA	VDA	Greedy
Total Profit ($)	$-123.81$	$-185.53$	$-221.02$
Emergency Purchase (kWh)	$30.33$	$34.00$	$34.37$
Total FIT (kWh)	$18.95$	$32.24$	$43.74$
P2P Trade Volume (kWh)	$10.58$	$9.96$	$9.55$
Average SoC (kWh)	$16.56$	$16.69$	$16.60$

TABLE II: Daily Performance Comparison of MMAPPO under Different Market Mechanisms

Power Systems

Power systems research studies the planning, operation, control, and protection of electrical grids and their integration with energy resources.

See all 231 papers

Multi-agent Reinforcement Learning-based Joint Design of Low-Carbon P2P Market and Bidding Strategy in Microgrids
Apr 3, 2026Junhao Ren, Honglin Gao, Sijie Wang +5Power Systems
School of Electrical and Electronic Engineering, Nanyang Technological University, 639798, Singapore · School of Information Science and Technology, University of Science and Technology of China, Hefei, 230026, China · MOYA Analytics, 051912, Singapore +1
MARS-DA: A Hierarchical Reinforcement Learning Framework for Risk-Aware Multi-Agent Bidding in Power Grids
May 4, 2026Jiayi Chen, Xuan Zhang, Guiling WangHierarchical RL Risk-Sensitive RL
Department of Computer Science, New Jersey Institute of Technology
Reinforcement Learning and Rule-Based Peer-to-Peer Pricing in Residential PV-BES Communities
Sep 1, 2026Pablo Benalcazar, Maciej Kalka, Wilian Guamán +1Reinforcement Learning Dynamic Pricing
Division of Energy Economics, Mineral and Energy Economy Research Institute, Polish Academy of Sciences, Kraków, Poland · GITEA, Escuela Superior Politécnica de Chimborazo (ESPOCH), Riobamba, Ecuador

Apr 3, 2026J/K move · Enter open · S save

Multi-agent Reinforcement Learning-based Joint Design of Low-Carbon P2P Market and Bidding Strategy in Microgrids

Junhao Ren, Honglin Gao, Sijie Wang, Lan Zhao+4

School of Electrical and Electronic Engineering, Nanyang Technological University, 639798, Singapore · School of Information Science and Technology, University of Science and Technology of China, Hefei, 230026, China · MOYA Analytics, 051912, Singapore+1

Power Systems

Open paper AskPDF arXiv

The challenges of the uncertainties in renewable energy generation and the instability of the real-time market limit the effective utilization of clean energy in microgrid communities. Existing peer-to-peer (P2P) and microgrid coordination approaches typically rely on certain centralized optimization or restrictive coordination rules which are difficult to be implemented in real-life applications. To address the challenge, we propose an intraday P2P trading framework that allows self-interested microgrids to pursue their economic benefits, while allowing the market operator to maximize the social welfare, namely the low carbon emission objective, of the entire community. Specifically, the decision-making processes of the microgrids are formulated as a Decentralized Partially Observable Markov Decision Process (DEC-POMDP) and solved using a Multi-Agent Reinforcement Learning (MARL) framework. Such an approach grants each microgrid a high degree of decision-making autonomy, while a novel market clearing mechanism is introduced to provide macro-regulation, incentivizing microgrids to prioritize local renewable energy consumption and hence reduce carbon emissions. Simulation results demonstrate that the combination of the self-interested bidding strategy and the P2P market design helps significantly improve renewable energy utilization and reduce reliance on external electricity with high carbon-emissions. The framework achieves a balanced integration of local autonomy, self-interest pursuit, and improved community-level economic and environmental benefits.

Abstract

Power Systems

Explore similar work

Multi-agent Reinforcement Learning-based Joint Design of Low-Carbon P2P Market and Bidding Strategy in Microgrids

MARS-DA: A Hierarchical Reinforcement Learning Framework for Risk-Aware Multi-Agent Bidding in Power Grids

Reinforcement Learning and Rule-Based Peer-to-Peer Pricing in Residential PV-BES Communities