eess.SYFeb 17, 2026

Harnessing Implicit Cooperation: A Multi-Agent Reinforcement Learning Approach Towards Decentralized Local Energy Markets

Authors: Nelson Salazar-Pena, Alejandra Tabares, Andres Gonzalez-Mancera

Organizations: Department of Mechanical Engineering, Universidad de los Andes, 111711, Bogotá D.C., Colombia · Department of Industrial Engineering, Universidad de los Andes, 111711, Bogotá D.C., Colombia

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

This paper proposes implicit cooperation, a framework enabling decentralized agents to approximate optimal coordination in local energy markets without explicit peer-to-peer communication. We formulate the problem as a decentralized partially observable Markov decision problem that is solved through a multi-agent reinforcement learning task in which agents use stigmergic signals (key performance indicators at the system level) to infer and react to global states. Through a 3x3 factorial design on an IEEE 34-node topology, we evaluated three training paradigms (CTCE, CTDE, DTDE) and three algorithms (PPO, APPO, SAC). Results identify APPO-DTDE as the optimal configuration, achieving a coordination score of 91.7% relative to the theoretical centralized benchmark (CTCE). However, a critical trade-off emerges between efficiency and stability: while the centralized benchmark maximizes allocative efficiency with a peer-to-peer trade ratio of 0.6, the fully decentralized approach (DTDE) demonstrates superior physical stability. Specifically, DTDE reduces the variance of grid balance by 31% compared to hybrid architectures, establishing a highly predictable, import-biased load profile that simplifies grid regulation. Furthermore, topological analysis reveals emergent spatial clustering, where decentralized agents self-organize into stable trading communities to minimize congestion penalties. While SAC excelled in hybrid settings, it failed in decentralized environments due to entropy-driven instability. This research proves that stigmergic signaling provides sufficient context for complex grid coordination, offering a robust, privacy-preserving alternative to expensive centralized communication infrastructure.

Figures & tables

ParadigmControllerCoordination SignalKey AdvantageKey DisadvantageScalabilityResilience
Centralized controlSingle central controllerDirect command and control signalsGlobal optimization, high controllabilitySingle point of failure, high computational burdenLowLow
Decentralized controlIndividual agentsLocal measurements (voltage, frequency)No single point of failure, plug-and-play capabilitySuboptimal global outcomes, system imbalanceHighHigh
Transactive energyIndividual agentsEconomic signals (prices, bids, asks)Fosters prosumer autonomy, economic efficiencyCommunication infrastructure, potential for market power abuseHighHigh
Table 1: Comparative analysis of control paradigms for distributed energy resources (DERs).
FeatureCTCECTDEDTDERef
Architecture1-to-NN-to-N (centralized critic)N-to-N (independent)[ 85 , 74 , 38 ]
Information (Training)Global stateGlobal state (centralized critic)Local observations[ 117 , 18 ]
Information (Execution)Global stateLocal observationsLocal observations[ 18 ]
Coordination mechanismCentralized optimizerCentralized criticIndependent learners[ 117 , 18 ]
StrengthOptimalityPerformanceScalability[ 117 , 18 ]
WeaknessCurse of dimensionalityCurse of dimensionalityNon-stationarity[ 83 , 74 , 38 ]
Table 2: Technical taxonomy of MARL training paradigms in LEMs.
FeatureOptimization ModelsGame Theoretic ModelsMARL
ObjectiveSystem-optimal (e.g., global welfare)Nash equilibriumAdaptive profit maximization
Uncertainty handlingRobust or stochasticDeterministicBelief state (experience-based)
Market dynamicsStatic with perfect competitionStatic equilibriumNon-stationary and emergent
ScalabilityHigh (linear/convex)Low (combinatorial)High (observations and reward shaping)
LimitationRequires perfect foresightComputational intractabilityNon-stationarity and training stability
Table 3: Comparative analysis of market modeling and control methods in LEMs.
FeaturePPO / APPOSACRef
Policy typeOn-policyOff-policy[ 102 , 44 ]
Key featureClip functionMax entropy[ 66 ]
Trade-offStability / Sample efficiencySample efficiency / Stale data-
Sample efficiencyLow (discards data after update)High (re-uses data via replay buffer)[ 102 ]
Update mechanismClipped surrogate objectiveActor-critic with entropy regularization[ 102 ]
Data sensitivityRobust to stale dataProne to stale data issues[ 107 ]
Table 4: Comparative analysis of PPO/APPO and SAC for continuous control in LEMS.
Market Signals Agent-Specific Signals Implicit Cooperation KPIs
1. Current step 2. Time of day 3. Clearing price 4. Clearing volume 5. Grid balance 6. DSO buy volume 7. DSO sell volume 8. DSO total volume9. P2P volume 10. DSO trade ratio 11. Net grid import 12. DSO buy price 13. DSO sell price 14. Mean local price 15. Price spread 16. Local price advantage 1. Energy generation 2. Energy demand 3. Cumulative demand satisfied 4. Cumulative demand deferred 5. Remaining demand 6. Cummulative supply satisfied 7. Cummulative supply deferred 8. Remaining supply9. Mean profit 10. Reputation 11. Battery energy level 12. Battery SoC 13. Battery available charge 14. Battery available discharge 15. Battery cummulative charge 16. Battery cummulative discharge 1. Social welfare 2. Market liquidity 3. Mean bid-ask spread 4. Price volatility 5. Supply-demand imbalance6. Grid congestion 7. Coordination score 8. Coordination convergence 9. DER self-consumption 10. Flexibility utilization
Table 5: Observation space vector for the formulation of the LEM as a Dec-POMDP.
AgentIEEE-34 NodeCapacity (kW)Battery (kW)Battery RatioDescription
Small Industry840180.0108.00.6Continuous demand with morning peak; relies on stored energy
Community Hospital890170.085.00.5High morning demand peak, transitioning to afternoon generation surplus
University Campus844140.098.00.7Bi-modal demand (morning/afternoon peaks); moderate midday generation
Shopping Mall816130.078.00.6High afternoon/evening demand; generation sink for midday surplus
Residential Complex80080.064.00.8Dual-peak demand (morning/evening); reliant on storage for evening ramp
Apartment Building83075.052.50.7Similar to residential complex; moderate generation capability
Table 6: Configuration of the agents for the case study.
Figure 1: Generation and demand profiles for DER agents in the case study.
Figure 2: The 34-node IEEE test feeder as the grid network for the case study.
Figure 3: Feed-in tariff and utility price profiles for DSO agent in the case study.
ParameterValueDescription
Max steps24Episode length representing a full 24-hour cycle (1 hour/step)
Grid topologyIEEE 34-BusStandard distribution network topology
Grid capacity1800 kWTotal grid capacity
Seed42Fixed random seed for reproducibility
Price range20 - 600.0Allowable bid range, designed to be wider than DSO prices
Quantity range0 - 180 kWhAllowable trade size per order
Table 7: Core simulation parameters for the case study.
HyperparameterPPOAPPOSAC
Learning rate1e-51e-5Actor: 1e-5, Critic: 1e-5
Discount factor ( ψ\psi )0.990.990.99
Entropy coefficient0.010.01Dynamic
Gradient clip0.50.50.5
Clip Param0.2N/AN/A
GAE / V-traceGAEV-trace-
Table 8: Initial hyperparameters of the MARL algorithm for the case study.
Figure 4: Mean episode reward for evaluating the 9 experiments over 10 000 training episodes. Only the first 300 episodes are shown.
 CTCE CTDE DTDE
 PPOAPPOSAC PPOAPPOSAC PPOAPPOSAC
Min2702,22641,22560,7 2618,927912730 989,7991,6-669,8
Mean3099,43143,93139,8 3004,83096,53158,2 2742,5299854,5
Std44,897,993,4 49,164,642,5 431,5431,4215,8
Max3123,33272,13189,7 3074,13181,43166,1 2935,23243,5708,4
Table 9: Mean episode reward for evaluating the 9 experiments over 10 000 training episodes.
 CTCE CTDE DTDE
 PPOAPPOSAC PPOAPPOSAC PPOAPPOSAC
Learning Rate9.2×10−69.2\times 10^{-6}3.3×10−43.3\times 10^{-4}4.6×10−34.6\times 10^{-3} 9.2×10−69.2\times 10^{-6}1.2×10−51.2\times 10^{-5}2.9×10−32.9\times 10^{-3} 4.4×10−34.4\times 10^{-3}3.7×10−53.7\times 10^{-5}1.0×10−31.0\times 10^{-3}
Entropy Coeff5.720.01- 0.040.01- 0.240.02-
Table 10: Final hyperparameters optimized by PBT after 10 000 training episodes.
Figure 5: Implicit cooperation KPIs and market dynamics for the best configurations identified: APPO-CTCE, SAC-CTDE, and APPO-DTDE.
Figure 6: Trading network for the case study. The blue circles represent buyers, while the black circles represent sellers. The thickness of the edges indicates the magnitude of the energy exchanged between the agents.
Figure 7: Computational scalability matrix of MARL configurations for different numbers of agents.
More in this topic

Multi-Agent Coordination

Multi-agent coordination studies how autonomous agents align actions, share information, and divide work to achieve individual or collective goals.

See all 511 papers

Explore similar work

CardsList
Feb 17, 2026

MARLEM: A Multi-Agent Reinforcement Learning Simulation Framework for Implicit Cooperation in Decentralized Local Energy Markets

This paper introduces a novel, open-source MARL simulation framework for studying implicit cooperation in LEMs, modeled as a decentralized partially observable Markov decision process and implemented as a Gymnasium environment for MARL. Our framework features a modular market platform with plug-and-play clearing mechanisms, physically constrained agent models (including battery storage), a realistic grid network, and a comprehensive analytics suite to evaluate emergent coordination. The main contribution is a novel method to foster implicit cooperation, where agents' observations and rewards are enhanced with system-level key performance indicators to enable them to independently learn strategies that benefit the entire system and aim for collectively beneficial outcomes without explicit communication. Through representative case studies (available in a dedicated GitHub repository in https://github.com/salazarna/marlem, we show the framework's ability to analyze how different market configurations (such as varying storage deployment) impact system performance. This illustrates its potential to facilitate emergent coordination, improve market efficiency, and strengthen grid stability. The proposed simulation framework is a flexible, extensible, and reproducible tool for researchers and practitioners to design, test, and validate strategies for future intelligent, decentralized energy systems.

Nelson Salazar-Pena, Alejandra Tabares, Andres Gonzalez-Mancera

Department of Mechanical Engineering, Universidad de los Andes, 111711, Bogotá D.C., Colombia · Department of Industrial Engineering, Universidad de los Andes, 111711, Bogotá D.C., Colombia

Multi-Agent CoordinationPower SystemsDecentralized MARLEmergent Behavior in Multi-Agent SystemsReward Shaping
AskPDF
Apr 22, 2026

A Hierarchical MARL-Based Approach for Coordinated Retail P2P Trading and Wholesale Market Participation of DERs

The ongoing shift towards decentralization of the electric energy sector, driven by the growing electrification across end-use sectors, and widespread adoption of distributed energy resources (DERs), necessitates their active participation in the electricity markets to support grid operations. Furthermore, with bi-directional energy and communication flows becoming standard, intelligent, easy-to-deploy, resource-conservative demand-side participation is expected to play a critical role in securing power grid operational flexibility and market efficiency. This work proposes a market engagement framework that leverages a hierarchical multi-agent deep reinforcement learning (MARL) approach to enable individual prosumers to participate in peer-to-peer retail auctions and further aggregate these intelligent prosumers to facilitate effective DER participation in wholesale markets. Ultimately, a Stackelberg game is proposed to coordinate this hierarchical MARL-based DER market participation framework toward enhanced market performance.

Patrick Wilk, Ethan Cantor, Yikui Liu +1

aDepartment Electrical and Computer Engineering, Rowan University, 201 Mullica Hill Rd, Glassboro, New Jersey, 08028, United States · bSchool of Electrical Engineering, Sichuan University, 24 South Section 1, 1st Ring Road, Chengdu, 610065, China

Multi-Agent Reinforcement LearningPower Systems
AskPDF
Apr 3, 2026

Multi-agent Reinforcement Learning-based Joint Design of Low-Carbon P2P Market and Bidding Strategy in Microgrids

The challenges of the uncertainties in renewable energy generation and the instability of the real-time market limit the effective utilization of clean energy in microgrid communities. Existing peer-to-peer (P2P) and microgrid coordination approaches typically rely on certain centralized optimization or restrictive coordination rules which are difficult to be implemented in real-life applications. To address the challenge, we propose an intraday P2P trading framework that allows self-interested microgrids to pursue their economic benefits, while allowing the market operator to maximize the social welfare, namely the low carbon emission objective, of the entire community. Specifically, the decision-making processes of the microgrids are formulated as a Decentralized Partially Observable Markov Decision Process (DEC-POMDP) and solved using a Multi-Agent Reinforcement Learning (MARL) framework. Such an approach grants each microgrid a high degree of decision-making autonomy, while a novel market clearing mechanism is introduced to provide macro-regulation, incentivizing microgrids to prioritize local renewable energy consumption and hence reduce carbon emissions. Simulation results demonstrate that the combination of the self-interested bidding strategy and the P2P market design helps significantly improve renewable energy utilization and reduce reliance on external electricity with high carbon-emissions. The framework achieves a balanced integration of local autonomy, self-interest pursuit, and improved community-level economic and environmental benefits.

Junhao Ren, Honglin Gao, Sijie Wang +5

School of Electrical and Electronic Engineering, Nanyang Technological University, 639798, Singapore · School of Information Science and Technology, University of Science and Technology of China, Hefei, 230026, China · MOYA Analytics, 051912, Singapore +1

Power Systems
AskPDF