eess.SYFeb 17, 2026

Harnessing Implicit Cooperation: A Multi-Agent Reinforcement Learning Approach Towards Decentralized Local Energy Markets

Authors: Nelson Salazar-Pena, Alejandra Tabares, Andres Gonzalez-Mancera

Organizations: Department of Mechanical Engineering, Universidad de los Andes, 111711, Bogotá D.C., Colombia · Department of Industrial Engineering, Universidad de los Andes, 111711, Bogotá D.C., Colombia

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

This paper proposes implicit cooperation, a framework enabling decentralized agents to approximate optimal coordination in local energy markets without explicit peer-to-peer communication. We formulate the problem as a decentralized partially observable Markov decision problem that is solved through a multi-agent reinforcement learning task in which agents use stigmergic signals (key performance indicators at the system level) to infer and react to global states. Through a 3x3 factorial design on an IEEE 34-node topology, we evaluated three training paradigms (CTCE, CTDE, DTDE) and three algorithms (PPO, APPO, SAC). Results identify APPO-DTDE as the optimal configuration, achieving a coordination score of 91.7% relative to the theoretical centralized benchmark (CTCE). However, a critical trade-off emerges between efficiency and stability: while the centralized benchmark maximizes allocative efficiency with a peer-to-peer trade ratio of 0.6, the fully decentralized approach (DTDE) demonstrates superior physical stability. Specifically, DTDE reduces the variance of grid balance by 31% compared to hybrid architectures, establishing a highly predictable, import-biased load profile that simplifies grid regulation. Furthermore, topological analysis reveals emergent spatial clustering, where decentralized agents self-organize into stable trading communities to minimize congestion penalties. While SAC excelled in hybrid settings, it failed in decentralized environments due to entropy-driven instability. This research proves that stigmergic signaling provides sufficient context for complex grid coordination, offering a robust, privacy-preserving alternative to expensive centralized communication infrastructure.

Figures & tables

ParadigmControllerCoordination SignalKey AdvantageKey DisadvantageScalabilityResilience
Centralized controlSingle central controllerDirect command and control signalsGlobal optimization, high controllabilitySingle point of failure, high computational burdenLowLow
Decentralized controlIndividual agentsLocal measurements (voltage, frequency)No single point of failure, plug-and-play capabilitySuboptimal global outcomes, system imbalanceHighHigh
Transactive energyIndividual agentsEconomic signals (prices, bids, asks)Fosters prosumer autonomy, economic efficiencyCommunication infrastructure, potential for market power abuseHighHigh
Table 1: Comparative analysis of control paradigms for distributed energy resources (DERs).
FeatureCTCECTDEDTDERef
Architecture1-to-NN-to-N (centralized critic)N-to-N (independent)[ 85 , 74 , 38 ]
Information (Training)Global stateGlobal state (centralized critic)Local observations[ 117 , 18 ]
Information (Execution)Global stateLocal observationsLocal observations[ 18 ]
Coordination mechanismCentralized optimizerCentralized criticIndependent learners[ 117 , 18 ]
StrengthOptimalityPerformanceScalability[ 117 , 18 ]
WeaknessCurse of dimensionalityCurse of dimensionalityNon-stationarity[ 83 , 74 , 38 ]
Table 2: Technical taxonomy of MARL training paradigms in LEMs.
FeatureOptimization ModelsGame Theoretic ModelsMARL
ObjectiveSystem-optimal (e.g., global welfare)Nash equilibriumAdaptive profit maximization
Uncertainty handlingRobust or stochasticDeterministicBelief state (experience-based)
Market dynamicsStatic with perfect competitionStatic equilibriumNon-stationary and emergent
ScalabilityHigh (linear/convex)Low (combinatorial)High (observations and reward shaping)
LimitationRequires perfect foresightComputational intractabilityNon-stationarity and training stability
Table 3: Comparative analysis of market modeling and control methods in LEMs.
FeaturePPO / APPOSACRef
Policy typeOn-policyOff-policy[ 102 , 44 ]
Key featureClip functionMax entropy[ 66 ]
Trade-offStability / Sample efficiencySample efficiency / Stale data-
Sample efficiencyLow (discards data after update)High (re-uses data via replay buffer)[ 102 ]
Update mechanismClipped surrogate objectiveActor-critic with entropy regularization[ 102 ]
Data sensitivityRobust to stale dataProne to stale data issues[ 107 ]
Table 4: Comparative analysis of PPO/APPO and SAC for continuous control in LEMS.
Market Signals Agent-Specific Signals Implicit Cooperation KPIs
1. Current step 2. Time of day 3. Clearing price 4. Clearing volume 5. Grid balance 6. DSO buy volume 7. DSO sell volume 8. DSO total volume9. P2P volume 10. DSO trade ratio 11. Net grid import 12. DSO buy price 13. DSO sell price 14. Mean local price 15. Price spread 16. Local price advantage 1. Energy generation 2. Energy demand 3. Cumulative demand satisfied 4. Cumulative demand deferred 5. Remaining demand 6. Cummulative supply satisfied 7. Cummulative supply deferred 8. Remaining supply9. Mean profit 10. Reputation 11. Battery energy level 12. Battery SoC 13. Battery available charge 14. Battery available discharge 15. Battery cummulative charge 16. Battery cummulative discharge 1. Social welfare 2. Market liquidity 3. Mean bid-ask spread 4. Price volatility 5. Supply-demand imbalance6. Grid congestion 7. Coordination score 8. Coordination convergence 9. DER self-consumption 10. Flexibility utilization
Table 5: Observation space vector for the formulation of the LEM as a Dec-POMDP.
AgentIEEE-34 NodeCapacity (kW)Battery (kW)Battery RatioDescription
Small Industry840180.0108.00.6Continuous demand with morning peak; relies on stored energy
Community Hospital890170.085.00.5High morning demand peak, transitioning to afternoon generation surplus
University Campus844140.098.00.7Bi-modal demand (morning/afternoon peaks); moderate midday generation
Shopping Mall816130.078.00.6High afternoon/evening demand; generation sink for midday surplus
Residential Complex80080.064.00.8Dual-peak demand (morning/evening); reliant on storage for evening ramp
Apartment Building83075.052.50.7Similar to residential complex; moderate generation capability
Table 6: Configuration of the agents for the case study.
Figure 1: Generation and demand profiles for DER agents in the case study.
Figure 2: The 34-node IEEE test feeder as the grid network for the case study.
Figure 3: Feed-in tariff and utility price profiles for DSO agent in the case study.
ParameterValueDescription
Max steps24Episode length representing a full 24-hour cycle (1 hour/step)
Grid topologyIEEE 34-BusStandard distribution network topology
Grid capacity1800 kWTotal grid capacity
Seed42Fixed random seed for reproducibility
Price range20 - 600.0Allowable bid range, designed to be wider than DSO prices
Quantity range0 - 180 kWhAllowable trade size per order
Table 7: Core simulation parameters for the case study.
HyperparameterPPOAPPOSAC
Learning rate1e-51e-5Actor: 1e-5, Critic: 1e-5
Discount factor ( ψ\psi )0.990.990.99
Entropy coefficient0.010.01Dynamic
Gradient clip0.50.50.5
Clip Param0.2N/AN/A
GAE / V-traceGAEV-trace-
Table 8: Initial hyperparameters of the MARL algorithm for the case study.
Figure 4: Mean episode reward for evaluating the 9 experiments over 10 000 training episodes. Only the first 300 episodes are shown.
 CTCE CTDE DTDE
 PPOAPPOSAC PPOAPPOSAC PPOAPPOSAC
Min2702,22641,22560,7 2618,927912730 989,7991,6-669,8
Mean3099,43143,93139,8 3004,83096,53158,2 2742,5299854,5
Std44,897,993,4 49,164,642,5 431,5431,4215,8
Max3123,33272,13189,7 3074,13181,43166,1 2935,23243,5708,4
Table 9: Mean episode reward for evaluating the 9 experiments over 10 000 training episodes.
 CTCE CTDE DTDE
 PPOAPPOSAC PPOAPPOSAC PPOAPPOSAC
Learning Rate9.2×10−69.2\times 10^{-6}3.3×10−43.3\times 10^{-4}4.6×10−34.6\times 10^{-3} 9.2×10−69.2\times 10^{-6}1.2×10−51.2\times 10^{-5}2.9×10−32.9\times 10^{-3} 4.4×10−34.4\times 10^{-3}3.7×10−53.7\times 10^{-5}1.0×10−31.0\times 10^{-3}
Entropy Coeff5.720.01- 0.040.01- 0.240.02-
Table 10: Final hyperparameters optimized by PBT after 10 000 training episodes.
Figure 5: Implicit cooperation KPIs and market dynamics for the best configurations identified: APPO-CTCE, SAC-CTDE, and APPO-DTDE.
Figure 6: Trading network for the case study. The blue circles represent buyers, while the black circles represent sellers. The thickness of the edges indicates the magnitude of the energy exchanged between the agents.
Figure 7: Computational scalability matrix of MARL configurations for different numbers of agents.
More in this topic

Multi-Agent Coordination

Multi-agent coordination studies how autonomous agents align actions, share information, and divide work to achieve individual or collective goals.

See all 511 papers

Explore similar work

CardsList

  1. MARLEM: A Multi-Agent Reinforcement Learning Simulation Framework for Implicit Cooperation in Decentralized Local Energy Markets

    Feb 17, 2026Nelson Salazar-Pena, Alejandra Tabares, Andres Gonzalez-ManceraMulti-Agent CoordinationPower Systems

    Department of Mechanical Engineering, Universidad de los Andes, 111711, Bogotá D.C., Colombia · Department of Industrial Engineering, Universidad de los Andes, 111711, Bogotá D.C., Colombia

  2. A Hierarchical MARL-Based Approach for Coordinated Retail P2P Trading and Wholesale Market Participation of DERs

    Apr 22, 2026Patrick Wilk, Ethan Cantor, Yikui Liu +1Multi-Agent Reinforcement LearningPower Systems

    aDepartment Electrical and Computer Engineering, Rowan University, 201 Mullica Hill Rd, Glassboro, New Jersey, 08028, United States · bSchool of Electrical Engineering, Sichuan University, 24 South Section 1, 1st Ring Road, Chengdu, 610065, China

  3. Multi-agent Reinforcement Learning-based Joint Design of Low-Carbon P2P Market and Bidding Strategy in Microgrids

    Apr 3, 2026Junhao Ren, Honglin Gao, Sijie Wang +5Power Systems

    School of Electrical and Electronic Engineering, Nanyang Technological University, 639798, Singapore · School of Information Science and Technology, University of Science and Technology of China, Hefei, 230026, China · MOYA Analytics, 051912, Singapore +1