Safe Decentralized Operation of EV Virtual Power Plant with Limited Network Visibility via Multi-Agent Reinforcement Learning

The electrification of transportation through electric vehicles introduces new challenges for power grid management, such as increased peak demand, voltage fluctuations, line overloads, and the integration of variable renewable energy sources. To enable efficient integration of EVs while minimizing costs for users and avoiding network overloads, implicit coordination between EVs is required. This work compares two independent multi-agent reinforcement learning approaches for optimizing such decentralized EV charging: contextual combinatorial bandits and policy gradient algorithms. Using a realistic simulation environment with autonomous agents making decisions based on local environmental information (including price signals, state-of-charge, and temporal constraints), we evaluate their performance across varying congestion levels, and mixed-strategy configurations with heterogeneous agent groups under dynamic electricity pricing derived from real photovoltaic production data.

By aggregating distributed energy resources (DERs) into a single dispatchable resource, virtual power plants (VPPs) enable the grid to use the flexibility of numerous small-scale DERs. Delivering an aggregate VPP commitment requires multi-agent coordination, where agents manage local DER models and operating constraints while jointly meeting a power commitment within limited communication rounds. Classical distributed optimization is computationally intensive, requiring iterative local optimization and repeated communication to reach consensus. Learning-based methods can accelerate coordination but often lack guarantees of local feasibility and accurate commitment tracking. This paper proposes LOOP-MAC (Learning to Optimize the Optimization Process for Multi-Agent Coordination), a fully decentralized learning-based framework that ensures local feasibility by construction while efficiently reducing the aggregate commitment gap. LOOP-MAC employs a two-stage decision process. Learned coordination uses neural optimizers equipped with a gauge layer to produce locally feasible schedules through a few rounds of message exchange. Local adjustment then rebalances each schedule toward feasible flexibility extremes using a commitment-gap estimate inferred from messages already received. The adjusted schedule remains locally feasible as a convex combination of feasible schedules, and the adjustment requires no additional communication round. On a 100-agent VPP, five communication rounds followed by local adjustment reduce the mean commitment gap from 5.1% to 0.75% of the agents' aggregate absolute power and the optimality gap from 3.93% to 3.07% relative to learned coordination alone. Per-agent online computation takes approximately 1.1 ms, including the adjustment.

The rapid growth of Electric Vehicle (EV) adoption challenges power distribution networks through peak load spikes, voltage instability, and transformer overloads from uncoordinated charging. While Model Predictive Control (MPC) and standard Reinforcement Learning (RL) methods have addressed these issues, existing approaches rarely treat real-time carbon intensity or fluctuating renewable energy (RE) availability as primary scheduling objectives, leaving substantial decarbonisation potential unrealised. This paper proposes an emission-aware RL strategy based on the Soft Actor Critic (SAC) algorithm, with a multi-objective reward that penalises carbon emissions, curtailed on-site renewables, and unmet user demand. The agent is trained within a unified benchmarking framework on the EV2Gym platform, incorporating behind-the-meter solar and wind profiles, time-varying EirGrid carbon intensity data, and realistic workplace EV behaviour across 25 Electric Vehicle Supply Equipment (EVSE) units. Nine control strategies, including heuristics, emission-aware MPC variants, and the proposed RL agent, are compared under five renewable penetration scenarios (0%-50%) over ten independent runs each. The RL agent achieves a carbon intensity as low as 23.96 grams of carbon dioxide per kilowatt-hour under 50% wind penetration, representing up to 87% emission reduction versus the uncontrolled baseline, and outperforms the external graph-based Power Distribution Network (PDN) benchmark. Transformer overload remains below 7 kWh across scenarios, against up to 1093 kWh for the As Fast As Possible (AFAP) heuristic, and renewable self-consumption reaches 52% under combined wind and solar supply. Embedding carbon intensity forecasts into the RL state and reward aligns charging with low-emission periods while preserving grid compliance and user satisfaction.