Cooperative EV Charging Assignment Using Multi-Agent Reinforcement Learning: A Comparison of QMIX and Independent Q-Learning Under Urban Traffic Uncertainty

Nour-Eddine Moumni; Rachid Alaoui; Driss Kiouach

cs.MAOct 27, 2026

Cooperative EV Charging Assignment Using Multi-Agent Reinforcement Learning: A Comparison of QMIX and Independent Q-Learning Under Urban Traffic Uncertainty

Authors: Nour-Eddine Moumni, Rachid Alaoui, Driss Kiouach

Organizations: Laboratory of Research in Computer Science and Telecommunications (LRIT), Mohammed V University, Rabat, Morocco · Higher School of Technology of Salé (EST Salé), Mohammed V University, Salé, Morocco · L2MASI Laboratory, Faculty of Sciences Dhar El Mahraz, Sidi Mohamed Ben Abdellah University, Fez, Morocco

Abstract

The allocation of electric vehicles to public charging stations is increasingly constrained by the fact that several heterogeneous facilities often serve the same stream of requests while observing only partial local information. An accept/reject decision that is defensible for one station may nevertheless reduce the efficiency of the charging district as a whole. We formulate this operational problem as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP) and examine whether Centralized Training with Decentralized Execution (CTDE), instantiated through QMIX, improves assignment performance relative to a cooperative Independent Q-Learning baseline (IQL). The evaluation uses a six-station urban network simulated with SUMO, five independent random seeds, 200 training episodes, and four traffic flux scenarios ranging from 100 to 250 vehicles/hour. Under this protocol, QMIX reaches a mean reward per evaluated decision of +0.776, versus +0.578 for IQL (delta = +0.198, seed-paired Cohens dz = 2.98, exact two-sided Wilcoxon p = 0.0625, the minimum attainable value at this sample size), and raises the acceptance rate from 72.4% to 94.0% (delta = +21.6 pp, dz = 5.24, Wilcoxon p = 0.0625). A five-variant ablation suggests this gain is primarily attributable to the CTDE architecture when holding cooperative reward terms fixed, though not formally isolated with a single seed. By contrast, cooperative reward shaping and experience sharing show no clear descriptive benefit and may slightly reduce performance. These findings support the view that, for multi-station charging assignment, coordination should be embedded in the learning architecture rather than added only through heuristic cooperation terms.

Figures & tables

Figure 1: Overview of the cooperative MARL framework. A centralized MARL controller trains six station agents (two fast, two medium, two slow) on a SUMO-simulated urban network through CTDE . Each agent observes a local $1{,}152$ -state discretization and selects accept/reject in real time. Training cycles through four traffic flux regimes; rewards combine success/failure outcomes with shaping terms. The figure contrasts the centralized training phase (purple) with the decentralized execution phase (blue).

ID	Variant	Algo	Coop	$L_{\text{coop}}$	$R_{\text{exp}}$	Reward
A0	QMIX (full)	QMIX	✓	0.20	0.00	$+0.759$
A1	QMIX (without coop)	QMIX	$\times$	0.00	0.00	$+0.809$
A2	IQL (baseline)	IQL	✓	0.20	0.15	$+0.517$
A3	IQL (without sharing)	IQL	✓	0.20	0.00	$+0.561$
A4	IQL (pure)	IQL	$\times$	0.00	0.00	$+0.584$

Table 1: Ablation design used to isolate the contribution of CTDE, cooperative reward shaping, and experience sharing. Each variant differs from the reference configuration by at most two controlled changes.

Metric	IQL	QMIX	$\Delta$	Stat.
Mean Reward	$+0.578$	$+0.776\,\dagger$	$+0.198$	$d_{z}=2.98$ ‡
Accept Rate (%)	$72.4$	$94.0\,\dagger$	$+21.6\,\text{pp}$	$d_{z}=5.24$ ‡
Success/Accept (%)	$100.0$	$100.0$	$0.0$	—
Travel Time (s)	$13.3\,\dagger$	$15.9$	$+2.5$	selection bias ∗
Waiting Time (s)	$51.4$	$47.9\,\dagger$	$-3.5$	—
Load Balance CV	$0.531$	$0.551$	$+0.020$	ns

Table 2: QoS comparison, pooled over every evaluated decision across 5 seeds

\times

200 episodes. Best value per metric marked with

\dagger

; ns = not significant.

Figure 2: Reward convergence curves (IQL = blue, QMIX = orange) per traffic flux scenario, showing per-episode mean reward across 5 seeds. QMIX converges faster and achieves higher asymptotic performance in all scenarios, with the largest absolute gap under High flux where station contention is greatest.

Figure 3: Performance heatmap across four traffic flux scenarios (rows) and five QoS dimensions (columns) for IQL (left) and QMIX (right), where cell values are raw means and color intensity is column-normalized. QMIX dominates on reward and accept rate across all scenarios, while load balance CV remains comparable.

Figure 4: Accept rate (left) and mean reward (right) by station type for IQL and QMIX (5 seeds $\times$ 200 episodes), where QMIX achieves uniformly high accept rates across all station types while IQL performance is most degraded at Med-1 and Slow-2, the two stations with the lowest accept rate under contention.

Panel of Figure 5 — Figure 5: ( a ) Mean reward and accept rate for the five ablation variants (seed = 42, 50 episodes) and ( b ) estimated contribution of each cooperative component to mean reward ( $\Delta=\text{variant\_with}-\text{variant\_without}$ ). Positive values denote an increase in reward after adding the component; negative values denote a decrease. The only positive component effect in this ablation is associated with CTDE ( $+0.198$ , A0 vs. A3, the single-factor contrast).

Figure 6: Training trajectories of the five ablation variants (seed = 42). The QMIX configurations (A0, A1) reach stable reward levels early in training and remain above the IQL configurations, which stay volatile throughout the 50 episodes without a clear ordering in variance among A2–A4.

Electric Vehicle Charging

Electric vehicle charging research studies how to forecast, plan, control, and secure charging while coordinating vehicles, stations, and power grids.

See all 40 papers

Jun 30, 2026

Smart charging of large fleets of Electric Vehicles: Independent Multi-Agent Reinforcement Learning approaches

The electrification of transportation through electric vehicles introduces new challenges for power grid management, such as increased peak demand, voltage fluctuations, line overloads, and the integration of variable renewable energy sources. To enable efficient integration of EVs while minimizing costs for users and avoiding network overloads, implicit coordination between EVs is required. This work compares two independent multi-agent reinforcement learning approaches for optimizing such decentralized EV charging: contextual combinatorial bandits and policy gradient algorithms. Using a realistic simulation environment with autonomous agents making decisions based on local environmental information (including price signals, state-of-charge, and temporal constraints), we evaluate their performance across varying congestion levels, and mixed-strategy configurations with heterogeneous agent groups under dynamic electricity pricing derived from real photovoltaic production data.

Xavier Rate, Eloann Le Guern, Raphaël Féraud +8

Orange · ENS Rennes · IMT Atlantique +1

Multi-Agent Coordination Electric Vehicle Charging Policy Gradient Methods Contextual Bandits Decentralized MARL Power Systems

Ask PDF

Sep 7, 2026

Emergent Charging Coordination in Electric Delivery Fleets

In electric delivery fleets, mid-shift charging is non-trivial: each vehicle must decide when, where and how much to charge to finish on time with battery above a safety floor. The choices are coupled: queues build where too many vehicles pick the same station. Prior work resolves this coupling with central dispatching, precomputed schedules or reservations, machinery that charging infrastructure rarely supports. Instead, we use a family of learning agents under purely local control: every vehicle runs the same policy, deciding alone from its time budgets and broadcast station occupancies, leading to emergent coordination without central control or messaging. We validate this paradigm in simulation on real OpenStreetMap networks of twenty cities, each with a frozen scenario calibrated by an omniscient Oracle (99.5% of shifts completed on time), whereas a naive greedy rule (nearest station on low battery) completes just 73%. Agents trained with neuroevolution (NEAT) and policy gradients (PPO) on four cities and deployed zero-shot across all twenty, sixteen never seen in training, complete 96.8% and 98.6% of shifts, with the policy-gradient controllers proving more robust when demand or vehicle characteristics drift beyond the trained regime. In contrast, tuned threshold heuristics that read vehicle urgency alone fall short in contended cities (~80%). Through training, these learning agents rediscover partial charging and short opportunistic sessions, and route around busy stations, cutting per-session queue waits from about 45 minutes to under 2. In summary, this coordination paradigm balances local urgency against public occupancy, reaching near-Oracle performance at minimal implementation cost.

Javier Vales-Alonso, Juan J. Alcaraz

Department of Information and Communication Technologies, Universidad Politécnica de Cartagena, Cartagena, Spain

Multi-Agent Coordination Electric Vehicle Charging Emergent Behavior in Multi-Agent Systems

Ask PDF

Apr 28, 2026

A Distributionally Robust Reinforcement Learning Framework for Constrained Urban EV Dispatch

We study city-scale control of electric-vehicle (EV) ride-hailing fleets where dispatch, repositioning, and charging decisions must respect charger and feeder limits under uncertain, spatially correlated demand and travel times. We formulate the problem as a hex-grid semi-Markov decision process (semi-MDP) with mixed actions -- discrete actions for serving, repositioning, and charging, together with continuous charging power -- and variable action durations. To guarantee physical feasibility during both training and deployment, the policy learns over high-level intentions produced by a masked, temperature-annealed actor. These intentions are projected at every decision step through a time-limited rolling mixed-integer linear program (MILP) that strictly enforces state-of-charge, port, and feeder constraints. To mitigate distributional shifts, we optimize a Soft Actor-Critic (SAC) agent against a Wasserstein-1 ambiguity set with a graph-aligned Mahalanobis ground metric that captures spatial correlations. The robust backup uses the Kantorovich-Rubinstein dual, a projected subgradient inner loop, and a primal-dual risk-budget update. Our architecture combines a two-layer Graph Convolutional Network (GCN) encoder, twin critics, and a value network that drives the adversary. Experiments on a large-scale EV fleet simulator built from NYC taxi data show that PD-RSAC achieves the highest net profit, reaching $1.22M, compared with $0.58M-$0.70M for strong heuristic, single-agent RL, and multi-agent RL baselines, including Greedy, SAC, MAPPO, and MADDPG, while maintaining zero feeder-limit violations.

An Nguyen, Hoang Nguyen, Phuong Le +3

College of Engineering and Computer Science, VinUniversity, Hanoi, Vietnam · Center for Environmental Intelligence, VinUniversity, Hanoi, Vietnam

Electric Vehicle Charging Constrained RL Distributionally Robust Optimization Distributionally Robust RL Robust RL

Ask PDF

Abstract

Electric Vehicle Charging

Explore similar work

Smart charging of large fleets of Electric Vehicles: Independent Multi-Agent Reinforcement Learning approaches

Emergent Charging Coordination in Electric Delivery Fleets

A Distributionally Robust Reinforcement Learning Framework for Constrained Urban EV Dispatch