Characterizing MARL for Energy Control: A Multi-KPI Benchmark on the CityLearn Environment

Aymen Khouja; Imen Jendoubi; Oumayma Mahjoub; Oussama Mahfoudhi; Ruan De Kock; Siddarth Singh; Claude Formanek

cs.AIFeb 22, 2026

Characterizing MARL for Energy Control: A Multi-KPI Benchmark on the CityLearn Environment

Authors: Aymen Khouja, Imen Jendoubi, Oumayma Mahjoub, Oussama Mahfoudhi, Ruan De Kock, Siddarth Singh, Claude Formanek

Organizations: InstaDeep

Abstract

The optimization of urban energy systems is crucial for the advancement of sustainable and resilient smart cities, which are becoming increasingly complex with multiple decision-making units. To address scalability and coordination concerns, Multi-Agent Reinforcement Learning (MARL) is a promising solution. This paper addresses the imperative need for comprehensive and reliable benchmarking of MARL algorithms on energy management tasks. CityLearn is used as a case study environment because it realistically simulates urban energy systems, incorporates multiple storage systems, and utilizes renewable energy sources. By doing so, our work sets a new standard for evaluation, conducting a comparative study across multiple key performance indicators (KPIs). This approach illuminates the key strengths and weaknesses of various algorithms, moving beyond traditional KPI averaging which often masks critical insights. Our experiments utilize widely accepted baselines such as Proximal Policy Optimization (PPO) and Soft Actor Critic (SAC), and encompass diverse training schemes including Decentralized Training with Decentralized Execution (DTDE) and Centralized Training with Decentralized Execution (CTDE) approaches and different neural network architectures. Our work also proposes novel KPIs that tackle real world implementation challenges such as individual building contribution and battery storage lifetime. Our findings show that DTDE consistently outperforms CTDE in both average and worst-case performance. Additionally, temporal dependency learning improved control on memory dependent KPIs such as ramping and battery usage, contributing to more sustainable battery operation. Results also reveal robustness to agent or resource removal, highlighting both the resilience and decentralizability of the learned policies.

Figures & tables

Panel of Figure 1 — Figure 1: An illustrative overview of our comprehensive MARL evaluation framework. (a) A multi-objective comparison showing algorithm performance trade-offs across nine distinct KPIs. (b) A sample analysis quantifying granular metrics, such as the relative contribution of individual agents. (c) A probabilistic rank distribution from our statistical robustness analysis, summarizing algorithm performance over multiple trial. (d) Pairwise improvement probabilities indicating how likely one algorithm is to outperform another.

Figure 2: Sample efficiency curves for algorithms on the average score metric, showing SAC’s strong early gains but weaker plateaued performance, while IPPO ultimately achieves the lowest average score and best overall results.

Panel of Figure 3 — Figure 3: Aggregate Performance Analysis of Control Algorithms. This figure presents a consolidated view of algorithm performance across all evaluation scenarios. (a) The IQM of the aggregate score, where lower is better. IPPO achieves the best average performance. (b) The CVaR of the score highlights worst-case outcomes, again showing IPPO’s superiority and indicating robust performance. The wide confidence intervals for MAPPO in both (a) and (b) reveal its high sensitivity to random seeds.

Panel of Figure 4 — Figure 4: Comparison of Key Performance Indicators (KPIs) Across Algorithms. Performance on three environmental metrics, where lower IQM scores (a-c) are better. (a) Ramping (IQM): Recurrent models significantly outperform their feedforward counterparts on this metric. (b) Carbon Emissions (IQM): All algorithms perform similarly, with no clear, consistent advantage from temporal dependency. (c) Discomfort Proportion (IQM): Recurrent-SAC and IPPO achieve the strongest performance, highlighting decentralized variants’ potential on this metric. (d–f) Probability of Improvement: These plots mirror the trends in (a–c), showing that recurrent models outperform feedforward variants on ramping (d), but offers no clear advantage on carbon emissions (e), and on discomfort reduction (f). Notably, beyond Recurrent-IPPO, the decentralized variants also exhibit a high probability of outperforming other approaches on the discomfort metric.

Panel of Figure 5 — Figure 5: Battery Usage Patterns for Electrical and Hot Water Storage. This figure compares the battery management strategies learned by the algorithms, focusing on Depth of Discharge (DoD) and average discharge duration. Lower DoD IQM scores indicate less strain on the batteries while higher duration IQM scores indicate longer discharges which is more beneficial. (a, c) The DoD for both electrical and hot water storage is consistently lower for recurrent independent learners (Rec-IPPO and Rec-SAC), indicating shallower and less stressful discharge cycles. (b, d) Similarly, these recurrent agents achieve longer average discharge durations for electrical storage, reflecting more frequent and responsive battery usage.

Panel of Figure 6 — Figure 6: Analysis of Individual Agent Contribution to Global Reward. (a) Agent importance scores, which quantify individual influence on the total reward, are consistently near zero (typically ±0.05). (b) Distribution of the total team reward per time step, shown for scale. The comparison reveals that individual agent contributions (a) are negligible in magnitude relative to the overall team rewards (b).

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Policy Type	Coordination	Network Type	Algorithm
On-policy	Centralized	Feed Forward	MAPPO
On-policy	Centralized	Recurrent	Rec-MAPPO
On-policy	Decentralized	Feed Forward	Independent PPO (IPPO)
On-policy	Decentralized	Recurrent	Rec IPPO
Off-policy	Decentralized	Feed Forward	SAC
Off-policy	Decentralized	Recurrent	Rec SAC

Appendix

Table 1: Overview of MARL Algorithms Considered in This Study

Hyperparameter	Values Swept
Update Epochs ( $\gamma$ )	[2, 4, 8]
Actor Learning rate ( $\alpha$ )	[1e-4, 2.5e-4, 5e-4]
Critic Learning rate ( $\alpha$ )	[1e-4, 2.5e-4, 5e-4]
Clipping value	[0.2, 0.1, 0.05]
Entropy Coefficient	[0.0, 1e-2, 1e-55]
Max grad norm	[0.5, 5, 10]

Appendix

Table 2: Hyperparameter Sweep for PPO systems

Hyperparameter	Values Swept
Minibatch size	[64, 128, 256, 512]
Learning rate ( $\alpha$ )	[1e-4, 2.5e-4, 5e-4]
Replay Buffer size	[1e6, 5e6]
Actor Parameter update frequency	[1, 2, 4]
Learner Start	[23040, 46080]
Actor Network Layers	[(128, 128), (256,256)]

Appendix

Table 3: Hyperparameter Sweep for SAC systems

Experimental setup	SAX	Rec SAX	IPPO	REC IPPO	MAPPO	REC MAPPO
Hyperparameters
Discount factor	0.95	0.95	0.99	0.99	0.99	0.99
Batch size	8192	8192	8192	8192	8192	8192
Replay buffer size	1M	1M	-	-	-	-
Minimum replay buffer size before updating	23040	46080	-	-	-	-
Target smoothing coefficient	0.001	0.001	-	-	-	-

Appendix

Table 4: Experimental Setup for Algorithms

Observation	Description
day type	Indicates the type of day (e.g., weekday, weekend, holiday), impacting building occupancy and energy consumption patterns.
hour	Represents the hour of the day, crucial for daily patterns in electricity consumption, solar generation, and pricing.
outdoor dry bulb temperature and variations	Represents outdoor temperature, informing anticipatory actions for heating or cooling systems.
diffuse solar irradiance and variations	Measures sunlight from the sky, excluding direct sunlight, impacting natural light and heat in a building.
direct solar irradiance and variations	Represents sunlight directly from the sun’s direction, affecting solar power generation and building cooling needs.
carbon intensity	Indicates carbon dioxide emitted per unit of electricity consumed, guiding decisions to reduce a building’s carbon footprint.

Appendix

Table 5: Description of the observations that the agent recieves.

KPI Name	Description
Carbon Emission (G)	Total carbon emissions resulting from imported electricity, quantifying the environmental impact of energy consumption. Lower values indicate more environmentally friendly patterns.
Discomfort (U)	Proportion of times when a building’s indoor temperature is outside the comfort zone. A higher value indicates more discomfort for occupants.
Ramping (R)	Average change in electricity consumption from one time step to another. Lower values suggest smoother transitions and fewer spikes, indicating a more stable power consumption profile.
Daily one minus load factor average (L)	Average measure of the ratio of daily average electricity consumption to daily peak consumption. Closer to 1 indicates more even consumption throughout the day, while closer to 0 suggests variability.
Daily peak average ( $P_{d}$ )	Average of the maximum electricity consumption recorded at any time step in a day.
Annual Peak Average ( $P_{n}$ )	Highest electricity consumption value recorded across all time steps in a year.

Appendix

Table 6: Description of KPIs used in the CityLearn environment.

Figure 7: Sebulba podracer Hessel et al. (2021) architecture diagram adapted from Picard et al. (2023) .

Figure 8: An example State of Charge (SoC) graph used to illustrate the application of the Rainflow Algorithm.

Figure 9: Rainflow Algorithm application example

Panel of Figure 10 — Figure 10: Performance Variance of MAPPO Algorithms in Best- vs. Worst-Case Runs. This figure isolates the top three and bottom three performing runs for each algorithm to analyze their stability and performance consistency. Lower scores are better.

Panel of Figure 11 — Figure 11: Performance Comparison on Secondary KPIs. This figure presents a selection of key performance indicators where all tested algorithms demonstrate strong and broadly comparable results. Lower scores indicate better performance for all metrics. The overlapping confidence intervals and similar Interquartile Mean (IQM) scores across these plots indicate that no single algorithm holds a significant advantage for these objectives.

Panel of Figure 12 — Figure 12: Trade-off Comparison between IPPO and MAPPO. These plots visualize the performance of IPPO and MAPPO across multiple min-max normalized KPIs, where lower values are better.

Panel of Figure 13 — Figure 13: Trade-off Comparison between IPPO and Rec-IPPO. These plots compare the performance profiles of the feedforward and recurrent versions of IPPO across normalized KPIs where lower values are better.

Panel of Figure 14 — Figure 14: Trade-off Comparison between SAC and Recurrent SAC (Rec-SAC). These plots visualize the strategic differences between the standard and recurrent versions of the SAC algorithm across multiple normalized KPIs where lower values are better.

Panel of Figure 15 — Figure 15: Best-Case Performance of Each Algorithm Across Key KPIs. This figure highlights the peak performance achieved by each algorithm on six different metrics. This shows that recurrent models frequently achieve the best individual scores suggesting that they are capable of finding superior solutions for specific objectives contrasting with an algorithm like IPPO, which has a strong average score but may not reach the same performance peaks on individual KPIs.

Figure 16: Performance of algorithms trained with 3 agents and tested on 6 agents

Figure 17: Sample efficiency curve of Rec-MAPPO for original observation vs modified observation

Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning studies how interacting agents learn coordinated or competitive policies through shared experience and rewards.

See all 373 papers

Scaling up Energy-Aware Multi-Agent Reinforcement Learning for Mission-Oriented Drone Networks with Individual Reward
May 24, 2026Changling Li, Ying LiReinforcement Learning Credit Assignment in RL
Department of Computer Science, ETH Zurich, 8092, Zurich, Switzerland · Department of Computer Science, Colby College, 04901, Waterville, ME, USA
PowerMarketJax: A JAX Benchmark Suite for Multi-Agent Reinforcement Learning in Power Markets
Sep 29, 2026Zhanhua Pan, Xin Qin, Xiao Liu +3RL Benchmarks Multi-Agent Reinforcement Learning
Nanyang Technological University, Singapore. · Cornell University, USA. · University of Bristol, UK.
Multi-Agent Reinforcement Learning for V2X Resource Allocation: Disentangling MARL Challenges Through Benchmarking
Feb 18, 2026Siyuan Wang, Lei Lei, Pranav Maheshwari +2RL Benchmarks Reinforcement Learning
College of Engineering, University of Guelph, Guelph, Ontario, Canada · College of Electrical Engineering and Computer Sciences, Ningbo University, Ningbo, 315211, China

May 24, 2026J/K move · Enter open · S save

Scaling up Energy-Aware Multi-Agent Reinforcement Learning for Mission-Oriented Drone Networks with Individual Reward

Changling Li, Ying Li

Department of Computer Science, ETH Zurich, 8092, Zurich, Switzerland · Department of Computer Science, Colby College, 04901, Waterville, ME, USA

Reinforcement Learning Credit Assignment in RL Multi-Robot Coordination Cooperative MARL

Open paper AskPDF arXiv

Multi-agent reinforcement learning (MARL) has shown wide applicability in collaborative systems such as autonomous driving and smart cities for its ability of learning through interaction. With the recent development of drone networks, researchers have also applied MARL to address the trajectory planning problems. However, the dynamic environment and the limited battery capacity are still challenging for using MARL to achieve efficient collaborative task execution. In this paper, we propose an energy-aware MARL model as an attempt to tackle these challenges, leveraging Deep Q-Networks (DQN) with individual reward functions driven by the task execution progress and the remaining battery of drones. We conduct a set of simulation studies for the proposed mode and compare it with the shared reward MARL~\cite{Li2022MARL} to explore the impact of credit assignment in MARL. The results indicate that our proposed model can achieve at least 80% success rate regardless of the task locations and lengths. Similar to the shared reward mode, the individual reward mode can achieve a better success rate when the task density is high, and it can hit nearly a 100% success rate when task density gets close to 40%. The true advantage of our proposed model with individual reward is revealed when scaling up the environment. The comparison to the shared reward MARL shows that the our proposed model is more robust towards the change of the environment size and agent numbers. It can achieve higher success rate with fewer steps due to the clarity of the goal which improves energy efficiency even better.

Abstract

Multi-Agent Reinforcement Learning

Explore similar work

Scaling up Energy-Aware Multi-Agent Reinforcement Learning for Mission-Oriented Drone Networks with Individual Reward

PowerMarketJax: A JAX Benchmark Suite for Multi-Agent Reinforcement Learning in Power Markets

Multi-Agent Reinforcement Learning for V2X Resource Allocation: Disentangling MARL Challenges Through Benchmarking