Authors: Tiziano Maisonhaute, Jean-Christophe Soulié, Fabien Michel
Organizations: LIRMM, Université de Montpellier, CNRS, Montpellier, France · CIRAD, UPR Recyclage et Risque, F-97743 Saint-Denis, La Réunion, France · Recyclage et Risque, CIRAD, Université de Montpellier, Montpellier, France
The growing complexity of Multi-Agent Reinforcement Learning (MARL) has fragmented the literature, obscuring underlying architectural choices behind diverse implementations. This fragmentation makes it difficult to evaluate critical modeling dimensions in isolation, identify how components affect the resulting system dynamics, and compare, reuse, or combine contributions across the literature. This paper proposes a modular perspective for modeling, analyzing, and implementing MARL systems. We decompose them into eight functional modules: agents, environments, learning, reward modeling, training and execution modes, communication, modeling of other agents, and evaluation criteria. For each module, we clarify its role and review representative approaches. We instantiate this decomposition in MARLKIT, a framework whose modules are explicit and interchangeable. MARLKIT demonstrates how the proposed perspective supports transparent experimental assumptions and the controlled variation and combination of MARL mechanisms. Experiments illustrate its ability to prototype, compare, reuse and combine modules while making differences between configurations explicit. Overall, the paper provides a structured perspective on MARL modeling and implementation, offering a common language for making modeling choices explicit, supporting clearer descriptions of MARL approaches, and supporting more controlled experimentation through the systematic isolation and combination of MARL mechanisms.
Figure 3: Partial knowledge representation. Red arrows represent missing information for Agent 1.
Figure 4: Influence/Reaction principle: agents produce influences that are combined into a joint action, which is used to compute the environment reaction (state transition). Then the rewards are compute in the reward phase.
Figure 5: policy can now take other input such as prediction of other agent action, or information that came from communication
Figure 6: Comparison of communication modes: broadcasting, networked, neighborhood and targeted. Broadcasting communication (a) involves all agents sending messages to all others. Networked communication (b) involves agents communicating only with their direct neighbors in a predefined network structure. Targeted communication (c) allows agents to send messages to specific other agents. Figures adapted from Gronauer and Diepold (2022) .
Figure 7: MARLKIT core modules.
Figure 8: Production of the rewards
Figure 9: Total of the raw rewards of all agents added together. Green is independent reward, Blue is fully cooperative reward, red is fair mixed reward and orange is logistical reward.
Figure 10: Total of the raw rewards of all agents added together. Blue is DDPG and orange is MADDPG.
Figure 11: Total of the raw rewards of all agents added together. Orange is no communication, green is policy parameter sharing, blue is observation sharing and red is both observation and policy parameter sharing.
Figure 12: Proportion of episodes in which one car passes, with one agent selecting Go and the other Yield .
Figure 13: System evaluation process.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 14: All collecting resources results. Green is independent reward, blue is fully cooperative reward, red is fair mixed reward and orange is logistical reward.
Figure 15: Results of the prey-hunter experiment. Blue is DDPG and orange is MADDPG.
Figure 16: Results of the prey-hunter experiment. Orange is no communication, green is policy parameter sharing, blue is observation sharing and red is both observation and policy parameter sharing.
Figure 17: Go or Yield reward matrix. In each cell, the reward on the left is received by Agent 1 and the reward on the right by Agent 2.
Figure 18: Results of Go or Yield experiment. Green is independent, orange is minimax and blue is action frequencies modeling.