The Role of Network Topology and Opponent Information in Shaping Cooperation in Multi-Agent Reinforcement Learning Systems
Authors: Seongho Son, Stephen Hailes, Mirco Musolesi
Organizations: Department of Computer Science, University College London · UCL Centre for Artificial Intelligence, University College London · Department of Computer Science and Engineering, University of Bologna
Several works have investigated the influence of graph topology on cooperation among artificial agents, while the majority of the literature has focused on modelling agents' adaptation through strategy imitation, which relies solely on the cumulative payoffs of others. This paper investigates scenarios in which each agent learns to play the two-player Iterated Prisoner's Dilemma (IPD) using deep reinforcement learning. Each agent is represented as a node in a graph, where its neighbours constitute the pool of opponents with whom it can interact. During each IPD episode, agents are provided with different types of information about their opponent, consisting of action history and opponent identity. Experimental results across different graph topologies show that the number of neighbours per node and the average path length are the main factors affecting the emergence of cooperation. We also show that, while partner selection fosters mutual cooperation by limiting the diversity of the opponent pool, providing agents with the identity of their opponent hinders the proliferation of cooperative strategies.
Figures & tables
PD
C
D
C
(R, R)
(S, T)
D
(T, S)
(P, P)
Table 1: (a) General format of payoff matrix used for an episode of Prisoner’s Dilemma (left). (b) Payoff matrix used in each interaction of Iterated Prisoner’s Dilemma (right).
Figure 1: Visual summary of IPD on graphs, from the perspective of i th agent. For each interaction round, each agent selects an opponent among its neighbours in the graph with uniform probability. Both experiences of playing with opponents that agent i selected or the agent being selected by one of its neighbours are used for training the dilemma-playing policy πi .
Figure 2: Visual summary of IPD with partner selection, from the perspective of i th agent. For each interaction round, each agent selects an opponent among the other N−1 agents using partner selection module ηsel,πi . The partner selection module is trained with only the experience of agent i selecting an opponent, as the action of partner selection aseli is the index of the selected opponent.
Figure 3: Experimental results per graph type with parameters for graph generation (ER, WS and BA from the left, respectively).
Figure 4: Experimental results per graph type with average path length per graph (ER, WS and BA from the left, respectively).
Figure 5: Relation between graph generation parameters and average path length (ER, WS and BA from the left, respectively).
Figure 6: Experimental results of partner selection experiment. Dots represent the average proportion of mutual cooperation in the last 100 interaction rounds of 20 runs, while vertical intervals represent 95% confidence intervals of each experiment setting. Experiments conducted without having identity information in the states of dilemma-playing agents are coloured in green, while experiments which included binary encoding of the opponent’s identity index are coloured in orange. Except for the experiment with identity information and action history length l=1 , which showed significantly lower level of mutual cooperation, all the other experiments’ confidence intervals overlap.
Figure 7: Learning dynamics of partner selection experiment without opponent identity ( l=1,5,10 from the left, respectively).
Figure 8: Learning dynamics of partner selection experiment with opponent identity ( l=1,5,10 from the left, respectively).
Q(sC,C)>Q(sC,D)
Q(sC,C)≤Q(sC,D)
Q(sD,C)>Q(sD,D)
cooperator
reverser
Q(sD,C)≤Q(sD,D)
retaliator
defector
Table 2: Possible strategies in experiments where l=1 , opponent identity excluded from state information.
Figure 9: Interaction patterns in the IPD across different graph structures (ER, WS and BA from the top, respectively). Each graph shows results of one of the runs in each experiment setting. Each node represents an agent in the population. Colours of the nodes represent types of interactions the agent has experienced the most in the last 100 interaction rounds. Colours of the edges represent types of interactions that have occurred the most between the two nodes each edge is connecting.
Figure 10: Interaction patterns in the IPD with partner selection without identity ( l=1,5,10 from the top, respectively). Each graph shows results of one of the 20 runs in each experiment setting. Edges have directions as each agent can select their interaction partners. Agents that mainly cooperate playing IPD (blue, purple nodes) are being selected from other agents, while agents that mainly exploit others (green nodes) are not being selected.
Figure 11: Dynamics of inequality (Gini coefficient) of partner selection between agents per 100 interaction rounds. Shaded areas show 95% confidence interval across 20 runs.
Figure 12: Dynamics of inequality (Gini coefficient) of proportion of different types of interactions. pmc (upper left), pue (upper right) and pmd (bottom left) denote the proportion of mutual cooperation, unilateral exploitation ( sucker defined in Section 2.1 ) and mutual defection in 100 interaction rounds respectively. Shaded areas show 95% confidence interval across 20 runs.
Figure 13: IPD interaction patterns with partner selection and binary-encoded identity ( l=1,5,10 from the top, respectively). Each graph shows results of one of the 20 runs in each experiment setting. When the length of the action history is 1, partner selections are highly concentrated. This concentration is alleviated as the history length gets longer, enabling more mutual cooperation to emerge in the population as experiment proceeds.
Cooperative multi-agent reinforcement learning (MARL) trains agents to coordinate toward shared goals under partial information and decentralized control.
Sustained cooperation among self-interested learning agents is essential in many shared environments. Reputation systems are among the best-studied mechanisms for promoting such cooperation. Existing multi-agent reinforcement learning approaches, however, typically assume that reputation is either globally available to all agents or derived solely from an agent's own observations, leaving largely unexplored how reputational information propagates between agents. We study gossip, the exchange of information about third parties, as a mechanism for reputation diffusion over dynamic communication networks. Agents form private opinions from limited observations and exchange them with their neighbors following DeGroot opinion dynamics, while the communication network changes as agents interact and move. A tunable broadcast radius controls the reach of information, spanning private reputation, local gossip, and globally shared reputation. We study this framework in two n-player social dilemmas with distinct network dynamics: a Prisoner's Dilemma with randomly resampled interaction groups and Harvest, where communication neighborhoods emerge from agents' movements. Our results show that the reach of reputational information shapes cooperation in both environments. Moreover, in Harvest, reputation diffusion increases environmental rewards and leads to more peaceful and sustainable behavior.
Achieving cooperation among self-interested agents remains a fundamental challenge in multi-agent reinforcement learning. Recent work showed that mutual cooperation can be induced between "learning-aware" agents that account for and shape the learning dynamics of their co-players. However, existing approaches typically rely on hardcoded, often inconsistent, assumptions about co-player learning rules or enforce a strict separation between "naive learners" updating on fast timescales and "meta-learners" observing these updates. Here, we demonstrate that the in-context learning capabilities of sequence models allow for co-player learning awareness without requiring hardcoded assumptions or explicit timescale separation. We show that training sequence model agents against a diverse distribution of co-players naturally induces in-context best-response strategies, effectively functioning as learning algorithms on the fast intra-episode timescale. We find that the cooperative mechanism identified in prior work-where vulnerability to extortion drives mutual shaping-emerges naturally in this setting: in-context adaptation renders agents vulnerable to extortion, and the resulting mutual pressure to shape the opponent's in-context learning dynamics resolves into the learning of cooperative behavior. Our results suggest that standard decentralized reinforcement learning on sequence models combined with co-player diversity provides a scalable path to learning cooperative behaviors.
Marissa A. Weis, Maciej Wołczyk, Rajai Nasser +4
Google, Paradigms of Intelligence Team · Santa Fe Institute
To promote cooperation in Multi-Agent Reinforcement Learning, the reward signals of all agents can be aggregated together, forming global rewards that are commonly known as the fully cooperative setting. However, global rewards are usually noisy because they contain the contributions of all agents, which have to be resolved in the credit assignment process. On the other hand, using local reward benefits from faster learning due to the separation of agents' contributions, but can be suboptimal as agents myopically optimize their own reward while disregarding the global optimality. In this work, we propose a method that combines the merits of both approaches. By using a graph of interaction between agents, our method discerns the individual agent contribution in a more fine-grained manner than a global reward, while alleviating the cooperation problem with agents' local reward. We also introduce a practical approach for approximating such a graph. Our experiments demonstrate the flexibility of the approach, enabling improvements over the traditional local and global reward settings.
Bang Giang Le, Viet Cuong Ta
Human-Machine Interaction Laboratory VNU University of Engineering and Technology, Hanoi, Vietnam