cs.AIFeb 18, 2026

Multi-agent cooperation through in-context co-player inference

Authors: Marissa A. Weis, Maciej Wołczyk, Rajai Nasser, Rif A. Saurous, Blaise Agüera y Arcas, João Sacramento, Alexander Meulemans

Organizations: Google, Paradigms of Intelligence Team · Santa Fe Institute

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Achieving cooperation among self-interested agents remains a fundamental challenge in multi-agent reinforcement learning. Recent work showed that mutual cooperation can be induced between "learning-aware" agents that account for and shape the learning dynamics of their co-players. However, existing approaches typically rely on hardcoded, often inconsistent, assumptions about co-player learning rules or enforce a strict separation between "naive learners" updating on fast timescales and "meta-learners" observing these updates. Here, we demonstrate that the in-context learning capabilities of sequence models allow for co-player learning awareness without requiring hardcoded assumptions or explicit timescale separation. We show that training sequence model agents against a diverse distribution of co-players naturally induces in-context best-response strategies, effectively functioning as learning algorithms on the fast intra-episode timescale. We find that the cooperative mechanism identified in prior work-where vulnerability to extortion drives mutual shaping-emerges naturally in this setting: in-context adaptation renders agents vulnerable to extortion, and the resulting mutual pressure to shape the opponent's in-context learning dynamics resolves into the learning of cooperative behavior. Our results suggest that standard decentralized reinforcement learning on sequence models combined with co-player diversity provides a scalable path to learning cooperative behaviors.

Figures & tables

Figure 1: Mixed training leads to robust cooperation. RL agents trained against a mix of tabular policies and learning agents converge to cooperation (solid lines). Ablations: Agents trained purely against other learning agents (dotted lines) or with access to explicit co-player identifications (dashed lines) converge to defection, highlighting that in-context inference is a critical factor for the learning of cooperative behaviors with standard decentralized MARL. Error bars indicate standard deviation across 10 random seeds.
Figure 2: A–B: Emergence of in-context best response. Performance of PPI agents (trained against random tabular opponents) when evaluated against specific fixed strategies. The agents demonstrate in-context learning, identifying the opponent and converging to the best response within the episode. C–D: Learning to extort in-context learners. Agents trained against a “Fixed In-Context Learner” (an agent pre-trained in Step 1 to best-respond to tabular policies) learn to extort it. The RL agent achieves a higher share of the reward by exploiting the in-context adaptation of its opponent. E–F: From mutual extortion to cooperation. When two agents initialized with extortion policies (from Step 2) play against each other, their mutual attempts to extort their co-player result in the shaping of each other’s policy towards more cooperative behavior, both within episodes through in-context learning ( F ) and across episodes through in-weight learning ( E ). Error bars indicate standard deviation across 10 random seeds.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

  Player 2
  CD
Player 1C(1, 1)(-1, 2)
D(2, -1)(0, 0)
Appendix
Table 1: Single-round IPD payoff matrix
RL HyperparameterStep 1Step 2Step 3Step 4
advantages_normalizationTrueFalseTrueTrue
batch size2048204840964096
reward_rescaling0.20.20.050.050.020.020.020.02
value_discount ( γ\gamma )0.990.990.990.990.990.990.990.99
td_lambda ( λtd\lambda_{\text{td}} )0.990.991.01.00.950.951.01.0
gae_lambda ( λgae\lambda_{\text{gae}} )0.990.991.01.00.950.951.01.0
Appendix
Table 2: A2C hyperparameters
Figure 3: Emergence of best-response in mixed training. We plot within-episode performance of models trained in Figure 1 before convergence. We observe that both A2C and PPI try to extort their counterpart at the beginning of the episode which subsequently leads to increased levels of cooperation. At the same time, identifying the opponent as a non-tit-for-tat-like tabular policy leads to high defection ratio. Error bars indicate standard deviation across 10 random seeds.
Figure 4: A-B: Emergence of in-context best response Performance of A2C trained against random tabular opponents and evaluated after convergence on a set of specific static policies. We denote the final agent as “Fixed In-Context Learner”. C-D: Learning to extort in-context learners. Performance of a randomly initialized A2C agent against the Fixed In-Context Learner. E-F: From mutual extortion to cooperation. Two A2C extortion agents initially converge to cooperation when playing against each other, but with time they might collapse to mutual defection depending on the random seed. Error bars correspond to standard deviation over 5 random initializations.
More in this topic

Cooperative MARL

Cooperative multi-agent reinforcement learning (MARL) trains agents to coordinate toward shared goals under partial information and decentralized control.

See all 244 papers

Explore similar work

CardsList
May 11, 2026

PC3D: Zero-Shot Cooperation Across Variable Rosters via Personalized Context Distillation

Cooperative multi-agent reinforcement learning often assumes a fixed execution team, yet many decentralized systems must operate with varying numbers of active agents during deployment. We study this setting under episodic roster variation: each episode is executed by a set of homogeneous agents, with the team size varying across episodes. Agents act only from local histories, without execution-time communication, privileged coordinators, or online retraining. Therefore, effective cooperation requires each agent to recover relevant context about the active team and adapt its behavior accordingly. To this end, we propose PC3D (Personalized Central Coordination Context Distillation), a method for training decentralized policies to recover and use personalized coordination context from local interaction histories. During training, a set-structured centralized teacher compresses the active team into coordination tokens and personalizes them into agent-specific contexts, which are distilled into decentralized policies. At execution, each agent predicts its own context from local history and adaptively uses it to condition decision-making. Across three cooperative MARL benchmarks, PC3D achieves higher returns than the evaluated baselines with both seen and unseen roster sizes, and ablations attribute these gains to both context distillation and adaptive context use.

Ahmet Onur Akman, Rafał Kucharski

Doctoral School of Exact and Natural Sciences, Jagiellonian University, Kraków, Poland · Faculty of Mathematics and Computer Science, Jagiellonian University, Kraków, Poland

Zero-Shot LearningDecentralized MARLCooperative MARLMulti-Agent Reinforcement Learning
AskPDF
Mar 6, 2026

Multi-Agent Reinforcement Learning with Submodular Reward

In this paper, we study cooperative multi-agent reinforcement learning (MARL) where the joint reward exhibits submodularity, which is a natural property capturing diminishing marginal returns when adding agents to a team. Unlike standard MARL with additive rewards, submodular rewards model realistic scenarios where agent contributions overlap (e.g., multi-drone surveillance, collaborative exploration). We provide the first formal framework for this setting and develop algorithms with provable guarantees on sample efficiency and regret bound. For known dynamics, our greedy policy optimization achieves a 1/21/2-approximation with polynomial complexity in the number of agents KK, overcoming the exponential curse of dimensionality inherent in joint policy optimization. For unknown dynamics, we propose a UCB-based learning algorithm achieving a 1/21/2-regret of O(H2KSAT)O(H^2KS\sqrt{AT}) over TT episodes.

Wenjing Chen, Chengyuan Qian, Shuo Xing +2

Department of Computer Science and Engineering, Texas A&M University

Cooperative MARLMulti-Agent Reinforcement LearningSubmodular Optimization
AskPDF
Mar 7, 2026

Learning When to Cooperate Under Heterogeneous Goals

A significant element of human cooperative intelligence lies in our ability to identify opportunities for fruitful collaboration; and conversely to recognise when the task at hand is better pursued alone. Research on flexible cooperation in machines has left this meta-level problem largely unexplored, despite its importance for successful collaboration in heterogeneous open-ended environments. Here, we extend the typical Ad Hoc Teamwork (AHT) setting to incorporate the idea of agents having heterogeneous goals that in any given scenario may or may not overlap. We introduce a novel approach to learning policies in this setting, based on a hierarchical combination of imitation and reinforcement learning, and show that it outperforms baseline methods across extended versions of two cooperative environments. We also investigate the contribution of an auxiliary component that learns to model teammates by predicting their actions, finding that its effect on performance is inversely related to the amount of observable information about teammate goals.

Max Taylor-Davies, Neil Bramley, Christopher G. Lucas

School of Informatics, The University of Edinburgh · Department of Psychology, The University of Edinburgh

Multi-Agent CollaborationAd Hoc TeamworkHeterogeneous MARL
AskPDF