A significant element of human cooperative intelligence lies in our ability to identify opportunities for fruitful collaboration; and conversely to recognise when the task at hand is better pursued alone. Research on flexible cooperation in machines has left this meta-level problem largely unexplored, despite its importance for successful collaboration in heterogeneous open-ended environments. Here, we extend the typical Ad Hoc Teamwork (AHT) setting to incorporate the idea of agents having heterogeneous goals that in any given scenario may or may not overlap. We introduce a novel approach to learning policies in this setting, based on a hierarchical combination of imitation and reinforcement learning, and show that it outperforms baseline methods across extended versions of two cooperative environments. We also investigate the contribution of an auxiliary component that learns to model teammates by predicting their actions, finding that its effect on performance is inversely related to the amount of observable information about teammate goals.
Figures & tables
Figure 1: An illustration of the goal space under three different scenarios, with the set of ‘worthwhile’ (rewarding and in-principle achievable) goals highlighted in green. Gego denotes the set of goals that would produce reward for the ego agent; Gteammates the set of goals that would produce reward for at least one teammate; and Gsolo the set of goals that can be achieved through the effort of a single agent.
Figure 2: (A) The hierarchical architecture of GRILL (B) The encoder-decoder architecture optimised offline in stage 1, from which the action decoder becomes the low-level policy πaction in stage 2 (C) The auxiliary modelling component used in GRILL-M (but not GRILL)
Figure 3: Example frames from the two AHT environments we extend.
Figure 4: Top: evaluation returns relative to oracle policy, measured over 1000 episodes × the 3 scenarios × 20 independent training runs. Mean values are given by the horizontal markers, while underlaid violin plots show the KDE. Agents were trained for 1e7 and 5e7 timesteps on CR and LBF respectively. Bottom: from the same set of evaluation episodes, the difference in proportion of non-solo goals pursued between the full- and no-overlap scenarios.
Figure 5: The distribution of goals attempted by the ego agent during evaluation. For CR, ‘attempt’ means the agent occupied one of the goal tiles; for LBF, it means the agent used the ‘collect’ action while adjacent to a fruit.
Figure 6: A visualisation of trajectories sampled from trained PPO and GRILL agents in the no- and full-overlap scenarios of the cooperative reaching environment. Trajectories are sampled from random initial locations, and a small amount of noise is applied to each path in order to better visualise the distributions.
Figure 7: Left: the percentage increase in mean evaluation return in the LBF environment for GRILL-M vs GRILL, as a function of the amount of noise in the ego agent’s observation of their teammate’s goal vector. Returns are averaged over 1000 episodes × the 3 scenarios × 20 independent training runs. Right: the same increase compared for the extreme cases of no noise and teammate goal information entirely absent.
Ad-hoc collaboration often relies on identifying and adhering to shared conventions. However, when partners can follow multiple conventions, agents must do more than simply adapt; they must actively steer the team toward the most effective joint strategy. We present ConventionPlay, a reinforcement learning-based approach that extends cognitive hierarchies to include a diverse population of adaptive followers. By training against partners with varied capability limits, our agent learns to probe its partner's repertoire, leading the team when possible and following when necessary. Our results in canonical coordination tasks show that ConventionPlay achieves superior coordination efficiency, particularly in settings where conventions have differentiated payoffs.
Abhishek Sriraman, Eleni Vasilaki, Robert Loftin
University of Sheffield · Sheffield, United Kingdom
Achieving cooperation among self-interested agents remains a fundamental challenge in multi-agent reinforcement learning. Recent work showed that mutual cooperation can be induced between "learning-aware" agents that account for and shape the learning dynamics of their co-players. However, existing approaches typically rely on hardcoded, often inconsistent, assumptions about co-player learning rules or enforce a strict separation between "naive learners" updating on fast timescales and "meta-learners" observing these updates. Here, we demonstrate that the in-context learning capabilities of sequence models allow for co-player learning awareness without requiring hardcoded assumptions or explicit timescale separation. We show that training sequence model agents against a diverse distribution of co-players naturally induces in-context best-response strategies, effectively functioning as learning algorithms on the fast intra-episode timescale. We find that the cooperative mechanism identified in prior work-where vulnerability to extortion drives mutual shaping-emerges naturally in this setting: in-context adaptation renders agents vulnerable to extortion, and the resulting mutual pressure to shape the opponent's in-context learning dynamics resolves into the learning of cooperative behavior. Our results suggest that standard decentralized reinforcement learning on sequence models combined with co-player diversity provides a scalable path to learning cooperative behaviors.
Marissa A. Weis, Maciej Wołczyk, Rajai Nasser +4
Google, Paradigms of Intelligence Team · Santa Fe Institute
Learning to collaborate with previously unseen partners is a fundamental generalization challenge, known as Ad Hoc Teamwork (AHT). Existing methods often adopt a two-stage pipeline: first, a fixed population of teammates is generated, and second, an AHT agent is trained to collaborate with them. This separation limits coverage of behaviors and ignores whether the generated teammates are informative for the AHT agent to learn from. On the other hand, AHT agents are typically trained under the assumption that the training teammate set is uncontrollable, despite the fact that its composition strongly influences generalization. This paper presents a unified framework for AHT by reformulating the problem as an open-ended learning process between an AHT agent and an adversarial teammate generator. We introduce ROTATE, a regret-driven, open-ended training algorithm that alternates between improving the AHT agent and generating teammates that probe its collaboration deficiencies. Experiments across Overcooked and Level-Based Foraging tasks demonstrate that ROTATE substantially outperforms baselines on an unseen set of teammates, establishing a new standard for robust, generalizable teamwork.
Caroline Wang, Arrasy Rahman, Benjamin Nativi +4
Department of Computer Science The University of Texas at Austin · Visa AI · College of Computing and Data Science Nanyang Technological University +1