Learning When to Cooperate Under Heterogeneous Goals

Max Taylor-Davies; Neil Bramley; Christopher G. Lucas

cs.MAMar 7, 2026

Learning When to Cooperate Under Heterogeneous Goals

Authors: Max Taylor-Davies, Neil Bramley, Christopher G. Lucas

Organizations: School of Informatics, The University of Edinburgh · Department of Psychology, The University of Edinburgh

Abstract

A significant element of human cooperative intelligence lies in our ability to identify opportunities for fruitful collaboration; and conversely to recognise when the task at hand is better pursued alone. Research on flexible cooperation in machines has left this meta-level problem largely unexplored, despite its importance for successful collaboration in heterogeneous open-ended environments. Here, we extend the typical Ad Hoc Teamwork (AHT) setting to incorporate the idea of agents having heterogeneous goals that in any given scenario may or may not overlap. We introduce a novel approach to learning policies in this setting, based on a hierarchical combination of imitation and reinforcement learning, and show that it outperforms baseline methods across extended versions of two cooperative environments. We also investigate the contribution of an auxiliary component that learns to model teammates by predicting their actions, finding that its effect on performance is inversely related to the amount of observable information about teammate goals.

Figures & tables

Figure 1: An illustration of the goal space under three different scenarios, with the set of ‘worthwhile’ (rewarding and in-principle achievable) goals highlighted in green.

\mathcal{G}^{\text{ego}}

denotes the set of goals that would produce reward for the ego agent;

\mathcal{G}^{\text{teammates}}

the set of goals that would produce reward for at least one teammate; and

\mathcal{G}^{\text{solo}}

the set of goals that can be achieved through the effort of a single agent.

Figure 2: (A) The hierarchical architecture of GRILL (B) The encoder-decoder architecture optimised offline in stage 1, from which the action decoder becomes the low-level policy $\pi_{\text{action}}$ in stage 2 (C) The auxiliary modelling component used in GRILL-M (but not GRILL)

Figure 3: Example frames from the two AHT environments we extend.

Figure 4: Top: evaluation returns relative to oracle policy, measured over 1000 episodes

\times

the 3 scenarios

\times

20 independent training runs. Mean values are given by the horizontal markers, while underlaid violin plots show the KDE. Agents were trained for 1e7 and 5e7 timesteps on CR and LBF respectively. Bottom: from the same set of evaluation episodes, the difference in proportion of non-solo goals pursued between the full- and no-overlap scenarios.

Figure 5: The distribution of goals attempted by the ego agent during evaluation. For CR, ‘attempt’ means the agent occupied one of the goal tiles; for LBF, it means the agent used the ‘collect’ action while adjacent to a fruit.

Figure 6: A visualisation of trajectories sampled from trained PPO and GRILL agents in the no- and full-overlap scenarios of the cooperative reaching environment. Trajectories are sampled from random initial locations, and a small amount of noise is applied to each path in order to better visualise the distributions.

Figure 7: Left: the percentage increase in mean evaluation return in the LBF environment for GRILL-M vs GRILL, as a function of the amount of noise in the ego agent’s observation of their teammate’s goal vector. Returns are averaged over 1000 episodes

\times

the 3 scenarios

\times

20 independent training runs. Right: the same increase compared for the extreme cases of no noise and teammate goal information entirely absent.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

method	description	value
LIAM	encoder hidden	(64, 64) (LSTM, dense)
LIAM	encoder output	10
LIAM	decoder hidden	(64, 64)
OMG	MLP hidden	128
OMG	MLP output	32
OMG	RNN hidden	(128, 128)

Appendix

Table 8

hyperparameter	value
batch size	1024
minibatches	8
epochs	2
max grad norm	0.5
learning rate	1e-4
discount factor ( $\gamma$ )	0.99

Appendix

Table 9

method	hyperparameter	value
LIAM	encoder-decoder learning rate	1e-3
OMG	RNN, VAE, CVAE learning rate	1e-3
GRILL-M	encoder-decoder learning rate	1e-3

Appendix

Table 10

Multi-Agent Collaboration

Multi-agent collaboration studies how autonomous agents coordinate, share information, and combine capabilities to solve tasks together.

See all 311 papers

ConventionPlay: Capability-Limited Training for Robust Ad-Hoc Collaboration
Apr 20, 2026Abhishek Sriraman, Eleni Vasilaki, Robert LoftinMulti-Agent Coordination Multi-Agent Collaboration
University of Sheffield · Sheffield, United Kingdom
Multi-agent cooperation through in-context co-player inference
Feb 18, 2026Marissa A. Weis, Maciej Wołczyk, Rajai Nasser +4Cooperative MARL Decentralized MARL
Google, Paradigms of Intelligence Team · Santa Fe Institute
ROTATE: Regret-driven Open-ended Training for Ad Hoc Teamwork
May 29, 2025Caroline Wang, Arrasy Rahman, Benjamin Nativi +4Multi-Agent Collaboration Ad Hoc Teamwork
Department of Computer Science The University of Texas at Austin · Visa AI · College of Computing and Data Science Nanyang Technological University +1

Apr 20, 2026J/K move · Enter open · S save

ConventionPlay: Capability-Limited Training for Robust Ad-Hoc Collaboration

Abhishek Sriraman, Eleni Vasilaki, Robert Loftin

University of Sheffield · Sheffield, United Kingdom

Multi-Agent Coordination Multi-Agent Collaboration Ad Hoc Teamwork Cooperative MARL

Open paper AskPDF arXiv

Ad-hoc collaboration often relies on identifying and adhering to shared conventions. However, when partners can follow multiple conventions, agents must do more than simply adapt; they must actively steer the team toward the most effective joint strategy. We present ConventionPlay, a reinforcement learning-based approach that extends cognitive hierarchies to include a diverse population of adaptive followers. By training against partners with varied capability limits, our agent learns to probe its partner's repertoire, leading the team when possible and following when necessary. Our results in canonical coordination tasks show that ConventionPlay achieves superior coordination efficiency, particularly in settings where conventions have differentiated payoffs.

Abstract

Multi-Agent Collaboration

Explore similar work

ConventionPlay: Capability-Limited Training for Robust Ad-Hoc Collaboration

Multi-agent cooperation through in-context co-player inference

ROTATE: Regret-driven Open-ended Training for Ad Hoc Teamwork