ConventionPlay: Capability-Limited Training for Robust Ad-Hoc Collaboration

Abhishek Sriraman; Eleni Vasilaki; Robert Loftin

cs.MAOct 8, 2026

ConventionPlay: Capability-Limited Training for Robust Ad-Hoc Collaboration

Authors: Abhishek Sriraman, Eleni Vasilaki, Robert Loftin

Organizations: Department of Computer Science Sheffield, UK · Pittsburgh, PA 15213, USA

Abstract

Ad-hoc collaboration often requires agents to identify and adhere to some shared convention within a cooperative task. Existing work on reinforcement learning (RL) for ad-hoc collaboration focuses on training agents that adapt to the conventions established by their partners. These methods fail to consider the possibility that while some partners might follow only a single fixed convention, others may themselves be capable of adapting to multiple conventions. Here we present ConventionPlay, an RL-based approach that teaches agents to discover their partner's optimal convention by training against a learned population of partners that exhibit different degrees of adaptability across conventions. Some of these partners follow a single, fixed convention, while others are able to adapt to a subset of the possible conventions for the task in question. The existence of partners that support a limited subset of conventions forces agents trained against this population to actively probe their partner's capabilities, and steer their partner towards the most effective joint strategy that they are capable of following. Our experimental results demonstrate that agents trained via ConventionPlay achieve superior performance to existing ad-hoc collaboration methods against test populations of partners that are compatible with multiple conventions.

Figures & tables

Figure 1: Comparison of the ConventionPlay pipeline with existing ad-hoc collaboration methods. Each circle represents an agent. Colors at the $K_{0}$ level indicate the convention to which each agent adheres. Adaptive agents are shown in red with an arrow from their training population. While baseline approaches diversify the $K_{0}$ population, we introduce diversity among $K_{1}$ agents to force the $K_{2}$ agent to infer its partner’s limited capability set.

Panel of Figure 2 — Figure 2: Payoff matrices for the Repeated Matrix Game. Left: Uniform payoffs across conventions. Middle: Partially differentiated payoffs across conventions. Right: Fully differentiated payoffs.

Environment	Var	FCP	SyKLRBR (L5)	SyKLRBR (L2)	CoMeDi (BC)	CoMeDi (BR)	LIPO	MEP	ConvPlay (matfact)	ConvPlay (spectral)
Matrix Game	Non-diff	95.16 $\pm$ 0.30	95.47 $\pm$ 0.17	95.55 $\pm$ 0.14	67.64 $\pm$ 0.21	68.86 $\pm$ 4.86	95.58 $\pm$ 0.09	69.44 $\pm$ 1.31	94.64 $\pm$ 0.15	94.42 $\pm$ 0.37
	Partial	41.76 $\pm$ 0.14	41.76 $\pm$ 0.20	41.82 $\pm$ 0.20	31.45 $\pm$ 0.11	30.57 $\pm$ 2.75	38.70 $\pm$ 0.10	34.18 $\pm$ 3.36	40.69 $\pm$ 1.31	41.21 $\pm$ 0.16
	Full	37.44 $\pm$ 0.06	37.23 $\pm$ 0.76	37.42 $\pm$ 0.52	29.00 $\pm$ 0.09	29.95 $\pm$ 0.67	33.90 $\pm$ 0.74	30.25 $\pm$ 0.83	36.35 $\pm$ 0.44	36.56 $\pm$ 0.57
PMR	Non-diff	89.42 $\pm$ 7.94	97.54 $\pm$ 0.24	97.46 $\pm$ 0.14	66.48 $\pm$ 3.18	80.17 $\pm$ 11.29	90.64 $\pm$ 6.80	52.11 $\pm$ 7.26	78.21 $\pm$ 7.14	91.07 $\pm$ 9.01
	Partial	60.33 $\pm$ 6.97	66.00 $\pm$ 0.15	65.98 $\pm$ 0.19	35.49 $\pm$ 5.25	53.16 $\pm$ 6.44	43.60 $\pm$ 0.27	53.96 $\pm$ 10.76	64.85 $\pm$ 2.12	65.80 $\pm$ 0.26
	Full	56.49 $\pm$ 6.22	54.27 $\pm$ 6.05	32.68 $\pm$ 7.56	16.56 $\pm$ 7.40	37.81 $\pm$ 10.94	47.17 $\pm$ 7.54	38.17 $\pm$ 2.51	57.92 $\pm$ 3.62	60.58 $\pm$ 0.11

Table 1: Mean normalised episodic return on

{\mathcal{P}}^{(0)}_{\text{test}}

partners. Each cell reports mean

\pm

1 s.d. (unbiased sample standard deviation,

n-1

denominator) across

n=5

seeds. Bold marks the row’s best value.

Environment	Var	FCP	SyKLRBR (L5)	SyKLRBR (L2)	CoMeDi (BC)	CoMeDi (BR)	LIPO	MEP	ConvPlay (matfact)	ConvPlay (spectral)
Matrix Game	Non-diff	65.93 $\pm$ 2.69	63.78 $\pm$ 2.06	63.64 $\pm$ 2.56	66.48 $\pm$ 1.59	70.66 $\pm$ 3.31	60.41 $\pm$ 5.12	71.98 $\pm$ 4.02	86.19 $\pm$ 1.76	85.52 $\pm$ 1.91
	Partial	26.03 $\pm$ 3.90	25.65 $\pm$ 2.41	25.56 $\pm$ 2.50	47.05 $\pm$ 0.21	38.23 $\pm$ 1.58	30.09 $\pm$ 0.91	39.86 $\pm$ 2.31	46.50 $\pm$ 0.93	50.47 $\pm$ 1.52
	Full	24.03 $\pm$ 1.31	25.74 $\pm$ 4.37	23.97 $\pm$ 4.24	48.12 $\pm$ 0.33	33.33 $\pm$ 1.22	33.47 $\pm$ 2.68	46.27 $\pm$ 0.89	51.52 $\pm$ 1.37	52.63 $\pm$ 0.41
PMR	Non-diff	89.33 $\pm$ 5.75	94.39 $\pm$ 0.37	94.43 $\pm$ 0.52	63.70 $\pm$ 3.95	80.00 $\pm$ 8.47	90.10 $\pm$ 5.53	62.91 $\pm$ 8.20	86.30 $\pm$ 4.50	93.46 $\pm$ 2.14
	Partial	58.44 $\pm$ 5.35	62.25 $\pm$ 1.10	62.93 $\pm$ 1.93	40.57 $\pm$ 3.44	57.89 $\pm$ 7.25	47.82 $\pm$ 1.19	56.30 $\pm$ 8.30	74.82 $\pm$ 1.16	72.58 $\pm$ 1.24
	Full	53.73 $\pm$ 5.81	53.08 $\pm$ 4.95	36.07 $\pm$ 3.68	20.99 $\pm$ 10.27	39.29 $\pm$ 9.71	47.62 $\pm$ 5.66	43.74 $\pm$ 2.96	65.54 $\pm$ 1.41	66.11 $\pm$ 1.28

Table 2: Mean normalised episodic return on

{\mathcal{P}}^{(1)}_{\text{sampled}}

partners. Each cell reports mean

\pm

1 s.d. (unbiased sample standard deviation,

n-1

denominator) across

n=5

seeds. Bold marks the row’s best value.

Figure 3: Normalized reward under full payoff differentiation across all three environments (Matrix Game, PMR, Task-Relay) and all three evaluation populations ( ${\mathcal{P}}^{(0)}_{\text{test}}$ , ${\mathcal{P}}^{(1)}_{\text{sampled}}$ , ${\mathcal{P}}^{(1)}_{\text{heur}}$ ). The same pattern can be observed in all settings: ConventionPlay shows a modest cost against fixed-convention partners, but outperforms baselines against various types of adaptive partners.

Figure 4: Left: fraction of various methods applying a probing (i.e., uncoordinated) action to a fixed-convention partner; ConventionPlay can be seen to probe for significantly longer, explaining its reduced performance with the ${\mathcal{P}}^{(0)}$ population. Right: performance on a modified ${\mathcal{P}}^{(1)}_{\text{sampled}}$ in which partners adapt for only a fixed number of steps before committing, tracing ConventionPlay ’s advantage across a spectrum of partner adaptivity.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Task-Relay task structure and an example resolved trajectory.

Method	Matrix Game	PMR	Task-Relay
MEP	20	12	10
LIPO	20	12	5
CoMeDi (BC) / CoMeDi (BR)	20	12	10
FCP	40	30	30
SyKLRBR	40	30	30
ConventionPlay	40	30	30

Appendix

Table 5:

K_{0}

population size for each method, by environment.

	Matrix Game	PMR	Task-Relay
$K_{1}$ test population size	50	12	10
$q$ (number of conventions)	8	6	5

Appendix

Table 6:

K_{1}

test population size and number of conventions

q

, by environment.

Figure 6: Makeup of the generated $K_{1}$ subsets. Top left: cross-play matrix of the $K_{0}$ population, clustered into convention sets. Top middle: sampling of individual policies from each convention to form training distributions for individual $K_{1}$ agents. Top right: distribution of maximum achievable returns across $K_{1}$ subsets, reflecting varying capability levels. Bottom left: Jaccard similarity between generated $K_{1}$ subsets. Bottom right: $K_{1}$ subset sizes and frequency with which each $K_{0}$ policy appears across all $K_{1}$ training distributions.

Panel of Figure 7 — Figure 7: Setup for the convention-rebalancing experiment: a matrix-game variant designed so that some conventions are harder to discover than others, and the resulting imbalance in a self-play $K_{0}$ population.

Figure 8: Left: $K_{2}$ partner-sampling marginal probability by true convention for the Matrix Game variant, comparing naive best-response sampling against matrix-factorization, spectral-clustering, and k-means rebalancing, relative to the uniform target (dotted line). Right: normalized episodic return for a best-response policy trained under each rebalancing scheme, evaluated against the uniform ${\mathcal{P}}^{(0)}_{\text{test}}$ population, with an oracle upper bound (dashed line) given by the balanced test population’s own self-crossplay.

Figure 9: Normalized episodic return for a best-response policy trained under each rebalancing scheme (MatFact, Spectral, k-means), as a function of the specified number of conventions $q$ , evaluated against the uniform ${\mathcal{P}}^{(0)}_{\text{test}}$ population for the Matrix Game variant of Appendix D.1 . The dashed vertical line marks the ground-truth number of conventions. All methods improve over the unrebalanced baseline even when $q$ is mis-specified, with the largest gains at the correct value; MatFact remains comparatively resilient to overspecification of $q$ .

Environment	Var	FCP	SyKLRBR (L5)	SyKLRBR (L2)	CoMeDi (BC)	CoMeDi (BR)	LIPO	MEP	ConvPlay (matfact)	ConvPlay (spectral)
Matrix Game	Non-diff	93.04 $\pm$ 0.83	93.47 $\pm$ 0.43	94.06 $\pm$ 0.50	44.25 $\pm$ 1.88	87.25 $\pm$ 2.39	94.02 $\pm$ 0.38	81.26 $\pm$ 18.89	93.62 $\pm$ 0.49	93.01 $\pm$ 0.68
	Partial	27.07 $\pm$ 0.98	27.77 $\pm$ 1.07	27.75 $\pm$ 0.92	9.11 $\pm$ 0.32	13.29 $\pm$ 1.66	27.60 $\pm$ 0.54	22.85 $\pm$ 5.02	28.75 $\pm$ 4.37	29.54 $\pm$ 0.60
	Full	14.87 $\pm$ 0.58	15.38 $\pm$ 2.55	15.91 $\pm$ 0.96	8.69 $\pm$ 0.17	12.97 $\pm$ 1.12	14.36 $\pm$ 2.18	14.25 $\pm$ 1.00	17.60 $\pm$ 2.15	17.24 $\pm$ 1.46
PMR	Non-diff	62.91 $\pm$ 6.71	68.17 $\pm$ 1.21	68.30 $\pm$ 0.93	51.17 $\pm$ 3.28	63.83 $\pm$ 7.42	68.52 $\pm$ 1.44	47.63 $\pm$ 2.14	60.78 $\pm$ 4.33	64.94 $\pm$ 7.64
	Partial	32.49 $\pm$ 3.39	32.18 $\pm$ 1.03	32.73 $\pm$ 1.06	21.83 $\pm$ 1.91	28.78 $\pm$ 3.08	26.10 $\pm$ 1.30	30.84 $\pm$ 4.90	39.18 $\pm$ 2.49	39.26 $\pm$ 1.33
	Full	27.88 $\pm$ 4.42	27.52 $\pm$ 2.34	27.20 $\pm$ 2.77	7.96 $\pm$ 7.46	23.64 $\pm$ 2.72	25.84 $\pm$ 1.61	19.74 $\pm$ 3.37	34.42 $\pm$ 1.48	36.64 $\pm$ 1.50

Appendix

Table 7: Mean normalised episodic return on

{\mathcal{P}}^{(1)}_{\text{heur}}

partners. Each cell reports mean

\pm

1 s.d. (unbiased sample standard deviation,

n-1

denominator) across

n=5

seeds. Bold marks the row’s best value.

Environment	Var	FCP	SyKLRBR (L5)	SyKLRBR (L2)	CoMeDi (BC)	CoMeDi (BR)	LIPO	MEP	ConvPlay (matfact)	ConvPlay (spectral)
Matrix Game	Non-diff	95.16 $\pm$ 0.30	95.47 $\pm$ 0.17	95.55 $\pm$ 0.14	67.64 $\pm$ 0.21	68.86 $\pm$ 4.86	95.58 $\pm$ 0.09	69.44 $\pm$ 1.31	94.64 $\pm$ 0.15	94.42 $\pm$ 0.37
	Partial	95.35 $\pm$ 0.55	95.25 $\pm$ 0.35	95.41 $\pm$ 0.40	55.77 $\pm$ 0.24	53.83 $\pm$ 6.64	83.15 $\pm$ 0.30	68.33 $\pm$ 10.25	90.19 $\pm$ 5.04	92.12 $\pm$ 0.37
	Full	95.26 $\pm$ 0.26	92.53 $\pm$ 5.88	93.07 $\pm$ 5.35	44.80 $\pm$ 0.16	48.01 $\pm$ 1.97	64.11 $\pm$ 4.55	56.18 $\pm$ 0.47	87.36 $\pm$ 4.78	86.02 $\pm$ 5.34
PMR	Non-diff	90.05 $\pm$ 7.96	98.20 $\pm$ 0.24	98.11 $\pm$ 0.14	66.95 $\pm$ 3.24	80.55 $\pm$ 11.43	91.27 $\pm$ 6.79	52.50 $\pm$ 7.26	78.71 $\pm$ 7.25	91.70 $\pm$ 9.04
	Partial	90.14 $\pm$ 8.64	98.20 $\pm$ 0.13	98.16 $\pm$ 0.19	51.02 $\pm$ 7.01	73.20 $\pm$ 8.43	63.66 $\pm$ 0.32	83.68 $\pm$ 13.72	94.31 $\pm$ 7.10	97.53 $\pm$ 0.24
	Full	87.40 $\pm$ 13.70	88.43 $\pm$ 8.62	78.20 $\pm$ 10.79	30.13 $\pm$ 9.03	63.62 $\pm$ 14.91	70.07 $\pm$ 8.67	55.67 $\pm$ 9.16	90.22 $\pm$ 8.88	96.69 $\pm$ 0.21

Appendix

Table 8: Coordination efficiency

\eta

(%) on

{\mathcal{P}}^{(0)}_{\text{test}}

partners. Each cell reports mean

\pm

1 s.d. (unbiased sample standard deviation,

n-1

denominator) across

n=5

seeds. Bold marks the row’s best value.

Environment	Var	FCP	SyKLRBR (L5)	SyKLRBR (L2)	CoMeDi (BC)	CoMeDi (BR)	LIPO	MEP	ConvPlay (matfact)	ConvPlay (spectral)
Matrix Game	Non-diff	66.88 $\pm$ 2.76	64.68 $\pm$ 2.10	64.54 $\pm$ 2.61	67.45 $\pm$ 1.62	71.73 $\pm$ 3.36	61.28 $\pm$ 5.21	73.10 $\pm$ 4.09	87.47 $\pm$ 1.79	86.81 $\pm$ 1.93
	Partial	46.69 $\pm$ 6.71	46.24 $\pm$ 3.65	46.04 $\pm$ 4.17	67.40 $\pm$ 0.44	59.71 $\pm$ 3.32	53.29 $\pm$ 1.11	63.24 $\pm$ 5.07	76.38 $\pm$ 1.84	84.23 $\pm$ 1.82
	Full	39.32 $\pm$ 1.84	42.34 $\pm$ 6.18	39.32 $\pm$ 6.25	59.74 $\pm$ 0.52	50.83 $\pm$ 1.96	51.35 $\pm$ 4.97	64.18 $\pm$ 2.11	73.99 $\pm$ 1.15	81.11 $\pm$ 0.92
PMR	Non-diff	93.71 $\pm$ 6.04	99.02 $\pm$ 0.39	99.07 $\pm$ 0.54	66.82 $\pm$ 4.12	83.90 $\pm$ 8.91	94.51 $\pm$ 5.83	65.86 $\pm$ 8.60	90.53 $\pm$ 4.72	98.03 $\pm$ 2.26
	Partial	73.24 $\pm$ 5.58	77.58 $\pm$ 1.22	78.43 $\pm$ 2.32	47.97 $\pm$ 5.34	71.37 $\pm$ 7.19	60.96 $\pm$ 1.51	70.84 $\pm$ 9.72	92.03 $\pm$ 1.65	90.17 $\pm$ 1.54
	Full	74.78 $\pm$ 7.65	73.55 $\pm$ 6.69	52.59 $\pm$ 5.85	31.71 $\pm$ 8.82	55.86 $\pm$ 11.29	67.04 $\pm$ 7.07	59.92 $\pm$ 3.88	88.94 $\pm$ 3.00	91.27 $\pm$ 1.73

Appendix

Table 9: Coordination efficiency

\eta

(%) on

{\mathcal{P}}^{(1)}_{\text{sampled}}

partners. Each cell reports mean

\pm

1 s.d. (unbiased sample standard deviation,

n-1

denominator) across

n=5

seeds. Bold marks the row’s best value.

Environment	Var	FCP	SyKLRBR (L5)	SyKLRBR (L2)	CoMeDi (BC)	CoMeDi (BR)	LIPO	MEP	ConvPlay (matfact)	ConvPlay (spectral)
Matrix Game	Non-diff	93.77 $\pm$ 0.83	94.22 $\pm$ 0.43	94.80 $\pm$ 0.50	44.62 $\pm$ 1.89	87.93 $\pm$ 2.41	94.76 $\pm$ 0.39	81.90 $\pm$ 19.03	94.36 $\pm$ 0.49	93.74 $\pm$ 0.68
	Partial	73.68 $\pm$ 1.64	74.50 $\pm$ 1.32	74.66 $\pm$ 1.25	10.26 $\pm$ 0.67	20.37 $\pm$ 4.23	74.51 $\pm$ 0.63	56.05 $\pm$ 15.59	68.33 $\pm$ 13.38	75.19 $\pm$ 1.00
	Full	49.06 $\pm$ 1.02	46.20 $\pm$ 9.34	48.91 $\pm$ 4.48	9.38 $\pm$ 0.31	26.12 $\pm$ 2.95	27.99 $\pm$ 3.60	32.67 $\pm$ 3.80	42.12 $\pm$ 8.41	51.01 $\pm$ 3.26
PMR	Non-diff	86.32 $\pm$ 9.15	93.47 $\pm$ 1.66	93.68 $\pm$ 1.30	70.08 $\pm$ 4.49	87.45 $\pm$ 10.39	93.94 $\pm$ 1.96	64.77 $\pm$ 2.91	83.32 $\pm$ 5.95	88.98 $\pm$ 10.65
	Partial	65.43 $\pm$ 7.36	65.73 $\pm$ 1.84	66.81 $\pm$ 1.76	36.71 $\pm$ 4.63	55.79 $\pm$ 6.80	52.83 $\pm$ 2.26	63.85 $\pm$ 8.86	75.41 $\pm$ 6.62	75.83 $\pm$ 2.72
	Full	62.86 $\pm$ 9.40	62.71 $\pm$ 4.36	61.84 $\pm$ 5.23	19.93 $\pm$ 18.25	53.51 $\pm$ 6.56	57.06 $\pm$ 3.58	44.10 $\pm$ 7.84	73.11 $\pm$ 3.92	77.68 $\pm$ 3.22

Appendix

Table 10: Coordination efficiency

\eta

(%) on

{\mathcal{P}}^{(1)}_{\text{heur}}

partners. Each cell reports mean

\pm

1 s.d. (unbiased sample standard deviation,

n-1

denominator) across

n=5

seeds. Bold marks the row’s best value.

Figure 10: Trajectories in PMR for the ad-hoc agent (red) paired with a heuristic partner (blue), comparing FCP (top row) and ConventionPlay (bottom row). Left column: a fixed $K_{0}$ partner committed to landmark 0. Remaining columns: tit-for-tat partners that default to landmark 0 before switching to landmark 2, 3, or 4 respectively, conditioned on the position of the partner agent. FCP agents infer their partner’s initially preferred convention and coordinate accordingly. ConventionPlay ’s trajectories are visibly more winding, showing its ability to both lead a capability-limited adaptive partner or follow a fixed-convention one.

Figure 11: Effect of $K_{1}$ population size on coordination efficiency with ${\mathcal{P}}^{(1)}_{\text{sampled}}$ partners. Coordination efficiency $\eta\in[0,1]$ is the fraction of a partner’s achievable return obtained (defined in Appendix D.3 ). ConventionPlay exhibits substantial gains up to moderate diversity, followed by diminishing returns. FCP—invariant to $K_{1}$ size—is shown as a horizontal line for reference. Error bars show $\pm$ 1 sample standard deviation, where each seed’s value is mean efficiency averaged over all test partners.

Explore similar work

Apr 20, 2026cs.MA

ConventionPlay: Capability-Limited Training for Robust Ad-Hoc Collaboration

Ad-hoc collaboration often relies on identifying and adhering to shared conventions. However, when partners can follow multiple conventions, agents must do more than simply adapt; they must actively steer the team toward the most effective joint strategy. We present ConventionPlay, a reinforcement learning-based approach that extends cognitive hierarchies to include a diverse population of adaptive followers. By training against partners with varied capability limits, our agent learns to probe its partner's repertoire, leading the team when possible and following when necessary. Our results in canonical coordination tasks show that ConventionPlay achieves superior coordination efficiency, particularly in settings where conventions have differentiated payoffs.

Abhishek Sriraman, Eleni Vasilaki, Robert Loftin

University of Sheffield · Sheffield, United Kingdom

Multi-Agent Coordination Multi-Agent Collaboration Ad Hoc Teamwork Cooperative MARL

Ask PDF

Jul 29, 2026cs.AI

Partner Capability Estimation for Task-Agnostic Adaptation in Ad-Hoc Teamwork

Effective collaboration with novel and diverse partners is a crucial skill for autonomous agents. Most current ad-hoc teamwork (AHT) approaches assume that agents will collaborate on a single, fixed task and that the partner's capabilities, their ability to successfully execute the desired action, are already known. In reality, a partner's true capabilities are often hidden, and human collaborators may act sub-optimally on tasks with multiple valid strategies. To address these limitations, we extend ad-hoc teamwork into a multi-task setting by re-framing it as a problem of joint planning with decentralised execution under hidden partner capabilities. We introduce CE-CM (Capability Estimation via Contextual Models), an approximate Bayesian method that infers task-invariant capability vectors. By using simulation-based sampling, the agent estimates capabilities and induces a contextual Multi-agent Markov Decision Processes for planning. This approach requires no population pre-training and refines its beliefs online from just a few tasks. To account for human unpredictability, we propose CE-CM-Div, an extension that evaluates capability hypotheses against diverse planner rollouts rather than a single optimal trajectory. Simulated experiments demonstrate that CE-CM rapidly recovers hidden capabilities, reduces infeasible action assignments, and adapts to changes over time. Furthermore, in an offline human study of 225 trajectories from 15 participants, CE-CM-Div substantially improved capability estimates over the baseline CE-CM method. Our results suggest capability-based modelling is a promising interpretable, task-agnostic representation in the studied settings, demonstrating that accounting for behavioural diversity is essential for robust human-AI teaming.

Peter Tisnikar, Maja Swieczkowska, Benteng Ma +2

aDepartment of Informatics, King’s College London, 30 Aldwych, London, WC2B 4BG, England, United Kingdom · bDepartment of Engineering Science, University of Oxford, Parks Road, Oxford, OX1 3PJ, England, United Kingdom

Bayesian Inference Ad Hoc Teamwork Multi-Agent Planning Human-AI Collaboration

Ask PDF

May 29, 2025cs.AI

ROTATE: Regret-driven Open-ended Training for Ad Hoc Teamwork

Learning to collaborate with previously unseen partners is a fundamental generalization challenge, known as Ad Hoc Teamwork (AHT). Existing methods often adopt a two-stage pipeline: first, a fixed population of teammates is generated, and second, an AHT agent is trained to collaborate with them. This separation limits coverage of behaviors and ignores whether the generated teammates are informative for the AHT agent to learn from. On the other hand, AHT agents are typically trained under the assumption that the training teammate set is uncontrollable, despite the fact that its composition strongly influences generalization. This paper presents a unified framework for AHT by reformulating the problem as an open-ended learning process between an AHT agent and an adversarial teammate generator. We introduce ROTATE, a regret-driven, open-ended training algorithm that alternates between improving the AHT agent and generating teammates that probe its collaboration deficiencies. Experiments across Overcooked and Level-Based Foraging tasks demonstrate that ROTATE substantially outperforms baselines on an unseen set of teammates, establishing a new standard for robust, generalizable teamwork.

Caroline Wang, Arrasy Rahman, Benjamin Nativi +4

Department of Computer Science The University of Texas at Austin · Visa AI · College of Computing and Data Science Nanyang Technological University +1

Multi-Agent Collaboration Ad Hoc Teamwork Cooperative MARL

Ask PDF