ConventionPlay: Capability-Limited Training for Robust Ad-Hoc Collaboration

Abhishek Sriraman; Eleni Vasilaki; Robert Loftin

cs.MAOct 8, 2026

ConventionPlay: Capability-Limited Training for Robust Ad-Hoc Collaboration

Authors: Abhishek Sriraman, Eleni Vasilaki, Robert Loftin

Organizations: Department of Computer Science Sheffield, UK · Pittsburgh, PA 15213, USA

Abstract

Ad-hoc collaboration often requires agents to identify and adhere to some shared convention within a cooperative task. Existing work on reinforcement learning (RL) for ad-hoc collaboration focuses on training agents that adapt to the conventions established by their partners. These methods fail to consider the possibility that while some partners might follow only a single fixed convention, others may themselves be capable of adapting to multiple conventions. Here we present ConventionPlay, an RL-based approach that teaches agents to discover their partner's optimal convention by training against a learned population of partners that exhibit different degrees of adaptability across conventions. Some of these partners follow a single, fixed convention, while others are able to adapt to a subset of the possible conventions for the task in question. The existence of partners that support a limited subset of conventions forces agents trained against this population to actively probe their partner's capabilities, and steer their partner towards the most effective joint strategy that they are capable of following. Our experimental results demonstrate that agents trained via ConventionPlay achieve superior performance to existing ad-hoc collaboration methods against test populations of partners that are compatible with multiple conventions.

Figures & tables

Figure 1: Comparison of the ConventionPlay pipeline with existing ad-hoc collaboration methods. Each circle represents an agent. Colors at the $K_{0}$ level indicate the convention to which each agent adheres. Adaptive agents are shown in red with an arrow from their training population. While baseline approaches diversify the $K_{0}$ population, we introduce diversity among $K_{1}$ agents to force the $K_{2}$ agent to infer its partner’s limited capability set.

Panel of Figure 2 — Figure 2: Payoff matrices for the Repeated Matrix Game. Left: Uniform payoffs across conventions. Middle: Partially differentiated payoffs across conventions. Right: Fully differentiated payoffs.

Environment	Var	FCP	SyKLRBR (L5)	SyKLRBR (L2)	CoMeDi (BC)	CoMeDi (BR)	LIPO	MEP	ConvPlay (matfact)	ConvPlay (spectral)
Matrix Game	Non-diff	95.16 $\pm$ 0.30	95.47 $\pm$ 0.17	95.55 $\pm$ 0.14	67.64 $\pm$ 0.21	68.86 $\pm$ 4.86	95.58 $\pm$ 0.09	69.44 $\pm$ 1.31	94.64 $\pm$ 0.15	94.42 $\pm$ 0.37
	Partial	41.76 $\pm$ 0.14	41.76 $\pm$ 0.20	41.82 $\pm$ 0.20	31.45 $\pm$ 0.11	30.57 $\pm$ 2.75	38.70 $\pm$ 0.10	34.18 $\pm$ 3.36	40.69 $\pm$ 1.31	41.21 $\pm$ 0.16
	Full	37.44 $\pm$ 0.06	37.23 $\pm$ 0.76	37.42 $\pm$ 0.52	29.00 $\pm$ 0.09	29.95 $\pm$ 0.67	33.90 $\pm$ 0.74	30.25 $\pm$ 0.83	36.35 $\pm$ 0.44	36.56 $\pm$ 0.57
PMR	Non-diff	89.42 $\pm$ 7.94	97.54 $\pm$ 0.24	97.46 $\pm$ 0.14	66.48 $\pm$ 3.18	80.17 $\pm$ 11.29	90.64 $\pm$ 6.80	52.11 $\pm$ 7.26	78.21 $\pm$ 7.14	91.07 $\pm$ 9.01
	Partial	60.33 $\pm$ 6.97	66.00 $\pm$ 0.15	65.98 $\pm$ 0.19	35.49 $\pm$ 5.25	53.16 $\pm$ 6.44	43.60 $\pm$ 0.27	53.96 $\pm$ 10.76	64.85 $\pm$ 2.12	65.80 $\pm$ 0.26
	Full	56.49 $\pm$ 6.22	54.27 $\pm$ 6.05	32.68 $\pm$ 7.56	16.56 $\pm$ 7.40	37.81 $\pm$ 10.94	47.17 $\pm$ 7.54	38.17 $\pm$ 2.51	57.92 $\pm$ 3.62	60.58 $\pm$ 0.11

Table 1: Mean normalised episodic return on

{\mathcal{P}}^{(0)}_{\text{test}}

partners. Each cell reports mean

\pm

1 s.d. (unbiased sample standard deviation,

n-1

denominator) across

n=5

seeds. Bold marks the row’s best value.

Environment	Var	FCP	SyKLRBR (L5)	SyKLRBR (L2)	CoMeDi (BC)	CoMeDi (BR)	LIPO	MEP	ConvPlay (matfact)	ConvPlay (spectral)
Matrix Game	Non-diff	65.93 $\pm$ 2.69	63.78 $\pm$ 2.06	63.64 $\pm$ 2.56	66.48 $\pm$ 1.59	70.66 $\pm$ 3.31	60.41 $\pm$ 5.12	71.98 $\pm$ 4.02	86.19 $\pm$ 1.76	85.52 $\pm$ 1.91
	Partial	26.03 $\pm$ 3.90	25.65 $\pm$ 2.41	25.56 $\pm$ 2.50	47.05 $\pm$ 0.21	38.23 $\pm$ 1.58	30.09 $\pm$ 0.91	39.86 $\pm$ 2.31	46.50 $\pm$ 0.93	50.47 $\pm$ 1.52
	Full	24.03 $\pm$ 1.31	25.74 $\pm$ 4.37	23.97 $\pm$ 4.24	48.12 $\pm$ 0.33	33.33 $\pm$ 1.22	33.47 $\pm$ 2.68	46.27 $\pm$ 0.89	51.52 $\pm$ 1.37	52.63 $\pm$ 0.41
PMR	Non-diff	89.33 $\pm$ 5.75	94.39 $\pm$ 0.37	94.43 $\pm$ 0.52	63.70 $\pm$ 3.95	80.00 $\pm$ 8.47	90.10 $\pm$ 5.53	62.91 $\pm$ 8.20	86.30 $\pm$ 4.50	93.46 $\pm$ 2.14
	Partial	58.44 $\pm$ 5.35	62.25 $\pm$ 1.10	62.93 $\pm$ 1.93	40.57 $\pm$ 3.44	57.89 $\pm$ 7.25	47.82 $\pm$ 1.19	56.30 $\pm$ 8.30	74.82 $\pm$ 1.16	72.58 $\pm$ 1.24
	Full	53.73 $\pm$ 5.81	53.08 $\pm$ 4.95	36.07 $\pm$ 3.68	20.99 $\pm$ 10.27	39.29 $\pm$ 9.71	47.62 $\pm$ 5.66	43.74 $\pm$ 2.96	65.54 $\pm$ 1.41	66.11 $\pm$ 1.28

Table 2: Mean normalised episodic return on

{\mathcal{P}}^{(1)}_{\text{sampled}}

partners. Each cell reports mean

\pm

1 s.d. (unbiased sample standard deviation,

n-1

denominator) across

n=5

seeds. Bold marks the row’s best value.

Figure 3: Normalized reward under full payoff differentiation across all three environments (Matrix Game, PMR, Task-Relay) and all three evaluation populations ( ${\mathcal{P}}^{(0)}_{\text{test}}$ , ${\mathcal{P}}^{(1)}_{\text{sampled}}$ , ${\mathcal{P}}^{(1)}_{\text{heur}}$ ). The same pattern can be observed in all settings: ConventionPlay shows a modest cost against fixed-convention partners, but outperforms baselines against various types of adaptive partners.

Figure 4: Left: fraction of various methods applying a probing (i.e., uncoordinated) action to a fixed-convention partner; ConventionPlay can be seen to probe for significantly longer, explaining its reduced performance with the ${\mathcal{P}}^{(0)}$ population. Right: performance on a modified ${\mathcal{P}}^{(1)}_{\text{sampled}}$ in which partners adapt for only a fixed number of steps before committing, tracing ConventionPlay ’s advantage across a spectrum of partner adaptivity.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Task-Relay task structure and an example resolved trajectory.

Method	Matrix Game	PMR	Task-Relay
MEP	20	12	10
LIPO	20	12	5
CoMeDi (BC) / CoMeDi (BR)	20	12	10
FCP	40	30	30
SyKLRBR	40	30	30
ConventionPlay	40	30	30

Appendix

Table 5:

K_{0}

population size for each method, by environment.

	Matrix Game	PMR	Task-Relay
$K_{1}$ test population size	50	12	10
$q$ (number of conventions)	8	6	5

Appendix

Table 6:

K_{1}

test population size and number of conventions

q

, by environment.

Figure 6: Makeup of the generated $K_{1}$ subsets. Top left: cross-play matrix of the $K_{0}$ population, clustered into convention sets. Top middle: sampling of individual policies from each convention to form training distributions for individual $K_{1}$ agents. Top right: distribution of maximum achievable returns across $K_{1}$ subsets, reflecting varying capability levels. Bottom left: Jaccard similarity between generated $K_{1}$ subsets. Bottom right: $K_{1}$ subset sizes and frequency with which each $K_{0}$ policy appears across all $K_{1}$ training distributions.

Panel of Figure 7 — Figure 7: Setup for the convention-rebalancing experiment: a matrix-game variant designed so that some conventions are harder to discover than others, and the resulting imbalance in a self-play $K_{0}$ population.

Figure 8: Left: $K_{2}$ partner-sampling marginal probability by true convention for the Matrix Game variant, comparing naive best-response sampling against matrix-factorization, spectral-clustering, and k-means rebalancing, relative to the uniform target (dotted line). Right: normalized episodic return for a best-response policy trained under each rebalancing scheme, evaluated against the uniform ${\mathcal{P}}^{(0)}_{\text{test}}$ population, with an oracle upper bound (dashed line) given by the balanced test population’s own self-crossplay.

Figure 9: Normalized episodic return for a best-response policy trained under each rebalancing scheme (MatFact, Spectral, k-means), as a function of the specified number of conventions $q$ , evaluated against the uniform ${\mathcal{P}}^{(0)}_{\text{test}}$ population for the Matrix Game variant of Appendix D.1 . The dashed vertical line marks the ground-truth number of conventions. All methods improve over the unrebalanced baseline even when $q$ is mis-specified, with the largest gains at the correct value; MatFact remains comparatively resilient to overspecification of $q$ .

Environment	Var	FCP	SyKLRBR (L5)	SyKLRBR (L2)	CoMeDi (BC)	CoMeDi (BR)	LIPO	MEP	ConvPlay (matfact)	ConvPlay (spectral)
Matrix Game	Non-diff	93.04 $\pm$ 0.83	93.47 $\pm$ 0.43	94.06 $\pm$ 0.50	44.25 $\pm$ 1.88	87.25 $\pm$ 2.39	94.02 $\pm$ 0.38	81.26 $\pm$ 18.89	93.62 $\pm$ 0.49	93.01 $\pm$ 0.68
	Partial	27.07 $\pm$ 0.98	27.77 $\pm$ 1.07	27.75 $\pm$ 0.92	9.11 $\pm$ 0.32	13.29 $\pm$ 1.66	27.60 $\pm$ 0.54	22.85 $\pm$ 5.02	28.75 $\pm$ 4.37	29.54 $\pm$ 0.60
	Full	14.87 $\pm$ 0.58	15.38 $\pm$ 2.55	15.91 $\pm$ 0.96	8.69 $\pm$ 0.17	12.97 $\pm$ 1.12	14.36 $\pm$ 2.18	14.25 $\pm$ 1.00	17.60 $\pm$ 2.15	17.24 $\pm$ 1.46
PMR	Non-diff	62.91 $\pm$ 6.71	68.17 $\pm$ 1.21	68.30 $\pm$ 0.93	51.17 $\pm$ 3.28	63.83 $\pm$ 7.42	68.52 $\pm$ 1.44	47.63 $\pm$ 2.14	60.78 $\pm$ 4.33	64.94 $\pm$ 7.64
	Partial	32.49 $\pm$ 3.39	32.18 $\pm$ 1.03	32.73 $\pm$ 1.06	21.83 $\pm$ 1.91	28.78 $\pm$ 3.08	26.10 $\pm$ 1.30	30.84 $\pm$ 4.90	39.18 $\pm$ 2.49	39.26 $\pm$ 1.33
	Full	27.88 $\pm$ 4.42	27.52 $\pm$ 2.34	27.20 $\pm$ 2.77	7.96 $\pm$ 7.46	23.64 $\pm$ 2.72	25.84 $\pm$ 1.61	19.74 $\pm$ 3.37	34.42 $\pm$ 1.48	36.64 $\pm$ 1.50

Appendix

Table 7: Mean normalised episodic return on

{\mathcal{P}}^{(1)}_{\text{heur}}

partners. Each cell reports mean

\pm

1 s.d. (unbiased sample standard deviation,

n-1

denominator) across

n=5

seeds. Bold marks the row’s best value.

Environment	Var	FCP	SyKLRBR (L5)	SyKLRBR (L2)	CoMeDi (BC)	CoMeDi (BR)	LIPO	MEP	ConvPlay (matfact)	ConvPlay (spectral)
Matrix Game	Non-diff	95.16 $\pm$ 0.30	95.47 $\pm$ 0.17	95.55 $\pm$ 0.14	67.64 $\pm$ 0.21	68.86 $\pm$ 4.86	95.58 $\pm$ 0.09	69.44 $\pm$ 1.31	94.64 $\pm$ 0.15	94.42 $\pm$ 0.37
	Partial	95.35 $\pm$ 0.55	95.25 $\pm$ 0.35	95.41 $\pm$ 0.40	55.77 $\pm$ 0.24	53.83 $\pm$ 6.64	83.15 $\pm$ 0.30	68.33 $\pm$ 10.25	90.19 $\pm$ 5.04	92.12 $\pm$ 0.37
	Full	95.26 $\pm$ 0.26	92.53 $\pm$ 5.88	93.07 $\pm$ 5.35	44.80 $\pm$ 0.16	48.01 $\pm$ 1.97	64.11 $\pm$ 4.55	56.18 $\pm$ 0.47	87.36 $\pm$ 4.78	86.02 $\pm$ 5.34
PMR	Non-diff	90.05 $\pm$ 7.96	98.20 $\pm$ 0.24	98.11 $\pm$ 0.14	66.95 $\pm$ 3.24	80.55 $\pm$ 11.43	91.27 $\pm$ 6.79	52.50 $\pm$ 7.26	78.71 $\pm$ 7.25	91.70 $\pm$ 9.04
	Partial	90.14 $\pm$ 8.64	98.20 $\pm$ 0.13	98.16 $\pm$ 0.19	51.02 $\pm$ 7.01	73.20 $\pm$ 8.43	63.66 $\pm$ 0.32	83.68 $\pm$ 13.72	94.31 $\pm$ 7.10	97.53 $\pm$ 0.24
	Full	87.40 $\pm$ 13.70	88.43 $\pm$ 8.62	78.20 $\pm$ 10.79	30.13 $\pm$ 9.03	63.62 $\pm$ 14.91	70.07 $\pm$ 8.67	55.67 $\pm$ 9.16	90.22 $\pm$ 8.88	96.69 $\pm$ 0.21

Appendix

Table 8: Coordination efficiency

\eta

(%) on

{\mathcal{P}}^{(0)}_{\text{test}}

partners. Each cell reports mean

\pm

1 s.d. (unbiased sample standard deviation,

n-1

denominator) across

n=5

seeds. Bold marks the row’s best value.

Environment	Var	FCP	SyKLRBR (L5)	SyKLRBR (L2)	CoMeDi (BC)	CoMeDi (BR)	LIPO	MEP	ConvPlay (matfact)	ConvPlay (spectral)
Matrix Game	Non-diff	66.88 $\pm$ 2.76	64.68 $\pm$ 2.10	64.54 $\pm$ 2.61	67.45 $\pm$ 1.62	71.73 $\pm$ 3.36	61.28 $\pm$ 5.21	73.10 $\pm$ 4.09	87.47 $\pm$ 1.79	86.81 $\pm$ 1.93
	Partial	46.69 $\pm$ 6.71	46.24 $\pm$ 3.65	46.04 $\pm$ 4.17	67.40 $\pm$ 0.44	59.71 $\pm$ 3.32	53.29 $\pm$ 1.11	63.24 $\pm$ 5.07	76.38 $\pm$ 1.84	84.23 $\pm$ 1.82
	Full	39.32 $\pm$ 1.84	42.34 $\pm$ 6.18	39.32 $\pm$ 6.25	59.74 $\pm$ 0.52	50.83 $\pm$ 1.96	51.35 $\pm$ 4.97	64.18 $\pm$ 2.11	73.99 $\pm$ 1.15	81.11 $\pm$ 0.92
PMR	Non-diff	93.71 $\pm$ 6.04	99.02 $\pm$ 0.39	99.07 $\pm$ 0.54	66.82 $\pm$ 4.12	83.90 $\pm$ 8.91	94.51 $\pm$ 5.83	65.86 $\pm$ 8.60	90.53 $\pm$ 4.72	98.03 $\pm$ 2.26
	Partial	73.24 $\pm$ 5.58	77.58 $\pm$ 1.22	78.43 $\pm$ 2.32	47.97 $\pm$ 5.34	71.37 $\pm$ 7.19	60.96 $\pm$ 1.51	70.84 $\pm$ 9.72	92.03 $\pm$ 1.65	90.17 $\pm$ 1.54
	Full	74.78 $\pm$ 7.65	73.55 $\pm$ 6.69	52.59 $\pm$ 5.85	31.71 $\pm$ 8.82	55.86 $\pm$ 11.29	67.04 $\pm$ 7.07	59.92 $\pm$ 3.88	88.94 $\pm$ 3.00	91.27 $\pm$ 1.73

Appendix

Table 9: Coordination efficiency

\eta

(%) on

{\mathcal{P}}^{(1)}_{\text{sampled}}

partners. Each cell reports mean

\pm

1 s.d. (unbiased sample standard deviation,

n-1

denominator) across

n=5

seeds. Bold marks the row’s best value.

Environment	Var	FCP	SyKLRBR (L5)	SyKLRBR (L2)	CoMeDi (BC)	CoMeDi (BR)	LIPO	MEP	ConvPlay (matfact)	ConvPlay (spectral)
Matrix Game	Non-diff	93.77 $\pm$ 0.83	94.22 $\pm$ 0.43	94.80 $\pm$ 0.50	44.62 $\pm$ 1.89	87.93 $\pm$ 2.41	94.76 $\pm$ 0.39	81.90 $\pm$ 19.03	94.36 $\pm$ 0.49	93.74 $\pm$ 0.68
	Partial	73.68 $\pm$ 1.64	74.50 $\pm$ 1.32	74.66 $\pm$ 1.25	10.26 $\pm$ 0.67	20.37 $\pm$ 4.23	74.51 $\pm$ 0.63	56.05 $\pm$ 15.59	68.33 $\pm$ 13.38	75.19 $\pm$ 1.00
	Full	49.06 $\pm$ 1.02	46.20 $\pm$ 9.34	48.91 $\pm$ 4.48	9.38 $\pm$ 0.31	26.12 $\pm$ 2.95	27.99 $\pm$ 3.60	32.67 $\pm$ 3.80	42.12 $\pm$ 8.41	51.01 $\pm$ 3.26
PMR	Non-diff	86.32 $\pm$ 9.15	93.47 $\pm$ 1.66	93.68 $\pm$ 1.30	70.08 $\pm$ 4.49	87.45 $\pm$ 10.39	93.94 $\pm$ 1.96	64.77 $\pm$ 2.91	83.32 $\pm$ 5.95	88.98 $\pm$ 10.65
	Partial	65.43 $\pm$ 7.36	65.73 $\pm$ 1.84	66.81 $\pm$ 1.76	36.71 $\pm$ 4.63	55.79 $\pm$ 6.80	52.83 $\pm$ 2.26	63.85 $\pm$ 8.86	75.41 $\pm$ 6.62	75.83 $\pm$ 2.72
	Full	62.86 $\pm$ 9.40	62.71 $\pm$ 4.36	61.84 $\pm$ 5.23	19.93 $\pm$ 18.25	53.51 $\pm$ 6.56	57.06 $\pm$ 3.58	44.10 $\pm$ 7.84	73.11 $\pm$ 3.92	77.68 $\pm$ 3.22

Appendix

Table 10: Coordination efficiency

\eta

(%) on

{\mathcal{P}}^{(1)}_{\text{heur}}

partners. Each cell reports mean

\pm

1 s.d. (unbiased sample standard deviation,

n-1

denominator) across

n=5

seeds. Bold marks the row’s best value.

Figure 10: Trajectories in PMR for the ad-hoc agent (red) paired with a heuristic partner (blue), comparing FCP (top row) and ConventionPlay (bottom row). Left column: a fixed $K_{0}$ partner committed to landmark 0. Remaining columns: tit-for-tat partners that default to landmark 0 before switching to landmark 2, 3, or 4 respectively, conditioned on the position of the partner agent. FCP agents infer their partner’s initially preferred convention and coordinate accordingly. ConventionPlay ’s trajectories are visibly more winding, showing its ability to both lead a capability-limited adaptive partner or follow a fixed-convention one.

Figure 11: Effect of $K_{1}$ population size on coordination efficiency with ${\mathcal{P}}^{(1)}_{\text{sampled}}$ partners. Coordination efficiency $\eta\in[0,1]$ is the fraction of a partner’s achievable return obtained (defined in Appendix D.3 ). ConventionPlay exhibits substantial gains up to moderate diversity, followed by diminishing returns. FCP—invariant to $K_{1}$ size—is shown as a horizontal line for reference. Error bars show $\pm$ 1 sample standard deviation, where each seed’s value is mean efficiency averaged over all test partners.

Explore similar work

ConventionPlay: Capability-Limited Training for Robust Ad-Hoc Collaboration
Apr 20, 2026Abhishek Sriraman, Eleni Vasilaki, Robert LoftinMulti-Agent Coordination Multi-Agent Collaboration
University of Sheffield · Sheffield, United Kingdom
Partner Capability Estimation for Task-Agnostic Adaptation in Ad-Hoc Teamwork
Jul 29, 2026Peter Tisnikar, Maja Swieczkowska, Benteng Ma +2Bayesian Inference Ad Hoc Teamwork
aDepartment of Informatics, King’s College London, 30 Aldwych, London, WC2B 4BG, England, United Kingdom · bDepartment of Engineering Science, University of Oxford, Parks Road, Oxford, OX1 3PJ, England, United Kingdom
ROTATE: Regret-driven Open-ended Training for Ad Hoc Teamwork
May 29, 2025Caroline Wang, Arrasy Rahman, Benjamin Nativi +4Multi-Agent Collaboration Ad Hoc Teamwork
Department of Computer Science The University of Texas at Austin · Visa AI · College of Computing and Data Science Nanyang Technological University +1

Apr 20, 2026, cs.MAJ/K move · Enter open · S save

ConventionPlay: Capability-Limited Training for Robust Ad-Hoc Collaboration

Abhishek Sriraman, Eleni Vasilaki, Robert Loftin

University of Sheffield · Sheffield, United Kingdom

Multi-Agent Coordination Multi-Agent Collaboration Ad Hoc Teamwork Cooperative MARL

Open paper AskPDF arXiv

Ad-hoc collaboration often relies on identifying and adhering to shared conventions. However, when partners can follow multiple conventions, agents must do more than simply adapt; they must actively steer the team toward the most effective joint strategy. We present ConventionPlay, a reinforcement learning-based approach that extends cognitive hierarchies to include a diverse population of adaptive followers. By training against partners with varied capability limits, our agent learns to probe its partner's repertoire, leading the team when possible and following when necessary. Our results in canonical coordination tasks show that ConventionPlay achieves superior coordination efficiency, particularly in settings where conventions have differentiated payoffs.

Abstract

Explore similar work

ConventionPlay: Capability-Limited Training for Robust Ad-Hoc Collaboration

Partner Capability Estimation for Task-Agnostic Adaptation in Ad-Hoc Teamwork

ROTATE: Regret-driven Open-ended Training for Ad Hoc Teamwork