PopGen: Generating Semantically Diverse Populations for Zero-Shot Coordination

Paul Teiletche; Kanghyun Ryu; Kyungjoon Park; Dhruva Tirumala; Markus Wulfmeier; Negar Mehr

cs.AIOct 11, 2026

PopGen: Generating Semantically Diverse Populations for Zero-Shot Coordination

Authors: Paul Teiletche, Kanghyun Ryu, Kyungjoon Park, Dhruva Tirumala, Markus Wulfmeier, Negar Mehr

Organizations: EPFL · UC Berkeley · Yonsei University · Google DeepMind ·

Abstract

In multi-agent reinforcement learning, exposing a cooperator to partners with diverse strategies during training is key to better collaboration at test time. However, existing methods typically rely on trajectory-level proxies to measure diversity, which may label two distinct trajectories of the same strategy as diverse. Instead, our key insight is to leverage LLMs' ability to describe behaviorally distinct agents in environments, similar to how humans can recognize high-level strategies. To this end, we introduce PopGen, a framework that synthesizes semantically diverse agents from scratch using LLMs. PopGen first generates a diverse set of textual descriptions of agent behaviors, then translates each behavior into a code policy that can be executed within the environment. We show that cooperators trained against PopGen populations achieve robust zero-shot cooperation with unseen partners, outperforming population-play baselines across four collaborative environments. We further show that semantic diversity scales with PopGen population size, improving the cooperator's worst-case performance, and we identify design choices that preserve population quality while maximizing diversity. Our code is available at https://github.com/labicon/PopGen.

Figures & tables

Figure 1: Overview of PopGen . Left (Stage 1): An LLM produces a diverse set of behaviors at the text level using oversampling. Right (Stage 2): Each behavior description enters an iterative refinement loop: a generator translates the behavior into an executable policy (Policy Code), which is evaluated with a VLM through environment rollouts. An advisor then analyzes rollout results and provides feedback to guide subsequent refinement attempts.

Figure 2: Zero-shot coordination with held-out partners. Each bar reports the average team reward (higher is better) when each cooperator is paired with the held-out partners, which do not appear in any training population. Error bars give the standard deviation across five runs, and dots mark individual runs. PopPlay attains the highest reward in all four environments. Table 5 reports the reward against each individual partner.

Panel of Figure 3 — Figure 3: Cooperator role adaptation in overcooked and robustness to held-out partners. (a) Rollouts show PopPlay taking semantically diverse behaviors interpretable to humans. (b) PopPlay spreads its actions across subtasks while the baselines tend to focus on a subset of actions. Per-partner counts are in Table 8 . (c) PopPlay achieves the highest worst-case reward in all environments.

Panel of Figure 4 — Figure 4: Scaling the PopGen population in overcooked . PopPlay cooperators trained against pools of $n$ distinct PopGen agents at a fixed budget. (a) Larger populations are more diverse and raise the worst-case team reward with held-out partners. (b) Higher diversity also raises the average team reward, with gains plateauing beyond $n=4$ . Error bars show the standard error over runs.

Figure 5: Diversity and quality of PopGen population agents. Quality and diversity of a generated set of 30 behaviors, averaged across the environments. PopGen matches the diversity of VS while improving quality, and is substantially more diverse than DIR and SemDiv.

Figure 6: Population play with policy code and reward-function partners. Mean team reward against held-out partners of cooperators whose training partners are modeled as executable code (like PopGen ) or as neural network policies trained with RL on LLM-generated reward functions (like SemDiv). The set of behaviors is kept identical, so that only agent modeling method changes. Error bars give the standard deviation across five runs, and dots mark individual runs.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Environments used for cooperator training. From left to right, PMR requires agents (red and purple) to navigate toward and converge on the same landmark (green). PP requires agents (red) to coordinate to capture prey, with different rewards between individually capturing a rabbit (green) and jointly capturing a stag (yellow). LBF requires agents to combine their carrying capacities to collect food items whose levels may exceed the level of a single agent. Finally, overcooked requires two chefs to coordinate the sequential subtasks of preparing, cooking, and delivering soups.

Env	Num. Agents	Action Space	Observation	Objective
LBF	2 (ego: 2)	Discrete (6)	Partial	Collect all food items
Overcooked	2 (ego: 2)	Discrete (6)	Global	Serve as many soups as possible
PMR	2 (ego: 2)	Continuous (2D)	Global	Meet at the same landmark
PP	7 (ego: 2, adv.: 5)	Continuous (2D)	Partial	Capture prey

Appendix

Table 1: Multi-agent environments in PopGen cover cooperative settings across discrete and continuous action spaces. We note that adversarial agents (prey) in PP follow fixed policies defined by the environment, so only the ego team is generated and trained.

Env.	Name	Description
LBF	Closest-Food Greedy	Moves to the nearest food and loads it when possible.
	Team-Center Food	Targets food near the center of the two agents.
	Teammate-Biased Food	Targets the food cheapest for the teammate to reach.
	Synchronized Loading	Waits at its food until the teammate is adjacent.
PP	Closest-Prey Greedy	Chases the nearest prey, whether it is a rabbit or a stag.
	Teammate-Biased Prey	Targets the prey the teammate is also pressuring.

Appendix

Table 2: Held-out evaluation partners for each environment: Level-Based Foraging (LBF), Predator-Prey (PP), Overcooked, and Point-Mass Rendezvous (PMR).

Shared across all environments
Discount factor $\gamma$	0.99
Learning rate	$10^{-4}$
PPO clip $\epsilon$	0.2
GAE $\lambda$	0.9
Policy network	GRU
Prioritized sampling $\beta$	1.0

Appendix

Table 3: Cooperator training hyperparameters. The upper block lists the settings shared by every environment and every method. The lower block lists the per-environment settings.

Method	Partners	PMR	LBF	PP	Overcooked
Self-Play (SP)	Eval	$1.01\pm 0.25$	$0.81\pm 0.04$	$1.21\pm 0.12$	$63.38\pm 65.37$
Self-Play (SP)	Train	$1.60\pm 0.22$	$0.78\pm 0.02$	$1.36\pm 0.14$	$133.79\pm 151.41$
FCP	Eval	$0.93\pm 0.31$	$0.80\pm 0.03$	$1.27\pm 0.09$	$40.53\pm 52.23$
FCP	Train	$\mathbf{1.68\pm 0.30}$	$0.80\pm 0.02$	$1.31\pm 0.19$	$157.29\pm 134.16$
MEP	Eval	$1.45\pm 0.46$	$0.72\pm 0.14$	$1.30\pm 0.17$	$33.79\pm 37.73$
MEP	Train	$1.32\pm 0.17$	$0.65\pm 0.30$	$1.31\pm 0.32$	$126.05\pm 104.18$

Appendix

Table 4: Average team reward of each cooperator against evaluation and training partners. We report reward against held-out eval behaviors and against the train partners the cooperator saw during training. We report the reward averaged across partners (mean

\pm

standard deviation over five runs). Best value per environment is in bold .

Held-out partner	PopPlay (ours)	SP	FCP	MEP	CoMeDi
Point-Mass Rendezvous (PMR)
Closest-Landmark Greedy	$\mathbf{1.58}_{\pm 0.26}$	$0.82_{\pm 0.08}$	$0.92_{\pm 0.16}$	$1.01_{\pm 0.02}$	$\mathbf{1.46}_{\pm 0.21}$
Farthest-Landmark Commitment	$\mathbf{1.60}_{\pm 0.40}$	$\mathbf{0.80}_{\pm 0.49}$	$0.40_{\pm 0.40}$	$\mathbf{1.20}_{\pm 0.49}$	$\mathbf{1.45}_{\pm 0.22}$
Teammate-Biased Landmark	$\mathbf{2.00}_{\pm 0.00}$	$\mathbf{2.00}_{\pm 0.00}$	$\mathbf{2.00}_{\pm 0.00}$	$\mathbf{1.99}_{\pm 0.01}$	$1.93_{\pm 0.04}$
Regroup then Diverge	$\mathbf{1.60}_{\pm 0.40}$	$0.40_{\pm 0.40}$	$0.40_{\pm 0.40}$	$\mathbf{1.60}_{\pm 0.40}$	$\mathbf{1.42}_{\pm 0.23}$
Level-Based Foraging (LBF)

Appendix

Table 5: Team reward against each held-out partner. We report mean

\pm

standard error of the team reward across 5 runs. For every partner, bold marks the best cooperator and every method whose error bar overlaps it. The final row counts the partners for which a method attains the highest average.

Landmark (%)	Run 1				Run 2				Run 3				Run 4				Run 5
	1	2	3	4	1	2	3	4	1	2	3	4	1	2	3	4	1	2	3	4
East	0	0	0	100	35	100	0	0	0	0	0	0	85	0	0	100	0	0	0	0
North	100	15	0	0	0	0	75	0	0	0	100	0	15	0	30	0	0	0	0	0
West	0	0	75	0	0	0	0	0	100	0	0	0	0	0	70	0	100	100	70	100
South	0	85	25	0	65	0	25	100	0	100	0	100	0	100	0	0	0	0	30	0

Appendix

Table 6: Self-Play in Point-Mass Rendezvous. Each column is one self-play policy paired with itself. We run 5 self-play experiments, each producing 4 policies from different seeds, and evaluate each policy for 20 episodes. Rows give the percentage of episodes in which the pair meets at each landmark, and the largest in each column is in bold. Counting each policy’s most frequent landmark, only Run 1 covers all four landmarks, and all four policies in Run 5 settle on West, since Self-Play has no principled way of promoting diversity across seeds.

Stage (%)	Run 1				Run 2				Run 3				Run 4				Run 5
	1	2	3	4	1	2	3	4	1	2	3	4	1	2	3	4	1	2	3	4	Mean
Fetch onion	37	40	51	38	58	51	40	54	41	49	35	46	43	42	57	34	51	47	35	39	44
Onion $\to$ pot	31	36	38	35	38	36	35	39	33	28	33	45	33	35	34	34	38	42	35	36	36
Start pot	9	12	0	12	0	0	12	0	8	8	11	0	11	10	0	10	0	0	12	12	6
Fetch dish	11	12	11	8	4	13	13	7	11	8	11	10	11	13	9	10	10	11	10	12	10
Scoop soup	11	0	0	8	0	0	0	0	8	8	11	0	2	0	0	10	0	0	10	1	3

Appendix

Table 7: Self-Play in Overcooked. Each column is one self-play policy paired with itself. We run 5 self-play experiments, each producing 4 policies from different seeds, and evaluate each policy for 20 episodes. Rows give each pipeline stage’s share of all stage actions performed by both agents (%), and the largest in each column is in bold.

		Cooperator
Method	Partner	Onion $\rightarrow$ pot	Start pot	Fetch dish	Scoop soup	Serve soup
PopPlay	Fill & Start	$0.23$	$0.22$	$10.62$	$10.35$	$8.14$
	Fill only	$0.23$	$0.78$	$2.34$	$0.74$	$0.72$
	Fetch & Serve	$5.18$	$4.85$	$0.36$	$0.02$	$0.00$
	Wait & Serve	$5.20$	$4.85$	$0.41$	$0.03$	$0.00$
Self-Play	Fill & Start	$0.00$	$0.00$	$7.96$	$7.16$	$5.55$

Appendix

Table 8: Cooperator task events per held-out partner in overcooked . Per-partner breakdown of Figure 3b . Fill & Start and Fill only load the pot but never serve. Fetch & Serve and Wait & Serve plate and serve but never fetch an onion. Counts are averaged per episode over 50 episodes.

Method	LLM Calls	Candidates	Turns	Prompt Example	Definition
1. Instance-level Prompt
Direct (DIR)	$N$	1	1	“Think step-by-step, generate a football player behavior”	$y_{i}\sim\pi(y\|x_{\text{CoT}})$
2. List-level Prompt
One-Pass (OP)	$1$	$N$	1	“Generate 30 football player behaviors”	$(y_{1},\ldots,y_{N})\sim\pi(y_{1},\ldots,y_{N}\|x_{\text{OP}})$
Sequence (SEQ)	$\lceil N/k\rceil$	$k$	1	“Generate 5 football player behaviors”	$(y_{1},\ldots,y_{k})\sim\pi(y_{1},\ldots,y_{k}\|x_{\text{seq}})$
Multi-Turn (MT)	$N$	1	$N$	Turn 1: “Generate a football player behavior” Turn 2+: “Generate another football player behavior”	$y_{i}\sim\pi(y\|x_{\text{multi}},h_{i-1})$

Appendix

Table 9: Comparison of prompting methods for generating

N

behaviors.

k

is the number of candidates per LLM call (in our experiments

k=5

y_{i}

denotes the

i

-th generated candidate,

\hat{p}_{i}

its verbalized probability, and

\pi(\cdot|x)

the LLM’s output distribution conditioned on prompt

x

. For Multi-Turn and VS-Multi,

h_{i-1}

denotes the conversation history up to turn

i-1

, and

t

the turn index.

Figure 8: Quality and diversity of all behavior generation methods. Quality and diversity of a generated set of 30 behaviors, averaged across the four environments, for every method in Table 9 . VS-OS is the method used in PopGen .

		Diversity ( $\uparrow$ )							Quality ( $\uparrow$ )
Type	Ratio	football	lbf	pmr	pp	smacv2	overcooked	avg	football	lbf	pmr	pp	smacv2	overcooked	avg
Baseline (VS)	$\times$ 1	0.730	0.726	0.636	0.691	0.756	0.612	0.692	12.77	11.16	13.52	12.20	12.00	13.32	12.50
One Pass	$\times$ 2	0.737	0.710	0.645	0.725	0.752	0.634	0.701	12.47	12.24	13.78	11.90	12.44	13.30	12.69
One Pass	$\times$ 4	0.729	0.684	0.644	0.718	0.747	0.684	0.701	12.50	12.18	13.84	13.04	12.57	13.20	12.89
Independent	$\times$ 2	0.737	0.654	0.637	0.735	0.738	0.616	0.686	12.44	11.97	13.84	12.93	12.20	13.26	12.77
Independent	$\times$ 4	0.726	0.672	0.643	0.721	0.749	0.614	0.688	13.01	12.57	13.88	12.77	12.11	13.10	12.91

Appendix

Table 10: Quality and diversity of oversampling variants across environments. Diversity is measured as average pairwise embedding distance; quality as total score. Best value per column in bold . The Sequential

\times

2 variant — referred to as VS-OS throughout the paper — achieves the best diversity–quality trade-off overall.

Panel of Figure 9 — Figure 9: Diversity breakdown and scaling properties of VS variants. (a) VS-MT sacrifices worst-case diversity (minimum pairwise distance drops to 0.35) and coverage (0.38) relative to single-pass VS (0.48, 0.44): iterative refinement tends to homogenize outputs. VS-OS recovers both metrics (0.47, 0.45) by decoupling exploration from selection—oversampling casts a wide net, then a dedicated merging step curates without collapsing variety. Lexical dissimilarity is comparable across all three. (b) The decoupled design of VS-OS trades additional latency for better scaling behavior. When moving from Flash to Pro, VS-MT gains quality ( $+1.00$ ) but loses diversity ( $-0.06$ ), confirming that multi-turn refinement with a stronger model amplifies the homogenization effect. VS-OS, by contrast, translates the stronger model into gains on both axes ( $+0.57$ quality, $+0.04$ diversity): because exploration and selection are separate stages, a more capable model improves the quality of each stage independently without one undermining the other.

Name	Abbr.	Description
LLM-Embodied Agent	LLM-A	Uses an LLM directly as the decision-making agent, conditioned on textualized observations and a behavior specification: $\pi_{\text{LLM-A}}\sim\text{LLM}(o,b)$
Reward Function	RF	An LLM generates a reward function $r_{b}:(o,a)\rightarrow\mathbb{R}$ , whose optimization via reinforcement learning yields a neural network policy $\pi_{\text{RF}}$ exhibiting the target behavior.
Policy Code	PC	An LLM synthesizes an executable policy $f_{b}:o\rightarrow a$ that directly maps observations to actions, serving as the policy.

Appendix

Table 11: Behavior modeling methods. Each method specifies a distinct mechanism for instantiating a target behavior

b

in an agent policy

\pi

o

is the observation of the environment, and

a

is the sampled action.

		Behavior Adherence (Dev)					Behavior Adherence (Test)
Method	Attempt	LBF	PMR	PP	Overcooked	Avg.	LBF	PMR	PP	Overcooked	Avg.
LLM-A	First	$13.3\pm 10.4$	$33.3\pm 7.7$	$58.3\pm 10.4$	$30.7\pm 8.0$	$33.9\pm 9.1$	$25.4\pm 1.9$	$26.7\pm 5.8$	$93.3\pm 5.2$	$30.7\pm 6.5$	$44.0\pm 4.9$
	Best	$\mathbf{13.3\pm 10.4}$	$\mathbf{50.7\pm 7.8}$	$\mathbf{76.7\pm 10.4}$	$\mathbf{62.7\pm 9.3}$	$\mathbf{50.9\pm 9.5}$	$\mathbf{25.4\pm 1.9}$	$\mathbf{32.7\pm 6.6}$	$\mathbf{93.3\pm 5.2}$	$\mathbf{45.3\pm 9.1}$	$\mathbf{49.2\pm 5.7}$
	$\Delta$	$0.0$	$+17.4$	$+18.3$	$+32.0$	$+16.9$	$0.0$	$+6.0$	$0.0$	$+14.6$	$+5.2$
PC	First	$43.0\pm 8.4$	$56.0\pm 8.3$	$95.0\pm 3.5$	$34.7\pm 10.8$	$57.2\pm 7.8$	$53.5\pm 3.9$	$58.0\pm 7.5$	$74.0\pm 8.4$	$41.3\pm 10.5$	$56.7\pm 7.6$
	Best	$\mathbf{71.0\pm 6.5}$	$\mathbf{84.0\pm 5.9}$	$\mathbf{98.0\pm 2.7}$	$\mathbf{68.0\pm 10.3}$	$\mathbf{80.3\pm 6.4}$	$\mathbf{61.0\pm 5.0}$	$\mathbf{66.7\pm 6.1}$	$\mathbf{74.0\pm 8.4}$	$\mathbf{63.3\pm 9.3}$	$\mathbf{66.3\pm 7.2}$

Appendix

Table 12: Adherence scores of PopGen agents across environments and splits. We use gemini-3-flash-preview as the agent generator or player in LLM-A. For each method, we report first-attempt and best-attempt adherence (out of 5 attempts) along with the improvement

\Delta

. Avg. columns show the mean across environments. Best values per method and split are in bold .

Method	Generation latency (s) $\downarrow$	Rollout latency (s) $\downarrow$
LLM-A	—	$110.1\pm 1.5$
RF	$2878.6\pm 239.5$	$11.7\pm 0.6$
PC	$\mathbf{8.1\pm 0.2}$	$\mathbf{9.8\pm 1.9}$

Appendix

Table 13: Average generation and rollout latency per method. Generation latency covers code synthesis (PC) or RL training (RF); rollout latency is for a single episode. LLM-A has no generation step (—) but incurs high rollout cost from per-timestep LLM inference. Best per column in bold . smacv2 is excluded due to engine overhead dominating latencies.

Split	Attempt	LBF	PMR	PP	Overcooked	Avg.
Dev	First	$43.0\pm 8.4$	$56.0\pm 8.3$	$95.0\pm 3.5$	$34.7\pm 10.8$	$57.2\pm 7.8$
	Best	$\mathbf{71.0\pm 6.5}$	$\mathbf{84.0\pm 5.9}$	$\mathbf{98.0\pm 2.7}$	$\mathbf{68.0\pm 10.3}$	$\mathbf{80.3\pm 6.4}$
	$\Delta$	$+28.0$	$+28.0$	$+3.0$	$+33.3$	$+23.1$
Test	First	$53.5\pm 3.9$	$58.0\pm 7.5$	$74.0\pm 8.4$	$41.3\pm 10.5$	$56.7\pm 7.6$
	Best	$\mathbf{61.0\pm 5.0}$	$\mathbf{66.7\pm 6.1}$	$\mathbf{74.0\pm 8.4}$	$\mathbf{63.3\pm 9.3}$	$\mathbf{66.3\pm 7.2}$
	$\Delta$	$+7.5$	$+8.7$	$0.0$	$+22.0$	$+9.6$

Appendix

Table 14: Policy iterative refinement. Improvement in adherence (first- vs. best-attempt, out of 5 attempts) on the dev and test splits. Avg. columns show the mean across environments. Best values per split are in bold .

Zero-Shot Coordination

See all 11 papers

Apr 28, 2026

Zero Shot Coordination for Sparse Reward Tasks with Diverse Reward Shapings

Many Multi-Agent Reinforcement Learning (MARL) agents fail to adapt properly to cooperating with agents trained with the same objectives but different seeds, algorithms, or other training differences. This is the problem of Zero-Shot Coordination (ZSC), which focuses on training agents to cooperate well with unknown agents. ZSC has been studied for a variety of tabular cases and simple games such as Hanabi, achieving excellent results. However, existing solutions to ZSC only consider identical rewards for your trained agents and all future partners. This is not realistic for the trained agents, as they do not consider the problem of cooperating with agents that have identical sparse objectives but shape the rewards for those objectives in different manner. To address this issue, we show how to train an ensemble of methods using randomized reward shapings chosen using 4 selection algorithms. Experiments done on the Overcooked environment demonstrate consistent improvements of 62.2%-119.2% in sparse reward over baseline ZSC algorithms when playing with agents that have identical sparse rewards but different reward shapings.

Keenan Powell, Peihong Yu, Pratap Tokekar

Department of Computer Science University of Maryland, College Park

Multi-Agent Coordination Reward Shaping Sparse-Reward RL Reward Design for RL Zero-Shot Coordination Multi-Agent Reinforcement Learning

Ask PDF

Aug 8, 2025

Unsupervised Partner Design Enables Robust Ad-hoc Teamwork

We introduce Unsupervised Partner Design (UPD), a population-free multi-agent reinforcement learning method for robust ad-hoc teamwork. UPD generates training partners on-the-fly and selects them adaptively based on a learnability criterion, removing the need for pre-trained partner populations or manual parameter tuning. We show that this simple mechanism enables effective partner diversity and can be extended to joint partner-environment selection when a procedural level generator is available. Across Level-Based Foraging, Overcooked-AI, and the Overcooked Generalisation Challenge, UPD consistently achieves strong performance compared to both population-based and population-free baselines. In a human-AI user study, agents trained with UPD achieve higher returns and are rated as more adaptive, more human-like, and less frustrating than all evaluated baseline methods.

Constantin Ruhdorfer, Matteo Bortoletto, Victor Oei +2

Collaborative Artificial Intelligence, University of Stuttgart, Stuttgart, Germany.

Multi-Agent Collaboration Ad Hoc Teamwork Robust RL Cooperative MARL

Ask PDF

Jun 26, 2026

LLawCo: Learning Laws of Cooperation for Modeling Embodied Multi-Agent Behavior

Embodied agents operating in decentralized and partially observable environments have attracted growing attention in recent years. However, existing large language model (LLM)-based agents often exhibit behaviors that are misaligned with their partners or inconsistent with the environment state, leading to inefficient cooperation and poor task success. To address this challenge, we propose a novel framework, Learning Laws of Cooperation (LLawCo), that enables embodied agents to autonomously align with both their partners and task objectives. Our framework allows agents to reflect on past failures to extract misaligned behavioral patterns, which are used to derive high-level behavioral laws, such as "Talk when necessary" and "Wait for partner." These laws are explicitly incorporated into the agents' chains of thought via supervised fine-tuning, aligning their reasoning with task requirements and the behavior of other agents. To evaluate our approach, we introduce PARTNR-Dialog, a large-scale multi-agent communicative and cooperative planning benchmark built on the PARTNR environment. Experiments on existing tasks and our new benchmark demonstrate significant improvements in cooperative efficiency and task success rates. Across four backbone LLMs, our method achieves average success rate improvements of 4.5% on the PARTNR-Dialog benchmark and 6.8% on the TDW-MAT benchmark over state-of-the-art open-source communicative agent frameworks. See the LLawCo project page for details: https://www.merl.com/research/highlights/LLawCo

Qinhong Zhou, Chuang Gan, Anoop Cherian

University of Massachusetts Amherst, USA. · 2Mitsubishi Electric Research Laboratories (MERL), USA.

Multi-Agent Coordination AI Alignment Multi-Agent Collaboration LLM Agent Training Multi-Agent Planning

Ask PDF

Abstract

Zero-Shot Coordination

Explore similar work

Zero Shot Coordination for Sparse Reward Tasks with Diverse Reward Shapings

Unsupervised Partner Design Enables Robust Ad-hoc Teamwork

LLawCo: Learning Laws of Cooperation for Modeling Embodied Multi-Agent Behavior