AgentWorld: Benchmarking Long-Horizon Collaboration of Multi-agent LLMs

Raphael Shu; Yusen Zhang; Young Min Cho; Jin Mo Yang; Yuan Yuan; Wenliang Zheng; Sharath Chandra Guntuku; Lyle Ungar

cs.MASep 25, 2026

AgentWorld: Benchmarking Long-Horizon Collaboration of Multi-agent LLMs

Authors: Raphael Shu, Yusen Zhang, Young Min Cho, Jin Mo Yang, Yuan Yuan, Wenliang Zheng, Sharath Chandra Guntuku, Lyle Ungar,

Organizations: OpenAgents · Columbia University · University of Pennsylvania · Seoul National University ·

Abstract

Existing multi-agent benchmarks primarily test in competitive settings, short-horizon interactions under 20 steps, or simply aggregate individual performance, failing to isolate and highlight genuine collaboration capabilities of LLM-based agents. We introduce AgentWorld, a benchmark of 100 human-annotated tasks (with 100 augmented variants) for evaluating long-horizon, multi-agent collaboration. Tasks span 50+ interaction rounds across a rich MMORPG sandbox and require 3-20 agents with asymmetric roles and abilities to coordinate through communication, joint planning, and resource sharing under a blackbox setting where each agent acts independently without access to others' internal states. To quantify collaboration effectiveness in addition to conventional binary task success, we propose Causal Collaboration Effectiveness (CCE), a graph-based metric that traces causal dependencies between agent actions and measures what fraction of a team's effort actually contributed to the outcome. Experiments with Gemini 3 Flash, Claude Haiku 4.5, GPT-5 Mini, and DeepSeek R1-70B show that even the best model achieves only 52.0% task success, with systematic failure modes including communication breakdowns, role confusion, and inability to maintain shared plans across rounds. AgentWorld is fully open-source.

Figures & tables

Figure 1: An example of the AgentWorld environment with 10 agents interacting through live chat.

Figure 2: Overview of AgentWorld. The MMORPG sandbox with diverse biomes and resources (left), blackbox multi-agent interaction model (center), and 10 representative high-level API tools (of 13 total) that abstract away low-level game mechanics (right).

Figure 3: Example task definition (Task 86: Forge Vanguard). Ten agents collaborate through a mine $\rightarrow$ smelt $\rightarrow$ craft pipeline: miners extract ore and coal, the smelter processes them into iron bars, and smiths forge heavy swords and an axe. Each agent has unique skills, equipment, inventory, and spawn location.

Platform	Max Agents	Blackbox	Domain	API Tools	Asymmetric Roles	Scalable	Open World
Overcooked ( Sun et al., 2025 )	4	No	Cooking	5	No	Yes	No
AgentBench ( Liu et al., 2023 )	5	Yes	Board games	2	No	Yes	No
SmallVille ( Park et al., 2023 )	25	Yes	Villager	10	No	Yes	No
Minecraft ( Wang et al., 2023 )	$\sim$ 50	No	Crafting	10	No	No	Yes
Kaetram (ours)	1,000	Yes	Role-playing	13	Yes	Yes	Yes
Benchmark	Tasks	Task Horizon	Agents	Roles	Task Type	Game Engine	Quantitative Eval

Table 1: Comparison of multi-agent simulators and benchmarks. Top: simulator capabilities. Bottom: benchmark characteristics. AgentWorld uniquely combines blackbox environments, asymmetric roles, long task horizons, and a world-level sandbox.

Figure 4: Dataset Statistics. (a) Task categories for main and augmented splits. (b) Agents per task. (c) Task horizon (max rounds). (d) Average success rate by category across four models (Gemini 3 Flash, Claude Haiku 4.5, GPT-5 Mini, DeepSeek R1-70B) on the main set.

Figure 5: CCE illustrated on Task 01. Colored nodes: contributing actions; gray dashed: non-contributing. Purple arrows: cross-agent causal links. The Lumberjack transfers logs to the Woodworker (R2) and Wizard (R4), enabling downstream crafting. CCE = 12/27 = 44.4%.

	Main Set			Augmented Set			Behavior
Model	SR%	PSR%	CCE	SR%	PSR%	CCE	Rounds	Chats
DeepSeek R1-70B	20.0	43.5	0.125	10.0	28.9	0.063	34.9	7.5
Claude Haiku 4.5	45.0	62.7	0.294	26.0	49.1	0.179	35.9	26.6
GPT-5 Mini	36.0	62.6	0.206	21.0	41.6	0.145	34.5	44.1
Gemini 3 Flash	52.0	71.5	0.320	24.0	61.3	0.113	26.4	11.0
Muse-Glimmer-30B	38.0	45.3	0.128	36.0	52.5	0.126	41.65	13.58

Table 2: Main results on the AgentWorld benchmark. SR: task success rate (%). PSR: partial success rate (%). CCE: causal collaboration effectiveness averaged over all tasks (failed tasks contribute CCE

=0

). Rounds and Chats are averages on the main set. Bold values identify the best reported scores excluding provisional results. The additional models use the 100-task augmented v1 split. Muse-Glimmer-30B results are final; † Gemma results are provisional and averaged over scored tasks.

Setting	SR%	Avg. Rounds	Avg. Chats
Random actions	5.7	34.1	8.3
Shared-plan-no-communication	17.6	30.7	0.0
No-communication	22.9	32.0	0.0
Single-agent	28.6	28.0	0.0
Vanilla multi-agent (LLM)	52.0	26.4	11.0
Oracle communication	60.0	25.2	7.9

Table 3: Baseline comparison on AgentWorld (Gemini 3 Flash). Random actions use no LLM. Oracle communication grants one agent visibility into all messages to coordinate the team. Vanilla multi-agent is the default blackbox setting reported in Table 2 .

Setting	SR%	Avg. Rounds	Avg. Chats
Full AgentWorld	54.3	18.8	4.7
w/o asymmetric roles	45.7	20.2	4.5
w/o long horizon	37.1	8.7	2.4
w/ random spawn	51.4	19.5	4.6
w/o documentation	29.4	30.7	3.9

Table 4: Ablation of key design components on a 35-task subset (Gemini 3 Flash). Every component contributes; removing task documentation or the long horizon has the largest effect.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Definition	Representative Example (Chat Snippet)	Count	Ratio
Stale & Redundant	Referencing completed tasks, repeating known states, or asking resolved questions.	Wizard had already crafted the staff in previous round “… do you have the sticks now?”	23	37.7%
Misidentification	Incorrect self-role labeling or requesting resources from the wrong agent.	… coming from the miner, smelter: “… send 1x coal to me.”	10	16.4%
Factual Error	Reporting incorrect item names, coordinates, or false readiness of a process.	“Transferred 12 gold ore (as goldnugget)… smith needs EXACTLY goldore.”	9	14.8%
Premature Completion	Unilaterally declaring the task finished before critical final steps are verified.	“All 7 jewelry items are finished… (before ring jeweler’s actions shown).”	7	11.5%
Non-Material Output	Polite but idle chatter (e.g., “cheering”) without providing coordinates or items.	“… let me know if you need any help.”	7	11.5%
Logical Inconsistency	Pursuing invalid strategies or issuing contradictory commands.	Everyone is already dead“… I’m searching for bowls… This is going to be tough alone.”	5	8.2%

Appendix

Table 5: Failure mode analysis of multi-agent collaboration: a communication perspective.

Agent model	GPT-4.1	Claude Sonnet 4	Llama-4 Maverick
Gemini 3 Flash	0.320	0.338	0.330
Claude Haiku 4.5	0.294	0.241	0.284
GPT-5 Mini	0.206	0.221	0.224
DeepSeek R1-70B	0.125	0.104	0.147

Appendix

Table 6: CCE under three different judge models (main set, averaged over all tasks). The ranking of agent models is fully preserved across all three judges; absolute values shift by at most

\sim

0.05.

Figure 6: Task annotation platform. (a) World map for selecting agent spawn locations. (b) Skill level configuration with sliders. (c) Searchable inventory item browser (133 items). (d) Equipment slots with character preview.

Figure 7: Task 94: Maritime Trading (20 agents, 6 rounds, success). The largest team succeeds with efficient coordination in minimal rounds.

Figure 8: Task 92: Trade Network (12 agents, 9 rounds, success). 19 resource transfers show intense cross-agent collaboration.

Figure 9: Task 85: Ice Convoy (10 agents, 9 rounds, success). Polar resource gathering with coordinated ice log collection.

Figure 10: Task 80: Festival Preparation (10 agents, 9 rounds, success). Parallel cooking and crafting with 17 transfers.

Figure 11: Task 67: Resource Caravan (8 agents, 6 rounds, success). Multi-region gathering with 11 transfers across biomes.

Figure 12: Task 60: Supply Chain (7 agents, 15 rounds, success). The most transfer-heavy task with 24 resource exchanges.

Multi-Agent Collaboration

Multi-agent collaboration studies how autonomous agents coordinate, share information, and combine capabilities to solve tasks together.

See all 309 papers

Benchmarking Open-Ended Multi-Agent Coordination in Language Agents
Jun 6, 2026Kale-ab Abebe Tessera, Andras Szecsenyi, Cameron Barker +7Multi-Agent Coordination LLM Agent Evaluation
University of Edinburgh · University of Oxford · University College London
CoCoBench: A Cooperative Coordination Benchmark for Embodied Multi-Agent Task Planning
Aug 28, 2026Yang Chen, Ye-Xin Xie, Lirong Che +8Multi-Agent Coordination AI Agent Benchmarks
Nanjing University · AgiBot · Tsinghua University
CollabBench: Benchmarking and Unleashing Collaborative Ability of LLMs with Diverse Players via Proactive Engagement
Jun 4, 2026Hong Qian, Yuanhao Liu, Zihan Zhou +7Game-Playing Agents Human Behavior Simulation
Shanghai Institute of AI for Education, and School of Computer Science and Technology, East China Normal University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · Tencent Inc., Shenzhen, China

Jun 6, 2026J/K move · Enter open · S save

Benchmarking Open-Ended Multi-Agent Coordination in Language Agents

Kale-ab Abebe Tessera, Andras Szecsenyi, Cameron Barker, Alexander Rutherford+6

University of Edinburgh · University of Oxford · University College London

Multi-Agent Coordination LLM Agent Evaluation AI Agent Benchmarks Long-Horizon LLM Agents Multi-Agent Communication Zero-Shot Coordination

Open paper AskPDF arXiv

As language models are increasingly deployed as autonomous agents, they must coordinate with others over long horizons in open-ended interactive tasks. Yet existing evaluations rarely test these demands together, instead emphasising single-agent tasks, short interactions, or highly structured multi-agent settings. We introduce

alem

, a JAX-based benchmark for open-ended multi-agent coordination built on Craftax-like dynamics. Alem embeds procedurally generated coordination tasks, soft specialisation, communication, and controllable coordination difficulty into a long-horizon survival world with exploration, crafting, trading, and combat. We evaluate

13

modern LLMs zero-shot within homogeneous teams, with trained MARL agents as reference points. Current LLM agents remain far from solving alem, averaging only ~6% normalised return, but their failures are not uniform. On the hardest coordination setting, zero-shot Gemini-3.1-Pro-High approaches MARL agents trained for one billion steps, while GPT-5.4-High achieves strong base-task reward but much lower coordination reward. This contrast shows that individual task competence does not imply coordination competence. Ablations show that communication is the largest contributor to coordination, while memory and reasoning help when used to maintain multi-step plans. Overall, our results identify coordination as a distinct bottleneck for frontier LLM agents, separate from single-agent capabilities. Alem makes this bottleneck measurable and provides a controlled testbed for developing agents that communicate, allocate roles, and execute shared plans. Code is available at https://github.com/alem-world/alem-env.

Abstract

Multi-Agent Collaboration

Explore similar work

Benchmarking Open-Ended Multi-Agent Coordination in Language Agents

CoCoBench: A Cooperative Coordination Benchmark for Embodied Multi-Agent Task Planning

CollabBench: Benchmarking and Unleashing Collaborative Ability of LLMs with Diverse Players via Proactive Engagement