Large Language Models (LLMs) have shown remarkable capabilities as autonomous agents, yet existing benchmarks either focus on single-agent tasks or are confined to narrow domains, failing to capture the dynamics of multi-agent coordination and competition. In this paper, we introduce MultiAgentBench, a comprehensive benchmark designed to evaluate LLM-based multi-agent systems across diverse, interactive scenarios. Our framework measures not only task completion but also the quality of collaboration and competition using novel, milestone-based key performance indicators. Moreover, we evaluate various coordination protocols (including star, chain, tree, and graph topologies) and innovative strategies such as group discussion and cognitive planning. Notably, gpt-4o-mini reaches the average highest task score, graph structure performs the best among coordination protocols in the research scenario, and cognitive planning improves milestone achievement rates by 3%. Code and datasets are public available at https://github.com/MultiagentBench/MARBLE.
Figures & tables
Figure 1: Overview of MultiAgentBench evaluation process : Multi-Agent System Coordination in various interactive environments, with a focus on task performance, and coordination.
Figure 2: MARBLE : showcasing interactions between task information, persona data, domain databases, memory modules, and the environment through the coordinate engine and cognitive module.
Figure 3: Illustration of coordination protocols and planning prompt strategies . (a) shows centralized and decentralized planning structures (e.g., star, tree, graph, and chain). (b) describes strategies like group discussions and cognitive prompts, incorporating iterative feedback and task updates for effective planning.
Figure 4: Illustration of our benchmark curation and the dynamic milestones detecting for KPI metric.
Model
Research
Minecraft
Database
Coding
Bargaining
WereWolf
TS
CS
TS
CS
TS
CS
TS
CS
TS
CS
TS
CS
Meta-Llama-3.1-8B
80.87
52.40
6.12
54.40
34.00
40.00
59.90
67.24
72.81
73.36
12.64
60.00
Meta-Llama-3.1-70B
80.80
49.50
0.21
75.00
53.00
37.70
62.10
67.18
72.13
71.46
19.82
60.60
Meta-Llama-3.3-70B
80.00
72.00
9.15
69.00
28.50
40.00
56.60
74.40
73.15
69.56
36.33
76.30
gpt3.5-turbo
70.20
55.90
5.05
63.60
45.00
60.89
55.50
76.20
71.67
72.00
15.69
75.90
gpt-4o-mini
84.13
52.00
33.60
61.50
45.00
43.22
65.10
66.30
74.47
74.20
14.06
60.10
Table 1: Average Task Score ( TS ) (%) and the Coordination Score ( CS ) for Minecraft, Database, Coding, Bargaining, and WereWolf, scores are multiplied by 20. We can see that model abilities are still the key factor for task completion. CS is a good indicator for TS given three pairs of scenarios having the one model having the highest TS and CS at the same time.
Figure 5: Comparison of Different Coordination Protocols.—Tree, Star, Graph, and Chain—across multiple evaluation metrics. Specially, the token usages are scaled such that the lowest value is 0 and the hightest value is 100 . Details about metrics used for research task can be found at A.4
Figure 6: Average Metrics for Research Tasks for different planning prompt strategies. Cognitive Evolve Planning show best result in CS.
Figure 7: Scores for gpt-4o-mini across different iterations in Minecraft. The figure presents Task Score (TS), Communication Score (CS), Planning Score (PS), and Collaboration Score (CoS) over multiple iterations.
Figure 8: Scaling on Agents Number in Research Scenario. This figure shows the impact of agent number on KPI, Coordination Score, and Task Score.
Model
Total
Strat. Info.
Trust Collab.
Role Iter.
gpt-4o-mini
100
97
100
42
gpt-3.5
100
100
100
66
llama-3.1-8b
100
96
98
38
llama-3.1-70b
100
100
100
55
llama-3.3-70b
100
100
100
89
Grand Total
500
493
498
290
Table 2: Summary of Emergent Behaviors in Werewolf Scenarios
File
Total
Strat. Info.
Trust Collab.
Role Iter.
Buyer Side
buyer-gpt-3.5-turbo
100
81
1
81
buyer-gpt-4o-mini
100
88
0
88
buyer-Llama-3.1-70B
100
81
2
80
buyer-Llama-3.1-8B
100
92
0
87
buyer-Llama-3.3-70B
100
84
1
82
Table 3: Summary of Emergent Behaviors in Bargaining Scenarios (Buyer vs. Seller).
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Comm (Human)
Plan (Human)
Comm (Machine)
Plan (Machine)
llama31_70b
3.19
3.19
3.12
3.00
llama33
3.94
3.44
3.89
3.89
gpt-4o-mini
3.61
3.33
3.00
3.00
gpt3.5-turbo
3.75
3.44
4.00
3.75
llama31_8b
2.62
3.06
3.00
3.00
Appendix
Table 4: Comparison of human vs. machine evaluation scores in a Werewolf scenario.
Metric
Mean (M.)
Std(M.)
Range (M.)
Mean (H.)
Std(H.)
Range (H.)
Communication
2.17
2.35
[–1.0, 5.0]
1.69
2.19
[–1.0, 5.0]
Planning
3.55
0.61
[2.0, 5.0]
3.41
0.73
[1.0, 5.0]
Appendix
Table 5: Summary statistics of coordination scores. The score of –1 in communication indicates runs with no communication. M. stands Machine and H. stands for Human.
Metric
Pearson’s r
p-value
Spearman’s ρ
p-value
Communication
0.8057
0.0000
0.7467
0.0000
Planning
0.2685
0.0000
0.3601
0.0000
Appendix
Table 6: Correlation between human and model-generated coordination metrics.
Correlation Type
Average Coefficient
Average p-value
Pearson
0.3290
0.0004
Spearman
0.3641
0.1923
Appendix
Table 7: Average inter-annotator correlation across metrics. Results indicate reasonable human agreement, especially in Communication.
Figure 9: Task Content Case.
Figure 10: Agent Profile Case.
Figure 11: 5Q cases.
Faction
Action/Outcome
Points
Villager
Villager candidate elected as sheriff
+2
Guard successfully protects a target from werewolf attack
+2
Witch successfully saves a target from werewolf attack
+2
Witch uses poison to kill a werewolf
+2
Werewolf is voted out during the day
+2
Each villager who votes for a werewolf
+0.2
Appendix
Table 8: Scoring Rules for Villagers and Werewolves in Full-Game Simulation
Figure 12: Net Score vs. Result Score (Scatter Plot)
Model
Completion Ratio
Villager Net Score
llama3.1-8B
0.2412
-1.2055
llama3.1-70B
0.3641
-1.0736
llama3.3-70B
0.3754
0.2802
gpt-3.5-turbo
0.2217
-0.7272
gpt-4o-mini
0.2503
-1.4207
Appendix
Table 9: Single-Day Simulation metrics for each model: completion ratio, villager net score.
Model
Net Score
Result Score
Win Rate
llama3.1-8B
-5.0839
-2.3793
0.0115
llama3.1-70B
-5.2892
-2.0000
0.0323
llama3.3-70B
0.4511
-0.1915
0.3511
gpt-3.5-turbo
-2.8230
-1.3448
0.0920
gpt-4o-mini
-4.6649
-2.0825
0.0309
gpt-4o(baseline)
-2.1946
-0.7742
0.2473
Appendix
Table 10: Full-Run Simulation metrics for each model: net score, result score, and villager win rate.
Figure 13: Internal Reasoning and Proposed Speech for the Llama3.1-8B Seer
Figure 14: The internal reasoning processes of the GPT-4o Seer (above) and Witch (below) when faced with critical decisions.
Figure 15: Key Nightly Actions and Sheriff Transitions in the Werewolf Game of LLama3.3-70B vs GPT-4o
Trait
Very Negative
Moderately Negative
Slightly Negative
Slightly Positive
Moderately Positive
Very Positive
OPE (Openness)
60%
71%
unimaginative (62%)
imaginative (55%)
77%
75%
CON (Conscientiousness)
68%
68%
irresponsible (66%)
responsible (67%)
63%
68%
EXT (Extraversion)
60%
67%
introverted (59%)
extraverted (58%)
75%
81%
AGR (Agreeableness)
71%
69%
distrustful (59%)
trustful (71%)
68%
62%
NEU (Neuroticism)
59%
59%
relaxed (81%)
nervous (70%)
55%
76%
Appendix
Table 11: Distribution of Personality Traits in Percentage
Figure 16: Task Content Case: One Happy Camper High Chair Banner.
Figure 18: Negotiation Result Summary (gpt 4o-mini) for One Happy Camper High Chair Banner.
Model
B-Comm
B-Plan
B-Collab Avg
S-Comm
S-Plan
S-Collab Avg
Final Bargaining
gpt-3.5-turbo
3.590
3.550
3.570
3.700
3.560
3.630
3.600
gpt-4o-mini
3.550
3.510
3.530
4.020
3.760
3.890
3.710
Llama-3.1-70B-Instruct-Turbo
3.030
3.480
3.255
4.180
3.600
3.890
3.573
Llama-3.1-8B-Instruct-Turbo
3.710
3.490
3.600
3.840
3.630
3.735
3.668
Llama-3.3-70B-Instruct-Turbo
3.010
3.430
3.220
3.930
3.540
3.735
3.478
Appendix
Table 12: Buyer and Seller detailed scores (Communication, Planning, and their Collab average), plus the Final Bargaining Score for each model.
Bargaining (TS)
Model
Buyer
Seller
Meta-Llama-3.1-8B
3.573
3.708
Meta-Llama-3.1-70B
3.557
3.656
Meta-Llama-3.3-70B
3.519
3.796
gpt3.5-turbo
3.535
3.632
gpt-4o-mini
3.578
3.869
Appendix
Table 13: Bargaining (TS) Performance
Figure 19: Distribution of Number of Blocks in Building Blue Prints
Figure 20: Number of Blocks vs Block Hit Rate for Different Models
Figure 21: Functin Call Executability by Model
Figure 22: Communication Prompt used to evaluate clarity, decision-making, and alignment with social relationships/profiles in a multiagent system.
Figure 23: Planning Prompt used to evaluate how well the agents define roles, assign tasks, and distribute workloads in a multiagent system, with automatic line wrapping.
Figure 24: KPI Prompt used to check if milestones such as “form 5q” or “improve 5q” have been achieved in a research iteration.
Figure 25: Task Score (5Q) Prompt for assessing the final research idea’s innovation, safety, and feasibility, with automatic line wrapping.
Figure 26: Good Communication Case #1. An example of two research agents dive deep into one potential research topics, and give insightful views.
Figure 27: Bad Communication Case #1. A snippet of repeated messages with minimal progress and various self-repetitions, reflecting inefficient communication patterns.
Agent systems powered by multimodal large language models (MLLMs) have advanced rapidly in recent years, yet existing embodied-agent benchmarks still lack fine-grained diagnostics for multi-agent coordination. Most benchmarks either focus on single-agent task completion or summarize multi-agent behavior with overall task success rates, which can obscure coordination failures such as duplicated work, violations of ordering constraints, resource contention, and desynchronized handoffs. In this paper, we introduce CoCoBench, a construct-level benchmark for evaluating multi-agent embodied coordination in executable household tasks. CoCoBench contains 897 oracle-validated instances organized around four recurring coordination constructs: task allocation, sequential ordering, mutual exclusion, and handoff coordination. In addition to task success rate, CoCoBench provides construct-level scores that measure whether agents coordinate effectively. We evaluate 11 leading MLLMs across different coordination modes, observation inputs, and numbers of agents. The results show that coordination ability is highly construct-specific: strong overall performance does not imply balanced competence across different coordination types. These findings point to new directions for designing targeted model architectures and improving multi-agent coordination ability.
Existing multi-agent benchmarks primarily test in competitive settings, short-horizon interactions under 20 steps, or simply aggregate individual performance, failing to isolate and highlight genuine collaboration capabilities of LLM-based agents. We introduce AgentWorld, a benchmark of 100 human-annotated tasks (with 100 augmented variants) for evaluating long-horizon, multi-agent collaboration. Tasks span 50+ interaction rounds across a rich MMORPG sandbox and require 3-20 agents with asymmetric roles and abilities to coordinate through communication, joint planning, and resource sharing under a blackbox setting where each agent acts independently without access to others' internal states. To quantify collaboration effectiveness in addition to conventional binary task success, we propose Causal Collaboration Effectiveness (CCE), a graph-based metric that traces causal dependencies between agent actions and measures what fraction of a team's effort actually contributed to the outcome. Experiments with Gemini 3 Flash, Claude Haiku 4.5, GPT-5 Mini, and DeepSeek R1-70B show that even the best model achieves only 52.0% task success, with systematic failure modes including communication breakdowns, role confusion, and inability to maintain shared plans across rounds. AgentWorld is fully open-source.
Raphael Shu, Yusen Zhang, Young Min Cho +7
OpenAgents · Columbia University · University of Pennsylvania +2
Large language model (LLM)-based multi-agent systems (MAS) tackle challenging tasks by dividing work among agents and coordinating their contributions toward shared objectives. Evaluating these systems therefore requires benchmarks that assess not only task-solving performance but also the ability to handle interdependent work. However, existing benchmarks often emphasize task difficulty without substantial collaboration requirements or evaluate only a limited range of predefined collaboration patterns. In this paper, we introduce MAC-Bench, a comprehensive benchmark for evaluating multi-agent collaboration through interconnected tasks. Rather than imposing predefined agent roles or communication protocols, MAC-Bench embeds collaboration requirements directly into task dependencies and shared-resource constraints. Specifically, we mine reusable collaboration structures from selected benchmark tasks at two levels. Local collaboration patterns capture recurring interactions among subtasks, while global task structures characterize how these interactions contribute to complete workflows. Through coarse-to-fine composition, these structures are recombined into task graphs combining different collaboration requirements. Tasks are instantiated in executable environments with automated verification of partial progress and complete task resolution.