Large Language Models (LLMs) have shown remarkable capabilities as autonomous agents, yet existing benchmarks either focus on single-agent tasks or are confined to narrow domains, failing to capture the dynamics of multi-agent coordination and competition. In this paper, we introduce MultiAgentBench, a comprehensive benchmark designed to evaluate LLM-based multi-agent systems across diverse, interactive scenarios. Our framework measures not only task completion but also the quality of collaboration and competition using novel, milestone-based key performance indicators. Moreover, we evaluate various coordination protocols (including star, chain, tree, and graph topologies) and innovative strategies such as group discussion and cognitive planning. Notably, gpt-4o-mini reaches the average highest task score, graph structure performs the best among coordination protocols in the research scenario, and cognitive planning improves milestone achievement rates by 3%. Code and datasets are public available at https://github.com/MultiagentBench/MARBLE.
Figures & tables
Figure 1: Overview of MultiAgentBench evaluation process : Multi-Agent System Coordination in various interactive environments, with a focus on task performance, and coordination.
Figure 2: MARBLE : showcasing interactions between task information, persona data, domain databases, memory modules, and the environment through the coordinate engine and cognitive module.
Figure 3: Illustration of coordination protocols and planning prompt strategies . (a) shows centralized and decentralized planning structures (e.g., star, tree, graph, and chain). (b) describes strategies like group discussions and cognitive prompts, incorporating iterative feedback and task updates for effective planning.
Figure 4: Illustration of our benchmark curation and the dynamic milestones detecting for KPI metric.
Model
Research
Minecraft
Database
Coding
Bargaining
WereWolf
TS
CS
TS
CS
TS
CS
TS
CS
TS
CS
TS
CS
Meta-Llama-3.1-8B
80.87
52.40
6.12
54.40
34.00
40.00
59.90
67.24
72.81
73.36
12.64
60.00
Meta-Llama-3.1-70B
80.80
49.50
0.21
75.00
53.00
37.70
62.10
67.18
72.13
71.46
19.82
60.60
Meta-Llama-3.3-70B
80.00
72.00
9.15
69.00
28.50
40.00
56.60
74.40
73.15
69.56
36.33
76.30
gpt3.5-turbo
70.20
55.90
5.05
63.60
45.00
60.89
55.50
76.20
71.67
72.00
15.69
75.90
gpt-4o-mini
84.13
52.00
33.60
61.50
45.00
43.22
65.10
66.30
74.47
74.20
14.06
60.10
Table 1: Average Task Score ( TS ) (%) and the Coordination Score ( CS ) for Minecraft, Database, Coding, Bargaining, and WereWolf, scores are multiplied by 20. We can see that model abilities are still the key factor for task completion. CS is a good indicator for TS given three pairs of scenarios having the one model having the highest TS and CS at the same time.
Figure 5: Comparison of Different Coordination Protocols.—Tree, Star, Graph, and Chain—across multiple evaluation metrics. Specially, the token usages are scaled such that the lowest value is 0 and the hightest value is 100 . Details about metrics used for research task can be found at A.4
Figure 6: Average Metrics for Research Tasks for different planning prompt strategies. Cognitive Evolve Planning show best result in CS.
Figure 7: Scores for gpt-4o-mini across different iterations in Minecraft. The figure presents Task Score (TS), Communication Score (CS), Planning Score (PS), and Collaboration Score (CoS) over multiple iterations.
Figure 8: Scaling on Agents Number in Research Scenario. This figure shows the impact of agent number on KPI, Coordination Score, and Task Score.
Model
Total
Strat. Info.
Trust Collab.
Role Iter.
gpt-4o-mini
100
97
100
42
gpt-3.5
100
100
100
66
llama-3.1-8b
100
96
98
38
llama-3.1-70b
100
100
100
55
llama-3.3-70b
100
100
100
89
Grand Total
500
493
498
290
Table 2: Summary of Emergent Behaviors in Werewolf Scenarios
File
Total
Strat. Info.
Trust Collab.
Role Iter.
Buyer Side
buyer-gpt-3.5-turbo
100
81
1
81
buyer-gpt-4o-mini
100
88
0
88
buyer-Llama-3.1-70B
100
81
2
80
buyer-Llama-3.1-8B
100
92
0
87
buyer-Llama-3.3-70B
100
84
1
82
Table 3: Summary of Emergent Behaviors in Bargaining Scenarios (Buyer vs. Seller).
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Comm (Human)
Plan (Human)
Comm (Machine)
Plan (Machine)
llama31_70b
3.19
3.19
3.12
3.00
llama33
3.94
3.44
3.89
3.89
gpt-4o-mini
3.61
3.33
3.00
3.00
gpt3.5-turbo
3.75
3.44
4.00
3.75
llama31_8b
2.62
3.06
3.00
3.00
Appendix
Table 4: Comparison of human vs. machine evaluation scores in a Werewolf scenario.
Metric
Mean (M.)
Std(M.)
Range (M.)
Mean (H.)
Std(H.)
Range (H.)
Communication
2.17
2.35
[–1.0, 5.0]
1.69
2.19
[–1.0, 5.0]
Planning
3.55
0.61
[2.0, 5.0]
3.41
0.73
[1.0, 5.0]
Appendix
Table 5: Summary statistics of coordination scores. The score of –1 in communication indicates runs with no communication. M. stands Machine and H. stands for Human.
Metric
Pearson’s r
p-value
Spearman’s ρ
p-value
Communication
0.8057
0.0000
0.7467
0.0000
Planning
0.2685
0.0000
0.3601
0.0000
Appendix
Table 6: Correlation between human and model-generated coordination metrics.
Correlation Type
Average Coefficient
Average p-value
Pearson
0.3290
0.0004
Spearman
0.3641
0.1923
Appendix
Table 7: Average inter-annotator correlation across metrics. Results indicate reasonable human agreement, especially in Communication.
Figure 9: Task Content Case.
Figure 10: Agent Profile Case.
Figure 11: 5Q cases.
Faction
Action/Outcome
Points
Villager
Villager candidate elected as sheriff
+2
Guard successfully protects a target from werewolf attack
+2
Witch successfully saves a target from werewolf attack
+2
Witch uses poison to kill a werewolf
+2
Werewolf is voted out during the day
+2
Each villager who votes for a werewolf
+0.2
Appendix
Table 8: Scoring Rules for Villagers and Werewolves in Full-Game Simulation
Figure 12: Net Score vs. Result Score (Scatter Plot)
Model
Completion Ratio
Villager Net Score
llama3.1-8B
0.2412
-1.2055
llama3.1-70B
0.3641
-1.0736
llama3.3-70B
0.3754
0.2802
gpt-3.5-turbo
0.2217
-0.7272
gpt-4o-mini
0.2503
-1.4207
Appendix
Table 9: Single-Day Simulation metrics for each model: completion ratio, villager net score.
Model
Net Score
Result Score
Win Rate
llama3.1-8B
-5.0839
-2.3793
0.0115
llama3.1-70B
-5.2892
-2.0000
0.0323
llama3.3-70B
0.4511
-0.1915
0.3511
gpt-3.5-turbo
-2.8230
-1.3448
0.0920
gpt-4o-mini
-4.6649
-2.0825
0.0309
gpt-4o(baseline)
-2.1946
-0.7742
0.2473
Appendix
Table 10: Full-Run Simulation metrics for each model: net score, result score, and villager win rate.
Figure 13: Internal Reasoning and Proposed Speech for the Llama3.1-8B Seer
Figure 14: The internal reasoning processes of the GPT-4o Seer (above) and Witch (below) when faced with critical decisions.
Figure 15: Key Nightly Actions and Sheriff Transitions in the Werewolf Game of LLama3.3-70B vs GPT-4o
Trait
Very Negative
Moderately Negative
Slightly Negative
Slightly Positive
Moderately Positive
Very Positive
OPE (Openness)
60%
71%
unimaginative (62%)
imaginative (55%)
77%
75%
CON (Conscientiousness)
68%
68%
irresponsible (66%)
responsible (67%)
63%
68%
EXT (Extraversion)
60%
67%
introverted (59%)
extraverted (58%)
75%
81%
AGR (Agreeableness)
71%
69%
distrustful (59%)
trustful (71%)
68%
62%
NEU (Neuroticism)
59%
59%
relaxed (81%)
nervous (70%)
55%
76%
Appendix
Table 11: Distribution of Personality Traits in Percentage
Figure 16: Task Content Case: One Happy Camper High Chair Banner.
Figure 18: Negotiation Result Summary (gpt 4o-mini) for One Happy Camper High Chair Banner.
Model
B-Comm
B-Plan
B-Collab Avg
S-Comm
S-Plan
S-Collab Avg
Final Bargaining
gpt-3.5-turbo
3.590
3.550
3.570
3.700
3.560
3.630
3.600
gpt-4o-mini
3.550
3.510
3.530
4.020
3.760
3.890
3.710
Llama-3.1-70B-Instruct-Turbo
3.030
3.480
3.255
4.180
3.600
3.890
3.573
Llama-3.1-8B-Instruct-Turbo
3.710
3.490
3.600
3.840
3.630
3.735
3.668
Llama-3.3-70B-Instruct-Turbo
3.010
3.430
3.220
3.930
3.540
3.735
3.478
Appendix
Table 12: Buyer and Seller detailed scores (Communication, Planning, and their Collab average), plus the Final Bargaining Score for each model.
Bargaining (TS)
Model
Buyer
Seller
Meta-Llama-3.1-8B
3.573
3.708
Meta-Llama-3.1-70B
3.557
3.656
Meta-Llama-3.3-70B
3.519
3.796
gpt3.5-turbo
3.535
3.632
gpt-4o-mini
3.578
3.869
Appendix
Table 13: Bargaining (TS) Performance
Figure 19: Distribution of Number of Blocks in Building Blue Prints
Figure 20: Number of Blocks vs Block Hit Rate for Different Models
Figure 21: Functin Call Executability by Model
Figure 22: Communication Prompt used to evaluate clarity, decision-making, and alignment with social relationships/profiles in a multiagent system.
Figure 23: Planning Prompt used to evaluate how well the agents define roles, assign tasks, and distribute workloads in a multiagent system, with automatic line wrapping.
Figure 24: KPI Prompt used to check if milestones such as “form 5q” or “improve 5q” have been achieved in a research iteration.
Figure 25: Task Score (5Q) Prompt for assessing the final research idea’s innovation, safety, and feasibility, with automatic line wrapping.
Figure 26: Good Communication Case #1. An example of two research agents dive deep into one potential research topics, and give insightful views.
Figure 27: Bad Communication Case #1. A snippet of repeated messages with minimal progress and various self-repetitions, reflecting inefficient communication patterns.