Large language model (LLM)-based multi-agent systems (MAS) tackle challenging tasks by dividing work among agents and coordinating their contributions toward shared objectives. Evaluating these systems therefore requires benchmarks that assess not only task-solving performance but also the ability to handle interdependent work. However, existing benchmarks often emphasize task difficulty without substantial collaboration requirements or evaluate only a limited range of predefined collaboration patterns. In this paper, we introduce MAC-Bench, a comprehensive benchmark for evaluating multi-agent collaboration through interconnected tasks. Rather than imposing predefined agent roles or communication protocols, MAC-Bench embeds collaboration requirements directly into task dependencies and shared-resource constraints. Specifically, we mine reusable collaboration structures from selected benchmark tasks at two levels. Local collaboration patterns capture recurring interactions among subtasks, while global task structures characterize how these interactions contribute to complete workflows. Through coarse-to-fine composition, these structures are recombined into task graphs combining different collaboration requirements. Tasks are instantiated in executable environments with automated verification of partial progress and complete task resolution.
Figures & tables
Figure 1: MAC-Bench overview. Left: a packaging task links work branches through intermediate artifacts. Upper right: six task categories across two domains. Lower right: overall Progress Score (PS) for five systems with two LLM backbones; the highest Resolved Rate (RR) is 6.25%.
Figure 2: Task structures in general-purpose agent benchmarks. Percentages over 50 sampled tasks per benchmark. Unresolved denotes uncertain classifications from available materials (Appendix G.3.1 ).
Figure 3: Construction pipeline of MAC-Bench. (a) Mining extracts reusable local patterns and global task structures from existing tasks. (b) Composition recombines global structures and expands them with local patterns. (c) Instantiation specifies concrete tasks, initial environments, and outcome verifiers.
Business Operations
Software Engineering
Overall
System
Finance
Service
Project
App Dev.
Lib Dev.
Dev. Tools
All
PS ↑
RR ↑
PS ↑
RR ↑
PS ↑
RR ↑
PS ↑
RR ↑
PS ↑
RR ↑
PS ↑
RR ↑
PS ↑
RR ↑
GPT-6 Luna
Single Agent (ReAct) ( Yao et al., 2023 )
21.08
0.00
24.93
0.00
26.08
12.50
7.79
0.00
8.08
0.00
18.50
12.50
17.74
4.17
AOrchestra ( Ruan et al., 2026 )
0.00
0.00
14.86
0.00
4.71
0.00
8.08
0.00
2.46
0.00
2.08
0.00
5.37
0.00
Magentic-One ( Fourney et al., 2024 )
0.00
0.00
1.12
0.00
15.69
0.00
0.00
0.00
4.00
0.00
18.20
0.00
6.50
0.00
Table 1: Main results on MAC-Bench. Progress Score (PS) and Resolved Rate (RR) are percentages ( ↑ ). Bold and underlined values indicate the best and second-best scores in each column within each backbone. Zero scores are left unmarked.
Figure 4: Task-relation outcomes with DeepSeek-Flash. (a) 14 dependency checks on nine tasks. (b) Six shared-resource checks on four tasks. Percentages are averaged within tasks, then across tasks, and rounded. Unavailable means at least one required output is missing; unassessable denotes inconclusive evidence.
Figure 5: Meta-Team collaboration cases. (a) Different workers leave booking and calendar dates inconsistent for six of 20 volunteers. (b) The planner’s CLI calls the developer’s unimplemented isolated-build environment constructor, blocking the isolated build.
Figure 6: Work structures in MAC-Bench. (a) Task frequencies by branch and cross-branch dependency counts. (b) Exclusive profiles: dependencies only (Dep.), integration without sharing (Integr.), or integration with sharing only within (Within) or across branches (Across). Dashes mark zero counts.
Task
PS after violation (%)
Missing work
Broken relation
Preservation violation
Business operations
Studio closeout
85.7
85.7
100.0 †
Workspace recovery
85.7
85.7
100.0 †
Housing allocation
75.0
75.0
100.0 †
Market allocation
83.3
83.3
100.0 †
Table 2: Scoring under controlled changes. All displayed results fail complete grading; † denotes PS of 100% despite a failed preservation check. References, legal alternatives, and repairs pass on all twelve tasks with PS of 100%.
Appendix figures & tables31 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Task distribution in MAC-Bench. Six equally sized categories contain eight test tasks each, totaling 48 tasks.
Category
Representative work and relations
Business Operations
Financial Management
Resolve credits, expenses, and balances; align recovery commitments with capacity and funding.
Service Management
Reconcile eligibility, appointments, and allocations; keep bookings and schedules consistent.
Project Management
Coordinate permissions, content, staffing, and costs; make delivery records reflect selected arrangements.
Software Engineering
Application Development
Implement interacting data, service, and user-facing features while retaining existing behavior.
Appendix
Table 3: Task categories in MAC-Bench. Each category contains eight test tasks. Examples summarize public requirements from the final collection.
Category
Branch range
Integr.
Sharing
Within
Across
Finance
7–10
8
8
0
Service
3–6
8
6
2
Projects
5–8
8
2
6
Applications
4–7
8
0
8
Libraries
5–5
7
1
5
Appendix
Table 4: Final-task structure summary. Each category contains eight tasks. Integr. counts tasks with multi-input integration. Share within counts tasks with within-branch constraints only; Share across counts tasks with cross-branch constraints.
Source
Mining
Held-out
MBABench ( Yen et al., 2026 )
17
9
UniPat SaaS-Bench ( Shi et al., 2026 )
12
5
RoadmapBench ( Xu et al., 2026b )
30
15
SaaSBench ( Ren et al., 2026 )
20
9
SWE-Milestone ( Deng et al., 2026 )
19
11
LHTB ( Li et al., 2026 )
2
1
Appendix
Table 5: Source-task composition of the mining and held-out corpora. Counts refer to accepted task graphs, rather than instantiated benchmark tasks.
Figure 8: Task-level PS differences from ReAct. Each system is compared on the same 48 tasks. Circle area indicates frequency; diamonds and lines show means and 95% task-resampling intervals. Win, Tie, and Loss count tasks with higher, equal, and lower PS than ReAct.
System
Median use
Episode stops
Tokens (M)
Calls
Cap
End
Other
ReAct
13.38
222
34
8
6
AOrchestra
15.00
995
44
3
1
Magentic-One
15.01
306
42
3
3
Meta-Team
14.49
297.5
35
13
0
Superagent
15.02
132
42
2
4
Appendix
Table 6: Resource use and stopping across the 48 tasks. Tokens and Calls are median total tokens and recorded model calls. Cap, End, and Other count token-limit stops, normal episode endings, and other stops (including blocked and failed); each row totals 48. End does not imply task resolution.
Figure 9: Dependent results by task relation. Cells give item counts in each state; split cells show multiple states. For unavailable results, ↑ and ↓ indicate that only the upstream or downstream side is available; unmarked gray cells indicate neither. Stars mark relations with complete paired outputs and distinct agent contributors, traced in Table 7 .
Cross-agent relation
Items
Inconsistent
First
Final
AOrchestra
Radio rights → text
24
24
24
Market eligibility → handover
84
72
72
Market allocation → inspections
48
39
39
Meta-Team
Appendix
Table 7: Changes in cross-agent consistency. First is the earliest state with comparable records for all listed items; Final is episode termination. The bold count reflects five corrected booking–calendar mismatches. Other rows are unchanged. Conservation counts 32 group placements across 16 sessions.
Condition
Change to the evaluated result
Expected to pass complete grading
Reference
Complete result from the reference solution.
Legal alternative
A different permitted choice or implementation, with dependent results updated.
Repair
Correct the inconsistent business records or restore the edited software component.
Expected to fail complete grading
Missing work
Remove a required result register or omit required functionality.
Appendix
Table 8: Controlled scoring protocol. Each of twelve tasks is evaluated under all six conditions. The expected decision follows the task requirements; Table 2 reports observed scores for the violations.
Condition
Goal passed
PS
Alloc.
Fin.
Appt.
Hand.
(%)
Reference
✓
✓
✓
✓
100
Stale handover
✓
✓
✓
×
75
Repaired handover
✓
✓
✓
✓
100
Remove an upstream register from the reference
Allocation
×
✓
×
×
25
Appendix
Table 9: Goal-level diagnosis in housing allocation. Columns show allocation, finance, appointments, and handover. A stale handover fails its own goal; missing upstream records also invalidate dependent goals. ✓ and × indicate pass and fail.
Figure 10: Task-structure reuse on 50 held-out tasks. (a) Coverage versus corpus size, with ranges across ten nested sequences. (b) Coverage distributions with 100 mining tasks.
Held-out source
Tasks
Local
Rules
MBABench
9
57.7
22.7
UniPat SaaS-Bench
5
23.8
2.2
RoadmapBench
15
72.9
34.9
SaaSBench
9
96.7
79.4
SWE-Milestone
11
76.4
51.2
LHTB
1
70.0
33.3
Appendix
Table 10: Held-out structure coverage by source (%). Each value averages task-level coverage; All tasks averages the 50 tasks rather than the source means.
Figure 11: Accepted outputs and distinct structures. Filled and open markers show distinct accepted structures and all accepted outputs from 60 requests, respectively. Their gap counts repeated structures; the remainder fails checks. No eligible larger fixed-global source exists (—).
Figure 12: Expected discovery of distinct accepted structures within each recorded 60-request batch. Each curve is computed from uniformly sampled subsets of the recorded requests, including failed requests.
Figure 13: Effect of the global operation cap on target attainment. Each point uses 60 paired requests; bands show 95% within-seed request-bootstrap intervals. The resource-backed check and target scales are the same as in Figure 11 .
Figure 14: Sensitivity of target attainment to the acceptance criterion. All four checks use the same 60 requests per method and target scale. Resource-backed, With delivery, and All subtasks denote the criteria in Appendix G.3.4 .
Generator
Smaller graphs
Larger graphs
Pass (%)
Time (s)
Pass (%)
Time (s)
Composition
80.0
0.071
91.7
0.212
Direct LLM
38.3
63.69
21.7
75.93
w/o guidance
5.0
0.347
26.7
0.352
Appendix
Table 11: Target attainment and recorded mean graph-generation time, with 60 requests per generator and scale. Pass uses the resource-backed criterion; time excludes mining and instantiation.
Construction
Acceptance (%)
Ours
40.0
Source + refinement
40.0
Direct LLM
46.7
Appendix
Table 12: Acceptance of generated scenario specifications under common structural checks and automatic review supported by specification excerpts. Each condition uses 60 requests across 30 paired backgrounds.
Method
Temperature
Max. output tokens
ReAct
0.0
8,192
AOrchestra
0.2
8,192
Meta-Team
0.2
32,768
Magentic-One
0.2
8,192
Superagent
Default
32,768
Appendix
Table 13: Standard model-call decoding parameters for both backbones. Default denotes the provider’s default temperature.
Control
Setting
AOrchestra
Delegation
Sequential
Coordinator-attempt limit
3,000
Worker-step limit
50
Worker memory / recent records
10 / 3
Meta-Team
Appendix
Table 14: Method-specific execution controls, shared by both backbones. Shared episode limits also apply.
Task
Dependencies
Shared
Budget
Exhibition recovery
1
0
15
Anthology release
2
0
15
Radio release
2
0
15
Conservation programme
2
1
15
Membership renewal
1
0
15
Housing handover
1
1
10
Appendix
Table 15: Tasks and relation checks in the collaboration analysis. Shared-resource checks include constraints among allocations within a work group. Budgets are in millions of tokens.
“I (planner) am taking src/build/main.py — do NOT edit it. You own src/build/env.py and src/build/_builder.py . Focus on env.py first”
5:03
Planner / edit
Writes the CLI, including from .env import DefaultIsolatedEnv and with DefaultIsolatedEnv(installer=installer) as env: .
5:06
Tool / local test
3 passed in 0.03s (CLI helper and module tests).
Appendix
Table 18: Chronological excerpts from a Meta-Team build-tool episode. Times are elapsed minutes and seconds. Message, code, and tool-output fragments reproduce the record; edit descriptions summarize operations. The final row reports complete task grading after the episode.
Task. Prepare a radio programme and corrected transcripts. Each transcript must retain the speaker label, attribution, and embargo specified for its segment.
Time
Actor / event
Selected trajectory excerpt
10:54
Coordinator → worker-4
Delegates the rights register: “Compute the correct rights manifest per task rules”.
13:22
Worker-4 / write
Creates NCR RESULT rights . For SEG001: speaker_label: Contributor 01 ; attribution: Northbank source SEG001 ; embargo: 2027-04-19 .
13:24
Tool / receipt
The rights-note creation succeeds.
37:12
Coordinator → worker-14
Delegates transcripts with fields “ speaker_label : exact allowed label” and “ attribution : exact source attribution”; requests that both new notes be read back.
40:16
Worker-14 / write
Creates NCR RESULT transcripts . For SEG001: speaker_label: Presenter ; attribution: Riverbend Voices archive ; embargo: 2027-04-19 .
Appendix
Table 19: Chronological excerpts from an AOrchestra radio episode. Times are elapsed minutes and seconds. Short quotations reproduce instruction fragments; operation rows show selected fields. The relation check compares final artifacts, separately from tool-call success.
Task. Restore a Python build frontend with a working CLI, build engine, and isolated environment. The delivered components must support independent consumer execution.
Time
Actor / event
Selected trajectory excerpt
12:13
Orchestrator → Coder
“Implement src/build/main.py in full” and “Emit each as one executable sh block for ComputerTerminal.”
12:25
Coder / proposal
“I’ll implement main.py in full”; returns a shell block writing the CLI and running tests.
12:28
Orchestrator → terminal
“Execute Coder’s latest fenced sh block exactly as written” and report stdout/stderr.
12:30
Tool / test
ERROR collecting tests/test_main.py : the flaky marker is unregistered. Tests stop during collection.
13:16
Orchestrator → terminal
Relays a later Coder block registering the marker and rerunning the utility and CLI tests.
Appendix
Table 20: Chronological excerpts from a Magentic-One build-tool episode. The Coder proposes code; ComputerTerminal executes it through the orchestrator. Message fragments reproduce the record, while descriptions summarize operations. Local test errors and final goal checks are reported separately.
Task. Relocate a museum exhibition after its planned venue becomes unavailable. Arrange exhibit inspections, installation, school visits, and public opening with qualified staff and non-overlapping shifts for each employee.
Time
Actor / event
Selected trajectory excerpt
16:46
Worker → subagent
Delegates a 219-entry work plan: “Each row must become ONE native submitted Shift Assignment”. States: “There are no duplicate rows requiring dedupe EXCEPT if you are resuming”.
16:55–17:08
Subagent / read
Reads the shared plan. Six separate rows, including inspections of exhibits OBJ001 and OBJ005, assign employee E002 to the same June 29 shift, 08:00–17:00.
17:31
Tool / receipt
The subagent’s first assignment for E002 on June 29 is submitted successfully.
17:51
Subagent / create
Requests another assignment for E002 on June 29 using the same shift. The application returns a conflict.
18:00–18:08
Subagent / submit
Submitting further shifts for E002 on dates already covered by submitted shifts returns three conflicts. The subagent inspects the saved records.
Appendix
Table 21: Superagent’s plan-to-execution handoff in an exhibition-recovery task. Times are elapsed minutes and seconds. Quotations reproduce delegation instructions; other rows summarize the plan, tool receipts, and final state.
Table 22: Prompt template for extracting task objectives, inputs, outputs, and resource accesses.
Table 23: Prompt template for recursive task decomposition.
Table 24: Prompt templates for scenario refinement and extraction of outcome requirements.
Table 25: Prompt templates for preliminary task-structure analysis and the Direct LLM baseline in the graph-composition experiment.
Large language models (LLMs) have progressively evolved into the core of autonomous agents. Building on this progress, LLM-based multi-agent systems (MAS) coordinate multiple agents into a synergistic team to accomplish complex tasks that exceed the capabilities of individual agents. The effectiveness of such systems depends not only on the agents themselves, but also on how collaboration mechanisms are designed and organized. Note that real-world collaboration is typically partially observable, where each agent can only access partial information about the environment due to physical or privacy-related constraints. However, many existing multi-agent benchmarks assume global observability, and leave limited support for systematically evaluating collaboration mechanisms. To bridge this gap, we introduce MASBench, a multi-agent collaboration benchmark designed under partially observable constraints. It is organized into three progressive task categories: Reasoning, Scheduling, and Game. Through this structure, we progressively evaluate three representative collaboration mechanisms: Protocol, Memory, and Routing. MASBench further provides deterministic evaluation metrics, including performance score, communication cost, and cost effectiveness, to characterize both collaboration outcomes and communication overhead. Experiments across diverse LLM backbones and mechanism configurations offer empirical guidance for effective MAS design. Code is available at: https://github.com/BUPT-GAMMA/MASBench
Qizhi Chu, Zekai Yu, Sijie Wen +5
Beijing University of Posts and Telecommunications · Shanghai Jiao Tong University · Tsinghua University
LLM-based multi-agent systems (MAS) have shown promise in complex problem solving. As MAS methods diversify, systematic evaluation becomes increasingly challenging. However, existing benchmarks largely focus on final outcomes, leaving unclear how collaboration gains arise, are preserved, or are lost. To address this limitation, we introduce MASTraceBench, a benchmark for diagnosing collaboration gains through proposal trajectories in MAS. Across six cooperative and competitive tasks, MASTraceBench tracks and grades proposal trajectories and provides a multi-layer metric suite covering Task Score, Collaboration Gain, proposal-trajectory indicators, and Token Cost. Using MASTraceBench, we systematically compare representative MAS methods not only by final performance, but also by how agent proposals evolve and are aggregated into the final answer. This analysis reveals a recurring pattern: final MAS answers rarely surpass the strongest initial proposal; interaction often lifts initially weaker proposals toward it, while strong initial proposals are seldom further improved and may regress. To reduce this risk, we propose CLEARS, which replaces whole-proposal exchange with claim-level evaluation across agents to guide reliable synthesis. CLEARS more often preserves or improves upon the strongest initial proposal and achieves the highest Collaboration Gain on five of the six tasks.
Yapeng Li, Songze Li, Shuang Yu +4
Harbin Institute of Technology, Harbin, China · Peking University, Beijing, China
Multi-agent systems (MAS) built on Large Language Models (LLMs) are proliferating rapidly, but their heterogeneous execution traces provide no common basis for evaluation across methods. Outcome-only benchmarks discard collaborations, whereas LLM-as-Judge evaluation requires additional, model-dependent inference and can vary with the LLM and rubric. We introduce a generalizable evaluation framework that maps native MAS traces into a shared space of unified collaboration graphs, enabling different methods to be evaluated under the same representation, reference set, and metric panel. Candidate graphs are compared with a query-specific reference forest. Each forest is a benchmark-provided collection of verified-success graphs: it records diverse ways in which representative MAS methods can complete the task, rather than prescribing a unique optimal process. Instantiating the framework as ForestBench, we filter 844 collaboration-necessary queries from seven public datasets, precompute ten successful target-conditioned reference graphs per query, and evaluate six representative MAS frameworks. Controlled backbone, reference-construction, and perturbation studies test the stability and scope of evaluation. Once the benchmark forests are built, ForestBench scores a trace in milliseconds without further LLM inference, providing a reusable structural basis for comparing diverse MAS collaboration traces.
Guo Chen, Ziwen Li, Reed Li +4
Southwest University Chongqing, China · Tencent Beijing, China · Institute of Computing Technology, Chinese Academy of Sciences Beijing, China