MAC-Bench: Benchmarking Multi-Agent Collaboration via Interconnected Tasks
Organizations: Xiamen University · Jilin University · ByteDance
Abstract
Large language model (LLM)-based multi-agent systems (MAS) tackle challenging tasks by dividing work among agents and coordinating their contributions toward shared objectives. Evaluating these systems therefore requires benchmarks that assess not only task-solving performance but also the ability to handle interdependent work. However, existing benchmarks often emphasize task difficulty without substantial collaboration requirements or evaluate only a limited range of predefined collaboration patterns. In this paper, we introduce MAC-Bench, a comprehensive benchmark for evaluating multi-agent collaboration through interconnected tasks. Rather than imposing predefined agent roles or communication protocols, MAC-Bench embeds collaboration requirements directly into task dependencies and shared-resource constraints. Specifically, we mine reusable collaboration structures from selected benchmark tasks at two levels. Local collaboration patterns capture recurring interactions among subtasks, while global task structures characterize how these interactions contribute to complete workflows. Through coarse-to-fine composition, these structures are recombined into task graphs combining different collaboration requirements. Tasks are instantiated in executable environments with automated verification of partial progress and complete task resolution.
Figures & tables
| Business Operations | Software Engineering | Overall | ||||||||||||
| System | Finance | Service | Project | App Dev. | Lib Dev. | Dev. Tools | All | |||||||
| PS | RR | PS | RR | PS | RR | PS | RR | PS | RR | PS | RR | PS | RR | |
| GPT-6 Luna | ||||||||||||||
| Single Agent (ReAct) ( Yao et al., 2023 ) | 21.08 | 0.00 | 24.93 | 0.00 | 26.08 | 12.50 | 7.79 | 0.00 | 8.08 | 0.00 | 18.50 | 12.50 | 17.74 | 4.17 |
| AOrchestra ( Ruan et al., 2026 ) | 0.00 | 0.00 | 14.86 | 0.00 | 4.71 | 0.00 | 8.08 | 0.00 | 2.46 | 0.00 | 2.08 | 0.00 | 5.37 | 0.00 |
| Magentic-One ( Fourney et al., 2024 ) | 0.00 | 0.00 | 1.12 | 0.00 | 15.69 | 0.00 | 0.00 | 0.00 | 4.00 | 0.00 | 18.20 | 0.00 | 6.50 | 0.00 |
| Task | PS after violation (%) | ||
|---|---|---|---|
| Missing work | Broken relation | Preservation violation | |
| Business operations | |||
| Studio closeout | 85.7 | 85.7 | 100.0 |
| Workspace recovery | 85.7 | 85.7 | 100.0 |
| Housing allocation | 75.0 | 75.0 | 100.0 |
| Market allocation | 83.3 | 83.3 | 100.0 |
Appendix figures & tables31 assets
Supplementary material from the paper’s appendix.
Appendix
| Category | Representative work and relations |
|---|---|
| Business Operations | |
| Financial Management | Resolve credits, expenses, and balances; align recovery commitments with capacity and funding. |
| Service Management | Reconcile eligibility, appointments, and allocations; keep bookings and schedules consistent. |
| Project Management | Coordinate permissions, content, staffing, and costs; make delivery records reflect selected arrangements. |
| Software Engineering | |
| Application Development | Implement interacting data, service, and user-facing features while retaining existing behavior. |
| Category | Branch range | Integr. | Sharing | |
|---|---|---|---|---|
| Within | Across | |||
| Finance | 7–10 | 8 | 8 | 0 |
| Service | 3–6 | 8 | 6 | 2 |
| Projects | 5–8 | 8 | 2 | 6 |
| Applications | 4–7 | 8 | 0 | 8 |
| Libraries | 5–5 | 7 | 1 | 5 |
| Source | Mining | Held-out |
|---|---|---|
| MBABench ( Yen et al., 2026 ) | 17 | 9 |
| UniPat SaaS-Bench ( Shi et al., 2026 ) | 12 | 5 |
| RoadmapBench ( Xu et al., 2026b ) | 30 | 15 |
| SaaSBench ( Ren et al., 2026 ) | 20 | 9 |
| SWE-Milestone ( Deng et al., 2026 ) | 19 | 11 |
| LHTB ( Li et al., 2026 ) | 2 | 1 |
| System | Median use | Episode stops | |||
|---|---|---|---|---|---|
| Tokens (M) | Calls | Cap | End | Other | |
| ReAct | 13.38 | 222 | 34 | 8 | 6 |
| AOrchestra | 15.00 | 995 | 44 | 3 | 1 |
| Magentic-One | 15.01 | 306 | 42 | 3 | 3 |
| Meta-Team | 14.49 | 297.5 | 35 | 13 | 0 |
| Superagent | 15.02 | 132 | 42 | 2 | 4 |
| Cross-agent relation | Items | Inconsistent | |
| First | Final | ||
| AOrchestra | |||
| Radio rights text | 24 | 24 | 24 |
| Market eligibility handover | 84 | 72 | 72 |
| Market allocation inspections | 48 | 39 | 39 |
| Meta-Team | |||
| Condition | Change to the evaluated result |
|---|---|
| Expected to pass complete grading | |
| Reference | Complete result from the reference solution. |
| Legal alternative | A different permitted choice or implementation, with dependent results updated. |
| Repair | Correct the inconsistent business records or restore the edited software component. |
| Expected to fail complete grading | |
| Missing work | Remove a required result register or omit required functionality. |
| Condition | Goal passed | PS | |||
| Alloc. | Fin. | Appt. | Hand. | (%) | |
| Reference | ✓ | ✓ | ✓ | ✓ | 100 |
| Stale handover | ✓ | ✓ | ✓ | 75 | |
| Repaired handover | ✓ | ✓ | ✓ | ✓ | 100 |
| Remove an upstream register from the reference | |||||
| Allocation | ✓ | 25 | |||
| Held-out source | Tasks | Local | Rules |
|---|---|---|---|
| MBABench | 9 | 57.7 | 22.7 |
| UniPat SaaS-Bench | 5 | 23.8 | 2.2 |
| RoadmapBench | 15 | 72.9 | 34.9 |
| SaaSBench | 9 | 96.7 | 79.4 |
| SWE-Milestone | 11 | 76.4 | 51.2 |
| LHTB | 1 | 70.0 | 33.3 |
| Generator | Smaller graphs | Larger graphs | ||
|---|---|---|---|---|
| Pass (%) | Time (s) | Pass (%) | Time (s) | |
| Composition | 80.0 | 0.071 | 91.7 | 0.212 |
| Direct LLM | 38.3 | 63.69 | 21.7 | 75.93 |
| w/o guidance | 5.0 | 0.347 | 26.7 | 0.352 |
| Construction | Acceptance (%) |
|---|---|
| Ours | 40.0 |
| Source + refinement | 40.0 |
| Direct LLM | 46.7 |
| Method | Temperature | Max. output tokens |
|---|---|---|
| ReAct | 0.0 | 8,192 |
| AOrchestra | 0.2 | 8,192 |
| Meta-Team | 0.2 | 32,768 |
| Magentic-One | 0.2 | 8,192 |
| Superagent | Default | 32,768 |
| Control | Setting |
|---|---|
| AOrchestra | |
| Delegation | Sequential |
| Coordinator-attempt limit | 3,000 |
| Worker-step limit | 50 |
| Worker memory / recent records | 10 / 3 |
| Meta-Team | |
| Task | Dependencies | Shared | Budget |
|---|---|---|---|
| Exhibition recovery | 1 | 0 | 15 |
| Anthology release | 2 | 0 | 15 |
| Radio release | 2 | 0 | 15 |
| Conservation programme | 2 | 1 | 15 |
| Membership renewal | 1 | 0 | 15 |
| Housing handover | 1 | 1 | 10 |
| Setting | Value |
|---|---|
| Objective embedding model | BAAI/bge-m3 |
| Semantic / interface weights | 0.45 / 0.25 |
| Hierarchy / graph weights | 0.15 / 0.15 |
| Candidate subtask-type counts | 8–20 |
| Initializations / maximum iterations | 5 / 25 |
| Stability refits | 4 |
| Component | Extracted | Shared candidates |
|---|---|---|
| Local structures | ||
| Subtask types | 14 | 14 |
| Local patterns | 300 | 150 |
| Attachment rules | 7,900 | 194 |
| Global structures | ||
| Global structures | 100 | 24 |
| Task. Restore a Python build frontend. Its CLI must invoke the build engine and isolated environment to produce build artifacts. | ||
|---|---|---|
| Time | Actor / event | Selected trajectory excerpt |
| 4:15 | Developer / edit | Patches _builder.py , including backend validation and exception handling, then runs component tests. |
| 4:18 | Tool / result | env.py:59: NotImplementedError: Implement capability: DefaultIsolatedEnv.init |
| 4:37 | Planner developer | “I (planner) am taking src/build/main.py — do NOT edit it. You own src/build/env.py and src/build/_builder.py . Focus on env.py first” |
| 5:03 | Planner / edit | Writes the CLI, including from .env import DefaultIsolatedEnv and with DefaultIsolatedEnv(installer=installer) as env: . |
| 5:06 | Tool / local test | 3 passed in 0.03s (CLI helper and module tests). |
| Task. Prepare a radio programme and corrected transcripts. Each transcript must retain the speaker label, attribution, and embargo specified for its segment. | ||
|---|---|---|
| Time | Actor / event | Selected trajectory excerpt |
| 10:54 | Coordinator worker-4 | Delegates the rights register: “Compute the correct rights manifest per task rules”. |
| 13:22 | Worker-4 / write | Creates NCR RESULT rights . For SEG001: speaker_label: Contributor 01 ; attribution: Northbank source SEG001 ; embargo: 2027-04-19 . |
| 13:24 | Tool / receipt | The rights-note creation succeeds. |
| 37:12 | Coordinator worker-14 | Delegates transcripts with fields “ speaker_label : exact allowed label” and “ attribution : exact source attribution”; requests that both new notes be read back. |
| 40:16 | Worker-14 / write | Creates NCR RESULT transcripts . For SEG001: speaker_label: Presenter ; attribution: Riverbend Voices archive ; embargo: 2027-04-19 . |
| Task. Restore a Python build frontend with a working CLI, build engine, and isolated environment. The delivered components must support independent consumer execution. | ||
|---|---|---|
| Time | Actor / event | Selected trajectory excerpt |
| 12:13 | Orchestrator Coder | “Implement src/build/main.py in full” and “Emit each as one executable sh block for ComputerTerminal.” |
| 12:25 | Coder / proposal | “I’ll implement main.py in full”; returns a shell block writing the CLI and running tests. |
| 12:28 | Orchestrator terminal | “Execute Coder’s latest fenced sh block exactly as written” and report stdout/stderr. |
| 12:30 | Tool / test | ERROR collecting tests/test_main.py : the flaky marker is unregistered. Tests stop during collection. |
| 13:16 | Orchestrator terminal | Relays a later Coder block registering the marker and rerunning the utility and CLI tests. |
| Task. Relocate a museum exhibition after its planned venue becomes unavailable. Arrange exhibit inspections, installation, school visits, and public opening with qualified staff and non-overlapping shifts for each employee. | ||
|---|---|---|
| Time | Actor / event | Selected trajectory excerpt |
| 16:46 | Worker subagent | Delegates a 219-entry work plan: “Each row must become ONE native submitted Shift Assignment”. States: “There are no duplicate rows requiring dedupe EXCEPT if you are resuming”. |
| 16:55–17:08 | Subagent / read | Reads the shared plan. Six separate rows, including inspections of exhibits OBJ001 and OBJ005, assign employee E002 to the same June 29 shift, 08:00–17:00. |
| 17:31 | Tool / receipt | The subagent’s first assignment for E002 on June 29 is submitted successfully. |
| 17:51 | Subagent / create | Requests another assignment for E002 on June 29 using the same shift. The application returns a conflict. |
| 18:00–18:08 | Subagent / submit | Submitting further shifts for E002 on dates already covered by submitted shifts returns three conflicts. The subagent inspects the saved records. |
Multi-Agent Collaboration
Multi-agent collaboration studies how autonomous agents coordinate, share information, and combine capabilities to solve tasks together.
See all 341 papers