OpenMAS-GApp: A Comprehensive Benchmark for Domain-Adaptive Multi-Agent Application Construction
Organizations: Beijing Institute of Technology
Abstract
Constructing multi-agent system (MAS) applications across domains requires translating diverse requirements into executable systems, involving task decomposition, agent and tool selection, and execution organization. Existing studies have explored individual aspects of MAS construction, including agent generation, architecture design, workflow generation, and component selection. However, these methods are often developed and evaluated independently, using different task specifications, component resources, and execution settings. This makes it difficult to compare their construction capabilities or determine how they can be applied to different domains. To address this problem, we propose OpenMAS-GApp, a One-for-All benchmark for MAS application construction across diverse domains. It standardizes task inputs, component resources, construction interfaces, and execution environments to support MAS construction under consistent conditions. Within OpenMAS-GApp, we provide Graph Harness as a reference construction method that derives task dependencies, selects compatible agents and tools, and organizes their interactions into executable graphs. We evaluate construction methods across 19 benchmark datasets under 20 evaluation settings spanning four application types and four professional domains.
Figures & tables
| Domain | Datasets |
|---|---|
| Medicine | MedQA [ 16 ] , PubMedQA [ 17 ] |
| Finance | FinQA [ 4 ] , FinanceBench [ 15 ] |
| Science | SciQ [ 29 ] , SciBench [ 26 ] |
| Law | CaseHOLD [ 35 ] |
| Application Type | Datasets and Settings |
|---|---|
| General Reasoning | MMLU [ 11 ] , MMLU-Pro [ 28 ] , BBH-Boolean, BBH-Mixed [ 24 ] , ARC [ 5 ] , StrategyQA [ 10 ] |
| Mathematical Reasoning | GSM8K [ 6 ] , MATH-500 |
| Reading and QA | HotpotQA [ 31 ] , MuSiQue [ 25 ] , DROP [ 9 ] |
| Code Generation | HumanEval [ 2 ] , MBPP [ 1 ] |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Domain or application type | Data sources |
|---|---|
| General reasoning | MMLU, MMLU-Pro, BBH-Boolean, BBH-Mixed, ARC, StrategyQA |
| Mathematical reasoning | GSM8K, MATH-500 |
| Reading and QA | HotpotQA, MuSiQue, DROP |
| Code generation | HumanEval, MBPP |
| Medicine | MedQA, PubMedQA |
| Finance | FinQA, FinanceBench |
| Data sources | Evaluation partition |
|---|---|
| MMLU [ 11 ] and MMLU-Pro [ 28 ] | Official test split of each source |
| ARC-Challenge [ 5 ] | Official Challenge test split of 1,172 problems |
| BBH-Boolean and BBH-Mixed [ 24 ] | Boolean-only and mixed-configuration settings from the same benchmark. Candidate pools contain 100 and 240 problems, respectively. The main comparisons and ablations use the first 10 Boolean and first 30 Mixed problems, respectively. |
| StrategyQA [ 10 ] | Official test partition through the source evaluation |
| GSM8K [ 6 ] | Released test split |
| MATH-500 | Released 500-problem evaluation subset of MATH [ 12 ] |
| Item | Content |
|---|---|
| Problem identity | Source dataset and problem identifier |
| Requirement | Objective, supplied inputs, output requirements, and application constraints |
| Execution inputs | Materials and resources available to the application |
| Component registry | Available agents and tools with capability and interface descriptions |
| Domain resources | Supplied domain concepts, constraints, and heuristics |
| Execution environment | Common runtime, tool permissions, and resource budgets |
| Operation | Cost | Conditions |
|---|---|---|
| Requirement coverage checks | Given coverage indicators | |
| Task dependency checks | Pairwise task and field comparisons | |
| Topological ordering | Given the selected task graph | |
| Component capability scoring | Cached cosine and support scores | |
| Candidate interface checks | Given candidate connections | |
| Path scoring | Cached component and edge scores |
| Record | Required configuration | Current entry |
|---|---|---|
| Language-model backbone | Provider, model identifier, revision, and serving endpoint | DeepSeek-V4-flash |
| Stage-specific models | Task parser, semantic evaluator, synthesizer, and planner | Shared backbone |
| Decoding | Temperature, sampling parameters, output limits, and seeds | Configuration D02 |
| Embedding and matching | Encoder revision, normalization, predicates, and thresholds | 3072-dimensional encoder |
| Runtime environment | Operating system, hardware, memory, and software versions | 16 vCPU, 64 GiB |
| Serving configuration | Local or remote inference, concurrency, and batching | Remote API, four calls |
| Comparison | Displayed datasets or applications |
|---|---|
| Construction paradigms and orchestration | MMLU-Pro, FinanceBench, MedQA, PubMedQA, MuSiQue |
| Construction ablation | GSM8K, MATH-500, MMLU-Pro, ARC, BBH-Boolean, BBH-Mixed, HotpotQA, DROP, MuSiQue, FinanceBench, SciBench |
| Component scalability | 100 problems each from GSM8K, MATH-500, and PubMedQA, with brief and detailed interface descriptions in Configuration D07. |
| Efficiency | FinanceBench, HotpotQA, PubMedQA, CaseHOLD |
| Robustness | BBH-Mixed, GSM8K, MuSiQue, SciBench |
| Domain case studies | Medical and legal applications. Case inputs and bindings are specified in Appendix E.4 . |
| Task group | Datasets | Full | Task structure | Component selection | Graph construction | I/O constraints |
|---|---|---|---|---|---|---|
| Mathematical reasoning | 2 | 84.25 | 4.95 | 6.81 | 4.90 | 5.66 |
| General/structured reasoning | 4 | 78.50 | 4.39 | 3.30 | 2.43 | 3.17 |
| Multi-hop QA | 3 | 66.27 | 3.60 | 3.49 | 3.35 | 2.36 |
| Domain-specific pair | 2 | 74.34 | 3.76 | 4.57 | 5.70 | 3.26 |
| Dataset | Task structure | Component selection | Graph construction | I/O constraints |
|---|---|---|---|---|
| GSM8K | 5.01 | 3.73 | 3.32 | 4.80 |
| MATH-500 | 4.88 | 9.88 | 6.48 | 6.52 |
| MMLU-Pro | 3.32 | 2.39 | 1.02 | 1.88 |
| ARC | 3.11 | 4.12 | 2.02 | 0.81 |
| BBH-Boolean | 6.67 | 3.34 | 3.34 | 6.67 |
| BBH-Mixed | 4.44 | 3.33 | 3.33 | 3.33 |
| Method | FinanceBench | HotpotQA | PubMedQA | CaseHOLD | ||||
|---|---|---|---|---|---|---|---|---|
| Score | Tokens | Score | Tokens | Score | Tokens | Score | Tokens | |
| Direct | 66.67 | 2,050.00 | 35.00 | 680.00 | 77.00 | 810.00 | 58.00 | 740.00 |
| Plan-Execute | 50.00 | 8,750.00 | 62.00 | 2,840.00 | 76.00 | 3,020.00 | 71.00 | 2,580.00 |
| Sequence | 69.07 | 5,370.00 | 52.31 | 2,190.00 | 76.60 | 2,250.00 | 77.70 | 1,910.00 |
| Tree | 37.47 | 5,400.00 | 52.56 | 3,870.00 | 84.40 | 3,810.00 | 55.61 | 3,640.00 |
| Flat | 42.13 | 6,050.00 | 45.61 | 1,050.00 | 73.40 | 1,570.00 | 61.42 | 1,170.00 |
| Evaluation Setting | Reference metric | Direct | Prompt Plan | Component | Plan | Workflow | Graph Harness |
|---|---|---|---|---|---|---|---|
| General reasoning | |||||||
| MMLU | Accuracy | ||||||
| MMLU-Pro | Accuracy | ||||||
| BBH-Boolean | Exact match | ||||||
| BBH-Mixed | Exact match | ||||||
| ARC | Accuracy | ||||||
| Evaluation Setting | Metric | AFlow | ADAS | GPTSwarm | AgentSquare | Graph Harness |
|---|---|---|---|---|---|---|
| General reasoning | ||||||
| MMLU | Accuracy | |||||
| MMLU-Pro | Accuracy | |||||
| BBH-Boolean | Exact match | |||||
| BBH-Mixed | Exact match | |||||
| ARC | Accuracy | |||||
| Dataset | Full | w/o Task Structure | w/o Component Selection | w/o Graph Construction | w/o I/O Constraints |
|---|---|---|---|---|---|
| HumanEval | |||||
| MBPP | |||||
| CaseHOLD |
| Dataset | Flat Selection | Sequence | Tree | Workflow | Agent Graph | Graph Harness |
|---|---|---|---|---|---|---|
| MMLU-Pro | ||||||
| GSM8K | ||||||
| HotpotQA | ||||||
| HumanEval | ||||||
| MedQA | ||||||
| FinanceBench |
| Dataset | Direct | Prompt Plan | Component | Plan | Workflow | Graph Harness |
|---|---|---|---|---|---|---|
| MMLU | ||||||
| MMLU-Pro | ||||||
| BBH-Boolean | ||||||
| BBH-Mixed | ||||||
| ARC | ||||||
| StrategyQA |
| Dataset | Direct | Prompt Plan | Component | Plan | Workflow | Graph Harness |
|---|---|---|---|---|---|---|
| MMLU | ||||||
| MMLU-Pro | ||||||
| BBH-Boolean | ||||||
| BBH-Mixed | ||||||
| ARC | ||||||
| StrategyQA |
| Dataset | Direct | Prompt Plan | Component | Plan | Workflow | Graph Harness |
|---|---|---|---|---|---|---|
| MMLU | ||||||
| MMLU-Pro | ||||||
| BBH-Boolean | ||||||
| BBH-Mixed | ||||||
| ARC | ||||||
| StrategyQA |
| Domain | Output summary | Review | Tokens | Time (s) |
|---|---|---|---|---|
| Medicine | Reviewed medical answer | Approved | 2,784 | 8.70 |
| Law | Conditional compensation report | Approved | 3,468 | 9.10 |
Multi-Agent Systems
Multi-Agent Systems studies how autonomous agents coordinate, compete, communicate, and learn in shared or interdependent tasks.
See all 727 papers