OpenMAS-GCom. A Diagnostic Benchmark for Graph-enhanced Multi-Agent Systems

Kairui Yang; Xunkai Li; Kaixiang Zhang; Minghao An; Zekai Chen; Yuxuan Ba; Rong-Hua Li

cs.LGSep 18, 2026

OpenMAS-GCom. A Diagnostic Benchmark for Graph-enhanced Multi-Agent Systems

Authors: Kairui Yang, Xunkai Li, Kaixiang Zhang, Minghao An, Zekai Chen, Yuxuan Ba, Rong-Hua Li

Abstract

Graph-enhanced multi-agent systems (G-MAS) coordinate large language model agents through communication graphs and role assignments, which determine how agents exchange information and divide responsibilities. However, final-score comparisons across systems combine differences in models, communication patterns, roles, and computation costs, making performance differences difficult to attribute to specific communication structures, role assignments, and information flows. To address this evaluation attribution problem, we introduce OpenMAS-GCom, a benchmark for diagnosing how these components affect G-MAS performance through controlled interventions. We represent systems through collaboration units, communication links, shared intermediate information, and execution rules. OpenMAS-GCom compares original systems with versions modified by changing one component while keeping tasks, models, prompts, and budget limits fixed. We rewire communication edges, remove specialist or critic agents, replace intermediate messages with incorrect content, and disable workers during execution. The benchmark evaluates 17 single-agent, ordinary multi-agent, and graph-enhanced configurations on 29 datasets across six domains. We add 400 G-MAS-Complex tasks requiring agents to combine information from multiple documents, resolve conflicting records, and return specified values with source identifiers. Experiments show larger mean losses after specialist removal than after critic removal, different performance degradation under incorrect messages and worker failures despite similar original scores, and different configurations achieving the highest accuracy and accuracy per token on G-MAS-Complex.

Figures & tables

Figure 1: OpenMAS-GCom model framework. The benchmark selects datasets, configures G-MAS organizations, executes graph guided collaboration, and applies four graph intervention protocols.

	Mathematical Reasoning		Knowledge & Commonsense		Code Generation		Biomedical QA		Financial Reasoning		Tabular Reasoning		Complex Collaboration
Method	GSM8K	AQuA	MMLU-Pro	StrategyQA	HumanEval	LiveCode	MedQA	MedMCQA	ConvFinQA	FinQA	TabFact	WTQ	G-MAS-Complex
DeepSeek	$93.03$	$89.89$	$73.07$	$84.60$	$83.54$	$21.90$	$88.58$	$81.27$	$60.13$	$38.10$	$91.47$	$76.73$	$26.75$
Qwen	$59.44$	$68.90$	$60.93$	$79.40$	$90.24$	$26.86$	$78.95$	$73.53$	$61.81$	$28.51$	$81.40$	$65.27$	$13.25$
AutoGen	$92.92$	$90.16$	$71.80$	$84.13$	$81.71$	$18.86$	$88.74$	$79.67$	$60.67$	$36.70$	$91.67$	$75.93$	$23.50$
MADebate	$93.20$	$89.89$	$71.87$	$84.20$	$79.88$	$20.38$	$89.45$	$79.67$	$61.01$	$36.97$	$91.47$	$76.67$	$14.00$
DyLAN	$93.33$	$89.89$	$73.87$	$86.67$	$85.98$	$22.86$	$88.45$	$79.93$	$61.10$	$38.88$	$92.53$	$77.40$	$4.50$

Table 1: Central-score comparison (%). Bold and underlined entries mark the highest and second-highest distinct displayed scores, including ties. Appendix A.8 defines the statistical sources.

Figure 2: Group-wise maximum reported scores over evaluated Ordinary MAS and Graph-enhanced MAS configurations. Each bar selects the highest score within its group and task subset, allowing different configurations to supply different bars.

Figure 3: Domain summaries with independent panel scales. Accuracy denotes the reported aggregate score. Appendix A.1 lists dataset-specific metrics.

Figure 4: Responses to topology rewiring, role removal, incorrect messages, and worker failures. Panels (a) and (b) show five datasets, while panels (c) and (d) compare the displayed methods. Section 3.3 defines the corresponding operations.

Figure 5: G-MAS-Complex efficiency. Bars show strict exact match per 1K tokens for all baselines.

Method	Strict EM (%)	Tokens/task (K)	Strict EM/K Tokens	Latency (s)	Nodes	Edges
GoAgent	50.25	26.47	1.90	45.46	17.25	67.15
VeriMap	29.50	4.73	6.24	7.09	4.00	4.00
DeepSeek	26.75	5.94	4.50	1.51	1.00	0.00
AutoGen	23.50	3.39	6.93	6.42	3.00	2.00
BigMAS	19.25	11.65	1.65	17.11	14.30	30.90
GPTSwarm	15.50	3.84	4.04	6.48	3.95	2.95

Table 2: Observed operating points on G-MAS-Complex. Strict EM is in percent, tokens per task in K, and latency in seconds. Nodes and edges are execution averages. Strict EM/K Tokens describes score per 1K tokens at the reported setting.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	#Datasets	Datasets	Primary evaluation target
Mathematics	7	GSM8K, SVAMP, MultiArith, AQuA, ASDiv-A, Game-of-24, MATH	Numeric answer, multiple-choice answer, valid expression, or competition-style final answer
Knowledge and commonsense	7	MMLU, MMLU-Redux, MMLU-Pro, StrategyQA, CommonsenseQA, ARC-Easy, ARC-Challenge	Multiple-choice or yes/no answer after robust answer extraction
Code	4	HumanEval, HumanEval++, LiveCodeBench-v6, MultiAgentBench-Coding	Unit-test pass rate, executable correctness, or code-task quality score
Biomedical	4	PubMedQA, MedQA, MedMCQA, MMLU-Med	Biomedical yes/no/maybe answer or medical multiple-choice answer
Finance	3	TAT-QA, ConvFinQA, FinQA	Normalized numerical or textual financial answer
Tabular	4	TabFact, WikiSQL, WikiTableQuestions, SQA	Table entailment, table-query answer, or sequential table answer

Appendix

Table 3: Broad-suite dataset summary. Domain-level statistics summarize the broad suite. Per-dataset sample counts are shown in Table 11 .

Tier	Main stress factor	Document and conflict pattern	Required behavior
Medium	Multi-document lookup	Several documents with local dependencies and light distractors	Retrieve relevant records and return a structured answer
Hard	Cross-document dependency	Chained records with competing revisions or aliases	Resolve dependencies before producing fields and references
Ultra	Conflict and provenance	Multiple plausible records, authority rules, and source checkpoints	Select authoritative records and justify structured fields
Extreme	End-to-end audit	Dense dependencies, distractors, revision conflicts, and checksum constraints	Combine retrieval, conflict resolution, ranking, and verification

Appendix

Table 4: G-MAS-Complex tier construction. Each tier contains 100 frozen tasks.

Family	Methods	Adapter representation
Single-agent	DeepSeek, Qwen ( DeepSeek-AI, 2025 , Qwen Team, 2025 )	One runnable unit, empty edge set, direct answer parser
Ordinary MAS	AutoGen, MADebate, DyLAN ( Wu et al., 2023 , Du et al., 2023 , Liu et al., 2024 )	Fixed role set with predefined debate, refinement, or dynamic selection policy
Graph construction and routing	GPTSwarm, GDesigner, GraphSearch, GraphR1, MASRouter ( Zhuge et al., 2024 , Zhang et al., 2025 , Liu et al., 2026b , Luo et al., 2026 , Yue et al., 2025 )	Explicit graph or search/routing module mapped to typed directed relations
Sparse or role-specialized G-MAS	SparseCT, R-GFM, BigMAS, ARGDes, EIB ( Li et al., 2024 , Liu et al., 2026a , Hao et al., 2026 , Li et al., 2026 , Shen et al., 2025 )	Specialist units and sparse communication structures mapped to runnable nodes and edges
Verification and recovery-oriented G-MAS	GoAgent, VeriMap ( Chen et al., 2026 , Xu et al., 2025 )	Verification, fallback, or rerouting policies exposed as execution-policy components

Appendix

Table 5: Baseline families and organization adapters.

Intervention	Samples	Backend	Prompts	Parser	Budget	Roles	Edges	Degree seq.
Structure rewiring	Yes	Yes	Yes	Yes	Yes	Yes	Changed	Yes
Node removal	Yes	Yes	Yes	Yes	Yes	Changed	Changed	No
Information injection	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Execution failure	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes

Appendix

Table 6: Controls preserved by diagnostic interventions. Yes means the factor is kept fixed relative to the reference run.

Perspective	Main comparison	Reported results
Necessity	Single-agent vs. ordinary MAS vs. graph-enhanced MAS	Main 13-dataset comparison and full 29-dataset table
Effectiveness	Method behavior across domains and complex-task tiers	Domain averages, G-MAS-Complex tiers, and complete method rankings
Robustness	Structure, node, information, and execution interventions	Performance changes under controlled organizational perturbations
Efficiency	Task quality relative to execution cost	Strict EM, tokens, latency, executed nodes, executed edges, and Strict EM/K tokens

Appendix

Table 7: Evaluation strategy summary.

Category	Recorded fields
Task state	Dataset id, sample id, split or frozen-list identifier, gold answer, parser version
Organization state	Method id, organization hash, node list, edge list, role configuration, routing policy
Backend state	Backend configuration id, prompt-template hash, runtime limits, random seed when used
Prediction state	Raw response, parsed prediction, invalid-parse flag, metric result, final answer
Cost state	Token usage, model calls, executed nodes, executed edges, retries, latency, termination reason
Intervention state	Reference organization hash, intervention type, changed nodes/edges, injected records, failed workers, recovery action

Appendix

Table 8: Recorded environment and run metadata. The record links task, organization, backend, prediction, cost, and intervention fields.

Component	Setting
Broad-suite sampling	Full evaluation split where used, otherwise fixed ordered subset, with three matched 500-instance samples for repeated-sampling datasets
G-MAS-Complex sampling	400 frozen tasks, 100 tasks each for Medium, Hard, Ultra, and Extreme
Structure rewiring	Original, 10%, 30%, 50%, and 70% degree-preserving rewiring
Node intervention	Base, No critic, No specialist, Expert only
Information intervention	Incorrect-information concentration $c\in\{0,.2,.4,.6,.8\}$
Execution intervention	Failed workers $k\in\{0,1,2,3,4\}$

Appendix

Table 9: Benchmark-level hyperparameter and sweep settings.

Code	Cells	Recorded quantity	Repetition information
$\mathrm{s}$	221	Mean and sample standard deviation across sampled task sets. Each set contains 500 instances.	Three sample sets
$\mathrm{r}$	152	Fixed reference score with an attached run-seed standard deviation.	Run count unrecorded
$\mathrm{e}$	34	Reference score with a recorded empirical estimate combining dataset and method components at weights 0.70 and 0.30.	Component counts unrecorded
$\mathrm{b}$	100	Reference score with a binomial standard-error estimate computed from that score and the displayed evaluation count $N$ .	Analytic estimate
$\mathrm{m}$	3	Backup run-seed mean and standard deviation for cells with an empty reference metric.	Run count unrecorded

Appendix

Table 10: Statistical sources and cell counts for the 510 entries in the detailed result tables.

Method	GSM8K Acc. $N=1319$	SVAMP Acc. $N=1000$	MultiArith Acc. $N=600$	AQuA Acc. $N=254$	ASDiv-A Acc. $N=500$	Game-of-24 Acc. $N=1362$	MATH Acc. $N=500$
DeepSeek	$\underline{93.71}$ $(0.33)^{\mathrm{r}}$	$94.00$ $(0.10)^{\mathrm{r}}$	$98.50$ $(0.25)^{\mathrm{r}}$	$89.76$ $(0.45)^{\mathrm{r}}$	$88.53$ $(1.10)^{\mathrm{s}}$	$58.88$ $(1.33)^{\mathrm{b}}$	$55.73$ $(3.64)^{\mathrm{s}}$
Qwen	$59.44$ $(0.56)^{\mathrm{e}}$	$90.90$ $(0.91)^{\mathrm{b}}$	$97.17$ $(0.68)^{\mathrm{b}}$	$68.90$ $(0.86)^{\mathrm{e}}$	$86.87$ $(0.42)^{\mathrm{s}}$	$21.00$ $(1.10)^{\mathrm{b}}$	$45.20$ $(2.27)^{\mathrm{s}}$
AutoGen	$92.87$ $(0.34)^{\mathrm{r}}$	$93.70$ $(0.77)^{\mathrm{b}}$	$98.50$ $(0.19)^{\mathrm{r}}$	$88.98$ $(1.72)^{\mathrm{r}}$	$88.53$ $(0.99)^{\mathrm{s}}$	$\underline{67.91}$ $(1.26)^{\mathrm{b}}$	$56.13$ $(2.66)^{\mathrm{s}}$
MADebate	$92.87$ $(0.24)^{\mathrm{r}}$	$94.20$ $(0.74)^{\mathrm{b}}$	$\mathbf{98.83}$ $(0.00)^{\mathrm{r}}$	$89.37$ $(0.23)^{\mathrm{r}}$	$88.53$ $(0.99)^{\mathrm{s}}$	$62.70$ $(1.31)^{\mathrm{b}}$	$55.27$ $(3.42)^{\mathrm{s}}$
DyLAN	$93.33$ $(0.41)^{\mathrm{e}}$	$94.00$ $(0.25)^{\mathrm{r}}$	$\underline{98.67}$ $(0.29)^{\mathrm{r}}$	$89.76$ $(0.99)^{\mathrm{r}}$	$\mathbf{88.80}$ $(1.40)^{\mathrm{s}}$	$62.41$ $(1.31)^{\mathrm{b}}$	$56.07$ $(3.24)^{\mathrm{s}}$
GPTSwarm	$93.56$ $(0.43)^{\mathrm{r}}$	$94.10$ $(0.46)^{\mathrm{r}}$	$98.50$ $(0.25)^{\mathrm{r}}$	$\mathbf{91.34}$ $(0.68)^{\mathrm{e}}$	$88.67$ $(0.81)^{\mathrm{s}}$	$65.71$ $(0.85)^{\mathrm{r}}$	$57.00$ $(2.84)^{\mathrm{s}}$

Appendix

Table 11: Mathematical reasoning results. Each entry places the central score above its uncertainty term and source code. Source codes are defined above.

Method	MMLU Acc. $N=500$	MMLU- Redux Acc. $N=500$	MMLU-Pro Acc. $N=500$	StrategyQA Acc. $N=500$	Commonsense QA Acc. $N=500$	ARC-Easy Acc. $N=500$	ARC- Challenge Acc. $N=1418$
DeepSeek	$85.40$ $(0.92)^{\mathrm{s}}$	$83.73$ $(0.50)^{\mathrm{s}}$	$73.07$ $(0.76)^{\mathrm{s}}$	$84.60$ $(0.80)^{\mathrm{s}}$	$85.40$ $(0.53)^{\mathrm{s}}$	$\underline{94.27}$ $(0.61)^{\mathrm{s}}$	$96.26$ $(0.50)^{\mathrm{b}}$
Qwen	$82.33$ $(2.76)^{\mathrm{s}}$	$79.80$ $(0.92)^{\mathrm{s}}$	$60.93$ $(0.90)^{\mathrm{s}}$	$79.40$ $(1.93)^{\mathrm{s}}$	$85.67$ $(1.33)^{\mathrm{s}}$	$94.07$ $(0.76)^{\mathrm{s}}$	$96.05$ $(0.52)^{\mathrm{b}}$
AutoGen	$85.80$ $(0.35)^{\mathrm{s}}$	$84.00$ $(1.73)^{\mathrm{s}}$	$71.80$ $(1.20)^{\mathrm{s}}$	$84.13$ $(0.70)^{\mathrm{s}}$	$85.67$ $(1.30)^{\mathrm{s}}$	$\underline{94.27}$ $(0.76)^{\mathrm{s}}$	$96.61$ $(0.15)^{\mathrm{r}}$
MADebate	$85.93$ $(0.12)^{\mathrm{s}}$	$83.60$ $(0.53)^{\mathrm{s}}$	$71.87$ $(1.14)^{\mathrm{s}}$	$84.07$ $(1.96)^{\mathrm{s}}$	$85.20$ $(1.56)^{\mathrm{s}}$	$94.07$ $(0.81)^{\mathrm{s}}$	$96.54$ $(0.49)^{\mathrm{b}}$
DyLAN	$86.00$ $(1.39)^{\mathrm{s}}$	$83.93$ $(1.50)^{\mathrm{s}}$	$\mathbf{73.87}$ $(1.21)^{\mathrm{s}}$	$\mathbf{86.67}$ $(0.70)^{\mathrm{s}}$	$85.73$ $(0.61)^{\mathrm{s}}$	$94.07$ $(0.76)^{\mathrm{s}}$	$\underline{96.76}$ $(0.19)^{\mathrm{r}}$
GPTSwarm	$85.60$ $(0.53)^{\mathrm{s}}$	$\underline{84.40}$ $(0.35)^{\mathrm{s}}$	$70.73$ $(0.61)^{\mathrm{s}}$	$85.67$ $(2.84)^{\mathrm{s}}$	$85.53$ $(0.64)^{\mathrm{s}}$	$\underline{94.27}$ $(0.76)^{\mathrm{s}}$	$\mathbf{96.83}$ $(0.19)^{\mathrm{r}}$

Appendix

Table 12: Knowledge and commonsense results. Each entry places the central score above its uncertainty term and source code. Source codes are defined above.

Method	HumanEval Acc. $N=164$	HumanEval++ Acc. $N=164$	LiveCodeBench v6 Pass@ $1$ $N=175$	MultiAgentBench Coding CodeQ $N=102$
DeepSeek	$84.76\,(0.00)^{\mathrm{r}}$	$\mathbf{80.49}\,(2.14)^{\mathrm{r}}$	$20.57\,(2.01)^{\mathrm{r}}$	$41.55\,(1.06)^{\mathrm{m}}$
Qwen	$\mathbf{90.24}\,(1.35)^{\mathrm{e}}$	$76.83\,(0.70)^{\mathrm{r}}$	$\mathbf{26.86}\,(1.39)^{\mathrm{e}}$	$\mathbf{60.54}\,(4.84)^{\mathrm{b}}$
AutoGen	$81.71\,(1.21)^{\mathrm{e}}$	$73.78\,(3.43)^{\mathrm{b}}$	$18.86\,(1.25)^{\mathrm{e}}$	$42.01\,(4.89)^{\mathrm{b}}$
MADebate	$83.54\,(2.11)^{\mathrm{r}}$	$77.44\,(2.46)^{\mathrm{r}}$	$21.71\,(2.88)^{\mathrm{r}}$	$40.69\,(4.86)^{\mathrm{b}}$
DyLAN	$85.98\,(1.19)^{\mathrm{e}}$	$78.66\,(3.20)^{\mathrm{b}}$	$22.86\,(1.23)^{\mathrm{e}}$	$37.06\,(4.78)^{\mathrm{b}}$
GPTSwarm	$84.76\,(0.93)^{\mathrm{r}}$	$\mathbf{80.49}\,(5.21)^{\mathrm{r}}$	$19.43\,(0.00)^{\mathrm{r}}$	$41.91\,(4.89)^{\mathrm{b}}$

Appendix

Table 13: Code generation results. Entries give score (uncertainty) source with the defined source codes. The recorded CodeQ metric is retained for MultiAgentBench-Coding.

Method	PubMedQA Acc. $N=3000$	MedQA Acc. $N=1273$	MedMCQA Acc. $N=500$	MMLU-Med Acc. $N=1561$
DeepSeek	$81.33\,(0.71)^{\mathrm{b}}$	$88.45\,(0.68)^{\mathrm{r}}$	$79.67\,(0.99)^{\mathrm{s}}$	$89.75\,(0.77)^{\mathrm{b}}$
Qwen	$80.70\,(1.25)^{\mathrm{b}}$	$78.95\,(0.72)^{\mathrm{e}}$	$73.53\,(0.61)^{\mathrm{s}}$	$87.51\,(0.84)^{\mathrm{b}}$
AutoGen	$81.43\,(0.71)^{\mathrm{b}}$	$89.08\,(0.55)^{\mathrm{r}}$	$79.67\,(1.70)^{\mathrm{s}}$	$89.94\,(0.15)^{\mathrm{r}}$
MADebate	$81.87\,(0.70)^{\mathrm{b}}$	$89.08\,(0.53)^{\mathrm{r}}$	$78.53\,(1.14)^{\mathrm{s}}$	$\underline{90.26}\,(0.75)^{\mathrm{b}}$
DyLAN	$\mathbf{82.43}\,(0.69)^{\mathrm{b}}$	$88.77\,(0.08)^{\mathrm{r}}$	$79.93\,(1.10)^{\mathrm{s}}$	$89.56\,(0.38)^{\mathrm{r}}$
GPTSwarm	$81.67\,(0.71)^{\mathrm{b}}$	$89.63\,(0.54)^{\mathrm{e}}$	$79.00\,(0.87)^{\mathrm{s}}$	$89.94\,(0.24)^{\mathrm{r}}$

Appendix

Table 14: Biomedical reasoning results. Entries give score (uncertainty) source with the defined source codes. PubMedQA retains the recorded source sample count.

Method	TAT-QA Acc. $N=1663$	ConvFinQA Acc. $N=1490$	FinQA Acc. $N=1147$
DeepSeek	$70.48\,(0.63)^{\mathrm{r}}$	$60.13\,(0.43)^{\mathrm{e}}$	$38.10\,(0.51)^{\mathrm{e}}$
Qwen	$69.51\,(1.13)^{\mathrm{b}}$	$\mathbf{61.81}\,(0.62)^{\mathrm{e}}$	$28.51\,(0.70)^{\mathrm{e}}$
AutoGen	$70.29\,(1.12)^{\mathrm{b}}$	$60.74\,(0.66)^{\mathrm{r}}$	$36.70\,(0.57)^{\mathrm{e}}$
MADebate	$70.66\,(1.12)^{\mathrm{b}}$	$61.01\,(0.52)^{\mathrm{e}}$	$36.97\,(0.61)^{\mathrm{e}}$
DyLAN	$70.48\,(0.23)^{\mathrm{r}}$	$60.60\,(0.55)^{\mathrm{r}}$	$\underline{38.88}\,(0.55)^{\mathrm{e}}$
GPTSwarm	$70.41\,(0.33)^{\mathrm{r}}$	$60.67\,(0.24)^{\mathrm{r}}$	$37.66\,(1.13)^{\mathrm{r}}$

Appendix

Table 15: Financial reasoning results with uncertainty terms and source codes.

Method	TabFact Acc. $N=500$	WikiSQL Acc. $N=500$	WikiTable Questions Acc. $N=500$	SQA Acc. $N=500$
DeepSeek	$92.00\,(1.11)^{\mathrm{s}}$	$74.07\,(0.61)^{\mathrm{s}}$	$76.73\,(2.16)^{\mathrm{s}}$	$55.20\,(1.44)^{\mathrm{s}}$
Qwen	$81.40\,(1.56)^{\mathrm{s}}$	$\underline{74.93}\,(1.27)^{\mathrm{s}}$	$65.27\,(1.50)^{\mathrm{s}}$	$46.53\,(2.14)^{\mathrm{s}}$
AutoGen	$91.80\,(1.22)^{\mathrm{s}}$	$74.20\,(1.78)^{\mathrm{s}}$	$75.93\,(1.03)^{\mathrm{s}}$	$54.67\,(1.97)^{\mathrm{s}}$
MADebate	$91.47\,(0.42)^{\mathrm{s}}$	$74.27\,(1.10)^{\mathrm{s}}$	$76.67\,(1.33)^{\mathrm{s}}$	$54.60\,(1.93)^{\mathrm{s}}$
DyLAN	$\mathbf{92.53}\,(1.53)^{\mathrm{s}}$	$73.87\,(2.53)^{\mathrm{s}}$	$\underline{77.40}\,(1.44)^{\mathrm{s}}$	$55.00\,(1.74)^{\mathrm{s}}$
GPTSwarm	$92.20\,(1.40)^{\mathrm{s}}$	$74.80\,(1.97)^{\mathrm{s}}$	$76.60\,(1.93)^{\mathrm{s}}$	$56.13\,(3.13)^{\mathrm{s}}$

Appendix

Table 16: Tabular reasoning results with uncertainty terms and source codes.

Method	G-MAS-Complex Strict EM $N=400$
DeepSeek	$26.75\,(2.21)^{\mathrm{b}}$
Qwen	$13.25\,(1.70)^{\mathrm{b}}$
AutoGen	$23.50\,(2.12)^{\mathrm{b}}$
MADebate	$14.00\,(1.73)^{\mathrm{b}}$
DyLAN	$4.50\,(1.04)^{\mathrm{b}}$
GPTSwarm	$15.50\,(1.81)^{\mathrm{b}}$

Appendix

Table 17: G-MAS-Complex results. Entries give strict exact match in percent with a binomial standard-error estimate in parentheses over 400 tasks.

Method	ASDiv-A	MATH
DeepSeek	$88.53\!\pm\!1.10$	$55.73\!\pm\!3.64$
Qwen	$86.87\!\pm\!0.42$	$45.20\!\pm\!2.27$
AutoGen	$88.53\!\pm\!0.99$	$56.13\!\pm\!2.66$
MADebate	$88.53\!\pm\!0.99$	$55.27\!\pm\!3.42$
DyLAN	$\mathbf{88.80}\!\pm\!1.40$	$56.07\!\pm\!3.24$
GPTSwarm	$88.67\!\pm\!0.81$	$57.00\!\pm\!2.84$

Appendix

Table 18: Mathematical reasoning under sample-set variation. Entries report mean

\pm

sample standard deviation in percent across three 500-instance sample sets.

Method	MMLU	MMLU- Redux	MMLU-Pro	StrategyQA	Commonsense QA	ARC-Easy
DeepSeek	$85.40\!\pm\!0.92$	$83.73\!\pm\!0.50$	$73.07\!\pm\!0.76$	$84.60\!\pm\!0.80$	$85.40\!\pm\!0.53$	$\underline{94.27}\!\pm\!0.61$
Qwen	$82.33\!\pm\!2.76$	$79.80\!\pm\!0.92$	$60.93\!\pm\!0.90$	$79.40\!\pm\!1.93$	$85.67\!\pm\!1.33$	$94.07\!\pm\!0.76$
AutoGen	$85.80\!\pm\!0.35$	$84.00\!\pm\!1.73$	$71.80\!\pm\!1.20$	$84.13\!\pm\!0.70$	$85.67\!\pm\!1.30$	$\underline{94.27}\!\pm\!0.76$
MADebate	$85.93\!\pm\!0.12$	$83.60\!\pm\!0.53$	$71.87\!\pm\!1.14$	$84.07\!\pm\!1.96$	$85.20\!\pm\!1.56$	$94.07\!\pm\!0.81$
DyLAN	$86.00\!\pm\!1.39$	$83.93\!\pm\!1.50$	$\mathbf{73.87}\!\pm\!1.21$	$\mathbf{86.67}\!\pm\!0.70$	$85.73\!\pm\!0.61$	$94.07\!\pm\!0.76$
GPTSwarm	$85.60\!\pm\!0.53$	$\underline{84.40}\!\pm\!0.35$	$70.73\!\pm\!0.61$	$85.67\!\pm\!2.84$	$85.53\!\pm\!0.64$	$\underline{94.27}\!\pm\!0.76$

Appendix

Table 19: Knowledge and commonsense under sample-set variation. Entries report mean

\pm

sample standard deviation in percent across three 500-instance sample sets.

Method	MedMCQA
DeepSeek	$79.67\!\pm\!0.99$
Qwen	$73.53\!\pm\!0.61$
AutoGen	$79.67\!\pm\!1.70$
MADebate	$78.53\!\pm\!1.14$
DyLAN	$79.93\!\pm\!1.10$
GPTSwarm	$79.00\!\pm\!0.87$

Appendix

Table 20: Biomedical reasoning under sample-set variation. Entries report mean

\pm

sample standard deviation in percent across three 500-instance sample sets.

Method	TabFact	WikiSQL	WikiTable Questions	SQA
DeepSeek	$92.00\!\pm\!1.11$	$74.07\!\pm\!0.61$	$76.73\!\pm\!2.16$	$55.20\!\pm\!1.44$
Qwen	$81.40\!\pm\!1.56$	$\underline{74.93}\!\pm\!1.27$	$65.27\!\pm\!1.50$	$46.53\!\pm\!2.14$
AutoGen	$91.80\!\pm\!1.22$	$74.20\!\pm\!1.78$	$75.93\!\pm\!1.03$	$54.67\!\pm\!1.97$
MADebate	$91.47\!\pm\!0.42$	$74.27\!\pm\!1.10$	$76.67\!\pm\!1.33$	$54.60\!\pm\!1.93$
DyLAN	$\mathbf{92.53}\!\pm\!1.53$	$73.87\!\pm\!2.53$	$\underline{77.40}\!\pm\!1.44$	$55.00\!\pm\!1.74$
GPTSwarm	$92.20\!\pm\!1.40$	$74.80\!\pm\!1.97$	$76.60\!\pm\!1.93$	$56.13\!\pm\!3.13$

Appendix

Table 21: Tabular and structured reasoning under sample-set variation. Entries report mean

\pm

sample standard deviation in percent across three 500-instance sample sets.

Multi-Agent LLM Systems

Multi-agent large language model systems coordinate multiple AI agents to solve tasks through communication, specialization, shared tools, and memory.

See all 1,001 papers

ForestBench: A Unified Graph Framework for Evaluating Multi-Agent Collaboration
Aug 9, 2026Guo Chen, Ziwen Li, Reed Li +4Multi-Agent LLM Systems AI Agent Evaluation
Southwest University Chongqing, China · Tencent Beijing, China · Institute of Computing Technology, Chinese Academy of Sciences Beijing, China
GraphMAS: A Systematic Benchmark of Multi-Agent Coordination for Graph Learning
Sep 30, 2026Jiayi Yang, Yifang Chen, Yuanfu Sun +2Multi-Agent Coordination Text-Attributed Graph Learning
New York University Shanghai · New York University · Northwestern University
MASBench: Benchmarking LLM-based Multi-Agent Collaboration under Partial Observability
Oct 3, 2026Qizhi Chu, Zekai Yu, Sijie Wen +5Multi-Agent LLM Systems Multi-Agent Collaboration
Beijing University of Posts and Telecommunications · Shanghai Jiao Tong University · Tsinghua University

Aug 9, 2026J/K move · Enter open · S save

ForestBench: A Unified Graph Framework for Evaluating Multi-Agent Collaboration

Guo Chen, Ziwen Li, Reed Li, Yu Lu+3

Southwest University Chongqing, China · Tencent Beijing, China · Institute of Computing Technology, Chinese Academy of Sciences Beijing, China

Multi-Agent LLM Systems AI Agent Evaluation Multi-Agent Collaboration AI Agent Benchmarks

Open paper AskPDF arXiv

Multi-agent systems (MAS) built on Large Language Models (LLMs) are proliferating rapidly, but their heterogeneous execution traces provide no common basis for evaluation across methods. Outcome-only benchmarks discard collaborations, whereas LLM-as-Judge evaluation requires additional, model-dependent inference and can vary with the LLM and rubric. We introduce a generalizable evaluation framework that maps native MAS traces into a shared space of unified collaboration graphs, enabling different methods to be evaluated under the same representation, reference set, and metric panel. Candidate graphs are compared with a query-specific reference forest. Each forest is a benchmark-provided collection of verified-success graphs: it records diverse ways in which representative MAS methods can complete the task, rather than prescribing a unique optimal process. Instantiating the framework as ForestBench, we filter

844

collaboration-necessary queries from seven public datasets, precompute ten successful target-conditioned reference graphs per query, and evaluate six representative MAS frameworks. Controlled backbone, reference-construction, and perturbation studies test the stability and scope of evaluation. Once the benchmark forests are built, ForestBench scores a trace in milliseconds without further LLM inference, providing a reusable structural basis for comparing diverse MAS collaboration traces.

Abstract

Multi-Agent LLM Systems

Explore similar work

ForestBench: A Unified Graph Framework for Evaluating Multi-Agent Collaboration

GraphMAS: A Systematic Benchmark of Multi-Agent Coordination for Graph Learning

MASBench: Benchmarking LLM-based Multi-Agent Collaboration under Partial Observability