AsynCodeBench: Benchmarking Collaboration of Asynchronous Multi-Agent Systems in Software Engineering

Kaituo Zhang; Zhen Xiong; Zhimeng Jiang; Mingyu Zhong; Zhouyuan Yuan; Zhecheng Li; Bowen Lin; Chia-Yuan Chang

cs.SESep 26, 2026

AsynCodeBench: Benchmarking Collaboration of Asynchronous Multi-Agent Systems in Software Engineering

Authors: Kaituo Zhang, Zhen Xiong, Zhimeng Jiang, Mingyu Zhong, Zhouyuan Yuan, Zhecheng Li, Bowen Lin, Chia-Yuan Chang,

Organizations: University of Houston · New York University · Texas A&M University · UCSD ·

Abstract

Multi-agent coding has emerged as an increasingly active direction in software engineering, where complex development tasks are decomposed across multiple specialized agents working on different parts of the problem. Despite the shift from individual problem solving to distributed collaboration, multi-agent systems still lack a direct measure of collaboration and are largely evaluated through task-level outcomes inherited from single-agent coding, conflating individual coding capability with cross-agent coordination. We introduce AsynCodeBench, a dependency-centric benchmark for asynchronous multi-agent software engineering that represents each task with an explicit dependency graph and executable Dependency Checkers. Through this dependency-tracking process, we propose two complementary measures: Asynchronous Dependency Pass Rate (ADPR), which measures how many cross-agent dependencies are ultimately satisfied, and Dependency Resolution Step (DRS), which measures when each dependency first becomes satisfied during execution. AsynCodeBench comprises 19 tasks from real-world repositories, exposing 52 directed dependencies as explicit units for evaluating cross-agent collaboration. Experiments across model families, scales, and generations reveal a clear gap between coding and collaboration capability: improvements in coding performance do not necessarily translate into stronger collaboration, and task-level metrics can diverge substantially from dependency-level collaboration measures. Dependency-trajectory analysis further reveals that successful coordination often emerges not gradually, but through concentrated bursts in which many dependencies become resolved over a short portion of the execution trajectory, a pattern we term a hopping window.

Figures & tables

Figure 1: A small illustration of the existing gap.

Figure 2: Collaboration gains and hidden dependency failures. (a) For each model, Single is compared with the highest task-macro Test Pass Rate achieved by any of the four multi-agent protocols; arrows show the signed gain. (b) Test Pass Rate versus ADPR for 190 Async-Manager-RO model–task outcomes. The dashed line indicates equality between the two metrics, and the 50% boundaries provide a visual partition.

Figure 3: Qwen scale and generation analysis under five matched execution protocols. The left region compares Qwen3.5-9B and Qwen3.5-27B, while the right region compares Qwen3.5, Qwen3.6, and Qwen3.8 at a nominal scale of 27B. Panels report Test Pass Rate, ADPR, and normalized DRS; lower DRS indicates earlier dependency resolution.

Figure 4: Protocol effects and dependency-closure dynamics. (a) ADPR change relative to Single across models and multi-agent protocols. (b) Cumulative dependency closure over normalized logical steps $k/T$ , showing a recurrent middle-to-late hopping window ( $0.50<k/T\leq 0.75$ ).

Figure 5: Representative dependency-closure trajectories across execution protocols. Each panel shows the cumulative fraction of dependencies ever resolved over normalized logical steps for one model. The shaded region marks the hopping window.

Figure 6: Quality–cost Pareto frontiers for Nemotron-3.5 under ADPR and Test Pass Rate . Both use identical protocols and token costs, yet identify different non-dominated configurations. Marker shapes indicate execution protocols, and lines connect their respective Pareto frontiers.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Task-level dependency graphs in AsynCodeBench. Across the benchmark, we identify nine types of dependency relations, each defined by explicit upstream and downstream implementation units. These dependencies form the basis for dependency-level evaluation with ADPR.

Figure 8: Domain distribution of tasks in AsynCodeBench.

Task	Source record	Domain	Lang.	Agents	Files	Edges	Tests	Ov.
cachetools	Commit0	F/L	Py	2	2	5	12	0
deprecated	Commit0	F/L	Py	2	2	3	8	0
portalocker	Commit0	S/S	Py	2	2	3	5	1
tinydb	Commit0	S/S	Py	2	7	3	8	1
wcwidth	Commit0	S/S	Py	2	2	2	4	0
requests	Commit0	N/P	Py	3	13	3	5	3

Appendix

Table 1: Complete AsynCodeBench task inventory and provenance. Agents denotes the number of specialists used in a multi-agent condition; Files denotes the number of distinct implementation files writable by specialists; Edges denotes the number of registered dependency edges; Tests denotes the number of test files included in the final evaluator; and Ov. denotes the number of checksum-pinned public overlays. Domain abbreviations are F/L (framework and libraries), S/S (systems and storage), C/IR (compiler and IR), N/P (network protocols), and SEC (security).

Task Review Sheet
Task	<task-name>
Reviewer	Reviewer 1
Include	$\square$ Yes $\square$ No
Parallelizability	$\square$ Parallelizable $\square$ Partially parallelizable $\square$ Effectively serial
Rationale	Reviewer justification for whether the task contains meaningful dependencies and can be naturally decomposed across agents.

Appendix

Table 2: Human-review sheet used for task validation. Each task is reviewed independently by three reviewers.

Protocol	Decomp.	Concurrent	Manager	Writable Manager
Single	–	–	–	–
Serial	✓	–	–	–
Async-Private	✓	✓	–	–
Async-Manager-RO	✓	✓	✓	–
Async-Manager	✓	✓	✓	✓

Appendix

Table 3: Execution protocols in AsynCodeBench. “Writable manager” denotes scope-constrained patches produced in a private manager worktree; the manager never directly modifies the integrated workspace.

Figure 9: Event-triggered checkpoint construction in AsynCodeBench. The artifact checkpoint evaluates the three checker groups in a private specialist workspace and is used only for local diagnostics. Integrated checker outcomes are recorded at the gold-bordered trajectory checkpoints after integration attempts, accepted state-changing manager interventions when applicable, and final evaluation. Only these ordered integrated-workspace outcomes form the dependency trajectory used by ADPR and DRS.

Paper name		Evaluated checkpoint	Archi- tecture	Total	Active
	Qwen3.5-9B	Qwen/ Qwen3.5-9B	Dense	9B	9B
	Qwen3.5-27B	Qwen/ Qwen3.5-27B	Dense	27B	27B
	Qwen3.6-27B	Qwen/ Qwen3.6-27B	Dense	27B	27B
	Qwen3.8-27B	Qwen/ Qwen3.8-27B	Dense	27B	27B
	Qwen3-Coder- Next	Qwen/ Qwen3-Coder- Next-FP8	MoE	80B	3B
	DeepSeek- V4-Flash	deepseek/ deepseek-v4- flash-0731	MoE	284B	13B

Appendix

Table 4: Models evaluated in AsynCodeBench. Active denotes the approximate number of parameters activated per token. For dense models, all language-model parameters are active.

Figure 10: Test Pass Rate versus ADPR under the four protocols not shown in Figure 2 (b). Each point represents a model–task outcome and colors denote models. The dashed line indicates equality between Test Pass Rate and ADPR, while the 50% boundaries provide a common visual partition. All panels use identical axes and encodings.

Protocol	Low Test / Low ADPR $\downarrow$	Low Test / High ADPR	High Test / Low ADPR $\downarrow$	High Test / High ADPR $\uparrow$
Single	97 (51.1%)	0 (0.0%)	42 (22.1%)	51 (26.8%)
Serial	104 (54.7%)	0 (0.0%)	44 (23.2%)	42 (22.1%)
Async-Private	116 (61.1%)	0 (0.0%)	49 (25.8%)	25 (13.2%)
Async-Manager-RO	105 (55.3%)	0 (0.0%)	52 (27.4%)	33 (17.4%)
Async-Manager	83 (43.7%)	3 (1.6%)	50 (26.3%)	54 (28.4%)

Appendix

Table 5: Distribution of model–task outcomes across the four regions in Figure 10 and Figure 2 (b). Each protocol contains 190 outcomes. Entries report count (percentage). The 50% threshold is used only as a descriptive visual partition. Blue cells denote the best value according to the indicated direction, while orange cells highlight values discussed in the accompanying analysis.

Metric	Single	Serial	Async-P.	Manager-RO	Manager
Pearson $r$	0.716	0.667	0.599	0.617	0.628
Spearman $\rho$	0.737	0.651	0.559	0.613	0.657
Mean Test Pass Rate $\uparrow$	51.0%	47.9%	41.9%	48.0%	57.4%
Mean ADPR $\uparrow$	27.4%	22.7%	14.2%	18.8%	31.3%
Mean signed gap $\downarrow$	23.7%	25.2%	27.8%	29.2%	26.1%
Test Pass Rate $>$ ADPR $\downarrow$	62.6%	70.5%	77.9%	71.6%	67.4%

Appendix

Table 6: Correlation and mismatch diagnostics by protocol. Percentages are computed over 190 model–task outcomes per protocol. The signed gap is Test Pass Rate minus ADPR; the absolute gap ignores direction. Arrows indicate the preferred direction where applicable. Blue cells denote the best value according to the indicated direction, while orange cells highlight values discussed in the accompanying analysis.

	Cumulative closure (%)
Protocol	By 50%	By 75%	Ever closed	Closure in $(0.50,0.75]$ (%)
Serial	4.9	20.3	22.7	67.8
Async-Private	4.6	12.6	14.2	56.5
Async-Manager-RO	3.0	11.9	18.9	47.3
Async-Manager	13.8	24.7	31.3	34.9

Appendix

Table 7: Cumulative dependency closure and hopping-window concentration across execution protocols.

Protocol	Peak events from $k=2,T=3$	First closure at raw $k=2$
Serial	68.9%	58.6%
Async-Private	73.9%	60.2%
Async-Manager-RO	66.7%	38.4%
Async-Manager	50.0%	48.6%

Appendix

Table 8: Effect of checkpoint normalization on the hopping-window pattern.

Figure 11: Model–protocol variation in hopping-window concentration. Each cell shows the share of eventual first closures occurring within the hopping window (0.50, 0.75]. N/A denotes no observed dependency closure.

Figure 12: Dependency-closure trajectories for the seven models not shown in the main-text case studies. Curves report cumulative dependencies ever resolved over normalized logical execution, and crosses denote final ADPR. The shaded region marks the representative aggregate concentration

0.50<x\leq 0.75

. Individual trajectories show that the onset and magnitude of concentrated closure vary substantially across models and protocols.

Figure 13: Metric-dependent quality–cost trade-offs for four additional models. Each panel shows the within-model Pareto frontiers under ADPR (blue, left axis) and Test Pass Rate (orange, right axis). Marker shapes distinguish execution protocols, and lines connect non-dominated configurations under each metric. Dominated configurations remain visible. Axis ranges are independently scaled across models.

Model	Same membership?	ADPR-only protocols	Test-Pass-only protocols
DeepSeek-V4	Yes	–	–
Qwen3.8-27B	Yes	–	–
Qwen3.6-27B	Yes	–	–
Qwen3.5-27B	Yes	–	–
Qwen3-Coder-Next	Yes	–	–
Qwen3.5-9B	No	–	Async-Manager

Appendix

Table 9: Comparison of within-model Pareto frontiers under ADPR and Test Pass Rate. The two quality metrics are evaluated using identical protocols and token costs. Only metric-exclusive configurations are listed.

Figure 14: Case study of DeepSeek-V4 under Async-Manager on TVM-20073 . Although both upstream requirements become available during execution, the shared downstream span-propagation logic remains unsatisfied after two attempts, leaving both dependencies unresolved. The final workspace nevertheless passes 24 of 28 evaluator items (85.7% Test Pass Rate), while ADPR remains 0%, illustrating how task-level performance can obscure unresolved cross-agent dependencies and the coordination failures behind them.

Protocol	Test Pass $\uparrow$	ADPR $\uparrow$	DRS $\downarrow$	Tokens (M) $\downarrow$
Qwen3.5-9B
Single	24.44	14.21	–	5.18
Serial	15.83	5.96	0.962	8.01
Async-Private	15.13	0.00	1.000	8.32
Async-Manager-RO	28.44	1.05	0.992	15.53
Async-Manager	36.93	2.81	0.988	$15.48^{\dagger}$

Appendix

Table 10: Complete experimental results for the Qwen model family. Test Pass Rate and ADPR are reported as percentages. Tokens denote mean reported tokens per task in millions.

Protocol	Test Pass $\uparrow$	ADPR $\uparrow$	DRS $\downarrow$	Tokens (M) $\downarrow$
DeepSeek-V4
Single	93.64	78.95	–	2.54
Serial	87.84	66.67	0.665	6.89
Async-Private	75.75	45.61	0.769	6.92
Async-Manager-RO	80.36	64.21	0.748	11.61
Async-Manager	89.85	71.93	0.690	13.65

Appendix

Table 11: Complete experimental results for the remaining model families. Test Pass Rate and ADPR are reported as percentages. Tokens denote mean reported tokens per task in millions.

Single	Serial	Async-Private	Async-Manager-RO	Async-Manager
Qwen3.5-9B
2/19 (10.5%)	0/19 (0.0%)	0/19 (0.0%)	0/19 (0.0%)	0/19 (0.0%)
Gemma-4-26B
0/19 (0.0%)	0/19 (0.0%)	0/19 (0.0%)	0/19 (0.0%)	1/19 (5.3%)
Nemotron-3.5
0/19 (0.0%)	0/19 (0.0%)	0/19 (0.0%)	0/19 (0.0%)	0/19 (0.0%)

Appendix

Table 12: Number of tasks with successful outcomes across models and execution protocols. Each entry reports the number of successful tasks out of 19, with the corresponding percentage in parentheses.

Multi-Agent Collaboration

Multi-agent collaboration studies how autonomous agents coordinate, share information, and combine capabilities to solve tasks together.

See all 320 papers

OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime
Sep 29, 2026Chun-Wah Hsu, Kai Gong, Yu Wu +12Multi-Agent Orchestration Multi-Agent Collaboration
Shanghai Jiao Tong University · University of Cambridge · Nanyang Technological University +4
Effective Strategies for Asynchronous Software Engineering Agents
Mar 23, 2026Jiayi Geng, Graham NeubigMulti-Agent Coordination Software Engineering Agents
Carnegie Mellon University, Language Technologies Institute
An Empirical Study of Coordination Mode as the First-Class Citizen in From-Scratch Multi-Agent Coding
Jul 30, 2026Yanyu Ren, Yunfeng Bai, Xizheng Wang +2Multi-Agent Coordination Multi-Agent Orchestration
1Tsinghua University · 2Zhongguancun Laboratory

Sep 29, 2026J/K move · Enter open · S save

OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime

Chun-Wah Hsu, Kai Gong, Yu Wu, Xianhe Chen+11

Shanghai Jiao Tong University · University of Cambridge · Nanyang Technological University+4

Multi-Agent Orchestration Multi-Agent Collaboration AI Agent Benchmarks Multi-Agent Coding

Open paper AskPDF arXiv

Multi-agent coding systems are designed to tackle complex software engineering tasks through collaboration. However, existing evaluations typically assume configured organizations are followed faithfully, whereas reality differs. This behavioral gap, combined with differences in underlying system components, prevents clear attribution of observed gains. To this end, we introduce OpenCollab, a multi-agent coding framework that provides a unified infrastructure for programmable collaboration and controllable runtime. Specifically, OpenCollab unifies organization design, enforces experimental control on a shared runtime, and tracks execution through fine-grained event streams. On this basis, we define Adherence to quantify whether the declared organization is actually realized. Our experiments reveal that agents collaborate very differently across configurations: changing any single dimension shifts Adherence, from 47.2% to as high as 97.2%. Furthermore, extensive agentic coding benchmarks show that OpenCollab (Duo) establishes new SOTA performance compared to the mainstream harnesses such as Mini-SWE-agent, OpenHands, Codex CLI, and Claude Code, showing that a well-designed organization can outperform strong existing harnesses, while OpenCollab's single-agent configuration uses the fewest tokens across all evaluated suites. OpenCollab establishes a unified multi-agent infrastructure for easy programmable collaboration and controlled causal evaluation.

Abstract

Multi-Agent Collaboration

Explore similar work

OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime

Effective Strategies for Asynchronous Software Engineering Agents

An Empirical Study of Coordination Mode as the First-Class Citizen in From-Scratch Multi-Agent Coding