Multi-agent coding has emerged as an increasingly active direction in software engineering, where complex development tasks are decomposed across multiple specialized agents working on different parts of the problem. Despite the shift from individual problem solving to distributed collaboration, multi-agent systems still lack a direct measure of collaboration and are largely evaluated through task-level outcomes inherited from single-agent coding, conflating individual coding capability with cross-agent coordination. We introduce AsynCodeBench, a dependency-centric benchmark for asynchronous multi-agent software engineering that represents each task with an explicit dependency graph and executable Dependency Checkers. Through this dependency-tracking process, we propose two complementary measures: Asynchronous Dependency Pass Rate (ADPR), which measures how many cross-agent dependencies are ultimately satisfied, and Dependency Resolution Step (DRS), which measures when each dependency first becomes satisfied during execution. AsynCodeBench comprises 19 tasks from real-world repositories, exposing 52 directed dependencies as explicit units for evaluating cross-agent collaboration. Experiments across model families, scales, and generations reveal a clear gap between coding and collaboration capability: improvements in coding performance do not necessarily translate into stronger collaboration, and task-level metrics can diverge substantially from dependency-level collaboration measures. Dependency-trajectory analysis further reveals that successful coordination often emerges not gradually, but through concentrated bursts in which many dependencies become resolved over a short portion of the execution trajectory, a pattern we term a hopping window.
Figures & tables
Figure 1: A small illustration of the existing gap.
Figure 2: Collaboration gains and hidden dependency failures. (a) For each model, Single is compared with the highest task-macro Test Pass Rate achieved by any of the four multi-agent protocols; arrows show the signed gain. (b) Test Pass Rate versus ADPR for 190 Async-Manager-RO model–task outcomes. The dashed line indicates equality between the two metrics, and the 50% boundaries provide a visual partition.
Figure 3: Qwen scale and generation analysis under five matched execution protocols. The left region compares Qwen3.5-9B and Qwen3.5-27B, while the right region compares Qwen3.5, Qwen3.6, and Qwen3.8 at a nominal scale of 27B. Panels report Test Pass Rate, ADPR, and normalized DRS; lower DRS indicates earlier dependency resolution.
Figure 4: Protocol effects and dependency-closure dynamics. (a) ADPR change relative to Single across models and multi-agent protocols. (b) Cumulative dependency closure over normalized logical steps k/T , showing a recurrent middle-to-late hopping window ( 0.50<k/T≤0.75 ).
Figure 5: Representative dependency-closure trajectories across execution protocols. Each panel shows the cumulative fraction of dependencies ever resolved over normalized logical steps for one model. The shaded region marks the hopping window.
Figure 6: Quality–cost Pareto frontiers for Nemotron-3.5 under ADPR and Test Pass Rate . Both use identical protocols and token costs, yet identify different non-dominated configurations. Marker shapes indicate execution protocols, and lines connect their respective Pareto frontiers.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Task-level dependency graphs in AsynCodeBench. Across the benchmark, we identify nine types of dependency relations, each defined by explicit upstream and downstream implementation units. These dependencies form the basis for dependency-level evaluation with ADPR.
Figure 8: Domain distribution of tasks in AsynCodeBench.
Task
Source record
Domain
Lang.
Agents
Files
Edges
Tests
Ov.
cachetools
Commit0
F/L
Py
2
2
5
12
0
deprecated
Commit0
F/L
Py
2
2
3
8
0
portalocker
Commit0
S/S
Py
2
2
3
5
1
tinydb
Commit0
S/S
Py
2
7
3
8
1
wcwidth
Commit0
S/S
Py
2
2
2
4
0
requests
Commit0
N/P
Py
3
13
3
5
3
Appendix
Table 1: Complete AsynCodeBench task inventory and provenance. Agents denotes the number of specialists used in a multi-agent condition; Files denotes the number of distinct implementation files writable by specialists; Edges denotes the number of registered dependency edges; Tests denotes the number of test files included in the final evaluator; and Ov. denotes the number of checksum-pinned public overlays. Domain abbreviations are F/L (framework and libraries), S/S (systems and storage), C/IR (compiler and IR), N/P (network protocols), and SEC (security).
Task Review Sheet
Task
<task-name>
Reviewer
Reviewer 1
Include
□ Yes □ No
Parallelizability
□ Parallelizable □ Partially parallelizable □ Effectively serial
Rationale
Reviewer justification for whether the task contains meaningful dependencies and can be naturally decomposed across agents.
Appendix
Table 2: Human-review sheet used for task validation. Each task is reviewed independently by three reviewers.
Protocol
Decomp.
Concurrent
Manager
Writable Manager
Single
–
–
–
–
Serial
✓
–
–
–
Async-Private
✓
✓
–
–
Async-Manager-RO
✓
✓
✓
–
Async-Manager
✓
✓
✓
✓
Appendix
Table 3: Execution protocols in AsynCodeBench. “Writable manager” denotes scope-constrained patches produced in a private manager worktree; the manager never directly modifies the integrated workspace.
Figure 9: Event-triggered checkpoint construction in AsynCodeBench. The artifact checkpoint evaluates the three checker groups in a private specialist workspace and is used only for local diagnostics. Integrated checker outcomes are recorded at the gold-bordered trajectory checkpoints after integration attempts, accepted state-changing manager interventions when applicable, and final evaluation. Only these ordered integrated-workspace outcomes form the dependency trajectory used by ADPR and DRS.
Paper name
Evaluated checkpoint
Archi- tecture
Total
Active
Qwen3.5-9B
Qwen/ Qwen3.5-9B
Dense
9B
9B
Qwen3.5-27B
Qwen/ Qwen3.5-27B
Dense
27B
27B
Qwen3.6-27B
Qwen/ Qwen3.6-27B
Dense
27B
27B
Qwen3.8-27B
Qwen/ Qwen3.8-27B
Dense
27B
27B
Qwen3-Coder- Next
Qwen/ Qwen3-Coder- Next-FP8
MoE
80B
3B
DeepSeek- V4-Flash
deepseek/ deepseek-v4- flash-0731
MoE
284B
13B
Appendix
Table 4: Models evaluated in AsynCodeBench. Active denotes the approximate number of parameters activated per token. For dense models, all language-model parameters are active.
Figure 10: Test Pass Rate versus ADPR under the four protocols not shown in Figure 2 (b). Each point represents a model–task outcome and colors denote models. The dashed line indicates equality between Test Pass Rate and ADPR, while the 50% boundaries provide a common visual partition. All panels use identical axes and encodings.
Protocol
Low Test / Low ADPR ↓
Low Test / High ADPR
High Test / Low ADPR ↓
High Test / High ADPR ↑
Single
97 (51.1%)
0 (0.0%)
42 (22.1%)
51 (26.8%)
Serial
104 (54.7%)
0 (0.0%)
44 (23.2%)
42 (22.1%)
Async-Private
116 (61.1%)
0 (0.0%)
49 (25.8%)
25 (13.2%)
Async-Manager-RO
105 (55.3%)
0 (0.0%)
52 (27.4%)
33 (17.4%)
Async-Manager
83 (43.7%)
3 (1.6%)
50 (26.3%)
54 (28.4%)
Appendix
Table 5: Distribution of model–task outcomes across the four regions in Figure 10 and Figure 2 (b). Each protocol contains 190 outcomes. Entries report count (percentage). The 50% threshold is used only as a descriptive visual partition. Blue cells denote the best value according to the indicated direction, while orange cells highlight values discussed in the accompanying analysis.
Metric
Single
Serial
Async-P.
Manager-RO
Manager
Pearson r
0.716
0.667
0.599
0.617
0.628
Spearman ρ
0.737
0.651
0.559
0.613
0.657
Mean Test Pass Rate ↑
51.0%
47.9%
41.9%
48.0%
57.4%
Mean ADPR ↑
27.4%
22.7%
14.2%
18.8%
31.3%
Mean signed gap ↓
23.7%
25.2%
27.8%
29.2%
26.1%
Test Pass Rate > ADPR ↓
62.6%
70.5%
77.9%
71.6%
67.4%
Appendix
Table 6: Correlation and mismatch diagnostics by protocol. Percentages are computed over 190 model–task outcomes per protocol. The signed gap is Test Pass Rate minus ADPR; the absolute gap ignores direction. Arrows indicate the preferred direction where applicable. Blue cells denote the best value according to the indicated direction, while orange cells highlight values discussed in the accompanying analysis.
Cumulative closure (%)
Protocol
By 50%
By 75%
Ever closed
Closure in (0.50,0.75] (%)
Serial
4.9
20.3
22.7
67.8
Async-Private
4.6
12.6
14.2
56.5
Async-Manager-RO
3.0
11.9
18.9
47.3
Async-Manager
13.8
24.7
31.3
34.9
Appendix
Table 7: Cumulative dependency closure and hopping-window concentration across execution protocols.
Protocol
Peak events from k=2,T=3
First closure at raw k=2
Serial
68.9%
58.6%
Async-Private
73.9%
60.2%
Async-Manager-RO
66.7%
38.4%
Async-Manager
50.0%
48.6%
Appendix
Table 8: Effect of checkpoint normalization on the hopping-window pattern.
Figure 11: Model–protocol variation in hopping-window concentration. Each cell shows the share of eventual first closures occurring within the hopping window (0.50, 0.75]. N/A denotes no observed dependency closure.
Figure 12: Dependency-closure trajectories for the seven models not shown in the main-text case studies. Curves report cumulative dependencies ever resolved over normalized logical execution, and crosses denote final ADPR. The shaded region marks the representative aggregate concentration 0.50<x≤0.75 . Individual trajectories show that the onset and magnitude of concentrated closure vary substantially across models and protocols.
Figure 13: Metric-dependent quality–cost trade-offs for four additional models. Each panel shows the within-model Pareto frontiers under ADPR (blue, left axis) and Test Pass Rate (orange, right axis). Marker shapes distinguish execution protocols, and lines connect non-dominated configurations under each metric. Dominated configurations remain visible. Axis ranges are independently scaled across models.
Model
Same membership?
ADPR-only protocols
Test-Pass-only protocols
DeepSeek-V4
Yes
–
–
Qwen3.8-27B
Yes
–
–
Qwen3.6-27B
Yes
–
–
Qwen3.5-27B
Yes
–
–
Qwen3-Coder-Next
Yes
–
–
Qwen3.5-9B
No
–
Async-Manager
Appendix
Table 9: Comparison of within-model Pareto frontiers under ADPR and Test Pass Rate. The two quality metrics are evaluated using identical protocols and token costs. Only metric-exclusive configurations are listed.
Figure 14: Case study of DeepSeek-V4 under Async-Manager on TVM-20073 . Although both upstream requirements become available during execution, the shared downstream span-propagation logic remains unsatisfied after two attempts, leaving both dependencies unresolved. The final workspace nevertheless passes 24 of 28 evaluator items (85.7% Test Pass Rate), while ADPR remains 0%, illustrating how task-level performance can obscure unresolved cross-agent dependencies and the coordination failures behind them.
Protocol
Test Pass ↑
ADPR ↑
DRS ↓
Tokens (M) ↓
Qwen3.5-9B
Single
24.44
14.21
–
5.18
Serial
15.83
5.96
0.962
8.01
Async-Private
15.13
0.00
1.000
8.32
Async-Manager-RO
28.44
1.05
0.992
15.53
Async-Manager
36.93
2.81
0.988
15.48†
Appendix
Table 10: Complete experimental results for the Qwen model family. Test Pass Rate and ADPR are reported as percentages. Tokens denote mean reported tokens per task in millions.
Protocol
Test Pass ↑
ADPR ↑
DRS ↓
Tokens (M) ↓
DeepSeek-V4
Single
93.64
78.95
–
2.54
Serial
87.84
66.67
0.665
6.89
Async-Private
75.75
45.61
0.769
6.92
Async-Manager-RO
80.36
64.21
0.748
11.61
Async-Manager
89.85
71.93
0.690
13.65
Appendix
Table 11: Complete experimental results for the remaining model families. Test Pass Rate and ADPR are reported as percentages. Tokens denote mean reported tokens per task in millions.
Single
Serial
Async-Private
Async-Manager-RO
Async-Manager
Qwen3.5-9B
2/19 (10.5%)
0/19 (0.0%)
0/19 (0.0%)
0/19 (0.0%)
0/19 (0.0%)
Gemma-4-26B
0/19 (0.0%)
0/19 (0.0%)
0/19 (0.0%)
0/19 (0.0%)
1/19 (5.3%)
Nemotron-3.5
0/19 (0.0%)
0/19 (0.0%)
0/19 (0.0%)
0/19 (0.0%)
0/19 (0.0%)
Appendix
Table 12: Number of tasks with successful outcomes across models and execution protocols. Each entry reports the number of successful tasks out of 19, with the corresponding percentage in parentheses.
Multi-agent coding systems are designed to tackle complex software engineering tasks through collaboration. However, existing evaluations typically assume configured organizations are followed faithfully, whereas reality differs. This behavioral gap, combined with differences in underlying system components, prevents clear attribution of observed gains. To this end, we introduce OpenCollab, a multi-agent coding framework that provides a unified infrastructure for programmable collaboration and controllable runtime. Specifically, OpenCollab unifies organization design, enforces experimental control on a shared runtime, and tracks execution through fine-grained event streams. On this basis, we define Adherence to quantify whether the declared organization is actually realized. Our experiments reveal that agents collaborate very differently across configurations: changing any single dimension shifts Adherence, from 47.2% to as high as 97.2%. Furthermore, extensive agentic coding benchmarks show that OpenCollab (Duo) establishes new SOTA performance compared to the mainstream harnesses such as Mini-SWE-agent, OpenHands, Codex CLI, and Claude Code, showing that a well-designed organization can outperform strong existing harnesses, while OpenCollab's single-agent configuration uses the fewest tokens across all evaluated suites. OpenCollab establishes a unified multi-agent infrastructure for easy programmable collaboration and controlled causal evaluation.
Chun-Wah Hsu, Kai Gong, Yu Wu +12
Shanghai Jiao Tong University · University of Cambridge · Nanyang Technological University +4
AI agents have become increasingly capable at isolated software engineering (SWE) tasks such as resolving issues on Github. Yet long-horizon tasks involving multiple interdependent subtasks still pose challenges both with respect to accuracy, and with respect to timely completion. A natural approach to solving these long-horizon tasks in a timely manner is asynchronous multi-agent collaboration, where multiple agents work on different parts of the task at the same time. But effective application of multi-agent systems has proven surprisingly difficult: concurrent edits by multiple agents interfere with each other, dependencies are difficult to synchronize, and combining partial progress into a coherent whole is challenging. On the other hand, human developers have long relied on mature collaboration infrastructure to manage these challenges in large software projects. Inspired by these collaboration primitives, we introduce Centralized Asynchronous Isolated Delegation (CAID), a structured multi-agent coordination paradigm grounded in three core SWE primitives: centralized task delegation, asynchronous execution, and isolated workspaces. CAID constructs dependency-aware task plans through a central manager, executes subtasks concurrently in isolated workspaces, and consolidates progress via structured integration with executable test-based verification. In empirical evaluation, we find that CAID improves accuracy over single-agent baselines by 25.6% absolute on paper reproduction tasks (PaperBench) and 14.7% on Python library development tasks (Commit0). Through systematic analysis, we find that branch-and-merge is a central coordination mechanism for multi-agent collaboration, and that SWE primitives such as git worktree, git commit, and git merge enable it to be realized in a reliable and executable manner.
Jiayi Geng, Graham Neubig
Carnegie Mellon University, Language Technologies Institute
Multi-agent vibe coding promises to accelerate software development, yet existing benchmarks rely on synthetic environments that ignore practical time and monetary costs, conflate reasoning with communication, and reward only superficial completion. We introduce multi-agent from-scratch evaluation benchmark, MSEval, evaluating multi-agent coding on real-world tasks. Grounded in 10 authentic, full-stack projects across 10 domains, MSEval scores performance using hierarchical requirements and deterministic rubrics. Its execution engine, LegoGent, tests 10 collaboration topologies where agents coordinate via periodic sync intervals and deploy through native CI/CD pipelines. Concurrently, the automated grader TAgent dynamically probes implementations to jointly measure functional success, latency, and prefix-cached token cost. Across 100 runs, MSEval reveals that organizational topology rivals model capability in shaping the speed--cost--quality trade-off. For identical tasks and models, varying the topology shifts scores by over 30 points and doubles wall-clock time. Structured pipelines converge fastest with the highest quality, whereas heavy managerial oversight degrades performance. Ultimately, MSEval establishes a rigorous, reproducible standard for measuring how multi-agent teams actually build software. The benchmark is released at https://github.com/robinren03/MSEval.