WorkflowArena: Benchmarking Executable Workflow Orchestration in LLM-Based Multi-Agent Systems
Organizations: Navers Lab, Einsia.AI, Tsinghua University
Abstract
Large language model agents are increasingly deployed as multi-agent systems rather than as a standalone model operating within a single context. Recent agent harnesses allow the model to instantiate worker agents and to author the executable workflow that coordinates them, shifting the design of system structure from developers to the model at run time. The quality of this organization therefore constitutes a distinct capability, yet existing benchmarks either measure end-to-end task completion or evaluate coordination under structures specified in advance. We formalize this capability as workflow orchestration and introduce WorkflowArena, a benchmark of 145 long-horizon tasks spanning 8 professional domains, from software engineering and formal mathematics to legal analysis and 3D production. Each task specifies an initial workspace and an integrated deliverable composed of interdependent components, which is assessed by a hidden verifier. Agents determine autonomously whether and how to delegate, and all delegation is recorded, enabling joint evaluation of deliverable quality and the organization of the underlying work. Experiments with nine models show that no single configuration performs best across all domains and that systems exhibit distinct delegation strategies: at high effort, Claude models in Claude Code invoke workflows less frequently but with larger worker groups, whereas GPT models in Codex delegate more frequently in smaller units. When the executing model is held fixed on a challenging subset of tasks, workflows authored by Claude Opus 5.5 substantially improve weaker executors, indicating that workflow organization is itself a measurable determinant of task performance.
Figures & tables
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Task form | Tasks | Source material | Instantiation and integration requirement | Acceptance evidence |
|---|---|---|---|---|
| SWE (17 tasks) | ||||
| Feature implementation | 11 | Open-source repository revisions and pull requests. | Implement a specified feature across an existing codebase while preserving unrelated behavior. | Hidden feature and regression tests |
| Subsystem reimplementation | 6 | Versioned open-source subsystems. | Remove implementation bodies while retaining the public interface and surrounding repository; reconstruct the missing behavior. | Behavior recovery and regression tests |
| OPS (18 tasks) | ||||
| Policy-constrained process execution | 6 | Public expenditure records with authored policies and exceptions. | Execute a queue of approvals or transactions under authorization, ordering, and resource constraints. | Outcome, action history, constraints |
| Cross-system business orchestration | 6 | Authored business scenarios grounded in operational records. | Coordinate records and obligations across business systems, including identity, access, finance, and case handling. | Final states and cross-system consistency |
| API model identifier | Harness | Uncached input | Cache read | Cache write | Output |
|---|---|---|---|---|---|
| claude-opus-5-5 | Claude Code | 4.00 | 0.20 | 5.00 | 20.00 |
| claude-sonnet-5-5 | Claude Code | 2.00 | 0.10 | 2.50 | 10.00 |
| gpt-6-astra | Codex | 10.00 | 1.00 | 12.50 | 50.00 |
| gpt-6.1-sol | Codex | 2.00 | 0.10 | 2.50 | 10.00 |
| gpt-6-luna | Codex | 0.10 | 0.01 | 0.125 | 0.50 |
| kimi-k3.1-pro | Claude Code | 2.00 | 0.20 | 2.00 | 10.00 |
| Domain | Acceptance criteria |
|---|---|
| SWE | Feature implementation: successful build and import; all required tests pass; verified expected test coverage, allowing only task-defined legitimate skips; no test-integrity violations. Subsystem reimplementation: all designated fail-to-pass tests pass, all previously passing behaviors remain intact, and no integrity violations or unresolved grader errors. |
| OPS | Business-goal check accuracy , process check accuracy , and all constraint checks pass. Each dimension uses its own check count as the denominator. |
| SCI | Blind predictive modeling: valid submissions with required target coverage, meeting the task-specific reference baseline under its prescribed metric comparison. Published result reproduction: all numerical targets meet their tolerances, with verified code and execution evidence supporting the reproduction. |
| MATH | Successful Lean build; all required declarations have the prescribed types; permitted-axiom and dependency restrictions hold; task-pinned definition values match. Applies to both formal proof construction and formal API development. |
| FIN | Cross-filing peer analysis: all required root checks pass. Multi-entity consolidation and audit: native full-success conditions hold, including required branch and aggregate comparisons and hard consistency checks. |
| LAW | Contract portfolio analysis: clause-extraction score and aggregation-question exact correctness . Multi-party case adjudication: correct legal determinations and satisfaction of the required sentencing and consistency criteria. |
LLM Agent Orchestration
Large language model agent orchestration coordinates models, tools, and workflows to complete complex tasks reliably and efficiently.
See all 489 papers