SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations
Organizations: TokenWave AI
Abstract
A coding agent working within a software organization must establish what to build, collaborate with colleagues, and verify what it delivers. We introduce SWE-CoWork, a benchmark comprising 101 development assignments across 101 real repositories, 11 programming languages, and 12 application domains. Each assignment places an agent in a simulated engineering organization with 26 colleagues who have distinct responsibilities, bounded knowledge, and controlled collaboration behavior. Starting from a brief handoff, agents must recover requirements distributed across colleagues and workplace records, reconcile conflicting evidence, and implement changes evaluated by private executable tests. We evaluate eleven models and find a substantial gap between partial implementation and complete task delivery. The two leading models satisfy approximately 63% of fail-to-pass (F2P) tests but complete fewer than 8% of assignments. Providing complete requirements upfront improves F2P by approximately 16 percentage points for Seed Evolving and GPT-6 Astra, yet leaves complete task success unchanged. Requirement-level trajectory analysis reveals failures throughout the delivery process: Claude-Opus-5.5 directs a mean of 93% of first questions about queried cards to their accountable owners across runs but obtains only 64% of colleague-held requirements. Deferred questions go unfollowed, implementations remain incomplete even after requirements are acquired, and 31% of requirements reported as completed fail their tests. These findings highlight sustained requirement tracking, from clarification through implementation, verification, and handoff, as a key challenge for reliable autonomous software engineering.
Figures & tables
| Benchmark | Engineering task | Initial information | Information sources | Information process | Evaluation |
|---|---|---|---|---|---|
| DeepSWE 1.1 ( Huang et al., 2026 ; Datacurve, 2026 ) | Repo change | Request Underspecified | Repo | n/r | Patch tests |
| SWE Atlas ( Raghavendra et al., 2026 ) | Code Q&A Test writing Refactor | Request Underspecified | Repo | No live Q&A | Execution Rubric |
| FrontierSWE v2 ( Kondra et al., 2026 ) | Build Optimize Research | Brief | Workspace Self-check | n/r | Task score |
| CursorBench 4.0 ( Cursor, 2026 ; Jain, 2026 ) | Edit Refactor Investigate | Request Underspecified | Code Tools | n/r | Agentic grader |
| Real-SWE (Specific Labs) ( Das et al., 2026 ) | Enterprise change | Brief Underspecified | Repo Docs Board Chat | n/r | n/r |
| RealSWE (paper v2) ( Kim et al., 2026 ) | Bug fix Feature | Issue variants Underspecified | Repo | Static request No live Q&A | Fixed target |
| Model | F2P (%) | P2P (%) | Pass rate (%) |
|---|---|---|---|
| Claude-Opus-5.5 | 63.84 | 93.24 | 6.93 |
| GPT-6 Astra | 62.91 | 89.15 | 7.92 |
| Seed Evolving | 39.09 | 94.48 | 0.99 |
| Qwen3.8-Max | 35.70 | 91.80 | 0.00 |
| GLM 5.3 | 31.27 | 87.72 | 0.99 |
| Qwen3.8-Flash | 30.70 | 93.40 | 0.00 |
| Seed Evolving | GPT-6 Astra | |||||
|---|---|---|---|---|---|---|
| Metric | CoWork | Oracle | Gain | CoWork | Oracle | Gain |
| F2P | 39.09 | 55.76 | +16.67 | 62.91 | 78.84 | +15.93 |
| P2P | 94.48 | 89.50 | 4.98 | 89.15 | 91.81 | +2.66 |
| Pass rate | 0.99 | 0.99 | +0.00 | 7.92 | 7.92 | +0.00 |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark | Focus | Information setting | Evaluation / control |
|---|---|---|---|
| SWE-bench ( Jimenez et al., 2024 ) | Issue resolution | Issue and repository | F2P/P2P tests |
| SWE-bench Pro ( Deng et al., 2025 ) | Complex software changes | Issue and repository | Repository tests |
| SWE-Lancer ( Miserendino et al., 2025 ) | Paid freelance work | Task description | End-to-end tests for implementation tasks |
| Active-SWE ( Li et al., 2026 ) | Proactive bug discovery and fixing | Repository without issue reports | Bug discovery and repair evaluation |
| TheAgentCompany ( Xu et al., 2025 ) | Professional workflows, including SWE | Company services and multiple colleagues | Programmatic checkpoints; some model-based criteria |
| LHAW ( Pu et al., 2026 ) | Controlled underspecification | Removed goals, constraints, inputs or context | Inherited evaluators and agent trials |
| Domain | Upstream repositories | |
|---|---|---|
| Web & Frontend | 26 | ada-url/ada, api-platform/core, collective/icalendar, emberjs/ember.js, feedbin/feedbin, fomantic/fomantic-ui, gioxx/marvelloussuspender, googleapis/google-api-php-client, hyperf/hyperf, ing-bank/lion, ionic-team/stencil, jasontaylordev/cleanarchitecture, jeremydaly/lambda-api, jnunemaker/httparty, jquery/jquery, kimai/kimai, mika-cn/maoxian-web-clipper, nolimits4web/swiper, python/pythondotorg, rubenlagus/telegrambots, saloonphp/saloon, shane32/qrcoder, tanstack/query, urllib3/urllib3, vacanza/holidays, webrecorder/pywb |
| Dev Tools & Compilers | 22 | ansible/ansible-lint, carapace-sh/carapace-bin, cargo-generate/cargo-generate, cocoapods/xcodeproj, embarkstudios/cargo-deny, firebase/firebase-tools, git-lfs/git-lfs, hotswapprojects/hotswapagent, isomorphic-git/isomorphic-git, jctools/jctools, jest-community/jest-extended, junit-team/junit-examples, kpdecker/jsdiff, mdx-js/mdx, microsoft/vscode-js-debug, nedbat/coveragepy, pallets/click, postcss/postcss, psf/black, serde-rs/serde, stoplightio/elements, thought-machine/please |
| Data & Databases | 11 | alibaba/jetcache, clockworklabs/spacetimedb, dolthub/go-mysql-server, elastic/elasticsearch-rails, friendsofsymfony/foselasticabundle, kotlin-orm/ktorm, mongodb/laravel-mongodb, qishibo/anotherredisdesktopmanager, rubyzip/rubyzip, ruflin/elastica, webtorrent/webtorrent |
| Games & Graphics | 10 | caffeinemc/sodium, hubs-foundation/hubs, mezz/justenoughitems, mikke89/rmlui, multitheftauto/mtasa-blue, nasa-ammos/3dtilesrendererjs, sergio0694/computesharp, steamre/steamkit, tencent/puerts, underminersteam/undertalemodtool |
| Desktop Applications | 7 | betterdiscord/betterdiscord, bobotig/python-mss, electronnet/electron.net, ipfs/ipfs-desktop, openutau/openutau, pimutils/khal, sozi-projects/sozi |
| Cloud & Infrastructure | 5 | boto/boto3, brightercommand/brighter, kubermatic/kubeone, robusta-dev/robusta, stackstorm/st2 |
| Quantity per task | Mean | Median | P25 | P75 | Max |
| Code task | |||||
| Verifier test nodes | 32.3 | 32 | 28 | 35 | 68 |
| Fail-to-pass test nodes | 26.7 | 27 | 23 | 31 | 67 |
| Pass-to-pass test nodes | 5.6 | 4 | 3 | 8 | 19 |
| Production files in golden patch | 97.5 | 60 | 37 | 133 | 524 |
| Production lines added | 4,590.3 | 2,955 | 1,687 | 5,645 | 26,605 |
| Ask rate | Targeting | Gain/Q | Follow-thr. | Redundancy | Unread | |
|---|---|---|---|---|---|---|
| Claude-Opus-5.5 ( CoWork ) | 80.4 | 93.3 | 0.64 | 78.3 | 13.7 | 0.2 |
| Spearman with F2P | +0.11 | +0.14 | +0.04 | +0.13 | +0.14 | +0.18 |
| Axis | Stratum | F2P (%) | Passed tasks | Disclosed (%) | |
|---|---|---|---|---|---|
| Verifier test nodes | 21 | 70.1 | 3 | 76.5 | |
| – | 55 | 62.5 | 2 | 61.3 | |
| 25 | 64.7 | 2 | 61.9 | ||
| Edit size | k production lines | 6 | 67.6 | 0 | 75.0 |
| k– k | 58 | 66.0 | 5 | 65.4 | |
| k | 37 | 62.0 | 2 | 61.7 |
| Median per run | Requirements obtained (%) | ||||||
|---|---|---|---|---|---|---|---|
| Model | Messages | Tool calls | Hours | Colleague | All | Run median | F2P (%) |
| GPT-6 Astra | 18.5 | 182 | 3.00 | 67.8 | 75.3 | 71.6 | 62.91 |
| Claude-Opus-5.5 | 22 | 213 | 5.01 | 64.2 | 70.7 | 72.0 | 63.84 |
| GLM 5.3 | 21 | 785 | 2.46 | 58.8 | 66.3 | 60.9 | 31.27 |
| Qwen3.8-Max | 6 | 696 | 8.00 | 46.9 | 56.6 | 48.7 | 35.70 |
| Seed Evolving | 9 | 503 | 4.46 | 44.3 | 50.6 | 38.1 | 39.09 |
| Dimension | Assessed | I | P | E | A |
|---|---|---|---|---|---|
| Search strategy | 101 | 0 | 6 | 90 | 5 |
| Decomposition/prioritization | 101 | 0 | 17 | 75 | 9 |
| Hypothesis testing | 101 | 0 | 2 | 33 | 66 |
| Information value | 101 | 0 | 3 | 85 | 13 |
| Abstraction/transfer | 101 | 0 | 2 | 57 | 42 |
| Causal debugging | 101 | 0 | 2 | 28 | 71 |
Software Engineering Agents
Software Engineering Agents are AI systems that use code, tools, and feedback to perform software development tasks.
See all 380 papers