cs.SEOct 14, 2026
SaveSWE-Nexus: Automated Composition of Software Engineering Tasks from Test Execution
Organizations: National University of Singapore, Singapore · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Independent Researchers · Tsinghua University, Beijing, China
Abstract
Historical code changes provide convenient task boundaries, but do not identify which production functions selected tests exercise or support composing validated behaviors beyond a single change. We present SWE-Nexus, a fully automated pipeline that constructs repository-level implementation tasks from test execution. Four construction agents select tests, adapt tracing and evaluation protocols to each repository, and generate behavioral specifications. For each selected test file, tracing records an observed execution footprint of production function bodies. A scope is retained only if all expected scoring points pass in the original repository and masking the traced bodies causes at least one failure or error. Compatible scopes are composed deterministically, with shared functions counted once and every constituent test file revalidated under the complete combined mask. The benchmark contains 600 tasks from 105 repositories across six programming languages, including 116 atomic and 484 composite tasks. We evaluate 12 models under three agent scaffolds. The best configuration resolves 37.8% of tasks in one attempt, and scope-weighted scoring measures partial recovery. SWE-Nexus enables task construction from validated behavior beyond individual historical changes.
Figures & tables
Figure 1: Task construction from historical PRs (a) and test execution (b). Edges show observed test–function associations. Deterministic composition counts shared functions once. Independent checks before and after composition require all expected points to pass originally ( ) and at least one failure or error per file under the complete mask ( ).
Figure 2: Statistics of the 600-task benchmark: (a) task composition and connectivity of composite tasks (connected or disconnected), (b) difficulty by language (100 tasks each), (c) repository coverage, (d) selected test-file counts, (e) changed lines in reference restoration patches (log scale), and (f) scoring points per task (10-point bins). Edges link test files that share observed production functions. Dashed lines mark medians.
Figure 3: Construction and evaluation in SWE-Nexus . The Selector, Trace, Validation, and Query Agents select tests, trace execution, adapt protocols, and generate specifications. Independent checks validate scopes before and after composition. The solver receives the specification and masked repository, and the evaluator restores hidden tests and applies the fixed protocol. Passing and gives a scope-weighted score of 50% and leaves the task unresolved.
Figure 4: Overall P@1 performance across 12 models and three agent scaffolds. Segments connect each model’s resolved rate (circle) and mean scope-weighted score (diamond). All panels share a 0–65% scale. Appendix D reports exact values.
| Model | Easy ( ) | Medium ( ) | Hard ( ) | Overall ( ) | ||||
| P@1 | P@4 | P@1 | P@4 | P@1 | P@4 | P@1 | P@4 | |
| Proprietary models | ||||||||
| Claude Opus 4.8 | 80.4 / 86.2 | 84.1 / 90.6 | 32.9 / 53.3 | 43.7 / 67.0 | 17.3 / 28.4 | 22.9 / 39.7 | 37.8 / 51.3 | 45.0 / 61.9 |
| Gemini 3.1 Pro | 56.5 / 70.4 | 64.5 / 79.5 | 14.3 / 46.5 | 23.4 / 60.5 | 2.2 / 25.7 | 8.2 / 37.4 | 19.3 / 44.0 | 27.0 / 55.9 |
| GPT-5.3 Codex | 60.9 / 78.7 | 71.7 / 86.2 | 29.0 / 60.2 | 39.8 / 72.4 | 10.0 / 32.1 | 13.9 / 41.2 | 29.0 / 53.7 | 37.2 / 63.6 |
| GPT-5.4 | 55.1 / 70.2 | 62.3 / 77.9 | 25.1 / 59.0 | 35.1 / 69.6 | 11.7 / 40.2 | 16.0 / 48.4 | 26.8 / 54.4 | 34.0 / 63.4 |
Table 1: Claude Code performance by task scope. Cells report resolved rate / scope-weighted score (%). P@1 and P@4 use one and four attempts. Bold marks metric maxima, including ties.
Figure 5: Performance by task scope and language. Panels (a,c) show P@1/P@4 means across 12 models under Claude Code. Panels (b,d) show P@1 means across models for each scaffold. Shading and ranges show the middle 50%. Dashed lines mark grand means. JS and TS denote JavaScript and TypeScript.
| Model | Go | Python | JavaScript | TypeScript | Java | Kotlin |
| Proprietary models | ||||||
| Claude Opus 4.8 | 34.0 / 47.1 | 40.0 / 52.0 | 30.0 / 53.7 | 41.0 / 52.5 | 51.0 / 64.5 | 31.0 / 38.0 |
| Gemini 3.1 Pro | 16.0 / 37.3 | 8.0 / 37.1 | 12.0 / 49.2 | 16.0 / 43.9 | 35.0 / 51.1 | 29.0 / 45.2 |
| GPT-5.3 Codex | 19.0 / 36.7 | 31.0 / 61.1 | 16.0 / 59.3 | 31.0 / 57.7 | 45.0 / 60.3 | 32.0 / 46.9 |
| GPT-5.4 | 14.0 / 36.7 | 27.0 / 61.6 | 19.0 / 63.1 | 26.0 / 57.6 | 44.0 / 57.8 | 31.0 / 49.5 |
| GPT-5.5 | 27.0 / 49.9 | 24.0 / 64.9 | 18.0 / 64.0 | 32.0 / 56.8 | 54.0 / 64.1 | 47.0 / 58.7 |
Table 2: Claude Code P@1 performance by language (100 tasks each). Cells report resolved rate / scope-weighted score (%). Bold marks maxima for each language, including ties.
Figure 6: Recorded activity in resolved and unresolved trajectories. Each point compares group medians for one of 126 configurations defined by model, scaffold, and language: (a) tool calls and (b) modified production files. Dashed diagonals mark equality.
Figure 7: A composite task for OpenAPI documentation generation in Kompendium, with an abridged description. Seven selected test files cover 94 production functions across 33 source files. Purple circles mark the 36 functions associated with multiple selected test files.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Language | Representative observation mechanism | Scope or mapping constraint |
|---|---|---|
| Python | Profiling callbacks during pytest execution, with source paths and qualified names. | A call-phase hook does not automatically cover setup, teardown, native extensions, or subprocesses. |
| Go | Function-entry instrumentation in source files supplied through a build overlay. | Package compilation and execution can exceed the selected physical test file. |
| JavaScript | Source instrumentation or test-runner transforms in the executing Node.js process. | Export-only instrumentation and worker coverage depend on the chosen backend. |
| TypeScript | Source-aware transforms integrated with Vite/Vitest or the repository runner. | Transpilation, expression bodies, and test ownership require explicit source mapping. |
| Java | Source probes or runtime bytecode instrumentation with test-framework hooks. | Constructors, generated methods, and thread boundaries affect observation and localization. |
| Kotlin | Source probes or runtime observation followed by mapping to Kotlin declarations. | Synthetic wrappers and declarations without independent source bodies require special handling. |
Table 3: Representative tracing mechanisms and coverage constraints. The mechanism is adapted to each repository’s test runner and source layout.
Figure 8: Specification generation and solver inputs. The Query Agent uses the original implementation and selected tests to describe behavior. The solver receives the specification and masked repository, which retains function declarations and signatures.
Figure 9: Atomic and composite tasks with their validated implementation scopes.
Figure 10: Recorded tool actions and outcomes from two trajectories. Visible checks guide repair, and independent evaluation determines complete and partial recovery.
| Language | Repository | Tier | Functions / lines | Calls | Resolved | Weighted | |
|---|---|---|---|---|---|---|---|
| Python | canonical/cloud-init | Easy | 2 | 22 / 216 | 47 | Yes | 100.0 |
| JavaScript | FlowFuse/flowfuse | Medium | 6 | 85 / 646 | 71 | No | 91.6 |
| TypeScript | minvws/NL COVID-19 Dashboard | Easy | 2 | 4 / 113 | 50 | Yes | 100.0 |
| Java | apache/avro | Hard | 7 | 632 / 5,198 | 89 | No | 0.0 |
| Kotlin | FasterXML/Jackson Kotlin | Medium | 6 | 49 / 553 | 192 | No | 90.4 |
Table 4: First-attempt Qwen3.5-Plus cases under Claude Code across five languages. counts selected test files. Functions / lines gives unique functions and recorded body spans. Weighted is the scope-weighted score (%).
| Model | Reference |
|---|---|
| Claude Opus 4.8 | ( Anthropic, 2026b ) |
| Gemini 3.1 Pro | ( Google DeepMind, 2026 ) |
| GPT-5.3 Codex | ( OpenAI, 2026a ) |
| GPT-5.4 | ( OpenAI, 2026b ) |
| GPT-5.5 | ( OpenAI, 2026c ) |
| Qwen3.5-Plus | ( Qwen Team, 2026 ) |
Table 7: Reference sources for the evaluated model variants.
| Setting | Value |
|---|---|
| Agent turns | 200 per scaffold invocation |
| Solver time limit | 6,600 seconds per attempt |
| Runtime timeout | 7,200 seconds |
| Output token limit | 32,000 per model request |
Table 8: Common execution limits across all evaluated models and agent scaffolds. These limits are fixed before solving.
| Model | Claude Code | Cline | OpenCode |
| Proprietary models | |||
| Claude Opus 4.8 | 37.8 / 51.3 | 26.5 / 43.5 | 29.7 / 53.9 |
| Gemini 3.1 Pro | 19.3 / 44.0 | 19.5 / 43.0 | 18.2 / 41.4 |
| GPT-5.3 Codex | 29.0 / 53.7 | 27.7 / 52.4 | 27.8 / 53.8 |
| GPT-5.4 | 26.8 / 54.4 | 31.2 / 55.4 | 23.8 / 51.3 |
| GPT-5.5 | 33.7 / 59.7 | 31.2 / 58.0 | 25.5 / 53.6 |
Table 9: Exact overall P@1 performance on SWE-Nexus (%). Cells report resolved rate / scope-weighted score. Bold marks metric maxima, including ties.
| Model | Go | Python | JavaScript | TypeScript | Java | Kotlin |
| Cline · Proprietary models | ||||||
| Claude Opus 4.8 | 28.0 / 39.4 | 20.0 / 47.2 | 21.0 / 38.9 | 22.0 / 41.2 | 40.0 / 56.2 | 28.0 / 38.2 |
| Gemini 3.1 Pro | 20.0 / 33.5 | 11.0 / 39.1 | 13.0 / 44.4 | 11.0 / 40.5 | 36.0 / 53.9 | 26.0 / 46.7 |
| GPT-5.3 Codex | 16.0 / 32.8 | 24.0 / 47.6 | 14.0 / 59.1 | 24.0 / 53.2 | 51.0 / 64.2 | 37.0 / 57.6 |
| GPT-5.4 | 20.0 / 36.4 | 21.0 / 56.3 | 25.0 / 64.8 | 24.0 / 53.9 | 47.0 / 58.3 | 50.0 / 62.4 |
| GPT-5.5 | 27.0 / 43.2 | 16.0 / 55.7 | 16.0 / 60.5 | 29.0 / 60.8 | 53.0 / 66.7 | 46.0 / 61.4 |
Table 10: Cline and OpenCode P@1 performance by language (100 tasks each). Cells report resolved rate / scope-weighted score (%). Bold marks maxima for each language and scaffold, including ties.
Figure 11: An adapted Go composite task from knative-extensions/eventing-kafka-broker .
Figure 12: An adapted Python composite task from encode/django-rest-framework .
Figure 13: An adapted JavaScript composite task from shipshapecode/shepherd .
Figure 14: An adapted TypeScript composite task from element-plus/element-plus .
Figure 15: An adapted Java composite task from apache/avro .
Figure 16: An adapted Kotlin composite task from adobe/S3Mock .
More in this topic
Software Engineering Benchmarks
Software engineering benchmarks evaluate how well AI systems understand, build, repair, test, and maintain software in realistic settings.
See all 156 papers