cs.CLMar 3, 2026
SaveBeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?
Abstract
Current code-agent benchmarks primarily evaluate localized issue resolution within a single target repository, leaving under-tested many software engineering tasks that require external knowledge or broader repository-level changes. We introduce BeyondSWE, a 500-instance benchmark drawn from 246 real-world GitHub repositories to evaluate code agents beyond single-repository bug fixing. BeyondSWE covers four representative settings: cross-repository issue resolution, domain-specific issue resolution, dependency-driven migration, and document-to-repository generation, spanning both broader knowledge scope and broader resolution scope. Our evaluation shows that BeyondSWE remains far from saturated: the best OpenHands-based agent reaches 46.12 average score, while the strongest Codex harness with GPT-5.4 (xhigh) reaches 56.65 under a search-aware prompt. To study whether external information access closes this gap, we use SearchSWE as a controlled diagnostic baseline for search-augmented coding. Search access improves most models and substantially helps some tasks, but the gains remain limited and uneven, showing that current agents still struggle to convert retrieved information into precise, version-compatible, and locally actionable code changes. These results suggest that deep search for coding remains an open problem: progress requires agents that can reliably combine external evidence with repository-local reasoning and execution-based verification.
Figures & tables
| Benchmark | Scope | Statistics | |||
| Resol. | Knowledge | #Repo | #Files | #Lines | |
| SWE-bench-Verified | Local Func | Within Repo | 12 | 1.3 | 11.6 |
| SWE-bench-Live | Local Func | Within Repo | 223 | 2.7 | 65.1 |
| SWE-bench Pro | Local Func | Within Repo | 41 | 4.1 | 107.4 |
| CrossRepo | Local Func | Cross Repo | 67 | 4.1 | 190.7 |
| DomainFix | Local Func | Domain | 12 | 4.2 | 157.6 |
Table 1: Comparison with existing SWE benchmarks
Figure 1: Overview of BeyondSWE. Our benchmark extends evaluation along two practical dimensions— knowledge scope and resolution scope : CrossRepo and DomainFix expand knowledge scope by requiring external software resources and domain expertise respectively; DepMigrate and Doc2Repo expand resolution scope from localized patches to codebase-wide transformations.
Figure 2: Environment Construction Pipeline for BeyondSWE. We collect task-specific candidates, build executable Docker environments through an agent-assisted setup process, and retain only instances whose tests exhibit stable fail-to-pass and pass-to-pass behavior across repeated runs.
Figure 3: Overview of SearchSWE as a controlled search-augmented coding baseline. The agent retains the local coding workflow, while adding web search and fetch tools under a blocklist that prevents direct access to solution-revealing target-repository artifacts.
| CrossRepo | DomainFix | DepMigrate | Doc2Repo | |||
| Model | %Resolved | %Resolved | %Resolved | Pass Rate | #(Alm.) Corr. | AVG |
| Codex | ||||||
| GPT-5.4 (xhigh) w/ SearchSWE Prompt | 55.17 +8.9 | 61.11 +19.4 | 48.59 +4.2 | 61.74 +0.1 | (7) / 2 | 56.65 +8.2 |
| GPT-5.4 (xhigh) w/ Default Prompt | 46.23 | 41.67 | 44.38 | 61.64 | (7) / 2 | 48.48 |
| OpenHands | ||||||
| DeepSeek-V4-Pro(Max) ( DeepSeek-AI, 2026 ) | 44.00 | 38.89 | 44.38 | 57.20 | (5) / 1 | 46.12 |
Table 2: Main Results on BeyondSWE. We organize the results by three evaluation settings: Codex, OpenHands, and SearchSWE. For Codex, we report both the default prompt setting and the SearchSWE-style prompt setting to highlight the sensitivity of frontier search-code agents to prompt design. Red / green values indicate gains/drops relative to the matched no-search setting: the default prompt for Codex and OpenHands for SearchSWE. Bold and underlined denote the best and second-best results across each evaluation settings.
Figure 4: Task-wise decomposition of SearchSWE outcomes relative to OpenHands. Each task row aggregates the eight matched model pairs. The Overall row aggregates all 32 task-model comparisons.
Figure 5: Average tool calls per instance for Codex with GPT-5.4 (xhigh) under the default prompt and the SearchSWE-style prompt. The search-oriented prompt substantially increases web-search use while keeping local coding-tool usage comparable.
Figure 6: Search effort versus SearchSWE gain on two contrasting tasks. Each point is an evaluated model with matched OpenHands and SearchSWE runs. The y-axis shows task-score of SearchSWE delta over OpenHands. Within each task, more search calls do not monotonically imply larger gains.
Figure 7: Cost and token-budget efficiency on BeyondSWE. Solid black lines denote the Pareto frontier among evaluated configurations. In the token plot, the dashed LOWESS curve summarizes the local performance trend: points below it are less token-productive than configurations with similar token budgets, and the high-token flattening shows that longer trajectories alone do not reliably improve performance.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Failure Mode | Instance | Search Behavior | Local Failure | Takeaway |
| Evidence availability | unidata_siphon_pr234 | Search returns high-level documentation instead of source-level backend logic | Agent implements a brittle interpretation of an ambiguous API description | Search must recover evidence at the right technical granularity |
| Version grounding | behave_behave-django_pr162 | Agent searches around an unsupported newer Django assumption instead of local constraints | Agent applies a method pattern incompatible with the repository’s legacy test lifecycle | External evidence must be filtered through local dependency versions |
| Evidence filtering | abravalheri_validate-pyproject_pr105 | Keyword-matched results drift into unrelated domains due to overloaded terminology | Agent falls back to a generic plugin-registration pattern that creates test side effects | Agents need semantic source discrimination, not only retrieval |
Table 3: Summary of search-augmented coding failure modes analyzed in the case studies.
| Component | Agent’s Intent / Target | Search Engine Response |
| Target Artifact | Backend Source Code | User-Facing Documentation |
| ( akrherz/iem/.../raob.py ) | ( .../json/raob.py?help ) | |
| Information Type | Precise Logic | Ambiguous Natural Language |
| Explicit conditionals, error handling, and parameter parsing logic. | High-level description: “approach this service… with just a timestamp .” | |
| Resulting Action | (Hypothetical) Implement robust parameter negotiation mirroring backend logic. | (Actual) Implemented a brittle solution based on literal interpretation of "just a timestamp". |
Table 4: Analysis of the Information Landscape Mismatch in unidata_siphon_pr234 . The search engine’s bias towards curated content obscures the precise logic required for robust code implementation.
| Context Source | Code Pattern & Logic | Status |
| Local Environment (Django 2.2/3.x) | Instance Method (Legacy) def _pre_setup(self): ... Constraint: Must maintain state on the test instance. | Ignored |
| Agent Search (Hallucinated "v5.2") | Class Method (Modern/Future) @classmethod def _pre_setup(cls): ... Bias: Assumes newer patterns apply universally. | Prioritized |
| Implementation (The Error) | Signature Mismatch Agent applies @classmethod to legacy hook. Result: Breaks MRO and instance state access. | ✗ |
Table 5: Analysis of Version Conflict in behave_behave-django_pr162 . The agent prioritizes hallucinated future specifications over explicit local constraints.
| Search Result Title | Domain | Snippet Content | Rel. |
| Families - repo-review 0.12.4.dev15 … | Target | “Families are a set of simple strings that group together similar checks…” | ✓ |
| Writing Family Instances with the Revit Writer | Architecture (BIM) | “…highlight some notable points on Revit family instances and the generic API…” | ✗ |
| RelativityOne - User Guide | Legal Tech | “…launch the Review interface from the Documents and Family card …” | ✗ |
| Maryland Workforce Innovation… | Gov. Policy | “…conducts an onsite review to examine all resource documents…” | ✗ |
Table 6: Analysis of Retrieval Noise in abravalheri_validate-pyproject_pr105 . The polysemy of the term "Family" causes the search engine to drift into unrelated technical domains.
| Research Field | Repositories |
| Scientific Computing | |
| Astronomy | astroplan |
| Bioinformatics | biotite, Biopython |
| Computational chemistry | cclib |
| Plasma physics | PlasmaPy |
| Quantum physics | qutip |
Table 7: Research fields and repositories included in the DomainFix task.
Figure 8: Distribution of lines of code across the 50 Doc2Repo repositories.
Figure 9: Lines of code for each Doc2Repo repository, sorted by size.
| Field | Description |
| instance_id | Unique identifier for the instance, typically in the format user_repo_prN . |
| dataset_id | Dataset identifier (e.g., realswe_bench ). |
| task | Task category: crossrepo , domainfix , depmigrate , or doc2repo . |
| user | GitHub organization or user name. |
| repo | Repository name. |
| language | Primary programming language of the repository. |
Table 8: Data fields for each instance in BeyondSWE.
| CrossRepo | DomainFix | DepMigrate | Doc2Repo | |||
| Model | %Resolved | %Resolved | %Resolved | Pass Rate | #(Alm.) Corr. | AVG |
| OpenHands | ||||||
| GLM-4.7 ( Z.ai Team, 2025 ) | 40.20 | 36.11 | 39.89 | 48.40 | (3) / 1 | 41.15 |
| DeepSeek-V3.2 ( Liu et al., 2025a ) | 38.00 | 30.56 | 36.52 | 54.99 | (3) / 0 | 40.02 |
| Kimi-K2 ( Team et al., 2025a ) | 37.00 | 27.78 | 39.53 | 54.91 | (6) / 2 | 39.81 |
| GPT-5.2 ( OpenAI, 2025b ) | 33.00 | 23.61 | 34.27 | 53.89 | (6) / 2 | 36.19 |
Table 9: Evaluation results of other models on BeyondSWE. This table includes BeyondSWE entries that are not reported in Table 2 . Bold and underlined values indicate the best and second-best results within each OpenHands/SearchSWE block, respectively. For SearchSWE rows, red / green deltas compare against the matched OpenHands result for the same model.
| Task | Candidate Source | Automatic Filtering | Human Verification | Final Instances |
| CrossRepo | Merged pull requests in Python-dominant repositories containing external links | Environment construction and stability inspection reduce about 3,000 candidates to about 800 executable candidates | Review external-link relevance and check that rewritten issue statements preserve task context without solution-specific details | 200 issues / 67 repos |
| DomainFix | Pull requests from expert-selected repositories across 11 scientific and engineering fields | Environment construction and stability inspection reduce about 800 candidates to about 200 executable candidates | Three domain experts independently verify environmental correctness, domain complexity, and solution non-triviality | 72 issues / 12 repos |
| DepMigrate | Pull requests mentioning 23 widely used packages and relevant version upgrades | LLM-based migration filtering plus environment construction reduce about 7,000 candidates to about 1,000 executable candidates | Four software engineering experts verify that each instance is a genuine dependency migration | 178 issues / 120 repos |
| Doc2Repo | Recently created Python repositories with continued activity, at least three contributors, and more than 20 stars | Repository masking, specification generation, test adaptation, and executable environment construction produce 60 candidates | Manual review selects high-quality specifications and checks alignment between adapted tests and documented behavior | 50 repos |
Table 10: Candidate collection, filtering, and verification pipeline for BeyondSWE.
| Task | Reviewers | Verification Criteria | Agreement / Retention Rule |
| CrossRepo | Senior software engineers and senior PhD auditors | External links are relevant to the issue; rewritten statements preserve task requirements; no solution-specific details are exposed; tests exhibit valid fail-to-pass behavior | Retain only instances passing manual relevance, leakage, and stability checks |
| DomainFix | Three domain experts, plus final senior PhD audit | Environment executes as expected; the issue requires domain-specific knowledge beyond general programming; the solution is non-trivial and cannot be inferred from error messages alone | Retain only instances accepted by all three domain experts |
| DepMigrate | Four software engineering experts, plus final senior PhD audit | The pull request corresponds to a real dependency migration; the upgraded dependency is installed in the evaluation environment; the required fix is not an incidental local patch | Retain only instances verified as genuine migration tasks |
| Doc2Repo | Senior software engineers and senior PhD auditors | Specification preserves public API and behavior while removing implementation details; adapted tests match the specification; stricter-than-source requirements are documented | Retain only specifications and tests judged aligned and executable |
| All tasks | Five senior software engineers and five senior PhD researchers | Environment construction, data cleaning, problem-statement quality, test validity, and final task-category fit | Reject instances with instability, solution leakage, underspecified behavior, or task-category mismatch |
Table 11: Human verification protocol for BeyondSWE.
Figure 10: Representative CrossRepo instance.
Figure 11: Representative DomainFix instance.
Figure 12: Representative DepMigrate instance.
Figure 13: Representative Doc2Repo instance.
| Prompt Component | Purpose | Key Constraints | Used For |
| Problem-statement generation | Convert raw pull requests into issue-style task descriptions | Remove solution-specific details while preserving reproduction steps, expected behavior, logs, and necessary context | CrossRepo, DomainFix, DepMigrate |
| SearchSWE system prompt | Define the shared code-agent workflow with search and local verification | Search only when local context is insufficient; fetch full pages before using results; prioritize local environment and user instructions | All SearchSWE runs |
| CrossRepo user prompt | Encourage use of external software artifacts when target-repository context is incomplete | Inspect related projects or documentation, but adapt evidence to the local repository and tests | CrossRepo |
| DomainFix user prompt | Guide agents to combine code reasoning with domain-specific knowledge | Verify scientific definitions, formulas, or domain conventions before applying a fix | DomainFix |
| DepMigrate user prompt | Frame the task as compatibility with upgraded dependencies | Do not downgrade dependencies; use migration guides or release notes only when compatible with installed versions | DepMigrate |
| Doc2Repo user prompt | Constrain clean-room repository construction from a specification | Treat the provided specification as authoritative; use search only for dependency syntax or missing imports, not to override the spec | Doc2Repo |
Table 12: Summary of prompt components used for benchmark construction and SearchSWE evaluation.