cs.SEOct 10, 2026
SaveRe-executing Coding-Agent Evaluations: Unchanged Decisions and Unresolved Policy Risk
Organizations: Independent Researchers
Abstract
Executable tests grade large language model (LLM) coding-agent patches for leaderboards, rewards and self-improvement. Two preregistered re-execution studies returned negative results. SWE-bench Pro v1's offline option rejected none of 59 accepted Go patches whose grading configuration changed. Their exploratory, unweighted upper confidence limit is 4.95% under candidate independence, leaving a 3% exposed-patch risk unresolved; the preregistered estimator uses inverse sampling-probability weights. An extension was inconclusive. In a selected stratum, a reconstructed module cache built from the reference solution's dependencies rejected accepted alternatives from four models that passed with their own dependencies. Re-executing 1,929 selected agent-version/task evaluations from the Darwin Gödel Machine's archive preserved best-agent selection and system-variant ranking, with search history and scores of evaluations not rerun fixed. Exploratory host, image and adapted-grader comparisons also preserved them. Seven of 23 disagreements with archived verdicts did not recur on a second host; rebuilt images changed 16 three-run majority verdicts (0.83%) under four-CPU execution. We provide an evidence-to-claim summary, lessons for grader audits and an anonymous replication package. These results concern accepted patches and fixed-history decisions; general policy safety and adaptive-learning robustness require broader evaluations.
Figures & tables
| Quantity | Value | Threshold |
| Evaluable candidates (of 312) | 281 | |
| Models among them | 7 | |
| Tasks contributing them | 110 | |
| Verdict losses (0 of 3 restricted runs resolved) | 0 | |
| Confirmed reversible losses | 0 | |
| Weighted loss rate | 0.00% |
Table 1: Study 1 confirmatory result (offline policy P BLOCK , 120-task prefix). Rates are over evaluable candidates; the descriptive sensitivity row also counts the 11 unevaluable ones on eligible tasks.
Figure 1: Exposure and post-hoc bounds in Study 1. (a) The 281 evaluable accepted patches by task language; only the 59 Go patches (25 tasks) faced a configuration that P BLOCK changes. (b) One-sided 95% Clopper-Pearson upper confidence limit after zero rejections by the number of independent units. The two filled points are unweighted bounds for different estimands: the patch rejection rate (59 patches) and the share of tasks with a rejection (25 tasks). The dashed line marks the 3% of the hypothesis as a reference level for both estimands, and the orange hollow marker is where the bound first reaches it (99 units); the grey point is the hypothetical bound if all 91 Go tasks of the frame were eligible and none rejected.
Figure 2: Where re-executing DGM’s archive disagreed with its recorded verdicts. A cell is one archived evaluation (arm, agent version, task), re-run three times nominally (four CPUs) and three times under contention (a quarter of a CPU). Stable drift cells differ from the record in all six original-host runs. Classifications come from logs and test lists; the Django cause remains unisolated and its seven drift cells did not reproduce in the second execution. Annotations give the tasks and direction.
| Evidence | Supported scope and boundary |
|---|---|
| Study 1 | 0 losses among 59 exposed patches on 25 tasks. Unweighted post-hoc upper bound 4.95% under candidate independence; 3% exposed risk unresolved. |
| Study 2 | Best-node selection (D1) and arm ranking (D2) unchanged in the original replay and follow-ups. Original drift partly unreproduced; adaptive learning untested. |
| Extension | 2 rejecting tasks among 59 eligible. Control failed; descriptive only, inconclusive. |
| Selected gold cache | 6 of 15 accepted patches rejected; gold passed on 11 of 11 tasks. Existence only; no prevalence estimate. |
Table 2: Evidence-to-claim summary. Units, exposure and conditioning differ across studies; their tallies cannot be pooled into a common risk estimate.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Study and object | Unit (count) | Contrast, event and outcome |
|---|---|---|
| Study 1: SWE-bench Pro, 120-task prefix | accepted patch (281 evaluable; 59 exposed) | emulated networked contract vs. offline policy; confirmed reversible loss. KILL : 0 losses |
| Extension: remaining 63 Go tasks | task (59 eligible) | same arms; any failed restricted run of a reproduced candidate. Inconclusive : control failed; 2 of 59 |
| Study 2: DGM archive | selected evaluation cell (1,929, six runs each) | record vs. nominal and contended reruns; decision reversal. KILL : 0 reversals |
| Feasibility: 11 selected Go tasks | accepted patch (15) | emulated networked contract vs. gold-bound cache. Existence only: 6 rejected |
Table 3: The studies, their units and their outcomes. Only Study 1 and Study 2 test a preregistered hypothesis with a kill rule; the extension reports a descriptive tally after its control failure, and the feasibility stratum was selected. Eligible, evaluable and exposed are defined in Sections 3 and 4 .
Figure 3: Study 1 protocol. Each artefact runs under every arm three times, each run in a fresh network-isolated container from the pinned image; A0 and A2 add module mirrors for Go tasks only. A confirmed reversible loss counts only on an eligible task. A1 (A1 BLOCK ) is the offline policy P BLOCK ; Go tasks whose gold patch edits the dependency manifest or checksum file also run under the gold-bound cache P LEGO (A1 LEGO , drawn as A1-LEGO, dashed) for a descriptive analysis with eligibility under that policy. A0 is the baseline and A2 restores candidate-requested modules outside the task’s own module path. An eligible task passes the gold patch and fails the empty patch in all three restricted repetitions. A confirmed loss additionally requires a baseline-reproduced candidate with finished control and candidate cells, rejection in all restricted runs and acceptance in all restored runs, with the registered network-error signature. Negative controls are not shown.
| System | Tested pin | Upstream on 2026-10-07 | Coverage class and reason |
|---|---|---|---|
| SWE-bench Pro data (Hugging Face ScaleAI/SWE-bench_Pro ) | revision 2d52cb3d , configuration v1 : 731 tasks; parquet SHA-256 c8cd7115 , byte-identical to tag v1.0 ( 307bd006 ) | main is still 2d52cb3d ; at that revision the default config is V2 (642 tasks, parquet 1067f72c ), with hard and v1 as further configs | CURRENT for v1: the pin is the head and the v1 config is unchanged |
| SWE-bench Pro v1 evaluator ( scaleapi/SWE-bench_Pro-os ) | swe_bench_pro_eval.py , run scripts and parsers at 66f92766 ; option --block_network sets network mode none | head 66f92766 is release v2.0.0 (2026-09-22); its V2 README keeps the v1 evaluator and images “for reproducing v1 numbers” and does not use them for V2 | CURRENT : the pin is the head and the option is unchanged |
| SWE-bench Pro V2 (Harbor format) | not tested | v2.0.0 (2026-09-22), now the default config: 642 tasks, every task’s verifier with network mode public , new images on ghcr.io/scaleapi/swe-bench_pro-v2 ; it drops 13 of our 120 prefix tasks and 4 of the 63 extension tasks and keeps all 28 prefix Go tasks | “latest protocol keeps network access” (Section 2 ): CURRENT . Study 1 on V2: UNVERIFIED-CURRENT ; V2 has no offline verifier, and its images and Harbor verifier were not run |
| Task images ( jefzda/sweap-images ) | by digest: 212 in images.json (182 used in feasibility and the prefix), 63 in images_appendix_m0c.json | all 212 and all 63 Docker Hub tag digests unchanged; V2 does not use these images | CURRENT for v1 |
| Gold patch fails in every arm ( flipt-5aef5a14 , tutanota-1ff82aa3 ) | v1 data and images | V2 drops the Tutanota task; V2’s release gate reports the reference patch resolving 642 of 642 tasks, the Flipt task included | UNVERIFIED-CURRENT : every gold run logs network errors, so replaying without a network may cause it; not re-run on V2 |
| Offline option fails the gold patch ( vuls-cc63a0ec ; four extension tasks) | v1 evaluator, --block_network | all five tasks are in V2, whose verifier is networked; the v1 option is unchanged | CURRENT for the v1 option |
Table 4: Study 1: versions tested and upstream state, checked live on 2026-10-07 (15:54 to 16:02 UTC). Classes as defined in Appendix H .
| System | Tested pin | Upstream on 2026-10-07 | Coverage class and reason |
|---|---|---|---|
| DGM code ( jennyzzt/dgm ): test-change filter, staged gate | a565fd2d (2025-08-13) | head is a565fd2d ; no releases; no open issue or pull request title concerns the SWE-bench evaluation behind D1 and D2 (the nearest, issue #31 and its open fix #32, concern parent selection) | CURRENT : the pin is the head |
| DGM archive ( swe_results.zip , linked from DGM’s README) | Google Drive file of 1,010,489,591 bytes; SHA-256 of its zip listing 26580b4d | same size, last modified 2025-05-23, same listing hash | CURRENT : unchanged |
| SWE-bench grader | get_eval_report at dc4c087c (2024-10-25), the commit DGM’s README pins | PyPI swebench 5.0.2 (2026-08-18); newest tag v5.0.1 ( 87ab1f6c ); main 02e7a74f (2026-09-02); 33 commits to grading.py and 24 to log_parsers/ since the pin; the 5.0.0 changelog lists grading fixes | TESTED-PIN : Study 2 holds this grader fixed; UNVERIFIED-CURRENT under 5.0.2 and main. The grader-version follow-up regrades adapted archived logs with 5.0.2, not current-harness outputs |
| SWE-bench Verified reference test lists | SWE-bench/SWE-bench_Verified rev 78f471bf (parquet SHA-256 030cfd7f ) | head is 78f471bf (2026-08-16), same parquet; its commit 96dae8db (2026-08-10) dropped test_compose_roundtrip[] from Astropy 7606’s PASS_TO_PASS list. princeton-nlp/SWE-bench_Verified , which DGM’s code loads, is at c104f840 (2025-02-18) and keeps it. Issue #341 on this task was closed on 2025-03-02, cross-referenced by #338, before this change | Astropy 7606 label: FIXED-UPSTREAM at 96dae8db , in the SWE-bench copy only. Regrade and drift: TESTED-PIN grader |
| Study 2 images ( swebench/sweb.eval.x86_64.* ) | 200 images pinned by image ID, all Docker Hub tag v2 (pushed 2025-09-10/11) | latest re-pushed for all 200 on 2026-08-13 (163) and 2026-08-16 (37); SWE-bench 5.0.0’s changelog is dated 2026-08-17; the current recipe ( SWE-bench/swe-bench-tasks at 3d07b464 ) still commits the setup edits | setup-commit detail: CURRENT at the recipe level. Image-dependent results: UNVERIFIED-CURRENT at this check; the image-comparison follow-up tests the 2026-10-05 image snapshot |
| SWE-bench pull request 662 (Section 7 ) | capture of 2026-09-25: open, unmerged | still open and unmerged; one commit, head 2cdaaa60 (2026-09-18); body unchanged; last updated 2026-09-28; retitled by its author on 2026-09-18 “[don’t merge, see comment]” | CURRENT |
Table 5: Study 2, the host and the cited system papers: versions tested and upstream state, checked live on 2026-10-07 (15:54 to 16:02 UTC). Classes as defined in Appendix H .