Edit, But Verify: An Empirical Audit of Instructed Code-Editing Benchmarks

Amir M. Ebrahimi; Gopi Krishnan Rajbahadur

cs.SEApr 6, 2026

Edit, But Verify: An Empirical Audit of Instructed Code-Editing Benchmarks

Authors: Amir M. Ebrahimi, Gopi Krishnan Rajbahadur

Organizations: Queen’s University, Kingston, Canada

Abstract

Instructed code editing, where an LLM modifies existing code based on a natural language instruction, accounts for roughly 19% of real-world coding assistant interactions. Yet very few benchmarks directly evaluate this capability. From a survey of over 150 code-related benchmarks, we find that only two, CanItEdit and EDIT-Bench, target instructed code editing with human-authored instructions and test-based evaluation. We audit both by comparing their programming languages, edit intents, and application domains against distributions observed in the wild (Copilot Arena, AIDev, GitHub Octoverse), and by measuring test counts, statement coverage, and test scope across all 213 problems. Both benchmarks concentrate over 90% of evaluation on Python while TypeScript, GitHub's most-used language, is absent. Backend and frontend development, which together constitute 46% of real-world editing activity, are largely missing, and documentation, testing, and maintenance edits (31.4% of human PRs) have zero representation. Both benchmarks have modest test counts (CanItEdit median 13, EDIT-Bench median 4), though CanItEdit compensates with near-complete whole-file coverage and fail-before/pass-after validation. 59% of EDIT-Bench's low-coverage suites would not detect modifications outside the edit region. EDIT-Bench has 15 problems that are not solved by any of 40 LLMs and 11 of these problems trace failures to poor benchmark artifacts rather than model limitations. Further, 29% of EDIT-Bench problems and 6% of CanItEdit problems share a codebase with at least one other problem within the benchmark. In summary, these benchmarks measure a narrower construct than deployment decisions require. We therefore propose six empirically grounded desiderata and release all audit artifacts so the community can build instructed code-editing benchmarks whose scores reliably reflect real-world editing capability.

Figures & tables

Figure 1. Edit-intent distribution across benchmarks and real-world AI-assisted development (AIDev).

Figure 2. Application-domain distribution across benchmarks and real-world AI-assisted editing (Copilot Arena).

ML Subcategory	EDIT-Bench (39)		CanItEdit (13)
	n	%	n	%
Traditional ML Dev	11	28.2	11	84.6
ML-Adjacent Tooling	18	46.2	1	7.7
AI-Native Application Dev	10	25.6	1	7.7

Table 1. Distribution of AI/ML problems by subcategory.

Figure 3. Number of tests per problem.

Figure 4. Test coverage distributions of EDIT-Bench and CanItEdit benchmarks.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5. Prompt template for classifying benchmark tasks into Conventional Commit categories (RQ1).

Panel of Figure 6 — Figure 6. Prompt template for classifying benchmark tasks into application domains (RQ1). Categories are aligned with those reported by Copilot Arena ( Chi et al., 2025a ) .

Panel of Figure 7 — Figure 7. Prompt template for classifying AI/ML problems into subcategories (RQ1).

Panel of Figure 8 — Figure 8. Prompt template for assessing test suite adequacy against edit intent (RQ2).

Problem	What the edit changes	Why the tests are inadequate	Consequence
104	Fixes add_edge logic, re-indents remove_vertex and remove_edge , and corrects print-string formatting.	Tests detect failures only because the original file contains a syntax/import error, not because they verify the repaired logic. None of the fixed methods are exercised.	Suite confirms the original is malformed but provides no confidence the fix is functionally correct.
29	Modifies pandas column-processing logic in visualize_results_generic .	The test intended to call the function with sample DataFrames is overwritten by a second test with the same name, leaving only a shallow existence check.	Modified column-handling code is never validated.
52	Rewrites DataFrame construction from scalar-based to list-per-column format.	Tests inspect source text and reconstruct a DataFrame in the harness rather than executing the implementation. Any non-list-of-dicts pattern is accepted as corrected.	Original broken version can still appear to pass; changed code is never executed.
87	Adds dynamic square colorization and a boundary check preventing squares from disappearing before the canvas is drawn.	One test checks for a fill parameter, but the original constant blue already satisfies it. The movement test covers only simple in-bounds moves and never triggers the new boundary logic.	Neither new behavior (dynamic color, boundary check) is exercised by the suite.

Appendix

Table 2. Manual analysis of four EDIT-Bench problems identified as having genuinely inadequate test suites (diff-region coverage below 75%).

ID	Primary Cause	Summary
Benchmark Artifacts
53	Test Unfairness	Tests use brittle string-matching on source code rather than validating behavior. A model implementing Python’s recommended logger.error(..., exc_info=True) is rejected because the source does not contain the exact string traceback.format_exc() .
30	Invalid Test Logic	test_basic_functionality contains mathematical errors (e.g., asserts $5=1^{2}+1^{2}+1^{2}+2^{2}$ when the actual sum is 7). Seven models correctly return False for these impossible cases but are penalized.
8	Test Bug + Underspecification	Instruction is a title fragment with no actionable guidance. Tests require module importability (unstated) and contain a mathematical error rejecting the valid decomposition $4=1^{2}+1^{2}+1^{2}+1^{2}$ .
126	Fragile Mocking	All 12 models correctly implement dictionary-based clustering, but extremely complex mocking ( patch.dict , nested try/except , patch.object ) breaks with minor implementation variations. Models score 0.167–0.333 despite correct behavior.
39	Infrastructure Bug	Test harness applies dependency mocks after module import has already failed. ModuleNotFoundError fires before any assertion executes; all 12 models score 0.0 across all 5 language variants.

Appendix

Table 3. Classification of universally unsolved EDIT-Bench problems.

Software Engineering Benchmarks

Software engineering benchmarks evaluate how well AI systems understand, build, repair, test, and maintain software in realistic settings.

See all 155 papers

SAFEdit: Does Multi-Agent Decomposition Resolve the Reliability Challenges of Instructed Code Editing?
Apr 28, 2026Noam Tarshish, Nofar Selouk, Daniel Hodisan +4Execution-Guided Code Generation Multi-Agent Coding
Ben-Gurion University of the Negev · Beer Sheva, Israel
Learning from 53.6K Real-World Developer Edits of AI-Generated Code
Jul 27, 2026Jenny T. Liang, Mihika Bairathi, Wayne Chi +3Code Generation Evaluation Code Language Models
Carnegie Mellon University
When Models Edit Too Much: On the Fidelity of Minimal Code Edits
Sep 3, 2026Tongyao Zhu, Wei Hern Lim, Min-Yen KanAutomated Program Repair Language Model Editing
National University of Singapore

Apr 28, 2026J/K move · Enter open · S save

SAFEdit: Does Multi-Agent Decomposition Resolve the Reliability Challenges of Instructed Code Editing?

Noam Tarshish, Nofar Selouk, Daniel Hodisan, Bar Ezra Gafniel+3

Ben-Gurion University of the Negev · Beer Sheva, Israel

Execution-Guided Code Generation Multi-Agent Coding LLM Agent Reliability

Open paper AskPDF arXiv

Instructed code editing is a significant challenge for large language models (LLMs). On the EditBench benchmark, 39 of 40 evaluated models obtain a task success rate (TSR) below 60 percent, highlighting a gap between general code generation and the ability to perform instruction-driven editing under executable test constraints. To address this, we propose SAFEdit, a multi-agent framework for instructed code editing that decomposes the editing process into specialized roles to improve reliability and reduce unintended code changes. A Planner Agent produces an explicit, visibility-aware edit plan, an Editor Agent applies minimal, literal code modifications, and a Verifier Agent executes real test runs. When tests fail, SAFEdit uses a Failure Abstraction Layer (FAL) to transform raw test logs into structured diagnostic feedback, which is fed back to the Editor to support iterative refinement. We compare SAFEdit against both prior single-model results reported for EditBench and an implemented ReAct single-agent baseline under the same evaluation conditions. We used EditBench to evaluate SAFEdit on 445 code editing instances in five languages (English, Polish, Spanish, Chinese, and Russian) under varying spatial context variants. SAFEdit achieved 68.6 percent TSR, outperforming the single-model baseline by 3.8 percentage points and the ReAct single-agent baseline by 8.6 percentage points. The iterative refinement loop was found to contribute 17.4 percentage points to SAFEdit's overall success rate. SAFEdit's automated error analysis further indicates a reduction in instruction-level hallucinations compared to single-agent approaches, providing an additional framework component for interpreting failures beyond pass or fail outcomes.

Abstract

Software Engineering Benchmarks

Explore similar work

SAFEdit: Does Multi-Agent Decomposition Resolve the Reliability Challenges of Instructed Code Editing?

Learning from 53.6K Real-World Developer Edits of AI-Generated Code

When Models Edit Too Much: On the Fidelity of Minimal Code Edits