cs.SEApr 6, 2026

Edit, But Verify: An Empirical Audit of Instructed Code-Editing Benchmarks

Authors: Amir M. Ebrahimi, Gopi Krishnan Rajbahadur

Organizations: Queen’s University, Kingston, Canada

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Instructed code editing, where an LLM modifies existing code based on a natural language instruction, accounts for roughly 19% of real-world coding assistant interactions. Yet very few benchmarks directly evaluate this capability. From a survey of over 150 code-related benchmarks, we find that only two, CanItEdit and EDIT-Bench, target instructed code editing with human-authored instructions and test-based evaluation. We audit both by comparing their programming languages, edit intents, and application domains against distributions observed in the wild (Copilot Arena, AIDev, GitHub Octoverse), and by measuring test counts, statement coverage, and test scope across all 213 problems. Both benchmarks concentrate over 90% of evaluation on Python while TypeScript, GitHub's most-used language, is absent. Backend and frontend development, which together constitute 46% of real-world editing activity, are largely missing, and documentation, testing, and maintenance edits (31.4% of human PRs) have zero representation. Both benchmarks have modest test counts (CanItEdit median 13, EDIT-Bench median 4), though CanItEdit compensates with near-complete whole-file coverage and fail-before/pass-after validation. 59% of EDIT-Bench's low-coverage suites would not detect modifications outside the edit region. EDIT-Bench has 15 problems that are not solved by any of 40 LLMs and 11 of these problems trace failures to poor benchmark artifacts rather than model limitations. Further, 29% of EDIT-Bench problems and 6% of CanItEdit problems share a codebase with at least one other problem within the benchmark. In summary, these benchmarks measure a narrower construct than deployment decisions require. We therefore propose six empirically grounded desiderata and release all audit artifacts so the community can build instructed code-editing benchmarks whose scores reliably reflect real-world editing capability.

Figures & tables

Figure 1. Edit-intent distribution across benchmarks and real-world AI-assisted development (AIDev).
Figure 2. Application-domain distribution across benchmarks and real-world AI-assisted editing (Copilot Arena).
ML SubcategoryEDIT-Bench (39)CanItEdit (13)
 n%n%
Traditional ML Dev1128.21184.6
ML-Adjacent Tooling1846.217.7
AI-Native Application Dev1025.617.7
Table 1. Distribution of AI/ML problems by subcategory.
Figure 3. Number of tests per problem.
Figure 4. Test coverage distributions of EDIT-Bench and CanItEdit benchmarks.
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5. Prompt template for classifying benchmark tasks into Conventional Commit categories (RQ1).
Figure 6. Prompt template for classifying benchmark tasks into application domains (RQ1). Categories are aligned with those reported by Copilot Arena ( Chi et al., 2025a ) .
Figure 7. Prompt template for classifying AI/ML problems into subcategories (RQ1).
Figure 8. Prompt template for assessing test suite adequacy against edit intent (RQ2).
ProblemWhat the edit changesWhy the tests are inadequateConsequence
104Fixes add_edge logic, re-indents remove_vertex and remove_edge , and corrects print-string formatting.Tests detect failures only because the original file contains a syntax/import error, not because they verify the repaired logic. None of the fixed methods are exercised.Suite confirms the original is malformed but provides no confidence the fix is functionally correct.
29Modifies pandas column-processing logic in visualize_results_generic .The test intended to call the function with sample DataFrames is overwritten by a second test with the same name, leaving only a shallow existence check.Modified column-handling code is never validated.
52Rewrites DataFrame construction from scalar-based to list-per-column format.Tests inspect source text and reconstruct a DataFrame in the harness rather than executing the implementation. Any non-list-of-dicts pattern is accepted as corrected.Original broken version can still appear to pass; changed code is never executed.
87Adds dynamic square colorization and a boundary check preventing squares from disappearing before the canvas is drawn.One test checks for a fill parameter, but the original constant blue already satisfies it. The movement test covers only simple in-bounds moves and never triggers the new boundary logic.Neither new behavior (dynamic color, boundary check) is exercised by the suite.
Appendix
Table 2. Manual analysis of four EDIT-Bench problems identified as having genuinely inadequate test suites (diff-region coverage below 75%).
IDPrimary CauseSummary
Benchmark Artifacts
53Test UnfairnessTests use brittle string-matching on source code rather than validating behavior. A model implementing Python’s recommended logger.error(..., exc_info=True) is rejected because the source does not contain the exact string traceback.format_exc() .
30Invalid Test Logictest_basic_functionality contains mathematical errors (e.g., asserts 5=12+12+12+225=1^{2}+1^{2}+1^{2}+2^{2} when the actual sum is 7). Seven models correctly return False for these impossible cases but are penalized.
8Test Bug + UnderspecificationInstruction is a title fragment with no actionable guidance. Tests require module importability (unstated) and contain a mathematical error rejecting the valid decomposition 4=12+12+12+124=1^{2}+1^{2}+1^{2}+1^{2} .
126Fragile MockingAll 12 models correctly implement dictionary-based clustering, but extremely complex mocking ( patch.dict , nested try/except , patch.object ) breaks with minor implementation variations. Models score 0.167–0.333 despite correct behavior.
39Infrastructure BugTest harness applies dependency mocks after module import has already failed. ModuleNotFoundError fires before any assertion executes; all 12 models score 0.0 across all 5 language variants.
Appendix
Table 3. Classification of universally unsolved EDIT-Bench problems.
More in this topic

Software Engineering Benchmarks

Software engineering benchmarks evaluate how well AI systems understand, build, repair, test, and maintain software in realistic settings.

See all 155 papers

Explore similar work

CardsList
Apr 28, 2026

SAFEdit: Does Multi-Agent Decomposition Resolve the Reliability Challenges of Instructed Code Editing?

Instructed code editing is a significant challenge for large language models (LLMs). On the EditBench benchmark, 39 of 40 evaluated models obtain a task success rate (TSR) below 60 percent, highlighting a gap between general code generation and the ability to perform instruction-driven editing under executable test constraints. To address this, we propose SAFEdit, a multi-agent framework for instructed code editing that decomposes the editing process into specialized roles to improve reliability and reduce unintended code changes. A Planner Agent produces an explicit, visibility-aware edit plan, an Editor Agent applies minimal, literal code modifications, and a Verifier Agent executes real test runs. When tests fail, SAFEdit uses a Failure Abstraction Layer (FAL) to transform raw test logs into structured diagnostic feedback, which is fed back to the Editor to support iterative refinement. We compare SAFEdit against both prior single-model results reported for EditBench and an implemented ReAct single-agent baseline under the same evaluation conditions. We used EditBench to evaluate SAFEdit on 445 code editing instances in five languages (English, Polish, Spanish, Chinese, and Russian) under varying spatial context variants. SAFEdit achieved 68.6 percent TSR, outperforming the single-model baseline by 3.8 percentage points and the ReAct single-agent baseline by 8.6 percentage points. The iterative refinement loop was found to contribute 17.4 percentage points to SAFEdit's overall success rate. SAFEdit's automated error analysis further indicates a reduction in instruction-level hallucinations compared to single-agent approaches, providing an additional framework component for interpreting failures beyond pass or fail outcomes.

Noam Tarshish, Nofar Selouk, Daniel Hodisan +4

Ben-Gurion University of the Negev · Beer Sheva, Israel

Execution-Guided Code GenerationMulti-Agent CodingLLM Agent Reliability
AskPDF
Jul 27, 2026

Learning from 53.6K Real-World Developer Edits of AI-Generated Code

Imperfections in AI-generated code require that software developers modify the generated code manually, or by re-prompting an AI programming assistant. Manual code edits provide more realistic and granular information on editing behavior than Git commits, which only contain final successful code snippets. Yet, due to a lack of high-quality, realistic code editing data, LLMs are mostly trained on publicly available Git data (e.g., commits). To address this gap, we introduce DECODE (Developer Edits of Code Dataset), a dataset of 53.6K real-world in-IDE code edits of AI-generated code in Python, TypeScript, and JavaScript, sourced from 1K+ developers. First, we demonstrate the utility of DECODE for data analysis, obtaining insights on when, why, and how AI-generated code is edited. We find that most edits occur within the first 15 minutes after accepting an AI completion, resulting in the removal of AI completions in 31% of edit trajectories. Second, we use DECODE to benchmark the ability of LLMs to predict code edits. We find that finetuning on DECODE enables open-source 3B models to perform code edit prediction tasks significantly better than frontier LLMs. We then discuss implications of this work, emphasizing the necessity of developer-centric machine learning approaches for future AI programming assistants.

Jenny T. Liang, Mihika Bairathi, Wayne Chi +3

Carnegie Mellon University

Code Generation EvaluationCode Language ModelsAI-Assisted Software Development
AskPDF
Sep 3, 2026

When Models Edit Too Much: On the Fidelity of Minimal Code Edits

Large language models (LLMs) are increasingly used to edit existing code, but correctness alone is not enough: useful repairs should also be minimal, reviewable, and faithful to the original implementation. We study over-editing, the tendency of a model to rewrite code beyond what is required to fix a bug. We construct an evaluation framework from 400 BigCodeBench problems by injecting controlled AST-level corruptions into reference solutions, giving each repair task a known minimal patch. Across frontier LLMs, over-editing is widespread even among strong models like GPT-5.5: high Pass@1 can coexist with unnecessarily large edits and added cognitive complexity. A preservation instruction substantially reduces this behavior, lowering average excess Levenshtein distance from 0.195 to 0.131, reducing added cognitive complexity by 26.6%, and increasing Pass@1 by 2.3 points. However, these gains do not simply follow from a larger reasoning budget or larger models. We next ask whether minimal editing can be learned directly during post-training. We observe that supervised fine-tuning overfits to seen corruption patterns, whereas reinforcement learning gives the best out-of-domain edit-fidelity and performance-retention trade-off. These results position edit fidelity as a distinct axis of code-repair quality and show that it can be measured and learned.

Tongyao Zhu, Wei Hern Lim, Min-Yen Kan

National University of Singapore

Automated Program RepairLanguage Model Editing
AskPDF