How Should Diffusion Language Models Edit Code?
Organizations: The University of Hong Kong · Huawei Technologies Ltd. · City University of Hong Kong · Peking University
Abstract
Code editing requires a model to decide where to make changes, generate the new content, and preserve everything else. We study how masked diffusion language models divide these responsibilities across four editing interfaces: whole-file rewriting, search-and-replace, locate-then-infill, and token-level editing. Experiments on CanItEdit reveal a composition gap: diffusion models can generate coordinated changes when the correct edit locations are supplied, but much of this capability is lost when those locations must be predicted. Access to the intact original code helps the model fill multiple edit regions, yet does not resolve the difficulty of selecting those regions. By varying the editable regions while holding the generation model and decoding procedure fixed, we identify two distinct requirements for successful editing: covering every required change and placing precise boundaries around it. Missing a required region prevents the corresponding change, while widening regions to ensure coverage can sharply reduce success by requiring unchanged code to be regenerated. A sentence-level Wiki editing probe shows the same qualitative gap between supplied and predicted locations beyond code. These findings show why strong infilling capability alone does not ensure reliable editing: the interface must expose all required changes while limiting regeneration of unchanged code.
Figures & tables
| Interface / structure | Overall | Single | Multi | SynErr | Exc. |
|---|---|---|---|---|---|
| Autonomous DiffuCoder-family editors | |||||
| Search-and-replace ( str_replace ) | 23.3 | 48.5 | 11.3 | 5.7 | 68.6 |
| Locate-then-infill | 12.9 | 27.9 | 5.6 | 15.2 | 70.0 |
| Rewrite | 11.4 | 27.9 | 3.5 | 23.8 | 63.3 |
| Token-level editing | 5.2 | — | — | 50.0 | — |
| Structure and conditioning diagnostics | |||||
| Scaffold | Recall (%) | pass@1 (%) |
|---|---|---|
| Clean gold | 100 | 25.4 |
| Drop one required span | 66.0 | 6.3 |
| Expand each range | 100 | 7.7 |
| Add one false-positive hole | 100 | 20.4 |
| Add two false-positive holes | 100 | 8.5 |
| Interface / locations | Exact | SARI | ROUGE-L | BLEU | Edit F1 | Coverage | Preserved |
|---|---|---|---|---|---|---|---|
| str_replace | 0.0 | 0.411 | 0.882 | 0.795 | 0.061 | 6.8 | 97.5 |
| Predicted locations infill | 3.1 | 0.437 | 0.807 | 0.728 | 0.376 | 45.3 | 82.3 |
| Gold locations infill | 9.4 | 0.597 | 0.808 | 0.755 | 0.927 | 89.3 | 95.0 |
| Interface | Text generated | Structure predicted |
|---|---|---|
| Rewrite | Implicit in full output | |
| Search-and-replace | Exact source anchors | |
| Locate-then-infill | Regions and per-hole length | |
| Token-level editing | Changed content | Operations across source tokens |
Appendix figures & tables50 assets
Supplementary material from the paper’s appendix.
Appendix
| Statistic | Mean | Median | Max |
| Source tokens | 174.6 | 174 | 546 |
| Target tokens | 204.1 | 204 | 730 |
| Instruction tokens | 28.0 | 27 | 118 |
| Edit distance (tokens) | 49.5 | 26 | 722 |
| Edit ratio ( ) | 0.23 | 0.16 | 1.00 |
| Field | Mean | Std | Min | P25 | Med | P75 | P90 | P95 | Max |
|---|---|---|---|---|---|---|---|---|---|
| Source tokens | 174.6 | 84.3 | 4 | 107 | 174 | 242 | 288 | 307 | 546 |
| Target tokens | 204.1 | 92.0 | 6 | 133 | 204 | 273 | 320 | 346 | 730 |
| Instruction tokens | 28.0 | 3.9 | 23 | 25 | 27 | 29 | 32 | 35 | 118 |
| Prompt + Source | 202.6 | 84.5 | 28 | 135 | 202 | 270 | 316 | 336 | 577 |
| Prompt + Target | 232.1 | 92.2 | 32 | 161 | 232 | 301 | 348 | 374 | 755 |
| Field | Mean | Std | Min | Med | P75 | P95 | Max |
|---|---|---|---|---|---|---|---|
| Source chars | 748.0 | 383.1 | 10 | 733 | 1033 | 1384 | 2937 |
| Target chars | 874.0 | 414.3 | 21 | 863 | 1175 | 1544 | 2947 |
| Source lines | 27.2 | 13.0 | 1 | 27 | 36 | 49 | 94 |
| Target lines | 31.1 | 14.0 | 1 | 31 | 41 | 55 | 164 |
| Instruction chars | 44.9 | 17.7 | 16 | 41 | 51 | 74 | 444 |
| Operation | Mean | Std | Min | Med | P75 | P95 | Max |
| keep tokens | 161.5 | 81.7 | 1 | 160 | 227 | 293 | 379 |
| insert tokens | 36.4 | 57.9 | 0 | 15 | 44 | 148 | 689 |
| delete tokens | 6.9 | 22.6 | 0 | 0 | 4 | 35 | 525 |
| edit tokens | 6.3 | 14.6 | 0 | 2 | 6 | 28 | 279 |
| Edit distance | 49.5 | 63.1 | 1 | 26 | 61 | 175 | 722 |
| Edit ratio | 0.23 | 0.21 | 0.003 | 0.16 | 0.33 | 0.70 | 1.00 |
| Change Type | Count | % |
|---|---|---|
| Mixed (insert+delete+edit) | 14,362 | 69.7 |
| Insert only | 4,984 | 24.2 |
| Delete only | 993 | 4.8 |
| Edit only (substitution) | 263 | 1.3 |
| Has insertions | 17,426 | 84.6 |
| Has deletions | 7,603 | 36.9 |
| Statistic | Mean | Std | Min | Med | P75 | P95 | Max |
|---|---|---|---|---|---|---|---|
| Groups per example | 2.3 | 2.5 | 0 | 2 | 3 | 7 | 31 |
| Tokens per group | 15.6 | 30.2 | 1 | 6 | 15 | 65 | 611 |
| # | Family | Supervised state |
|---|---|---|
| 1 | Fill pair | Fully expanded [MASK] slots; parallel fill; keep/edit/delete on source. |
| 2 | Expand trigger | Pre-mask source; <|expand|> on the last source token before each insert group. |
| 3 | Intermediate expand | Random partition of each group; every slot expand vs. fill; doubling. |
| 4 | Over-expand recovery | slots; extra slot labelled EOS . |
| 5 | Partial-fill | Some insert slots already committed; remaining masks filled. |
| 6 | Spurious-expand | Bogus masks on non-insert positions, labelled EOS . |
| Interface | Length unit | Other |
|---|---|---|
| Stage-1 plans | block-decision, size 32 | confidence ; numbered old code in the best runs |
| Stage-2 fill | block-decision, size 16 | confidence ; joint denoising of all holes |
| Rewrite SFT | fixed 256 or 512 steps | full-file generation; no Stage-1 plan |
| str_replace (headline) | block-append, step 1685 | serialized ; runtime apply-in-order |
| str_replace (budget) | 256-slot, step 1515 | same interface, smaller generation budget |
| str_replace (pad) | random-pad, step 1685 | same recipe as headline except padding |
| Role | Step | Init / path |
|---|---|---|
| Rewrite SFT | 640 | DiffuCoder-Instruct; ml-diffucoder/checkpoints/sft_editpackft_baseline_rewrite_8gpu |
| str_replace SFT | 1685 | DiffuCoder-Instruct; checkpoints/str_replace_sft_block (block-append) |
| Shared Stage-1/2 multitask | 1935 | oracle Stage-2 specialist 1610; checkpoints/stage23_multitask_sft_from_stage3_step1610 |
| Oracle Stage-2 specialist | 1610 | DiffuCoder-Instruct; checkpoints/stage3_oracle_scaffold_diffucoder_block_decision_ep5 |
| Attempt | Result | Lesson |
|---|---|---|
| Token-level Levenshtein editing | Oracle masks 30–32.5% exact match; model-chosen insert points 0%. | Bottleneck is where to insert, not fill given where. |
| Header reweight, skeleton weight, header-only targets, multi-hunk oversampling | All failed location gates (exact line-op 25%; mean edits toward gold 2.78). Header-only raised count but produced out-of-bounds locations. | Upweighting format tokens is not a substitute for learning the discrete structure those tokens encode. |
| Frozen-backbone dense ScaffoldHead | CanItEdit exact line-op 0.48%; in-domain exact event-tape 6.6%. Stopped: no fill, no unfreeze. | Coarse metrics (nearby anchors, non-zero edits) can look calibrated while the scaffold remains unusable. |
| OpenCoder 50/50 Stage-2 mix; hole markers; per-token DreamOn expand | Gold-loc / Qwen 38.1% vs. ep5 39.5%. Markers swapped EOS-error types; DreamOn SyntaxError 38%. | More generic code did not raise the fill ceiling. Block-decision was the production length harness because it avoided the worst syntax collapse. |
| Round-consistent / block-state-balanced Stage-2 sampler | Exact realized length ; under-generation ; pass@1 ; multi-span . Long 1-hole ( – / ) unchanged. | Matching gold trajectory states more closely improved length statistics, not functional editing (Table 15 ). |
| Stage-2 training | pass@1 | SynErr | 1-span | Multi | Exact len | Under-gen |
|---|---|---|---|---|---|---|
| Aligned old-visible | 27.1 | 27.1 | 30.9 | 25.4 | 16.4 | 71.1 |
| Round-consistent | 23.8 | 29.0 | 33.8 | 19.0 | 21.5 | 65.9 |
| Setting | pass@1 | SynErr | 1-hole | multi | 2 | 3 | |
|---|---|---|---|---|---|---|---|
| Unlocked | 27.1 | 27.1 | 30.9 / 35.3 | 25.4 / 23.2 | 44.4 / 11.1 | 7.1 / 28.6 | 16.7 / 31.7 |
| forced occupancy | 28.1 | 42.4 | 47.1 / 32.4 | 19.0 / 47.2 | 33.3 / 35.2 | 21.4 / 60.7 | 5.0 / 51.7 |
| 27.6 | 46.2 | 42.6 / 29.4 | 20.4 / 54.2 | 40.7 / 37.0 | 17.9 / 60.7 | 3.3 / 66.7 | |
| 24.8 | 50.5 | 41.2 / 38.2 | 16.9 / 56.3 | 35.2 / 38.9 | 14.3 / 60.7 | 1.7 / 70.0 | |
| 14.8 | 56.7 | 26.5 / 44.1 | 9.2 / 62.7 | 20.4 / 48.1 | 7.1 / 60.7 | 0.0 / 76.7 | |
| 4.3 | 71.0 | 1.5 / 69.1 | 5.6 / 71.8 | 11.1 / 70.4 | 3.6 / 67.9 | 1.7 / 75.0 |
| unlocked / | Pass | SynErr | Other |
|---|---|---|---|
| Pass | 38 | 15 | 4 |
| SynErr | 12 | 30 | 15 |
| Other | 9 | 44 | 43 |
| Holes | changed | Pass SynErr | SynErr Pass | |
|---|---|---|---|---|
| 1 | 68 | 28 | 0 | 9 |
| 2 | 54 | 21 | 9 | 3 |
| 3 | 28 | 17 | 0 | 0 |
| 60 | 33 | 6 | 0 |
| Slice | Max gold hole | pass@1 | SynErr | |
| 1-hole | 1–16 | 14 | 57.1 | 14.3 |
| 1-hole | 17–32 | 22 | 54.5 | 18.2 |
| 1-hole | 33–64 | 12 | 8.3 | 33.3 |
| 1-hole | 20 | 0.0 | 70.0 | |
| All examples | 74 | 6.8 | 47.3 | |
| Total gold tokens | — | 60 | 8.3 | 45.0 |
| Split | cont. acc. | stop acc. | all holes | mean | |
|---|---|---|---|---|---|
| Overall | 210 | 57.3 | 73.3 | 50 | 0.281 |
| 1-hole | 68 | 32.4 | 76.5 | 25 | 0.247 |
| 2-hole | 54 | 69.9 | 71.3 | 16 | 0.308 |
| 3-hole | 28 | 51.5 | 70.2 | 5 | 0.312 |
| 60 | 65.0 | 74.0 | 4 | 0.270 | |
| Multi SynErr | 67 | 74.0 | 63.3 | 5 | 0.375 |
| Placement | Layout | exact-match | pass@1 |
|---|---|---|---|
| Middle | prefix [MASK] suffix | 79.2 | 93.6 |
| End-prompted | chat(PREFIX, SUFFIX) then [MASK] | 31.9 | 17.7 |
| Evaluation | Middle | End | |
|---|---|---|---|
| Off-the-shelf single_line pass@1 | 93.6 | 17.7 | |
| Matched-SFT single_line pass@1 | 94.6 | 94.3 | |
| exact match | 84.2 | 82.5 | |
| Matched-SFT multi_line pass@1 | 61.9 | 55.2 | |
| exact match | 37.3 | 33.1 |
| Evaluation | Metric | Middle | End |
|---|---|---|---|
| EditPackFT | exact match | 19.2 | 19.8 |
| syntax-ok | 97.0 | 96.8 | |
| mean token distance | 12.2 | 12.2 | |
| CanItEdit single-span | pass@1 | 42.4 | 47.0 |
| string exact match | 24.2 | 27.3 | |
| SyntaxError | 28.8 | 25.8 |
| Task | Output tokens | M2T | Replace | Delete | Insert | Structural share |
|---|---|---|---|---|---|---|
| Mathematics | 121 | 137 | 4.4 | 1.6 | 1.2 | 5.0% |
| Prose | 107 | 124 | 9.0 | 2.8 | 1.6 | 9.7% |
| Code writing | 128 | 136 | 1.4 | 0.2 | 0.2 | 1.3% |
| Code editing | 104 | 117 | 3.4 | 1.2 | 0.4 | 4.1% |
| Interface | Explicit | Exact | Edit | Length | Multi-span | Source copied |
|---|---|---|---|---|---|---|
| location | boundary | count | allocation | serialization | by runtime | |
| Whole-file rewrite | — | — | — | — | — | — |
| Search-and-replace | ✓ | ✓ | ✓ | — | ✓ | ✓ |
| Locate-then-infill | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Token-level in-place | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Locator | Parse | Exact | Pred./gold | pass@1 | Multi |
|---|---|---|---|---|---|
| DiffuCoder | 91.0 | 2.4 | 1.89 / 2.78 | 13.3 | 12.0 |
| iLLaDA-8B | 94.3 | 4.3 | 1.93 / 2.78 | 13.8 | 11.3 |
| LLaDA-8B | 86.7 | 3.3 | 1.81 / 2.78 | 11.9 | 10.6 |
| Knob | Rewrite | str_replace | Locate |
|---|---|---|---|
| Init | Instruct | Instruct | Stage-2 |
| LR | |||
| Batch / epochs | 128 / 4 | 64 / 5 | 64 / 3 |
| Decode temp | two-stage |
| Interface | Setup | pass@1 | SynErr | 1-span | multi |
|---|---|---|---|---|---|
| Rewrite | SFT, 256 steps | 9.0 | 36.2 | 20.6 | 3.5 |
| Rewrite | SFT, 768 steps | 10.0 | 13.8 | 20.6 | 4.9 |
| str_replace | 256-slot, step 1515 | 9.0 | 5.7 | 23.5 | 2.1 |
| str_replace | random-pad, step 1685 | 4.3 | 11.9 | 7.4 | 2.8 |
| Unified diff | DiffuCoder FT | 4.3 | 1.0 | 13.2 | 0.0 |
| Bucket | str_replace | Locate | Rewrite | Gold loc Qwen | |
|---|---|---|---|---|---|
| 1–5 changed lines | 54 | 46.3 | 29.6 | 29.6 | 77.8 |
| 6–20 changed lines | 112 | 20.5 | 9.8 | 7.1 | 39.3 |
| changed lines | 44 | 2.3 | 0.0 | 0.0 | 9.1 |
| Changed-line ratio | 28 | 21.4 | 21.4 | 10.7 | 89.3 |
| Length similar | 108 | 34.3 | 20.4 | 21.3 | 62.0 |
| Length grow | 100 | 12.0 | 5.0 | 1.0 | 22.0 |
| Interface | Checkpoints | pass@1 (%) |
| Locate same-ckpt | 645 / 1000 / 1290 / 1935 | 6.7 / 11.4 / 13.3 / 12.9 |
| Rewrite 640 decode | 256 / 512 / 768 steps | 9.0 / 11.4 / 10.0 |
| Rewrite sibling 0522 | 428 / 856 (512 decode) | 10.5 / 11.4 |
| str_replace 256-slot | 303 / 606 / 909 / 1212 / 1515 | 11.9 / 8.1 / 8.1 / 7.6 / 9.0 |
| Model | pass@1 | SynErr | 1-span | multi |
|---|---|---|---|---|
| GPT-4o | 72.4 | 0 | 79.4 | 69.0 |
| Qwen3-8B | 51.4 | 0 | 67.6 | 43.7 |
| Qwen2.5-Coder-7B-Instruct | 48.6 | 0 | 58.8 | 43.7 |
| Interface | OK | SynErr | Assert. | Other | Timeout |
|---|---|---|---|---|---|
| Stable-DiffCoder rewrite | 127 | 0 | 56 | 26 | 1 |
| str_replace block-append | 49 | 12 | 71 | 73 | 5 |
| Locate+infill same ckpt | 27 | 32 | 66 | 81 | 4 |
| Rewrite SFT, 512 steps | 24 | 50 | 37 | 96 | 3 |
| Oracle loc empty plan | 52 | 57 | 45 | 56 | 0 |
| Oracle loc Qwen CoT | 90 | 45 | 15 | 59 | 1 |
| Interface | Fmt | Apply | No-op | Syn | Loc | Sem | Pass |
|---|---|---|---|---|---|---|---|
| Rewrite SFT, 512 steps | 0 | 0 | 11 | 59 | 0 | 116 | 24 |
| str_replace block-append | 23 | 29 | 6 | 14 | 0 | 89 | 49 |
| Locate+infill same ckpt | 9 | 0 | 9 | 42 | 112 | 11 | 27 |
| Oracle loc empty plan | 0 | 0 | 12 | 64 | 0 | 82 | 52 |
| Oracle loc Qwen CoT | 0 | 0 | 1 | 58 | 0 | 61 | 90 |
| str_replace <old> spans, 142 multi-span cases | |
|---|---|
| Mean gold-span recall | 42.7% |
| All gold spans covered | 21.8% (31/142) |
| Predicted | 45/142 |
| Enter hybrid Stage 2 branch | 53/142 |
| Metric | Value |
|---|---|
| Parseable plans | 95.7% |
| Zero predicted edits | 4.3% |
| Exact line-op match | 13.3% |
| Nearest-anchor exact | 57.9% |
| Mean predicted / gold edits | 1.55 / 2.78 |
| Scaffold | Dist. | pass@1 | SynErr | 1-span | Multi |
|---|---|---|---|---|---|
| Exact gold ranges | 0.00 | 42.9% | 21.4% | 57.4% | 35.9% |
| Local typed | |||||
| Shift insert-gap | 0.69 | 38.6% | 21.0% | 50.0% | 33.1% |
| Shift insert-gap | 0.71 | 37.1% | 22.9% | 50.0% | 31.0% |
| Shift start (where valid) | 0.56 | 36.7% | 21.4% | 54.4% | 28.2% |
| Shift end | 1.66 | 33.3% | 31.4% | 55.9% | 22.5% |
| Plan / conditioning | pass@1 | 1-span | Multi | SynErr |
|---|---|---|---|---|
| Gold loc. + Qwen CoT | 42.9% | 57.4% | 35.9% | 21.4% |
| Gold loc. + location-free gold-aware Qwen | 42.9% | 57.4% | 35.9% | 22.4% |
| Gold loc. + Qwen CoT (ep5 filler) | 39.5% | 48.5% | 35.2% | 27.1% |
| Gold loc. + empty plan (multitask 1935) | 24.8% | 42.6% | 16.2% | 27.1% |
| Gold loc. + input-only Qwen3-8B | 24.8% | 41.2% | 16.9% | 21.4% |
| Gold loc. + Qwen headers only | 22.4% | 42.6% | 12.7% | 16.7% |
| Interface | 1 hole | 2 | 3 | |
|---|---|---|---|---|
| 68 | 54 | 28 | 60 | |
| Oracle loc. Qwen CoT (C) | 57.4% | 50.0% | 50.0% | 16.7% |
| Oracle loc. empty (C) | 42.6% | 22.2% | 21.4% | 8.3% |
| str_replace block-append (A) | 48.5% | 16.7% | 17.9% | 3.3% |
| Locate-and-infill same ckpt (A) | 27.9% | 9.3% | 7.1% | 1.7% |
| Rewrite SFT 512 (A) | 27.9% | 7.4% | 3.6% | 0.0% |
| Interface | 1 hole | 2 | 3 | |
|---|---|---|---|---|
| 68 | 54 | 28 | 60 | |
| Oracle loc. Qwen CoT (1935) | 57.4 / 20.6 | 50.0 / 22.2 | 50.0 / 21.4 | 16.7 / 21.7 |
| Oracle loc. empty (1935) | 42.6 / 16.2 | 22.2 / 25.9 | 21.4 / 28.6 | 8.3 / 40.0 |
| Aligned empty 1610 | 33.8 / 27.9 | 27.8 / 20.4 | 3.6 / 17.9 | 6.7 / 35.0 |
| Aligned old-visible 1610 | 30.9 / 35.3 | 44.4 / 11.1 | 7.1 / 28.6 | 16.7 / 31.7 |
| Intervention | pass@1 | 1-span | Multi |
|---|---|---|---|
| Exact gold spans | 42.9% | 57.4% | 35.9% |
| Extra false-positive span | 39.5% | 48.5% | 35.2% |
| Shift first span line | 19.5% | 29.4% | 14.8% |
| Drop last gold span | 5.7% | 0.0% | 8.5% |
| Planner | Parse | Exact op | Mean pred | pass@1 |
|---|---|---|---|---|
| Unconstrained Stage 1 | 95.7% | 13.3% | 1.55 | 12.9% |
| Gold- prompt hint | 89.0% | 14.3% | 1.80 | 11.9% |
| Arm | Full | Unaffected | Affected |
|---|---|---|---|
| D pre-glue | 20.5 | 34.7 | 0.0 |
| D ′ glue-fix | 26.2 | 34.7 | 14.0 |
| D ′′ canonicalize | 28.1 | 34.7 | 18.6 |
| Glue-fix oracle length | 27.1 | 30.6 | 22.1 |
| Arm | pass@1 | 1-span | Multi | SynErr |
|---|---|---|---|---|
| D pre-glue integrated | 20.5 | 30.9 | 15.5 | 20.5 |
| D ′ glue-fixed integrated | 26.2 | 33.8 | 22.5 | 26.7 |
| D ′′ canonicalized integrated | 28.1 | 33.8 | 25.4 | 24.8 |
| Local prefix control | 28.1 | 38.2 | 23.2 | 21.9 |
| Scaffold | Passed | pass@1 (%) |
|---|---|---|
| Predicted baseline | 15/142 | 10.6 |
| (coverage) | 19/142 | 13.4 |
| (boundaries) | 20/142 | 14.1 |
| (both) | 31/142 | 21.8 |
| Gold | 36/142 | 25.4 |
| Interface | Setup | pass@1 | SynErr | 1-span | Multi |
|---|---|---|---|---|---|
| Direct str_replace | block-append 1685 | 23.3 | 5.7 | 48.5 | 11.3 |
| Hybrid-multi | predicted <old> old-visible | 22.9 | 9.0 | 48.5 | 10.6 |
| Hybrid-all | all unique <old> old-visible | 13.3 | 23.8 | 22.1 | 9.2 |
| Gold loc. old-visible | aligned 1610 | 27.1 | 27.1 | 30.9 | 25.4 |
| Gold loc. aligned empty | aligned 1610 | 20.5 | 26.7 | 33.8 | 14.1 |
| Method | All 31 | Routed 19 | Fallback 12 |
|---|---|---|---|
| Direct str_replace | 10/31 | 7/19 | 3/12 |
| Predicted scaffold filler | 8/31 | 5/19 | 3/12 |
| Gold scaffold same filler | 13/31 | 7/19 | 6/12 |
| Relation | Direct | Pred. scaf. | Gold scaf. | |
|---|---|---|---|---|
| Merged multiple gold spans | 12 | 3/12 | 3/12 | 6/12 |
| Strict superset | 10 | 5/10 | 2/10 | 3/10 |
| Exact boundaries | 3 | 2/3 | 1/3 | 1/3 |
| Strict subset | 3 | 0/3 | 2/3 | 3/3 |
| Overlap / fragmented | 3 | 0/3 | 0/3 | 0/3 |
| Predicted gold | 17 | 7/17 | 5/17 | 5/17 |
| Scaffold | pass@1 | Multi | SynErr | Appl./imposs. | |
|---|---|---|---|---|---|
| Clean gold | 27.1 | 25.4 | — | 27.1 | 210/0 |
| Drop one gold span | 4.3 | 6.3 | 12.9 | 210/0 | |
| Drop 25% of gold spans | 3.3 | 4.9 | 11.0 | 210/0 | |
| Drop 50% of gold spans | 0.5 | 0.7 | 10.0 | 210/0 | |
| Expand | 15.2 | 7.7 | 39.0 | 210/0 | |
| Expand | 13.8 | 7.0 | 42.9 | 210/0 |
| Family | Mode | Recall | Extra | Multi | Overall | SynErr |
|---|---|---|---|---|---|---|
| gold | clean scaffold | 100.0 | 0.00 | 25.4 | 27.1 | 27.1 |
| missing-span | drop-one | 66.0 | 0.00 | 6.3 | 4.3 | 12.9 |
| missing-span | drop-25% | 62.2 | 0.00 | 4.9 | 3.3 | 11.0 |
| missing-span | drop-50% | 45.5 | 0.00 | 0.7 | 0.5 | 10.0 |
| shift | line | 29.6 | 2.62 | 9.9 | 11.9 | 15.2 |
| shift | line | 32.9 | 2.46 | 4.9 | 8.1 | 32.4 |
| Method | Overall | 1-span | Multi-span |
| Direct one-shot str_replace | 23.3 | 48.5 | 11.3 |
| One-edit prompt, max 5 rounds | 14.8 | 35.3 | 4.9 |
| First-valid executor, max 1 round | 19.5 | 48.5 | 5.6 |
| First-valid executor, max 2 rounds | 18.1 | 38.2 | 8.5 |
| First-valid executor, max 3 rounds | 17.1 | 30.9 | 10.6 |
| First-valid executor, max 4 rounds | 19.0 | 36.8 | 10.6 |
| Stop reason | Rate | |
|---|---|---|
| noop_edit | 59 | 28.1% |
| Malformed edit structure | 56 | 26.7% |
| repeated_edit | 29 | 13.8% |
| max_rounds | 27 | 12.9% |
| code_state_loop | 22 | 10.5% |
| ambiguous_old | 13 | 6.2% |