Large language models used for code editing can be trained and deployed in at least two output regimes: direct generation, where the model emits the entire modified file in one shot, and iterative diff-based generation ("steps"), where the model emits a sequence of localized search/replace edits applied one at a time until it signals completion or a step budget is exhausted. The diff-based regime is attractive because it mirrors how developers edit code and should require far fewer generated tokens per turn. We train two code models - a 100M-parameter model trained from scratch (Rainbow-Pony-100M) and a fine-tuned Qwen2.5-Coder-0.5B - in both regimes on a shared Flutter/Dart dataset, and evaluate all four resulting models on a held-out set of approx 1,790 tasks per model. Direct generation substantially outperforms diff-based generation on every metric we measure - compilation/static-analysis pass rate, bits-per-byte, character-level similarity to the reference, and blinded LLM-judge ratings of goal fulfillment, correctness, and code quality - and the gap persists after controlling for task difficulty via a matched-ID comparison and when restricting to code that compiles on both sides. We then identify a single, architecture-independent mechanism behind the conditions where diff-based generation does win: it is competitive on short, spatially localized edits, and its category-level wins concentrate in exactly the two task categories - refactoring and error-handling/edge-case fixes - with the lowest mean edit-step count in our dataset. We term this task locality and discuss its implications for when an edit-based training regime is and is not the right choice for a code-editing model.
Figures & tables
Arm
Params
Steps
Epochs
Fine-tune tokens
Final train loss
Final val loss
rainbow-pony-direct
98.16M
5,000
3.06
5.03M / 0.54M (val)
0.157
0.189
rainbow-pony-steps
98.16M
43,000
2.99
146.1M processed ∗
0.221
0.292
qwen-direct
493.81M
5,000
3.05
4.71M / 0.52M (val)
0.074
0.162
qwen-steps
493.81M
43,000
2.96
138.3M processed ∗
0.202
0.247
Table 1: Fine-tuning configuration and outcome per arm, as logged by the training harness. Batch size 8 and block size 1,024 throughout.
Architecture
Mode
n
dart_pass (95% CI)
bits/byte
similarity
Rainbow-Pony-100M
direct
1,789
0.802 [0.783, 0.820]
0.107
0.511
Rainbow-Pony-100M
steps
1,789
0.347 [0.325, 0.369]
0.180
0.439
Qwen2.5-Coder-0.5B
direct
1,792
0.900 [0.885, 0.913]
0.088
0.568
Qwen2.5-Coder-0.5B
steps
1,792
0.501 [0.478, 0.524]
0.142
0.493
Table 2: Aggregate evaluation metrics by architecture and mode. n is the held-out set size per architecture (shared across its direct/steps arms).
Figure 1: Summary panel: dart_pass rate, bits_per_byte , similarity_ratio , stop-reason and fallback breakdowns, the matched-ID best-case comparison, and blinded judge scores, across both architectures.
Architecture
done
apply_failed
max_steps
malformed
Rainbow-Pony-100M
1,525 (85.2%)
116 (6.5%)
144 (8.1%)
4 (0.2%)
Qwen2.5-Coder-0.5B
1,456 (81.3%)
118 (6.6%)
213 (11.9%)
5 (0.3%)
Table 3: Steps-mode trajectory outcomes ( stop_reason ) by architecture.
Architecture
done
apply_failed
max_steps
malformed ( n )
Rainbow-Pony-100M
0.356
0.336
0.236
1.00 (4)
Qwen2.5-Coder-0.5B
0.569
0.229
0.183
0.60 (5)
Table 4: dart_pass rate conditional on stop_reason , steps mode. malformed has n≤5 per architecture and is noise.
Architecture
no fallback ( n )
fallback used ( n )
Rainbow-Pony-100M
0.570 (795)
0.123 (730)
Qwen2.5-Coder-0.5B
0.800 (919)
0.175 (537)
Table 5: dart_pass rate within done -only trajectories, split by whether the trajectory required at least one ambiguity-fallback edit.
Architecture
clean IDs ( n )
clean-steps dart_pass
matched-direct dart_pass
gap (pp)
Rainbow-Pony-100M
792
57.2%
82.1%
24.9
Qwen2.5-Coder-0.5B
919
80.0%
92.4%
12.4
Table 6: Matched-ID comparison: clean steps-mode trajectories vs. direct mode on the identical sample IDs. Recomputed directly from the released evaluation datasets (Section 8 ).
Architecture
Dimension
steps
direct
t (Welch)
p
Rainbow-Pony-100M
goal fulfillment
3.44
3.89
−4.27
2.2×10−5
correctness
3.71
4.14
−4.60
5.0×10−6
code quality
3.96
4.46
−6.87
<10−6
Qwen2.5-Coder-0.5B
goal fulfillment
4.36
4.72
−6.95
<10−6
correctness
4.36
4.75
−8.17
<10−6
code quality
4.52
4.82
−8.25
<10−6
Table 7: Blinded LLM-judge scores (1–5 scale) among matched-ID rows where both steps and direct pass dart_pass ( n=390 paired rows for Rainbow-Pony-100M, n=688 for Qwen2.5-Coder-0.5B). All six pairwise differences are significant under Welch’s t -test at p<0.001 . Recomputed directly from the released evaluation datasets (Section 8 ).
Figure 2: Heatmap summary of all four arms across six key metrics (darker = better, per-column normalized).
Figure 3: Same heatmap restricted to the best-case steps population (done, no fallback, < 20 steps, dart_pass=True ) vs. matched direct-mode rows.
Metric
steps win rate
dart_pass (steps passes, direct fails)
3.5% – 4.8%
bits_per_byte (steps lower)
0.1% – 0.3%
similarity_ratio (steps higher)
33.7% – 34.8%
judge dimensions (steps higher)
4.8% – 9.2%
Table 8: Steps-mode per-row win rate by metric, across both architectures (ranges span the two architectures).
Category
Rainbow-Pony share of wins
Qwen share of wins
refactoring_edits
18.6%
14.3%
error_handling_and_edge_cases
15.1%
15.9%
(baseline share, all categories)
∼ 11.2%
Table 9: Task categories overrepresented among dart_pass steps-mode wins, relative to their ∼ 11.2% baseline share of the dataset, in both architectures. Win counts are small ( n=63 –86 total wins per architecture); other categories disagreed in direction between architectures and are treated as noise.
Architecture
wins mean
wins median
rest mean
rest median
t
Rainbow-Pony-100M
5.54
5
8.94
8
−8.60
Qwen2.5-Coder-0.5B
6.65
6
9.02
8
−4.54
Table 10: Trajectory length ( num_steps ) for composite judge-majority wins vs. all other rows. All differences significant under Welch’s t -test, p≈0 .
Architecture
Dimension
t
p
Rainbow-Pony-100M
goal fulfillment
−8.48
<0.001
correctness
−9.99
<0.001
code quality
−5.09
<0.001
Qwen2.5-Coder-0.5B
goal fulfillment
−3.85
0.0002
correctness
−5.39
<0.001
code quality
−3.72
0.0003
Table 11: Per-dimension Welch’s t -tests: trajectory length of steps-mode judge wins vs. non-wins. All p<0.001 .
Figure 4: Judge win rate by trajectory length ( num_steps ), all steps rows vs. best-case-only steps rows, per architecture and judge dimension.
Figure 5: Mean judge score by trajectory length: steps (all data), steps (best-case IDs), direct (all data), direct (best-case IDs).
Figure 6: Win rate by trajectory length for both directions (steps-wins, direct-wins), all data vs. best-case IDs.
Figure 7: Judge score by trajectory length, best-case steps (done, no fallback, < 20 steps, dart_pass=True ) vs. matched direct-mode rows on the same sample IDs. Shown for reference alongside Figure 5 .
Category
Rainbow-Pony mean steps
Qwen mean steps
refactoring_edits (median 2–3)
4.14
4.49
error_handling_and_edge_cases
6.84
7.24
Table 12: Mean num_steps for the two win-overrepresented categories vs. the full 9-category distribution. Both are the lowest or near-lowest mean-step-count categories in both architectures.
Diffusion large language models (dLLMs) are appealing for code generation because they can update many tokens in parallel, yet many are optimized for sequence synthesis rather than edit prediction. This setting is poorly matched to modern AI-assisted IDEs, where edit prediction must infer the next code change from recent edits, the current file, and cursor-local context. Such changes are usually sparse and preserve most source tokens, so regenerating a full code segment wastes computation. We propose Diff Diffusion, a diffusion-inspired LLM framework for code editing that predicts edit paths from an original snippet to its revised form. Diff Diffusion represents denoising as structured edit operations, and combines this formulation with multi-stage training and efficient parallel decoding. By focusing computation on changed regions, Diff Diffusion supports accurate low-latency edits and multi-position parallel modification. Across public edit-completion benchmarks, edit-prediction evaluations, and online validation, Diff Diffusion matches or leads autoregressive baselines while delivering 2.78 times faster inference on public edit-completion benchmarks and a 39.0% P50 model-latency reduction in production. This work provides a new modeling perspective for AI applications that require high accuracy and low latency.
Code editing requires a model to decide where to make changes, generate the new content, and preserve everything else. We study how masked diffusion language models divide these responsibilities across four editing interfaces: whole-file rewriting, search-and-replace, locate-then-infill, and token-level editing. Experiments on CanItEdit reveal a composition gap: diffusion models can generate coordinated changes when the correct edit locations are supplied, but much of this capability is lost when those locations must be predicted. Access to the intact original code helps the model fill multiple edit regions, yet does not resolve the difficulty of selecting those regions. By varying the editable regions while holding the generation model and decoding procedure fixed, we identify two distinct requirements for successful editing: covering every required change and placing precise boundaries around it. Missing a required region prevents the corresponding change, while widening regions to ensure coverage can sharply reduce success by requiring unchanged code to be regenerated. A sentence-level Wiki editing probe shows the same qualitative gap between supplied and predicted locations beyond code. These findings show why strong infilling capability alone does not ensure reliable editing: the interface must expose all required changes while limiting regeneration of unchanged code.
Xijia Tao, Ziru Liu, Shansan Gong +7
The University of Hong Kong · Huawei Technologies Ltd. · City University of Hong Kong +1
Large Language Models (LLMs) are increasingly used for code editing, yet the prevalent full-code generation paradigm suffers from severe efficiency bottlenecks, posing challenges for interactive coding assistants that demand low latency and cost. Despite the predominant focus on scaling model capabilities, the edit format itself has been largely overlooked in model training. In this paper, we begin with a systematic study of conventional diff formats and reveal that fragile offsets and fragmented hunks make generation highly unnatural for LLMs. To address it, we introduce BlockDiff and FuncDiff, two structure-aware diff formats that represent changes as block-level rewrites of syntactically coherent units such as control structures and functions. Furthermore, we propose AdaEdit, a general adaptive edit strategy that trains LLMs to dynamically choose the most token-efficient format between a given diff format and full code. Extensive experiments demonstrate that AdaEdit paired with structure-aware diff formats consistently matches the accuracy of full-code generation, while reducing both latency and cost by over 30% on long-code editing tasks.
Wei Cheng, Yongchang Cao, Chen Shen +4
State Key Laboratory for Novel Software Technology, Nanjing University, China · Tongyi Lab, Alibaba Group, China