Diffs vs. Whole Files: An Empirical Comparison of Iterative Edit-Based and Direct Generation for Flutter/Dart Code Models

Andrej Andrejev

cs.SESep 5, 2026

Diffs vs. Whole Files: An Empirical Comparison of Iterative Edit-Based and Direct Generation for Flutter/Dart Code Models

Authors: Andrej Andrejev

Organizations: Independent researcher (bbidpa)

Abstract

Large language models used for code editing can be trained and deployed in at least two output regimes: direct generation, where the model emits the entire modified file in one shot, and iterative diff-based generation ("steps"), where the model emits a sequence of localized search/replace edits applied one at a time until it signals completion or a step budget is exhausted. The diff-based regime is attractive because it mirrors how developers edit code and should require far fewer generated tokens per turn. We train two code models - a 100M-parameter model trained from scratch (Rainbow-Pony-100M) and a fine-tuned Qwen2.5-Coder-0.5B - in both regimes on a shared Flutter/Dart dataset, and evaluate all four resulting models on a held-out set of approx 1,790 tasks per model. Direct generation substantially outperforms diff-based generation on every metric we measure - compilation/static-analysis pass rate, bits-per-byte, character-level similarity to the reference, and blinded LLM-judge ratings of goal fulfillment, correctness, and code quality - and the gap persists after controlling for task difficulty via a matched-ID comparison and when restricting to code that compiles on both sides. We then identify a single, architecture-independent mechanism behind the conditions where diff-based generation does win: it is competitive on short, spatially localized edits, and its category-level wins concentrate in exactly the two task categories - refactoring and error-handling/edge-case fixes - with the lowest mean edit-step count in our dataset. We term this task locality and discuss its implications for when an edit-based training regime is and is not the right choice for a code-editing model.

Figures & tables

Arm	Params	Steps	Epochs	Fine-tune tokens	Final train loss	Final val loss
rainbow-pony-direct	98.16M	5,000	3.06	5.03M / 0.54M (val)	0.157	0.189
rainbow-pony-steps	98.16M	43,000	2.99	146.1M processed ∗	0.221	0.292
qwen-direct	493.81M	5,000	3.05	4.71M / 0.52M (val)	0.074	0.162
qwen-steps	493.81M	43,000	2.96	138.3M processed ∗	0.202	0.247

Table 1: Fine-tuning configuration and outcome per arm, as logged by the training harness. Batch size 8 and block size 1,024 throughout.

Architecture	Mode	$n$	dart_pass (95% CI)	bits/byte	similarity
Rainbow-Pony-100M	direct	1,789	0.802 [0.783, 0.820]	0.107	0.511
Rainbow-Pony-100M	steps	1,789	0.347 [0.325, 0.369]	0.180	0.439
Qwen2.5-Coder-0.5B	direct	1,792	0.900 [0.885, 0.913]	0.088	0.568
Qwen2.5-Coder-0.5B	steps	1,792	0.501 [0.478, 0.524]	0.142	0.493

Table 2: Aggregate evaluation metrics by architecture and mode.

n

is the held-out set size per architecture (shared across its direct/steps arms).

Figure 1: Summary panel: dart_pass rate, bits_per_byte , similarity_ratio , stop-reason and fallback breakdowns, the matched-ID best-case comparison, and blinded judge scores, across both architectures.

Architecture	done	apply_failed	max_steps	malformed
Rainbow-Pony-100M	1,525 (85.2%)	116 (6.5%)	144 (8.1%)	4 (0.2%)
Qwen2.5-Coder-0.5B	1,456 (81.3%)	118 (6.6%)	213 (11.9%)	5 (0.3%)

Table 3: Steps-mode trajectory outcomes ( stop_reason ) by architecture.

Architecture	done	apply_failed	max_steps	malformed ( $n$ )
Rainbow-Pony-100M	0.356	0.336	0.236	1.00 (4)
Qwen2.5-Coder-0.5B	0.569	0.229	0.183	0.60 (5)

Table 4: dart_pass rate conditional on stop_reason , steps mode. malformed has

n\leq 5

per architecture and is noise.

Architecture	no fallback ( $n$ )	fallback used ( $n$ )
Rainbow-Pony-100M	0.570 (795)	0.123 (730)
Qwen2.5-Coder-0.5B	0.800 (919)	0.175 (537)

Table 5: dart_pass rate within done -only trajectories, split by whether the trajectory required at least one ambiguity-fallback edit.

Architecture	clean IDs ( $n$ )	clean-steps dart_pass	matched-direct dart_pass	gap (pp)
Rainbow-Pony-100M	792	57.2%	82.1%	24.9
Qwen2.5-Coder-0.5B	919	80.0%	92.4%	12.4

Table 6: Matched-ID comparison: clean steps-mode trajectories vs. direct mode on the identical sample IDs. Recomputed directly from the released evaluation datasets (Section 8 ).

Architecture	Dimension	steps	direct	$t$ (Welch)	$p$
Rainbow-Pony-100M	goal fulfillment	3.44	3.89	$-4.27$	$2.2\times 10^{-5}$
	correctness	3.71	4.14	$-4.60$	$5.0\times 10^{-6}$
	code quality	3.96	4.46	$-6.87$	$<10^{-6}$
Qwen2.5-Coder-0.5B	goal fulfillment	4.36	4.72	$-6.95$	$<10^{-6}$
	correctness	4.36	4.75	$-8.17$	$<10^{-6}$
	code quality	4.52	4.82	$-8.25$	$<10^{-6}$

Table 7: Blinded LLM-judge scores (1–5 scale) among matched-ID rows where both steps and direct pass dart_pass (

n=390

paired rows for Rainbow-Pony-100M,

n=688

for Qwen2.5-Coder-0.5B). All six pairwise differences are significant under Welch’s

t

-test at

p<0.001

. Recomputed directly from the released evaluation datasets (Section 8 ).

Figure 2: Heatmap summary of all four arms across six key metrics (darker = better, per-column normalized).

Figure 3: Same heatmap restricted to the best-case steps population (done, no fallback, $<$ 20 steps, dart_pass=True ) vs. matched direct-mode rows.

Metric	steps win rate
dart_pass (steps passes, direct fails)	3.5% – 4.8%
bits_per_byte (steps lower)	0.1% – 0.3%
similarity_ratio (steps higher)	33.7% – 34.8%
judge dimensions (steps higher)	4.8% – 9.2%

Table 8: Steps-mode per-row win rate by metric, across both architectures (ranges span the two architectures).

Category	Rainbow-Pony share of wins	Qwen share of wins
refactoring_edits	18.6%	14.3%
error_handling_and_edge_cases	15.1%	15.9%
(baseline share, all categories)	$\sim$ 11.2%

Table 9: Task categories overrepresented among dart_pass steps-mode wins, relative to their

\sim

11.2% baseline share of the dataset, in both architectures. Win counts are small (

n=63

–86 total wins per architecture); other categories disagreed in direction between architectures and are treated as noise.

Architecture	wins mean	wins median	rest mean	rest median	$t$
Rainbow-Pony-100M	5.54	5	8.94	8	$-8.60$
Qwen2.5-Coder-0.5B	6.65	6	9.02	8	$-4.54$

Table 10: Trajectory length ( num_steps ) for composite judge-majority wins vs. all other rows. All differences significant under Welch’s

t

-test,

p\approx 0

Architecture	Dimension	$t$	$p$
Rainbow-Pony-100M	goal fulfillment	$-8.48$	$<0.001$
	correctness	$-9.99$	$<0.001$
	code quality	$-5.09$	$<0.001$
Qwen2.5-Coder-0.5B	goal fulfillment	$-3.85$	$0.0002$
	correctness	$-5.39$	$<0.001$
	code quality	$-3.72$	$0.0003$

Table 11: Per-dimension Welch’s

t

-tests: trajectory length of steps-mode judge wins vs. non-wins. All

p<0.001

Figure 4: Judge win rate by trajectory length ( num_steps ), all steps rows vs. best-case-only steps rows, per architecture and judge dimension.

Figure 5: Mean judge score by trajectory length: steps (all data), steps (best-case IDs), direct (all data), direct (best-case IDs).

Figure 6: Win rate by trajectory length for both directions (steps-wins, direct-wins), all data vs. best-case IDs.

Figure 7: Judge score by trajectory length, best-case steps (done, no fallback, $<$ 20 steps, dart_pass=True ) vs. matched direct-mode rows on the same sample IDs. Shown for reference alongside Figure 5 .

Category	Rainbow-Pony mean steps	Qwen mean steps
refactoring_edits (median 2–3)	4.14	4.49
error_handling_and_edge_cases	6.84	7.24

Table 12: Mean num_steps for the two win-overrepresented categories vs. the full 9-category distribution. Both are the lowest or near-lowest mean-step-count categories in both architectures.

Code Language Models

Code language models are AI models that generate, explain, edit, and reason about source code across programming tasks.

See all 115 papers

Diff Diffusion: Predicting Edit Paths with Diffusion Models for Efficient Code Editing
Oct 13, 2026Xiaolong Xu, Hao Jiang, Bo Jiang +5Diffusion Language Models Diffusion Language Model Decoding
ByteDance, Shenzhen, China
How Should Diffusion Language Models Edit Code?
Sep 29, 2026Xijia Tao, Ziru Liu, Shansan Gong +7Masked Diffusion Models Language Model Editing
The University of Hong Kong · Huawei Technologies Ltd. · City University of Hong Kong +1
To Diff or Not to Diff? Structure-Aware and Adaptive Output Formats for Efficient LLM-based Code Editing
Apr 30, 2026Wei Cheng, Yongchang Cao, Chen Shen +4Code Generation LLM Inference Acceleration
State Key Laboratory for Novel Software Technology, Nanjing University, China · Tongyi Lab, Alibaba Group, China

Oct 13, 2026J/K move · Enter open · S save

Diff Diffusion: Predicting Edit Paths with Diffusion Models for Efficient Code Editing

Xiaolong Xu, Hao Jiang, Bo Jiang, Zhiwen Deng+4

ByteDance, Shenzhen, China

Diffusion Language Models Diffusion Language Model Decoding Efficient Language Model Inference

Open paper AskPDF arXiv

Diffusion large language models (dLLMs) are appealing for code generation because they can update many tokens in parallel, yet many are optimized for sequence synthesis rather than edit prediction. This setting is poorly matched to modern AI-assisted IDEs, where edit prediction must infer the next code change from recent edits, the current file, and cursor-local context. Such changes are usually sparse and preserve most source tokens, so regenerating a full code segment wastes computation. We propose Diff Diffusion, a diffusion-inspired LLM framework for code editing that predicts edit paths from an original snippet to its revised form. Diff Diffusion represents denoising as structured edit operations, and combines this formulation with multi-stage training and efficient parallel decoding. By focusing computation on changed regions, Diff Diffusion supports accurate low-latency edits and multi-position parallel modification. Across public edit-completion benchmarks, edit-prediction evaluations, and online validation, Diff Diffusion matches or leads autoregressive baselines while delivering 2.78 times faster inference on public edit-completion benchmarks and a 39.0% P50 model-latency reduction in production. This work provides a new modeling perspective for AI applications that require high accuracy and low latency.

Abstract

Code Language Models

Explore similar work

Diff Diffusion: Predicting Edit Paths with Diffusion Models for Efficient Code Editing

How Should Diffusion Language Models Edit Code?

To Diff or Not to Diff? Structure-Aware and Adaptive Output Formats for Efficient LLM-based Code Editing