DEdit: Iterative Draft Editing for Speculative Decoding

Longxuan Yu; Bingsen Chen; Peng Shi; Dongkyu Lee; Yi Xiang; Hideo Kobayashi; Sheng Zhang; Shuaichen Chang

cs.CLSep 29, 2026

DEdit: Iterative Draft Editing for Speculative Decoding

Authors: Longxuan Yu, Bingsen Chen, Peng Shi, Dongkyu Lee, Yi Xiang, Hideo Kobayashi, Sheng Zhang, Shuaichen Chang,

Organizations: University of California, Riverside · Amazon Web Services · New York University

Abstract

Speculative decoding accelerates autoregressive LLMs by having a lightweight drafter propose tokens that the target model verifies in parallel. Diffusion-based drafters further reduce drafting latency by proposing multiple tokens at once. However, these tokens are predicted independently, so a single early error causes prefix verification to discard the rest of the draft, even when it contains useful downstream predictions. We introduce DEdit, a diffusion-based drafter that can not only draft by conventional parallel unmasking but also iteratively edit its draft through token-to-token predictions. Through editing, later predictions can serve as bidirectional context for repairing earlier errors and extending the accepted prefix. To teach the model to repair errors while preserving correct predictions, we propose ProposalMix, a training scheme that mixes draft predictions with ground-truth tokens based on first-pass confidence during training. Across seven benchmarks on Qwen3-4B and Qwen3-8B, DEdit achieves the highest macro-average token acceptance and speedup among the evaluated drafters, reaching macro-average speedups of $5.72\times$ and $5.97\times$ over autoregressive generation under greedy decoding, respectively. Further analysis shows that acceptance improves with more editing passes and wider drafting windows, and that ProposalMix halves harmful edits that shorten the accepted prefix. Moreover, restricting the editor to causal attention lowers acceptance, especially on highly predictable outputs, indicating that future context is a key source of these gains.

Figures & tables

Figure 1: Bidirectional editing before verification extends the accepted prefix.

Figure 2: DEdit inference and training. (a) At inference time, drafter generates and edits its proposal tokens through

K

passes, sending only the final proposal to the target model for verification. (b) ProposalMix constructs P2 inputs for joint proposal-and-edit training.

Model	Method	Math						Code						Chat		Overall
Model	Method	GSM8K		MATH-500		AIME25		HumanEval		MBPP		LCB		MT-Bench		Avg.
Temperature: 0		Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$
Qwen3-4B	DFlash	4.99 $\times$	6.04	6.52 $\times$	7.76	5.82 $\times$	7.07	4.45 $\times$	5.32	4.08 $\times$	5.06	4.81 $\times$	5.83	2.98 $\times$	4.17	4.81 $\times$	5.89
	DSpark	5.37 $\times$	7.13	6.52 $\times$	9.02	6.06 $\times$	8.23	4.61 $\times$	6.23	4.37 $\times$	6.11	5.32 $\times$	7.14	2.67 $\times$	4.53	4.99 $\times$	6.91
	Domino	5.53 $\times$	6.76	7.21 $\times$	8.87	6.65 $\times$	8.12	4.76 $\times$	5.84	4.56 $\times$	5.70	5.30 $\times$	6.39	3.13 $\times$	4.53	5.30 $\times$	6.60
	DEdit P3	6.51 $\times$	7.75	8.18 $\times$	10.67	6.84 $\times$	9.28	4.95 $\times$	6.48	4.71 $\times$	6.13	5.71 $\times$	7.61	3.17 $\times$	5.11	5.72 $\times$	7.58

Table 1: End-to-end speedup and mean token acceptance (

\tau

) under the evaluation protocol in Section 4.1 ; Avg. is the seven-suite macro average. Baselines operate at

W=16

(Domino and DSpark draft 16 candidates; DFlash drafts 15), while DEdit uses the

W=32

P3 model after a 3K-step continuation from

W=16

. Bold marks the best value in each model/temperature block.

Method	Pass	$W=16$	$W=24$	$W=32$
DFlash	P1	$5.89$	$6.01$	$6.01$
Domino	P1	$6.60$	$6.91$	$6.93$
DEdit	P1	$5.82$	$5.93$	$5.93$
	P2	$6.63$	$7.02$	$7.12$
	P3	$6.87$	$7.39$	7.58

Table 2: Window scaling and training ablations on Qwen3-4B. Both panels report macro-average

\tau

across seven benchmarks. (a) Proposal width and editing passes at

T_{p}=0

; Domino drafts

W

candidates and the other methods

W-1

. (b) Training components in a controlled 100K setting: a frozen DFlash drafter supplies P1, and a separately trained editor is evaluated at P2 with

W=16

;

\Delta\tau

is relative to Full.

Panel of Figure 3 — Figure 3: Editing outcomes and future context on Qwen3-4B. (a) Edit-gain distribution $A_{\mathrm{P2}}-A_{\mathrm{P1}}$ for the Full and w/o ProposalMix editors of Table 2 (b). Percentages use all rounds; zero-gain rounds are omitted (60.0% without ProposalMix , 66.5% with it). (b) $W=16$ DEdit P2-minus-Domino gain versus correct future-token count. Points and bands show response-cluster means and 95% cluster-bootstrap CIs; gray bars show the number of rounds per bin on a log scale, from 56k to 127. Fully accepted rounds (7.7%) are excluded.

		DEdit P2			DEdit P3
Task slice	Domino	Causal	Bidir.	$\Delta$	Causal	Bidir.	$\Delta$
General (7-suite avg.)	5.168	4.997	5.219	+0.222	5.192	5.482	+0.290
Reasoning	3.260	3.224	3.422	+0.199	3.371	3.624	+0.253
Summary	3.756	3.718	3.913	+0.195	3.881	4.141	+0.260
Copy-heavy	8.560	9.374	10.098	+0.724	9.819	10.717	+0.897

Table 3: Fixed-weight visibility intervention. Values are

A

over the first 16 candidate positions, averaged over rounds within each slice; General averages the seven suite-level means. Causal and bidirectional DEdit share a

W=32

checkpoint and P1 proposal. Domino (

W=16

) takes the first token of this proposal and drafts the remaining 15 with its own backbone and corrector.

\Delta

: bidirectional minus causal; underlined: below Domino. Details in Appendix A.3 .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: A full

W=32

editing trace. On an MBPP prompt, successive passes extend the accepted prefix from 8 to 18 to 30 of 31 proposal tokens. Blue cells already match the target but remain blocked by an earlier error.

Figure 5: Fixed-weight intervention on future proposal visibility. Lines report response-cluster mean bidirectional-minus-causal accepted tokens with 95% response-bootstrap intervals; gray bars report each bin’s share of 181,205 eligible rounds. P2 compares both masks on the same P1 proposal, while P3 applies the next edit to each configuration. The hatched bar denotes the 10,382 rounds in which all 16 shared proposal tokens are accepted.

Source / partition	800K (Qwen3-4B)	800K (Qwen3-8B)	100K (Qwen3-4B)
Nemotron / chat	365,375 (45.68%)	365,511 (45.69%)	28,791 (28.79%)
Nemotron / code	175,000 (21.88%)	175,000 (21.88%)	2,195 (2.20%)
Nemotron / math	239,467 (29.94%)	239,467 (29.93%)	6,698 (6.70%)
Nemotron / STEM	–	–	23,197 (23.20%)
CodeAlpaca-20k	20,022 (2.50%)	20,022 (2.50%)	–
OpenR1 Math	–	–	12,429 (12.43%)

Appendix

Table 4: Training-corpus composition: records (percentage of each corpus). Counts are computed from the released source field and, for 100K, the joint source / split fields. Nemotron rows denote v2 partitions for 800K and the mirror’s nemotron labels for 100K. A dash denotes no records with that source label.

Setting	Controlled	Main
Role	Training-design ablations	Main system comparison
Target(s)	Qwen3-4B	Qwen3-4B and Qwen3-8B
Corpus	100K target-aligned	800K Nemotron-based mixture
Global batch	8	16
Epochs	6
Optimizer	AdamW, LR $6\times 10^{-4}$ , warmup 0.04, clip 1.0

Appendix

Table 5: Training configurations for the controlled ablations and main results. Shared entries span both columns.

Step	P1 accuracy	GT selection	Overwrite
12,221	33.19	28.10	5.79
36,663	42.80	37.35	5.84
73,326	49.87	44.91	5.84

Appendix

Table 6: ProposalMix input rates (%;

\eta=0.5

) when each checkpoint generates its own P1 on a fixed probe set. GT selection marks positions chosen for ground-truth conditioning; overwrite counts those whose tokens change.

Figure 6: Trade-off across editing passes for the same

W=32

checkpoint at

T_{p}=0

. Each configuration is evaluated twice with a shared autoregressive baseline. Drafting uses CUDA Graphs and target verification remains eager. Training supervises P1 and P2; P3 and P4 reuse the trained editor at inference.

Figure 7: Host enqueue-ahead during CUDA Graph replay. (a) The CPU submits eager target operations while the GPU executes the captured drafter; draft and target GPU operations remain sequential. A forced host barrier delays target submission. (b) Nsight traces decompose each round into the pre-target draft interval, target compute, and gaps between target operations.

Efficient Language Model Inference

Efficient language model inference reduces the compute, memory, latency, and energy needed to generate useful outputs.

See all 308 papers

D^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models
Jun 3, 2026Liyuan Zhang, Jiarui Zhang, Jinwei Yao +6Speculative Decoding for Diffusion Language Models Speculative Decoding
Peking University · Tsinghua University · HKUST +2
DLoop: Looped Speculative Decoding
Oct 6, 2026Geonmo Gu, Byeongho Heo, HeeJae Jun +4Speculative Decoding LLM Inference Acceleration
NAVER AI Lab · Korea University · NAVER AI Search Platform
Accelerating Speculative Decoding with Block Diffusion Draft Trees
Apr 14, 2026Liran Ringel, Yaniv RomanoSpeculative Decoding Parallel Speculative Decoding
Department of Computer Science, Technion – Israel Institute of Technology. · Department of Electrical and Computer Engineering, Technion – Israel Institute of Technology.

Jun 3, 2026J/K move · Enter open · S save

D^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models

Liyuan Zhang, Jiarui Zhang, Jinwei Yao, Ran Yan+5

Peking University · Tsinghua University · HKUST+2

Speculative Decoding for Diffusion Language Models Speculative Decoding Diffusion Language Model Inference LLM Inference Acceleration Diffusion Language Model Decoding Parallel Speculative Decoding

Open paper AskPDF arXiv

Speculative decoding accelerates autoregressive large language model inference by drafting multiple tokens and verifying them in a single target-model forward pass. Recent diffusion-based drafters generate an entire block of tokens in parallel but usually commit to a single draft sequence per verification: once the first mismatch occurs, all subsequent draft tokens are discarded, resulting in a limited acceptance rate. Naively batching more draft candidate sequences only introduces a marginal improvement, as redundant or poorly placed branches increase the cost of drafting and verification without proportionally increasing the number of accepted tokens. We propose D^2SD, a dual diffusion draft speculative decoding framework that organizes candidates into a confidence-guided prefix tree, where the first diffusion drafter generates a block along with per-position confidence scores that are used to identify the most likely rejection boundary and select the top-K prefix ranges for recovery; the second variable-prefix diffusion drafter re-anchors at each selected prefix and proposes alternative continuations in one batched pass; the resulting shared-prefix candidates are jointly verified via cascade attention. Empirically, D^2SD shows clear improvements over both the underlying diffusion approach and strong autoregressive speculative decoding baselines.

Abstract

Efficient Language Model Inference

Explore similar work

D^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models

DLoop: Looped Speculative Decoding

Accelerating Speculative Decoding with Block Diffusion Draft Trees