Self-Decomposing Diffusion for Code Generation

Lize Shao; Zichen Xie; Zhizhen Chen; Zhe Zeng; Wenxi Wang

cs.CLOct 20, 2026

Self-Decomposing Diffusion for Code Generation

Authors: Lize Shao, Zichen Xie, Zhizhen Chen, Zhe Zeng, Wenxi Wang

Organizations: University of Virginia

Abstract

Diffusion language models (DLMs) are emerging as a promising paradigm for code generation, offering the potential for fast, parallel decoding. However, generating complex programs remains challenging, as their interdependent components must maintain consistent interfaces, data flows, and shared assumptions. Aggressive parallel decoding can compromise these dependencies and undermine program correctness, while conservative decoding sacrifices efficiency. To address this trade-off, we propose planning program decomposition before implementation to coordinate interdependent units during parallel generation. We introduce Self-Decomposing Diffusion for Code (SDDC), a novel framework that decomposes complex programs and implements their units in parallel, entirely through diffusion. SDDC first generates a contract specifying the program's decomposition and inter-unit dependencies, then uses it to guide coordinated parallel implementation. Across five benchmarks and two 7B diffusion backbones, SDDC substantially improves the accuracy--latency trade-off, raising pass@1 by up to 31.1 percentage points while decoding 7.5--13.7 $\times$ faster than the base models, and outperforming a fine-tuning baseline by up to 23.5 points with lower latency.

Figures & tables

Panel of Figure 1 — Figure 1: SDDC is more accurate and faster. Dream-Coder 7B on the program-level benchmarks of Table 1 : (a) pass@1 (test pass rate on ClassEval and ClassEval-Pro); (b) latency per problem and speedup over Vanilla.

Figure 2: Overview of SDDC on a held-out KodCode task (abridged and adapted from the model’s output). (1) Contract synthesis denoises entries that give a budget, an index, a role, and the identifiers that each unit defines ( def ) and requires ( use ). (2) Canvas instantiation fixes each entry’s header $h_{k}$ at the start of its canvas. (3) Coupled unit decoding denoises all canvases jointly; they also coordinate a decision that the contract leaves open, nesting both helpers inside sudoku_solver , whose final return the third canvas writes (highlighted). (4) Program assembly removes headers and padding and concatenates the units.

		KodCode (held-out)		ClassEval				ClassEval-Pro
				pass@1		pass@5		pass@1		pass@5
Model	Method	pass@1	pass@5	Class	Test	Class	Test	Class	Test	Class	Test
Dream-Coder 7B	Vanilla	12.5	6.7	9.0	22.6	6.0	23.2	3.3	4.7	4.0	6.8
		61.9	308.0	61.3	61.3	306.9	306.9	123.6	123.6	619.7	619.7
	Diffusion SFT	15.0 (+2.5)	18.3 (+11.6)	7.0 ( $-$ 2.0)	42.3 (+19.7)	8.0 (+2.0)	48.3 (+25.1)	1.3 ( $-$ 2.0)	19.6 (+14.9)	3.0 ( $-$ 1.0)	32.0 (+25.2)
		9.7 (6.4 $\times$ )	51.3 (6.0 $\times$ )	10.4 (5.9 $\times$ )	10.4 (5.9 $\times$ )	55.5 (5.5 $\times$ )	55.5 (5.5 $\times$ )	42.8 (2.9 $\times$ )	42.8 (2.9 $\times$ )	220.8 (2.8 $\times$ )	220.8 (2.8 $\times$ )

Table 1: Program-level composition: pass@1 and pass@5 (%) on KodCode (held-out), ClassEval, and ClassEval-Pro. Class: classes that pass all of their tests; Test: unit tests (ClassEval) or test classes (ClassEval-Pro) passed. Blue rows give the mean latency per problem in seconds and the speedup over Vanilla; pass@5 latency covers five samples. Values next to the scores give the change from Vanilla ( green

\geq

10 points, yellow

<

10 points, red a decrease). pass@5 samples at temperature 0.4 and pass@1 at 0.2 (Appendix A.4 ). Vanilla uses 512 iterations and Diffusion SFT threshold decoding (256 iterations for DiffuCoder on KodCode). The best score per backbone is bolded .

Model	Method	pass@1	pass@5	pass@1	pass@5
		HumanEval Pro		MBPP Pro
Dream- Coder 7B	Vanilla	40.2	19.5	51.9	33.9
		39.4	197.3	34.7	173.4
	Diffusion SFT	26.8 ( $-$ 13.4)	31.7 (+12.2)	36.0 ( $-$ 15.9)	38.9 (+5.0)
		4.2 (9.4 $\times$ )	21.7 (9.1 $\times$ )	2.3 (14.9 $\times$ )	12.2 (14.2 $\times$ )
	SDDC (Ours)	44.5 (+4.3)	42.1 (+22.6)	52.6 (+0.7)	56.1 (+22.2)

Table 2: Self-invoking problem solving (pass@1 and pass@5, %), formatted as in Table 1 . Blue rows give the mean latency per problem in seconds and the speedup over Vanilla. Values next to the scores give the change from Vanilla ( green

\geq

10 points, yellow

<

10 points, red a decrease). Diffusion SFT uses threshold decoding (256 iterations for DiffuCoder on HumanEval Pro); Table 5 adds further configurations.

Panel of Figure 3 — Figure 3: Latency–quality trade-off on KodCode (held-out). Each point is one decoding configuration, labeled with its number of iterations; the upper left is better. On both backbones, the default configuration of SDDC (star) dominates every baseline configuration, and its variants with other contract schedules and single commitment (circles) lie above all of them. Threshold decoding of the Dream-Coder base model (0.8 pass@1, not timed) is omitted.

Attention	Binding	pass@1	Misbound	Overrun
isolated		4.2	32.3	10.1
coupled		11.2	11.0	4.9
isolated	✓	13.3	0.1	1.0
coupled	✓	14.2	0.0	0.9

Table 3: Binding (headers and length-scaled padding) and coupled attention on KodCode (curated subset, left-to-right contracts,

\lambda=10

; held-out). Misbound: canvases whose first definition belongs to another entry; Overrun: canvases that continue into the next unit (%).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Optimizer	paged 8-bit AdamW
Learning rate	$10^{-5}$
Schedule	cosine, 3% warm-up
Effective batch size	32
Epochs (steps)	6 (8,520)
Sequence length	2,048 tokens
Precision	bf16

Appendix

Table 4: Training hyperparameters. The optimizer is paged 8-bit AdamW [ 49 , 50 ] .

	HumanEval Pro		MBPP Pro
Decoding	Dream-Coder	DiffuCoder	Dream-Coder	DiffuCoder
Threshold 0.8	26.8	14.0	36.0	31.5
	4.2	5.5	2.3	2.8
Threshold 0.9	31.1	15.2	40.2	33.3
	6.6	9.2	3.7	4.6
Threshold 0.95	30.5	14.6	40.5	33.9

Appendix

Table 5: Diffusion SFT under different decoding configurations on HumanEval Pro and MBPP Pro: pass@1 (%), with the mean latency per problem in seconds in the blue row below. Vanilla (512 iterations) and SDDC are shown for reference; the most accurate entry of each column is bolded .

	Dream-Coder		DiffuCoder
Contract	pass@1	Lat.	pass@1	Lat.
Default	25.8	12.46	22.1	12.37
End-after-entry	22.9	9.81	21.7	9.55
Pad-scan	21.7	8.56	22.5	8.90
Left-to-right	22.1	9.07	19.6	8.92

Appendix

Table 6: Contract schedules on KodCode (held-out) with single commitment per canvas: pass@1 (%) and latency (s).

$\gamma_{u}$	pass@1	Iter.	pass@1	Iter.
	Dream-Coder 7B		DiffuCoder 7B
single	22.4	137	20.0	143
0.99	24.8	56	20.5	64
0.95	19.5	33	18.6	39
0.90	17.1	25	15.2	28
0.80	10.5	17	9.5	20

Appendix

Table 7: Commitment threshold

\gamma_{u}

on the KodCode validation split: pass@1 (%) and mean unit-decoding iterations (Iter.); “single” commits one token per canvas per iteration.

Code Generation

Code generation studies methods for turning natural-language, visual, or formal specifications into executable programs and code edits.

See all 313 papers

Locally Coherent Parallel Decoding in Diffusion Language Models
Mar 3, 2026Michael Hersche, Nicolas Menet, Ronan Tanios +1Parallel Decoding for Diffusion Language Models Diffusion Language Model Decoding
IBM Research - Zurich · Department of Computer Science, ETH Zürich
Mean-Field Parallel Decoding for Discrete Diffusion Language Models
Jun 14, 2026Tamim Zoabi, Ameen Ali, Liran Ringel +1Parallel Decoding Diffusion Language Model Decoding
School of Electrical & Computer Engineering Tel Aviv University, Israel · School of Computer Science and AI Tel Aviv University, Israel · Department of Computer Science Technion, Israel Institute of Technology, Israel
Divide and Conquer: Accelerating Diffusion-Based Large Language Models via Adaptive Parallel Decoding
Feb 27, 2026Xiangzhong Luo, Yilin An, Zhicheng Yu +2Diffusion Language Model Inference Diffusion Language Model Decoding
Southeast University · Nanyang Technological University

Mar 3, 2026J/K move · Enter open · S save

Locally Coherent Parallel Decoding in Diffusion Language Models

Michael Hersche, Nicolas Menet, Ronan Tanios, Abbas Rahimi

IBM Research - Zurich · Department of Computer Science, ETH Zürich

Parallel Decoding for Diffusion Language Models Diffusion Language Model Decoding LLM Code Generation Blockwise Diffusion Language Models

Open paper AskPDF arXiv

Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive (AR) models, offering sub-linear generation latency and bidirectional capabilities that are particularly appealing for code generation and editing. Achieving sub-linear latency in discrete DLMs requires predicting multiple tokens in parallel. However, standard DLMs sample tokens independently from conditional marginal distributions, failing to capture the joint dependencies among concurrently generated tokens. As a result, they often lead to syntactic inconsistencies and break multi-token structures. In this work, we introduce CoDiLA (Coherent Diffusion with Local Autoregression), a method that reconciles parallel sampling with local dependency modeling. Rather than forcing the DLM to resolve fine-grained syntax, CoDiLA delegates local decoding to a small, auxiliary AR model operating on the diffusion latents. This design allows for parallel generation while ensuring sequential validity within a block and maintaining core DLM capabilities, including bidirectional modeling across blocks. We demonstrate that using a highly compact auxiliary AR model (e.g., 0.6B parameters) effectively eliminates coherence artifacts, establishing a new Pareto frontier for accuracy and speed in code generation benchmarks.

Abstract

Code Generation

Explore similar work

Locally Coherent Parallel Decoding in Diffusion Language Models

Mean-Field Parallel Decoding for Discrete Diffusion Language Models

Divide and Conquer: Accelerating Diffusion-Based Large Language Models via Adaptive Parallel Decoding