Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive (AR) models, offering sub-linear generation latency and bidirectional capabilities that are particularly appealing for code generation and editing. Achieving sub-linear latency in discrete DLMs requires predicting multiple tokens in parallel. However, standard DLMs sample tokens independently from conditional marginal distributions, failing to capture the joint dependencies among concurrently generated tokens. As a result, they often lead to syntactic inconsistencies and break multi-token structures. In this work, we introduce CoDiLA (Coherent Diffusion with Local Autoregression), a method that reconciles parallel sampling with local dependency modeling. Rather than forcing the DLM to resolve fine-grained syntax, CoDiLA delegates local decoding to a small, auxiliary AR model operating on the diffusion latents. This design allows for parallel generation while ensuring sequential validity within a block and maintaining core DLM capabilities, including bidirectional modeling across blocks. We demonstrate that using a highly compact auxiliary AR model (e.g., 0.6B parameters) effectively eliminates coherence artifacts, establishing a new Pareto frontier for accuracy and speed in code generation benchmarks.
Figures & tables
Figure 1: Our CoDiLA in action. a) An example of incoherent text generated by Dream-Coder-Instruct-7B in the first iteration. Due to independent modeling of marginal distributions, it predicts the incoherent token “problem” (Top-1). b) This work enforces local coherence using a block-wise AR model conditioned on soft local tokens. In this example, it recovers coherence by retrieving the correct token “(list” from the Top-3 candidates. Displayed prompt was simplified for illustrative purposes.
Figure 2: CoDiLA with a block size of B=4 . This example depicts the prediction of the first block ( b1 ). First, the DLM computes the token-wise conditional marginal probability vectors ( πtj ). Next, we perform soft-conditioning by computing the expected embedding ( etj ) over the AR model’s embedding matrix ( Eϕ ), weighted by these marginals. Finally, the AR model receives these soft tokens, encapsulated by <think> and < \ think> boundary tokens, to autoregressively decode a locally coherent sequence.
Figure 3: Larger block sizes ( B ) reduce the training loss. We compute the average perplexity weighted by the masking ratio (see Equation ( 2 )), and display the moving average over 10 samples. The forward process always masks blocks of 32 contiguous tokens.
Figure 4: Inference with static parallelism. We report on Pass@1 (%) vs. Throughput (tokens/sec, batch-size 1) on a single NVIDIA A100-80GB GPU. We compare the base DLM ( Xie et al., 2025 ) , ADJUST ( Bansal and Sanghavi, 2025 ) , and our CoDiLA, all built on Dream-Coder-Instruct-7B. Parallelism is controlled by unmasking a fixed number of tokens per iteration. CoDiLA consistently achieves higher accuracy at equivalent throughput levels.
Model
K = B = 2
K = B = 4
K = B = 8
Dream-Coder-Instruct-7B
18
38
70
CoDiLA
4
13
16
Table 1: Syntax errors (%) on HumanEval.
Figure 5: Inference with dynamic parallelism. We operate a dynamic CoDiLA ( B=4 ) with different entropy thresholds ( τ ).
P@1 (%)
Tokens/step
Deepseek-Coder-6.7B †
45.7
1
Seed-Coder-8B †
59.3
1
Qwen2.5-Coder-7B †
58.7
1
DreamOn ( K=1 )
62.5
1
DreamOn ( K=2 )
53.1
2
DreamOn+CoDiLA ( τ=0.2 )
62.5
1.3
Table 2: Multi-line code infilling on HumanEval-Infilling. † Baseline results are taken from Wu et al. (2025) .
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
HumanEval/+
MBPP/+
BigCodeBench
Max Generation Length
768
512
1024
Temperature
0.1
0.1
0.1
Top- p
0.8
0.8
0.8
Block Size B
{2,4,8}
{2,4,8}
{2,4,8}
Appendix
Table 3: Hyperparameter configurations across different benchmarks.
K=1
K=B=2
K=B=4
K=B=8
P@1
TPS
P@1
TPS
P@1
TPS
P@1
TPS
HumanEval
Dream-Coder
74.4
9
54.9
19
24.4
37
5.5
74
CoDiLA (Softmax)
68.9
18
51.8
33
29.9
60
CoDiLA (Top-5)
63.4
18
56.1
33
28.0
58
CoDiLA (Top-3)
61.0
18
50.0
31
31.7
59
Appendix
Table 4: Impact of conditioning strategy. Comparison of Pass@1 (%) scores and throughput (TPS) between baseline Dream-Coder-Instruct-7B and CoDiLA with both soft- and Top-K-conditioning.
Figure 6: Ablation of the AR model size. CoDiLA does not require large AR models.
Candidate Scope (Blocks)
Pass@1
TPS
10 (default)
51.8%
34.4
20
49.3%
32.7
30
45.1%
32.1
50
42.1%
29.5
Appendix
Table 5: Increasing the lookahead window for block decoding selection has a negligible impact on the throughput (TPS), but impacts accuracy due to premature EOS decoding. We report CoDiLA with B=4 using static parallelism on HumanEval.
Figure 7: Selecting the blocks to decode globally based on DLM entropy performs comparable to a fixed lookahead window.
B=2
B=4
B=8
Pass@1
TPS
Pass@1
TPS
Pass@1
TPS
1 Block ( K=B )
68.9
27
55.6
47
39.3
81
2 Blocks ( K=2B )
53.3
54
31.2
95
22.3
159
Appendix
Table 6: Impact of generating multiple blocks. Comparison of Pass@1 (%) and throughput (TPS) on HumanEval when decoding one versus two blocks in parallel. For a given block size B , generating two blocks doubles the total parallelism ( K=2B ).
Block Size ( B=K )
Dream-Coder-7B
CoDiLA
2
0.9998
0.9186
4
0.9999
0.9021
8
0.9994
0.8344
Appendix
Table 7: Any-order generation behavior. Spearman correlation between the model’s empirical sampling order and a strict left-to-right sequence on MBPP. Lower values indicate stronger non-causal, any-order decoding behavior.
Batch Size
Dream-Coder-7B
CoDiLA
1
74.6
61.4
2
74.4
66.5
4
72.6
70.4
8
71.0
72.8
Appendix
Table 8: Throughput across varying batch sizes. Generation speed (tokens/sec) measured on a single NVIDIA A100-80GB GPU on the HumanEval benchmark using highly parallel decoding ( K=B=8 ).
K=1
K=2
K=4
K=8
K=16
Dream-Instruct-7B
85.1
85.5
63.2
52.6
43.3
CoDiLA
88.6
85.3
68.6
63.1
50.3
Appendix
Table 9: ParallelBench Waiting Line. Comparison of accuracy (%) between Dream-Instruct-7B and CoDiLA acting as a verifier.
Decoding iterations
MGDM (repr.)
CoDiLA ( B=4 )
CoDiLA ( B=8 )
2
94.4
95.3
95.4
4
96.5
96.8
96.9
8
97.8
98.2
98.2
16
99.2
99.3
99.3
20
99.3
99.5
99.4
Appendix
Table 10: Accuracy (%) on graph traversal (planning)
School of Electrical & Computer Engineering Tel Aviv University, Israel · School of Computer Science and AI Tel Aviv University, Israel · Department of Computer Science Technion, Israel Institute of Technology, Israel
The Hong Kong Polytechnic University, Hong Kong, China · ELLIS Institute Tübingen, Tübingen, Germany · Max Planck Institute for Intelligent Systems, Tübingen, Germany +3