Revisiting the 16x Pretraining Compute Gap between Diffusion and Autoregressive Language Models

Guanghan Wang; Bing He; Zhan Shi; Benoit Dumoulin; Hanqing Lu; Volodymyr Kuleshov

cs.LGOct 13, 2026

Revisiting the 16x Pretraining Compute Gap between Diffusion and Autoregressive Language Models

Authors: Guanghan Wang, Bing He, Zhan Shi, Benoit Dumoulin, Hanqing Lu, Volodymyr Kuleshov

Organizations: Cornell University, Cornell Tech · Amazon

Abstract

Diffusion language models (DLMs) have attracted wide attention for their fast, parallel decoding, but they are believed to be far less efficient to train than autoregressive language models (ARMs): a widely cited study reports that a DLM needs roughly 16x the training compute of an ARM to match its likelihood, a figure that has made pretraining DLMs from scratch look prohibitive at scale. We find that the methodology behind this figure has fallen out of step with current DLM practice: it scores the model with its evidence lower bound (ELBO) and trains a full-sequence, decoder-only model. We therefore revisit the scaling laws of DLMs, adopting the exact likelihood under the model's own decoding order, block diffusion training, an encoder-decoder architecture, and a sweep over the block sizes used in practice. Under this new protocol, we study across ten model variants, about 1,700 models in total, and find that pretraining DLMs is far cheaper than previously thought. Concretely, scoring with the exact likelihood reduces the compute multiple to 12.8x; block diffusion training and an encoder-decoder architecture further reduce it to 7.8x at block size 32, and smaller block sizes narrow it to 4.6x at block size 8 and 3.0x at block size 4. We complement these scaling analyses with an empirical rule for the optimal decoder depth of encoder-decoder block diffusion, a held-out compute budget beyond the fitting range at which the gap between variants persists, and downstream benchmark results where likelihood improvements carry over to accuracy. Overall, the 16x figure substantially overstates the pretraining cost gap between DLMs and ARMs, and pretraining DLMs from scratch is far more viable than previously assumed.

Figures & tables

Figure 2: IsoFLOP curves for eight main model variants (DUEL for diffusion variants, exact NLL for the ARM; for E2D2, each point is the best

L_{\rm dec}

at that size). Darker shades mark larger budgets, and stars mark fitted vertices.

Figure 4: The decoder depth given by Eq. 2 is near-optimal for E2D2 at

b{=}8

16

and

32

. Each bar represents one grid point

(C,N)

and spans the decoder fractions

L_{\rm dec}/L

whose DUEL lies within

5

m - nats of the minimum of that grid point’s sweep. A bar is green if the DUEL of the decoder depth given by Eq. 2 , i.e., its prediction multiplied by

L

and rounded to an integer, is within

5

m - nats of the minimum.

	$b{=}4$	$b{=}8$	$b{=}16$	$b{=}32$
MDM	$+0.4\pm 0.1$
BD3LM	$+1.0\pm 0.4$	$+0.9\pm 0.2$	$+0.9\pm 0.2$	$+0.8\pm 0.2$
E2D2	$+1.0\pm 0.5$	$+1.1\pm 0.4$	$+1.0\pm 0.3$	$+1.0\pm 0.2$

Table 3: Difference between DUEL and the ELBO in the eight-benchmark mean accuracy, as mean

\pm

s.d. over the five budgets of each variant. DUEL outperforms the ELBO with statistical significance.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Size	Block size
LLaDA † ( Nie et al., 2026 )	8B	32
Dream † ( Ye et al., 2025 )	7B	32
LLaDA2.0 ( Bie et al., 2025 )	16B, 100B	32
Fast-dLLM v2 ( Wu et al., 2026a )	1.5B, 7B	32
Nemotron-Labs-Diffusion ( Fu et al., 2026 )	3B, 8B, 14B	32
Nemotron-TwoTower ( Reda et al., 2026 )	30B-A3B	16

Appendix

Table 4: Block sizes adopted by large-scale masked diffusion language models. † Trained with full-sequence masking; the block size is the semi-autoregressive block length used at inference (for Dream, the setting chosen by Fast-dLLM ( Wu et al., 2026b ) ).

Figure 7: Attention masks of the four model variants on a six-token sequence with block size

b{=}2

. Rows are queries and columns are keys; filled cells attend. (a) ARM: causal self-attention. (b) MDM: full bidirectional self-attention over the partially masked sequence. (c) BD3LM: a single tower over the concatenation of the clean sequence

\mathbf{x}

and the noised sequence

\mathbf{x}_{t}

; the clean half attends block-causally, and each noised block attends to itself and to the clean tokens of all earlier blocks. (d, e) E2D2: a block-causal encoder over the clean sequence, and a decoder with block-diagonal self-attention within each noised block plus cross-attention to the encoder outputs

h

of the strictly preceding blocks. M marks a masked (noised) token; arrows mark the token predicted at that position.

Size (M)	$d$	$L$	ARM	MDM	BD3LM	E2D2
19	384	8	16.8	28.8	20.3	16.0
34	512	8	14.4	25.1	16.9	13.6
66	640	10	13.2	23.4	14.9	12.4
113	768	12	12.0	21.5	13.2	11.2
170	768	18	12.7	22.6	13.6	11.8
268	1024	16	10.0	18.2	10.7	9.2

Appendix

Table 5: Attention score FLOPs as a percentage of per-token training FLOPs, by variant and model size (

b{=}32

; E2D2 decoder

=L/2

; sequence length

2048

variant	$C$ (FLOPs)	trained sizes $N$
ARM	$6{\times}10^{18}$	$\{66,113,170,231,336,472,629\}$
	$10^{19}$	$\{85,142,231,336,472,666,944\}$
	$3{\times}10^{19}$	$\{142,231,336,472,666,944,1233\}$
	$6{\times}10^{19}$	$\{206,302,472,666,944,1233,1678\}$
	$10^{20}$	$\{268,472,666,944,1233,1678,2121\}$
	$3{\times}10^{20}$ (held-out)	$\{1678,2121,2718\}$

Appendix

Table 6: Ladder sizes

N

(millions of non-embedding parameters) trained at each budget for the ARM, the MDM and BD3LM.

$C$ (FLOPs)	$N$ (M)	$L$	$b{=}4$	$b{=}8$	$b{=}16$	$b{=}32$
$6{\times}10^{18}$	34	8	2–7	2–7	2–7	2–7
	48	9	2–8	2–8	2–8	2–8
	66	10	2–9	2–9	2–9	3–9
	85	13	2–12	4–12	4–12	5–12
	113	12	2–11	4–11	3–11	3–11
	142	15	—	—	6–14	5–14

Appendix

Table 7: E2D2 grid: for each budget and ladder size

N

(with total depth

L

), the range of decoder depths

L_{\rm dec}

trained at each block size. “—” means the size was not trained at that block size.

variant	runs	training	DUEL evaluation	total
ARM	43	1,507	—	1,507
MDM	80	2,393	8,384	10,777
BD3LM $b{=}32$	43	3,901	292	4,193
BD3LM $b{=}16$	43	3,903	147	4,050
BD3LM $b{=}8$	43	3,903	74	3,977
BD3LM $b{=}4$	40	3,504	39	3,543

Appendix

Table 8: Compute cost in B200 GPU hours by variant, split into training and DUEL evaluation. E2D2 rows include the decoder-depth sweep.

Figure 8: IsoFLOP curves for BD3LM and E2D2 at

b{=}4

(DUEL; for E2D2, each point is the best

L_{\rm dec}

at that size). Darker shades mark larger budgets, and stars mark fitted vertices.

Table 13

$b$	4	8	16	32
argmin, all columns	$0.645\pm 0.085$	$0.716\pm 0.040$	$0.728\pm 0.062$	$0.780\pm 0.055$
vertex, all columns	$0.643\pm 0.064$	$0.682\pm 0.029$	$0.723\pm 0.030$	$0.771\pm 0.049$
at compute-optimal $N$	$0.630\pm 0.045$	$0.706\pm 0.048$	$0.724\pm 0.034$	$0.756\pm 0.018$

Appendix

Table 11: Optimal decoder fraction

L^{\ast}_{\rm dec}/L

of E2D2 at each block size, read off in three ways: the discrete argmin of DUEL over all columns, the vertex of a parabola fitted near the minimum over all columns, and the argmin restricted to the compute-optimal

N

of each budget. Entries are mean

\pm

s.d.

Figure 9: Same as Figure 4 , for E2D2 at

b{=}4

$b$	columns	within $5$ m - nats	within $10$ m - nats	compute-optimal columns within $10$ m - nats
4	38	$76\%$	$87\%$	5/5
8	32	$81\%$	$88\%$	5/5
16	36	$89\%$	$97\%$	5/5
32	36	$89\%$	$100\%$	5/5
all	142	$84\%$	$93\%$	20/20

Appendix

Table 12: Rounded Eq. 2 against the column minimum, by block size.

variant	law $\mathcal{L}^{\ast}(3{\times}10^{20})$	measured best cell	miss
ARM	2.2231	2.2468	$+23.7$ m - nats ( $+1.07\%$ )
E2D2 $b{=}16$	2.4800	2.5084	$+28.4$ m - nats ( $+1.15\%$ )
E2D2 $b{=}32$	2.5167	2.5436	$+26.9$ m - nats ( $+1.07\%$ )

Appendix

Table 13: The held-out budget at

3{\times}10^{20}

FLOPs. Power laws fitted on the five budgets

\leq 10^{20}

are compared with the best measured cell.

Figure 10: Pure power laws (dashed) and shared-

E

laws

\mathcal{L}^{\ast}=E+a\,C^{\alpha}

(solid), both fitted on

\leq 10^{20}

, against the held-out budget at

3{\times}10^{20}

FLOPs (stars).

$E$	rms (m - nats)	$\alpha_{\rm ARM}$	$\alpha_{\text{E2D2 }16}$	miss ARM	miss E2D2 16	miss E2D2 32
0.0	5.9	$-0.062$	$-0.063$	$+23.7$	$+28.4$	$+26.9$
1.0	4.1	$-0.101$	$-0.097$	$+9.2$	$+14.2$	$+13.2$
1.2	3.7	$-0.115$	$-0.108$	$+4.1$	$+9.5$	$+8.7$
1.4	3.2	$-0.134$	$-0.122$	$-2.5$	$+3.8$	$+3.2$
1.5	3.1	$-0.147$	$-0.131$	$-6.7$	$+0.4$	$-0.0$
1.6	3.2	$-0.161$	$-0.141$	$-11.5$	$-3.5$	$-3.7$

Appendix

Table 14: Profile of the shared-

E

fit on the five budgets

\leq 10^{20}

: in-sample rms, exponents, and the miss at the held-out budget (measured minus predicted, m - nats).

Figure 11: Benchmark screening. For each benchmark, the accuracy averaged over the ten variants at their compute-optimal cells, relative to its value at

6{\times}10^{18}

FLOPs. Dotted lines mark the three benchmarks we exclude.

variant	$C$	HellaSwag	ARC-e	PIQA	SciQ	RACE	SIQA	OBQA	LAMBADA	mean
ARM	$6{\times}10^{18}$	28.9	35.1	61.1	60.1	27.0	37.0	26.2	16.6	36.5
	$10^{19}$	30.2	37.7	60.8	60.6	28.3	37.0	27.4	19.0	37.6
	$3{\times}10^{19}$	34.1	40.4	63.2	65.9	29.8	37.4	27.2	26.3	40.5
	$6{\times}10^{19}$	37.1	41.8	65.6	67.3	29.1	37.1	29.4	27.2	41.8
	$10^{20}$	40.1	42.9	67.7	72.4	33.3	38.3	30.2	30.8	44.5
MDM	$6{\times}10^{18}$	28.5	32.1	56.2	63.1	27.0	36.7	25.4	21.1	36.3

Appendix

Table 15: Zero-shot accuracy (%) on the eight retained benchmarks at the compute-optimal cell of every variant at every budget (DUEL scoring for the diffusion variants; exact NLL for the ARM), and their mean. This is the data plotted in Figure 6 .

asset	license	asset	license
Code		Data
SMDM ( Nie et al., 2025 )	Apache-2.0	SlimPajama-627B ( Soboleva et al., 2023 )	Apache-2.0
TinyLlama ( Zhang et al., 2024 ) / lit-gpt	Apache-2.0	Benchmarks
nanoGPT ( Karpathy, 2022 )	MIT	HellaSwag ( Zellers et al., 2019 )	MIT
GPT-NeoX ( Black et al., 2022 )	Apache-2.0	ARC-easy ( Clark et al., 2018 )	CC BY-SA 4.0
FlashAttention 2.6.3 ( Dao, 2024 )	BSD-3-Clause	PIQA ( Bisk et al., 2020 )	AFL-3.0

Appendix

Table 16: Third-party assets used in this study and their licenses.

		ARM	MDM		BD3LM $b{=}32$		BD3LM $b{=}16$		BD3LM $b{=}8$		BD3LM $b{=}4$
$C$	$N$	NLL	DUEL	ELBO	DUEL	ELBO	DUEL	ELBO	DUEL	ELBO	DUEL	ELBO
$6{\times}10^{18}$	19	—	—	—	3.4143	3.5110	3.3783	3.4803	3.3263	3.4305	—	—
	34	—	3.3979	3.4465	3.3510	3.4422	3.3074	3.4069	3.2588	3.3601	3.1909	3.2756
	48	—	3.3491	3.4045	3.3057	3.4000	3.2640	3.3654	3.2125	3.3146	3.1442	3.2268
	66	2.8782	3.3355	3.3806	3.2863	3.3809	3.2504	3.3511	3.1987	3.2975	3.1310	3.2150
	85	—	—	—	3.2943	3.3836	3.2611	3.3577	3.1953	3.2954	3.1331	3.2176

Appendix

Table 17: Per-cell results of the IsoFLOP sweeps for the ARM, the MDM and BD3LM.

N

is the nominal model size in millions of non-embedding parameters, as named in Table 6 ; entries are validation NLLs in nats per token (exact for the ARM; DUEL and ELBO for the diffusion variants), and “—” marks a size that was not trained for that variant. The last three ARM rows are the held-out budget of Section 4.3 . Within each budget, the smallest value in each column, i.e. the minimum of each IsoFLOP curve, is in bold.

$N$	$L_{\rm dec}/L$	DUEL	ELBO
$C={6{\times}10^{18}}$
34	2/8	3.5005	3.5703
34	3/8	3.3989	3.4660
34	4/8	3.3500	3.4175
34	5/8	3.3105	3.3794
34	6/8	3.2909	3.3593

Appendix

Table 18: Decoder-depth sweep of E2D2

b{=}32

(part 1 of 2): every trained model, one row per decoder depth.

N

is the nominal model size in millions of non-embedding parameters, as named in Table 6 ; DUEL and ELBO are validation NLLs in nats per token.

L_{\rm dec}/L

is the decoder depth over the total depth. The best

L_{\rm dec}

of each (budget, size) group is in bold. The rows at

3{\times}10^{20}

FLOPs are the held-out budget of Section 4.3 .

$C={10^{20}}$ (cont.)
$N$	$L_{\rm dec}/L$	DUEL	ELBO
666	19/21	2.7170	2.7904
944	16/25	2.7424	2.8160
944	17/25	2.7359	2.8078
944	18/25	2.7294	2.8019
944	19/25	2.7291	2.8018

Appendix

Table 19: Decoder-depth sweep of E2D2

b{=}32

(part 2 of 2): every trained model, one row per decoder depth.

$N$	$L_{\rm dec}/L$	DUEL	ELBO
$C={6{\times}10^{18}}$
34	2/8	3.4250	3.4991
34	3/8	3.3382	3.4073
34	4/8	3.2968	3.3658
34	5/8	3.2738	3.3441
34	6/8	3.2583	3.3294

Appendix

Table 20: Decoder-depth sweep of E2D2

b{=}16

(part 1 of 2): every trained model, one row per decoder depth.

N

is the nominal model size in millions of non-embedding parameters, as named in Table 6 ; DUEL and ELBO are validation NLLs in nats per token.

L_{\rm dec}/L

is the decoder depth over the total depth. The best

L_{\rm dec}

of each (budget, size) group is in bold. The rows at

3{\times}10^{20}

FLOPs are the held-out budget of Section 4.3 .

$C={6{\times}10^{19}}$ (cont.)
$N$	$L_{\rm dec}/L$	DUEL	ELBO
666	13/21	2.7724	2.8466
666	14/21	2.7700	2.8447
666	15/21	2.7675	2.8432
666	16/21	2.7751	2.8503
666	17/21	2.7710	2.8465

Appendix

Table 21: Decoder-depth sweep of E2D2

b{=}16

(part 2 of 2): every trained model, one row per decoder depth.

$C={6{\times}10^{18}}$
$N$	$L_{\rm dec}/L$	DUEL	ELBO
34	2/8	3.3391	3.4108
34	3/8	3.2682	3.3356
34	4/8	3.2355	3.3051
34	5/8	3.2167	3.2879
34	6/8	3.2092	3.2822

Appendix

Table 22: Decoder-depth sweep of E2D2

b{=}8

(part 1 of 2): every trained model, one row per decoder depth.

N

is the nominal model size in millions of non-embedding parameters, as named in Table 6 ; DUEL and ELBO are validation NLLs in nats per token.

L_{\rm dec}/L

is the decoder depth over the total depth. The best

L_{\rm dec}

of each (budget, size) group is in bold.

$N$	$L_{\rm dec}/L$	DUEL	ELBO
$C={10^{20}}$ (cont.)
944	20/25	2.6426	2.7151
944	21/25	2.6398	2.7117
944	22/25	2.6566	2.7302
1233	12/24	2.6782	2.7507
1233	13/24	2.6743	2.7464

Appendix

Table 23: Decoder-depth sweep of E2D2

b{=}8

(part 2 of 2): every trained model, one row per decoder depth.

$N$	$L_{\rm dec}/L$	DUEL	ELBO
$C={6{\times}10^{18}}$
34	2/8	3.2262	3.2826
34	3/8	3.1761	3.2335
34	4/8	3.1530	3.2133
34	5/8	3.1417	3.2042
34	6/8	3.1368	3.2019

Appendix

Table 24: Decoder-depth sweep of E2D2

b{=}4

(part 1 of 3): every trained model, one row per decoder depth.

N

is the nominal model size in millions of non-embedding parameters, as named in Table 6 ; DUEL and ELBO are validation NLLs in nats per token.

L_{\rm dec}/L

is the decoder depth over the total depth. The best

L_{\rm dec}

of each (budget, size) group is in bold.

$C={3{\times}10^{19}}$ (cont.)
$N$	$L_{\rm dec}/L$	DUEL	ELBO
472	8/18	2.7557	2.8136
472	9/18	2.7522	2.8107
472	10/18	2.7473	2.8059
472	11/18	2.7401	2.7998
472	12/18	2.7457	2.8048

Appendix

Table 25: Decoder-depth sweep of E2D2

b{=}4

(part 2 of 3): every trained model, one row per decoder depth.

$N$	$L_{\rm dec}/L$	DUEL	ELBO
$C={10^{20}}$ (cont.)
1233	22/24	2.6232	2.6821
1233	23/24	2.6418	2.7035
1678	4/25	2.6946	2.7491
1678	5/25	2.6936	2.7485
1678	6/25	2.6715	2.7271

Appendix

Table 26: Decoder-depth sweep of E2D2

b{=}4

(part 3 of 3): every trained model, one row per decoder depth.

Diffusion Language Models

Diffusion language models generate and refine text through iterative denoising, enabling bidirectional context and parallel token updates.

See all 473 papers

Oct 3, 2026

ALoDLM: Adaptively Looped Diffusion Language Models

Diffusion language models (DLMs) enable fast generation by predicting multiple tokens in parallel, but their practical adoption remains limited by a persistent quality gap relative to comparably sized autoregressive (AR) models. We attribute this gap to a computation-difficulty mismatch: within a partially observed sequence, some unknown tokens are easy to predict, while others require substantially more computation. Existing DLMs nevertheless apply uniform computational depth to all unknown positions at each denoising step. We introduce ALoDLM, which replaces uniform computation with token-adaptive latent recurrence. At each denoising step, ALoDLM iteratively refines latent representations and allocates computation according to token difficulty. Tokens ready to commit are fed back as discrete context, while unresolved tokens retain and further refine their latent states through additional recurrent passes. To learn token prediction and computation allocation jointly, we formulate token-wise computation schedules as latent variables and derive a conditional negative evidence lower bound (NELBO). We train ALoDLM at 1.7B and 8B parameter scales. Across eleven benchmarks, ALoDLM outperforms all evaluated DLMs and the corresponding AR baselines in average benchmark score at both scales. ALoDLM also retains fast parallel decoding, yielding a strong quality-efficiency trade-off among evaluated autoregressive and diffusion models under optimized inference engines.

Liancheng Fang, Zhuowei Li, Youngeun Kim +10

University of Illinois Chicago · Amazon AGI · Korea University

Adaptive Computation for Diffusion Models Diffusion Language Models Diffusion Language Model Inference Diffusion Language Model Decoding Diffusion Language Model Training Parallel Decoding for Diffusion Language Models

Ask PDF

Mar 23, 2026

Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison

We present a controlled empirical comparison between autoregressive (AR) and masked diffusion (MDLM) language models. Both models are trained on identical data (50M tokens from TinyStories), identical compute budget (20,000 steps, batch size 32, sequence length 512), and identical hardware (NVIDIA H100 80GB), isolating the generation paradigm as the sole variable. We report three findings. First, both paradigms achieve comparable training throughput (~50K tokens/second), with MDLM requiring only 4.7% more wall-clock time. Second, AR converges faster and begins overfitting by step 14,000, while MDLM converges more slowly and is still improving at step 20,000, suggesting different compute-optimal training regimes. Third, quantitative diversity analysis over 1,000 generated samples reveals a structural diversity-fluency trade-off: AR produces fluent but repetitive outputs (99.8% begin with the same word), while MDLM generates more diverse narratives (93.4% unique 5-word openings, higher Distinct-n, lower Self-BLEU), at the cost of occasional grammatical inconsistencies. All code, trained checkpoints, and data pipelines are released for reproducibility.

Caio Vicentino

Independent Researcher

Diffusion Language Models Autoregressive Language Modeling Diffusion Language Model Benchmarking LLM Training Diffusion Language Model Training Language Model Generation Evaluation

Ask PDF

May 7, 2026

Don't Retrain, Align: Adapting Autoregressive LMs to Diffusion LMs via Representation Alignment

Diffusion language models (DLMs) have recently demonstrated capabilities that complement standard autoregressive (AR) models, particularly in non-sequential generation and bidirectional editing. Although recent work has shown that pretrained autoregressive checkpoints can be converted into diffusion language models, existing recipes primarily transfer parameters through continued denoising training with objective- and attention-level modifications. We instead ask whether the internal representation geometry learned by next-token prediction can be explicitly preserved during AR-to-DLM conversion. We hypothesize that much of the semantic structure learned by AR pretraining can transfer across generation orders, and thus DLM training should be viewed as relearning the decoding path rather than relearning language representations. To investigate this, we introduce REPR-ALIGN, a representation alignment objective that adapts a bidirectional masked diffusion model to reuse representations from a pretrained AR model of identical architecture. Concretely, we align the hidden states of the DLM to the frozen AR model at every layer using cosine similarity, while optimizing the standard masked denoising objective. This simple alignment, with no adapters and no architectural changes beyond the attention mask, yields up to 4x training acceleration in our setting and is particularly effective in low-data regimes. Our results suggest that linguistic representations can transfer across generation order, and that representation alignment provides a simple and effective technique for training diffusion language models. Code is available at https://github.com/pengzhangzhi/Open-dLLM.

Fred Zhangzhi Peng, Alexis Fox, Anru R. Zhang +1

Duke University · AITHYRA

Masked Diffusion Models Diffusion Language Models Representation Geometry in Language Models Representation Alignment Diffusion Language Model Training

Ask PDF

Abstract

Diffusion Language Models

Explore similar work

ALoDLM: Adaptively Looped Diffusion Language Models

Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison

Don't Retrain, Align: Adapting Autoregressive LMs to Diffusion LMs via Representation Alignment