Less Uniform Discrete Diffusion is More Powerful and Scalable

Kaibo Wang; Ding Ding; Fangyu Ding; Zijin Feng; Han Shi; Haili Bai; Jiacheng Sun; Yang Xiang

cs.CLSep 21, 2026

Less Uniform Discrete Diffusion is More Powerful and Scalable

Authors: Kaibo Wang, Ding Ding, Fangyu Ding, Zijin Feng, Han Shi, Haili Bai, Jiacheng Sun, Yang Xiang

Organizations: The Hong Kong University of Science and Technology · Huawei Foundation Model Department

Abstract

Although uniform diffusion language models (UDLMs) represent a promising diffusion paradigm, scaling them remains challenging. We identify the core obstacle as an over-uniform training objective and condition-target confusion during sampling. To address these, we propose Less Uniform Diffusion (LUDI), a novel UDLM framework. Specifically, we (i) introduce a less uniform loss that directs each reverse transition toward the clean token, and (ii) equip the model with per-token time embeddings that supply token-level corruption hints, enabling confidence-based few-step sampling. Experiments across scales show that LUDI yields cleaner supervision and improves few-step generation. We further continue-train a 7B autoregressive model into LUDI-7B, resulting in a UDLM capable of complex reasoning. It achieves a 3-token-per-step speedup over AR decoding and competitive performance compared with masked diffusion baselines, revealing that the full potential of UDLMs for complex generation remains to be unlocked.

Figures & tables

Panel of Figure 1 — Figure 1: Schematic comparison between masked and uniform diffusion.

Panel of Figure 2 — Figure 2: Illustration of Less Uniform Diffusion (LUDI). (a) LU Loss. The ELBO loss ( left ) matches the reverse transition to $p(\mathbf{x}_{t-\Delta t}\mid\mathbf{x}_{t},\mathbf{x}_{0})$ , leading to over-uniform predictions by encouraging all token probabilities ( $\sum_{i}$ , ➜ ). Our LU loss ( right ) removes the smoothing term and retains only the suppression of the erroneous token ( $m$ ➜ ) and promotion of the target token ( $y$ ➜ ). (b) Per-token time embedding. The model receives token-level corruption hints via per-token time embeddings. Combined with the LU loss, this allows scaling LUDI to 7B parameters from AR initialization, enabling confidence-based decoding for complex reasoning tasks.

Task	200K Steps				500K Steps
Task	LUDI	SDDLM	SDDLM-v1	Duo	LUDI	SDDLM	SDDLM-v1	Duo
PIQA	60.12	57.40	55.33	58.11	62.19	61.15	60.28	61.37
SIQA	36.13	35.11	34.70	35.67	36.54	37.26	35.47	36.03
ARC-E	42.46	34.74	31.93	34.04	50.35	50.18	48.60	48.07
HellaSwag	34.42	30.85	30.05	31.82	35.31	34.22	33.31	35.28
OBQA	26.60	25.80	24.80	26.40	28.60	29.80	27.20	28.00

Table 2: Zero-shot downstream evaluation of 1B models at 200k and 500k pretraining checkpoints. Full evaluation details are provided in Appendix C.1 .

Baselines
Model	Size	Type	HumanEval		MBPP		GSM8K	IFEval	MMLU	GPQA	Avg.
Model	Size	Type	Base	Plus	Base	Plus	GSM8K	IFEval	MMLU	GPQA	Avg.
LLaDA	8B	MDLM	35.4	31.7	31.5	28.6	78.6	59.9	65.5	31.8	45.4
LLaDA-1.5	8B	MDLM	52.4	–	42.8	–	83.3	58.2	66.0	36.9	–
LLaDA-MoE	7B	MDLM	61.6	–	70.0	–	82.4	59.3	67.2	–	–
Dream	7B	AR $\to$ M	57.9	53.7	68.3	56.1	81.0	62.5	67.0	33.0	59.9

Table 3: LUDI-7B results and ablations. Metrics cover code generation (HumanEval, MBPP, HumanEval+, MBPP+), mathematical reasoning (GSM8K), instruction following (IFEval), knowledge-intensive question answering (MMLU, GPQA), and an overall average score (Avg.). Code benchmarks are evaluated using EvalPlus. The highest score in each column is marked in bold , and the second highest is underlined . AR

\to

M/U denotes MDLM/UDLMs continue-trained from AR. In ablation, LUDI is based on Fast dLLM v2 (FD) and incorporates the LU loss, per-token time embedding, and NBDiff (ND) techniques. Full evaluation details are provided in Appendix C.2 .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Context-Causal attention masks for block-diffusion adaptation. (a) Training mask: the prefix uses a lower-triangular causal mask; the active block attends bidirectionally within itself and causally to the prefix. (b) Inference mask: generated blocks form a causal prefix; only the current block is denoised bidirectionally.

	GSM8K		MATH500
Method	Dolci	Dolci-Nemo	Dolci	Dolci-Nemo
SDDLM	74.7	80.8	48.8	51.9
SDDLM-v1	73.5	80.8	48.1	51.9
Fast-dLLM v2	71.3	83.7	51.1	51.7
LUDI	75.2	82.1	49.0	52.6

Appendix

Table 4: Mathematical-reasoning accuracy with Dolci and Dolci-Nemo mixture.

Batch size	Latency (seconds)	Tokens per second	Tokens per step	Peak memory (GB)	Latency (seconds)	Tokens per second	Peak memory (GB)	Throughput speedup
	LUDI				Autoregressive baseline
1	494.46	132.97	3.19	31.05	572.95	69.31	15.35	$1.92\times$
4	371.99	229.20	5.76	31.46	227.70	176.32	15.56	$1.30\times$
16	179.68	477.50	11.92	31.93	86.08	464.84	16.33	$1.03\times$
32	106.97	572.60	14.20	32.95	63.43	629.86	17.39	$0.91\times$

Appendix

Table 5: End-to-end efficiency comparison between LUDI (confidence threshold 0.7) and the autoregressive baseline Qwen2.5-7B-Instruct on GSM8K across batch sizes 1–32. Throughput speedup is computed as LUDI tokens per second divided by autoregressive tokens per second.

Panel of Figure 5 — Figure 5: Computational efficiency simulation in a long-sequence, large-vocabulary setting with batch size 10, sequence length 1024, and vocabulary size 50,000.

Panel of Figure 6 — Figure 6: Generative perplexity over training checkpoints. Lower PPL is better. Lines connect all 10K-spaced checkpoints from 10K to 500K steps.

Panel of Figure 7 — Figure 7: Density of per-sample generative PPL and sample entropy for the four losses at the 500K checkpoint, evaluated with 1024-step sampling and 1024 generated samples per loss. Each panel uses its own axis range and density scale for readability. Darker cells indicate more samples within the corresponding panel, and the white dot marks the method mean.

Diffusion Language Models

Diffusion language models generate and refine text through iterative denoising, enabling bidirectional context and parallel token updates.

See all 466 papers

Looped Uniform Diffusion Language Models
Oct 21, 2026Saiteja Utpala, Chieh-Hsin Lai, Yuki Mitsufuji +1Diffusion Language Models Diffusion Language Model Training
MBZUAI · Sony Group Corporation · MIT
Consistent Diffusion Language Models
Apr 30, 2026Hasan Amin, Yuan Gao, Yaser Souri +4Few-Step Diffusion Sampling Diffusion Model Acceleration
Department of Computer Science, Purdue University · Microsoft
Sumi: Open Uniform Diffusion Language Model from Scratch
Jun 17, 2026Mengyu Ye, Keito Kudo, Wataru Ikeda +3Language Model Pretraining Diffusion Language Models
1Tohoku University

Oct 21, 2026J/K move · Enter open · S save

Looped Uniform Diffusion Language Models

Saiteja Utpala, Chieh-Hsin Lai, Yuki Mitsufuji, Praneeth Vepakomma

MBZUAI · Sony Group Corporation · MIT

Diffusion Language Models Diffusion Language Model Training Diffusion Language Model Inference

Open paper AskPDF arXiv

Uniform diffusion language models (UDLMs) generate text by iteratively denoising sequences of ordinary vocabulary tokens, any of which may be corrupted and subsequently revised. During training, UDLMs must denoise from a context that may itself be unreliable, whereas masked diffusion language models (MDLMs) always condition on a clean context. This challenging training objective makes UDLMs more FLOP-intensive to train than MDLMs, but it yields stronger downstream reasoning performance. Recently, selective looping, which recursively applies a few early Transformer layers within each denoising step, has been shown to improve training efficiency in MDLMs under an iso-FLOP budget. We investigate looping in UDLMs while keeping the training objective and sampler unchanged. We find that looping significantly improves training efficiency: looped UDLMs reach the validation NLL of a non-looped UDLM trained on about 0.5 trillion tokens with up to 2.8x fewer training FLOPs on OpenWebText and 3.3x fewer on FineWeb-Edu. We also find that, at a matched NFE budget, looped UDLMs reach up to 9.8% lower generation perplexity than the non-looped UDLM, with most gains achieved at moderate loop counts. Finally, we study different loop-count distributions and find that sampling smaller loop counts more frequently during training improves inference efficiency: on downstream tasks, this strategy uses 12% fewer loops than uniform loop-count sampling without degrading accuracy.

Abstract

Diffusion Language Models

Explore similar work

Looped Uniform Diffusion Language Models

Consistent Diffusion Language Models

Sumi: Open Uniform Diffusion Language Model from Scratch