Uniform-state discrete diffusion models excel at few-step generation and guidance due to their ability to self-correct, making them preferred over autoregressive or Masked diffusion models in these settings. However, their sampling quality plateaus with ancestral samplers as the number of steps increases. We introduce a family of Predictor-Corrector (PC) samplers for discrete diffusion that generalize prior methods and apply to arbitrary noise processes. When paired with uniform-state diffusion, our samplers outperform ancestral sampling on both language and image modeling, achieving lower generative perplexity at matched unigram entropy on OpenWebText and better FID/IS scores on CIFAR10. Crucially, unlike conventional samplers, our PC methods continue to improve with more sampling steps. Taken together, these findings call into question the assumption that Masked diffusion is the inevitable future of diffusion-based language modeling. Beyond sampling, we develop a memory-efficient curriculum for the Gaussian relaxation training phase, reducing training time by 25% and memory by 33% compared to Duo while maintaining comparable perplexity on OpenWebText and LM1B and strong downstream performance. We release code, checkpoints, and a video-tutorial on: https://s-sahoo.com/duo-ch2
Figures & tables
Figure 1: Performance on Language Modeling and Image Modeling. Ψ -samplers generalize ReMDM ( Wang et al., 2025 ) to arbitrary noise distributions. (Left): Generative perplexity (Gen. PPL; ↓ ) as a function of NFEs, with nucleus sampling p=0.9 . Ψ -samplers consistently improve with more steps, unlike ancestral sampling which plateaus. Curves are annotated with the average unigram entropy per sequence as a proxy for diversity. (Right): On CIFAR-10, Ψ -samplers achieve better FID ( ↓ ) than MDLM (with ReMDM).
Figure 2: Ψ -samplers combine predictor and corrector steps. The predictor transitions from zt to zs via qs∣t , but fails to remask tokens in MDMs. The corrector steps inject noise via qs , to revise earlier predictions. For κt<1 , noise injection enables error correction while preserving the forward process marginals. Our framework extends prior PC methods ( Campbell et al., 2022 ; Gat et al., 2024 ; Wang et al., 2025 ) to arbitrary priors π .
Figure 3: Efficient Curriculum for USDMs. Duo ( Sahoo et al., 2025a ) replaces discrete lookups with linear combinations of all K embeddings: (1) Gaussian diffusion on one-hot representations, (2) Low-temperature {\color[rgb]{0.6055,0,0}\mathrm{softmax}} , (3) weighted sum. Duo++ exploits the sparsity of the tempered softmax (most weights are effectively zero), and simulate the k largest entries (out of K) using ordered statistics. The approximate normalizer {\color[rgb]{0.6055,0,0}\tilde{Z}} admits a closed form expression ( 14 ). Duo++ has a 33% lower memory and 25% faster training than Duo.
Arc-e
Arc-c
HSwag
WinoG
PIQA
MathQA
OQA
AR Transformer
38.55
23.04
30.55
53.19
63.60
22.41
32.4
MDLM †
34.26
24.66
31.54
51.93
57.89
20.70
28.60
Duo
28.11
25.43
26.46
47.20
51.14
20.00
23.40
Duo++ ( k=2 )
27.32
26.11
26.26
49.64
52.12
20.40
27.80
Duo++ ( k=3 )
28.28
25.00
25.89
47.36
50.65
21.01
23.00
Duo++ ( k=5 )
28.03
25.77
26.90
50.12
51.25
20.20
25.40
Table 1: Accuracy on multiple-choice question answering datasets. Abbreviations: Arc-e (ARC-Easy), Arc-c (ARC-Challenge), HSwag (HellaSwag), WinoG (Winogrande), PIQA (Physical Intelligence Question Answering), OQA (OpenBookQA). † Results from Deschenaux et al. (2025) . Duo++ ( k=2 ) achieves slightly higher accuracy than Duo on 4 out of 6 tasks. Overall, Duo++ matches Duo’s performance while using 25% fewer flops . The highest accuracy among USDMs is bolded . The absolute best per column is underlined .
Figure 4: Illustration of the possible evolution of t and the associated κt . In practice, we use κt close to 1 during the PC phase.
LM1B
OWT
Autoregressive
Transformer †
22.3
17.5
Masked Diffusion
SEDD Absorb †
32.7
24.1
MDLM †
27.0
23.2
Uniform-state Diffusion
Table 2: Test perplexity (PPL) on LM1B and OWT. Lower is better. † Results from Sahoo et al. (2025a) . Best Uniform-state diffusion numbers are bolded . Duo and Duo++ achieve comparable performance across both datasets while requiring 25% fewer GPU-hours (Table 4 ), demonstrating the effectiveness of our memory-efficient curriculum.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Polynomial approximation and approximation error, compared to the series approximation, truncated at 150 terms. The degree- 9 polynomial (left) achieves orders of magnitude lower error than the degree- 5 polynomial (center) and sigmoid (right) approximations.
Component
Value
Vocab size
256
Number of ResNet blocks per scale
2
Base channels
128
Channel multiplier per scale
(1,2,2,2)
Attention resolutions
16
Conditional embedding dimension
128
Appendix
Table 3: Model architecture on CIFAR10
Figure 6: Ψ -samplers, which generalize ReMDM, significantly improve the Inception Score on CIFAR-10, compared to ancestral sampling.
Method
Throughput
Peak Memory
(samples/s) ↑
(GiB) ↓
Duo
81.8
94.3
Duo (after the CL)
122.4
63.3
Duo++ ( k∈{2,3,5} )
121.9
63.4
Appendix
Table 4: Training efficiency comparison between Duo and Duo++ on 138M parameter models. All measurements are conducted on a training job on 8 NVIDIA GH200-120GB GPU with batch size 32. We report the average throughput in sequence per second. The row “Duo (after CL)” denotes the resources consumption of Duo after the Curriculum phase. The impact of k is minimal when k∈{2,3,5} , and Duo++ uses similar resources.
PTB
Wiki
LM1B
LBD
AG News
PubMed
ArXiv
Autoregressive
Transformer †
82.05
25.75
51.25
51.28
52.09
49.01
41.73
Diffusion (138M)
SEDD Uniform †
105.51
41.10
82.62
57.29
82.64
55.89
50.86
UDLM †
112.82
39.42
77.59
53.57
80.96
50.98
44.08
Duo †
89.35
33.57
73.86
49.78
67.81
44.48
40.39
Appendix
Table 5: Zero-shot perplexity (PPL) on seven datasets. Lower is better. † Results taken from Sahoo et al. (2025a) . Duo++ ( k=2 ) achieves a slightly lower zero-shot perplexity than Duo on 6 of 7 datasets.
Figure 7: Marginal distributions of the top-5 entries using a tokenizer with 100k tokens, inverse temperature 100, and log signal-to-noise ratio −2 . The histograms of the efficient and naive implementation match closely.
Figure 8: Marginal distributions of the top-5 entries using a tokenizer with 100k tokens, inverse temperature 1000, and log signal-to-noise ratio −1 . The histograms of the efficient and naive implementation match closely.
Figure 9: Marginal distributions of the top-5 entries using a tokenizer with 100k tokens, inverse temperature 1000, and log signal-to-noise ratio −2 . The histograms of the efficient and naive implementation match closely.
Figure 10: Marginal distributions of the top-5 entries using a tokenizer with 100k tokens, inverse temperature 1000, and log signal-to-noise ratio −4 . The histograms of the efficient and naive implementation match closely.
Figure 11: Marginal distributions of the top-5 entries using the GPT-2 tokenizer, inverse temperature 100, and log signal-to-noise ratio −2 . The histograms of the efficient and naive implementation match closely.
Figure 12: Marginal distributions of the top-5 entries using the GPT-2 tokenizer, inverse temperature 1000, and log signal-to-noise ratio −1 . The histograms of the efficient and naive implementation match closely.
Figure 13: Marginal distributions of the top-5 entries using the GPT-2 tokenizer, inverse temperature 1000, and log signal-to-noise ratio −2 . The histograms of the efficient and naive implementation match closely.
Figure 14: Marginal distributions of the top-5 entries using the GPT-2 tokenizer, inverse temperature 1000, and log signal-to-noise ratio −4 . The histograms of the efficient and naive implementation match closely.
Scheduler
Time
PPL ↓
FID ↓ (Cosine)
FID ↓ (Log-linear)
64
256
1024
2048
64
256
1024
2048
MDLM
Cosine
✗
8.86
42.60
27.71
24.90
24.56
107.62
40.81
27.65
25.73
Cosine
✓
8.72
41.89
27.03
24.67
24.24
114.56
40.60
27.08
25.50
Log-linear
✗
8.76
43.95
29.01
26.11
25.67
111.77
42.15
28.85
26.89
Log-linear
✓
8.75
49.36
32.10
28.76
28.21
122.70
41.79
27.89
26.02
Appendix
Table 6: FID on CIFAR-10 with ancestral sampling. We train and sample with the log-linear and cosine scheduler. MDLM performs best with time-conditioning while Duo does not. We sample with discrete classifier-free guidance ( Schiff et al., 2025 ) with strength 1 , and greedy predictions on the last step.
Algo
Train
Sample
p
FID ↓
32
64
128
256
512
1024
2048
4096
Duo
log-lin
log-lin
1.0
42.71
33.44
29.18
27.46
26.62
26.08
25.87
25.79
Duo
log-lin
log-lin
0.9
28.53
24.24
22.89
22.41
22.56
22.35
22.54
22.41
Duo
log-lin
cos
1.0
39.65
31.78
28.55
27.03
26.03
25.89
25.75
25.63
Duo
log-lin
cos
0.9
25.96
23.13
22.68
22.21
22.26
22.58
22.49
22.49
MDLM
cos
log-lin
1.0
212.95
114.56
62.86
40.60
31.05
27.08
25.50
24.73
Appendix
Table 7: FID on CIFAR-10 with ancestral sampling and a finer grid. We pick the variant with the best FID from Table 6 .
Number of steps
32
64
128
256
512
1024
2048
4096
ReMDM cap (p=1.0)
η=0.005
215.67
116.24
63.37
40.82
31.40
27.28
24.97
24.78
η=0.010
218.41
118.25
64.50
41.77
32.40
28.68
27.91
33.68
η=0.020
224.20
122.61
66.95
44.54
36.26
35.39
46.01
92.48
η=0.050
242.25
143.21
84.41
64.10
73.89
132.13
210.60
203.14
Appendix
Table 8: FID on CIFAR-10 with ReMDM (best checkpoints, as shown in Table 7 ). We sample with/without nucleus sampling, and with the 3 schedules of Wang et al. (2025) (cap, loop, rescale). For the loop schedule, we use ton=0.55 , toff=0.05 , αon=0.9 , following ReMDM. Sampling experiments are executed in the original codebase of Wang et al. (2025) .
Algo
κ
Train
Sample
ton
toff
FID ↓
32
64
128
256
512
1024
2048
4096
Duo
0.02
log-lin
cos
0.2
0.15
40.64
33.06
30.36
29.85
31.31
34.36
39.06
38.38
Duo
0.02
log-lin
cos
0.5
0.45
41.81
33.67
29.50
26.55
24.83
25.12
31.63
51.83
Duo
0.02
log-lin
cos
0.8
0.7
43.99
37.41
35.68
38.88
46.76
59.68
75.46
91.73
Duo
0.5
log-lin
cos
0.2
0.1
39.95
32.14
28.86
27.18
26.57
26.46
27.29
28.35
Duo
0.5
log-lin
cos
0.6
0.4
39.54
29.40
23.46
20.77
23.72
38.42
72.97
105.75
Appendix
Table 9: FID on CIFAR-10 with Ψ -samplers, where Ψ -samplers are activated for steps with t∈[toff,ton] , when κt is kept constant (according to the κ column, 1 otherwise). We use the same checkpoints as in Table 7 . Using a cosine sampling schedule and light noise injection ( κ close to 1) generally perform best. The CIFAR-10 curves in Fig. 1 show the best FID per number of steps.
Algo
κ
Train
Sample
ton
toff
Inception Score ↑
32
64
128
256
512
1024
2048
4096
Duo
0.02
log-lin
cos
0.2
0.15
7.02
7.25
7.35
7.48
7.52
7.47
7.38
7.63
Duo
0.02
log-lin
cos
0.5
0.45
7.09
7.44
7.64
8.04
8.32
8.59
8.57
7.94
Duo
0.02
log-lin
cos
0.8
0.7
6.84
6.99
7.00
6.91
6.64
6.16
5.67
5.19
Duo
0.5
log-lin
cos
0.2
0.1
6.96
7.21
7.28
7.39
7.45
7.48
7.56
7.73
Duo
0.5
log-lin
cos
0.6
0.4
7.31
7.73
8.14
8.51
8.46
7.91
6.40
5.39
Appendix
Table 10: Inception Score on CIFAR-10 with Ψ -samplers, where Ψ -samplers are activated for steps with t∈[toff,ton] , when κt is kept constant (according to the κ column, 1 otherwise). We use the same checkpoints as in Table 7 . The CIFAR-10 curves in Fig. 6 show the best Inception Score per number of steps.
Algo
Train
Sample
FID ↓
32
64
128
256
512
1024
2048
4096
Duo with the ReMDM rescale schedule
Duo
log-lin
cos
39.64
32.03
28.49
26.95
26.16
25.71
25.25
25.02
Duo
log-lin
log-lin
42.27
33.58
29.49
27.36
26.33
25.86
25.07
25.21
ReMDM Rescale ( η=0.01 )
MDLM
cos
cos
70.64
41.94
31.60
27.31
25.27
24.61
23.41
23.25
Appendix
Table 11: FID on CIFAR-10 using Ψ -samplers whose κt schedulers are equivalent to ReMDM. We use no nucleus sampling, no temperature scaling, and cfg=1 . As expected, with the log-linear scheduler, we reach a similar FID as when using the ReMDM codebase (Table 8 ). However, note that by using a log-linear scheduler, using a constant κt=0.99 , we reach a better FID than with the original ReMDM scheduler.
Algo
Dist.
p
Sched.
Gen. PPL ( ↓ )
32
64
128
256
512
1024
2048
4096
Duo
✗
1.0
cos
87.23 (5.54)
79.94 (5.55)
75.87 (5.53)
73.95 (5.54)
72.13 (5.54)
71.41 (5.53)
72.29 (5.53)
70.77 (5.52)
Duo
✗
1.0
log-lin
96.76 (5.57)
86.01 (5.56)
79.97 (5.55)
78.46 (5.53)
76.93 (5.54)
75.02 (5.53)
75.65 (5.52)
75.39 (5.52)
Duo
✗
0.9
cos
42.42 (5.36)
39.26 (5.37)
37.62 (5.35)
36.52 (5.35)
35.21 (5.34)
35.37 (5.34)
35.39 (5.34)
34.91 (5.33)
Duo
✗
0.9
log-lin
44.24 (5.40)
40.08 (5.40)
37.93 (5.39)
36.66 (5.37)
35.77 (5.37)
34.79 (5.35)
34.93 (5.35)
34.75 (5.35)
Duo
✓
1.0
cos
67.04 (5.47)
61.09 (5.45)
59.65 (5.42)
57.76 (5.42)
57.90 (5.42)
56.81 (5.43)
56.39 (5.41)
57.32 (5.42)
Appendix
Table 12: Generative Perplexity (Gen. PPL) and Unigram Entropy on OpenWebText ( Gokaslan and Cohen, 2019 ) with ancestral sampling (no nucleus, no temperature scaling). We train using the log-linear noise scheduler, and sampling with the cosine scheduler is slightly better. We stick to the log-linear schedule for sampling in further experiments, to follow prior work, and since the cosine schedule only marginally reduces the Gen. PPL.
Algo
Eta
Nucleus P
Gen. PPL ( ↓ )
32
64
128
256
512
1024
2048
4096
Ancestral Sampling
Duo
N.A
1.0
96.76 (5.57)
86.01 (5.56)
79.97 (5.55)
78.46 (5.53)
76.93 (5.54)
75.02 (5.53)
75.65 (5.52)
75.39 (5.52)
Duo
N.A
0.95
56.65 (5.49)
50.78 (5.48)
48.68 (5.48)
47.26 (5.46)
45.42 (5.45)
45.11 (5.44)
45.12 (5.44)
44.84 (5.44)
Duo
N.A
0.9
44.24 (5.40)
40.08 (5.40)
37.93 (5.39)
36.66 (5.37)
35.77 (5.37)
34.79 (5.35)
34.93 (5.35)
34.75 (5.35)
MDLM
N.A
1.0
194.09 (5.74)
141.67 (5.69)
120.95 (5.67)
111.85 (5.65)
107.89 (5.64)
105.64 (5.64)
105.40 (5.63)
105.03 (5.62)
Appendix
Table 13: Generative Perplexity (Gen. PPL) and Unigram Entropy on OpenWebText ( Gokaslan and Cohen, 2019 ) with Ψ -samplers using κt schedules matching ReMDM (log-linear step size) and non-distilled models (as in Table 12 ). We experiment with nucleus sampling, following Wang et al. (2025) . The rescale schedule is most effective to improve the Gen. PPL while retaining the unigram entropy. The lightblue rows are the ones plotted in Fig. 1 (left).
Algo
Eta
Nucleus P
Gen. PPLL ( ↓ )
32
64
128
256
512
1024
2048
4096
Ancestral Sampling
Duo
N.A
1.0
68.35 (5.54)
62.92 (5.54)
59.82 (5.50)
58.77 (5.46)
58.32 (5.46)
57.82 (5.45)
55.39 (5.43)
55.89 (5.42)
Duo
N.A
0.95
44.94 (5.47)
41.78 (5.46)
40.32 (5.43)
38.93 (5.39)
38.69 (5.37)
38.45 (5.36)
36.92 (5.33)
37.26 (5.33)
Duo
N.A
0.9
35.92 (5.41)
32.98 (5.40)
31.49 (5.36)
30.32 (5.31)
30.06 (5.29)
30.00 (5.28)
28.90 (5.25)
29.19 (5.25)
MDLM
N.A
1.0
68.61 (5.48)
55.26 (5.45)
49.51 (5.44)
46.13 (5.42)
45.61 (5.42)
44.87 (5.42)
44.53 (5.41)
44.38 (5.42)
Appendix
Table 14: Generative Perplexity (Gen. PPL) and Unigram Entropy on OpenWebText ( Gokaslan and Cohen, 2019 ) with Ψ -samplers using κt schedules matching ReMDM (log-linear step size) and distilled models (as in Table 12 ). We experiment with nucleus sampling, following Wang et al. (2025) .
Is the uniform-state diffusion framework a more powerful paradigm for discrete diffusion? Recent studies indicate that this may be the case. In combination with predictor-corrector samplers, uniform-state diffusion models (USDMs) produce samples of higher-quality than masked diffusion models (MDMs), and USDMs equal or outperform MDMs in downstream tasks, even though they exhibit greater perplexity. Two issues remain unresolved. First, existing work compares uniform and masked diffusion with un-informed correctors that re-inject noise at random positions, rather than targeting tokens most likely to be wrong. Second, prior work compares full-sequence diffusion models, so we do not know whether the same conclusion holds when tokens are generated block by block. To address these issues, we introduce BlockGen, a blockwise sequence model that we instantiate with both masked and uniform diffusion. BlockGen trains on a mixture of block sizes and its likelihood interpolates between AR and pure diffusion more finely than models with a fixed block size. BlockGen enables AR-informed predictor-corrector sampling (ARPC), which combines AR and diffusion predictions to re-generate unlikely tokens without an auxiliary verifier. Under ancestral sampling, uniform outperforms masked in the block-by-block setting, especially in the few-step regime. Under ARPC, the gap closes and reverses at high NFE. With block size 16 on GSM8K, MDMs reach slightly higher accuracy than USDMs, and we observe a similar trend in Generative Perplexity on OpenWebText. Find our code at https://github.com/jdeschena/blockgen.
Justin Deschenaux, Caglar Gulcehre
EPFL Lausanne, Switzerland · EPFL, Lausanne, Switzerland · Microsoft AI
Discrete diffusion language models (dLLMs) provide a fast and flexible alternative to autoregressive models (ARMs) via iterative denoising with parallel updates. However, their evaluation is challenging: existing metrics conflate denoiser approximation error with sampler-induced error from the sampling dynamics, a problem that does not arise for ARMs whose autoregressive sampling exactly reflects the learned probability model. We introduce a sampler-centric oracle framework that replaces learned denoisers with an exact Hidden Markov Model posterior derived from a ground-truth Markov chain, isolating sampler-induced error in a controlled setting. We show that few-step discrete diffusion samplers are not distributionally correct even under an oracle denoiser, with transition-level mismatch that vanishes only as the number of steps approaches the sequence length. Moreover, improvements in negative log-likelihood (NLL), generative perplexity (GenPPL), or MAUVE do not imply correct sampling. Code is available at https://luhantang.github.io/dllm_sampler
Diffusion language models are a promising alternative to autoregressive models due to their potential for faster generation. Among discrete diffusion approaches, Masked diffusion currently dominates, largely driven by strong perplexity on language modeling benchmarks. In this work, we present the first scaling law study of uniform-state and interpolating discrete diffusion methods. We also show that Masked diffusion models can be made approximately 12% more FLOPs-efficient when trained with a simple cross-entropy objective. We find that perplexity is informative within a diffusion family but can be misleading across families, where models with worse likelihood scaling may be preferable due to faster and more practical sampling, as reflected by the speed-quality Pareto frontier. These results challenge the view that Masked diffusion is categorically the future of diffusion language modeling and that perplexity alone suffices for cross-algorithm comparison. Scaling all methods to 1.7B parameters, we show that uniform-state diffusion remains competitive on likelihood-based benchmarks and outperforms autoregressive and Masked diffusion models on GSM8K, despite worse validation perplexity. We provide the code, model checkpoints, and video tutorials on the project page: http://s-sahoo.github.io/scaling-dllms
Subham Sekhar Sahoo, Jean-Marie Lemercier, Zhihan Yang +4
NVIDIA, Santa Clara, USA · Department of Computer Science, Cornell Tech, NYC, USA · Department of Computer Science, Cornell University, Ithaca, USA +2